[mvapich-discuss] MVAPICH2 Invalid communicator errors

[Nathan Dauchy]

Shaun Rowland wrote:
> Nathan Dauchy wrote:
>> We have also run into a very similar sounding problem, with
>> mvapich2-0.9.8-2007.08.30 and intel-9.1.
>>
>> mpiexec -np 54 /rt1/rtruc/13km_wjet/exec/hybcst_sp
>> Fatal error in MPI_Comm_rank: Invalid communicator, error stack:
>> MPI_Comm_rank(105): MPI_Comm_rank(comm=0x5b, rank=0x7fbfffc898) failed
>> MPI_Comm_rank(64).: Invalid communicatorFatal error in MPI_Comm_rank:
>> Invalid communicator, error stack:
>>
>> Unfortunately, I haven't found an /usr/include/mpi.h file or other quick
>> fix yet.
>>
>> Is there supposed to be "-I/usr/include" in the output of "mpicc -show"?
>>   Perhaps something went wrong in the build process?  Here is the output
>> on the system with the Invalid communicator errors:
>>
>> $ mpicc -show
>> icc -D_EM64T_ -D_SMP_ -DUSE_HEADER_CACHING -DONE_SIDED
>> -DMPID_USE_SEQUENCE_NUMBERS -D_SHMEM_COLL_ -I/usr/include -O2
>> -I/opt/mvapich/2-0.9.8-2007.08.30/include
>> -L/opt/mvapich/2-0.9.8-2007.08.30/lib -lmpich -L/usr/lib64 -libverbs
>> -libumad -lpthread
>>
>> Whereas another system using mvapich-0.9.9 works fine and does not have
>> "-I/usr/include":
> 
> Hmm... if you take the /usr/include out of the MVAPICH2 mpicc script,
> does it then work?  It might not strictly be mpi.h.

I'll try this tomorrow, but I don't think it will make a difference
since the application is almost entirely fortran, and mpif90 doesn't
seem to have the /usr/include part.

$ mpif90 -show
ifort -O2 -I/opt/mvapich/2-0.9.8-2007.08.30/include
-I/opt/mvapich/2-0.9.8-2007.08.30/include
-L/opt/mvapich/2-0.9.8-2007.08.30/lib -lmpichf90 -lmpichf90 -lmpich
-L/usr/lib64 -libverbs -libumad -lpthread

Also, please note that our system has no "mpi.h" anywhere in
/usr/include, so I agree that something else is causing the problem.

Thanks,
Nathan