[mvapich-discuss] MVAPICH2 Invalid communicator errors

[Tom Mitchell]

On Sep 12 03:41, Mark Potts wrote:
> Hi,
>    Taking my first steps with MVAPICH2-0.9.8 from OFED 1.2.
> 
>    I am able to build and run apps using gcc to build MVAPICH2 and apps.
> 
>    I am also (apparently) successfully building MVAPICH2 and simple C
>    code apps. with an icc Intel 9.1 compiler.  However, any
>    built-with-Intel app that I attempt to run fails with an output
>    similar to that shown at the bottom.  The same apps build and run
>    quite nicely using gcc.  The failure occurs at the very beginning
>    of the pgm, after:

Are you compiling MVAPICH2 with one compiler and the application
with another?  In general you do not want to do this.

When building any MPI the helper scripts (mpicc and friends)
have specific knowledge of the compiler and libs.  After
building the binary, it can be important to ensure that paths
(both PATH and LD_LIBRARY_PATH etc. (csh path .vs. PATH too))
find all the correct bits first.   These environment variables
are important both on the command line and also on compute
nodes including localhost.

When you get to Fortran there is a risk of subtle binary
incompatibility issues to confound you if you are mixing
compilers and other common libs.  So always start with
all the bits being compiled with a single compiler suite.




> 
>       MPI_Init (&argc, &argv);
> 
>    and before the completion of the next statement:
> 
>       MPI_Comm_rank (MPI_COMM_WORLD, &rank);
> 
>    Since we need both gcc and Intel versions of MVAPICH2, I would
>    appreciate any help understanding what is going wrong during
>    startup of these apps.
> 
>    The output from mpich2version is:
> Version:           MVAPICH2-0.9.8
> Device:            osu_ch3:mrail
> Configure Options: --prefix=/usr/mpi/mvapich2-0.9.8-12/intel 
> --with-device=osu_ch3:mrail --with-rdma=gen2 --with-pm=mpd 
> --enable-romio --enable-sharedlibs=gcc --without-mpe
> 
> 
>   Typical failed mpiexec output:
> 
>  mpiexec -machinefile hostdews -n 3 ./mpisanity_mv2
> Fatal error in MPI_Comm_rank: Invalid communicator, error stack:
> MPI_Comm_rank(105): MPI_Comm_rank(comm=0x1, rank=0x7fff4fb01cd8) failed
> MPI_Comm_rank(64).: Invalid communicatorrank 2 in job 1  dewberry4_32925 
>   caused collective abort of all ranks
>   exit status of rank 2: killed by signal 9
> Fatal error in MPI_Comm_rank: Invalid communicator, error stack:
> MPI_Comm_rank(105): MPI_Comm_rank(comm=0x1, rank=0x7fffddc500c8) failed
> MPI_Comm_rank(64).: Invalid communicatorrank 1 in job 1  dewberry4_32925 
>   caused collective abort of all ranks
>   exit status of rank 1: killed by signal 9
> Fatal error in MPI_Comm_rank: Invalid communicator, error stack:
> MPI_Comm_rank(105): MPI_Comm_rank(comm=0x1, rank=0x7fff13581138) failed
> MPI_Comm_rank(64).: Invalid communicatorrank 0 in job 1  dewberry4_32925 
>   caused collective abort of all ranks
>   exit status of rank 0: killed by signal 9
> 
>    regards,
> -- 
> ***********************************
> >> Mark J. Potts, PhD
> >>
> >> HPC Applications Inc.
> >> phone: 410-992-8360 Bus
> >>        410-313-9318 Home
> >>        443-418-4375 Cell
> >> email: potts at hpcapplications.com
> >>        potts at excray.com
> ***********************************
> _______________________________________________
> mvapich-discuss mailing list
> mvapich-discuss at cse.ohio-state.edu
> http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss

-- 
	T o m   M i t c h e l l
	Host Solutions Group, QLogic Corp.  
	http://www.qlogic.com   http://support.qlogic.com