[mvapich-discuss] MVAPICH2 Invalid communicator errors

[Shaun Rowland]

Mark Potts wrote:
> Hi,
>    Our build guru here pointed out that the /usr/include that
>    I find in the line in the MVAPICH2 mpicc script:
>    MPI_CFLAGS="-D_EM64T_ -D_SMP_ -DUSE_HEADER_CACHING  -DONE_SIDED 
> -DMPID_USE_SEQUENCE_NUMBERS -D_SHMEM_COLL_ -DRDMA_CM   -I/usr/i nclude -O2"
>    is _not_ a result of any changes here.  The same line occurs
>    in our MVAPICH2 gcc and icc mpicc scripts.  Unfortunately, MPI_CFLAGS
>    goes into the icc command line first, before the MVAPICH2 include
>    directory reference.
> 
>    The injection of the MPI_CFLAGS definition in the mpicc script
>    apparently occurs as a result of the MVAPICH2 build sequence.
>    There is no similar inclusion of a "-I/usr/include" in the
>    MVAPICH-0.9.9 mpicc script(s).

It happens because the CFLAGS used to build MVPAICH2 end up being
captured, and these include the -I/usr/include, I assume because you
used the /usr prefix install for OFED 1.2 (which is the default, but
wasn't at first).

>    I'm also puzzled now why this seeming error in the MVAPICH2 mpicc
>    script doesn't cause problems for gcc- as well as icc-built
>    codes.

I thought it was causing a problem for icc? If you are seeing a
situation where gcc is working with the same mpicc script, and icc
fails, I believe it is because gcc will ignore -I paths that are already
defaults, of which /usr/include is one. Therefore, with gcc, the
inclusion of -I/usr/include before the MVAPICH2 install path makes no
difference because it is silently ignored (this is according to the gcc
manual page). I am guessing icc does not do that.

I am guessing that /usr/include/mpi.h is from the lam package - if you
are running a Red Hat based system. You can see with:

rpm -q -f /usr/include/mpi.h

If this is from the lam RPM, then I can try to exactly duplicate it (we
have at least one system with the same, but we never use /usr as the
prefix for test builds).

>    I now know how to manually fix the problem but I would appreciate
>    some more input from Shaun Rowland or other developer at OSU about
>    this issue.  If /usr/include should be in mpicc, then it must be
>    later in the command line.  Probably, it should not even be in the
>    script.

Technically, it does not need to be there - since that's a standard
search path. I am not sure how easy that is to fix yet. It might be
simple. I think there's still a potential issue if the mpi.h files don't
set the same header include related defines possibly - the ones that
keep a file from being included twice.

I'll take a look at this and get back to you.
-- 
Shaun Rowland	rowland at cse.ohio-state.edu
http://www.cse.ohio-state.edu/~rowland/