[mvapich-discuss] mvapich2-1.0.3 + charm++ error

[Matthew Koop]

Sangamesh,

In the past I have given these instructions for MVAPICH:
http://mail.cse.ohio-state.edu/pipermail/mvapich-discuss/2006-June/000189.html

MVAPICH2 should be the same with the exception of mpiCC for mpicxx. The
settings you have below look fine to me. If you have the proper wrapper
scripts (mpicc, etc) there should be no need to specify the OFED libraries
there.

Can you verify that 'which mpicc' and 'which mpicxx' do point to the
working MVAPICH2 installation?

Thanks,

Matt

On Mon, 4 Aug 2008, Sangamesh B wrote:

> Hi,
>
>     I've installed mvapich2-1.0.3 (make.mvapich2.ofa stack) with Intel
> compilers on Cent OS on a Rocks 4.3 cluster.
>
>    Installed charm5.9 (a parallel programming OO language library)
>
>   The charm build is successful with following options:
>
> ./build charm++ mpi-linux iccstatic pthreads --no-build-shared
> --incdir=/opt/MPI/mvapich2_intel/include
> --libdir=/opt/MPI/mvapich2_intel/lib --libdir=/usr/lib
>
>  But when I try to execute tests/charm++/simplearrayhello program:
>
> [root at local simplearrayhello]# make
> ../../../bin/charmc   hello.ci
> ../../../bin/charmc  -c hello.C
> ../../../bin/charmc  -language charm++ -o hello hello.o
> icpc: command line remark #10010: option '-static-libcxa' is deprecated and
> will be removed in a future release. See '-help deprecated'
> ld: skipping incompatible /usr/lib/libpthread.so when searching for
> -lpthread
> ld: skipping incompatible /usr/lib/libpthread.a when searching for -lpthread
> ld: skipping incompatible /usr/lib/libpthread.so when searching for
> -lpthread
>
> ..
> ..
> ld: skipping incompatible /usr/lib/libc.so when searching for -lc
> ld: skipping incompatible /usr/lib/libc.a when searching for -lc
> /opt/MPI/mvapich2_intel/lib/libmpich.a(rdma_iba_init.o)(.text+0xa7): In
> function `MPIDI_CH3I_CM_Finalize':
> : undefined reference to `ibv_dereg_mr'
> /opt/MPI/mvapich2_intel/lib/libmpich.a(rdma_iba_init.o)(.text+0xb9): In
> function `MPIDI_CH3I_CM_Finalize':
> : undefined reference to `ibv_dereg_mr'
> /opt/MPI/mvapich2_intel/lib/libmpich.a(rdma_iba_init.o)(.text+0x112): In
> function `MPIDI_CH3I_CM_Finalize':
> : undefined reference to `ibv_destroy_qp'
>
>
> ...
> Fatal Error by charmc in directory
> /opt/apps/namd_intel/charm-5.9/tests/charm++/simplearrayhello
>    Command icpc -static-libcxa -L/opt/MPI/mvapich2_intel/lib -L/usr/lib
> -L/opt/MPI/mvapich2_intel/lib -L/usr/lib -o hello -L../../../bin/../lib
> -I../../../bin/../include ../../../bin/../lib/libldb-rand.o hello.o
> moduleinit3173.o ../../../bin/../lib/libmemory-default.o
> ../../../bin/../lib/libthreads-default.o -lck -lconv-cplus-y -lconv-core
> -lconv-util -L/opt/MPI/mvapich2_intel/lib -L/usr/lib -lckqt -lmpich
> -lpthread -lpthread -ldl -lpthread -lm returned error code 1
> charmc exiting...
> make: *** [hello] Error 1
>
> It looks, charmc is not able to link libibverbs.
>
> With MPICH2, there is no problem charm++ execution.
>
> The src/arch/mpi-linux/conv-mach.sh is:
>
> CMK_CPP_CHARM='/lib/cpp -P'
> CMK_CPP_C='mpicc -E'
> CMK_CC='mpicc '
> CMK_CXX='mpicxx '
> CMK_CXXPP='mpicxx -E '
> CMK_CF77='ifort'
> CMK_CF90='ifort'
> CMK_RANLIB='ranlib'
> CMK_LIBS='-L/opt/MPI/mvapich2_intel/lib -L/usr/lib -lckqt -lmpich -lpthread'
> CMK_LD_LIBRARY_PATH="-Wl,-rpath,$CHARMLIBSO/"
> CMK_NATIVE_LIBS=''
> CMK_NATIVE_CC='icc '
> CMK_NATIVE_LD='icpc'
> CMK_NATIVE_CXX='icpc '
> CMK_NATIVE_LDXX='icpc'
> CMK_NATIVE_CC='icc '
> CMK_NATIVE_CXX='icpc '
> CMK_F90LIBS='-L/usr/absoft/lib -L/opt/absoft/lib -lf90math -lfio -lU77
> -lf77math '
> CMK_MOD_NAME_ALLCAPS=1
> CMK_MOD_EXT="mod"
> CMK_F90_USE_MODDIR=1
> CMK_F90_MODINC="-p
>
> Is this setting is ok or I need to add some more library paths (i.e. OFED
> libs)?
>
> Thank you,
> Sangamesh
>