[mvapich-discuss] MVAPICH2 not working with latest Intel 12 compiler on Mellanox IB Gen2

[Dan Kokron]

I had some issue with the 12.1 compiler and mvapich2-1.7.  The
resolution went something like this.


Intel(R) Fortran Intel(R) 64 Compiler XE for applications running on Intel(R) 64, Version 12.1.2.273 Build 20111128
Copyright (C) 1985-2011 Intel Corporation.  All rights reserved.

setenv ac_cv_func_malloc_0_nonnull yes
setenv ac_cv_func_realloc_0_nonnull yes
./configure CC=icc CXX=icpc F77=ifort FC=ifort CFLAGS=-fpic CXXFLAGS=-fpic FFLAGS=-fpic FCFLAGS=-fpic --prefix=/xxx/mvapich2-1.7/install/12.1.2.273 --enable-f77 --enable-fc --enable-cxx --enable-romio --with-file-system=lustre --enable-threads=default --with-device=ch3:mrail --with-rdma=gen2 --with-hwloc -disable-multi-aliases -enable-xrc=yes -enable-hybrid
make
make install




On Mon, 2012-04-23 at 22:33 -0500, Dhabaleswar Panda wrote:
> Hi Mohammad,
> 
> Thanks for your note. Glad to know that you are moving from MVAPICH1 to
> MVAPICH2.
> 
> Regarding the Intel compiler issue, which exact version of Intel 12
> compiler you are using.
> 
> A few months back, a similar posting was made on mvapich-discuss list.
> The poster also indicated that later versions of Intel 12.1 compiler
> didn't have this issue.
> 
> http://mail.cse.ohio-state.edu/pipermail/mvapich-discuss/2011-November/003642.html
> 
> Please take a look at this posting and see if any later version of the
> compiler helps to resolve the issue you are seeing.
> 
> Regarding the configuration flags, there is no need to use
> --enable-threads and --enable-smpcoll. You can drop these.
> 
> Let us know if these suggestions help to resolve the issues you are
> seeing.
> 
> Thanks,
> 
> DK
> 
> 
> On Mon, 23 Apr 2012, sindimo at gmail.com wrote:
> 
> > Dear Support,
> >
> >                    We are trying to get MVAPICH2 to work with the Intel 12
> > compiler on an older cluster having Mellanox IB Gen2. This seems to work
> > fine with the older Intel 10 compiler but not with Intel 12. It actually
> > builds but we are having issues when running the application.
> >
> > The configuration we are using:
> > export CC=${CC:-/usr/local/intel/ics12/ics12/bin/icc}
> > export CXX=${CXX:-/usr/local/intel/ics12/ics12/bin/icpc}
> > export F77=${F77:-/usr/local/intel/ics12/ics12/bin/ifort}
> > export FC=${FC:-/usr/local/intel/ics12/ics12/bin/ifort}
> >
> > ./configure --prefix=/usr/local/mpi/mvapich2/intel10/1.7 --enable-g=dbg
> > --enable-debuginfo --enable-romio  --with-file-system=panfs+nfs+ufs
> > --with-rdma=gen2 --with-ib-libpath=/usr/lib64 --enable-threads
> > --enable-smpcoll
> >
> > make
> > make install
> >
> > The PATH and LD_LIBRARY_PATH are all clean and setup properly.
> >
> > We use the below to launch the job:
> > mpirun_rsh -n 4 -hostfile ./nodes myapp.exe
> >
> > With MVAPICH2 1.7 + Intel 10 it runs fine.
> >
> > With MVAPICH2 1.7 + Intel 12 we get the below error immediately after we
> > lunch the job:
> >
> > [plcf023:mpispawn_0][child_handler] MPI process (rank: 0, pid: 15976)
> > terminated with signal 11 -> abort job
> > [plcf:mpirun_rsh][process_mpispawn_connection] mpispawn_0 from node plcf023
> > aborted: MPI process error (1)
> >
> >
> > We tried setting MV2_DEBUG_SHOW_BACKTRACE but it's not showing anything
> > more than the above error.
> >
> > We also tried MVAPICH2 1.8rc1 with Intel 12 and it's showing the same
> > problem.
> >
> > We ran out of ideas to trouble shoot this and we would appreciate any help
> > with it.
> >
> > Also for reference, we have newer clusters with Qlogic IB with PSM and the
> > MVAPICH2 + Intel 12 combination runs fine there, so it seems more of an
> > issue with the Mellanox Gen 2 IB. Obviously with the PSM clusters we
> > configure with "--with-device=ch3:psm   --with-psm-include=/usr/include
> > --with-psm=/usr/lib64".
> >
> > This is part of an activity we are doing in our HPC computer center to move
> > from MVAPICH1 to MVAPICH2 and currently this is the only show stopper we
> > have.
> >
> > Thank you for your help.
> >
> > Mohammad Sindi
> > HPC Group
> > EXPEC Computer Center
> > Saudi Aramco
> >
> 
> _______________________________________________
> mvapich-discuss mailing list
> mvapich-discuss at cse.ohio-state.edu
> http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss

-- 
Dan Kokron
Global Modeling and Assimilation Office, Code 610.1
NASA Goddard Space Flight Center
Greenbelt, MD 20771 USA
Daniel.S.Kokron at nasa.gov
Phone: (301) 614-5192
Fax:   (301) 614-5304