[mvapich-discuss] Problem with MVAPICH2 (1.9, r6338) on InfiniBand

[Devendar Bureddy]

Hi Steven

The ch3:mrail is the default configuration. You can see more details here:
http://mvapich.cse.ohio-state.edu/support/user_guide_mvapich2-1.9.html#x1-110004.4
 .

Please do not confuse with "mrail" configuration with multi-rail network
support. The use of multiple IB network lanes(HCA) is controlled with
MV2_NUM_HCAS run-time parameter.

The wrong usage of HCAs could show similar errors. Are basic osu-benchmarks
running correctly in your setup?

-Devendar


On Mon, Jul 8, 2013 at 12:21 PM, Steven Vancoillie <
steven.vancoillie at gmail.com> wrote:

> Hi,
>
> when running an application (Global Arrays test suite) on top of
> MVAPICH2 (via ARMCI-MPI), I get the following error:
>
> [0->3] send desc error, wc_opcode=0
> [0->3] wc.status=12, wc.wr_id=0x6cf5c8, wc.opcode=0, vbuf->phead->type=54
> = MPIDI_CH3_PKT_CLOSE
> [r5i1n6:mpi_rank_0][MPIDI_CH3I_MRAILI_Cq_poll]
> src/mpid/ch3/channels/mrail/src/gen2/ibv_channel_manager.c:586: [] Got
> completion with error 12, vendor code=0x81, dest rank=3
>
> I would be grateful if someone can help me resolve this issue.
>
> The is the output of mpiexec --version:
>
> HYDRA build details:
>     Version:                                 3.0.3
>     Release Date:                            unreleased development copy
>     CC:                              icc
>     CXX:                             icpc
>     F77:                             ifort
>     F90:                             ifort
>     Configure options:
> '--disable-option-checking' '--prefix=/apps/leuven/mvapich2/1.9_intel'
> 'CC=icc' 'CXX=icpc' 'F77=ifort' '--enable-f77' '--enable-fc'
> '--enable-cxx' '--enable-romio' '--enable-debuginfo' '--enable-mpe'
> '--enable-shared' '--without-ftb' '--with-mpe' '--disable-ckpt'
> '--disable-mcast' '--disable-checkerrors' '--enable-embedded-mode'
> '--cache-file=/dev/null' '--srcdir=.' 'CFLAGS= -DNDEBUG -DNVALGRIND
> -O2' 'LDFLAGS=-L/lib -L/lib -Wl,-rpath,/lib -L/lib' 'LIBS=-libumad
> -libverbs -lrt -lnuma -lpthread ' 'CPPFLAGS=
> -I/data/leuven/source/mvapich2/1.9_intel/mvapich2-1.9-r6338/src/mpl/include
> -I/data/leuven/source/mvapich2/1.9_intel/mvapich2-1.9-r6338/src/mpl/include
> -I/data/leuven/source/mvapich2/1.9_intel/mvapich2-1.9-r6338/src/openpa/src
> -I/data/leuven/source/mvapich2/1.9_intel/mvapich2-1.9-r6338/src/openpa/src
>
> -I/data/leuven/source/mvapich2/1.9_intel/mvapich2-1.9-r6338/src/mpi/romio/include
> -I/include -I/include'
>     Process Manager:                         pmi
>     Launchers available:                     ssh rsh fork slurm ll lsf
> sge manual persist
>     Topology libraries available:            hwloc
>     Resource management kernels available:   user slurm ll lsf sge pbs
> cobalt
>     Checkpointing libraries available:
>     Demux engines available:                 poll select
>
> Furthermore, someone suggested to me to build MVAPICH2 without mrail
> support: "you should probably use the default build of mvapich instead
> of the mrail build, unless you want to use multiple network lanes
> simultaneously". Unfortunately, I can't really understand from the
> user guide how I should do this, or even what multiple rail means. Is
> there some online documentation that addresses this or how I should
> build it?
>
> kind regards,
> Steven
> _______________________________________________
> mvapich-discuss mailing list
> mvapich-discuss at cse.ohio-state.edu
> http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss
>



-- 
Devendar
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.cse.ohio-state.edu/pipermail/mvapich-discuss/attachments/20130708/8d1a040c/attachment.html