[mvapich-discuss] Unknown packet type 40 in MRAILI_Process_send: No such file or directory

[Jeff Hammond]

An ARMCI-MPI user reported the following error to me when trying to
run NWChem over ARMCI-MPI3 with MVAPICH2 1.9.  Is this related to
incomplete MPI-3 support in MVAPICH2 1.9 or something else?  If it's
just an issue with an older version of MVAPICH2, then the solution is
easy enough.  But I want to make sure it's not something that could
affect newer versions, since in that case it is wasted effort to walk
the user through the MVAPICH2 build process.

Thanks!

Jeff

[n092:mpi_rank_23][MRAILI_Process_send]
src/mpid/ch3/channels/mrail/src/gen2/ibv_send.c:1748: Unknown packet
type 40 in MRAILI_Process_send: No such file or directory (2)
[n092:mpi_rank_20][MRAILI_Process_send]
src/mpid/ch3/channels/mrail/src/gen2/ibv_send.c:1748: Unknown packet
type 40 in MRAILI_Process_send: No such file or directory (2)
[n092:mpi_rank_22][MRAILI_Process_send]
src/mpid/ch3/channels/mrail/src/gen2/ibv_send.c:1748: Unknown packet
type 40 in MRAILI_Process_send: No such file or directory (2)
[n092:mpi_rank_16][MRAILI_Process_send]
src/mpid/ch3/channels/mrail/src/gen2/ibv_send.c:1748: Unknown packet
type 40 in MRAILI_Process_send: No such file or directory (2)
[n092:mpi_rank_17][MRAILI_Process_send]
src/mpid/ch3/channels/mrail/src/gen2/ibv_send.c:1748: Unknown packet
type 40 in MRAILI_Process_send: No such file or directory (2)
[n092:mpi_rank_14][MRAILI_Process_send]
src/mpid/ch3/channels/mrail/src/gen2/ibv_send.c:1748: Unknown packet
type 40 in MRAILI_Process_send: No such file or directory (2)
[n092:mpi_rank_13][MRAILI_Process_send]
src/mpid/ch3/channels/mrail/src/gen2/ibv_send.c:1748: Unknown packet
type 40 in MRAILI_Process_send: No such file or directory (2)
[n092:mpi_rank_18][MRAILI_Process_send]
src/mpid/ch3/channels/mrail/src/gen2/ibv_send.c:1748: Unknown packet
type 40 in MRAILI_Process_send: No such file or directory (2)
[n092:mpi_rank_19][MRAILI_Process_send]
src/mpid/ch3/channels/mrail/src/gen2/ibv_send.c:1748: Unknown packet
type 40 in MRAILI_Process_send: No such file or directory (2)
[n092:mpi_rank_21][MRAILI_Process_send]
src/mpid/ch3/channels/mrail/src/gen2/ibv_send.c:1748: Unknown packet
type 40 in MRAILI_Process_send: No such file or directory (2)
[n092:mpi_rank_15][MRAILI_Process_send]
src/mpid/ch3/channels/mrail/src/gen2/ibv_send.c:1748: Unknown packet
type 40 in MRAILI_Process_send: No such file or directory (2)
[n092:mpi_rank_12][MRAILI_Process_send]
src/mpid/ch3/channels/mrail/src/gen2/ibv_send.c:1748: Unknown packet
type 40 in MRAILI_Process_send: No such file or directory (2)
[proxy:0:0 at n059] HYD_pmcd_pmip_control_cmd_cb
(./pm/pmiserv/pmip_cb.c:913): assert (!closed) failed
[proxy:0:0 at n059] HYDT_dmxu_poll_wait_for_event
(./tools/demux/demux_poll.c:77): callback returned error status
[proxy:0:0 at n059] main (./pm/pmiserv/pmip.c:206): demux engine error
waiting for event
srun: error: n059: task 0: Exited with exit code 7
[mpiexec at n059] HYDT_bscu_wait_for_completion
(./tools/bootstrap/utils/bscu_wait.c:76): one of the processes
terminated badly; aborting
[mpiexec at n059] HYDT_bsci_wait_for_completion
(./tools/bootstrap/src/bsci_wait.c:23): launcher returned error
waiting for completion
[mpiexec at n059] HYD_pmci_wait_for_completion
(./pm/pmiserv/pmiserv_pmci.c:217): launcher returned error waiting for
completion
[mpiexec at n059] main (./ui/mpich/mpiexec.c:331): process manager error
waiting for completion



module show MVAPICH2

---------------------------------------------------------------------------------------------------------------------------------------------------
   /gpfs/buildsets/eb140915/modules/all/MVAPICH2/1.9-iccifort-2011.13.367:
---------------------------------------------------------------------------------------------------------------------------------------------------
whatis("Description: This is an MPI 3.0 implementation.  It is based
on MPICH2 and MVICH. - Homepage:
http://mvapich.cse.ohio-state.edu/overview/mvapich2/ ")
conflict("MVAPICH2")
prepend_path("CPATH","/gpfs/buildsets/eb140915/software/MVAPICH2/1.9-iccifort-2011.13.367/include")
prepend_path("LD_LIBRARY_PATH","/gpfs/buildsets/eb140915/software/MVAPICH2/1.9-iccifort-2011.13.367/lib")
prepend_path("LIBRARY_PATH","/gpfs/buildsets/eb140915/software/MVAPICH2/1.9-iccifort-2011.13.367/lib")
prepend_path("MANPATH","/gpfs/buildsets/eb140915/software/MVAPICH2/1.9-iccifort-2011.13.367/share/man")
prepend_path("PATH","/gpfs/buildsets/eb140915/software/MVAPICH2/1.9-iccifort-2011.13.367/bin")
prepend_path("PKG_CONFIG_PATH","/gpfs/buildsets/eb140915/software/MVAPICH2/1.9-iccifort-2011.13.367/lib/pkgconfig")
setenv("EBROOTMVAPICH2","/gpfs/buildsets/eb140915/software/MVAPICH2/1.9-iccifort-2011.13.367")
setenv("EBVERSIONMVAPICH2","1.9")
setenv("EBDEVELMVAPICH2","/gpfs/buildsets/eb140915/software/MVAPICH2/1.9-iccifort-2011.13.367/easybuild/MVAPICH2-1.9-iccifort-2011.13.367-easybuild-devel")
help([[   This is an MPI 3.0 implementation.  It is based on MPICH2
and MVICH. - Homepage:
http://mvapich.cse.ohio-state.edu/overview/mvapich2/    ]])


/gpfs/buildsets/eb140915/software/MVAPICH2/1.9-iccifort-2011.13.367/bin/mpichversion
MVAPICH2 Version:       1.9
MVAPICH2 Release date:  Mon May  6 12:25:08 EDT 2013
MVAPICH2 Device:        ch3:mrail
MVAPICH2 configure:
--prefix=/gpfs/buildsets/eb140915/software/MVAPICH2/1.9-iccifort-2011.13.367
--with-rdma=gen2 --with-thread-package=pthreads --enable-fast
--enable-shared --enable-sharedlibs=gcc --enable-f77 --enable-fc
--enable-cxx
MVAPICH2 CC:    icc -fPIC -O2 -xHOST -ftz -fp-speculation=safe
-fp-model source   -DNDEBUG -DNVALGRIND -O2
MVAPICH2 CXX:   icpc -fPIC -O2 -xHOST -ftz -fp-speculation=safe
-fp-model source  -DNDEBUG -DNVALGRIND -O2
MVAPICH2 F77:   ifort -L/lib -L/lib -fPIC -O2 -xHOST -ftz
-fp-speculation=safe -fp-model source  -O2
MVAPICH2 FC:    ifort -fPIC -O2 -xHOST -ftz -fp-speculation=safe
-fp-model source  -O2


-- 
Jeff Hammond
jeff.science at gmail.com
http://jeffhammond.github.io/