[mvapich-discuss] Unknown packet type 40 in MRAILI_Process_send: No such file or directory

Sun Mar 22 21:02:24 EDT 2015

Hello Jeff,

We've not seen this error before. It could be due to a memory corruption or
because of the incomplete support you mentioned. In either case, could you
please request the user to try with the latest release and see if it solves
the issue?

Best Regards,
Hari.

On Sun, Mar 22, 2015 at 7:36 PM, Jeff Hammond <jeff.science at gmail.com>
wrote:

> An ARMCI-MPI user reported the following error to me when trying to
> run NWChem over ARMCI-MPI3 with MVAPICH2 1.9.  Is this related to
> incomplete MPI-3 support in MVAPICH2 1.9 or something else?  If it's
> just an issue with an older version of MVAPICH2, then the solution is
> easy enough.  But I want to make sure it's not something that could
> affect newer versions, since in that case it is wasted effort to walk
> the user through the MVAPICH2 build process.
>
> Thanks!
>
> Jeff
>
> [n092:mpi_rank_23][MRAILI_Process_send]
> src/mpid/ch3/channels/mrail/src/gen2/ibv_send.c:1748: Unknown packet
> type 40 in MRAILI_Process_send: No such file or directory (2)
> [n092:mpi_rank_20][MRAILI_Process_send]
> src/mpid/ch3/channels/mrail/src/gen2/ibv_send.c:1748: Unknown packet
> type 40 in MRAILI_Process_send: No such file or directory (2)
> [n092:mpi_rank_22][MRAILI_Process_send]
> src/mpid/ch3/channels/mrail/src/gen2/ibv_send.c:1748: Unknown packet
> type 40 in MRAILI_Process_send: No such file or directory (2)
> [n092:mpi_rank_16][MRAILI_Process_send]
> src/mpid/ch3/channels/mrail/src/gen2/ibv_send.c:1748: Unknown packet
> type 40 in MRAILI_Process_send: No such file or directory (2)
> [n092:mpi_rank_17][MRAILI_Process_send]
> src/mpid/ch3/channels/mrail/src/gen2/ibv_send.c:1748: Unknown packet
> type 40 in MRAILI_Process_send: No such file or directory (2)
> [n092:mpi_rank_14][MRAILI_Process_send]
> src/mpid/ch3/channels/mrail/src/gen2/ibv_send.c:1748: Unknown packet
> type 40 in MRAILI_Process_send: No such file or directory (2)
> [n092:mpi_rank_13][MRAILI_Process_send]
> src/mpid/ch3/channels/mrail/src/gen2/ibv_send.c:1748: Unknown packet
> type 40 in MRAILI_Process_send: No such file or directory (2)
> [n092:mpi_rank_18][MRAILI_Process_send]
> src/mpid/ch3/channels/mrail/src/gen2/ibv_send.c:1748: Unknown packet
> type 40 in MRAILI_Process_send: No such file or directory (2)
> [n092:mpi_rank_19][MRAILI_Process_send]
> src/mpid/ch3/channels/mrail/src/gen2/ibv_send.c:1748: Unknown packet
> type 40 in MRAILI_Process_send: No such file or directory (2)
> [n092:mpi_rank_21][MRAILI_Process_send]
> src/mpid/ch3/channels/mrail/src/gen2/ibv_send.c:1748: Unknown packet
> type 40 in MRAILI_Process_send: No such file or directory (2)
> [n092:mpi_rank_15][MRAILI_Process_send]
> src/mpid/ch3/channels/mrail/src/gen2/ibv_send.c:1748: Unknown packet
> type 40 in MRAILI_Process_send: No such file or directory (2)
> [n092:mpi_rank_12][MRAILI_Process_send]
> src/mpid/ch3/channels/mrail/src/gen2/ibv_send.c:1748: Unknown packet
> type 40 in MRAILI_Process_send: No such file or directory (2)
> [proxy:0:0 at n059] HYD_pmcd_pmip_control_cmd_cb
> (./pm/pmiserv/pmip_cb.c:913): assert (!closed) failed
> [proxy:0:0 at n059] HYDT_dmxu_poll_wait_for_event
> (./tools/demux/demux_poll.c:77): callback returned error status
> [proxy:0:0 at n059] main (./pm/pmiserv/pmip.c:206): demux engine error
> waiting for event
> srun: error: n059: task 0: Exited with exit code 7
> [mpiexec at n059] HYDT_bscu_wait_for_completion
> (./tools/bootstrap/utils/bscu_wait.c:76): one of the processes
> terminated badly; aborting
> [mpiexec at n059] HYDT_bsci_wait_for_completion
> (./tools/bootstrap/src/bsci_wait.c:23): launcher returned error
> waiting for completion
> [mpiexec at n059] HYD_pmci_wait_for_completion
> (./pm/pmiserv/pmiserv_pmci.c:217): launcher returned error waiting for
> completion
> [mpiexec at n059] main (./ui/mpich/mpiexec.c:331): process manager error
> waiting for completion
>
>
>
> module show MVAPICH2
>
>
> ---------------------------------------------------------------------------------------------------------------------------------------------------
>    /gpfs/buildsets/eb140915/modules/all/MVAPICH2/1.9-iccifort-2011.13.367:
>
> ---------------------------------------------------------------------------------------------------------------------------------------------------
> whatis("Description: This is an MPI 3.0 implementation.  It is based
> on MPICH2 and MVICH. - Homepage:
> http://mvapich.cse.ohio-state.edu/overview/mvapich2/ ")
> conflict("MVAPICH2")
>
> prepend_path("CPATH","/gpfs/buildsets/eb140915/software/MVAPICH2/1.9-iccifort-2011.13.367/include")
>
> prepend_path("LD_LIBRARY_PATH","/gpfs/buildsets/eb140915/software/MVAPICH2/1.9-iccifort-2011.13.367/lib")
>
> prepend_path("LIBRARY_PATH","/gpfs/buildsets/eb140915/software/MVAPICH2/1.9-iccifort-2011.13.367/lib")
>
> prepend_path("MANPATH","/gpfs/buildsets/eb140915/software/MVAPICH2/1.9-iccifort-2011.13.367/share/man")
>
> prepend_path("PATH","/gpfs/buildsets/eb140915/software/MVAPICH2/1.9-iccifort-2011.13.367/bin")
>
> prepend_path("PKG_CONFIG_PATH","/gpfs/buildsets/eb140915/software/MVAPICH2/1.9-iccifort-2011.13.367/lib/pkgconfig")
>
> setenv("EBROOTMVAPICH2","/gpfs/buildsets/eb140915/software/MVAPICH2/1.9-iccifort-2011.13.367")
> setenv("EBVERSIONMVAPICH2","1.9")
>
> setenv("EBDEVELMVAPICH2","/gpfs/buildsets/eb140915/software/MVAPICH2/1.9-iccifort-2011.13.367/easybuild/MVAPICH2-1.9-iccifort-2011.13.367-easybuild-devel")
> help([[   This is an MPI 3.0 implementation.  It is based on MPICH2
> and MVICH. - Homepage:
> http://mvapich.cse.ohio-state.edu/overview/mvapich2/    ]])
>
>
>
> /gpfs/buildsets/eb140915/software/MVAPICH2/1.9-iccifort-2011.13.367/bin/mpichversion
> MVAPICH2 Version:       1.9
> MVAPICH2 Release date:  Mon May  6 12:25:08 EDT 2013
> MVAPICH2 Device:        ch3:mrail
> MVAPICH2 configure:
>
> --prefix=/gpfs/buildsets/eb140915/software/MVAPICH2/1.9-iccifort-2011.13.367
> --with-rdma=gen2 --with-thread-package=pthreads --enable-fast
> --enable-shared --enable-sharedlibs=gcc --enable-f77 --enable-fc
> --enable-cxx
> MVAPICH2 CC:    icc -fPIC -O2 -xHOST -ftz -fp-speculation=safe
> -fp-model source   -DNDEBUG -DNVALGRIND -O2
> MVAPICH2 CXX:   icpc -fPIC -O2 -xHOST -ftz -fp-speculation=safe
> -fp-model source  -DNDEBUG -DNVALGRIND -O2
> MVAPICH2 F77:   ifort -L/lib -L/lib -fPIC -O2 -xHOST -ftz
> -fp-speculation=safe -fp-model source  -O2
> MVAPICH2 FC:    ifort -fPIC -O2 -xHOST -ftz -fp-speculation=safe
> -fp-model source  -O2
>
>
> --
> Jeff Hammond
> jeff.science at gmail.com
> http://jeffhammond.github.io/
> _______________________________________________
> mvapich-discuss mailing list
> mvapich-discuss at cse.ohio-state.edu
> http://mailman.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.cse.ohio-state.edu/pipermail/mvapich-discuss/attachments/20150322/221f437c/attachment-0002.html>