[mvapich-discuss] Bad termination

[Albino A. Aveleda]

Dear All,

I have installed the mvapich2-2.0a. When I´ve been testing the job run but 
returned the error message below.

My command line inside the torque/PBS job file is 
mpirun -launcher rsh -f ${PBS_NODEFILE} -n ${NUM_PROC} ./mpitest

How do I fix this?

Best regards,
Albino

--- output ---
Hello world!  I am process number: 12 on host r1i1n8
Hello world!  I am process number: 13 on host r1i1n8
Hello world!  I am process number: 14 on host r1i1n8
Hello world!  I am process number: 15 on host r1i1n8
Hello world!  I am process number: 10 on host r1i1n8
Hello world!  I am process number: 11 on host r1i1n8
Hello world!  I am process number: 8 on host r1i1n8
Hello world!  I am process number: 9 on host r1i1n8
Hello world!  I am process number: 3 on host r1i1n11
Hello world!  I am process number: 2 on host r1i1n11
Hello world!  I am process number: 6 on host r1i1n11
Hello world!  I am process number: 7 on host r1i1n11
Hello world!  I am process number: 4 on host r1i1n11
Hello world!  I am process number: 5 on host r1i1n11
Hello world!  I am process number: 0 on host r1i1n11
Hello world!  I am process number: 1 on host r1i1n11
[0->0] send desc error, wc_opcode=0
[0->0] wc.status=12, wc.wr_id=0x705078, wc.opcode=0, vbuf->phead->type=54 = MPIDI_CH3_PKT_CLOSE
[r1i1n8:mpi_rank_11][MPIDI_CH3I_MRAILI_Cq_poll] src/mpid/ch3/channels/mrail/src/gen2/ibv_channel_manager.c:587: [] Got completion with error 12, v
endor code=0x81, dest rank=0
: No such file or directory (2)
[0->6] send desc error, wc_opcode=0
[0->6] wc.status=12, wc.wr_id=0x70d058, wc.opcode=0, vbuf->phead->type=54 = MPIDI_CH3_PKT_CLOSE
[r1i1n8:mpi_rank_9][MPIDI_CH3I_MRAILI_Cq_poll] src/mpid/ch3/channels/mrail/src/gen2/ibv_channel_manager.c:587: [] Got completion with error 12, ve
ndor code=0x81, dest rank=6
: No such file or directory (2)
[0->4] send desc error, wc_opcode=0
[0->4] wc.status=12, wc.wr_id=0x70cf60, wc.opcode=0, vbuf->phead->type=54 = MPIDI_CH3_PKT_CLOSE
[r1i1n8:mpi_rank_13][MPIDI_CH3I_MRAILI_Cq_poll] src/mpid/ch3/channels/mrail/src/gen2/ibv_channel_manager.c:587: [] Got completion with error 12, v
endor code=0x81, dest rank=4
: No such file or directory (2)
[0->4] send desc error, wc_opcode=0
[0->4] wc.status=12, wc.wr_id=0x70cf60, wc.opcode=0, vbuf->phead->type=54 = MPIDI_CH3_PKT_CLOSE
[r1i1n8:mpi_rank_15][MPIDI_CH3I_MRAILI_Cq_poll] src/mpid/ch3/channels/mrail/src/gen2/ibv_channel_manager.c:587: [] Got completion with error 12, v
endor code=0x81, dest rank=4
: No such file or directory (2)

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   EXIT CODE: 252
=   CLEANING UP REMAINING PROCESSES
=   YOU CAN IGNORE THE BELOW CLEANUP MESSAGES
===================================================================================
[proxy:0:0 at r1i1n11] HYD_pmcd_pmip_control_cmd_cb (./pm/pmiserv/pmip_cb.c:913): assert (!closed) failed
[proxy:0:0 at r1i1n11] HYDT_dmxu_poll_wait_for_event (./tools/demux/demux_poll.c:77): callback returned error status
[proxy:0:0 at r1i1n11] main (./pm/pmiserv/pmip.c:206): demux engine error waiting for event
[mpiexec at r1i1n11] HYDT_bscu_wait_for_completion (./tools/bootstrap/utils/bscu_wait.c:76): one of the processes terminated badly; aborting
[mpiexec at r1i1n11] HYDT_bsci_wait_for_completion (./tools/bootstrap/src/bsci_wait.c:23): launcher returned error waiting for completion
[mpiexec at r1i1n11] HYD_pmci_wait_for_completion (./pm/pmiserv/pmiserv_pmci.c:217): launcher returned error waiting for completion
[mpiexec at r1i1n11] main (./ui/mpich/mpiexec.c:331): process manager error waiting for completion