[mvapich-discuss] mvapich2 1.9a fails to cleanup after failed jobs

[Jonathan Perkins]

Bhupender:
Thanks for your report.  This could be a problem with mvapich2.  Can you
tell us a bit more about the problem you're facing.  Which processes in
particular are being left behind (mpispawn and/or other processes?)
Also, does this also happen when using mpiexec?

On Mon, Nov 19, 2012 at 05:05:26PM +0000, Bhupender Thakur wrote:
> Hi,
> 
> We are working on implementing mvapich2 one out new cluster but have run into some issues
> with mvapich2 unable to cleanup frequently when jobs fails
> 
> We are usign mellanox infiniband
> $ ibv_devinfo
> hca_id:    mlx4_0
>     transport:            InfiniBand (0)
>     fw_ver:                2.10.4492
>     node_guid:            0002:c903:00ff:25b0
>     sys_image_guid:            0002:c903:00ff:25b3
>     vendor_id:            0x02c9
>     vendor_part_id:            4099
>     hw_ver:                0x0
>     board_id:            DEL0A30000019
>     phys_port_cnt:            1
>         port:    1
>             state:            PORT_ACTIVE (4)
>             max_mtu:        2048 (4)
>             active_mtu:        2048 (4)
>             sm_lid:            1
>             port_lid:        300
>             port_lmc:        0x00
>             link_layer:        IB
> 
> 
> mvapich2 1.9a
> $ mpiname -a
> MVAPICH2 1.9a Sat Sep  8 15:01:35 EDT 2012 ch3:mrail
> 
> Compilation
> CC: /usr/local/compilers/Intel/composer_xe_2013.0.079/bin/intel64/icc -O2 -fPIC   -g -DNDEBUG -DNVALGRIND -O2
> CXX: /usr/local/compilers/Intel/composer_xe_2013.0.079/bin/intel64/icpc -O2 -fPIC  -g -DNDEBUG -DNVALGRIND -O2
> F77: /usr/local/compilers/Intel/composer_xe_2013/bin/ifort   -g -O2 -L/usr/lib64
> FC: /usr/local/compilers/Intel/composer_xe_2013.0.079/bin/intel64/ifort -O2 -fPIC  -g -O2
> 
> Configuration
> --prefix=/usr/local/packages/mvapich2/1.9a/Intel-13.0.0 \
> FC=/usr/local/compilers/Intel/composer_xe_2013.0.079/bin/intel64/ifort \
> CC=/usr/local/compilers/Intel/composer_xe_2013.0.079/bin/intel64/icc \
> CXX=/usr/local/compilers/Intel/composer_xe_2013.0.079/bin/intel64/icpc \
> CFLAGS=-O2 -fPIC FCFLAGS=-O2 -fPIC CXXFLAGS=-O2 -fPIC \
> LDFLAGS=-L/usr/local/compilers/Intel/composer_xe_2013/lib -L/usr/local/compilers/Intel/composer_xe_2013/lib/intel64 \
> LIBS= CPPFLAGS= \
> --enable-rdma-cm --enable-g=dbg --enable-romio --with-file-system=lustre+nfs \
> --with-ib-include=/usr/include --with-ib-libpath=/usr/lib64 \
> --enable-threads=runtime --enable-mpe --enable-smpcoll --enable-shared --enable-xrc --with-hwloc
> 
> $ pbs_mom --version
> version: 3.0.6
> pbs moms are threaded with the default 3 threads.
> 
> This does not happen with openmpi. A sample hello world is being run with different parameters
> 
> program dummy
>   use mpi
>   character*10 name
> ! Init MPI
>     call MPI_Init(mpierr)
> ! Get Rank Size
>     call MPI_COMM_Rank(MPI_COMM_WORLD, nrank, mpierr)
>     call MPI_COMM_Size(MPI_COMM_WORLD, nproc, mpierr)
> ! Get Date
>     if (nrank==0) then
>     write(*,*)'System date: Running mpirun_rsh'
>     call system('date')
>     end if
> ! Print rank
>     call MPI_Barrier(MPI_COMM_WORLD, mpierr)
>     !
>     call MPI_Get_processor_name(name, nlen, mpierr)
>     write(*,*)"    I am ", nrank, " of " ,nproc, " on ", name
>     !
>     call MPI_Barrier(MPI_COMM_WORLD, mpierr)
> ! Finalize
>     call MPI_Finalize(mpierr)
> end
> 
> ===========
> #
>   cat $PBS_NODEFILE > hostfile
>   cat $PBS_NODEFILE | uniq > hosts
>   mpi_width=`cat hostfile | wc -l`
>   mpi_nodes=`cat hosts | wc -l`
> 
> for mpi_pn in 8 16
>   do
>     let sum_mpi=$mpi_pn*$mpi_nodes
>     let OMP_NUM_THREADS=$mpi_width/$sum_mpi
> 
>     for param in "MV2_USE_XRC=1"  "MV2_USE_RoCE=1" "MV2_USE_RDMA_CM=1"
>      do
>        echo "    $param"
>        echo "    nodes:$mpi_nodes  mpi-per-node:$mpi_pn  omp:$OMP_NUM_THREADS"
>        time mpirun_rsh -np $sum_mpi -hostfile hosts $param ./dummy
>     done
> done
> ===============
> using parameter "MV2_USE_RoCE=1" and "MV2_USE_RDMA_CM=1" should fail as they
> not been configured yet(openib.conf), nevertheless, the program does not exit cleanly.
> We are seeing this with some other applications where the process seems to have crashed
> and is not producting any useful output, but there are threads lingering on ling after the program has
> crashed.
> 
> At this stage we are not sure if this is infiniband or torque or mvapich2 issue. Please let us know if
> you have seen this behaviour and if there is a way to resolve this.
> 
> Best,
> Bhupender.
> 
> 
> Bhupender Thakur.
> IT- Analyst,
> High Performance Computing, LSU.
> Ph (225)578-5934

> _______________________________________________
> mvapich-discuss mailing list
> mvapich-discuss at cse.ohio-state.edu
> http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss


-- 
Jonathan Perkins
http://www.cse.ohio-state.edu/~perkinjo