[mvapich-discuss] mvapich2 1.9a fails to cleanup after failed jobs

[Bhupender Thakur]

Hi,

We are working on implementing mvapich2 one out new cluster but have run into some issues
with mvapich2 unable to cleanup frequently when jobs fails

We are usign mellanox infiniband
$ ibv_devinfo
hca_id:    mlx4_0
    transport:            InfiniBand (0)
    fw_ver:                2.10.4492
    node_guid:            0002:c903:00ff:25b0
    sys_image_guid:            0002:c903:00ff:25b3
    vendor_id:            0x02c9
    vendor_part_id:            4099
    hw_ver:                0x0
    board_id:            DEL0A30000019
    phys_port_cnt:            1
        port:    1
            state:            PORT_ACTIVE (4)
            max_mtu:        2048 (4)
            active_mtu:        2048 (4)
            sm_lid:            1
            port_lid:        300
            port_lmc:        0x00
            link_layer:        IB


mvapich2 1.9a
$ mpiname -a
MVAPICH2 1.9a Sat Sep  8 15:01:35 EDT 2012 ch3:mrail

Compilation
CC: /usr/local/compilers/Intel/composer_xe_2013.0.079/bin/intel64/icc -O2 -fPIC   -g -DNDEBUG -DNVALGRIND -O2
CXX: /usr/local/compilers/Intel/composer_xe_2013.0.079/bin/intel64/icpc -O2 -fPIC  -g -DNDEBUG -DNVALGRIND -O2
F77: /usr/local/compilers/Intel/composer_xe_2013/bin/ifort   -g -O2 -L/usr/lib64
FC: /usr/local/compilers/Intel/composer_xe_2013.0.079/bin/intel64/ifort -O2 -fPIC  -g -O2

Configuration
--prefix=/usr/local/packages/mvapich2/1.9a/Intel-13.0.0 \
FC=/usr/local/compilers/Intel/composer_xe_2013.0.079/bin/intel64/ifort \
CC=/usr/local/compilers/Intel/composer_xe_2013.0.079/bin/intel64/icc \
CXX=/usr/local/compilers/Intel/composer_xe_2013.0.079/bin/intel64/icpc \
CFLAGS=-O2 -fPIC FCFLAGS=-O2 -fPIC CXXFLAGS=-O2 -fPIC \
LDFLAGS=-L/usr/local/compilers/Intel/composer_xe_2013/lib -L/usr/local/compilers/Intel/composer_xe_2013/lib/intel64 \
LIBS= CPPFLAGS= \
--enable-rdma-cm --enable-g=dbg --enable-romio --with-file-system=lustre+nfs \
--with-ib-include=/usr/include --with-ib-libpath=/usr/lib64 \
--enable-threads=runtime --enable-mpe --enable-smpcoll --enable-shared --enable-xrc --with-hwloc

$ pbs_mom --version
version: 3.0.6
pbs moms are threaded with the default 3 threads.

This does not happen with openmpi. A sample hello world is being run with different parameters

program dummy
  use mpi
  character*10 name
! Init MPI
    call MPI_Init(mpierr)
! Get Rank Size
    call MPI_COMM_Rank(MPI_COMM_WORLD, nrank, mpierr)
    call MPI_COMM_Size(MPI_COMM_WORLD, nproc, mpierr)
! Get Date
    if (nrank==0) then
    write(*,*)'System date: Running mpirun_rsh'
    call system('date')
    end if
! Print rank
    call MPI_Barrier(MPI_COMM_WORLD, mpierr)
    !
    call MPI_Get_processor_name(name, nlen, mpierr)
    write(*,*)"    I am ", nrank, " of " ,nproc, " on ", name
    !
    call MPI_Barrier(MPI_COMM_WORLD, mpierr)
! Finalize
    call MPI_Finalize(mpierr)
end

===========
#
  cat $PBS_NODEFILE > hostfile
  cat $PBS_NODEFILE | uniq > hosts
  mpi_width=`cat hostfile | wc -l`
  mpi_nodes=`cat hosts | wc -l`

for mpi_pn in 8 16
  do
    let sum_mpi=$mpi_pn*$mpi_nodes
    let OMP_NUM_THREADS=$mpi_width/$sum_mpi

    for param in "MV2_USE_XRC=1"  "MV2_USE_RoCE=1" "MV2_USE_RDMA_CM=1"
     do
       echo "    $param"
       echo "    nodes:$mpi_nodes  mpi-per-node:$mpi_pn  omp:$OMP_NUM_THREADS"
       time mpirun_rsh -np $sum_mpi -hostfile hosts $param ./dummy
    done
done
===============
using parameter "MV2_USE_RoCE=1" and "MV2_USE_RDMA_CM=1" should fail as they
not been configured yet(openib.conf), nevertheless, the program does not exit cleanly.
We are seeing this with some other applications where the process seems to have crashed
and is not producting any useful output, but there are threads lingering on ling after the program has
crashed.

At this stage we are not sure if this is infiniband or torque or mvapich2 issue. Please let us know if
you have seen this behaviour and if there is a way to resolve this.

Best,
Bhupender.


Bhupender Thakur.
IT- Analyst,
High Performance Computing, LSU.
Ph (225)578-5934
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.cse.ohio-state.edu/pipermail/mvapich-discuss/attachments/20121119/6244db91/attachment.html