[mvapich-discuss] mvapich2 warning: Rndv Receiver is receiving less than as expected

[Bernd Kallies]

I'd like to report a problem with mvapich2, which I did not manage to
nail down to a particular issue so far. So I'd like to report it at this
stage to the mvapich2 dev team.

We use the program package cp2k (http://cp2k.berlios.de/). We compiled
it with Intel compilers v10 or v11, and linked with mvapich2-1.2.0,
which was generated with the same compiler. In addition, the package
uses blacs, scalapack, and blas/lapack, fft.

We have an SGI ICE cluster equipped with Intel Xeon Harpertown (E5472, 8
cores per node) and 2-port 4xDDR Mellanox ConnectX adapters (Mellanox
MT26418, ConnectX IB DDR, PCIe 2.0 5GT/s). We use SLES10 SP2 for x86_64,
and OFED 1.3 built by SGI.

When calculating a particular time series with cp2k, the run aborts
after about 30 timesteps (takes a couple of hours by using 128 MPI
tasks) with the mvapich2 message

Warning! Rndv Receiver is receiving (49152 < 110592) less than as
expected

with varying recv/expected message sizes.

The simulation step where this message appears is not deterministic. The
only thing I know is that it appears for sure between the 27th and the
32rd time step after a restart. Sometimes a run aborts with SIGSEGV.
Sometimes the run only hangs until running into some time limit. The
problem cannot be reproduced by starting from restart files written by
cp2k inbetween the runs. One always has to wait about 30 time steps. So
it does not seem to depend on the input data, but on wallclock time.

The problem does not occur when using another MPI lib. I successfully
tried SGI MPT v1.22, Intel MPI v3.1.026, MPICH2 v1.0.7 and v1.0.8 (using
IPoIB). It only shows up with MVAPICH2. I tried v1.2.0, v1.2p1 from dev
snapshot 2009-03-02, v1.2rc1, v1.2rc2. Failures occur independently from
the compiler and optimization level used for the executable and
self-compiled libs. I tried Intel and gcc, -O0 .. -O2. Failures are also
independent from the blacs/scalapack/blas/lapack combination (I used
self-compiled with Intel compilers or gcc, or what is shipped with MKL
v10 or 11), and are not related to the use of shared vs. static libs. No
special mvapich2 tuning environment variables are set.

The only thing I figured out so far is that the warning messages and
following aborts seem to originate from MPI communication initiated by
blacs routines. A recent call stack trace produced by the Intel Fortran
RTE while running an executable with blacs debugging on, and using
static mpi/blacs/scalapack/mkl libs gives:

Warning! Rndv Receiver is receiving (49152 < 110592) less than as expected
BLACS ERROR 'MPI error 480874510 on call to MPI_Recv'
from {-1,-1}, pnum=15, Contxt=-1, on line 8 of file 'BI_Srecv.c'.

application called MPI_Abort(MPI_COMM_WORLD, 1) - process 15
forrtl: error (78): process killed (SIGTERM)
Image              PC                Routine            Line        Source             
cp2k.popt          0000000001A7BB4D  Unknown               Unknown  Unknown
cp2k.popt          0000000001A6B48A  Unknown               Unknown  Unknown
cp2k.popt          0000000001A6A634  Unknown               Unknown  Unknown
cp2k.popt          0000000001A5B701  Unknown               Unknown  Unknown
cp2k.popt          00000000011980E0  BI_Srecv                    8  BI_Srecv.c
cp2k.popt          0000000001197AE5  BI_IdringBR                12  BI_IdringBR.c
cp2k.popt          0000000001193ACE  Czgebr2d                  192  zgebr2d_.c
...
forrtl: error (78): process killed (SIGTERM)
Image              PC                Routine            Line        Source             
libmlx4-rdmav2.so  00002AF569D3414D  Unknown               Unknown  Unknown
...


Another run on the same input data by using the same exe and same task
geom gave

Warning! Rndv Receiver is receiving (512 < 221184) less than as expected
forrtl: severe (174): SIGSEGV, segmentation fault occurred
Image              PC                Routine            Line        Source             
cp2k.popt          0000000001A709D2  Unknown               Unknown  Unknown
cp2k.popt          0000000001A6D62D  Unknown               Unknown  Unknown
cp2k.popt          0000000001A6A5FC  Unknown               Unknown  Unknown
cp2k.popt          0000000001A5B701  Unknown               Unknown  Unknown
cp2k.popt          00000000011980E0  BI_Srecv                    8  BI_Srecv.c
cp2k.popt          0000000001197AE5  BI_IdringBR                12  BI_IdringBR.c
cp2k.popt          00000000011936E5  Cdgebr2d                  192  dgebr2d_.c
...
forrtl: error (78): process killed (SIGTERM)
Image              PC                Routine            Line        Source             
libmlx4-rdmav2.so  00002AEF72F07A00  Unknown               Unknown  Unknown
...


Currently I have no idea how to nail down the problem to a particular
mvapich2 issue. The time needed to reach the point of failure, the large
number of tasks, the complexity of the numerics, and the apparent
dependency on some runtime component make it hard to debug this.

Please let me know if you need further information.

Sincerely, BK

-- 
Dr. Bernd Kallies
Konrad-Zuse-Zentrum für Informationstechnik Berlin
Takustr. 7
14195 Berlin
Tel: +49-30-84185-270
Fax: +49-30-84185-311
e-mail: kallies at zib.de