[mvapich-discuss] mvapich2 warning: Rndv Receiver is receiving less than as expected

[Dhabaleswar Panda]

Thanks for reporting this issue in-depth. We will try to take a look at it
and get back to you. In the mean time, could you try disabling the runtime
environmental variable MV2_USE_SHMEM_COLL (MV2_USE_SHMEM_COLL=0) and let
us know whether the problem still persists or not.

Thanks,

DK

On Fri, 6 Mar 2009, Bernd Kallies wrote:

> I'd like to report a problem with mvapich2, which I did not manage to
> nail down to a particular issue so far. So I'd like to report it at this
> stage to the mvapich2 dev team.
>
> We use the program package cp2k (http://cp2k.berlios.de/). We compiled
> it with Intel compilers v10 or v11, and linked with mvapich2-1.2.0,
> which was generated with the same compiler. In addition, the package
> uses blacs, scalapack, and blas/lapack, fft.
>
> We have an SGI ICE cluster equipped with Intel Xeon Harpertown (E5472, 8
> cores per node) and 2-port 4xDDR Mellanox ConnectX adapters (Mellanox
> MT26418, ConnectX IB DDR, PCIe 2.0 5GT/s). We use SLES10 SP2 for x86_64,
> and OFED 1.3 built by SGI.
>
> When calculating a particular time series with cp2k, the run aborts
> after about 30 timesteps (takes a couple of hours by using 128 MPI
> tasks) with the mvapich2 message
>
> Warning! Rndv Receiver is receiving (49152 < 110592) less than as
> expected
>
> with varying recv/expected message sizes.
>
> The simulation step where this message appears is not deterministic. The
> only thing I know is that it appears for sure between the 27th and the
> 32rd time step after a restart. Sometimes a run aborts with SIGSEGV.
> Sometimes the run only hangs until running into some time limit. The
> problem cannot be reproduced by starting from restart files written by
> cp2k inbetween the runs. One always has to wait about 30 time steps. So
> it does not seem to depend on the input data, but on wallclock time.
>
> The problem does not occur when using another MPI lib. I successfully
> tried SGI MPT v1.22, Intel MPI v3.1.026, MPICH2 v1.0.7 and v1.0.8 (using
> IPoIB). It only shows up with MVAPICH2. I tried v1.2.0, v1.2p1 from dev
> snapshot 2009-03-02, v1.2rc1, v1.2rc2. Failures occur independently from
> the compiler and optimization level used for the executable and
> self-compiled libs. I tried Intel and gcc, -O0 .. -O2. Failures are also
> independent from the blacs/scalapack/blas/lapack combination (I used
> self-compiled with Intel compilers or gcc, or what is shipped with MKL
> v10 or 11), and are not related to the use of shared vs. static libs. No
> special mvapich2 tuning environment variables are set.
>
> The only thing I figured out so far is that the warning messages and
> following aborts seem to originate from MPI communication initiated by
> blacs routines. A recent call stack trace produced by the Intel Fortran
> RTE while running an executable with blacs debugging on, and using
> static mpi/blacs/scalapack/mkl libs gives:
>
> Warning! Rndv Receiver is receiving (49152 < 110592) less than as expected
> BLACS ERROR 'MPI error 480874510 on call to MPI_Recv'
> from {-1,-1}, pnum=15, Contxt=-1, on line 8 of file 'BI_Srecv.c'.
>
> application called MPI_Abort(MPI_COMM_WORLD, 1) - process 15
> forrtl: error (78): process killed (SIGTERM)
> Image              PC                Routine            Line        Source
> cp2k.popt          0000000001A7BB4D  Unknown               Unknown  Unknown
> cp2k.popt          0000000001A6B48A  Unknown               Unknown  Unknown
> cp2k.popt          0000000001A6A634  Unknown               Unknown  Unknown
> cp2k.popt          0000000001A5B701  Unknown               Unknown  Unknown
> cp2k.popt          00000000011980E0  BI_Srecv                    8  BI_Srecv.c
> cp2k.popt          0000000001197AE5  BI_IdringBR                12  BI_IdringBR.c
> cp2k.popt          0000000001193ACE  Czgebr2d                  192  zgebr2d_.c
> ...
> forrtl: error (78): process killed (SIGTERM)
> Image              PC                Routine            Line        Source
> libmlx4-rdmav2.so  00002AF569D3414D  Unknown               Unknown  Unknown
> ...
>
>
> Another run on the same input data by using the same exe and same task
> geom gave
>
> Warning! Rndv Receiver is receiving (512 < 221184) less than as expected
> forrtl: severe (174): SIGSEGV, segmentation fault occurred
> Image              PC                Routine            Line        Source
> cp2k.popt          0000000001A709D2  Unknown               Unknown  Unknown
> cp2k.popt          0000000001A6D62D  Unknown               Unknown  Unknown
> cp2k.popt          0000000001A6A5FC  Unknown               Unknown  Unknown
> cp2k.popt          0000000001A5B701  Unknown               Unknown  Unknown
> cp2k.popt          00000000011980E0  BI_Srecv                    8  BI_Srecv.c
> cp2k.popt          0000000001197AE5  BI_IdringBR                12  BI_IdringBR.c
> cp2k.popt          00000000011936E5  Cdgebr2d                  192  dgebr2d_.c
> ...
> forrtl: error (78): process killed (SIGTERM)
> Image              PC                Routine            Line        Source
> libmlx4-rdmav2.so  00002AEF72F07A00  Unknown               Unknown  Unknown
> ...
>
>
> Currently I have no idea how to nail down the problem to a particular
> mvapich2 issue. The time needed to reach the point of failure, the large
> number of tasks, the complexity of the numerics, and the apparent
> dependency on some runtime component make it hard to debug this.
>
> Please let me know if you need further information.
>
> Sincerely, BK
>
> --
> Dr. Bernd Kallies
> Konrad-Zuse-Zentrum für Informationstechnik Berlin
> Takustr. 7
> 14195 Berlin
> Tel: +49-30-84185-270
> Fax: +49-30-84185-311
> e-mail: kallies at zib.de
>
>
> _______________________________________________
> mvapich-discuss mailing list
> mvapich-discuss at cse.ohio-state.edu
> http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss
>