[mvapich-discuss] Consistent and reproducible vbuf problems (fwd)

[wei huang]

Hi Jeff,

Would you please let us know the size of the job you are running (number
of processes)? And are you running your broadcasts from the same user
buffer or different ones? Also, could you please verify the memory load
one the system when the failure happens? Thanks.

Regards,
Wei Huang

774 Dreese Lab, 2015 Neil Ave,
Dept. of Computer Science and Engineering
Ohio State University
OH 43210
Tel: (614)292-8501


On Sun, 6 Apr 2008, Dhabaleswar Panda wrote:

> I am using MVAPICH2 on a small set of workstations equipped with infiniband.
> I am using a GPU device library known as CUDA. CUDA uses page-locked memory
> areas, and I believe this is conflicting with MVAPICH2. If I run a series of
> broadcasts of size (1024, 2048, ..., 2MB) and run each size of broadcast a
> number of times (30 seems to work), nodes consistenly abort. A typical call
> stack:
>
> 2:  0x00002adab034db90 in memset () from /lib/libc.so.6
> 2:  (gdb) 2:  (gdb) bt
> 2:  #0  0x00002adab034db90 in memset () from /lib/libc.so.6
> 2:  #1  0x0000000000443bdd in allocate_vbuf_region ()
> 2:  #2  0x0000000000443fe5 in get_vbuf ()
> 2:  #3  0x000000000043716f in MRAILI_Get_Vbuf ()
> 2:  #4  0x000000000043739e in MPIDI_CH3I_MRAILI_Eager_send ()
> 2:  #5  0x000000000042f483 in MPIDI_CH3_Rendezvous_r3_push ()
> 2:  #6  0x000000000042f721 in MPIDI_CH3_Rendezvous_push ()
> 2:  #7  0x000000000042f9b1 in MPIDI_CH3I_MRAILI_Process_rndv ()
> 2:  #8  0x000000000042d904 in MPIDI_CH3I_Progress ()
> 2:  #9  0x0000000000420b6e in MPIC_Wait ()
> 2:  #10 0x00000000004215b9 in MPIC_Send ()
> 2:  #11 0x000000000041f62e in MPIR_Bcast ()
> 2:  #12 0x000000000042097e in PMPI_Bcast ()
> 2:  #13 0x000000000041dff9 in
> dcgn::BroadcastRequest::performCollectiveGlobal (
> 2:      this=0xc5c6f0) at infrastructure/src/BroadcastRequest.cpp:18
> 2:  #14 0x000000000041e7f0 in dcgn::CollectiveRequest::poll (this=0xc5c6f0,
> 2:      ioRequests=@0x5f3708) at infrastructure/src/CollectiveRequest.cpp:75
> 2:  #15 0x000000000041e75c in dcgn::CollectiveRequest::poll (this=0xc5c6f0,
> 2:      ioRequests=@0x5f3708) at infrastructure/src/CollectiveRequest.cpp:84
> 2:  #16 0x00000000004146e3 in dcgn::MPIWorker::serviceRequest
> (this=0x5f3680,
> 2:      req=0xc5c6f0, isShutdown=<value optimized out>)
> 2:      at infrastructure/src/MPIWorker.cpp:118
> 2:  #17 0x000000000041499f in dcgn::MPIWorker::loop (this=0x5f3680)
> 2:      at infrastructure/src/MPIWorker.cpp:78
> 2:  #18 0x0000000000415986 in dcgn::MPIWorker::launchThread (
> 2:      param=<value optimized out>) at infrastructure/src/MPIWorker.cpp:221
> 2:  #19 0x000000000041ce1a in dcgn::Thread::run (p=0x801de0)
> 2:      at infrastructure/src/Thread.cpp:18
> 2:  #20 0x00002adaaf78e297 in start_thread () from /lib/libpthread.so.0
> 2:  #21 0x00002adab039a51e in clone () from /lib/libc.so.6
> 2:  (gdb) up 13
> 2:  #13 0x000000000041dff9 in
> dcgn::BroadcastRequest::performCollectiveGlobal (
> 2:      this=0xc5c6f0) at infrastructure/src/BroadcastRequest.cpp:18
> 2:  18      MPI_Bcast(buf, numBytes, MPI_BYTE,
> mpiWorker->getMPIRankByTarget(root), MPI_COMM_WORLD);
> 2:  Current language:  auto; currently c++
> 2:  (gdb) print numBytes
> 2:  $1 = 2097152
>
>
> Before this crash, successful broadcasts were performed with sizes 1K, 2K,
> all the way to 1MB. The next broadcast is a 2MB broadcast and causes this
> crash. I am not sure if any 2MB broadcasts were performed before this, or if
> the first 2MB broadcast is what fails.
>
> Is there any way to either disable VBUF usage (I know this will cause
> performance degredation, but then again, slow performance is better than no
> performance) or limit vbuf usage? I tried limiting the number of vbufs to
> 1024 with a size of 16K (mpiexec -gdb -env MV2_VBUF_MAX 1024 -env
> MV2_VBUF_TOTAL_SIZE 16384) and I end up with
>
> 0:  [0] Abort: VBUF alloc failure, limit exceeded at line 138 in file vbuf.c
>