[mvapich-discuss] Equivalence of OMPI's mpi_leave_pinned parameter

[Jens Glaser]

(copying list, forgot that)

Hi Sreeram,

just tried out the parameter you mentioned. It results in a direct crash of my code.
The crash happens in an MPI_Isend/Irecv pair of two processes actually sending CUDA buffers (ok, this is not directly related to page-locked memory, then).
The exchange is mutual, i.e both process send and receive data to and from each other simultaneously.
Process 0 sends from its buffer A to buffer B of process 1.
Process 1 sends from its buffer A to buffer B of process 0.

This is the error message of the process that aborts during MPI_Wait:

[0->3] send desc error, wc_opcode=0
[0->3] wc.status=9, wc.wr_id=0x2411ed8, wc.opcode=0, vbuf->phead->type=21 = MPIDI_CH3_PKT_RNDV_R3_DATA
[2] Abort: [] Got completion with error 9, vendor code=0x8a, dest rank=3
 at line 586 in file ibv_channel_manager.c

Quite a message :)

On the other side, MPI_Wait() returns error 700, which I looked up using MPI_Error_string:

Unknown error class, error stack:
(unknown)(): Insufficient ports in active state.

Any clues? Or am I doing something wrong?

Jens

On Nov 3, 2012, at 11:41 PM, sreeram potluri wrote:

> Hi Jens, 
> 
> This is interesting. From what we have seen, registration in MVAPICH2 should not interfere with any page-locked buffers you use in the application. Can you explain your scenario a bit more or give a test case where you see such a failures?
> 
> MVAPICH2 does provide an option to disable registration cache by setting the parameter MV2_USE_LAZY_MEM_UNREGISTER to 0.  But this can have a negative impact on the application's performance. We would prefer we get to the root cause of the issue and resolve it if possible. 
> 
> Sreeram Potluri
> 
> On Sat, Nov 3, 2012 at 11:51 PM, Jens Glaser <jglaser at umn.edu> wrote:
> Hi,
> 
> I am developing an MPI/CUDA application which uses both page-locked host buffers and CUDA buffers for communication.
> Using CUDA buffers directly with MVAPICH2 works great, however I had a terrible time tracking down the occurrence of bad
> program behavior when using also page-locked memory (allocated using cudaHostAlloc). It seem's MVAPICH2's internal
> handling of these buffers interferes with device-host RDMA (GPUDirect), if the latter is not left entirely to the library.
> The same is true for OpenMPI, but fortunately OpenMPI has an MCA option mpi_leave_pinned, which can be turned off, and which in turn prevents the crashes.
> Also, the OpenMPI docs state that "other" MPI libraries have this parameter turned on by default (including MVAPICH2, I suspect).
> 
> Since I appreciate MVAPICH2's CUDA capabilities, I wonder if the library offers a similar way of turning off the optimizations that lead to my application
> failing.
> 
> Thanks
> Jens
> 
> 
> _______________________________________________
> mvapich-discuss mailing list
> mvapich-discuss at cse.ohio-state.edu
> http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss
> 
> 
> 


-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.cse.ohio-state.edu/pipermail/mvapich-discuss/attachments/20121104/d4e6c5d6/attachment.html