[mvapich-discuss] Equivalence of OMPI's mpi_leave_pinned parameter

[Jens Glaser]

Hi,

I am notifying the list that both problems have been fixed. The MVAPICH2 developers came up with a patch to fix the crash when simultaneously 
setting MV2_USE_LAZY_MEM_UNREGISTER=0 and MV2_USE_CUDA=1, and I suppose they will publicize it soon.

The original problem, undefined program behavior, was traced back to the use of cudaHostAlloc()/cudaFreeHost() in combination with registration cache. I modified my application
to use standard posix_memalign()/cudaHostRegister() calls instead. This way, the memory allocations are intercepted by the MPI library as they should, and my code works correctly.

Thanks,
Jens

On Nov 4, 2012, at 12:38 AM, Jens Glaser wrote:

> Hi,
> 
> brief update:
> 
> when using host buffers instead of device buffers, and your option (MV2_USE_LAZY_MEM_UNREGISTER=0),  the program runs just fine,
> as with OpenMPI.
> 
> I am really perplexed ....
> 
> Jens
> 
> On Nov 4, 2012, at 12:26 AM, Jens Glaser wrote:
> 
>> Hi Sreeram,
>> 
>> just tried out the parameter you mentioned. It results in a direct crash of my code.
>> The crash happens in an MPI_Isend/Irecv pair of two processes actually sending CUDA buffers (ok, this is not directly related to page-locked memory, then).
>> The exchange is mutual, i.e both process send and receive data to and from each other simultaneously.
>> Process 0 sends from its buffer A to buffer B of process 1.
>> Process 1 sends from its buffer A to buffer B of process 0.
>> 
>> This is the error message of the process that aborts during MPI_Wait:
>> 
>> [0->3] send desc error, wc_opcode=0
>> [0->3] wc.status=9, wc.wr_id=0x2411ed8, wc.opcode=0, vbuf->phead->type=21 = MPIDI_CH3_PKT_RNDV_R3_DATA
>> [2] Abort: [] Got completion with error 9, vendor code=0x8a, dest rank=3
>>  at line 586 in file ibv_channel_manager.c
>> 
>> Quite a message :)
>> 
>> On the other side, MPI_Wait() returns error 700, which I looked up using MPI_Error_string:
>> 
>> Unknown error class, error stack:
>> (unknown)(): Insufficient ports in active state.
>> 
>> Any clues? Or am I doing something wrong?
>> 
>> Jens
>> 
>> On Nov 3, 2012, at 11:41 PM, sreeram potluri wrote:
>> 
>>> Hi Jens, 
>>> 
>>> This is interesting. From what we have seen, registration in MVAPICH2 should not interfere with any page-locked buffers you use in the application. Can you explain your scenario a bit more or give a test case where you see such a failures?
>>> 
>>> MVAPICH2 does provide an option to disable registration cache by setting the parameter MV2_USE_LAZY_MEM_UNREGISTER to 0.  But this can have a negative impact on the application's performance. We would prefer we get to the root cause of the issue and resolve it if possible. 
>>> 
>>> Sreeram Potluri
>>> 
>>> On Sat, Nov 3, 2012 at 11:51 PM, Jens Glaser <jglaser at umn.edu> wrote:
>>> Hi,
>>> 
>>> I am developing an MPI/CUDA application which uses both page-locked host buffers and CUDA buffers for communication.
>>> Using CUDA buffers directly with MVAPICH2 works great, however I had a terrible time tracking down the occurrence of bad
>>> program behavior when using also page-locked memory (allocated using cudaHostAlloc). It seem's MVAPICH2's internal
>>> handling of these buffers interferes with device-host RDMA (GPUDirect), if the latter is not left entirely to the library.
>>> The same is true for OpenMPI, but fortunately OpenMPI has an MCA option mpi_leave_pinned, which can be turned off, and which in turn prevents the crashes.
>>> Also, the OpenMPI docs state that "other" MPI libraries have this parameter turned on by default (including MVAPICH2, I suspect).
>>> 
>>> Since I appreciate MVAPICH2's CUDA capabilities, I wonder if the library offers a similar way of turning off the optimizations that lead to my application
>>> failing.
>>> 
>>> Thanks
>>> Jens
>>> 
>>> 
>>> _______________________________________________
>>> mvapich-discuss mailing list
>>> mvapich-discuss at cse.ohio-state.edu
>>> http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss
>>> 
>>> 
>>> 
>> 
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.cse.ohio-state.edu/pipermail/mvapich-discuss/attachments/20121107/f78b08ab/attachment.html