[mvapich-discuss] Equivalence of OMPI's mpi_leave_pinned parameter

[sreeram potluri]

Hi Jens,

This is interesting. From what we have seen, registration in MVAPICH2
should not interfere with any page-locked buffers you use in the
application. Can you explain your scenario a bit more or give a test case
where you see such a failures?

MVAPICH2 does provide an option to disable registration cache by setting
the parameter MV2_USE_LAZY_MEM_UNREGISTER to 0.  But this can have a
negative impact on the application's performance. We would prefer we get to
the root cause of the issue and resolve it if possible.

Sreeram Potluri

On Sat, Nov 3, 2012 at 11:51 PM, Jens Glaser <jglaser at umn.edu> wrote:

> Hi,
>
> I am developing an MPI/CUDA application which uses both page-locked host
> buffers and CUDA buffers for communication.
> Using CUDA buffers directly with MVAPICH2 works great, however I had a
> terrible time tracking down the occurrence of bad
> program behavior when using also page-locked memory (allocated using
> cudaHostAlloc). It seem's MVAPICH2's internal
> handling of these buffers interferes with device-host RDMA (GPUDirect), if
> the latter is not left entirely to the library.
> The same is true for OpenMPI, but fortunately OpenMPI has an MCA option
> mpi_leave_pinned, which can be turned off, and which in turn prevents the
> crashes.
> Also, the OpenMPI docs state that "other" MPI libraries have this
> parameter turned on by default (including MVAPICH2, I suspect).
>
> Since I appreciate MVAPICH2's CUDA capabilities, I wonder if the library
> offers a similar way of turning off the optimizations that lead to my
> application
> failing.
>
> Thanks
> Jens
>
>
> _______________________________________________
> mvapich-discuss mailing list
> mvapich-discuss at cse.ohio-state.edu
> http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.cse.ohio-state.edu/pipermail/mvapich-discuss/attachments/20121104/2b4d1073/attachment.html