[mvapich-discuss] [cuda_stage_free] cudaMemcpy failed with 11 at 1564

[Maxim Milakov]

Hi all,

I am building and running code with MVAPICH 1.9a. The code is built with no CUDA support - it is not using CUDA at all.

Still MVAPICH2 is configured with CUDA support:

CPPFLAGS="-D x86_64 -D__align__\(n\)=__attribute__\(\(aligned\(n\)\)\) -D__location__\(a\)=__annotate__\(a\) -DCUDARTAPI=" F77=pgf77 FC=pgf90 CC=pgcc CXX=pgcpp ./configure --prefix=/usr/local/pgi_specific/mvapich2 --enable-f77 --enable-fc --enable-cxx --enable-cuda --with-cuda=$CUDA_INSTALL_PATH --disable-mcast

Whenever I run the code with no specific options I get no errors, the code runs fine, showing significant performance improvement on configurations with IB (in comparison with MPICH).

If I set MV2_USE_CUDA=1 (I tried it as the next step would be to run the version of the code ported to GPU) then the code aborts with the following error message:

[wm040:mpi_rank_0][cuda_stage_free] cudaMemcpy failed with 11 at 1564
[wm040:mpi_rank_1][cuda_stage_free] cudaMemcpy failed with 11 at 1564

It seems that the MVAPICH2 falsely detects that the pointer passed to mpi_send is referencing GPU memory.

I am using CUDA 5.0, still I experience the error when I use CUDA 4.2 (with new driver from installed with CUDA 5.0).
The error occurs even if the code runs just a single rank on each node.

I found similar issue here, with no resolution: http://mail.cse.ohio-state.edu/pipermail/mvapich-discuss/2012-August/004014.html

Thank you.


Best regards,
Maxim Milakov


-----------------------------------------------------------------------------------
This email message is for the sole use of the intended recipient(s) and may contain
confidential information.  Any unauthorized review, use, disclosure or distribution
is prohibited.  If you are not the intended recipient, please contact the sender by
reply email and destroy all copies of the original message.
-----------------------------------------------------------------------------------
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.cse.ohio-state.edu/pipermail/mvapich-discuss/attachments/20121106/6a66f4ee/attachment.html