[mvapich-discuss] Interleaved Isend/Irecv with CUDA vs. asynchronous

[Jens Glaser]

Hi all,

I am using MVAPICH2 1.9a. In my MPI cuda application, I have pairs non-blocking Isend/Irecvs. When visualizing profiling output (using VampirTrace) it appears that during the call to MPI_Waitall(),
the send/recv operations still seem to be interleaved, e.g. process 0 waits to recceive data from process 1 before it fill its send buffer using a cudaMemcpyAsync.
I tried to set MPICH_ASYNC_PROGRESS=1 and MV2_ENABLE_AFFINITY=0, but then I am getting the following error

[2] Abort: cudaMemcpyAsync from device to host failed
 at line 2244 in file ch3_smp_progress.c

Is asynchronous Isend/Irecv not completely supported by MVAPICH2?

Jens