[Mvapich-discuss] Possible bug in OSU Micro-Benchmarks 5.7.1 with cuda

[Goldman, Adam]

Hello,

While running the latest osu latency (5.7.1) benchmark with cuda support enabled, we encountered a possible bug in the OSU benchmark code.
It appears that when running osu_latency with one side using "MH" and the other "H", the non-cuda managed side will attempt to call cuda calls above message size 131072.

I am using OpenMPI v4.1.1 compiled with cuda support on RHEL 8.1

# mpirun -np 2 --host host1,host2 ./mpi/pt2pt/osu_latency -m 131072: MH H
...
131072                512.90
[../../util/osu_util_mpi.c:1691] CUDA call 'cudaMemPrefetchAsync(buf, length, devid, um_stream)' failed with 1: invalid argument

>From some debugging, it appears to be passing in a pointer to code allocated without cuda calls on the node that is not using cuda.
This issue appears to be new to v5.7.1. 

Not sure if this is the fix, but this seemed to fix the issue on osu_latency.c:
================================
@@ -134,9 +134,9 @@

         for(i = 0; i < options.iterations + options.skip; i++) {
 #ifdef _ENABLE_CUDA_
-            if (options.src == 'M') {
+            if (myid == 0) {
                 touch_managed_src(s_buf, size);
-            } else if (options.dst == 'M') {
+            } else {
                 touch_managed_dst(s_buf, size);
             }
 #endif
@@ -149,8 +149,8 @@
                 MPI_CHECK(MPI_Send(s_buf, size, MPI_CHAR, 1, 1, MPI_COMM_WORLD));
                 MPI_CHECK(MPI_Recv(r_buf, size, MPI_CHAR, 1, 1, MPI_COMM_WORLD, &reqstat));
 #ifdef _ENABLE_CUDA_
-                if (options.src == 'M') {
-                    touch_managed_src(r_buf, size);
-                }
+                touch_managed_src(r_buf, size);
 #endif

@@ -161,9 +161,7 @@
             } else if (myid == 1) {
                 MPI_CHECK(MPI_Recv(r_buf, size, MPI_CHAR, 0, 1, MPI_COMM_WORLD, &reqstat));
 #ifdef _ENABLE_CUDA_
-                if (options.dst == 'M') {
-                    touch_managed_dst(r_buf, size);
-                }
+                touch_managed_dst(r_buf, size);
 #endif

                 MPI_CHECK(MPI_Send(s_buf, size, MPI_CHAR, 0, 1, MPI_COMM_WORLD));
================================
Only the 1st change is required, but the last 2 are more just cleanups to avoid calling the same if expression twice. Once outside and once inside the functions.

Thank you,

Adam Goldman
HPC Fabric Software Engineer
Intel Corporation
adam.goldman at intel.com