<div dir="ltr">Hello,<div><br></div><div>We have been struggling to get MVAPICH2 to work with cuda-aware support and RDMA. We have compiled MVAPICH2 from source, with the --enable-cuda option, but when we run the osu_bibw bandwidth test using Device to Device communication, we get a segmentation fault.</div><div><br></div><div>Below is the output from osu_bibw using MVAPICH2:</div><div> MVAPICH2-2.3.6 Parameters<br>---------------------------------------------------------------------<br>        PROCESSOR ARCH NAME            : MV2_ARCH_AMD_EPYC_7401_48<br>        PROCESSOR FAMILY NAME          : MV2_CPU_FAMILY_AMD<br>        PROCESSOR MODEL NUMBER         : 1<br>        HCA NAME                       : MV2_HCA_MLX_CX_HDR<br>        HETEROGENEOUS HCA              : NO<br>        MV2_EAGERSIZE_1SC              : 0<br>        MV2_SMP_EAGERSIZE              : 16385<br>        MV2_SMP_QUEUE_LENGTH           : 65536<br>        MV2_SMP_NUM_SEND_BUFFER        : 16<br>        MV2_SMP_BATCH_SIZE             : 8<br>        Tuning Table:                  : MV2_ARCH_AMD_EPYC_7401_48 MV2_HCA_MLX_CX_HDR<br>---------------------------------------------------------------------<br># OSU MPI-CUDA Bi-Directional Bandwidth Test v5.7.1<br># Send Buffer on DEVICE (D) and Receive Buffer on DEVICE (D)<br># Size      Bandwidth (MB/s)<br>1                       0.07<br>2                       0.15<br>4                       0.29<br>8                       0.57<br>16                      1.12<br>32                      2.30<br>64                      4.75<br>128                     9.41<br>256                    18.44<br>512                    37.22<br>1024                   74.82<br>2048                  144.70<br>4096                  289.96<br>8192                  577.33<br>[cell3:mpi_rank_0][error_sighandler] Caught error: Segmentation fault (signal 11)<br>[cell3:mpi_rank_1][error_sighandler] Caught error: Segmentation fault (signal 11)<br><br>===================================================================================<br>=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES<br>=   PID 471850 RUNNING AT cell3<br>=   EXIT CODE: 139<br>=   CLEANING UP REMAINING PROCESSES<br>=   YOU CAN IGNORE THE BELOW CLEANUP MESSAGES<br>===================================================================================<br></div><div>And this is with OpenMPI:</div><div><span style="color:rgb(29,28,29);font-family:Slack-Lato,appleLogo,sans-serif;font-size:15px;font-variant-ligatures:common-ligatures;background-color:rgb(248,248,248)"># OSU MPI-CUDA Bi-Directional Bandwidth Test v5.8</span><br style="box-sizing:inherit;color:rgb(29,28,29);font-family:Slack-Lato,appleLogo,sans-serif;font-size:15px;font-variant-ligatures:common-ligatures;background-color:rgb(248,248,248)"><span style="color:rgb(29,28,29);font-family:Slack-Lato,appleLogo,sans-serif;font-size:15px;font-variant-ligatures:common-ligatures;background-color:rgb(248,248,248)"># Send Buffer on DEVICE (D) and Receive Buffer on DEVICE (D)</span><br style="box-sizing:inherit;color:rgb(29,28,29);font-family:Slack-Lato,appleLogo,sans-serif;font-size:15px;font-variant-ligatures:common-ligatures;background-color:rgb(248,248,248)"><span style="color:rgb(29,28,29);font-family:Slack-Lato,appleLogo,sans-serif;font-size:15px;font-variant-ligatures:common-ligatures;background-color:rgb(248,248,248)"># Size      Bandwidth (MB/s)</span><br style="box-sizing:inherit;color:rgb(29,28,29);font-family:Slack-Lato,appleLogo,sans-serif;font-size:15px;font-variant-ligatures:common-ligatures;background-color:rgb(248,248,248)"><span style="color:rgb(29,28,29);font-family:Slack-Lato,appleLogo,sans-serif;font-size:15px;font-variant-ligatures:common-ligatures;background-color:rgb(248,248,248)">1                       0.43</span><br style="box-sizing:inherit;color:rgb(29,28,29);font-family:Slack-Lato,appleLogo,sans-serif;font-size:15px;font-variant-ligatures:common-ligatures;background-color:rgb(248,248,248)"><span style="color:rgb(29,28,29);font-family:Slack-Lato,appleLogo,sans-serif;font-size:15px;font-variant-ligatures:common-ligatures;background-color:rgb(248,248,248)">2                       0.83</span><br style="box-sizing:inherit;color:rgb(29,28,29);font-family:Slack-Lato,appleLogo,sans-serif;font-size:15px;font-variant-ligatures:common-ligatures;background-color:rgb(248,248,248)"><span style="color:rgb(29,28,29);font-family:Slack-Lato,appleLogo,sans-serif;font-size:15px;font-variant-ligatures:common-ligatures;background-color:rgb(248,248,248)">4                       1.68</span><br style="box-sizing:inherit;color:rgb(29,28,29);font-family:Slack-Lato,appleLogo,sans-serif;font-size:15px;font-variant-ligatures:common-ligatures;background-color:rgb(248,248,248)"><span style="color:rgb(29,28,29);font-family:Slack-Lato,appleLogo,sans-serif;font-size:15px;font-variant-ligatures:common-ligatures;background-color:rgb(248,248,248)">8                       3.37</span><br style="box-sizing:inherit;color:rgb(29,28,29);font-family:Slack-Lato,appleLogo,sans-serif;font-size:15px;font-variant-ligatures:common-ligatures;background-color:rgb(248,248,248)"><span style="color:rgb(29,28,29);font-family:Slack-Lato,appleLogo,sans-serif;font-size:15px;font-variant-ligatures:common-ligatures;background-color:rgb(248,248,248)">16                      6.72</span><br style="box-sizing:inherit;color:rgb(29,28,29);font-family:Slack-Lato,appleLogo,sans-serif;font-size:15px;font-variant-ligatures:common-ligatures;background-color:rgb(248,248,248)"><span style="color:rgb(29,28,29);font-family:Slack-Lato,appleLogo,sans-serif;font-size:15px;font-variant-ligatures:common-ligatures;background-color:rgb(248,248,248)">32                     13.42</span><br style="box-sizing:inherit;color:rgb(29,28,29);font-family:Slack-Lato,appleLogo,sans-serif;font-size:15px;font-variant-ligatures:common-ligatures;background-color:rgb(248,248,248)"><span style="color:rgb(29,28,29);font-family:Slack-Lato,appleLogo,sans-serif;font-size:15px;font-variant-ligatures:common-ligatures;background-color:rgb(248,248,248)">64                     27.02</span><br style="box-sizing:inherit;color:rgb(29,28,29);font-family:Slack-Lato,appleLogo,sans-serif;font-size:15px;font-variant-ligatures:common-ligatures;background-color:rgb(248,248,248)"><span style="color:rgb(29,28,29);font-family:Slack-Lato,appleLogo,sans-serif;font-size:15px;font-variant-ligatures:common-ligatures;background-color:rgb(248,248,248)">128                    53.78</span><br style="box-sizing:inherit;color:rgb(29,28,29);font-family:Slack-Lato,appleLogo,sans-serif;font-size:15px;font-variant-ligatures:common-ligatures;background-color:rgb(248,248,248)"><span style="color:rgb(29,28,29);font-family:Slack-Lato,appleLogo,sans-serif;font-size:15px;font-variant-ligatures:common-ligatures;background-color:rgb(248,248,248)">256                   107.88</span><br style="box-sizing:inherit;color:rgb(29,28,29);font-family:Slack-Lato,appleLogo,sans-serif;font-size:15px;font-variant-ligatures:common-ligatures;background-color:rgb(248,248,248)"><span style="color:rgb(29,28,29);font-family:Slack-Lato,appleLogo,sans-serif;font-size:15px;font-variant-ligatures:common-ligatures;background-color:rgb(248,248,248)">512                   219.45</span><br style="box-sizing:inherit;color:rgb(29,28,29);font-family:Slack-Lato,appleLogo,sans-serif;font-size:15px;font-variant-ligatures:common-ligatures;background-color:rgb(248,248,248)"><span style="color:rgb(29,28,29);font-family:Slack-Lato,appleLogo,sans-serif;font-size:15px;font-variant-ligatures:common-ligatures;background-color:rgb(248,248,248)">1024                  437.81</span><br style="box-sizing:inherit;color:rgb(29,28,29);font-family:Slack-Lato,appleLogo,sans-serif;font-size:15px;font-variant-ligatures:common-ligatures;background-color:rgb(248,248,248)"><span style="color:rgb(29,28,29);font-family:Slack-Lato,appleLogo,sans-serif;font-size:15px;font-variant-ligatures:common-ligatures;background-color:rgb(248,248,248)">2048                  875.12</span><br style="box-sizing:inherit;color:rgb(29,28,29);font-family:Slack-Lato,appleLogo,sans-serif;font-size:15px;font-variant-ligatures:common-ligatures;background-color:rgb(248,248,248)"><span style="color:rgb(29,28,29);font-family:Slack-Lato,appleLogo,sans-serif;font-size:15px;font-variant-ligatures:common-ligatures;background-color:rgb(248,248,248)">4096                 1747.23</span><br style="box-sizing:inherit;color:rgb(29,28,29);font-family:Slack-Lato,appleLogo,sans-serif;font-size:15px;font-variant-ligatures:common-ligatures;background-color:rgb(248,248,248)"><span style="color:rgb(29,28,29);font-family:Slack-Lato,appleLogo,sans-serif;font-size:15px;font-variant-ligatures:common-ligatures;background-color:rgb(248,248,248)">8192                 3528.97</span><br style="box-sizing:inherit;color:rgb(29,28,29);font-family:Slack-Lato,appleLogo,sans-serif;font-size:15px;font-variant-ligatures:common-ligatures;background-color:rgb(248,248,248)"><span style="color:rgb(29,28,29);font-family:Slack-Lato,appleLogo,sans-serif;font-size:15px;font-variant-ligatures:common-ligatures;background-color:rgb(248,248,248)">16384                7015.15</span><br style="box-sizing:inherit;color:rgb(29,28,29);font-family:Slack-Lato,appleLogo,sans-serif;font-size:15px;font-variant-ligatures:common-ligatures;background-color:rgb(248,248,248)"><span style="color:rgb(29,28,29);font-family:Slack-Lato,appleLogo,sans-serif;font-size:15px;font-variant-ligatures:common-ligatures;background-color:rgb(248,248,248)">32768               13973.59</span><br style="box-sizing:inherit;color:rgb(29,28,29);font-family:Slack-Lato,appleLogo,sans-serif;font-size:15px;font-variant-ligatures:common-ligatures;background-color:rgb(248,248,248)"><span style="color:rgb(29,28,29);font-family:Slack-Lato,appleLogo,sans-serif;font-size:15px;font-variant-ligatures:common-ligatures;background-color:rgb(248,248,248)">65536               27702.68</span><br style="box-sizing:inherit;color:rgb(29,28,29);font-family:Slack-Lato,appleLogo,sans-serif;font-size:15px;font-variant-ligatures:common-ligatures;background-color:rgb(248,248,248)"><span style="color:rgb(29,28,29);font-family:Slack-Lato,appleLogo,sans-serif;font-size:15px;font-variant-ligatures:common-ligatures;background-color:rgb(248,248,248)">131072              51877.67</span><br style="box-sizing:inherit;color:rgb(29,28,29);font-family:Slack-Lato,appleLogo,sans-serif;font-size:15px;font-variant-ligatures:common-ligatures;background-color:rgb(248,248,248)"><span style="color:rgb(29,28,29);font-family:Slack-Lato,appleLogo,sans-serif;font-size:15px;font-variant-ligatures:common-ligatures;background-color:rgb(248,248,248)">262144              94556.99</span><br style="box-sizing:inherit;color:rgb(29,28,29);font-family:Slack-Lato,appleLogo,sans-serif;font-size:15px;font-variant-ligatures:common-ligatures;background-color:rgb(248,248,248)"><span style="color:rgb(29,28,29);font-family:Slack-Lato,appleLogo,sans-serif;font-size:15px;font-variant-ligatures:common-ligatures;background-color:rgb(248,248,248)">524288             157755.18</span><br style="box-sizing:inherit;color:rgb(29,28,29);font-family:Slack-Lato,appleLogo,sans-serif;font-size:15px;font-variant-ligatures:common-ligatures;background-color:rgb(248,248,248)"><span style="color:rgb(29,28,29);font-family:Slack-Lato,appleLogo,sans-serif;font-size:15px;font-variant-ligatures:common-ligatures;background-color:rgb(248,248,248)">1048576            236772.67</span><br style="box-sizing:inherit;color:rgb(29,28,29);font-family:Slack-Lato,appleLogo,sans-serif;font-size:15px;font-variant-ligatures:common-ligatures;background-color:rgb(248,248,248)"><span style="color:rgb(29,28,29);font-family:Slack-Lato,appleLogo,sans-serif;font-size:15px;font-variant-ligatures:common-ligatures;background-color:rgb(248,248,248)">2097152            333635.13</span><br style="box-sizing:inherit;color:rgb(29,28,29);font-family:Slack-Lato,appleLogo,sans-serif;font-size:15px;font-variant-ligatures:common-ligatures;background-color:rgb(248,248,248)"><span style="color:rgb(29,28,29);font-family:Slack-Lato,appleLogo,sans-serif;font-size:15px;font-variant-ligatures:common-ligatures;background-color:rgb(248,248,248)">4194304            408865.93</span><br></div><div><br></div><div><span style="color:rgb(29,28,29);font-family:Slack-Lato,appleLogo,sans-serif;font-size:15px;font-variant-ligatures:common-ligatures;background-color:rgb(248,248,248)"><br></span></div><div><font color="#1d1c1d" face="Slack-Lato, appleLogo, sans-serif"><span style="font-size:15px;font-variant-ligatures:common-ligatures;background-color:rgb(248,248,248)">Can GDR support be obtained by compiling from source like we are trying to do or do we have to use an RPM? We export MV2_USE_CUDA=1. Any recommendations would be greatly appreciated.</span></font></div><div><font color="#1d1c1d" face="Slack-Lato, appleLogo, sans-serif"><span style="font-size:15px;font-variant-ligatures:common-ligatures;background-color:rgb(248,248,248)"><br></span></font></div><div><font color="#1d1c1d" face="Slack-Lato, appleLogo, sans-serif"><span style="font-size:15px;font-variant-ligatures:common-ligatures;background-color:rgb(248,248,248)">Thanks,</span></font></div><div><font color="#1d1c1d" face="Slack-Lato, appleLogo, sans-serif"><span style="font-size:15px;font-variant-ligatures:common-ligatures;background-color:rgb(248,248,248)">John</span></font></div></div>