<div dir="ltr">Hi DK,<div><br></div><div>Do the CUDA and GCC versions on our system need to match the RPM exactly? We are running on Ubuntu, and there is no GCC 8.4.1 on Ubuntu.</div><div><br></div><div>Thank you,</div><div>John</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Jan 11, 2022 at 2:55 PM Panda, Dhabaleswar <<a href="mailto:panda@cse.ohio-state.edu">panda@cse.ohio-state.edu</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hi,<br>
<br>
Thanks for your note. For GPU support with MVAPICH2, it is strongly recommended to use the MVAPICH2-GDR package. This package supports many features related to GPUs and delivers the best performance and scalability on GPU clusters. Please use a suitable RPM package from the MVAPICH2-GDR download page for your system. Please refer to the corresponding user guide also. The MVAPICH2-GDR package can also be installed through Spack. Let us know if you experience any issues in using the MVAPICH2-GDR package on your GPU cluster.<br>
<br>
Thanks,<br>
<br>
DK<br>
<br>
<br>
________________________________________<br>
From: Mvapich-discuss <mvapich-discuss-bounces+panda.2=<a href="mailto:osu.edu@lists.osu.edu" target="_blank">osu.edu@lists.osu.edu</a>> on behalf of John Moore via Mvapich-discuss <<a href="mailto:mvapich-discuss@lists.osu.edu" target="_blank">mvapich-discuss@lists.osu.edu</a>><br>
Sent: Tuesday, January 11, 2022 2:48 PM<br>
To: <a href="mailto:mvapich-discuss@lists.osu.edu" target="_blank">mvapich-discuss@lists.osu.edu</a><br>
Cc: Maitham Alhubail<br>
Subject: [Mvapich-discuss] MVAPICH2 GDR from source code?<br>
<br>
Hello,<br>
<br>
We have been struggling to get MVAPICH2 to work with cuda-aware support and RDMA. We have compiled MVAPICH2 from source, with the --enable-cuda option, but when we run the osu_bibw bandwidth test using Device to Device communication, we get a segmentation fault.<br>
<br>
Below is the output from osu_bibw using MVAPICH2:<br>
MVAPICH2-2.3.6 Parameters<br>
---------------------------------------------------------------------<br>
PROCESSOR ARCH NAME : MV2_ARCH_AMD_EPYC_7401_48<br>
PROCESSOR FAMILY NAME : MV2_CPU_FAMILY_AMD<br>
PROCESSOR MODEL NUMBER : 1<br>
HCA NAME : MV2_HCA_MLX_CX_HDR<br>
HETEROGENEOUS HCA : NO<br>
MV2_EAGERSIZE_1SC : 0<br>
MV2_SMP_EAGERSIZE : 16385<br>
MV2_SMP_QUEUE_LENGTH : 65536<br>
MV2_SMP_NUM_SEND_BUFFER : 16<br>
MV2_SMP_BATCH_SIZE : 8<br>
Tuning Table: : MV2_ARCH_AMD_EPYC_7401_48 MV2_HCA_MLX_CX_HDR<br>
---------------------------------------------------------------------<br>
# OSU MPI-CUDA Bi-Directional Bandwidth Test v5.7.1<br>
# Send Buffer on DEVICE (D) and Receive Buffer on DEVICE (D)<br>
# Size Bandwidth (MB/s)<br>
1 0.07<br>
2 0.15<br>
4 0.29<br>
8 0.57<br>
16 1.12<br>
32 2.30<br>
64 4.75<br>
128 9.41<br>
256 18.44<br>
512 37.22<br>
1024 74.82<br>
2048 144.70<br>
4096 289.96<br>
8192 577.33<br>
[cell3:mpi_rank_0][error_sighandler] Caught error: Segmentation fault (signal 11)<br>
[cell3:mpi_rank_1][error_sighandler] Caught error: Segmentation fault (signal 11)<br>
<br>
===================================================================================<br>
= BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES<br>
= PID 471850 RUNNING AT cell3<br>
= EXIT CODE: 139<br>
= CLEANING UP REMAINING PROCESSES<br>
= YOU CAN IGNORE THE BELOW CLEANUP MESSAGES<br>
===================================================================================<br>
And this is with OpenMPI:<br>
# OSU MPI-CUDA Bi-Directional Bandwidth Test v5.8<br>
# Send Buffer on DEVICE (D) and Receive Buffer on DEVICE (D)<br>
# Size Bandwidth (MB/s)<br>
1 0.43<br>
2 0.83<br>
4 1.68<br>
8 3.37<br>
16 6.72<br>
32 13.42<br>
64 27.02<br>
128 53.78<br>
256 107.88<br>
512 219.45<br>
1024 437.81<br>
2048 875.12<br>
4096 1747.23<br>
8192 3528.97<br>
16384 7015.15<br>
32768 13973.59<br>
65536 27702.68<br>
131072 51877.67<br>
262144 94556.99<br>
524288 157755.18<br>
1048576 236772.67<br>
2097152 333635.13<br>
4194304 408865.93<br>
<br>
<br>
Can GDR support be obtained by compiling from source like we are trying to do or do we have to use an RPM? We export MV2_USE_CUDA=1. Any recommendations would be greatly appreciated.<br>
<br>
Thanks,<br>
John<br>
</blockquote></div>