<div dir="ltr">Hello,<div><br></div><div>While we wait for the RPM, we are trying to get regular MVAPICH2 to work across two of our nodes. We compiled version 2.3.6 from source. We can run the osu_bibw test locally, within a node without errors.<div><br></div><div>However, when we try to run across two nodes, we get the following error:</div></div><div><br></div><div><span style="color:rgb(29,28,29);font-family:Slack-Lato,appleLogo,sans-serif;font-size:15px;font-variant-ligatures:common-ligatures;background-color:rgb(248,248,248)">john@cell3:/shared_data/john_dev/osu-micro-benchmarks-5.8/mpi/pt2pt$ MV2_SMP_USE_CMA=0 mpirun -np 2 -hostfile hostfile ./osu_bibw</span><br style="box-sizing:inherit;color:rgb(29,28,29);font-family:Slack-Lato,appleLogo,sans-serif;font-size:15px;font-variant-ligatures:common-ligatures;background-color:rgb(248,248,248)"><span style="color:rgb(29,28,29);font-family:Slack-Lato,appleLogo,sans-serif;font-size:15px;font-variant-ligatures:common-ligatures;background-color:rgb(248,248,248)">[cell3:mpi_rank_0][rdma_param_handle_heterogeneity] All nodes involved in the job were detected to be homogeneous in terms of processors and interconnects. Setting MV2_HOMOGENEOUS_CLUSTER=1 can improve job startup performance on such systems. The following link has more details on enhancing job startup performance. </span><a target="_blank" class="gmail-c-link" href="http://mvapich.cse.ohio-state.edu/performance/job-startup/" rel="noopener noreferrer" tabindex="-1" style="box-sizing:inherit;text-decoration-line:none;font-family:Slack-Lato,appleLogo,sans-serif;font-size:15px;font-variant-ligatures:common-ligatures;background-color:rgb(248,248,248)">http://mvapich.cse.ohio-state.edu/performance/job-startup/</a><span style="color:rgb(29,28,29);font-family:Slack-Lato,appleLogo,sans-serif;font-size:15px;font-variant-ligatures:common-ligatures;background-color:rgb(248,248,248)">.</span><br style="box-sizing:inherit;color:rgb(29,28,29);font-family:Slack-Lato,appleLogo,sans-serif;font-size:15px;font-variant-ligatures:common-ligatures;background-color:rgb(248,248,248)"><span style="color:rgb(29,28,29);font-family:Slack-Lato,appleLogo,sans-serif;font-size:15px;font-variant-ligatures:common-ligatures;background-color:rgb(248,248,248)">[cell3:mpi_rank_0][rdma_param_handle_heterogeneity] To suppress this warning, please set MV2_SUPPRESS_JOB_STARTUP_PERFORMANCE_WARNING to 1</span><br style="box-sizing:inherit;color:rgb(29,28,29);font-family:Slack-Lato,appleLogo,sans-serif;font-size:15px;font-variant-ligatures:common-ligatures;background-color:rgb(248,248,248)"><span style="color:rgb(29,28,29);font-family:Slack-Lato,appleLogo,sans-serif;font-size:15px;font-variant-ligatures:common-ligatures;background-color:rgb(248,248,248)">[src/mpid/ch3/channels/mrail/src/gen2/vbuf.c 487] Cannot register vbuf region</span><br style="box-sizing:inherit;color:rgb(29,28,29);font-family:Slack-Lato,appleLogo,sans-serif;font-size:15px;font-variant-ligatures:common-ligatures;background-color:rgb(248,248,248)"><span style="color:rgb(29,28,29);font-family:Slack-Lato,appleLogo,sans-serif;font-size:15px;font-variant-ligatures:common-ligatures;background-color:rgb(248,248,248)">[cell3:mpi_rank_0][get_vbuf_by_offset] src/mpid/ch3/channels/mrail/src/gen2/vbuf.c:632: vbuf pool allocation failed: Cannot allocate memory (12)</span><br style="box-sizing:inherit;color:rgb(29,28,29);font-family:Slack-Lato,appleLogo,sans-serif;font-size:15px;font-variant-ligatures:common-ligatures;background-color:rgb(248,248,248)"><span style="color:rgb(29,28,29);font-family:Slack-Lato,appleLogo,sans-serif;font-size:15px;font-variant-ligatures:common-ligatures;background-color:rgb(248,248,248)">[src/mpid/ch3/channels/mrail/src/gen2/vbuf.c 487] Cannot register vbuf region</span><br style="box-sizing:inherit;color:rgb(29,28,29);font-family:Slack-Lato,appleLogo,sans-serif;font-size:15px;font-variant-ligatures:common-ligatures;background-color:rgb(248,248,248)"><span style="color:rgb(29,28,29);font-family:Slack-Lato,appleLogo,sans-serif;font-size:15px;font-variant-ligatures:common-ligatures;background-color:rgb(248,248,248)">[cell4:mpi_rank_1][get_vbuf_by_offset] src/mpid/ch3/channels/mrail/src/gen2/vbuf.c:632: vbuf pool allocation failed: Cannot allocate memory (12)</span><br></div><div><span style="color:rgb(29,28,29);font-family:Slack-Lato,appleLogo,sans-serif;font-size:15px;font-variant-ligatures:common-ligatures;background-color:rgb(248,248,248)"><br></span></div><div><span style="color:rgb(29,28,29);font-family:Slack-Lato,appleLogo,sans-serif;font-size:15px;font-variant-ligatures:common-ligatures;background-color:rgb(248,248,248)">We found some documentation that said this may be due to the value of the </span>log_num_mtt for OFED. We've found documentation for how to change this, and it involves changing the parameter in <span style="background-color:rgb(227,230,232);color:rgb(35,38,41);font-family:ui-monospace,"Cascadia Mono","Segoe UI Mono","Liberation Mono",Menlo,Monaco,Consolas,monospace;font-size:13px;white-space:pre-wrap">/etc/modprobe.d/mlx4_en.conf.</span></div><div><br></div><div><font color="#232629" face="ui-monospace, Cascadia Mono, Segoe UI Mono, Liberation Mono, Menlo, Monaco, Consolas, monospace"><span style="white-space:pre-wrap;background-color:rgb(255,255,255)">However, we do not have any mlx4_* under /etc/modprobe.d, only mlx5_. We are using</span><span style="white-space:pre-wrap;background-color:rgb(227,230,232)"> </span></font> MLNX_OFED_LINUX-5.5-1.0.3.2 as mentioned above. <span style="background-color:rgb(227,230,232);color:rgb(35,38,41);font-family:ui-monospace,"Cascadia Mono","Segoe UI Mono","Liberation Mono",Menlo,Monaco,Consolas,monospace;font-size:13px;white-space:pre-wrap"><br></span></div><div><br></div><div>The output for ulimit -a on both nodes is:</div><div><br>core file size (blocks, -c) 0<br>data seg size (kbytes, -d) unlimited<br>scheduling priority (-e) 0<br>file size (blocks, -f) unlimited<br>pending signals (-i) 4126989<br>max locked memory (kbytes, -l) 65536<br>max memory size (kbytes, -m) unlimited<br>open files (-n) 1024<br>pipe size (512 bytes, -p) 8<br>POSIX message queues (bytes, -q) 819200<br>real-time priority (-r) 0<br>stack size (kbytes, -s) 8192<br>cpu time (seconds, -t) unlimited<br>max user processes (-u) 4126989<br>virtual memory (kbytes, -v) unlimited<br>file locks (-x) unlimited<br></div><div><br></div><div>Any advice on how to resolve this error would be greatly appreciated.</div><div><br></div><div>Thanks,</div><div>John</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Jan 12, 2022 at 11:21 AM John Moore <<a href="mailto:john@flexcompute.com">john@flexcompute.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Great, thank you.</div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Jan 12, 2022 at 11:20 AM Shineman, Nat <<a href="mailto:shineman.5@osu.edu" target="_blank">shineman.5@osu.edu</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0);background-color:rgb(255,255,255)">
John, <br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0);background-color:rgb(255,255,255)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0);background-color:rgb(255,255,255)">
Thanks, we will get started on generating this RPM shortly. <br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0);background-color:rgb(255,255,255)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0);background-color:rgb(255,255,255)">
Nat<br>
</div>
<div id="gmail-m_7751912176042941799gmail-m_-7934138782485899441appendonsend"></div>
<hr style="display:inline-block;width:98%">
<div id="gmail-m_7751912176042941799gmail-m_-7934138782485899441divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> John Moore <<a href="mailto:john@flexcompute.com" target="_blank">john@flexcompute.com</a>><br>
<b>Sent:</b> Wednesday, January 12, 2022 11:19<br>
<b>To:</b> Shineman, Nat <<a href="mailto:shineman.5@osu.edu" target="_blank">shineman.5@osu.edu</a>><br>
<b>Cc:</b> Panda, Dhabaleswar <<a href="mailto:panda@cse.ohio-state.edu" target="_blank">panda@cse.ohio-state.edu</a>>; Maitham Alhubail <<a href="mailto:maitham@flexcompute.com" target="_blank">maitham@flexcompute.com</a>>; <a href="mailto:mvapich-discuss@lists.osu.edu" target="_blank">mvapich-discuss@lists.osu.edu</a> <<a href="mailto:mvapich-discuss@lists.osu.edu" target="_blank">mvapich-discuss@lists.osu.edu</a>><br>
<b>Subject:</b> Re: [Mvapich-discuss] MVAPICH2 GDR from source code?</font>
<div> </div>
</div>
<div>
<div dir="ltr">Hi Nat,
<div><br>
</div>
<div>we are using: MLNX_OFED_LINUX-5.5-1.0.3.2-ubuntu20.04-x86_64</div>
<div><br>
</div>
<div>Thanks,</div>
<div>John</div>
</div>
<br>
<div>
<div dir="ltr">On Wed, Jan 12, 2022 at 11:16 AM Shineman, Nat <<a href="mailto:shineman.5@osu.edu" target="_blank">shineman.5@osu.edu</a>> wrote:<br>
</div>
<blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0);background-color:rgb(255,255,255)">
Hi John, <br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0);background-color:rgb(255,255,255)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0);background-color:rgb(255,255,255)">
Can you tell us the ofed version on your system? <br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0);background-color:rgb(255,255,255)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0);background-color:rgb(255,255,255)">
Thanks, <br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0);background-color:rgb(255,255,255)">
Nat<br>
</div>
<div id="gmail-m_7751912176042941799gmail-m_-7934138782485899441x_gmail-m_9018559978451863016appendonsend"></div>
<hr style="display:inline-block;width:98%">
<div id="gmail-m_7751912176042941799gmail-m_-7934138782485899441x_gmail-m_9018559978451863016divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" color="#000000" style="font-size:11pt"><b>From:</b> John Moore <<a href="mailto:john@flexcompute.com" target="_blank">john@flexcompute.com</a>><br>
<b>Sent:</b> Wednesday, January 12, 2022 11:14<br>
<b>To:</b> Shineman, Nat <<a href="mailto:shineman.5@osu.edu" target="_blank">shineman.5@osu.edu</a>><br>
<b>Cc:</b> Panda, Dhabaleswar <<a href="mailto:panda@cse.ohio-state.edu" target="_blank">panda@cse.ohio-state.edu</a>>; Maitham Alhubail <<a href="mailto:maitham@flexcompute.com" target="_blank">maitham@flexcompute.com</a>>;
<a href="mailto:mvapich-discuss@lists.osu.edu" target="_blank">mvapich-discuss@lists.osu.edu</a> <<a href="mailto:mvapich-discuss@lists.osu.edu" target="_blank">mvapich-discuss@lists.osu.edu</a>><br>
<b>Subject:</b> Re: [Mvapich-discuss] MVAPICH2 GDR from source code?</font>
<div> </div>
</div>
<div>
<div dir="ltr">HI Nat,
<div><br>
</div>
<div>We have been struggling to get the RPM to work for us -- we've been working on it for about a week. We are using this RPM:</div>
<div><a href="http://mvapich.cse.ohio-state.edu/download/mvapich/gdr/2.3.6/mofed5.4/mvapich2-gdr-cuda11.3.mofed5.4.gnu8.4.1-2.3.6-1.el8.x86_64.rpm" target="_blank">http://mvapich.cse.ohio-state.edu/download/mvapich/gdr/2.3.6/mofed5.4/mvapich2-gdr-cuda11.3.mofed5.4.gnu8.4.1-2.3.6-1.el8.x86_64.rpm</a><br>
</div>
<div><br>
</div>
<div>If you could build us a custom RPM for our system, that would be very helpful.</div>
<div><br>
</div>
<div>We're running Ubuntu 20.04 kernel 5.4.0-92-generic</div>
<div><br>
</div>
<div>GCC version is: gcc (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0</div>
<div><br>
</div>
<div>CUDA version is CUDA 11.4</div>
<div>CUDA driver: 470.82.01</div>
<div><br>
</div>
<div>Please let me know if there is any other information that you need.</div>
<div><br>
</div>
<div>Thanks,</div>
<div>John</div>
<div><br>
</div>
</div>
<br>
<div>
<div dir="ltr">On Wed, Jan 12, 2022 at 9:26 AM Shineman, Nat <<a href="mailto:shineman.5@osu.edu" target="_blank">shineman.5@osu.edu</a>> wrote:<br>
</div>
<blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0);background-color:rgb(255,255,255)">
Hi John, <br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0);background-color:rgb(255,255,255)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0);background-color:rgb(255,255,255)">
You should be able to use the RPMs on Ubuntu by converting them with alien. Regarding the CUDA and compiler versioning, you will want to make sure CUDA is an exact match, but the compiler should only need to be the same major version. You will also want to
make sure that you match the mofed major version as well, though we recommend matching the exact version if possible. Please take a look at the download page and see if any of the RPMs there match your needs. Otherwise, we would be happy to generate a custom
RPM based on your system specifications. </div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0);background-color:rgb(255,255,255)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0);background-color:rgb(255,255,255)">
Thanks, <br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0);background-color:rgb(255,255,255)">
Nat<br>
</div>
<div id="gmail-m_7751912176042941799gmail-m_-7934138782485899441x_gmail-m_9018559978451863016x_gmail-m_953132348344130423appendonsend"></div>
<hr style="display:inline-block;width:98%">
<div id="gmail-m_7751912176042941799gmail-m_-7934138782485899441x_gmail-m_9018559978451863016x_gmail-m_953132348344130423divRplyFwdMsg" dir="ltr">
<font face="Calibri, sans-serif" color="#000000" style="font-size:11pt"><b>From:</b> Mvapich-discuss <mvapich-discuss-bounces+shineman.5=<a href="mailto:osu.edu@lists.osu.edu" target="_blank">osu.edu@lists.osu.edu</a>> on behalf of John Moore via Mvapich-discuss
<<a href="mailto:mvapich-discuss@lists.osu.edu" target="_blank">mvapich-discuss@lists.osu.edu</a>><br>
<b>Sent:</b> Tuesday, January 11, 2022 14:58<br>
<b>To:</b> Panda, Dhabaleswar <<a href="mailto:panda@cse.ohio-state.edu" target="_blank">panda@cse.ohio-state.edu</a>><br>
<b>Cc:</b> Maitham Alhubail <<a href="mailto:maitham@flexcompute.com" target="_blank">maitham@flexcompute.com</a>>;
<a href="mailto:mvapich-discuss@lists.osu.edu" target="_blank">mvapich-discuss@lists.osu.edu</a> <<a href="mailto:mvapich-discuss@lists.osu.edu" target="_blank">mvapich-discuss@lists.osu.edu</a>><br>
<b>Subject:</b> Re: [Mvapich-discuss] MVAPICH2 GDR from source code?</font>
<div> </div>
</div>
<div>
<div dir="ltr">Hi DK,
<div><br>
</div>
<div>Do the CUDA and GCC versions on our system need to match the RPM exactly? We are running on Ubuntu, and there is no GCC 8.4.1 on Ubuntu.</div>
<div><br>
</div>
<div>Thank you,</div>
<div>John</div>
</div>
<br>
<div>
<div dir="ltr">On Tue, Jan 11, 2022 at 2:55 PM Panda, Dhabaleswar <<a href="mailto:panda@cse.ohio-state.edu" target="_blank">panda@cse.ohio-state.edu</a>> wrote:<br>
</div>
<blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
Hi,<br>
<br>
Thanks for your note. For GPU support with MVAPICH2, it is strongly recommended to use the MVAPICH2-GDR package. This package supports many features related to GPUs and delivers the best performance and scalability on GPU clusters. Please use a suitable RPM
package from the MVAPICH2-GDR download page for your system. Please refer to the corresponding user guide also. The MVAPICH2-GDR package can also be installed through Spack. Let us know if you experience any issues in using the MVAPICH2-GDR package on your
GPU cluster.<br>
<br>
Thanks,<br>
<br>
DK<br>
<br>
<br>
________________________________________<br>
From: Mvapich-discuss <mvapich-discuss-bounces+panda.2=<a href="mailto:osu.edu@lists.osu.edu" target="_blank">osu.edu@lists.osu.edu</a>> on behalf of John Moore via Mvapich-discuss <<a href="mailto:mvapich-discuss@lists.osu.edu" target="_blank">mvapich-discuss@lists.osu.edu</a>><br>
Sent: Tuesday, January 11, 2022 2:48 PM<br>
To: <a href="mailto:mvapich-discuss@lists.osu.edu" target="_blank">mvapich-discuss@lists.osu.edu</a><br>
Cc: Maitham Alhubail<br>
Subject: [Mvapich-discuss] MVAPICH2 GDR from source code?<br>
<br>
Hello,<br>
<br>
We have been struggling to get MVAPICH2 to work with cuda-aware support and RDMA. We have compiled MVAPICH2 from source, with the --enable-cuda option, but when we run the osu_bibw bandwidth test using Device to Device communication, we get a segmentation fault.<br>
<br>
Below is the output from osu_bibw using MVAPICH2:<br>
MVAPICH2-2.3.6 Parameters<br>
---------------------------------------------------------------------<br>
PROCESSOR ARCH NAME : MV2_ARCH_AMD_EPYC_7401_48<br>
PROCESSOR FAMILY NAME : MV2_CPU_FAMILY_AMD<br>
PROCESSOR MODEL NUMBER : 1<br>
HCA NAME : MV2_HCA_MLX_CX_HDR<br>
HETEROGENEOUS HCA : NO<br>
MV2_EAGERSIZE_1SC : 0<br>
MV2_SMP_EAGERSIZE : 16385<br>
MV2_SMP_QUEUE_LENGTH : 65536<br>
MV2_SMP_NUM_SEND_BUFFER : 16<br>
MV2_SMP_BATCH_SIZE : 8<br>
Tuning Table: : MV2_ARCH_AMD_EPYC_7401_48 MV2_HCA_MLX_CX_HDR<br>
---------------------------------------------------------------------<br>
# OSU MPI-CUDA Bi-Directional Bandwidth Test v5.7.1<br>
# Send Buffer on DEVICE (D) and Receive Buffer on DEVICE (D)<br>
# Size Bandwidth (MB/s)<br>
1 0.07<br>
2 0.15<br>
4 0.29<br>
8 0.57<br>
16 1.12<br>
32 2.30<br>
64 4.75<br>
128 9.41<br>
256 18.44<br>
512 37.22<br>
1024 74.82<br>
2048 144.70<br>
4096 289.96<br>
8192 577.33<br>
[cell3:mpi_rank_0][error_sighandler] Caught error: Segmentation fault (signal 11)<br>
[cell3:mpi_rank_1][error_sighandler] Caught error: Segmentation fault (signal 11)<br>
<br>
===================================================================================<br>
= BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES<br>
= PID 471850 RUNNING AT cell3<br>
= EXIT CODE: 139<br>
= CLEANING UP REMAINING PROCESSES<br>
= YOU CAN IGNORE THE BELOW CLEANUP MESSAGES<br>
===================================================================================<br>
And this is with OpenMPI:<br>
# OSU MPI-CUDA Bi-Directional Bandwidth Test v5.8<br>
# Send Buffer on DEVICE (D) and Receive Buffer on DEVICE (D)<br>
# Size Bandwidth (MB/s)<br>
1 0.43<br>
2 0.83<br>
4 1.68<br>
8 3.37<br>
16 6.72<br>
32 13.42<br>
64 27.02<br>
128 53.78<br>
256 107.88<br>
512 219.45<br>
1024 437.81<br>
2048 875.12<br>
4096 1747.23<br>
8192 3528.97<br>
16384 7015.15<br>
32768 13973.59<br>
65536 27702.68<br>
131072 51877.67<br>
262144 94556.99<br>
524288 157755.18<br>
1048576 236772.67<br>
2097152 333635.13<br>
4194304 408865.93<br>
<br>
<br>
Can GDR support be obtained by compiling from source like we are trying to do or do we have to use an RPM? We export MV2_USE_CUDA=1. Any recommendations would be greatly appreciated.<br>
<br>
Thanks,<br>
John<br>
</blockquote>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</blockquote></div>
</blockquote></div>