<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body>
<div dir="ltr">
<div dir="ltr" style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt;">
Hi Reyhan,</div>
<div dir="ltr" style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt;">
It looks like it’s just an expired password. You can reset it at my.osc.edu</div>
<div dir="ltr" style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt;">
Karen</div>
<div dir="ltr"><br>
</div>
<div id="ms-outlook-mobile-signature">
<div style="page:WordSection1">
<div>
<p style="margin:0in;font-size:11pt;font-family:Aptos, sans-serif"><b><span style="font-family: Arial, sans-serif; color: rgb(14, 63, 117);">Karen Tomko<span> </span></span></b><span style="font-size: 12pt; font-family: Arial, sans-serif; color: rgb(14, 63, 117);"><o:p></o:p></span></p>
<p style="margin:0in;font-size:11pt;font-family:Aptos, sans-serif;line-height:13.5pt">
<span style="font-size: 9pt; font-family: Arial, sans-serif; color: rgb(82, 80, 81);">Director of Research Software Applications<span> </span><br>
<i>Ohio Supercomputer Center</i><br>
Ohio Department of Higher Education<span> </span><br>
1224 Kinnear Road, Columbus OH<span> </span><br>
Columbus, Ohio 43212<span> </span><br>
614.292.1091<span> </span><br>
<a href="mailto:ktomko@osc.edu" style="text-decoration: underline; color: rgb(70, 120, 134);"><span style="text-decoration:none">ktomko@osc.edu</span></a><br>
<a href="http://osc.edu" style="text-decoration: underline; color: rgb(70, 120, 134);"><span style="text-decoration:none">osc.edu</span></a><br style="box-sizing:border-box;outline-color:-webkit-focus-ring-color">
<o:p></o:p></span></p>
<p style="margin:0in;font-size:11pt;font-family:Aptos, sans-serif"><o:p> </o:p></p>
</div>
</div>
</div>
<div id="mail-editor-reference-message-container">
<div class="ms-outlook-mobile-reference-message skipProofing"><span style="mso-bookmark:_MailOriginalBody">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<div style="font-family: Aptos; font-size: 12pt; text-align: left; border-width: 1pt medium medium; border-style: solid none none; border-color: rgb(181, 196, 223) currentcolor currentcolor; padding: 3pt 0in 0in; color: black;">
<span style="font-weight:bold">From: </span>Mvapich-discuss <mvapich-discuss-bounces+ktomko=osc.edu@lists.osu.edu> on behalf of Motlagh, Reyhan via Mvapich-discuss <mvapich-discuss@lists.osu.edu><br>
<span style="font-weight:bold">Date: </span>Monday, January 13, 2025 at 5:01 PM<br>
<span style="font-weight:bold">To: </span>You, Zhi-Qiang <zyou@osc.edu>, Announcement about MVAPICH (MPI over InfiniBand, RoCE, Omni-Path, Slingshot, iWARP and EFA) Libraries developed at NBCL/OSU <mvapich-discuss@lists.osu.edu>, Panda, Dhabaleswar <panda@cse.ohio-state.edu><br>
<span style="font-weight:bold">Subject: </span>Re: [Mvapich-discuss] Failed to unpack MVAPICH-Plus RPM<br>
<br>
</div>
<div class="WordSection1">
<p class="MsoNormal" style="margin: 0in; font-size: 10pt; font-family: Aptos, sans-serif;">
<span style="font-size:11.0pt">Hi ZQ, </span></p>
<p class="MsoNormal" style="margin: 0in; font-size: 10pt; font-family: Aptos, sans-serif;">
<span style="font-size:11.0pt"> </span></p>
<p class="MsoNormal" style="margin: 0in; font-size: 10pt; font-family: Aptos, sans-serif;">
<span style="font-size:11.0pt">By default we build with the version of slurm included with the OS package manager (slurm 22 for rhel9). It looks like Cardinal uses slurm 24, so this may be causing some incompatibilities. Can you try out the RPM below to see
if that’s the resolution? We’re also looking into this on our end. </span></p>
<p class="MsoNormal" style="margin: 0in; font-size: 10pt; font-family: Aptos, sans-serif;">
<span style="font-size:11.0pt"> </span></p>
<p class="MsoNormal" style="margin: 0in; font-size: 10pt; font-family: Aptos, sans-serif;">
<span style="font-size:11.0pt"><a href="https://mvapich.cse.ohio-state.edu/download/mvapich/plus/4.0/cuda/UCX/mofed24.10/mvapich-plus-4.0-cuda12.4.rhel9.ofed24.10.ucx.gcc13.2.0.slurm24-4.0-1.x86_64.rpm" style="text-decoration: underline; color: rgb(70, 120, 134);">https://mvapich.cse.ohio-state.edu/download/mvapich/plus/4.0/cuda/UCX/mofed24.10/mvapich-plus-4.0-cuda12.4.rhel9.ofed24.10.ucx.gcc13.2.0.slurm24-4.0-1.x86_64.rpm</a></span></p>
<p class="MsoNormal" style="margin: 0in; font-size: 10pt; font-family: Aptos, sans-serif;">
<span style="font-size:11.0pt"> </span></p>
<p class="MsoNormal" style="margin: 0in; font-size: 10pt; font-family: Aptos, sans-serif;">
<span style="font-size:11.0pt">It looks like my osc account has been disabled, to help with this troubleshooting, who can I reach out to for reactivation (I assume this is all on Cardinal)? Username is rmotlagh.<br>
<br>
Regarding your questions:</span></p>
<ol start="1" type="1" style="margin-bottom: 0in;margin-top:0in" data-listchain="__List_Chain_147">
<li class="MsoListParagraph" style="margin: 0in 0in 0in 0.5in; font-size: 10pt; font-family: Aptos, sans-serif;margin-left:0in">
<span style="font-size:11.0pt">Yes, we are hoping to have MVAPICH 4.0 released within the month.
</span></li><li class="MsoListParagraph" style="margin: 0in 0in 0in 0.5in; font-size: 10pt; font-family: Aptos, sans-serif;margin-left:0in">
<span style="font-size:11.0pt">We have unified redundant envs (like having separate envs for HIP and CUDA) and made naming conventions more consistent for our CVARs. So yes, replace that with MVP_ENABLE_GPU</span></li><li class="MsoListParagraph" style="margin: 0in 0in 0in 0.5in; font-size: 10pt; font-family: Aptos, sans-serif;margin-left:0in">
<span style="font-size:11.0pt">Some of these are done in the netmod layer now. You can set IB devices with “UCX_NET_DEVICES=mlx5_0:1” and “UCX_SOCKADDR_TLS_PRIORITY=rdmacm” (rdmacm may require a new rpm with --with-rdmacm ucx configure flag, I will update the
website rpms to allow for this if it passes our testing). MVP_HOMOGENEOUS_CLUSTER’s equivalent is irrelevant now, performance is good regardless of this flag.
</span></li></ol>
<p class="MsoNormal" style="margin: 0in; font-size: 10pt; font-family: Aptos, sans-serif;">
<span style="font-size:11.0pt"> </span></p>
<p class="MsoNormal" style="margin: 0in; font-size: 10pt; font-family: Aptos, sans-serif;">
<span style="font-size:11.0pt">Best,</span></p>
<p class="MsoNormal" style="margin: 0in; font-size: 10pt; font-family: Aptos, sans-serif;">
<span style="font-size:11.0pt">Reyhan</span></p>
<p class="MsoNormal" style="margin: 0in; font-size: 10pt; font-family: Aptos, sans-serif;">
<span style="font-size:11.0pt"> </span></p>
<div id="mail-editor-reference-message-container">
<div>
<div style="border:none; border-top:solid #B5C4DF 1.0pt; padding:3.0pt 0in 0in 0in">
<p class="MsoNormal" style="margin: 0in; font-size: 10pt; font-family: Aptos, sans-serif;margin-bottom:12.0pt">
<b><span style="font-size: 12pt; color: black;">From: </span></b><span style="font-size: 12pt; color: black;">Mvapich-discuss <mvapich-discuss-bounces@lists.osu.edu> on behalf of You, Zhi-Qiang via Mvapich-discuss <mvapich-discuss@lists.osu.edu><br>
<b>Date: </b>Saturday, January 11, 2025 at 9:32</span><span style="font-size: 12pt; font-family: Arial, sans-serif; color: black;"> </span><span style="font-size: 12pt; color: black;">PM<br>
<b>To: </b>Panda, Dhabaleswar <panda@cse.ohio-state.edu>, Announcement about MVAPICH (MPI over InfiniBand, RoCE, Omni-Path, Slingshot, iWARP and EFA) Libraries developed at NBCL/OSU <mvapich-discuss@lists.osu.edu><br>
<b>Subject: </b>Re: [Mvapich-discuss] Failed to unpack MVAPICH-Plus RPM</span></p>
</div>
<div>
<p class="MsoNormal" style="margin: 0in; font-size: 10pt; font-family: Aptos, sans-serif;">
<span style="font-size:11.0pt">Hi DK,</span></p>
<p class="MsoNormal" style="margin: 0in; font-size: 10pt; font-family: Aptos, sans-serif;">
<span style="font-size:11.0pt"> </span></p>
<p class="MsoNormal" style="margin: 0in; font-size: 10pt; font-family: Aptos, sans-serif;">
<span style="font-size:11.0pt">Thank you for the prompt fix. The RPM is now functioning correctly. However, I encountered the following error while running a simple ping-pong MPI test over two nodes:</span></p>
<p class="MsoNormal" style="margin: 0in; font-size: 10pt; font-family: Aptos, sans-serif;">
<span style="font-size:11.0pt"><br>
slurmstepd: error: pmijobid missing in fullinit command</span></p>
<p class="MsoNormal" style="margin: 0in; font-size: 10pt; font-family: Aptos, sans-serif;">
<span style="font-size:11.0pt"><br>
I suspected this might be due to PMI incompatibility. I referred to </span><a href="https://urldefense.com/v3/__https:/mvapich-docs.readthedocs.io/en/latest/cvar.html*mvapich-environment-variables__;Iw!!KGKeukY!3fo-CIZdjSLr3Qr4T-N801LdCwjo-3DZiuA5KjZOvLaCn4id5M3xni5dWZHrZEnZrHIvm_FdrzIPC23DUe4941agsuMkFyC1$" style="text-decoration: underline; color: rgb(70, 120, 134);"><span style="font-size:11.0pt">this
documentation</span></a><span style="font-size:11.0pt"> and learned about setting MVP_PMI_VERSION to 2 to align with our SLURM configuration. However, the issue persists. I also checked the output of mpichversion -a and confirmed that the --with-pmi=pmi2 option
is enabled, leading me to conclude that this is not a PMI compatibility issue.</span></p>
<p class="MsoNormal" style="margin: 0in; font-size: 10pt; font-family: Aptos, sans-serif;">
<span style="font-size:11.0pt"> </span></p>
<p class="MsoNormal" style="margin: 0in; font-size: 10pt; font-family: Aptos, sans-serif;">
<span style="font-size:11.0pt">Additionally, I have a few related questions:</span></p>
<ol start="1" type="1" style="margin-bottom: 0in;margin-top:0in" data-listchain="__List_Chain_148">
<li class="MsoListParagraph" style="margin: 0in 0in 0in 0.5in; font-size: 10pt; font-family: Aptos, sans-serif;margin-left:0in">
<span style="font-size:11.0pt">Will there be an MVAPICH 4.0 release, or will it be replaced by the MVAPICH-Plus CPU-only version?</span></li><li class="MsoListParagraph" style="margin: 0in 0in 0in 0.5in; font-size: 10pt; font-family: Aptos, sans-serif;margin-left:0in">
<span style="font-size:11.0pt">The documentation linked above lists many environment variables that I haven’t encountered before when using MVAPICH2-GDR. Are these new variables specific to MVAPICH 4.0? Are variables like MV2_USE_CUDA/MVP_USE_CUDA still available,
or should they be replaced with MVP_ENABLE_GPU?</span></li><li class="MsoListParagraph" style="margin: 0in 0in 0in 0.5in; font-size: 10pt; font-family: Aptos, sans-serif;margin-left:0in">
<span style="font-size:11.0pt">Could you help confirm if the following variables are still supported in MVAPICH?</span></li></ol>
<ol start="3" type="1" style="margin-bottom: 0in;margin-top:0in" data-listchain="__List_Chain_149">
<ul type="disc" style="margin-bottom: 0in;margin-top:0in">
<li class="MsoListParagraph" style="margin: 0in 0in 0in 0.5in; font-size: 10pt; font-family: Aptos, sans-serif;margin-left:0in">
<span style="font-size:11.0pt">MVP_USE_RDMA_CM</span></li><li class="MsoListParagraph" style="margin: 0in 0in 0in 0.5in; font-size: 10pt; font-family: Aptos, sans-serif;margin-left:0in">
<span style="font-size:11.0pt">MVP_HOMOGENEOUS_CLUSTER</span></li><li class="MsoListParagraph" style="margin: 0in 0in 0in 0.5in; font-size: 10pt; font-family: Aptos, sans-serif;margin-left:0in">
<span style="font-size:11.0pt">MVP_IBA_HCA</span></li></ul>
</ol>
<p class="MsoNormal" style="margin: 0in; font-size: 10pt; font-family: Aptos, sans-serif;">
<span style="font-size:11.0pt"> </span></p>
<p class="MsoNormal" style="margin: 0in; font-size: 10pt; font-family: Aptos, sans-serif;">
<span style="font-size:11.0pt">Thank you for your time and assistance!</span></p>
<p class="MsoNormal" style="margin: 0in; font-size: 10pt; font-family: Aptos, sans-serif;">
<span style="font-size:11.0pt"> </span></p>
<p class="MsoNormal" style="margin: 0in; font-size: 10pt; font-family: Aptos, sans-serif;">
<span style="font-size:11.0pt">Best regards,</span></p>
<p class="MsoNormal" style="margin: 0in; font-size: 10pt; font-family: Aptos, sans-serif;">
<span style="font-size:11.0pt">ZQ</span></p>
<p class="MsoNormal" style="margin: 0in; font-size: 10pt; font-family: Aptos, sans-serif;">
<span style="font-size:11.0pt"> </span></p>
<p class="MsoNormal" style="margin: 0in; font-size: 10pt; font-family: Aptos, sans-serif;">
<span style="font-size:11.0pt"> </span></p>
<div id="mail-editor-reference-message-container">
<div>
<div>
<div style="border:none; border-top:solid #B5C4DF 1.0pt; padding:3.0pt 0in 0in 0in">
<p class="MsoNormal" style="margin: 0in; font-size: 10pt; font-family: Aptos, sans-serif;margin-bottom:12.0pt">
<b><span style="font-size: 12pt; color: black;">From: </span></b><span style="font-size: 12pt; color: black;">Panda, Dhabaleswar <panda@cse.ohio-state.edu><br>
<b>Date: </b>Saturday, January 11, 2025 at 3:14</span><span style="font-size: 12pt; font-family: Arial, sans-serif; color: black;"> </span><span style="font-size: 12pt; color: black;">AM<br>
<b>To: </b>You, Zhi-Qiang <zyou@osc.edu>, Announcement about MVAPICH (MPI over InfiniBand, RoCE, Omni-Path, Slingshot, iWARP and EFA) Libraries developed at NBCL/OSU <mvapich-discuss@lists.osu.edu><br>
<b>Subject: </b>RE: Failed to unpack MVAPICH-Plus RPM</span></p>
</div>
<div>
<p class="MsoNormal" style="margin: 0in; font-size: 10pt; font-family: Aptos, sans-serif;">
<span style="font-size:11.0pt; font-family:"Calibri",sans-serif">Hi ZQ, </span></p>
<p class="MsoNormal" style="margin: 0in; font-size: 10pt; font-family: Aptos, sans-serif;">
<span style="font-size:11.0pt; font-family:"Calibri",sans-serif"> </span></p>
<p class="MsoNormal" style="margin: 0in; font-size: 10pt; font-family: Aptos, sans-serif;">
<span style="font-size:11.0pt; font-family:"Calibri",sans-serif">As we have communicated with you separately, a new RPM has been uploaded. Please try this version and let us know whether you see any additional issues.
</span></p>
<p class="MsoNormal" style="margin: 0in; font-size: 10pt; font-family: Aptos, sans-serif;">
<span style="font-size:11.0pt; font-family:"Calibri",sans-serif"> </span></p>
<p class="MsoNormal" style="margin: 0in; font-size: 10pt; font-family: Aptos, sans-serif;">
<span style="font-size:11.0pt; font-family:"Calibri",sans-serif">DK</span></p>
<p class="MsoNormal" style="margin: 0in; font-size: 10pt; font-family: Aptos, sans-serif;">
<span style="font-size:11.0pt; font-family:"Calibri",sans-serif"> </span></p>
<div>
<div style="border:none; border-top:solid #E1E1E1 1.0pt; padding:3.0pt 0in 0in 0in">
<p class="MsoNormal" style="margin: 0in; font-size: 10pt; font-family: Aptos, sans-serif;">
<b><span style="font-size:11.0pt; font-family:"Calibri",sans-serif">From:</span></b><span style="font-size:11.0pt; font-family:"Calibri",sans-serif"> Mvapich-discuss <mvapich-discuss-bounces@lists.osu.edu>
<b>On Behalf Of </b>You, Zhi-Qiang via Mvapich-discuss<br>
<b>Sent:</b> Thursday, January 2, 2025 1:54 PM<br>
<b>To:</b> mvapich-discuss@lists.osu.edu<br>
<b>Subject:</b> [Mvapich-discuss] Failed to unpack MVAPICH-Plus RPM</span></p>
</div>
</div>
<p class="MsoNormal" style="margin: 0in; font-size: 10pt; font-family: Aptos, sans-serif;">
<span style="font-size:12.0pt"> </span></p>
<p class="MsoNormal" style="margin: 0in; font-size: 10pt; font-family: Aptos, sans-serif;">
<span style="font-size:11.0pt">Hello,</span></p>
<p class="MsoNormal" style="margin: 0in; font-size: 10pt; font-family: Aptos, sans-serif;">
<span style="font-size:11.0pt"> </span></p>
<p class="MsoNormal" style="margin: 0in; font-size: 10pt; font-family: Aptos, sans-serif;">
<span style="font-size:11.0pt">I downloaded the MVAPICH-Plus 4.0 RPM from the following link:</span></p>
<p class="MsoNormal" style="margin: 0in; font-size: 10pt; font-family: Aptos, sans-serif;">
<a href="https://mvapich.cse.ohio-state.edu/download/mvapich/plus/4.0/cuda/UCX/mofed5.0/mvapich-plus-4.0-cuda12.4.rhel9.ofed24.10.ucx.gcc13.2.0.slurm-4.0-1.x86_64.rpm" style="text-decoration: underline; color: rgb(70, 120, 134);"><span style="font-size:11.0pt">https://mvapich.cse.ohio-state.edu/download/mvapich/plus/4.0/cuda/UCX/mofed5.0/mvapich-plus-4.0-cuda12.4.rhel9.ofed24.10.ucx.gcc13.2.0.slurm-4.0-1.x86_64.rpm</span></a><span style="font-size:11.0pt">,
but I encountered an issue when trying to unpack it using cpio. The process failed with the error:</span></p>
<p class="MsoNormal" style="margin: 0in; font-size: 10pt; font-family: Aptos, sans-serif;">
<span style="font-size:11.0pt"><br>
cpio: premature end of file</span></p>
<p class="MsoNormal" style="margin: 0in; font-size: 10pt; font-family: Aptos, sans-serif;">
<span style="font-size:11.0pt"> </span></p>
<p class="MsoNormal" style="margin: 0in; font-size: 10pt; font-family: Aptos, sans-serif;">
<span style="font-size:11.0pt">I have no issues unpacking other RPMs, so it seems this file might be corrupted. Could you please check and confirm?</span></p>
<p class="MsoNormal" style="margin: 0in; font-size: 10pt; font-family: Aptos, sans-serif;">
<span style="font-size:11.0pt"> </span></p>
<p class="MsoNormal" style="margin: 0in; font-size: 10pt; font-family: Aptos, sans-serif;">
<span style="font-size:11.0pt">Thank you,</span></p>
<p class="MsoNormal" style="margin: 0in; font-size: 10pt; font-family: Aptos, sans-serif;">
<span style="font-size:11.0pt">ZQ</span></p>
<p class="MsoNormal" style="margin: 0in; font-size: 10pt; font-family: Aptos, sans-serif;">
<span style="font-size:11.0pt"> </span></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</span></div>
</div>
</div>
</body>
</html>