<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Latha;
panose-1:2 0 4 0 0 0 0 0 0 0;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:10.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
span.EmailStyle21
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="#0563C1" vlink="#954F72" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt">Hi, Ron.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Glad to hear that it worked for you.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Thanks for pointing us to the fact that these are not exposed on Spack. We will take care of it shortly.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Best,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Hari.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:11.0pt">From:</span></b><span style="font-size:11.0pt"> Rahaman, Ronald O <rrahaman6@gatech.edu>
<br>
<b>Sent:</b> Monday, April 24, 2023 1:01 PM<br>
<b>To:</b> Subramoni, Hari <subramoni.1@osu.edu>; Announcement about MVAPICH2 (MPI over InfiniBand, RoCE, Omni-Path, iWARP and EFA) Libraries developed at NBCL/OSU <mvapich-discuss@lists.osu.edu><br>
<b>Subject:</b> Re: Avoid linking to libXNVCtrl.so<o:p></o:p></span></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal" style="mso-line-height-alt:.75pt"><span style="font-size:1.0pt;color:white">Hi Hari, Thanks! The extra flags (--disable-opencl --disable-gl --disable-nvml) solved my issue. I can see that MVAPICH2 was not configured to link to XNVCtrl on
the dev node, and the OSU benchmark apps are working fine on the prod node. As <o:p>
</o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-line-height-alt:.75pt"><span style="font-size:1.0pt;color:white">ZjQcmQRYFpfptBannerStart<o:p></o:p></span></p>
</div>
<table class="MsoNormalTable" border="0" cellspacing="0" cellpadding="0" width="100%" style="width:100.0%;border-radius:4px">
<tbody>
<tr>
<td style="padding:12.0pt 0in 12.0pt 0in">
<table class="MsoNormalTable" border="1" cellspacing="0" cellpadding="0" width="100%" style="width:100.0%;background:#CFD3D7;border:none;border-top:solid #8C8E91 3.0pt">
<tbody>
<tr>
<td valign="top" style="border:none;padding:0in 7.5pt 3.75pt 4.5pt">
<table class="MsoNormalTable" border="0" cellspacing="0" cellpadding="0" align="left">
<tbody>
<tr>
<td style="padding:3.0pt 6.0pt 3.0pt 6.0pt">
<p class="MsoNormal"><b><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:black">This Message Is From an External Sender
<o:p></o:p></span></b></p>
</td>
</tr>
<tr>
<td style="padding:3.0pt 6.0pt 3.0pt 6.0pt">
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Arial",sans-serif;color:black">This message came from outside your organization.
<o:p></o:p></span></p>
</td>
</tr>
</tbody>
</table>
<table class="MsoNormalTable" border="0" cellspacing="0" cellpadding="0" align="right">
<tbody>
<tr>
<td style="padding:3.0pt 0in 3.0pt 0in">
<p class="MsoNormal"> <a href="https://us-phishalarm-ewt.proofpoint.com/EWT/v1/KGKeukY!vwQd0oZND6YAhRdwHo7Fd6qJj2BONI0bhC_7C1hITdyOahSYlySkv_EWTddgJ6-UO6O0sqWKVrtqIZjABiLgfJT9p85e9W79c5mWrxwIWYHNADMQRxEqVoFaBYeNPifqtdzA02NE_MmxaA$" target="_blank"><strong><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:black;border:solid #666666 1.0pt;padding:6.0pt;font-weight:normal;text-decoration:none"> Report Suspicious </span></strong></a>
<span style="font-size:11.0pt"><o:p></o:p></span></p>
</td>
</tr>
</tbody>
</table>
</td>
</tr>
</tbody>
</table>
</td>
</tr>
</tbody>
</table>
<div>
<p class="MsoNormal" style="mso-line-height-alt:.75pt"><span style="font-size:1.0pt;color:white">ZjQcmQRYFpfptBannerEnd<o:p></o:p></span></p>
</div>
<p class="MsoNormal"><span style="font-size:11.0pt">Hi Hari,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Thanks! The extra flags (<span style="mso-ligatures:standardcontextual">--disable-opencl --disable-gl --disable-nvml) solved my issue. I can see that MVAPICH2 was not configured to link to XNVCtrl on the
dev node, and the OSU benchmark apps are working fine on the prod node. <o:p></o:p></span></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;mso-ligatures:standardcontextual"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;mso-ligatures:standardcontextual">As a follow-up, these flags aren’t exposed for the mvapich2 package in Spack. I have enough working knowledge of Spack to update package.py for our site. It would be nice
to have this fix in the main Spack distribution. Would y’all be able to contribute that?
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;mso-ligatures:standardcontextual"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;mso-ligatures:standardcontextual">Best,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;mso-ligatures:standardcontextual">Ron</span><span style="font-size:11.0pt"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;color:black">--------<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:black">Ron Rahaman</span><span style="font-size:11.0pt"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Research Scientist II, Research Software Engineer<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Partnership for an Advanced Computing Environment (PACE)<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><span style="font-size:11.0pt">Georgia Institute of Technology
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal" style="margin-bottom:12.0pt"><b><span style="font-size:12.0pt;color:black">From:
</span></b><span style="font-size:12.0pt;color:black">Subramoni, Hari <<a href="mailto:subramoni.1@osu.edu">subramoni.1@osu.edu</a>><br>
<b>Date: </b>Thursday, April 20, 2023 at 9:00 AM<br>
<b>To: </b>Rahaman, Ronald O <<a href="mailto:rrahaman6@gatech.edu">rrahaman6@gatech.edu</a>>, Announcement about MVAPICH2 (MPI over InfiniBand, RoCE, Omni-Path, iWARP and EFA) Libraries developed at NBCL/OSU <<a href="mailto:mvapich-discuss@lists.osu.edu">mvapich-discuss@lists.osu.edu</a>><br>
<b>Cc: </b>Subramoni, Hari <<a href="mailto:subramoni.1@osu.edu">subramoni.1@osu.edu</a>><br>
<b>Subject: </b>RE: Avoid linking to libXNVCtrl.so<o:p></o:p></span></p>
</div>
<p class="MsoNormal"><span style="font-size:11.0pt;mso-ligatures:standardcontextual">Hi, Ron.</span><span style="mso-ligatures:standardcontextual"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;mso-ligatures:standardcontextual"> </span><span style="mso-ligatures:standardcontextual"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;mso-ligatures:standardcontextual">Can you please add the following to the configure line and retry the build?</span><span style="mso-ligatures:standardcontextual"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;mso-ligatures:standardcontextual"> </span><span style="mso-ligatures:standardcontextual"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;mso-ligatures:standardcontextual">--disable-opencl --disable-gl --disable-nvml</span><span style="mso-ligatures:standardcontextual"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;mso-ligatures:standardcontextual"> </span><span style="mso-ligatures:standardcontextual"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;mso-ligatures:standardcontextual">Best,</span><span style="mso-ligatures:standardcontextual"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;mso-ligatures:standardcontextual">Hari.</span><span style="mso-ligatures:standardcontextual"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;mso-ligatures:standardcontextual"> </span><span style="mso-ligatures:standardcontextual"><o:p></o:p></span></p>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:11.0pt">From:</span></b><span style="font-size:11.0pt"> Mvapich-discuss <<a href="mailto:mvapich-discuss-bounces@lists.osu.edu">mvapich-discuss-bounces@lists.osu.edu</a>>
<b>On Behalf Of </b>Rahaman, Ronald O via Mvapich-discuss<br>
<b>Sent:</b> Thursday, April 20, 2023 9:18 AM<br>
<b>To:</b> <a href="mailto:mvapich-discuss@lists.osu.edu">mvapich-discuss@lists.osu.edu</a><br>
<b>Subject:</b> [Mvapich-discuss] Avoid linking to libXNVCtrl.so</span><span style="mso-ligatures:standardcontextual"><o:p></o:p></span></p>
</div>
<p class="MsoNormal"><span style="mso-ligatures:standardcontextual"> <o:p></o:p></span></p>
<div>
<p class="MsoNormal" style="mso-line-height-alt:.75pt"><span style="font-size:1.0pt;color:white;mso-ligatures:standardcontextual">Hi all, I need to build MVAPICH2 on a dev server with a GPU; and then deploy it to a prod cluster where many nodes do not have
GPUs. However, when I build on the dev server, MVAPICH2 is linked to the available libXNVCtrl. so. Thus on the GPU-less
</span><span style="mso-ligatures:standardcontextual"><o:p></o:p></span></p>
</div>
<p class="MsoNormal"><span style="font-size:11.0pt;mso-ligatures:standardcontextual">Hi all,</span><span style="mso-ligatures:standardcontextual"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;mso-ligatures:standardcontextual"> </span><span style="mso-ligatures:standardcontextual"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;mso-ligatures:standardcontextual">I need to build MVAPICH2 on a dev server with a GPU; and then deploy it to a prod cluster where many nodes do not have GPUs. However, when I build on the dev server, MVAPICH2
is linked to the available libXNVCtrl.so. Thus on the GPU-less nodes on prod cluster, there are load-time errors since libXNVCtrl.so is not available.
</span><span style="mso-ligatures:standardcontextual"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;mso-ligatures:standardcontextual"> </span><span style="mso-ligatures:standardcontextual"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;mso-ligatures:standardcontextual">Is there a way to configure MVAPICH so it will not link to libXNVCtrl.so?
</span><span style="mso-ligatures:standardcontextual"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;mso-ligatures:standardcontextual"> </span><span style="mso-ligatures:standardcontextual"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;mso-ligatures:standardcontextual">I already have `--disable-cuda` set when I configure. If I need to build and deploy a differently-configured hwloc, I can definitely manage that.</span><span style="mso-ligatures:standardcontextual"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;mso-ligatures:standardcontextual"> </span><span style="mso-ligatures:standardcontextual"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;mso-ligatures:standardcontextual">Many thanks,</span><span style="mso-ligatures:standardcontextual"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;mso-ligatures:standardcontextual">Ron</span><span style="mso-ligatures:standardcontextual"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;mso-ligatures:standardcontextual"> </span><span style="mso-ligatures:standardcontextual"><o:p></o:p></span></p>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;color:black">--------</span><span style="mso-ligatures:standardcontextual"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:black">Ron Rahaman</span><span style="mso-ligatures:standardcontextual"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Research Scientist II, Research Software Engineer</span><span style="mso-ligatures:standardcontextual"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Partnership for an Advanced Computing Environment (PACE)</span><span style="mso-ligatures:standardcontextual"><o:p></o:p></span></p>
</div>
<p class="MsoNormal"><span style="font-size:11.0pt">Georgia Institute of Technology
</span><span style="mso-ligatures:standardcontextual"><o:p></o:p></span></p>
</div>
</body>
</html>