<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Hi Aditya,</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
We have found that typically Cray systems use a different version of PMI/PMI2 than is found on other slurm installations. Can you please try building with
<code>--with-pmi=cray</code> instead of giving the pmi2 path? This has been more successful in most tests that we have tried. If you are still having issues, sometimes you need to also add
<code>--with-craypmi=<path/to/craypmi/dir></code> to ensure the right version is picked up.</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Thanks,</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Nat</div>
<div id="appendonsend"></div>
<hr style="display: inline-block; width: 98%;">
<div id="divRplyFwdMsg" dir="ltr"><span style="font-family: Calibri, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);"><b>From:</b> Mvapich-discuss <mvapich-discuss-bounces+shineman.5=osu.edu@lists.osu.edu> on behalf of Kashi, Aditya via Mvapich-discuss <mvapich-discuss@lists.osu.edu><br>
<b>Sent:</b> Tuesday, January 9, 2024 18:18<br>
<b>To:</b> mvapich-discuss@lists.osu.edu <mvapich-discuss@lists.osu.edu><br>
<b>Cc:</b> Matheson, Michael <mathesonma@ornl.gov><br>
<b>Subject:</b> [Mvapich-discuss] libpmi2 could not be found while building for Slurm</span>
<div> </div>
</div>
<div dir="ltr" lang="en" id="x_pfptBannerzmut7kn" style="visibility: visible !important; opacity: 1 !important; max-width: none !important; max-height: none !important; display: block !important; text-align: left !important; margin: 16px 0px !important; padding: 8px 16px !important; border-radius: 4px !important; min-width: 200px !important; border-top: 4px solid rgb(140, 142, 145) !important;">
<div id="x_pfptBannerzmut7kn" style="visibility: visible !important; opacity: 1 !important; max-height: none !important; float: left !important; display: block !important; margin: 0px 0px 1px !important; max-width: 600px !important;">
<div id="x_pfptBannerzmut7kn" style="opacity: 1 !important; max-width: none !important; max-height: none !important; display: block !important; visibility: visible !important; font-family: "Arial", sans-serif !important; font-weight: bold !important; font-size: 14px !important; line-height: 18px !important;">
This Message Is From an External Sender</div>
<div id="x_pfptBannerzmut7kn" style="opacity: 1 !important; max-width: none !important; max-height: none !important; display: block !important; visibility: visible !important; font-weight: normal; font-family: "Arial", sans-serif !important; font-size: 12px !important; line-height: 18px !important; margin-top: 2px !important;">
This message came from outside your organization.</div>
</div>
<div id="x_pfptBannerzmut7kn" style="visibility: visible !important; opacity: 1 !important; max-width: none !important; max-height: none !important; float: right !important; display: block !important; margin: 0px 0px 0px 16px !important; text-align: right !important; width: fit-content !important;">
<div style="white-space: nowrap; margin: 3px 0px 3px 16px; padding: 7.5px 16px; border-width: 1px; border-style: solid; border-color: rgb(102, 102, 102); border-radius: 2px; width: fit-content; display: inline-block;">
<span style="font-family: "Arial", sans-serif; font-size: 14px; color: rgb(0, 0, 0); background-color: rgb(207, 211, 215);"><a href="https://us-phishalarm-ewt.proofpoint.com/EWT/v1/KGKeukY!siQZ16bhKayAAJnRn86E3cDMzsH8lNbryju8HZ7WNS4aDJ0T0cxv1kzTpsKK5ugjGyuGPD1VUmTJs6tjOQrCvUk4ZjkielXdW2XhjtqF6Zg4Ot3gIlXbQ1E$" id="x_pfptBannerzmut7kn" class="OWAAutoLink" style="text-decoration: none; display: inline-block; max-width: none; max-height: none; background-color: rgb(207, 211, 215);" data-auth="NotApplicable" data-loopstyle="linkonly">Report Suspicious</a></span></div>
</div>
<div style="line-height: 0; height: 0px; display: block; font-size: 0.01px;"> </div>
</div>
<div><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">Hi everyone,</span></div>
<div><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"><br>
</span></div>
<div><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">I'm trying to build MVAPICH 3.0rc on a Cray Shasta system with Slurm. This is my current build line:</span></div>
<div><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"><br>
</span></div>
<div><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">FFLAGS=-fallow-argument-mismatch ./configure --with-pmi=pmi2 --enable-slurm --with-pm=slurm --enable-rocm --with-rocm=$ROCM_PATH
--with-libfabric=/opt/cray/libfabric/1.15.2.0 --with-pmi2-libdir=/usr/lib64/slurmpmi</span></div>
<div><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"><br>
</span></div>
<div><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">It's able to find pmi2.h, but not libpmi2.so:</span></div>
<div><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"><br>
</span></div>
<div><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">...</span></div>
<div><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">configure: RUNNING CONFIGURE FOR src/pm/slurm</span></div>
<div><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">checking for srun... /usr/bin/srun</span></div>
<div><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">checking slurm/pmi2.h usability... yes</span></div>
<div><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">checking slurm/pmi2.h presence... yes</span></div>
<div><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">checking for slurm/pmi2.h... yes</span></div>
<div><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">checking for /usr/include/slurm/pmi2.h... yes</span></div>
<div><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">./configure: line 60908: found: command not found</span></div>
<div><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">checking for PMI2_Init in -lpmi2... no</span></div>
<div><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">configure: error: could not find the slurm libpmi2. Configure aborted</span></div>
<div><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"><br>
</span></div>
<div><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">However, I can see the file /usr/lib64/slurmpmi/libpmi2.so, which is symlinked to /usr/lib64/slurmpmi/libpmi2.so.0.0.0.
I've tried variations of the last flag like "--with-pmi-libdir", "--with-pmi2-lib=.../libpmi2.so" etc.</span></div>
<div><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"><br>
</span></div>
<div><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">Is there a known way to build MVAPICH with a scalable backend on this kind of system? Getting the best possible
performance at scale is absolutely necessary for what I'm trying to do.</span></div>
<div><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"><br>
</span></div>
<div><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">Thanks,</span></div>
<div><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">Aditya Kashi</span></div>
<div><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">Analytics and AI Methods at Scale</span></div>
<div><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">Oak Ridge National Laboratory</span></div>
</body>
</html>