<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Aptos;
panose-1:2 11 0 4 2 2 2 2 2 4;}
@font-face
{font-family:Menlo;
panose-1:2 11 6 9 3 8 4 2 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
code
{mso-style-priority:99;
font-family:"Courier New";}
pre
{mso-style-priority:99;
mso-style-link:"HTML Preformatted Char";
margin:0in;
margin-bottom:.0001pt;
font-size:10.0pt;
font-family:"Courier New";}
span.EmailStyle21
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
p.p1, li.p1, div.p1
{mso-style-name:p1;
margin:0in;
font-size:8.5pt;
font-family:Menlo;
color:black;}
span.s1
{mso-style-name:s1;}
span.HTMLPreformattedChar
{mso-style-name:"HTML Preformatted Char";
mso-style-priority:99;
mso-style-link:"HTML Preformatted";
font-family:"Courier New";}
span.hljs-string
{mso-style-name:hljs-string;}
span.hljs-variable
{mso-style-name:hljs-variable;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal">Hi John,<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I hope this message finds you well. My name is Pouya Kousha, and I'm the lead developer for OSU INAM. Thank you for reaching out to us. To provide you with the most accurate support, we need to gather more detailed information about the
issue you're encountering.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Could you please assist us by enabling additional debugging features? This can be done by setting specific variables in your osu-inamd.conf file, which is located in your INAM installation directory. Please update the file with the following
settings:<o:p></o:p></p>
<p class="p1"><span class="s1"><span style="color:#548235;mso-style-textfill-fill-color:#548235;mso-style-textfill-fill-alpha:100.0%"><span style="color:windowtext">INAM_DEBUG_INIT_VERBOSE=1</span></span></span><span style="color:#548235;mso-style-textfill-fill-color:#548235;mso-style-textfill-fill-alpha:100.0%"><o:p></o:p></span></p>
<p class="p1"><span class="s1"><span style="color:#548235;mso-style-textfill-fill-color:#548235;mso-style-textfill-fill-alpha:100.0%"><span style="color:windowtext">INAM_DEBUG_TIME_VERBOSE=1</span></span></span><span style="color:#548235;mso-style-textfill-fill-color:#548235;mso-style-textfill-fill-alpha:100.0%"><o:p></o:p></span></p>
<p class="p1"><span class="s1"><span style="color:#548235;mso-style-textfill-fill-color:#548235;mso-style-textfill-fill-alpha:100.0%"><span style="color:windowtext">INAM_DEBUG_MAIN_VERBOSE=2</span></span></span><span style="color:#548235;mso-style-textfill-fill-color:#548235;mso-style-textfill-fill-alpha:100.0%"><o:p></o:p></span></p>
<p class="p1"><span class="s1"><span style="color:#548235;mso-style-textfill-fill-color:#548235;mso-style-textfill-fill-alpha:100.0%"><span style="color:windowtext">INAM_DEBUG_DB_VERBOSE=2</span></span></span><span style="color:#548235;mso-style-textfill-fill-color:#548235;mso-style-textfill-fill-alpha:100.0%"><o:p></o:p></span></p>
<p class="p1"><span class="s1"><span style="color:#548235;mso-style-textfill-fill-color:#548235;mso-style-textfill-fill-alpha:100.0%"><span style="color:windowtext">INAM_DEBUG_NW_VERBOSE=1</span></span></span><span style="color:#548235;mso-style-textfill-fill-color:#548235;mso-style-textfill-fill-alpha:100.0%"><o:p></o:p></span></p>
<p class="p1"><span class="s1"><span style="color:#548235;mso-style-textfill-fill-color:#548235;mso-style-textfill-fill-alpha:100.0%"><span style="color:windowtext">INAM_DEBUG_FB_VERBOSE=1</span></span></span><span style="color:#548235;mso-style-textfill-fill-color:#548235;mso-style-textfill-fill-alpha:100.0%"><o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">After updating these settings, kindly restart the INAM daemon to apply the changes.<o:p></o:p></p>
<p class="MsoNormal">Furthermore, we would appreciate it if you could share the log file segment from
<span style="font-size:10.0pt;font-family:"Courier New"">/var/log/messages</span> that pertains to INAM, as well as the updated
<span style="font-size:10.0pt;font-family:"Courier New"">osu-inam.conf</span> file. This information will greatly aid in our diagnostics.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">From the srun add the followings as well. You can set <span style="font-size:8.5pt;font-family:Menlo;color:red">
MV2_DEBUG_TOOL_VERBOSE=1 </span>to check if from Mvapich2, we are sending information to INAM or not. It would be nice if you could send us the output (will be lengthy) as well. It should be good with 5-10 minutes of log for the application.<o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:8.5pt;font-family:Menlo;color:red"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.5pt;font-family:Menlo;color:red"><o:p> </o:p></span></p>
<p class="MsoNormal">Additionally, to assess if there's an interaction issue with Mvapich2, please add the following environment variables to your
<span style="font-size:10.0pt;font-family:"Courier New"">srun</span> command:<o:p></o:p></p>
<p class="MsoNormal"><span style="color:black"><o:p> </o:p></span></p>
<p class="MsoNormal" style="background:white"><span style="font-size:8.5pt;font-family:Menlo;color:#FC4708">export</span><span style="font-size:8.5pt;font-family:Menlo;color:#FB0074"> common=</span><span style="font-size:8.5pt;font-family:Menlo;color:#FC4708">"</span><span style="font-size:8.5pt;font-family:Menlo;color:#0E71CD">MV2_TWO_LEVEL_COMM_THRESHOLD=1
MV2_USE_RDMA_CM=0 MV2_TOOL_REPORT_PVARS=1 </span><span style="font-size:8.5pt;font-family:Menlo;color:red">MV2_DEBUG_TOOL_VERBOSE=1</span><span style="font-size:8.5pt;font-family:Menlo;color:#0E71CD"> MV2_USE_SHMEM_COLL=1 MV2_ENABLE_PVAR_TIMER=1 MV2_ENABLE_PVAR_COUNTER=1
MV2_ENABLE_PVAR_TIMER_BUCKETS=1 MV2_ENABLE_PVAR_COUNTER_BUCKETS=1 MV2_TOOL_REPORT_SESSIONS=1 MV2_TOOL_SESSIONS_DEFAULT_ALL_HANDLES=1 MV2_TOOL_REPORT_LUSTRE_STATS=0</span><span style="font-size:8.5pt;font-family:Menlo;color:#FC4708">"</span><span style="font-size:8.5pt;font-family:Menlo;color:#0E71CD"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New""><o:p> </o:p></span></p>
<pre><code><span style="font-size:9.0pt;font-family:Menlo;color:#0070C0">srun … </span></code><span class="hljs-variable"><span style="font-size:9.0pt;font-family:Menlo;color:#0070C0">$common</span></span><code><span style="font-size:9.0pt;font-family:Menlo;color:#0070C0"> \<o:p></o:p></span></code></pre>
<pre><code><span style="font-size:9.0pt;font-family:Menlo;color:#0070C0"> MV2_DEBUG_TOOL_VERBOSE=1 \<o:p></o:p></span></code></pre>
<pre><code><span style="font-size:9.0pt;font-family:Menlo;color:#0070C0"> MV2_TWO_LEVEL_COMM_THRESHOLD=1 \<o:p></o:p></span></code></pre>
<pre><code><span style="font-size:9.0pt;font-family:Menlo;color:#0070C0"> MV2_ON_DEMAND_THRESHOLD=1 \<o:p></o:p></span></code></pre>
<pre><code><span style="font-size:9.0pt;font-family:Menlo;color:#0070C0"> MV2_USE_RDMA_CM=0 \<o:p></o:p></span></code></pre>
<pre><code><span style="font-size:9.0pt;font-family:Menlo;color:#0070C0"> MV2_TOOL_INFO_FILE_PATH=</span></code><span class="hljs-string"><span style="font-size:9.0pt;font-family:Menlo;color:#0070C0">"<path to osu-inam.conf>"</span></span><code><span style="font-size:9.0pt;font-family:Menlo;color:#0070C0"> \<o:p></o:p></span></code></pre>
<pre><code><span style="font-size:9.0pt;font-family:Menlo;color:#0070C0"> application</span></code><span style="font-size:9.0pt;font-family:Menlo;color:#0070C0"><o:p></o:p></span></pre>
<p class="MsoNormal" style="background:white"><span style="font-size:8.5pt;font-family:Menlo;color:#FB0007"><o:p> </o:p></span></p>
<p class="MsoNormal">Please ensure you replace <span style="font-size:10.0pt;font-family:"Courier New"">
<path to osu-inam.conf></span> with the actual path to your configuration file. The output from this operation might be extensive, but it's needed for our investigation.
<u>You can set MV2_DEBUG_TOOL_VERBOSE=0 for future runs</u>. This runtime variable is for debugging purposes only.
<o:p></o:p></p>
<p class="MsoNormal" style="background:white"><span style="font-size:8.5pt;font-family:Menlo;color:black"><o:p> </o:p></span></p>
<p class="MsoNormal">Once you have gathered all the requested information, please send it to us and we will evaluate and get back to you.<o:p></o:p></p>
<p class="MsoNormal">If you have any questions or encounter any difficulties along the way, do not hesitate to reach out.<o:p></o:p></p>
<p class="MsoNormal" style="background:white"><span style="font-size:8.5pt;font-family:Menlo;color:#FC7208"><o:p> </o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal"><span style="color:black">Best,<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><span style="color:black">Pouya Kousha </span><o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal" style="margin-bottom:12.0pt"><b><span style="font-size:12.0pt;color:black">From:
</span></b><span style="font-size:12.0pt;color:black">Mvapich-discuss <mvapich-discuss-bounces@lists.osu.edu> on behalf of evancervj via Mvapich-discuss <mvapich-discuss@lists.osu.edu><br>
<b>Date: </b>Tuesday, February 13, 2024 at 1:41 PM<br>
<b>To: </b>mvapich-discuss@lists.osu.edu <mvapich-discuss@lists.osu.edu>, Lieber, Matt <lieber.31@osu.edu><br>
<b>Subject: </b>Re: [Mvapich-discuss] Issue while Installation of Mvapich2-x-advanced for OSU-INAM<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-line-height-alt:.75pt"><span style="font-size:1.0pt;color:white">Hi Matt, I have installed Slurm package (slurm-23. 11. 3-1. el7. x86_64) on the system. Next I used srun for running the application(IMB) using the following
command providing the MV2 options as mentioned in the section 'Running Example' <o:p>
</o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-line-height-alt:.75pt"><span style="font-size:1.0pt;color:white">ZjQcmQRYFpfptBannerStart<o:p></o:p></span></p>
</div>
<div style="border:none;border-top:solid #8C8E91 3.0pt;padding:0in 0in 0in 0in;display:block!important;text-align:left!important;margin:0px!important;padding:16px!important;border-radius:4px!important;min-width:200px!important;background-color:#CFD3D7!important;border-top:#8c8e91!important" id="pfptBannerw0ux1yz">
<div id="pfptBannerw0ux1yz">
<div id="pfptBannerw0ux1yz">
<p class="MsoNormal" style="line-height:13.5pt;background:#CFD3D7"><b><span lang="EN" style="font-family:"Arial",sans-serif;color:black">This Message Is From an External Sender
<o:p></o:p></span></b></p>
</div>
<div id="pfptBannerw0ux1yz">
<p class="MsoNormal" style="line-height:13.5pt;background:#CFD3D7"><span lang="EN" style="font-family:"Arial",sans-serif;color:black">This message came from outside your organization.
<o:p></o:p></span></p>
</div>
</div>
<div id="pfptBannerw0ux1yz">
<p class="MsoNormal" style="background:#CFD3D7"><span lang="EN" style="color:black"><a href="https://us-phishalarm-ewt.proofpoint.com/EWT/v1/KGKeukY!vaQTnUZgI6ZAq5mxfE7ktw8ETzxr_ESPGclUA8J1RLMU97_XY4Ve2k44z3Qyr_6pCJxNPhlOG9STYYXE-KPzOtId1PvsX5OVaCobNKZmqqLZiJ11wyrE5d8ZBaafWqUC4PtvFHk$"><span lang="EN-US" style="text-decoration:none"><o:p></o:p></span></a></span></p>
<div style="border:solid #666666 1.0pt;padding:0in 0in 0in 0in;display:inline-block!important;display:inline-block;visibility:visible!important;opacity:1!important;color:#000000!important;font-size:14px!important;font-weight:normal!important;text-decoration:none!important;border-radius:2px!important;padding:16px!important;margin:16px!important;white-space:nowrap!important;width:fit-content!important">
<p class="MsoNormal" style="background:#CFD3D7"><span lang="EN" style="font-family:"Arial",sans-serif;color:black"><a href="https://us-phishalarm-ewt.proofpoint.com/EWT/v1/KGKeukY!vaQTnUZgI6ZAq5mxfE7ktw8ETzxr_ESPGclUA8J1RLMU97_XY4Ve2k44z3Qyr_6pCJxNPhlOG9STYYXE-KPzOtId1PvsX5OVaCobNKZmqqLZiJ11wyrE5d8ZBaafWqUC4PtvFHk$"><span style="color:black;text-decoration:none">Report Suspicious
</span><span lang="EN-US" style="color:black;text-decoration:none"><o:p></o:p></span></a></span></p>
</div>
<p class="MsoNormal" style="background:#CFD3D7"><span lang="EN"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="background:#CFD3D7"><span lang="EN" style="color:black"> </span><span lang="EN"><o:p></o:p></span></p>
</div>
</div>
<div>
<p class="MsoNormal" style="mso-line-height-alt:.75pt"><span style="font-size:1.0pt;color:white">ZjQcmQRYFpfptBannerEnd<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt">Hi Matt,</span> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt"> </span> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt">I have installed Slurm package (<em><span style="font-family:"Calibri",sans-serif">slurm-23.11.3-1.el7.x86_64)</span></em> on the system. Next I used srun for running the application(IMB) using the following
command providing the MV2 options as mentioned in the section 'Running Example' of the userguide:</span>
<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt"> </span> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt"> <em><span style="font-family:"Calibri",sans-serif"> srun --mpi=pmi2 -n16 \</span></em></span>
<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><em><span style="font-size:12.0pt;font-family:"Calibri",sans-serif"> --export=MV2_ON_DEMAND_THRESHOLD=1,\</span></em>
<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><em><span style="font-size:12.0pt;font-family:"Calibri",sans-serif"> MV2_TOOL_INFO_FILE_PATH=/etc/osu-inam/osu-inam.conf,\
</span></em><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><em><span style="font-size:12.0pt;font-family:"Calibri",sans-serif"> MV2_TWO_LEVEL_COMM_THRESHOLD=1,MV2_USE_RDMA_CM=0,\</span></em><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><em><span style="font-size:12.0pt;font-family:"Calibri",sans-serif"> MV2_TOOL_REPORT_PVARS=1,MV2_ENABLE_PVAR_TIMER=1,\</span></em><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><em><span style="font-size:12.0pt;font-family:"Calibri",sans-serif"> MV2_TOOL_REPORT_PVARS=1,MV2_ENABLE_PVAR_TIMER=1,\</span></em><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><em><span style="font-size:12.0pt;font-family:"Calibri",sans-serif"> MV2_ENABLE_PVAR_COUNTER=1,MV2_ENABLE_PVAR_TIMER_BUCKETS=1,\</span></em><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><em><span style="font-size:12.0pt;font-family:"Calibri",sans-serif"> MV2_ENABLE_PVAR_COUNTER=1,MV2_ENABLE_PVAR_TIMER_BUCKETS=1,\</span></em><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><em><span style="font-size:12.0pt;font-family:"Calibri",sans-serif"> MV2_TOOL_SESSIONS_DEFAULT_ALL_HANDLES=1,MV2_TOOL_REPORT_LUSTRE_STATS=1 \</span></em>
<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><em><span style="font-size:12.0pt;font-family:"Calibri",sans-serif"> ./IMB-MPI1</span></em><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><i><span style="font-size:12.0pt"><br>
</span></i><span style="font-size:12.0pt">In addition to the packet counter information earlier availbale, this time Job information like Job-ID is visible and is being updated on the OSU_INAM web interface.
</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt">However, MPI information is not exported. The fields that are not exported like cpu and memory usage for the Jobs show the message "no data from mpi process to this job". MPI related graphs like Global or
inter-node communication graph also show the message "no data to display ". </span>
<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt">Is there something I'm missing for process level and MPI-level information to be exported. How can I get MPI-level information to be seen on the OSU-INAM interface?</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt"> </span> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt">Thanks</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt">John</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><br>
On January 24, 2024 at 9:45 PM "Lieber, Matt" <lieber.31@osu.edu> wrote: <o:p></o:p></p>
</div>
<div>
<blockquote style="border:none;border-left:solid blue 1.0pt;padding:0in 0in 0in 8.0pt;margin-left:0in;margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Aptos",sans-serif;color:black">Hi John,
<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Aptos",sans-serif;color:black">First you will need Slurm. Second the rpms that have slurm in their name for mvapich2x will use srun as their job launcher and not mpirun_rsh. Also, our user guide
under section 5 has some other useful information that will be required for the information to show up
<a href="https://mvapich.cse.ohio-state.edu/userguide/osu-inam/#_running_example">
https://mvapich.cse.ohio-state.edu/userguide/osu-inam/#_running_example</a> . </span>
<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Aptos",sans-serif;color:black">-Matt</span>
<o:p></o:p></p>
</div>
<div id="appendonsend">
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<div class="MsoNormal" align="center" style="text-align:center">
<hr size="0" width="100%" align="center">
</div>
<div id="divRplyFwdMsg">
<p class="MsoNormal"><strong><span style="font-family:"Calibri",sans-serif;color:black">From:</span></strong><span style="color:black"> evancervj <evancervj@cdac.in><br>
<strong><span style="font-family:"Calibri",sans-serif">Sent:</span></strong> Wednesday, January 24, 2024 1:55 AM<br>
<strong><span style="font-family:"Calibri",sans-serif">To:</span></strong> mvapich-discuss@lists.osu.edu <mvapich-discuss@lists.osu.edu>; Lieber, Matt <lieber.31@osu.edu><br>
<strong><span style="font-family:"Calibri",sans-serif">Subject:</span></strong> Re: [Mvapich-discuss] Issue while Installation of Mvapich2-x-advanced for OSU-INAM</span>
<o:p></o:p></p>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal" style="mso-line-height-alt:.75pt"><span style="font-size:1.0pt;color:white">HI Matt, Thanks for the suggestions. I had tried MVAPICH2X-Basic as well as using the flag --nodeps for MVAPICH2-X-Advanced. The installation is successful. I
was able to see the fabric related information like network topology and the <o:p>
</o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-line-height-alt:.75pt"><span style="font-size:1.0pt;color:white">
<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt">HI Matt,</span> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt">Thanks for the suggestions. I had tried MVAPICH2X-Basic as well as using the flag --nodeps for MVAPICH2-X-Advanced. The installation is successful. I was able to see the fabric related information like network
topology and the run-time packet counter information on the INAM web interface. </span>
<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt"> </span> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt">However process level and MPI related information was not seen on the INAM web interface. To export MPI related information to OSU_INAM, I tried running application using MVAPICH2X, but the mpirun_rsh file
was missing in the MVAPICH2X installation directory. </span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt">Both the MVAPICH2X packages(basic and advanced) did not provide the files like mpirun_rsh, mpirun.., to run applications.</span>
<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt"> </span> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><em><span style="font-size:12.0pt;font-family:"Calibri",sans-serif">rpm -Uvh --nodeps mvapich2-x-advanced-mofed5.0-gnu4.8.5-slurm-2.3-1.el7.x86_64.rpm</span></em>
<br>
<em><span style="font-size:12.0pt;font-family:"Calibri",sans-serif">Preparing... ################################# [100%]</span></em>
<br>
<em><span style="font-size:12.0pt;font-family:"Calibri",sans-serif">Updating / installing...</span></em>
<br>
<em><span style="font-size:12.0pt;font-family:"Calibri",sans-serif"> 1:mvapich2-x-advanced-mofed5.0-gnu4################################# [100%]</span></em>
<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><em><span style="font-size:12.0pt;font-family:"Calibri",sans-serif">ll /opt/mvapich2-x/gnu4.8.5/mofed5.0/advanced/slurm/bin/</span></em>
<br>
<em><span style="font-size:12.0pt;font-family:"Calibri",sans-serif">total 132</span></em>
<br>
<em><span style="font-size:12.0pt;font-family:"Calibri",sans-serif">lrwxrwxrwx 1 root root 6 Jan 23 10:27 mpic++ -> mpicxx</span></em>
<br>
<em><span style="font-size:12.0pt;font-family:"Calibri",sans-serif">-rwxr-xr-x 1 root root 10970 Jan 23 10:27 mpicc</span></em>
<br>
<em><span style="font-size:12.0pt;font-family:"Calibri",sans-serif">-rwxr-xr-x 1 root root 12856 Jun 2 2021 mpichversion</span></em>
<br>
<em><span style="font-size:12.0pt;font-family:"Calibri",sans-serif">-rwxr-xr-x 1 root root 10503 Jan 23 10:27 mpicxx</span></em>
<br>
<em><span style="font-size:12.0pt;font-family:"Calibri",sans-serif">-rwxr-xr-x 1 root root 14218 Jan 23 10:27 mpif77</span></em>
<br>
<em><span style="font-size:12.0pt;font-family:"Calibri",sans-serif">-rwxr-xr-x 1 root root 14218 Jan 23 10:27 mpif90</span></em>
<br>
<em><span style="font-size:12.0pt;font-family:"Calibri",sans-serif">-rwxr-xr-x 1 root root 14278 Jun 2 2021 mpifort</span></em>
<br>
<em><span style="font-size:12.0pt;font-family:"Calibri",sans-serif">-rwxr-xr-x 1 root root 12928 Jun 2 2021 mpiname</span></em>
<br>
<em><span style="font-size:12.0pt;font-family:"Calibri",sans-serif">-rwxr-xr-x 1 root root 23256 Jun 2 2021 mpivars</span></em>
<br>
<em><span style="font-size:12.0pt;font-family:"Calibri",sans-serif">-rwxr-xr-x 1 root root 3430 Jun 2 2021</span></em><span style="font-size:12.0pt"> parkill</span>
<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt"> </span> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt"> </span> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><em><span style="font-size:12.0pt;font-family:"Calibri",sans-serif">rpm -Uvh mvapich2-x-basic-mofed5.0-gnu4.8.5-slurm-2.3-1.el7.x86_64.rpm
</span></em><span style="font-size:12.0pt"><br>
<em><span style="font-family:"Calibri",sans-serif">Preparing... ################################# [100%]</span></em><br>
<em><span style="font-family:"Calibri",sans-serif">Updating / installing...</span></em><br>
<em><span style="font-family:"Calibri",sans-serif"> 1:mvapich2-x-basic-mofed5.0-gnu4.8.################################# [100%]</span></em><br>
<br>
<em><span style="font-family:"Calibri",sans-serif">ll /opt/mvapich2-x/gnu4.8.5/mofed5.0/basic/slurm/bin/</span></em><br>
<em><span style="font-family:"Calibri",sans-serif">total 132</span></em><br>
<em><span style="font-family:"Calibri",sans-serif">lrwxrwxrwx 1 root root 6 Jan 24 11:54 mpic++ -> mpicxx</span></em><br>
<em><span style="font-family:"Calibri",sans-serif">-rwxr-xr-x 1 root root 10756 Jan 24 11:54 mpicc</span></em><br>
<em><span style="font-family:"Calibri",sans-serif">-rwxr-xr-x 1 root root 12856 May 20 2021 mpichversion</span></em><br>
<em><span style="font-family:"Calibri",sans-serif">-rwxr-xr-x 1 root root 10324 Jan 24 11:54 mpicxx</span></em><br>
<em><span style="font-family:"Calibri",sans-serif">-rwxr-xr-x 1 root root 14036 Jan 24 11:54 mpif77</span></em><br>
<em><span style="font-family:"Calibri",sans-serif">-rwxr-xr-x 1 root root 14036 Jan 24 11:54 mpif90</span></em><br>
<em><span style="font-family:"Calibri",sans-serif">-rwxr-xr-x 1 root root 14096 May 20 2021 mpifort</span></em><br>
<em><span style="font-family:"Calibri",sans-serif">-rwxr-xr-x 1 root root 12928 May 20 2021 mpiname</span></em><br>
<em><span style="font-family:"Calibri",sans-serif">-rwxr-xr-x 1 root root 23256 May 20 2021 mpivars</span></em><br>
<em><span style="font-family:"Calibri",sans-serif">-rwxr-xr-x 1 root root 3430 May 20 2021 parkill</span></em></span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt"> </span> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt">I was able to access the fabric information on the OSU-INAM interface. I wanted to leverage the OSU-INAM features to observe the process level and MPI level information as well, but failed because of the above
mentioned issue. Any pointers on this will be helpful. </span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt"> </span> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt">Also my current environment do not have Slurm or any other job scheduler installed. So Is Slurm necessary for OSU-INAM to export MPI level information?
</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt"> </span> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt">Thanks</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt">John</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><br>
On January 20, 2024 at 6:36 AM "Lieber, Matt" <lieber.31@osu.edu> wrote: <o:p></o:p></p>
</div>
<div>
<blockquote style="border:none;border-left:solid blue 1.0pt;padding:0in 0in 0in 8.0pt;margin-left:0in;margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Aptos",sans-serif;color:black">Hi John,
<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Aptos",sans-serif;color:black">Sorry for the delay in getting back to you. There are multiple options that could fix this. If you do not plan to use the sharp functionality adding the flag --nodeps
should fix the issue you are seeing. Also MVAPICH2-X basic should also work with INAM. If you do wish to use sharp we will have to send a new rpm. Please let let us know how these options work for you.</span>
<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Aptos",sans-serif;color:black">Thanks,</span>
<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Aptos",sans-serif;color:black">Matt</span>
<o:p></o:p></p>
</div>
<div id="x_appendonsend">
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<div class="MsoNormal" align="center" style="text-align:center">
<hr size="0" width="100%" align="center">
</div>
<div id="x_divRplyFwdMsg">
<p class="MsoNormal"><strong><span style="font-family:"Calibri",sans-serif;color:black">From:</span></strong><span style="color:black"> Mvapich-discuss <mvapich-discuss-bounces@lists.osu.edu> on behalf of V John via Mvapich-discuss <mvapich-discuss@lists.osu.edu><br>
<strong><span style="font-family:"Calibri",sans-serif">Sent:</span></strong> Tuesday, January 16, 2024 7:41 AM<br>
<strong><span style="font-family:"Calibri",sans-serif">To:</span></strong> mvapich-discuss@lists.osu.edu <mvapich-discuss@lists.osu.edu><br>
<strong><span style="font-family:"Calibri",sans-serif">Subject:</span></strong> [Mvapich-discuss] Issue while Installation of Mvapich2-x-advanced for OSU-INAM</span>
<o:p></o:p></p>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal" style="mso-line-height-alt:.75pt"><span style="font-size:1.0pt;color:white">Hi everyone, I'm trying to use OSU-INAM on our cluster with Mellanox-IB ConnectX-6 interconnect with machines having Centos7. 6 OS. I downloaded the osu-inam(osu-inam-mysql-1.
0-1. el7. x86_64. rpm) package matching the environment which also required <o:p>
</o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-line-height-alt:.75pt"><span style="font-size:1.0pt;color:white">
<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal">Hi everyone, <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">I'm trying to use OSU-INAM on our cluster with Mellanox-IB ConnectX-6 interconnect with machines having Centos7.6 OS. I downloaded the osu-inam(osu-inam-mysql-1.0-1.el7.x86_64.rpm) package matching the environment which also required MOFED
5+. So I installed MLNX_OFED_LINUX-5.0-1.0.0.0 package on the systems. <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Osu-Inam user-guide mentioned the requirement of Mvapich2-x-advanced for getting MPI usage related information. Hence, I tried to install MVAPICH2-X-advanced. The only binary matching the environment available on the Mvapich website is
mvapich2-x-advanced-mofed5.0-gnu4.8.5-slurm-2.3-1.el7.x86_64.rpm. The installation fails however due to dependency on libsharp_coll.so.4.
<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><em><span style="font-family:"Calibri",sans-serif">rpm -Uvh mvapich2-x-advanced-mofed5.0-gnu4.8.5-slurm-2.3-1.el7.x86_64.rpm
</span></em><br>
<em><span style="font-family:"Calibri",sans-serif">error: Failed dependencies:</span></em>
<br>
<em><span style="font-family:"Calibri",sans-serif">libsharp_coll.so.4()(64bit) is needed by mvapich2-x-advanced-mofed5.0-gnu4.8.5-slurm-2.3-1.el7.x86_64</span></em>
<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">The libsharp_coll library versions under the /opt/mellanox/sharp/lib directory created post the installation of MLNX_OFED_LINUX-5.0-1.0.0.0 misses the required version, libsharp_coll.so.4.
<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><em><span style="font-family:"Calibri",sans-serif">ls /opt/mellanox/sharp/lib | grep libsharp_coll.so</span></em>
<br>
<em><span style="font-family:"Calibri",sans-serif">libsharp_coll.so</span></em> <br>
<em><span style="font-family:"Calibri",sans-serif">libsharp_coll.so.5</span></em>
<br>
<em><span style="font-family:"Calibri",sans-serif">libsharp_coll.so.5.0.1</span></em>
<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Is there a fix to this issue..? <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Thanks <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">John <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">HPC Technologies Group <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">C-DAC pune <o:p></o:p></p>
</div>
<p class="MsoNormal"><br>
------------------------------------------------------------------------------------------------------------
<br>
[ C-DAC is on Social-Media too. Kindly follow us at: <br>
Facebook: <a href="https://urldefense.com/v3/__https:/www.facebook.com/CDACINDIA__;!!KGKeukY!2KJphDWGFEcrgBZNFD0pyWfL_tLCH64l43moQyPTHdZMlRXL2sAhibE3ALe1Uzxh39TVZU1r-xrOX_sKpGycM_VAyzT0TQ$">
https://www.facebook.com/CDACINDIA</a> & Twitter: @cdacindia ] <br>
<br>
This e-mail is for the sole use of the intended recipient(s) and may <br>
contain confidential and privileged information. If you are not the <br>
intended recipient, please contact the sender by reply e-mail and destroy <br>
all copies and the original message. Any unauthorized review, use, <br>
disclosure, dissemination, forwarding, printing or copying of this email <br>
is strictly prohibited and appropriate legal action will be taken. <br>
------------------------------------------------------------------------------------------------------------
<o:p></o:p></p>
</div>
</blockquote>
<p class="MsoNormal"><br>
<o:p></o:p></p>
</div>
<p class="MsoNormal"><br>
------------------------------------------------------------------------------------------------------------
<br>
[ C-DAC is on Social-Media too. Kindly follow us at: <br>
Facebook: <a href="https://urldefense.com/v3/__https:/www.facebook.com/CDACINDIA__;!!KGKeukY!xP_sp9xIt5MlF-mGh9nukFmdaGDl-o3tQ8iYqzLQpVIRRub9-TeUbJ4uW-GqCJC54Y1hZ38BrLaABtnLmSWZ$">
https://www.facebook.com/CDACINDIA</a> & Twitter: @cdacindia ] <br>
<br>
This e-mail is for the sole use of the intended recipient(s) and may <br>
contain confidential and privileged information. If you are not the <br>
intended recipient, please contact the sender by reply e-mail and destroy <br>
all copies and the original message. Any unauthorized review, use, <br>
disclosure, dissemination, forwarding, printing or copying of this email <br>
is strictly prohibited and appropriate legal action will be taken. <br>
------------------------------------------------------------------------------------------------------------
<o:p></o:p></p>
</div>
</blockquote>
<p class="MsoNormal"><br>
<o:p></o:p></p>
</div>
<p class="MsoNormal"><br>
------------------------------------------------------------------------------------------------------------
<br>
[ C-DAC is on Social-Media too. Kindly follow us at: <br>
Facebook: <a href="https://urldefense.com/v3/__https:/www.facebook.com/CDACINDIA__;!!KGKeukY!xDnLo72OA_j_Ed4dReyJgNnnLbKCQdAh6avvk7JI6XHItqFiW5Z46E9g9a0A4WphLH9i1qstvTDycmA65H5Qj0GtzVFwXQ$">
https://www.facebook.com/CDACINDIA</a> & Twitter: @cdacindia ] <br>
<br>
This e-mail is for the sole use of the intended recipient(s) and may <br>
contain confidential and privileged information. If you are not the <br>
intended recipient, please contact the sender by reply e-mail and destroy <br>
all copies and the original message. Any unauthorized review, use, <br>
disclosure, dissemination, forwarding, printing or copying of this email <br>
is strictly prohibited and appropriate legal action will be taken. <br>
------------------------------------------------------------------------------------------------------------
<o:p></o:p></p>
</div>
</body>
</html>