<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd"><!-- BaNnErBlUrFlE-BoDy-start -->
<!-- Preheader Text : BEGIN -->
<div style="display:none !important;display:none;visibility:hidden;mso-hide:all;font-size:1px;color:#ffffff;line-height:1px;height:0px;max-height:0px;opacity:0;overflow:hidden;">
Hi Pouya Kousha, Thanks for the reply. I will try the suggestion given. Thanks John On April 5, 2024 at 10: 03 PM "Kousha, Pouya" <kousha. 2@ buckeyemail. osu. edu> wrote: Hi John, Sorry in the delay of
</div>
<!-- Preheader Text : END -->
<!-- Email Banner : BEGIN -->
<div style="display:none !important;display:none;visibility:hidden;mso-hide:all;font-size:1px;color:#ffffff;line-height:1px;height:0px;max-height:0px;opacity:0;overflow:hidden;">ZjQcmQRYFpfptBannerStart</div>
<!--[if ((ie)|(mso))]>
<table border="0" cellspacing="0" cellpadding="0" width="100%" style="padding: 16px 0px 16px 0px; direction: ltr" lang="en"><tr><td>
<table border="0" cellspacing="0" cellpadding="0" style="padding: 0px 10px 5px 6px; width: 100%; border-radius:4px; border-top:4px solid #8c8e91;background-color:#CFD3D7;"><tr><td valign="top">
<table align="left" border="0" cellspacing="0" cellpadding="0" style="padding: 4px 8px 4px 8px">
<tr><td style="color:#000000; font-family: 'Arial', sans-serif; font-weight:bold; font-size:14px; direction: ltr">
This Message Is From an External Sender
</td></tr>
<tr><td style="color:#000000; font-weight:normal; font-family: 'Arial', sans-serif; font-size:12px; direction: ltr">
This message came from outside your organization.
</td></tr>
</table>
<![if ie]><br clear="all"><![endif]>
<table align="right" border="0" cellspacing="0" cellpadding="0" style="padding: 4px 0px 4px 0px"><tr>
<td style="direction: ltr"> <a target="_blank" href="https://us-phishalarm-ewt.proofpoint.com/EWT/v1/KGKeukY!vcQTnQagIycgChgx3E6lV79uDqOdpFXEiHnbA4U2tE_N8KgwrDF-Tg4mD-sAazOP5toflAYf5BDpXWysZe-Qu6IrkDQi_gCfD9O0LDMeA5z0Vtuw3lksFfStrGapjKpytlLK_W78$" style="mso-padding-alt: 7.5px; padding: 7.5px; border-radius: 2px; border: 1.5px solid #666666; "><strong style="font-weight: normal; color: #000000; text-decoration: none; font-family: 'Arial', sans-serif; font-size:14px; line-height: 40px; "> Report Suspicious </strong></a> </td>
</tr></table>
</td></tr></table>
</td></tr></table>
<![endif]-->
<![if !((ie)|(mso))]>
<div dir="ltr" lang="en" id="pfptBanner97zweow" style="all: revert !important; display:block !important; text-align: left !important; margin:16px 0px 16px 0px !important; padding:8px 16px 8px 16px !important; border-radius: 4px !important; min-width: 200px !important; background-color: #CFD3D7 !important; background-color: #CFD3D7; border-top: 4px solid #8c8e91 !important; border-top: 4px solid #8c8e91;">
<div id="pfptBanner97zweow" style="all: unset !important; float:left !important; display:block !important; margin: 0px 0px 1px 0px !important; max-width: 600px !important;">
<div id="pfptBanner97zweow" style="all: unset !important; display:block !important; visibility: visible !important; background-color: #CFD3D7 !important; color:#000000 !important; color:#000000; font-family: 'Arial', sans-serif !important; font-family: 'Arial', sans-serif; font-weight:bold !important; font-weight:bold; font-size:14px !important; line-height:18px !important; line-height:18px">
This Message Is From an External Sender
</div>
<div id="pfptBanner97zweow" style="all: unset !important; display:block !important; visibility: visible !important; background-color: #CFD3D7 !important; color:#000000 !important; color:#000000; font-weight:normal; font-family: 'Arial', sans-serif !important; font-family: 'Arial', sans-serif; font-size:12px !important; line-height:18px !important; line-height:18px; margin-top:2px !important;">
This message came from outside your organization.
</div>
</div>
<div id="pfptBanner97zweow" style="all: unset !important; float: right !important; display: block !important; display: block; margin: 0px 0px 0px 16px !important; text-align: right !important; width: fit-content !important;">
<a id="pfptBanner97zweow" href="https://us-phishalarm-ewt.proofpoint.com/EWT/v1/KGKeukY!vcQTnQagIycgChgx3E6lV79uDqOdpFXEiHnbA4U2tE_N8KgwrDF-Tg4mD-sAazOP5toflAYf5BDpXWysZe-Qu6IrkDQi_gCfD9O0LDMeA5z0Vtuw3lksFfStrGapjKpytlLK_W78$"
style="all: unset !important; display: inline-block !important; text-decoration: none">
<div class="pfptPrimaryButton97zweow" style="display: inline-block !important; display: inline-block; visibility: visible !important; opacity: 1 !important; color: #000000 !important; color: #000000; font-family: 'Arial', sans-serif !important; font-family: 'Arial', sans-serif; font-size: 14px !important; font-weight: normal !important; text-decoration: none !important; border-radius: 2px !important; padding: 7.5px 16px !important; margin: 3px 0 3px 16px !important; white-space: nowrap !important; width: fit-content !important;
border: 1px solid #666666">
Report Suspicious
</div>
</a>
</div>
<div style="clear: both !important; display: block !important; visibility: hidden !important; line-height: 0 !important; font-size: 0.01px !important; height: 0px"> </div>
</div>
<![endif]>
<div style="display:none !important;display:none;visibility:hidden;mso-hide:all;font-size:1px;color:#ffffff;line-height:1px;height:0px;max-height:0px;opacity:0;overflow:hidden;">ZjQcmQRYFpfptBannerEnd</div>
<!-- Email Banner : END -->
<!-- BaNnErBlUrFlE-BoDy-end -->
<html xmlns="http://www.w3.org/1999/xhtml"><head><!-- BaNnErBlUrFlE-HeAdEr-start -->
<style>
#pfptBanner97zweow { all: revert !important; display: block !important;
visibility: visible !important; opacity: 1 !important;
background-color: #CFD3D7 !important;
max-width: none !important; max-height: none !important }
.pfptPrimaryButton97zweow:hover, .pfptPrimaryButton97zweow:focus {
background-color: #adb0b4 !important; }
.pfptPrimaryButton97zweow:active {
background-color: #8c8e91 !important; }
</style>
<!-- BaNnErBlUrFlE-HeAdEr-end -->
<meta content="text/html; charset=UTF-8" http-equiv="Content-Type"/>
<style type="text/css">v\:* {}
o\:* {}
w\:* {}
.shape {}
<!-- /* Font Definitions */ @font-face {font-family: "Cambria Math"; }
@font-face {font-family: Calibri; }
@font-face {font-family: Aptos; }
@font-face {font-family: Consolas; }
@font-face {font-family: Menlo; }
/* Style Definitions */ p.MsoNormal, li.MsoNormal, div.MsoNormal {margin: 0in; font-size: 11.0pt; font-family: "Calibri",sans-serif;}
a:link, span.MsoHyperlink { color: blue; text-decoration: underline;}
pre { margin: 0in; margin-bottom: .0001pt; font-size: 10.0pt; font-family: "Courier New";}
span.HTMLPreformattedChar { font-family: Consolas;}
p.p1, li.p1, div.p1 { margin: 0in; font-size: 8.5pt; font-family: Menlo; color: black;}
span.s1 {}
span.hljs-variable {}
span.hljs-string {}
.MsoChpDefault { font-size: 10.0pt;}
@page WordSection1 { margin: 1.0in 1.0in 1.0in 1.0in;}
div.WordSection1 {}
-->
</style></head><body style="">
<div>
<span style="font-size: 12.0pt;"><span style="color: black;">Hi Pouya Kousha</span>,</span>
</div>
<div>
<span style="font-size: 12.0pt;"> </span>
</div>
<div>
<span style="font-size: 12.0pt;">Thanks for the reply. I will try the suggestion given.<br/></span>
</div>
<div>
<span style="font-size: 12.0pt;"> </span>
</div>
<div>
<span style="font-size: 12.0pt;">Thanks<br/></span>
</div>
<div>
<span style="font-size: 12.0pt;">John<br/></span>
</div>
<div>
<span style="font-size: 12.0pt;"> </span>
</div>
<div>
<br/>On April 5, 2024 at 10:03 PM "Kousha, Pouya" <kousha.2@buckeyemail.osu.edu> wrote:
</div>
<div style="position: relative;">
<blockquote style="margin-left: 0px; padding-left: 10px; border-left: solid 1px blue;" type="cite">
<!-- [if !mso]> -->
<!-- <![endif] -->
<!-- [if gte mso 9]> -->
<!-- <![endif] -->
<!-- [if gte mso 9]> -->
<!-- <![endif] -->
<div class="WordSection1">
<p class="MsoNormal">Hi John,</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">Sorry in the delay of getting back to you.</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">I tested our RPMs using <a href="http://mvapich.cse.ohio-state.edu/download/mvapich/mv2x/2.3/mofed5.0/mvapich2-x-basic-mofed5.0-gnu4.8.5-slurm-2.3-1.el7.x86_64.rpm"> this link</a> as well as building from source.</p>
<p class="MsoNormal">With srun instead of exporting common, we need to export each variable individually. So instead of this line below export each variable individually.</p>
<p class="MsoNormal"><span style="font-size: 10.5pt; color: #4472c4;"> export common="MV2_TWO_LEVEL_COMM_THRESHOLD=1,MV2_USE_RDMA_CM=0,\ </span></p>
<p class="MsoNormal"><span style="font-size: 10.5pt; color: #4472c4;"> MV2_TOOL_REPORT_PVARS=1,MV2_DEBUG_TOOL_VERBOSE=1,\ </span></p>
<p class="MsoNormal"><span style="font-size: 10.5pt; color: #4472c4;"> MV2_USE_SHMEM_COLL=1,MV2_ENABLE_PVAR_TIMER=1,\ </span></p>
<p class="MsoNormal"><span style="font-size: 10.5pt; color: #4472c4;"> MV2_ENABLE_PVAR_COUNTER=1,MV2_ENABLE_PVAR_TIMER_BUCKETS=1,\ </span></p>
<p class="MsoNormal"><span style="font-size: 10.5pt; color: #4472c4;"> MV2_ENABLE_PVAR_COUNTER_BUCKETS=1,MV2_TOOL_REPORT_SESSIONS=1,\ </span></p>
<p class="MsoNormal"><span style="font-size: 10.5pt; color: #4472c4;"> MV2_TOOL_SESSIONS_DEFAULT_ALL_HANDLES=1,\ </span></p>
<p class="MsoNormal"><span style="font-size: 10.5pt; color: #4472c4;"> MV2_TOOL_REPORT_LUSTRE_STATS=0,MV2_ON_DEMAND_THRESHOLD=1,\ </span></p>
<p class="MsoNormal"><span style="font-size: 10.5pt; color: #4472c4;"> MV2_TOOL_INFO_FILE_PATH=/etc/osu-inam/osu-inam.conf"</span></p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">I tested against INAM with both MySQL and ClickHouse Databases and I was able to see the information on the INAM web page. The RPMs are available <a href="http://mvapich.cse.ohio-state.edu/downloads/">here</a></p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">Kindly let me know if this would resolve your issue. Thanks!</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">I am listing the content of the test file I used to run MVAPICH2-x and sending information to OSU INAM below:</p>
<p class="MsoNormal"> </p>
<p style="background: white;" class="MsoNormal"><span style="font-size: 8.5pt; font-family: Menlo; color: #fc4708;">export</span><span style="font-size: 8.5pt; font-family: Menlo; color: #fb0074;"> PPN=</span><span style="font-size: 8.5pt; font-family: Menlo; color: #0d6eff;">8</span><span style="font-size: 8.5pt; font-family: Menlo; color: black;"> <span style="background: #FECDCE;"> </span></span></p>
<p style="background: white;" class="MsoNormal"><span style="font-size: 8.5pt; font-family: Menlo; color: #fc4708;">export</span><span style="font-size: 8.5pt; font-family: Menlo; color: #fb0074;"> N=</span><span style="font-size: 8.5pt; font-family: Menlo; color: #fb0007;">$SLURM_NNODES</span><span style="font-size: 8.5pt; font-family: Menlo; color: black;"> <span style="background: #FECDCE;"> </span></span></p>
<p style="background: white;" class="MsoNormal"><span style="font-size: 8.5pt; font-family: Menlo; color: #fc4708;">export</span><span style="font-size: 8.5pt; font-family: Menlo; color: #fb0074;"> n=</span><span style="font-size: 8.5pt; font-family: Menlo; color: #fb0007;">$(($N</span><span style="font-size: 8.5pt; font-family: Menlo; color: #fc7208;">*</span><span style="font-size: 8.5pt; font-family: Menlo; color: #fb0007;">$PPN))</span><span style="font-size: 8.5pt; font-family: Menlo; color: black;"> <span style="background: #FECDCE;"> </span></span></p>
<p style="background: white;" class="MsoNormal"><span style="font-size: 8.5pt; font-family: Menlo; color: black;"> <span style="background: #FECDCE;"> </span></span></p>
<p style="background: white;" class="MsoNormal"><span style="font-size: 8.5pt; font-family: Menlo; color: #fc4708;">export</span><span style="font-size: 8.5pt; font-family: Menlo; color: #fb0074;"> LD_LIBRARY_PATH=</span><span style="font-size: 8.5pt; font-family: Menlo; color: black;">/home/kousha.2/</span><span style="font-size: 8.5pt; font-family: Menlo; color: #fc4708;">test</span><span style="font-size: 8.5pt; font-family: Menlo; color: black;">-</span><span style="font-size: 8.5pt; font-family: Menlo; color: #fc4708;">rpm</span><span style="font-size: 8.5pt; font-family: Menlo; color: black;">/opt/mvapich2-x/gnu4.8.5/mofed5.0/basic/slu<span style="background: #FECDCE;">r</span>m/lib64:</span><span style="font-size: 8.5pt; font-family: Menlo; color: #fb0007;">$LD_LIBRARY_PATH</span><span style="font-size: 8.5pt; font-family: Menlo; color: black;"></span></p>
<p style="background: white;" class="MsoNormal"><span style="font-size: 8.5pt; font-family: Menlo; color: #fc4708;">export</span><span style="font-size: 8.5pt; font-family: Menlo; color: #fb0074;"> PATH=</span><span style="font-size: 8.5pt; font-family: Menlo; color: black;">/home/kousha.2/</span><span style="font-size: 8.5pt; font-family: Menlo; color: #fc4708;">test</span><span style="font-size: 8.5pt; font-family: Menlo; color: black;">-</span><span style="font-size: 8.5pt; font-family: Menlo; color: #fc4708;">rpm</span><span style="font-size: 8.5pt; font-family: Menlo; color: black;">/opt/mvapich2-x/gnu4.8.5/mofed5.0/basic/slurm/bin:</span><span style="font-size: 8.5pt; font-family: Menlo; color: #fb0007;">$PAT<span style="background: #FECDCE;">H</span></span><span style="font-size: 8.5pt; font-family: Menlo; color: black;"></span></p>
<p style="background: white;" class="MsoNormal"><strong><span style="font-size: 8.5pt; font-family: Menlo; color: #16a403;">#export LD_LIBRARY_PATH=/home/kousha.2/projects/master-x/mvapich2/install-dir/lib:$LD_LIB<span style="background: #FECDCE;">R</span>ARY_PATH</span></strong><span style="font-size: 8.5pt; font-family: Menlo; color: #16a403;"></span></p>
<p style="background: white;" class="MsoNormal"><strong><span style="font-size: 8.5pt; font-family: Menlo; color: #16a403;">#export PATH=/home/kousha.2/projects/master-x/mvapich2/install-dir/bin:$PATH </span></strong><span style="font-size: 8.5pt; font-family: Menlo; color: black;"> <span style="background: #FECDCE;"> </span></span><span style="font-size: 8.5pt; font-family: Menlo; color: #16a403;"></span></p>
<p style="background: white;" class="MsoNormal"><span style="font-size: 8.5pt; font-family: Menlo; color: #fc4708;">export</span><span style="font-size: 8.5pt; font-family: Menlo; color: #fb0074;"> MPI=</span><span style="font-size: 8.5pt; font-family: Menlo; color: #fc4708;">"</span><span style="font-size: 8.5pt; font-family: Menlo; color: #0e71cd;">/home/kousha.2/projects/master-x/mvapich2/install-dir</span><span style="font-size: 8.5pt; font-family: Menlo; color: #fc4708;">"</span><span style="font-size: 8.5pt; font-family: Menlo; color: black;"> <span style="background: #FECDCE;"> </span></span><span style="font-size: 8.5pt; font-family: Menlo; color: #0e71cd;"></span></p>
<p style="background: white;" class="MsoNormal"><span style="font-size: 8.5pt; font-family: Menlo; color: #fc4708;">export</span><span style="font-size: 8.5pt; font-family: Menlo; color: #fb0074;"> OMB=</span><span style="font-size: 8.5pt; font-family: Menlo; color: #fc4708;">"</span><span style="font-size: 8.5pt; font-family: Menlo; color: #0e71cd;">/home/kousha.2/projects/master-x/mvapich2/install-dir</span><span style="font-size: 8.5pt; font-family: Menlo; color: #fc4708;">"</span><span style="font-size: 8.5pt; font-family: Menlo; color: black;"> <span style="background: #FECDCE;"> </span></span><span style="font-size: 8.5pt; font-family: Menlo; color: #0e71cd;"></span></p>
<p style="background: white;" class="MsoNormal"><span style="font-size: 8.5pt; font-family: Menlo; color: black;"> <span style="background: #FECDCE;"> </span></span></p>
<p style="background: white;" class="MsoNormal"><span style="font-size: 8.5pt; font-family: Menlo; color: #fc4708;">export</span><span style="font-size: 8.5pt; font-family: Menlo; color: #fb0074;"> MV2_TOOL_INFO_FILE_PATH=</span><span style="font-size: 8.5pt; font-family: Menlo; color: #fc4708;">"</span><span style="font-size: 8.5pt; font-family: Menlo; color: #0e71cd;">/home/kousha.2/inam.conf</span><span style="font-size: 8.5pt; font-family: Menlo; color: #fc4708;">"</span><span style="font-size: 8.5pt; font-family: Menlo; color: black;"> <span style="background: #FECDCE;"> </span></span></p>
<p style="background: white;" class="MsoNormal"><span style="font-size: 8.5pt; font-family: Menlo; color: #fc4708;">export</span><span style="font-size: 8.5pt; font-family: Menlo; color: #fb0074;"> HOSTFILE=</span><span style="font-size: 8.5pt; font-family: Menlo; color: #fb0007;">$PWD</span><span style="font-size: 8.5pt; font-family: Menlo; color: black;">/hostfile.</span><span style="font-size: 8.5pt; font-family: Menlo; color: #fb0007;">$SLURM_JOBID</span><span style="font-size: 8.5pt; font-family: Menlo; color: black;"> <span style="background: #FECDCE;"> </span></span></p>
<p style="background: white;" class="MsoNormal"><span style="font-size: 8.5pt; font-family: Menlo; color: black;"> <span style="background: #FECDCE;"> </span></span></p>
<p style="background: white;" class="MsoNormal"><span style="font-size: 8.5pt; font-family: Menlo; color: #fc4708;">echo</span><span style="font-size: 8.5pt; font-family: Menlo; color: #0e71cd;"> </span><span style="font-size: 8.5pt; font-family: Menlo; color: #fc4708;">"</span><span style="font-size: 8.5pt; font-family: Menlo; color: #0e71cd;">mpi=</span><span style="font-size: 8.5pt; font-family: Menlo; color: #fb0007;">$MPI</span><span style="font-size: 8.5pt; font-family: Menlo; color: #fc4708;">"</span><span style="font-size: 8.5pt; font-family: Menlo; color: black;"> <span style="background: #FECDCE;"> </span></span></p>
<p style="background: white;" class="MsoNormal"><span style="font-size: 8.5pt; font-family: Menlo; color: #fc4708;">echo</span><span style="font-size: 8.5pt; font-family: Menlo; color: #0e71cd;"> </span><span style="font-size: 8.5pt; font-family: Menlo; color: #fc4708;">"</span><span style="font-size: 8.5pt; font-family: Menlo; color: #0e71cd;">(</span><span style="font-size: 8.5pt; font-family: Menlo; color: #fb0007;">$(</span><span style="font-size: 8.5pt; font-family: Menlo; color: #fc7208;">which mpicc</span><span style="font-size: 8.5pt; font-family: Menlo; color: #fb0007;">)</span><span style="font-size: 8.5pt; font-family: Menlo; color: #0e71cd;">)</span><span style="font-size: 8.5pt; font-family: Menlo; color: #fc4708;">"</span><span style="font-size: 8.5pt; font-family: Menlo; color: black;"> <span style="background: #FECDCE;"> </span></span></p>
<p style="background: white;" class="MsoNormal"><span style="font-size: 8.5pt; font-family: Menlo; color: black;"> <span style="background: #FECDCE;"> </span></span></p>
<p style="background: white;" class="MsoNormal"><span style="font-size: 8.5pt; font-family: Menlo; color: #fc4708;">echo</span><span style="font-size: 8.5pt; font-family: Menlo; color: #0e71cd;"> </span><span style="font-size: 8.5pt; font-family: Menlo; color: #fc4708;">"</span><span style="font-size: 8.5pt; font-family: Menlo; color: #0e71cd;">path=</span><span style="font-size: 8.5pt; font-family: Menlo; color: #fb0007;">$PATH</span><span style="font-size: 8.5pt; font-family: Menlo; color: #fc4708;">"</span><span style="font-size: 8.5pt; font-family: Menlo; color: black;"> <span style="background: #FECDCE;"> </span></span></p>
<p style="background: white;" class="MsoNormal"><span style="font-size: 8.5pt; font-family: Menlo; color: #fc4708;">echo</span><span style="font-size: 8.5pt; font-family: Menlo; color: #0e71cd;"> </span><span style="font-size: 8.5pt; font-family: Menlo; color: #fc4708;">"</span><span style="font-size: 8.5pt; font-family: Menlo; color: #0e71cd;">LD=</span><span style="font-size: 8.5pt; font-family: Menlo; color: #fb0007;">$LD_LIBRARY_PATH</span><span style="font-size: 8.5pt; font-family: Menlo; color: #fc4708;">"</span><span style="font-size: 8.5pt; font-family: Menlo; color: black;"> <span style="background: #FECDCE;"> </span></span></p>
<p style="background: white;" class="MsoNormal"><span style="font-size: 8.5pt; font-family: Menlo; color: #fc4708;">echo</span><span style="font-size: 8.5pt; font-family: Menlo; color: #0e71cd;"> </span><span style="font-size: 8.5pt; font-family: Menlo; color: #fc4708;">'</span><span style="font-size: 8.5pt; font-family: Menlo; color: #0e71cd;">START</span><span style="font-size: 8.5pt; font-family: Menlo; color: #fc4708;">'</span><span style="font-size: 8.5pt; font-family: Menlo; color: black;"> <span style="background: #FECDCE;"> </span></span></p>
<p style="background: white;" class="MsoNormal"><span style="font-size: 8.5pt; font-family: Menlo; color: black;"> <span style="background: #FECDCE;"> </span></span></p>
<p style="background: white;" class="MsoNormal"><span style="font-size: 8.5pt; font-family: Menlo; color: #fc4708;">export</span><span style="font-size: 8.5pt; font-family: Menlo; color: #fb0074;"> MV2_TWO_LEVEL_COMM_THRESHOLD=</span><span style="font-size: 8.5pt; font-family: Menlo; color: #0d6eff;">1</span><span style="font-size: 8.5pt; font-family: Menlo; color: black;"> <span style="background: #FECDCE;"> </span></span></p>
<p style="background: white;" class="MsoNormal"><span style="font-size: 8.5pt; font-family: Menlo; color: #fc4708;">export</span><span style="font-size: 8.5pt; font-family: Menlo; color: #fb0074;"> MV2_USE_RDMA_CM=</span><span style="font-size: 8.5pt; font-family: Menlo; color: #0d6eff;">0</span><span style="font-size: 8.5pt; font-family: Menlo; color: black;"> <span style="background: #FECDCE;"> </span></span></p>
<p style="background: white;" class="MsoNormal"><span style="font-size: 8.5pt; font-family: Menlo; color: #fc4708;">export</span><span style="font-size: 8.5pt; font-family: Menlo; color: #fb0074;"> MV2_TOOL_REPORT_PVARS=</span><span style="font-size: 8.5pt; font-family: Menlo; color: #0d6eff;">1</span><span style="font-size: 8.5pt; font-family: Menlo; color: black;"> </span></p>
<p style="background: white;" class="MsoNormal"><strong><span style="font-size: 8.5pt; font-family: Menlo; color: #16a403;">#Disable DEBUG running in production</span></strong><span style="font-size: 8.5pt; font-family: Menlo; color: black;"> <span style="background: #FECDCE;"> </span></span></p>
<p style="background: white;" class="MsoNormal"><span style="font-size: 8.5pt; font-family: Menlo; color: #fc4708;">export</span><span style="font-size: 8.5pt; font-family: Menlo; color: #fb0074;"> MV2_DEBUG_TOOL_VERBOSE=</span><span style="font-size: 8.5pt; font-family: Menlo; color: #0d6eff;">2</span><span style="font-size: 8.5pt; font-family: Menlo; color: black;"> <span style="background: #FECDCE;"> </span></span></p>
<p style="background: white;" class="MsoNormal"><span style="font-size: 8.5pt; font-family: Menlo; color: #fc4708;">export</span><span style="font-size: 8.5pt; font-family: Menlo; color: #fb0074;"> MV2_USE_SHMEM_COLL=</span><span style="font-size: 8.5pt; font-family: Menlo; color: #0d6eff;">1</span><span style="font-size: 8.5pt; font-family: Menlo; color: black;"> <span style="background: #FECDCE;"> </span></span></p>
<p style="background: white;" class="MsoNormal"><span style="font-size: 8.5pt; font-family: Menlo; color: #fc4708;">export</span><span style="font-size: 8.5pt; font-family: Menlo; color: #fb0074;"> MV2_ENABLE_PVAR_TIMER=</span><span style="font-size: 8.5pt; font-family: Menlo; color: #0d6eff;">1</span><span style="font-size: 8.5pt; font-family: Menlo; color: black;"> <span style="background: #FECDCE;"> </span></span></p>
<p style="background: white;" class="MsoNormal"><span style="font-size: 8.5pt; font-family: Menlo; color: #fc4708;">export</span><span style="font-size: 8.5pt; font-family: Menlo; color: #fb0074;"> MV2_ENABLE_PVAR_COUNTER=</span><span style="font-size: 8.5pt; font-family: Menlo; color: #0d6eff;">1</span><span style="font-size: 8.5pt; font-family: Menlo; color: black;"> <span style="background: #FECDCE;"> </span></span></p>
<p style="background: white;" class="MsoNormal"><span style="font-size: 8.5pt; font-family: Menlo; color: #fc4708;">export</span><span style="font-size: 8.5pt; font-family: Menlo; color: #fb0074;"> MV2_ENABLE_PVAR_TIMER_BUCKETS=</span><span style="font-size: 8.5pt; font-family: Menlo; color: #0d6eff;">1</span><span style="font-size: 8.5pt; font-family: Menlo; color: black;"> <span style="background: #FECDCE;"> </span></span></p>
<p style="background: white;" class="MsoNormal"><span style="font-size: 8.5pt; font-family: Menlo; color: #fc4708;">export</span><span style="font-size: 8.5pt; font-family: Menlo; color: #fb0074;"> MV2_ENABLE_PVAR_COUNTER_BUCKETS=</span><span style="font-size: 8.5pt; font-family: Menlo; color: #0d6eff;">1</span><span style="font-size: 8.5pt; font-family: Menlo; color: black;"> <span style="background: #FECDCE;"> </span></span></p>
<p style="background: white;" class="MsoNormal"><span style="font-size: 8.5pt; font-family: Menlo; color: #fc4708;">export</span><span style="font-size: 8.5pt; font-family: Menlo; color: #fb0074;"> MV2_TOOL_REPORT_SESSIONS=</span><span style="font-size: 8.5pt; font-family: Menlo; color: #0d6eff;">1</span><span style="font-size: 8.5pt; font-family: Menlo; color: black;"> <span style="background: #FECDCE;"> </span></span></p>
<p style="background: white;" class="MsoNormal"><span style="font-size: 8.5pt; font-family: Menlo; color: #fc4708;">export</span><span style="font-size: 8.5pt; font-family: Menlo; color: #fb0074;"> MV2_TOOL_SESSIONS_DEFAULT_ALL_HANDLES=</span><span style="font-size: 8.5pt; font-family: Menlo; color: #0d6eff;">1</span><span style="font-size: 8.5pt; font-family: Menlo; color: black;"> <span style="background: #FECDCE;"> </span></span></p>
<p style="background: white;" class="MsoNormal"><span style="font-size: 8.5pt; font-family: Menlo; color: #fc4708;">export</span><span style="font-size: 8.5pt; font-family: Menlo; color: #fb0074;"> MV2_TOOL_REPORT_LUSTRE_STATS=</span><span style="font-size: 8.5pt; font-family: Menlo; color: #0d6eff;">0</span><span style="font-size: 8.5pt; font-family: Menlo; color: black;"> <span style="background: #FECDCE;"> </span></span></p>
<p style="background: white;" class="MsoNormal"><span style="font-size: 8.5pt; font-family: Menlo; color: #fc4708;">export</span><span style="font-size: 8.5pt; font-family: Menlo; color: #fb0074;"> MV2_ON_DEMAND_THRESHOLD=</span><span style="font-size: 8.5pt; font-family: Menlo; color: #0d6eff;">1</span><span style="font-size: 8.5pt; font-family: Menlo; color: black;"> <span style="background: #FECDCE;"> </span></span></p>
<p style="background: white;" class="MsoNormal"><span style="font-size: 8.5pt; font-family: Menlo; color: black;"> <span style="background: #FECDCE;"> </span></span></p>
<p style="background: white;" class="MsoNormal"><span style="font-size: 8.5pt; font-family: Menlo; color: black;"> <span style="background: #FECDCE;"> </span></span></p>
<p style="background: white;" class="MsoNormal"><span style="font-size: 8.5pt; font-family: Menlo; color: #fc4708;">for</span><span style="font-size: 8.5pt; font-family: Menlo; color: #fb0007;">((</span><span style="font-size: 8.5pt; font-family: Menlo; color: #fb0074;">i</span><span style="font-size: 8.5pt; font-family: Menlo; color: black;">=</span><span style="font-size: 8.5pt; font-family: Menlo; color: #0d6eff;">0</span><span style="font-size: 8.5pt; font-family: Menlo; color: black;">; i<</span><span style="font-size: 8.5pt; font-family: Menlo; color: #0d6eff;">100</span><span style="font-size: 8.5pt; font-family: Menlo; color: black;">; i++</span><span style="font-size: 8.5pt; font-family: Menlo; color: #fb0007;">))</span><span style="font-size: 8.5pt; font-family: Menlo; color: black;"> <span style="background: #FECDCE;"> </span></span></p>
<p style="background: white;" class="MsoNormal"><span style="font-size: 8.5pt; font-family: Menlo; color: #fc4708;">do</span><span style="font-size: 8.5pt; font-family: Menlo; color: black;"> <span style="background: #FECDCE;"> </span></span></p>
<p style="background: white;" class="MsoNormal"><span style="font-size: 8.5pt; font-family: Menlo; color: black;"> </span><span style="font-size: 8.5pt; font-family: Menlo; color: #fc4708;">echo</span><span style="font-size: 8.5pt; font-family: Menlo; color: #0e71cd;"> </span><span style="font-size: 8.5pt; font-family: Menlo; color: #fc4708;">"</span><span style="font-size: 8.5pt; font-family: Menlo; color: #0e71cd;"> srun --mpi=pmi2 --export=ALL -N </span><span style="font-size: 8.5pt; font-family: Menlo; color: #fb0007;">$N</span><span style="font-size: 8.5pt; font-family: Menlo; color: #0e71cd;"> -n </span><span style="font-size: 8.5pt; font-family: Menlo; color: #fb0007;">$n</span><span style="font-size: 8.5pt; font-family: Menlo; color: #0e71cd;"> </span><span style="font-size: 8.5pt; font-family: Menlo; color: #fb0007;">$OMB</span><span style="font-size: 8.5pt; font-family: Menlo; color: #0e71cd;">/libexec/osu-micro-benchmarks/mpi<span style="background: #FECDCE;">/</span>collective/osu_allreduce -i 5000 </span><span style="font-size: 8.5pt; font-family: Menlo; color: #fc4708;">"</span><span style="font-size: 8.5pt; font-family: Menlo; color: #0e71cd;"></span></p>
<p style="background: white;" class="MsoNormal"><span style="font-size: 8.5pt; font-family: Menlo; color: black;"> <span style="background: #FECDCE;"> </span></span></p>
<p style="background: white;" class="MsoNormal"><span style="font-size: 8.5pt; font-family: Menlo; color: black;"> srun </span><span style="font-size: 8.5pt; font-family: Menlo; color: #fc7208;">--mpi=pmi2</span><span style="font-size: 8.5pt; font-family: Menlo; color: black;"> </span><span style="font-size: 8.5pt; font-family: Menlo; color: #fc7208;">--export=ALL</span><span style="font-size: 8.5pt; font-family: Menlo; color: black;"> </span><span style="font-size: 8.5pt; font-family: Menlo; color: #fc4708;">-N</span><span style="font-size: 8.5pt; font-family: Menlo; color: black;"> </span><span style="font-size: 8.5pt; font-family: Menlo; color: #fb0007;">$N</span><span style="font-size: 8.5pt; font-family: Menlo; color: black;"> </span><span style="font-size: 8.5pt; font-family: Menlo; color: #fc4708;">-n</span><span style="font-size: 8.5pt; font-family: Menlo; color: black;"> </span><span style="font-size: 8.5pt; font-family: Menlo; color: #fb0007;">$n</span><span style="font-size: 8.5pt; font-family: Menlo; color: black;"> </span><span style="font-size: 8.5pt; font-family: Menlo; color: #fb0007;">$OMB</span><span style="font-size: 8.5pt; font-family: Menlo; color: black;">/libexec/osu-micro-benchmarks/mpi/collec<span style="background: #FECDCE;">t</span>ive/osu_allreduce </span><span style="font-size: 8.5pt; font-family: Menlo; color: #fc4708;">-i</span><span style="font-size: 8.5pt; font-family: Menlo; color: black;"> </span><span style="font-size: 8.5pt; font-family: Menlo; color: #0d6eff;">5000</span><span style="font-size: 8.5pt; font-family: Menlo; color: black;"></span></p>
<p style="background: white;" class="MsoNormal"><span style="font-size: 8.5pt; font-family: Menlo; color: #fc4708;">done</span><span style="font-size: 8.5pt; font-family: Menlo; color: black;"> <span style="background: #FECDCE;"> </span></span></p>
<p class="MsoNormal"> </p>
<p class="MsoNormal"> </p>
<div>
<div>
<p class="MsoNormal"><span style="color: black;">Best,</span></p>
</div>
</div>
<p class="MsoNormal"><span style="color: black;">Pouya Kousha </span></p>
<p class="MsoNormal"> </p>
<div style="border: none; border-top: solid #B5C4DF 1.0pt; padding: 3.0pt 0in 0in 0in;">
<p style="margin-bottom: 12.0pt;" class="MsoNormal"><strong><span style="font-size: 12.0pt; color: black;">From: </span></strong><span style="font-size: 12.0pt; color: black;">evancervj <evancervj@cdac.in><br/> <strong>Date: </strong>Tuesday, March 26, 2024 at 6:16 AM<br/> <strong>To: </strong>Kousha, Pouya <kousha.2@buckeyemail.osu.edu>, Announcement about MVAPICH2 (MPI over InfiniBand, RoCE, Omni-Path, iWARP and EFA) Libraries developed at NBCL/OSU <mvapich-discuss@lists.osu.edu>, Lieber, Matt <lieber.31@osu.edu><br/> <strong>Subject: </strong>GENTLE_REMINDER: [Mvapich-discuss] Issue while Installation of Mvapich2-x-advanced for OSU-INAM</span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size: 1.0pt; color: white;"> On February 21, 2024 at 2: 44 PM evancervj <evancervj@ cdac. in> wrote: Hi Pouya Kousha, Sorry for delay in replying with the log files gathered during the trials. The following changes were made in inamd. conf. Once </span></p>
</div>
<div>
</div>
<div>
<p class="MsoNormal"> </p>
</div>
<div>
<p class="MsoNormal"><br/> On February 21, 2024 at 2:44 PM evancervj <evancervj@cdac.in> wrote:</p>
</div>
<div>
<blockquote style="border: none; border-left: solid blue 1.0pt; padding: 0in 0in 0in 8.0pt; margin-left: 0in; margin-top: 5.0pt; margin-bottom: 5.0pt;">
<div>
<p class="MsoNormal"><span style="font-size: 12.0pt;">Hi <span style="color: black;"> Pouya Kousha</span>,</span></p>
</div>
<div>
<p class="MsoNormal"> </p>
</div>
<div>
<p class="MsoNormal"><span style="font-size: 12.0pt;">Sorry for delay in replying with the log files gathered during the trials. </span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size: 12.0pt;"> </span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size: 12.0pt;">The following changes were made in inamd.conf. Once the changes were made, the osu-inamd and osu-inamweb services were restarted.</span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size: 12.0pt;"> ##### INAM debug flags #####<br/> <br/> INAM_DEBUG_INIT_VERBOSE=1<br/> INAM_DEBUG_TIME_VERBOSE=1<br/> INAM_DEBUG_MAIN_VERBOSE=2<br/> INAM_DEBUG_DB_VERBOSE=2<br/> INAM_DEBUG_NW_VERBOSE=1<br/> INAM_DEBUG_FB_VERBOSE=1<br/> <br/> The trial is being done on a single node where the needed packages and dependencies are installed. This test node(rt03d) is connected to a Mellanox switch and this switch is connected to three other systems.</span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size: 12.0pt;">The application used in the trial is IMB on the test node. The following commands were fired to run the application using srun with the MV2 flags that were suggested:</span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size: 12.0pt;"> </span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size: 12.0pt;"> export common="MV2_TWO_LEVEL_COMM_THRESHOLD=1,MV2_USE_RDMA_CM=0,\</span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size: 12.0pt;"> MV2_TOOL_REPORT_PVARS=1,MV2_DEBUG_TOOL_VERBOSE=1,\</span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size: 12.0pt;"> MV2_USE_SHMEM_COLL=1,MV2_ENABLE_PVAR_TIMER=1,\</span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size: 12.0pt;"> MV2_ENABLE_PVAR_COUNTER=1,MV2_ENABLE_PVAR_TIMER_BUCKETS=1,\</span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size: 12.0pt;"> MV2_ENABLE_PVAR_COUNTER_BUCKETS=1,MV2_TOOL_REPORT_SESSIONS=1,\</span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size: 12.0pt;"> MV2_TOOL_SESSIONS_DEFAULT_ALL_HANDLES=1,\</span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size: 12.0pt;"> MV2_TOOL_REPORT_LUSTRE_STATS=0,MV2_ON_DEMAND_THRESHOLD=1,\</span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size: 12.0pt;"> MV2_TOOL_INFO_FILE_PATH=/etc/osu-inam/osu-inam.conf"</span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size: 12.0pt;"> </span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size: 12.0pt;"> (time srun --mpi=pmi2 -n12 --export=$common -v \</span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size: 12.0pt;"> ./IMB-MPI1 -msglen 1Mfile_4K_inc -npmin 12 2>&1) 2>&1 | tee /tmp/log_IMB-MPI1_OSU_INAM</span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size: 12.0pt;"> </span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size: 12.0pt;">The osu-inam.conf file created is as follows:</span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size: 12.0pt;"> </span></p>
</div>
<div>
<p style="margin-bottom: 12.0pt;" class="MsoNormal"><span style="font-size: 12.0pt;"> MV2_TOOL_QPN=4719<br/> MV2_TOOL_LID=3<br/> MV2_TOOL_COUNTER_INTERVAL=3<br/> MV2_TOOL_REPORT_CPU_UTIL=1<br/> MV2_TOOL_REPORT_MEM_UTIL=1<br/> MV2_TOOL_REPORT_IO_UTIL=1<br/> MV2_TOOL_REPORT_COMM_GRID=1<br/> MV2_TOOL_REPORT_LUSTRE_STATS=0<br/> MV2_TOOL_REPORT_PVARS=1</span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size: 12.0pt;">Please find attached the log files related to osu-inam.conf, osu-inamd.conf, /var/log/messages and IMB(application) output.</span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size: 12.0pt;"> </span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size: 12.0pt;">Thanks for the continued support in this issue.</span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size: 12.0pt;"> </span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size: 12.0pt;">-John</span></p>
</div>
<div>
<p class="MsoNormal"> </p>
</div>
<div>
<p class="MsoNormal"><br/> On February 14, 2024 at 9:43 PM "Kousha, Pouya" <kousha.2@buckeyemail.osu.edu> wrote:</p>
</div>
<div>
<blockquote style="border: none; border-left: solid blue 1.0pt; padding: 0in 0in 0in 8.0pt; margin-left: 0in; margin-top: 5.0pt; margin-bottom: 5.0pt;">
<div>
<p class="MsoNormal">Hi John,</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">I hope this message finds you well. My name is Pouya Kousha, and I'm the lead developer for OSU INAM. Thank you for reaching out to us. To provide you with the most accurate support, we need to gather more detailed information about the issue you're encountering.</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">Could you please assist us by enabling additional debugging features? This can be done by setting specific variables in your osu-inamd.conf file, which is located in your INAM installation directory. Please update the file with the following settings:</p>
<p class="p1"><span class="s1"><span style="color: windowtext;">INAM_DEBUG_INIT_VERBOSE=1</span></span></p>
<p class="p1"><span class="s1"><span style="color: windowtext;">INAM_DEBUG_TIME_VERBOSE=1</span></span></p>
<p class="p1"><span class="s1"><span style="color: windowtext;">INAM_DEBUG_MAIN_VERBOSE=2</span></span></p>
<p class="p1"><span class="s1"><span style="color: windowtext;">INAM_DEBUG_DB_VERBOSE=2</span></span></p>
<p class="p1"><span class="s1"><span style="color: windowtext;">INAM_DEBUG_NW_VERBOSE=1</span></span></p>
<p class="p1"><span class="s1"><span style="color: windowtext;">INAM_DEBUG_FB_VERBOSE=1</span></span></p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">After updating these settings, kindly restart the INAM daemon to apply the changes.</p>
<p class="MsoNormal">Furthermore, we would appreciate it if you could share the log file segment from <span style="font-size: 10.0pt; font-family: 'Courier New';">/var/log/messages</span> that pertains to INAM, as well as the updated <span style="font-size: 10.0pt; font-family: 'Courier New';">osu-inam.conf</span> file. This information will greatly aid in our diagnostics.</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">From the srun add the followings as well. You can set <span style="font-size: 8.5pt; font-family: Menlo; color: red;"> MV2_DEBUG_TOOL_VERBOSE=1 </span>to check if from Mvapich2, we are sending information to INAM or not. It would be nice if you could send us the output (will be lengthy) as well. It should be good with 5-10 minutes of log for the application.</p>
<p class="MsoNormal"><span style="font-size: 8.5pt; font-family: Menlo; color: red;"> </span></p>
<p class="MsoNormal"><span style="font-size: 8.5pt; font-family: Menlo; color: red;"> </span></p>
<p class="MsoNormal">Additionally, to assess if there's an interaction issue with Mvapich2, please add the following environment variables to your <span style="font-size: 10.0pt; font-family: 'Courier New';">srun</span> command:</p>
<p class="MsoNormal"><span style="color: black;"> </span></p>
<p style="background: white;" class="MsoNormal"><span style="font-size: 8.5pt; font-family: Menlo; color: #fc4708;">export</span><span style="font-size: 8.5pt; font-family: Menlo; color: #fb0074;"> common=</span><span style="font-size: 8.5pt; font-family: Menlo; color: #fc4708;">"</span><span style="font-size: 8.5pt; font-family: Menlo; color: #0e71cd;">MV2_TWO_LEVEL_COMM_THRESHOLD=1 MV2_USE_RDMA_CM=0 MV2_TOOL_REPORT_PVARS=1 </span><span style="font-size: 8.5pt; font-family: Menlo; color: red;">MV2_DEBUG_TOOL_VERBOSE=1</span><span style="font-size: 8.5pt; font-family: Menlo; color: #0e71cd;"> MV2_USE_SHMEM_COLL=1 MV2_ENABLE_PVAR_TIMER=1 MV2_ENABLE_PVAR_COUNTER=1 MV2_ENABLE_PVAR_TIMER_BUCKETS=1 MV2_ENABLE_PVAR_COUNTER_BUCKETS=1 MV2_TOOL_REPORT_SESSIONS=1 MV2_TOOL_SESSIONS_DEFAULT_ALL_HANDLES=1 MV2_TOOL_REPORT_LUSTRE_STATS=0</span><span style="font-size: 8.5pt; font-family: Menlo; color: #fc4708;">"</span></p>
<p class="MsoNormal"><span style="font-size: 10.0pt; font-family: 'Courier New';"> </span></p>
<pre><span style="font-size: 9.0pt; font-family: Menlo; color: #0070c0;">srun … <span class="hljs-variable">$common</span> \</span></pre>
<pre><span style="font-size: 9.0pt; font-family: Menlo; color: #0070c0;"> MV2_DEBUG_TOOL_VERBOSE=1 \</span></pre>
<pre><span style="font-size: 9.0pt; font-family: Menlo; color: #0070c0;"> MV2_TWO_LEVEL_COMM_THRESHOLD=1 \</span></pre>
<pre><span style="font-size: 9.0pt; font-family: Menlo; color: #0070c0;"> MV2_ON_DEMAND_THRESHOLD=1 \</span></pre>
<pre><span style="font-size: 9.0pt; font-family: Menlo; color: #0070c0;"> MV2_USE_RDMA_CM=0 \</span></pre>
<pre><span style="font-size: 9.0pt; font-family: Menlo; color: #0070c0;"> MV2_TOOL_INFO_FILE_PATH=<span class="hljs-string">"<path to osu-inam.conf>"</span> \</span></pre>
<pre><span style="font-size: 9.0pt; font-family: Menlo; color: #0070c0;"> application</span></pre>
<p style="background: white;" class="MsoNormal"><span style="font-size: 8.5pt; font-family: Menlo; color: #fb0007;"> </span></p>
<p class="MsoNormal">Please ensure you replace <span style="font-size: 10.0pt; font-family: 'Courier New';"> <path to osu-inam.conf></span> with the actual path to your configuration file. The output from this operation might be extensive, but it's needed for our investigation. <span style="text-decoration: underline;">You can set MV2_DEBUG_TOOL_VERBOSE=0 for future runs</span>. This runtime variable is for debugging purposes only.</p>
<p style="background: white;" class="MsoNormal"><span style="font-size: 8.5pt; font-family: Menlo; color: black;"> </span></p>
<p class="MsoNormal">Once you have gathered all the requested information, please send it to us and we will evaluate and get back to you.</p>
<p class="MsoNormal">If you have any questions or encounter any difficulties along the way, do not hesitate to reach out.</p>
<p style="background: white;" class="MsoNormal"><span style="font-size: 8.5pt; font-family: Menlo; color: #fc7208;"> </span></p>
<p class="MsoNormal"> </p>
<div>
<div>
<p class="MsoNormal"><span style="color: black;">Best,</span></p>
</div>
</div>
<p class="MsoNormal"><span style="color: black;">Pouya Kousha </span></p>
<p class="MsoNormal"> </p>
<div style="border: none; border-top: solid #B5C4DF 1.0pt; padding: 3.0pt 0in 0in 0in;">
<p style="margin-bottom: 12.0pt;" class="MsoNormal"><strong><span style="font-size: 12.0pt; font-family: 'Calibri',sans-serif; color: black;">From: </span></strong><span style="font-size: 12.0pt; color: black;">Mvapich-discuss <mvapich-discuss-bounces@lists.osu.edu> on behalf of evancervj via Mvapich-discuss <mvapich-discuss@lists.osu.edu><br/> <strong><span style="font-family: 'Calibri',sans-serif;">Date: </span></strong>Tuesday, February 13, 2024 at 1:41 PM<br/> <strong><span style="font-family: 'Calibri',sans-serif;">To: </span></strong>mvapich-discuss@lists.osu.edu <mvapich-discuss@lists.osu.edu>, Lieber, Matt <lieber.31@osu.edu><br/> <strong><span style="font-family: 'Calibri',sans-serif;">Subject: </span></strong>Re: [Mvapich-discuss] Issue while Installation of Mvapich2-x-advanced for OSU-INAM</span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size: 1.0pt; color: white;">Hi Matt, I have installed Slurm package (slurm-23. 11. 3-1. el7. x86_64) on the system. Next I used srun for running the application(IMB) using the following command providing the MV2 options as mentioned in the section 'Running Example' </span></p>
</div>
<div>
<p class="MsoNormal"> </p>
</div>
<div>
<p class="MsoNormal"><span style="font-size: 12.0pt;">Hi Matt,</span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size: 12.0pt;"> </span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size: 12.0pt;">I have installed Slurm package (<em><span style="font-family: 'Calibri',sans-serif;">slurm-23.11.3-1.el7.x86_64)</span></em> on the system. Next I used srun for running the application(IMB) using the following command providing the MV2 options as mentioned in the section 'Running Example' of the userguide:</span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size: 12.0pt;"> </span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size: 12.0pt;"> <em><span style="font-family: 'Calibri',sans-serif;"> srun --mpi=pmi2 -n16 \</span></em></span></p>
</div>
<div>
<p class="MsoNormal"><em><span style="font-size: 12.0pt; font-family: 'Calibri',sans-serif;"> --export=MV2_ON_DEMAND_THRESHOLD=1,\</span></em></p>
</div>
<div>
<p class="MsoNormal"><em><span style="font-size: 12.0pt; font-family: 'Calibri',sans-serif;"> MV2_TOOL_INFO_FILE_PATH=/etc/osu-inam/osu-inam.conf,\ </span></em></p>
</div>
<div>
<p class="MsoNormal"><em><span style="font-size: 12.0pt; font-family: 'Calibri',sans-serif;"> MV2_TWO_LEVEL_COMM_THRESHOLD=1,MV2_USE_RDMA_CM=0,\</span></em></p>
</div>
<div>
<p class="MsoNormal"><em><span style="font-size: 12.0pt; font-family: 'Calibri',sans-serif;"> MV2_TOOL_REPORT_PVARS=1,MV2_ENABLE_PVAR_TIMER=1,\</span></em></p>
</div>
<div>
<p class="MsoNormal"><em><span style="font-size: 12.0pt; font-family: 'Calibri',sans-serif;"> MV2_TOOL_REPORT_PVARS=1,MV2_ENABLE_PVAR_TIMER=1,\</span></em></p>
</div>
<div>
<p class="MsoNormal"><em><span style="font-size: 12.0pt; font-family: 'Calibri',sans-serif;"> MV2_ENABLE_PVAR_COUNTER=1,MV2_ENABLE_PVAR_TIMER_BUCKETS=1,\</span></em></p>
</div>
<div>
<p class="MsoNormal"><em><span style="font-size: 12.0pt; font-family: 'Calibri',sans-serif;"> MV2_ENABLE_PVAR_COUNTER=1,MV2_ENABLE_PVAR_TIMER_BUCKETS=1,\</span></em></p>
</div>
<div>
<p class="MsoNormal"><em><span style="font-size: 12.0pt; font-family: 'Calibri',sans-serif;"> MV2_TOOL_SESSIONS_DEFAULT_ALL_HANDLES=1,MV2_TOOL_REPORT_LUSTRE_STATS=1 \</span></em></p>
</div>
<div>
<p class="MsoNormal"><em><span style="font-size: 12.0pt; font-family: 'Calibri',sans-serif;"> ./IMB-MPI1</span></em></p>
</div>
<div>
<p class="MsoNormal"><em><span style="font-size: 12.0pt;"><br/> </span></em><span style="font-size: 12.0pt;">In addition to the packet counter information earlier availbale, this time Job information like Job-ID is visible and is being updated on the OSU_INAM web interface. </span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size: 12.0pt;">However, MPI information is not exported. The fields that are not exported like cpu and memory usage for the Jobs show the message "no data from mpi process to this job". MPI related graphs like Global or inter-node communication graph also show the message "no data to display ". </span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size: 12.0pt;">Is there something I'm missing for process level and MPI-level information to be exported. How can I get MPI-level information to be seen on the OSU-INAM interface?</span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size: 12.0pt;"> </span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size: 12.0pt;">Thanks</span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size: 12.0pt;">John</span></p>
</div>
<div>
<p class="MsoNormal"><br/> On January 24, 2024 at 9:45 PM "Lieber, Matt" <lieber.31@osu.edu> wrote:</p>
</div>
<div>
<blockquote style="border: none; border-left: solid blue 1.0pt; padding: 0in 0in 0in 8.0pt; margin-left: 0in; margin-top: 5.0pt; margin-bottom: 5.0pt;">
<div>
<p class="MsoNormal"><span style="font-size: 12.0pt; font-family: 'Aptos',sans-serif; color: black;">Hi John, </span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size: 12.0pt; font-family: 'Aptos',sans-serif; color: black;">First you will need Slurm. Second the rpms that have slurm in their name for mvapich2x will use srun as their job launcher and not mpirun_rsh. Also, our user guide under section 5 has some other useful information that will be required for the information to show up </span><a href="https://mvapich.cse.ohio-state.edu/userguide/osu-inam/#_running_example"><span style="font-size: 12.0pt; font-family: 'Aptos',sans-serif;">https://mvapich.cse.ohio-state.edu/userguide/osu-inam/#_running_example</span></a><span style="font-size: 12.0pt; font-family: 'Aptos',sans-serif; color: black;"> . </span></p>
</div>
<div>
<p class="MsoNormal"> </p>
</div>
<div>
<p class="MsoNormal"><span style="font-size: 12.0pt; font-family: 'Aptos',sans-serif; color: black;">-Matt</span></p>
</div>
<div id="appendonsend">
<p class="MsoNormal"> </p>
</div>
<div style="text-align: center;" align="center" class="MsoNormal">
<hr width="100%" size="0"/>
</div>
<div id="divRplyFwdMsg">
<p class="MsoNormal"><strong><span style="font-family: 'Calibri',sans-serif; color: black;">From:</span></strong><span style="color: black;"> evancervj <evancervj@cdac.in><br/> <strong><span style="font-family: 'Calibri',sans-serif;">Sent:</span></strong> Wednesday, January 24, 2024 1:55 AM<br/> <strong><span style="font-family: 'Calibri',sans-serif;">To:</span></strong> mvapich-discuss@lists.osu.edu <mvapich-discuss@lists.osu.edu>; Lieber, Matt <lieber.31@osu.edu><br/> <strong><span style="font-family: 'Calibri',sans-serif;">Subject:</span></strong> Re: [Mvapich-discuss] Issue while Installation of Mvapich2-x-advanced for OSU-INAM</span></p>
<div>
<p class="MsoNormal"> </p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size: 1.0pt; color: white;">HI Matt, Thanks for the suggestions. I had tried MVAPICH2X-Basic as well as using the flag --nodeps for MVAPICH2-X-Advanced. The installation is successful. I was able to see the fabric related information like network topology and the </span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size: 1.0pt; color: white;"> </span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size: 12.0pt;">HI Matt,</span></p>
</div>
<div>
<p class="MsoNormal"> </p>
</div>
<div>
<p class="MsoNormal"><span style="font-size: 12.0pt;">Thanks for the suggestions. I had tried MVAPICH2X-Basic as well as using the flag --nodeps for MVAPICH2-X-Advanced. The installation is successful. I was able to see the fabric related information like network topology and the run-time packet counter information on the INAM web interface. </span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size: 12.0pt;"> </span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size: 12.0pt;">However process level and MPI related information was not seen on the INAM web interface. To export MPI related information to OSU_INAM, I tried running application using MVAPICH2X, but the mpirun_rsh file was missing in the MVAPICH2X installation directory. </span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size: 12.0pt;">Both the MVAPICH2X packages(basic and advanced) did not provide the files like mpirun_rsh, mpirun.., to run applications.</span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size: 12.0pt;"> </span></p>
</div>
<div>
<p class="MsoNormal"> </p>
</div>
<div>
<p class="MsoNormal"><em><span style="font-size: 12.0pt; font-family: 'Calibri',sans-serif;">rpm -Uvh --nodeps mvapich2-x-advanced-mofed5.0-gnu4.8.5-slurm-2.3-1.el7.x86_64.rpm</span></em> <br/> <em><span style="font-size: 12.0pt; font-family: 'Calibri',sans-serif;">Preparing... ################################# [100%]</span></em> <br/> <em><span style="font-size: 12.0pt; font-family: 'Calibri',sans-serif;">Updating / installing...</span></em> <br/> <em><span style="font-size: 12.0pt; font-family: 'Calibri',sans-serif;"> 1:mvapich2-x-advanced-mofed5.0-gnu4################################# [100%]</span></em></p>
</div>
<div>
<p class="MsoNormal"> </p>
</div>
<div>
<p class="MsoNormal"><em><span style="font-size: 12.0pt; font-family: 'Calibri',sans-serif;">ll /opt/mvapich2-x/gnu4.8.5/mofed5.0/advanced/slurm/bin/</span></em> <br/> <em><span style="font-size: 12.0pt; font-family: 'Calibri',sans-serif;">total 132</span></em> <br/> <em><span style="font-size: 12.0pt; font-family: 'Calibri',sans-serif;">lrwxrwxrwx 1 root root 6 Jan 23 10:27 mpic++ -> mpicxx</span></em> <br/> <em><span style="font-size: 12.0pt; font-family: 'Calibri',sans-serif;">-rwxr-xr-x 1 root root 10970 Jan 23 10:27 mpicc</span></em> <br/> <em><span style="font-size: 12.0pt; font-family: 'Calibri',sans-serif;">-rwxr-xr-x 1 root root 12856 Jun 2 2021 mpichversion</span></em> <br/> <em><span style="font-size: 12.0pt; font-family: 'Calibri',sans-serif;">-rwxr-xr-x 1 root root 10503 Jan 23 10:27 mpicxx</span></em> <br/> <em><span style="font-size: 12.0pt; font-family: 'Calibri',sans-serif;">-rwxr-xr-x 1 root root 14218 Jan 23 10:27 mpif77</span></em> <br/> <em><span style="font-size: 12.0pt; font-family: 'Calibri',sans-serif;">-rwxr-xr-x 1 root root 14218 Jan 23 10:27 mpif90</span></em> <br/> <em><span style="font-size: 12.0pt; font-family: 'Calibri',sans-serif;">-rwxr-xr-x 1 root root 14278 Jun 2 2021 mpifort</span></em> <br/> <em><span style="font-size: 12.0pt; font-family: 'Calibri',sans-serif;">-rwxr-xr-x 1 root root 12928 Jun 2 2021 mpiname</span></em> <br/> <em><span style="font-size: 12.0pt; font-family: 'Calibri',sans-serif;">-rwxr-xr-x 1 root root 23256 Jun 2 2021 mpivars</span></em> <br/> <em><span style="font-size: 12.0pt; font-family: 'Calibri',sans-serif;">-rwxr-xr-x 1 root root 3430 Jun 2 2021</span></em><span style="font-size: 12.0pt;"> parkill</span></p>
</div>
<div>
<p class="MsoNormal"> </p>
</div>
<div>
<p class="MsoNormal"><span style="font-size: 12.0pt;"> </span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size: 12.0pt;"> </span></p>
</div>
<div>
<p style="margin-bottom: 12.0pt;" class="MsoNormal"><em><span style="font-size: 12.0pt; font-family: 'Calibri',sans-serif;">rpm -Uvh mvapich2-x-basic-mofed5.0-gnu4.8.5-slurm-2.3-1.el7.x86_64.rpm </span></em><span style="font-size: 12.0pt;"><br/> <em><span style="font-family: 'Calibri',sans-serif;">Preparing... ################################# [100%]</span></em><br/> <em><span style="font-family: 'Calibri',sans-serif;">Updating / installing...</span></em><br/> <em><span style="font-family: 'Calibri',sans-serif;"> 1:mvapich2-x-basic-mofed5.0-gnu4.8.################################# [100%]</span></em><br/> <br/> <em><span style="font-family: 'Calibri',sans-serif;">ll /opt/mvapich2-x/gnu4.8.5/mofed5.0/basic/slurm/bin/</span></em><br/> <em><span style="font-family: 'Calibri',sans-serif;">total 132</span></em><br/> <em><span style="font-family: 'Calibri',sans-serif;">lrwxrwxrwx 1 root root 6 Jan 24 11:54 mpic++ -> mpicxx</span></em><br/> <em><span style="font-family: 'Calibri',sans-serif;">-rwxr-xr-x 1 root root 10756 Jan 24 11:54 mpicc</span></em><br/> <em><span style="font-family: 'Calibri',sans-serif;">-rwxr-xr-x 1 root root 12856 May 20 2021 mpichversion</span></em><br/> <em><span style="font-family: 'Calibri',sans-serif;">-rwxr-xr-x 1 root root 10324 Jan 24 11:54 mpicxx</span></em><br/> <em><span style="font-family: 'Calibri',sans-serif;">-rwxr-xr-x 1 root root 14036 Jan 24 11:54 mpif77</span></em><br/> <em><span style="font-family: 'Calibri',sans-serif;">-rwxr-xr-x 1 root root 14036 Jan 24 11:54 mpif90</span></em><br/> <em><span style="font-family: 'Calibri',sans-serif;">-rwxr-xr-x 1 root root 14096 May 20 2021 mpifort</span></em><br/> <em><span style="font-family: 'Calibri',sans-serif;">-rwxr-xr-x 1 root root 12928 May 20 2021 mpiname</span></em><br/> <em><span style="font-family: 'Calibri',sans-serif;">-rwxr-xr-x 1 root root 23256 May 20 2021 mpivars</span></em><br/> <em><span style="font-family: 'Calibri',sans-serif;">-rwxr-xr-x 1 root root 3430 May 20 2021 parkill</span></em></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size: 12.0pt;"> </span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size: 12.0pt;">I was able to access the fabric information on the OSU-INAM interface. I wanted to leverage the OSU-INAM features to observe the process level and MPI level information as well, but failed because of the above mentioned issue. Any pointers on this will be helpful. </span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size: 12.0pt;"> </span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size: 12.0pt;">Also my current environment do not have Slurm or any other job scheduler installed. So Is Slurm necessary for OSU-INAM to export MPI level information? </span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size: 12.0pt;"> </span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size: 12.0pt;">Thanks</span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size: 12.0pt;">John</span></p>
</div>
<div>
<p class="MsoNormal"> </p>
</div>
<div>
<p class="MsoNormal"><br/> On January 20, 2024 at 6:36 AM "Lieber, Matt" <lieber.31@osu.edu> wrote:</p>
</div>
<div>
<blockquote style="border: none; border-left: solid blue 1.0pt; padding: 0in 0in 0in 8.0pt; margin-left: 0in; margin-top: 5.0pt; margin-bottom: 5.0pt;">
<div>
<p class="MsoNormal"><span style="font-size: 12.0pt; font-family: 'Aptos',sans-serif; color: black;">Hi John, </span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size: 12.0pt; font-family: 'Aptos',sans-serif; color: black;">Sorry for the delay in getting back to you. There are multiple options that could fix this. If you do not plan to use the sharp functionality adding the flag --nodeps should fix the issue you are seeing. Also MVAPICH2-X basic should also work with INAM. If you do wish to use sharp we will have to send a new rpm. Please let let us know how these options work for you.</span></p>
</div>
<div>
<p class="MsoNormal"> </p>
</div>
<div>
<p class="MsoNormal"><span style="font-size: 12.0pt; font-family: 'Aptos',sans-serif; color: black;">Thanks,</span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size: 12.0pt; font-family: 'Aptos',sans-serif; color: black;">Matt</span></p>
</div>
<div id="x_appendonsend">
<p class="MsoNormal"> </p>
</div>
<div style="text-align: center;" align="center" class="MsoNormal">
<hr width="100%" size="0"/>
</div>
<div id="x_divRplyFwdMsg">
<p class="MsoNormal"><strong><span style="font-family: 'Calibri',sans-serif; color: black;">From:</span></strong><span style="color: black;"> Mvapich-discuss <mvapich-discuss-bounces@lists.osu.edu> on behalf of V John via Mvapich-discuss <mvapich-discuss@lists.osu.edu><br/> <strong><span style="font-family: 'Calibri',sans-serif;">Sent:</span></strong> Tuesday, January 16, 2024 7:41 AM<br/> <strong><span style="font-family: 'Calibri',sans-serif;">To:</span></strong> mvapich-discuss@lists.osu.edu <mvapich-discuss@lists.osu.edu><br/> <strong><span style="font-family: 'Calibri',sans-serif;">Subject:</span></strong> [Mvapich-discuss] Issue while Installation of Mvapich2-x-advanced for OSU-INAM</span></p>
<div>
<p class="MsoNormal"> </p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size: 1.0pt; color: white;">Hi everyone, I'm trying to use OSU-INAM on our cluster with Mellanox-IB ConnectX-6 interconnect with machines having Centos7. 6 OS. I downloaded the osu-inam(osu-inam-mysql-1. 0-1. el7. x86_64. rpm) package matching the environment which also required </span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size: 1.0pt; color: white;"> </span></p>
</div>
<div>
<p class="MsoNormal">Hi everyone,</p>
</div>
<div>
<p class="MsoNormal"> </p>
</div>
<div>
<p class="MsoNormal">I'm trying to use OSU-INAM on our cluster with Mellanox-IB ConnectX-6 interconnect with machines having Centos7.6 OS. I downloaded the osu-inam(osu-inam-mysql-1.0-1.el7.x86_64.rpm) package matching the environment which also required MOFED 5+. So I installed MLNX_OFED_LINUX-5.0-1.0.0.0 package on the systems.</p>
</div>
<div>
<p class="MsoNormal"> </p>
</div>
<div>
<p class="MsoNormal"> </p>
</div>
<div>
<p class="MsoNormal">Osu-Inam user-guide mentioned the requirement of Mvapich2-x-advanced for getting MPI usage related information. Hence, I tried to install MVAPICH2-X-advanced. The only binary matching the environment available on the Mvapich website is mvapich2-x-advanced-mofed5.0-gnu4.8.5-slurm-2.3-1.el7.x86_64.rpm. The installation fails however due to dependency on libsharp_coll.so.4.</p>
</div>
<div>
<p class="MsoNormal"> </p>
</div>
<div>
<p class="MsoNormal"><em><span style="font-family: 'Calibri',sans-serif;">rpm -Uvh mvapich2-x-advanced-mofed5.0-gnu4.8.5-slurm-2.3-1.el7.x86_64.rpm </span></em><br/> <em><span style="font-family: 'Calibri',sans-serif;">error: Failed dependencies:</span></em> <br/> <em><span style="font-family: 'Calibri',sans-serif;">libsharp_coll.so.4()(64bit) is needed by mvapich2-x-advanced-mofed5.0-gnu4.8.5-slurm-2.3-1.el7.x86_64</span></em></p>
</div>
<div>
<p class="MsoNormal"> </p>
</div>
<div>
<p class="MsoNormal"> </p>
</div>
<div>
<p class="MsoNormal">The libsharp_coll library versions under the /opt/mellanox/sharp/lib directory created post the installation of MLNX_OFED_LINUX-5.0-1.0.0.0 misses the required version, libsharp_coll.so.4.</p>
</div>
<div>
<p class="MsoNormal"> </p>
</div>
<div>
<p class="MsoNormal"><em><span style="font-family: 'Calibri',sans-serif;">ls /opt/mellanox/sharp/lib | grep libsharp_coll.so</span></em> <br/> <em><span style="font-family: 'Calibri',sans-serif;">libsharp_coll.so</span></em> <br/> <em><span style="font-family: 'Calibri',sans-serif;">libsharp_coll.so.5</span></em> <br/> <em><span style="font-family: 'Calibri',sans-serif;">libsharp_coll.so.5.0.1</span></em></p>
</div>
<div>
<p class="MsoNormal"> </p>
</div>
<div>
<p class="MsoNormal"> </p>
</div>
<div>
<p class="MsoNormal">Is there a fix to this issue..?</p>
</div>
<div>
<p class="MsoNormal"> </p>
</div>
<div>
<p class="MsoNormal">Thanks</p>
</div>
<div>
<p class="MsoNormal">John</p>
</div>
<div>
<p class="MsoNormal">HPC Technologies Group</p>
</div>
<div>
<p class="MsoNormal">C-DAC pune</p>
</div>
<p class="MsoNormal"><br/> ------------------------------------------------------------------------------------------------------------ <br/> [ C-DAC is on Social-Media too. Kindly follow us at: <br/> Facebook: <a href="https://urldefense.com/v3/__https:/www.facebook.com/CDACINDIA__;!!KGKeukY!2KJphDWGFEcrgBZNFD0pyWfL_tLCH64l43moQyPTHdZMlRXL2sAhibE3ALe1Uzxh39TVZU1r-xrOX_sKpGycM_VAyzT0TQ$"> https://www.facebook.com/CDACINDIA</a> & Twitter: @cdacindia ] <br/> <br/> This e-mail is for the sole use of the intended recipient(s) and may <br/> contain confidential and privileged information. If you are not the <br/> intended recipient, please contact the sender by reply e-mail and destroy <br/> all copies and the original message. Any unauthorized review, use, <br/> disclosure, dissemination, forwarding, printing or copying of this email <br/> is strictly prohibited and appropriate legal action will be taken. <br/> ------------------------------------------------------------------------------------------------------------</p>
</div>
</blockquote>
<p class="MsoNormal"><br/> </p>
</div>
<p class="MsoNormal"><br/> ------------------------------------------------------------------------------------------------------------ <br/> [ C-DAC is on Social-Media too. Kindly follow us at: <br/> Facebook: <a href="https://urldefense.com/v3/__https:/www.facebook.com/CDACINDIA__;!!KGKeukY!xP_sp9xIt5MlF-mGh9nukFmdaGDl-o3tQ8iYqzLQpVIRRub9-TeUbJ4uW-GqCJC54Y1hZ38BrLaABtnLmSWZ$"> https://www.facebook.com/CDACINDIA</a> & Twitter: @cdacindia ] <br/> <br/> This e-mail is for the sole use of the intended recipient(s) and may <br/> contain confidential and privileged information. If you are not the <br/> intended recipient, please contact the sender by reply e-mail and destroy <br/> all copies and the original message. Any unauthorized review, use, <br/> disclosure, dissemination, forwarding, printing or copying of this email <br/> is strictly prohibited and appropriate legal action will be taken. <br/> ------------------------------------------------------------------------------------------------------------</p>
</div>
</blockquote>
<p class="MsoNormal"><br/> </p>
</div>
<p class="MsoNormal"><br/> ------------------------------------------------------------------------------------------------------------ <br/> [ C-DAC is on Social-Media too. Kindly follow us at: <br/> Facebook: <a href="https://urldefense.com/v3/__https:/www.facebook.com/CDACINDIA__;!!KGKeukY!xDnLo72OA_j_Ed4dReyJgNnnLbKCQdAh6avvk7JI6XHItqFiW5Z46E9g9a0A4WphLH9i1qstvTDycmA65H5Qj0GtzVFwXQ$"> https://www.facebook.com/CDACINDIA</a> & Twitter: @cdacindia ] <br/> <br/> This e-mail is for the sole use of the intended recipient(s) and may <br/> contain confidential and privileged information. If you are not the <br/> intended recipient, please contact the sender by reply e-mail and destroy <br/> all copies and the original message. Any unauthorized review, use, <br/> disclosure, dissemination, forwarding, printing or copying of this email <br/> is strictly prohibited and appropriate legal action will be taken. <br/> ------------------------------------------------------------------------------------------------------------</p>
</div>
</blockquote>
<p class="MsoNormal"><br/> </p>
</div>
</blockquote>
<p class="MsoNormal"><br/> </p>
</div>
<p class="MsoNormal"><br/> ------------------------------------------------------------------------------------------------------------ <br/> [ C-DAC is on Social-Media too. Kindly follow us at: <br/> Facebook: <a href="https://urldefense.com/v3/__https:/www.facebook.com/CDACINDIA__;!!KGKeukY!0HCbYervR8slKdz0w6qcJcDj9LfEhykuN9AHiSOxKc5Clsi7OWmIEOPAIu1QXtEhJIfrYGHXV0EYc91Fdy8ljeZ6ND1MGA$"> https://www.facebook.com/CDACINDIA</a> & Twitter: @cdacindia ] <br/> <br/> This e-mail is for the sole use of the intended recipient(s) and may <br/> contain confidential and privileged information. If you are not the <br/> intended recipient, please contact the sender by reply e-mail and destroy <br/> all copies and the original message. Any unauthorized review, use, <br/> disclosure, dissemination, forwarding, printing or copying of this email <br/> is strictly prohibited and appropriate legal action will be taken. <br/> ------------------------------------------------------------------------------------------------------------</p>
</div>
</blockquote>
<br/>
</div>
<br />------------------------------------------------------------------------------------------------------------
<br />[ C-DAC is on Social-Media too. Kindly follow us at:
<br />Facebook: <a href="https://urldefense.com/v3/__https://www.facebook.com/CDACINDIA__;!!KGKeukY!3y68wPgmnnjeXgQC8FuAGn2AY7xYMAFGiTuTj1cQWQfhYKGUTRi5IXEuQyekk_N5jHQRoRp4yXN4xR7rO5pgKphltbSC0A$">https://www.facebook.com/CDACINDIA</a> & Twitter: @cdacindia ]
<br />
<br />This e-mail is for the sole use of the intended recipient(s) and may
<br />contain confidential and privileged information. If you are not the
<br />intended recipient, please contact the sender by reply e-mail and destroy
<br />all copies and the original message. Any unauthorized review, use,
<br />disclosure, dissemination, forwarding, printing or copying of this email
<br />is strictly prohibited and appropriate legal action will be taken.
<br />------------------------------------------------------------------------------------------------------------
</body></html>