<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body>
<div>
<div dir="ltr">Hi Matt,</div>
<div dir="ltr"><br>
</div>
<div dir="ltr">Thanks for your note. Good to know that your model is working with MVAPICH 4.0. Sorry to know that you are getting variable performance here. </div>
<div dir="ltr"><br>
</div>
<div dir="ltr">I am sending a follow-up note to you. It will be good to have a short discussion with you and some of the MVAPICH team members to understand your environment and issues in detail and investigate these further. </div>
<div dir="ltr"><br>
</div>
<div dir="ltr">Thanks,</div>
<div dir="ltr"><br>
</div>
<div dir="ltr">DK</div>
</div>
<div id="ms-outlook-mobile-body-separator-line" dir="auto"><br>
</div>
<div id="ms-outlook-mobile-signature">Get <a href="https://aka.ms/o0ukef">Outlook for iOS</a></div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> Mvapich-discuss <mvapich-discuss-bounces@lists.osu.edu> on behalf of Thompson, Matt (GSFC-610.1)[SCIENCE SYSTEMS AND APPLICATIONS INC] via
Mvapich-discuss <mvapich-discuss@lists.osu.edu><br>
<b>Sent:</b> Tuesday, March 25, 2025 2:52:22 PM<br>
<b>To:</b> mvapich-discuss@lists.osu.edu <mvapich-discuss@lists.osu.edu><br>
<b>Subject:</b> [Mvapich-discuss] Transitioning to MVAPICH 4: Environment Variables</font>
<div> </div>
</div>
<div>
<div style="display:none!important; display:none; visibility:hidden; font-size:1px; color:#ffffff; line-height:1px; height:0px; max-height:0px; opacity:0; overflow:hidden">
All, First, congrats to the MVAPICH Team for v4. 0! Now, in the past MVAPICH2 + our model never seemed to work on a cluster (well, 10 years ago it did, but then it. . . stopped working). MVAPICH 3 also had issues, but MVAPICH 4. 0 seems to be working. . . albeit</div>
<div style="display:none!important; display:none; visibility:hidden; font-size:1px; color:#ffffff; line-height:1px; max-height:0px; opacity:0; overflow:hidden">
ZjQcmQRYFpfptBannerStart</div>
<div dir="ltr" lang="en" id="x_pfptBanneryp3rnor" style="display:block!important; text-align:left!important; margin:16px 0px 16px 0px!important; padding:8px 16px 8px 16px!important; border-radius:4px!important; min-width:200px!important; background-color:#CFD3D7!important; background-color:#CFD3D7; border-top:4px solid #8c8e91!important; border-top:4px solid #8c8e91">
<div id="x_pfptBanneryp3rnor" style="float:left!important; display:block!important; margin:0px 0px 1px 0px!important; max-width:600px!important">
<div id="x_pfptBanneryp3rnor" style="display:block!important; visibility:visible!important; background-color:#CFD3D7!important; color:#000000!important; color:#000000; font-family:'Arial',sans-serif!important; font-family:'Arial',sans-serif; font-weight:bold!important; font-weight:bold; font-size:14px!important; line-height:18px!important; line-height:18px">
This Message Is From an External Sender </div>
<div id="x_pfptBanneryp3rnor" style="display:block!important; visibility:visible!important; background-color:#CFD3D7!important; color:#000000!important; color:#000000; font-weight:normal; font-family:'Arial',sans-serif!important; font-family:'Arial',sans-serif; font-size:12px!important; line-height:18px!important; line-height:18px; margin-top:2px!important">
This message came from outside your organization. </div>
</div>
<div id="x_pfptBanneryp3rnor" style="float:right!important; display:block!important; display:block; margin:0px 0px 0px 16px!important; text-align:right!important; width:fit-content!important">
<a id="x_pfptBanneryp3rnor" href="https://us-phishalarm-ewt.proofpoint.com/EWT/v1/KGKeukY!vwQdMidND6YBRRdx3-4leYPCCGtCZUVIgQ8Tbd9q0E8oX2cw6znhgW_PFqU9whzUZy_dJ3khpDk1tmXs3S2Vqdhd_bvcFEXBXCAEAUUvfYei1z4Q8W1u5UUzhuEod-A_2UygNcCaXO7vAr8sdKpQ0w$" style="display:inline-block!important; text-decoration:none">
<div class="x_pfptPrimaryButtonyp3rnor" style="display:inline-block!important; display:inline-block; visibility:visible!important; opacity:1!important; color:#000000!important; color:#000000; font-family:'Arial',sans-serif!important; font-family:'Arial',sans-serif; font-size:14px!important; font-weight:normal!important; text-decoration:none!important; border-radius:2px!important; padding:7.5px 16px!important; margin:3px 0 3px 16px!important; white-space:nowrap!important; width:fit-content!important; border:1px solid #666666">
Report Suspicious </div>
</a></div>
<div style="clear:both!important; display:block!important; visibility:hidden!important; line-height:0!important; font-size:0.01px!important; height:0px">
</div>
</div>
<div style="display:none!important; display:none; visibility:hidden; font-size:1px; color:#ffffff; line-height:1px; max-height:0px; opacity:0; overflow:hidden">
ZjQcmQRYFpfptBannerEnd</div>
<style>
<!--
#x_pfptBanneryp3rnor
{display:block!important;
visibility:visible!important;
opacity:1!important;
background-color:#CFD3D7!important;
max-width:none!important;
max-height:none!important}
-->
</style>
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style>
<!--
@font-face
{font-family:Helvetica}
@font-face
{font-family:"Cambria Math"}
@font-face
{font-family:Calibri}
@font-face
{font-family:"Calibri Light"}
@font-face
{font-family:Aptos}
@font-face
{font-family:Consolas}
@font-face
{}
p.x_MsoNormal, li.x_MsoNormal, div.x_MsoNormal
{margin:0in;
font-size:12.0pt;
font-family:"Aptos",sans-serif}
a:link, span.x_MsoHyperlink
{color:#467886;
text-decoration:underline}
span.x_EmailStyle17
{font-family:Consolas;
color:windowtext;
font-weight:normal;
font-style:normal}
.x_MsoChpDefault
{}
@page WordSection1
{margin:1.0in 1.0in 1.0in 1.0in}
div.x_WordSection1
{}
-->
</style>
<div class="x_WordSection1">
<p class="x_MsoNormal"><span style="font-family:Consolas">All,</span></p>
<p class="x_MsoNormal"><span style="font-family:Consolas"><br>
First, congrats to the MVAPICH Team for v4.0!</span></p>
<p class="x_MsoNormal"><span style="font-family:Consolas"> </span></p>
<p class="x_MsoNormal"><span style="font-family:Consolas">Now, in the past MVAPICH2 + our model never seemed to work on a cluster (well, 10 years ago it did, but then it...stopped working). MVAPICH 3 also had issues, but MVAPICH 4.0 seems to be working...albeit
a bit more "wobbly" than Open MPI and Intel MPI.[1]</span></p>
<p class="x_MsoNormal"><span style="font-family:Consolas"> </span></p>
<p class="x_MsoNormal"><span style="font-family:Consolas">In the past, we found these:</span></p>
<p class="x_MsoNormal"><span style="font-family:Consolas"> </span></p>
<p class="x_MsoNormal"><span style="font-family:Consolas"> MV2_ENABLE_AFFINITY=0</span></p>
<p class="x_MsoNormal"><span style="font-family:Consolas"> MV2_MPIRUN_TIMEOUT=100</span></p>
<p class="x_MsoNormal"><span style="font-family:Consolas"> MV2_GATHERV_SSEND_THRESHOLD=256</span></p>
<p class="x_MsoNormal"><span style="font-family:Consolas"> MV2_HOMOGENEOUS_CLUSTER=1</span></p>
<p class="x_MsoNormal"><span style="font-family:Consolas"> MV2_ON_DEMAND_THRESHOLD=1</span></p>
<p class="x_MsoNormal"><span style="font-family:Consolas"> </span></p>
<p class="x_MsoNormal"><span style="font-family:Consolas">were useful, but I also see from
<a href="https://urldefense.com/v3/__https://mvapich-docs.readthedocs.io/en/mvapich-open/cvar.html__;!!KGKeukY!xaP_6bqO7gmKV9aKYNbYE5fguOeRX9CtljhsK8bPEiYXDkrZHUVWK3kxy_GlOk6jmPLnXE6J45FBTT_vl_tuzDAQ9CHKUqjp3fAsdHs$">
https://mvapich-docs.readthedocs.io/en/mvapich-open/cvar.html</a> that the MVAPICH environment variables might have changed.</span></p>
<p class="x_MsoNormal"><span style="font-family:Consolas"> </span></p>
<p class="x_MsoNormal"><span style="font-family:Consolas">So, is there a page/pages like
<a href="https://mvapich.cse.ohio-state.edu/performance/job-startup/">https://mvapich.cse.ohio-state.edu/performance/job-startup/</a> that might have useful tips? (Though I don't think our startup was all that slow, but similar.) Or a page translating old flags
into new?</span></p>
<p class="x_MsoNormal"><span style="font-family:Consolas"> </span></p>
<p class="x_MsoNormal"><span style="font-family:Consolas">Thanks,<br>
Matt</span></p>
<p class="x_MsoNormal"><span style="font-family:Consolas"> </span></p>
<p class="x_MsoNormal"><span style="font-family:Consolas">[1]: "Wobbly" in that at "high-ish" core count (3456 processes on 28 nodes) Intel MPI seems to be around 110 d/d (model days per wall clock day) per step, Open MPI around 120 d/d per step (until it seems
to have crashed weirdly). But MVAPICH 4 is varying from 50 to 110 d/d. No rhyme or reason, just different throughput randomly at each time step.</span></p>
<p class="x_MsoNormal"><span style="font-family:Consolas"> </span></p>
<div>
<div style="border:none; border-bottom:solid windowtext 1.0pt; padding:0in 0in 1.0pt 0in">
<p class="x_MsoNormal"><span style="font-size:11.0pt; font-family:Consolas"> </span></p>
</div>
<p class="x_MsoNormal"><span style="font-size:11.0pt; font-family:Consolas"> </span></p>
<table class="x_MsoNormalTable" border="0" cellspacing="0" cellpadding="0" width="540" style="width:405.0pt; border-collapse:collapse">
<tbody>
<tr style="height:91.3pt">
<td width="151" style="width:113.15pt; padding:2.15pt 1.45pt 2.15pt 1.45pt; height:91.3pt">
<p class="x_MsoNormal" align="center" style="text-align:center"><a href="https://urldefense.com/v3/__http://www.ssaihq.com/__;!!KGKeukY!xaP_6bqO7gmKV9aKYNbYE5fguOeRX9CtljhsK8bPEiYXDkrZHUVWK3kxy_GlOk6jmPLnXE6J45FBTT_vl_tuzDAQ9CHKUqjpqLHNs90$"><span style="font-size:11.0pt; font-family:"Calibri",sans-serif; color:#212121; text-decoration:none"><img border="0" width="131" height="131" id="x_Picture_x0020_2" src="cid:image001.png@01DB9D8C.E3777280" alt="signature_1230636052" style="width:1.3645in; height:1.3645in"></span></a><span style="font-size:11.0pt; font-family:"Calibri",sans-serif; color:#212121"></span></p>
</td>
<td width="26" valign="bottom" style="width:19.55pt; padding:2.15pt 1.45pt 2.15pt 1.45pt; height:91.3pt">
<p class="x_MsoNormal" align="center" style="text-align:center; line-height:12.65pt">
<span style="font-size:11.0pt; font-family:"Calibri Light",sans-serif; color:#212121"><img border="0" width="2" height="145" id="x_Picture_x0020_1" src="cid:image002.png@01DB9D8C.E3777280" alt="Image" style="width:.0208in; height:1.5104in"></span><span style="font-size:11.0pt; font-family:"Calibri",sans-serif; color:#212121"></span></p>
</td>
<td width="363" style="width:272.3pt; padding:2.15pt 1.45pt 2.15pt 1.45pt; height:91.3pt">
<p class="x_MsoNormal" style="line-height:12.65pt"><b><span style="font-size:11.0pt; font-family:Helvetica; color:black">Matt Thompson</span></b><span style="font-size:11.0pt; font-family:Helvetica; color:#212121"></span></p>
<p class="x_MsoNormal" style="line-height:12.65pt"><b><span style="font-size:9.0pt; font-family:Helvetica; color:black">Lead Scientific Software Engineer/Supervisor</span></b><span style="font-size:11.0pt; font-family:Helvetica; color:#212121"></span></p>
<p class="x_MsoNormal" style="line-height:12.65pt"><span style="font-size:9.0pt; font-family:Helvetica; color:#212121">Global Modeling and Assimilation Office</span></p>
<p class="x_MsoNormal" style="line-height:12.65pt"><span style="font-size:9.0pt; font-family:Helvetica; color:#212121">Science Systems and Applications, Inc.
</span></p>
<p class="x_MsoNormal" style="line-height:12.65pt"><span style="font-size:9.0pt; font-family:Helvetica; color:#212121">Code 610.1, 8800 Greenbelt Rd, Greenbelt, MD 20771</span></p>
<p class="x_MsoNormal" style="line-height:12.65pt"><span style="font-size:9.0pt; font-family:Helvetica; color:black">o: 301-614-6712</span><span style="font-size:11.0pt; font-family:Helvetica; color:#212121">
</span></p>
<p class="x_MsoNormal"><span style="font-size:9.0pt; font-family:Helvetica; color:#212121"><a href="mailto:matthew.thompson@nasa.gov"><span style="color:#467886">matthew.thompson@nasa.gov</span></a></span><span style="font-size:11.0pt; font-family:Helvetica; color:#212121"></span></p>
</td>
</tr>
</tbody>
</table>
</div>
<p class="x_MsoNormal"> </p>
</div>
</div>
</body>
</html>