<head><!-- BaNnErBlUrFlE-HeAdEr-start -->
<style>
#pfptBannerfwm57j6 { all: revert !important; display: block !important;
visibility: visible !important; opacity: 1 !important;
background-color: #CFD3D7 !important;
max-width: none !important; max-height: none !important }
.pfptPrimaryButtonfwm57j6:hover, .pfptPrimaryButtonfwm57j6:focus {
background-color: #adb0b4 !important; }
.pfptPrimaryButtonfwm57j6:active {
background-color: #8c8e91 !important; }
html:root, html:root>body { all: revert !important; display: block !important;
visibility: visible !important; opacity: 1 !important; }
</style>
<!-- BaNnErBlUrFlE-HeAdEr-end -->
</head><!-- BaNnErBlUrFlE-BoDy-start -->
<!-- Preheader Text : BEGIN -->
<div style="display:none !important;display:none;visibility:hidden;mso-hide:all;font-size:1px;color:#ffffff;line-height:1px;height:0px;max-height:0px;opacity:0;overflow:hidden;">
Hi, Recently I've compared mvapich (based on mpich 4. 3. 0 as I recall) and mpich 4. 3. 1 on single node Intel Xeon 6972p (it has mellanox fabric but since it's single node it's not relevant). The application is quite tricky but similar</div>
<!-- Preheader Text : END -->
<!-- Email Banner : BEGIN -->
<div style="display:none !important;display:none;visibility:hidden;mso-hide:all;font-size:1px;color:#ffffff;line-height:1px;max-height:0px;opacity:0;overflow:hidden;">ZjQcmQRYFpfptBannerStart</div>
<!--[if ((ie)|(mso))]>
<table border="0" cellspacing="0" cellpadding="0" width="100%" style="padding: 16px 0px 16px 0px; direction: ltr" lang="en"><tr><td>
<table border="0" cellspacing="0" cellpadding="0" style="padding: 0px 10px 5px 6px; width: 100%; border-radius:4px; border-top:4px solid #8c8e91;background-color:#CFD3D7;"><tr><td valign="top">
<table align="left" border="0" cellspacing="0" cellpadding="0" style="padding: 4px 8px 4px 8px">
<tr><td style="color:#000000; font-family: 'Arial', sans-serif; font-weight:bold; font-size:14px; direction: ltr">
This Message Is From an External Sender
</td></tr>
<tr><td style="color:#000000; font-weight:normal; font-family: 'Arial', sans-serif; font-size:12px; direction: ltr">
This message came from outside your organization.
</td></tr>
</table>
<![if ie]><br clear="all"><![endif]>
<table align="right" border="0" cellspacing="0" cellpadding="0" style="padding: 4px 0px 4px 0px"><tr>
<td style="direction: ltr"> <a target="_blank" href="https://us-phishalarm-ewt.proofpoint.com/EWT/v1/KGKeukY!vYQd06pJjE9AAhTaFS-M1rCxuGsuMAuBgcubeYoCIzfnLzi-eNInMvY0TONXlt2qBcPsx1kQkSd44s2C_U_rvIjthLybFDLcVaaOkMcJ_yXZ8RVfFiAN0NqO-okPEYu4Zsg4pA$" style="mso-padding-alt: 7.5px; padding: 7.5px; border-radius: 2px; border: 1.5px solid #666666; "><strong style="font-weight: normal; color: #000000; text-decoration: none; font-family: 'Arial', sans-serif; font-size:14px; line-height: 40px; "> Report Suspicious </strong></a> </td>
</tr></table>
</td></tr></table>
</td></tr></table>
<![endif]-->
<![if !((ie)|(mso))]>
<div dir="ltr" lang="en" id="pfptBannerfwm57j6" style="all: revert !important; display:block !important; text-align: left !important; margin:16px 0px 16px 0px !important; padding:8px 16px 8px 16px !important; border-radius: 4px !important; min-width: 200px !important; background-color: #CFD3D7 !important; background-color: #CFD3D7; border-top: 4px solid #8c8e91 !important; border-top: 4px solid #8c8e91;">
<div id="pfptBannerfwm57j6" style="all: unset !important; float:left !important; display:block !important; margin: 0px 0px 1px 0px !important; max-width: 600px !important;">
<div id="pfptBannerfwm57j6" style="all: unset !important; display:block !important; visibility: visible !important; background-color: #CFD3D7 !important; color:#000000 !important; color:#000000; font-family: 'Arial', sans-serif !important; font-family: 'Arial', sans-serif; font-weight:bold !important; font-weight:bold; font-size:14px !important; line-height:18px !important; line-height:18px">
This Message Is From an External Sender
</div>
<div id="pfptBannerfwm57j6" style="all: unset !important; display:block !important; visibility: visible !important; background-color: #CFD3D7 !important; color:#000000 !important; color:#000000; font-weight:normal; font-family: 'Arial', sans-serif !important; font-family: 'Arial', sans-serif; font-size:12px !important; line-height:18px !important; line-height:18px; margin-top:2px !important;">
This message came from outside your organization.
</div>
</div>
<div id="pfptBannerfwm57j6" style="all: unset !important; float: right !important; display: block !important; display: block; margin: 0px 0px 0px 16px !important; text-align: right !important; width: fit-content !important;">
<a id="pfptBannerfwm57j6" href="https://us-phishalarm-ewt.proofpoint.com/EWT/v1/KGKeukY!vYQd06pJjE9AAhTaFS-M1rCxuGsuMAuBgcubeYoCIzfnLzi-eNInMvY0TONXlt2qBcPsx1kQkSd44s2C_U_rvIjthLybFDLcVaaOkMcJ_yXZ8RVfFiAN0NqO-okPEYu4Zsg4pA$"
style="all: unset !important; display: inline-block !important; text-decoration: none">
<div class="pfptPrimaryButtonfwm57j6" style="display: inline-block !important; display: inline-block; visibility: visible !important; opacity: 1 !important; color: #000000 !important; color: #000000; font-family: 'Arial', sans-serif !important; font-family: 'Arial', sans-serif; font-size: 14px !important; font-weight: normal !important; text-decoration: none !important; border-radius: 2px !important; padding: 7.5px 16px !important; margin: 3px 0 3px 16px !important; white-space: nowrap !important; width: fit-content !important;
border: 1px solid #666666">
Report Suspicious
</div>
</a>
</div>
<div style="clear: both !important; display: block !important; visibility: hidden !important; line-height: 0 !important; font-size: 0.01px !important; height: 0px"> </div>
</div>
<![endif]>
<div style="display:none !important;display:none;visibility:hidden;mso-hide:all;font-size:1px;color:#ffffff;line-height:1px;max-height:0px;opacity:0;overflow:hidden;">ZjQcmQRYFpfptBannerEnd</div>
<!-- Email Banner : END -->
<!-- BaNnErBlUrFlE-BoDy-end -->
<div dir="ltr"><div>Hi,</div><div>Recently I've compared mvapich (based on mpich 4.3.0 as I recall) and mpich 4.3.1 on single node Intel Xeon 6972p (it has mellanox fabric but since it's single node it's not relevant). The application is quite tricky but similar issue is observed in IMB: the more rank you start the longer the delay is (2 ranks start almost instantly). The test was as follows:<br></div><div>1. Both MPIs is configured similarly:</div><div>./configure --prefix=$HOMEINIT/mvapich/4.0x-mt-ucx --enable-silent-rules \<br> --with-device=ch4:ucx:shm --with-pm=hydra --enable-romio --with-ch3-rank-bits=32 --enable-threads=multiple --without-ze --with-file-system=lustre+nfs \<br> --enable-shared --with-hwloc=embedded --with-ucx=embedded --with-libfabric=embedded --enable-fortran=all --with-ch4-shmmods=posix \<br> CC=icx F77=ifx FC=ifx CXX=icpx \<br> MPICHLIB_CPPFLAGS="-I$WORKINIT/misc.libs/lustre-release/lustre/include -I$WORKINIT/misc.libs/lustre-release/lustre/include/uapi" \<br> MPICHLIB_CFLAGS='-Wno-unused-but-set-variable -Wno-tautological-constant-compare -Wno-initializer-overrides' \<br> MPICHLIB_FCFLAGS='-Wno-unused-but-set-variable -Wno-tautological-constant-compare -Wno-initializer-overrides' \<br> MPICHLIB_CXXFLAGS='-Wno-unused-but-set-variable -Wno-tautological-constant-compare -Wno-initializer-overrides' \<br> 2>&1 | tee configure.log</div><div>(the only difference is the installation path)</div><div>2. Execute the application (mpiexec.hydra -launcher ssh -genvall -bind-to core:1 -np 192 ./app)<br></div><div>3. Review its report.</div><div>4. Recompile MVAPICH without --with-ch4-shmmods=posix</div><div>5. Repeat MVAPICH test.</div><div>So the results are as follows:</div><div>1. MPICH 4.3.1 <br></div><div> Initialization time : 4.02 s<br> Elapsed time : 94.39 s</div><div>2. MVAPICH</div><div> Initialization time : 55.06 s<br> Elapsed time : 131.06 s</div><div>3. MVAPICH with no posix shmem</div><div> Initialization time : 4.03 s<br> Elapsed time : 108.99 s</div><div><br></div><div>As you can see MVAPCIH is quite faster on execution stage (numbers are inclusive) but startup ruinis the "picture". <br></div><div><br></div><div>Is there any differences in shmem (apart from having its own MV_SHM or so) and how it can be fixed?<br></div><div>As I said earlier you can observe the same issue on IMB (presumably on all high pppn runs). The only reason I took this application is because it writes its init phase :).<br></div></div>