<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
Alex,</div>
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<br>
</div>
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
Thanks for reporting this. First, as a general rule we suggest not forcing the posix shmmod when building with UCX. UCX supports its own shared memory module and due to conflicts between the UCX and internal shared memory implementations, both MPICH and MVAPICH
perform best when allowing the UCX shared memory to operate independently. This is the behavior observed when you do not set
<code>--with-ch4-shmmod.</code></div>
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<br>
</div>
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
That said, if you need to use the internal posix shmmod for some reason, I have attached a patch. MVAPICH and MVAPICH-Plus support a different eager module than the standard MPICH version that includes some optimizations for weakly ordered architectures. This
is why the difference is observed in MVAPICH only. Looks like we were doing an unnecessary memset of the entire shmem region which grows exponentially with the local process count. However, based on my investigation, this is unnecessary.</div>
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<br>
</div>
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
Please let me know if you experence any issues with the patch. Otherwise we will include it in our next release with proper acknowledgement.</div>
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<br>
</div>
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
Thanks,</div>
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
Nat</div>
<div class="elementToProof" id="appendonsend"></div>
<hr style="display: inline-block; width: 98%;">
<div class="elementToProof" id="divRplyFwdMsg">
<div class="elementToProof" style="direction: ltr; font-family: Calibri, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
<b>From:</b> Mvapich-discuss <mvapich-discuss-bounces@lists.osu.edu> on behalf of Alex via Mvapich-discuss <mvapich-discuss@lists.osu.edu><br>
<b>Sent:</b> Saturday, August 23, 2025 05:06<br>
<b>To:</b> mvapich-discuss@lists.osu.edu <mvapich-discuss@lists.osu.edu><br>
<b>Subject:</b> [Mvapich-discuss] Very slow startup on mvapich 4.0</div>
<div class="elementToProof" style="direction: ltr;"> </div>
</div>
<div class="elementToProof" style="line-height: 1px; max-height: 0px; display: none; font-size: 1px; color: rgb(255, 255, 255);">
Hi, Recently I've compared mvapich (based on mpich 4. 3. 0 as I recall) and mpich 4. 3. 1 on single node Intel Xeon 6972p (it has mellanox fabric but since it's single node it's not relevant). The application is quite tricky but similar</div>
<div class="elementToProof" style="line-height: 1px; max-height: 0px; display: none; font-size: 1px; color: rgb(255, 255, 255);">
ZjQcmQRYFpfptBannerStart</div>
<div class="elementToProof" id="x_pfptBannerfwm57j6" style="background-color: rgb(207, 211, 215); margin: 16px 0px; padding: 8px 16px; border-top: 4px solid rgb(140, 142, 145); border-radius: 4px; max-width: none; max-height: none; min-width: 200px; display: block;">
<div class="elementToProof" id="x_pfptBannerfwm57j6" style="background-color: rgb(207, 211, 215); margin: 0px 0px 1px; max-width: 600px; max-height: none; display: block;">
<div class="elementToProof" id="x_pfptBannerfwm57j6" style="direction: ltr; text-align: left; line-height: 18px; background-color: rgb(207, 211, 215); max-width: none; max-height: none; display: block; font-family: "Arial", sans-serif; font-size: 14px; color: rgb(0, 0, 0);">
<b>This Message Is From an External Sender</b></div>
<div class="elementToProof" id="x_pfptBannerfwm57j6" style="direction: ltr; text-align: left; line-height: 18px; background-color: rgb(207, 211, 215); margin-top: 2px; max-width: none; max-height: none; display: block; font-family: "Arial", sans-serif; font-size: 12px; color: rgb(0, 0, 0);">
This message came from outside your organization.</div>
</div>
<div class="elementToProof" id="x_pfptBannerfwm57j6" style="background-color: rgb(207, 211, 215); margin: 0px 0px 0px 16px; max-width: none; max-height: none; display: block;">
<div class="elementToProof" style="direction: ltr; text-align: right; white-space: nowrap; margin: 3px 0px 3px 16px; padding: 7.5px 16px; border-width: 1px; border-style: solid; border-color: rgb(102, 102, 102); border-radius: 2px; display: inline-block; font-family: "Arial", sans-serif; font-size: 14px; color: rgb(0, 0, 0);">
<span style="background-color: rgb(207, 211, 215);"><a data-auth="NotApplicable" style="text-decoration: none; display: inline-block; background-color: rgb(207, 211, 215); max-width: none; max-height: none;" class="OWAAutoLink" id="x_pfptBannerfwm57j6" href="https://us-phishalarm-ewt.proofpoint.com/EWT/v1/KGKeukY!vYQd06pJjE9AAhTaFS-M1rCxuGsuMAuBgcubeYoCIzfnLzi-eNInMvY0TONXlt2qBcPsx1kQkSd44s2C_U_rvIjthLybFDLcVaaOkMcJ_yXZ8RVfFiAN0NqO-okPEYu4Zsg4pA$">Report Suspicious</a></span></div>
</div>
<div class="elementToProof" style="direction: ltr; text-align: left; line-height: 0; display: block; font-size: 0.01px;">
</div>
</div>
<div class="elementToProof" style="line-height: 1px; max-height: 0px; display: none; font-size: 1px; color: rgb(255, 255, 255);">
ZjQcmQRYFpfptBannerEnd</div>
<div class="elementToProof" style="direction: ltr;">Hi,</div>
<div class="elementToProof" style="direction: ltr;">Recently I've compared mvapich (based on mpich 4.3.0 as I recall) and mpich 4.3.1 on single node Intel Xeon 6972p (it has mellanox fabric but since it's single node it's not relevant). The application is quite
tricky but similar issue is observed in IMB: the more rank you start the longer the delay is (2 ranks start almost instantly). The test was as follows:</div>
<div class="elementToProof" style="direction: ltr;">1. Both MPIs is configured similarly:</div>
<div class="elementToProof" style="direction: ltr;">./configure --prefix=$HOMEINIT/mvapich/4.0x-mt-ucx --enable-silent-rules \<br>
--with-device=ch4:ucx:shm --with-pm=hydra --enable-romio --with-ch3-rank-bits=32 --enable-threads=multiple --without-ze --with-file-system=lustre+nfs \<br>
--enable-shared --with-hwloc=embedded --with-ucx=embedded --with-libfabric=embedded --enable-fortran=all --with-ch4-shmmods=posix \<br>
CC=icx F77=ifx FC=ifx CXX=icpx \<br>
MPICHLIB_CPPFLAGS="-I$WORKINIT/misc.libs/lustre-release/lustre/include -I$WORKINIT/misc.libs/lustre-release/lustre/include/uapi" \<br>
MPICHLIB_CFLAGS='-Wno-unused-but-set-variable -Wno-tautological-constant-compare -Wno-initializer-overrides' \<br>
MPICHLIB_FCFLAGS='-Wno-unused-but-set-variable -Wno-tautological-constant-compare -Wno-initializer-overrides' \<br>
MPICHLIB_CXXFLAGS='-Wno-unused-but-set-variable -Wno-tautological-constant-compare -Wno-initializer-overrides' \<br>
2>&1 | tee configure.log</div>
<div class="elementToProof" style="direction: ltr;">(the only difference is the installation path)</div>
<div class="elementToProof" style="direction: ltr;">2. Execute the application (mpiexec.hydra -launcher ssh -genvall -bind-to core:1 -np 192 ./app)</div>
<div class="elementToProof" style="direction: ltr;">3. Review its report.</div>
<div class="elementToProof" style="direction: ltr;">4. Recompile MVAPICH without --with-ch4-shmmods=posix</div>
<div class="elementToProof" style="direction: ltr;">5. Repeat MVAPICH test.</div>
<div class="elementToProof" style="direction: ltr;">So the results are as follows:</div>
<div class="elementToProof" style="direction: ltr;">1. MPICH 4.3.1</div>
<div class="elementToProof" style="direction: ltr;"> Initialization time : 4.02 s<br>
Elapsed time : 94.39 s</div>
<div class="elementToProof" style="direction: ltr;">2. MVAPICH</div>
<div class="elementToProof" style="direction: ltr;"> Initialization time : 55.06 s<br>
Elapsed time : 131.06 s</div>
<div class="elementToProof" style="direction: ltr;">3. MVAPICH with no posix shmem</div>
<div class="elementToProof" style="direction: ltr;"> Initialization time : 4.03 s<br>
Elapsed time : 108.99 s</div>
<div class="elementToProof" style="direction: ltr;"><br>
</div>
<div class="elementToProof" style="direction: ltr;">As you can see MVAPCIH is quite faster on execution stage (numbers are inclusive) but startup ruinis the "picture".</div>
<div class="elementToProof" style="direction: ltr;"><br>
</div>
<div class="elementToProof" style="direction: ltr;">Is there any differences in shmem (apart from having its own MV_SHM or so) and how it can be fixed?</div>
<div class="elementToProof" style="direction: ltr;">As I said earlier you can observe the same issue on IMB (presumably on all high pppn runs). The only reason I took this application is because it writes its init phase :).</div>
</body>
</html>