<head><!-- BaNnErBlUrFlE-HeAdEr-start -->
<style>
#pfptBannerohhcjig { all: revert !important; display: block !important;
visibility: visible !important; opacity: 1 !important;
background-color: #CFD3D7 !important;
max-width: none !important; max-height: none !important }
.pfptPrimaryButtonohhcjig:hover, .pfptPrimaryButtonohhcjig:focus {
background-color: #adb0b4 !important; }
.pfptPrimaryButtonohhcjig:active {
background-color: #8c8e91 !important; }
</style>
<!-- BaNnErBlUrFlE-HeAdEr-end -->
</head><!-- BaNnErBlUrFlE-BoDy-start -->
<!-- Preheader Text : BEGIN -->
<div style="display:none !important;display:none;visibility:hidden;mso-hide:all;font-size:1px;color:#ffffff;line-height:1px;height:0px;max-height:0px;opacity:0;overflow:hidden;">
Hi Hari, Thanks for the reply. Could I know what algorithm is used when the CMA support is disabled? Is it by copying to shared memory and copying out by the other process? If this process is abstracted by a queue, which queue algorithm does
</div>
<!-- Preheader Text : END -->
<!-- Email Banner : BEGIN -->
<div style="display:none !important;display:none;visibility:hidden;mso-hide:all;font-size:1px;color:#ffffff;line-height:1px;height:0px;max-height:0px;opacity:0;overflow:hidden;">ZjQcmQRYFpfptBannerStart</div>
<!--[if ((ie)|(mso))]>
<table border="0" cellspacing="0" cellpadding="0" width="100%" style="padding: 16px 0px 16px 0px; direction: ltr" lang="en"><tr><td>
<table border="0" cellspacing="0" cellpadding="0" style="padding: 0px 10px 5px 6px; width: 100%; border-radius:4px; border-top:4px solid #8c8e91;background-color:#CFD3D7;"><tr><td valign="top">
<table align="left" border="0" cellspacing="0" cellpadding="0" style="padding: 4px 8px 4px 8px">
<tr><td style="color:#000000; font-family: 'Arial', sans-serif; font-weight:bold; font-size:14px; direction: ltr">
This Message Is From an External Sender
</td></tr>
<tr><td style="color:#000000; font-weight:normal; font-family: 'Arial', sans-serif; font-size:12px; direction: ltr">
This message came from outside your organization.
</td></tr>
</table>
<![if ie]><br clear="all"><![endif]>
<table align="right" border="0" cellspacing="0" cellpadding="0" style="padding: 4px 0px 4px 0px"><tr>
<td style="direction: ltr"> <a target="_blank" href="https://us-phishalarm-ewt.proofpoint.com/EWT/v1/KGKeukY!vYQd06Uha8Yg4lZ8U2FE37LdUXAaoApKOM0wnU2hX-LDSvSZI89CViTJE9_NH091rj7GMl5uKUf-ReGFDusjwIYGgLNTtZZ3fD4qFNr2s6FHi8no6OWsvQdRo_b_Q84D6b5NyA$" style="mso-padding-alt: 7.5px; padding: 7.5px; border-radius: 2px; border: 1.5px solid #666666; "><strong style="font-weight: normal; color: #000000; text-decoration: none; font-family: 'Arial', sans-serif; font-size:14px; line-height: 40px; "> Report Suspicious </strong></a> </td>
</tr></table>
</td></tr></table>
</td></tr></table>
<![endif]-->
<![if !((ie)|(mso))]>
<div dir="ltr" lang="en" id="pfptBannerohhcjig" style="all: revert !important; display:block !important; text-align: left !important; margin:16px 0px 16px 0px !important; padding:8px 16px 8px 16px !important; border-radius: 4px !important; min-width: 200px !important; background-color: #CFD3D7 !important; background-color: #CFD3D7; border-top: 4px solid #8c8e91 !important; border-top: 4px solid #8c8e91;">
<div id="pfptBannerohhcjig" style="all: unset !important; float:left !important; display:block !important; margin: 0px 0px 1px 0px !important; max-width: 600px !important;">
<div id="pfptBannerohhcjig" style="all: unset !important; display:block !important; visibility: visible !important; background-color: #CFD3D7 !important; color:#000000 !important; color:#000000; font-family: 'Arial', sans-serif !important; font-family: 'Arial', sans-serif; font-weight:bold !important; font-weight:bold; font-size:14px !important; line-height:18px !important; line-height:18px">
This Message Is From an External Sender
</div>
<div id="pfptBannerohhcjig" style="all: unset !important; display:block !important; visibility: visible !important; background-color: #CFD3D7 !important; color:#000000 !important; color:#000000; font-weight:normal; font-family: 'Arial', sans-serif !important; font-family: 'Arial', sans-serif; font-size:12px !important; line-height:18px !important; line-height:18px; margin-top:2px !important;">
This message came from outside your organization.
</div>
</div>
<div id="pfptBannerohhcjig" style="all: unset !important; float: right !important; display: block !important; display: block; margin: 0px 0px 0px 16px !important; text-align: right !important; width: fit-content !important;">
<a id="pfptBannerohhcjig" href="https://us-phishalarm-ewt.proofpoint.com/EWT/v1/KGKeukY!vYQd06Uha8Yg4lZ8U2FE37LdUXAaoApKOM0wnU2hX-LDSvSZI89CViTJE9_NH091rj7GMl5uKUf-ReGFDusjwIYGgLNTtZZ3fD4qFNr2s6FHi8no6OWsvQdRo_b_Q84D6b5NyA$"
style="all: unset !important; display: inline-block !important; text-decoration: none">
<div class="pfptPrimaryButtonohhcjig" style="display: inline-block !important; display: inline-block; visibility: visible !important; opacity: 1 !important; color: #000000 !important; color: #000000; font-family: 'Arial', sans-serif !important; font-family: 'Arial', sans-serif; font-size: 14px !important; font-weight: normal !important; text-decoration: none !important; border-radius: 2px !important; padding: 7.5px 16px !important; margin: 3px 0 3px 16px !important; white-space: nowrap !important; width: fit-content !important;
border: 1px solid #666666">
Report Suspicious
</div>
</a>
</div>
<div style="clear: both !important; display: block !important; visibility: hidden !important; line-height: 0 !important; font-size: 0.01px !important; height: 0px"> </div>
</div>
<![endif]>
<div style="display:none !important;display:none;visibility:hidden;mso-hide:all;font-size:1px;color:#ffffff;line-height:1px;height:0px;max-height:0px;opacity:0;overflow:hidden;">ZjQcmQRYFpfptBannerEnd</div>
<!-- Email Banner : END -->
<!-- BaNnErBlUrFlE-BoDy-end -->
<div dir="ltr">Hi Hari,<div><br></div><div>Thanks for the reply. Could I know what algorithm is used when the CMA support is disabled? Is it by copying to shared memory and copying out by the other process? If this process is abstracted by a queue, which queue algorithm does it use? </div><div><br></div><div>Is there any way to find out what collective algorithm variant it's running? Many MPI libraries have many variants of allreduce/reduce algorithm. I would like to know which algorithm is used for a specific message size. I'm comparing the collective algorithm for intra machine communication between OpenMPI and MVAPICH2. The latency comparison is similar but the allreduce performance has a large gap. </div><div><br></div><div>Best,</div><div>Zhiting</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Apr 13, 2023 at 4:09 PM Subramoni, Hari <<a href="mailto:subramoni.1@osu.edu">subramoni.1@osu.edu</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class="msg1901956199875631360">
<div lang="EN-US" style="overflow-wrap: break-word;">
<div class="m_1901956199875631360WordSection1">
<p class="MsoNormal">Hi, Zhiting.<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">Please accept our sincere apologies for the delay.<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">The correct channel for MVAPICH2 2.3.7 is ch3:mrail. The Nemesis netmod is an older, depreciated version.<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">The environment variable MV2_USE_SHMEM_COLL enables and disables the use of our hierarchical process topology aware collective communication schemes.<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">If you would like to disable CMA support, please set the environment variable MV2_USE_SMP_CMA=0 at runtime.<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">Several of these designs have been implemented in various MVAPICH libraries.<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">Best,<u></u><u></u></p>
<p class="MsoNormal">Hari.<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<div style="border-right:none;border-bottom:none;border-left:none;border-top:1pt solid rgb(225,225,225);padding:3pt 0in 0in">
<p class="MsoNormal"><b>From:</b> Mvapich-discuss <mvapich-discuss-bounces+subramon=<a href="mailto:cse.ohio-state.edu@lists.osu.edu" target="_blank">cse.ohio-state.edu@lists.osu.edu</a>>
<b>On Behalf Of </b>zhiting zhu via Mvapich-discuss<br>
<b>Sent:</b> Friday, April 7, 2023 7:58 PM<br>
<b>To:</b> <a href="mailto:Mvapich-discuss@lists.osu.edu" target="_blank">Mvapich-discuss@lists.osu.edu</a><br>
<b>Subject:</b> [Mvapich-discuss] Question on shared memory communication<u></u><u></u></p>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<p class="MsoNormal"><span style="font-size:1pt;color:white">Hi, I'm a new user of the MPI lib. I have a few questions. - I notice that in the feature page for shared memory communication, there're two options: Shared-Memory-CH3
and Shared-Memory-Nemesis. The descriptions are exactly the same. <u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:1pt;color:white"><u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal">Hi,<u></u><u></u></p>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">I'm a new user of the MPI lib. I have a few questions. <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">- I notice that in the feature page for shared memory communication, there're two options: Shared-Memory-CH3 and Shared-Memory-Nemesis. The descriptions are exactly the same. What's the difference between these two? I find that in the documentation
MV2_USE_SHMEM_COLL applies to OFA-IB-CH3 and OFA-iWARP-CH3. It seems I have to use shared-memory-ch3? <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">FFLAGS="-w -fallow-argument-mismatch -O2" ../configure --with-device=ch3:mrail --disable-shared --prefix=$SCRIPT_DIR/mvapich2_install/<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">- I found a description on what nemesis is: <a href="https://urldefense.com/v3/__https:/www.mcs.anl.gov/uploads/cels/papers/TM-292.pdf__;!!KGKeukY!x9GebWlzOwJDIaxL9-bETHPm371fheTuZHtVSJ15G3kugdUOeEYmtRp9BpmzRGMtI9WUiPHAwYNL56dSdR9oXMi1td2HJ6HR$" target="_blank">https://www.mcs.anl.gov/uploads/cels/papers/TM-292.pdf</a>
Is there any documentation about the shared memory algorithms used in ch3:mrail? I'm looking for the case that doesn't use CMA/XPMEM/LiMiC. <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">- I also found the paper: Design and Evaluation of Shared Memory CommunicationBenchmarks on Emerging Architectures using MVAPICH2 <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Is any of these designs incorporated in the MVAPICH2? I'm trying to look at the high level api mentioned in this paper(msg_enqueue/codemsg_dequeue) but I find nothing. <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Best,<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Zhiting<u></u><u></u></p>
</div>
</div>
</div>
</div>
</div></blockquote></div>