<head><!-- BaNnErBlUrFlE-HeAdEr-start -->
<style>
#pfptBanner2rkz7yt { all: revert !important; display: block !important;
visibility: visible !important; opacity: 1 !important;
background-color: #CFD3D7 !important;
max-width: none !important; max-height: none !important }
.pfptPrimaryButton2rkz7yt:hover, .pfptPrimaryButton2rkz7yt:focus {
background-color: #adb0b4 !important; }
.pfptPrimaryButton2rkz7yt:active {
background-color: #8c8e91 !important; }
</style>
<!-- BaNnErBlUrFlE-HeAdEr-end -->
</head><!-- BaNnErBlUrFlE-BoDy-start -->
<!-- Preheader Text : BEGIN -->
<div style="display:none !important;display:none;visibility:hidden;mso-hide:all;font-size:1px;color:#ffffff;line-height:1px;height:0px;max-height:0px;opacity:0;overflow:hidden;">
Hi Hari, Thank you, It is actually rather hard to reproduce in a standalone example, in my experience. I wrote a simple standalone example with the same partitioning as the actual case, and was not able to reproduce it. We have several MPI_Gatherv
</div>
<!-- Preheader Text : END -->
<!-- Email Banner : BEGIN -->
<div style="display:none !important;display:none;visibility:hidden;mso-hide:all;font-size:1px;color:#ffffff;line-height:1px;height:0px;max-height:0px;opacity:0;overflow:hidden;">ZjQcmQRYFpfptBannerStart</div>
<!--[if ((ie)|(mso))]>
<table border="0" cellspacing="0" cellpadding="0" width="100%" style="padding: 16px 0px 16px 0px; direction: ltr" lang="en"><tr><td>
<table border="0" cellspacing="0" cellpadding="0" style="padding: 0px 10px 5px 6px; width: 100%; border-radius:4px; border-top:4px solid #8c8e91;background-color:#CFD3D7;"><tr><td valign="top">
<table align="left" border="0" cellspacing="0" cellpadding="0" style="padding: 4px 8px 4px 8px">
<tr><td style="color:#000000; font-family: 'Arial', sans-serif; font-weight:bold; font-size:14px; direction: ltr">
This Message Is From an External Sender
</td></tr>
<tr><td style="color:#000000; font-weight:normal; font-family: 'Arial', sans-serif; font-size:12px; direction: ltr">
This message came from outside your organization.
</td></tr>
</table>
<![if ie]><br clear="all"><![endif]>
<table align="right" border="0" cellspacing="0" cellpadding="0" style="padding: 4px 0px 4px 0px"><tr>
<td style="direction: ltr"> <a target="_blank" href="https://us-phishalarm-ewt.proofpoint.com/EWT/v1/KGKeukY!vYQd06mLz2qhajGen88lX-izeqy6zAzfjdnZH-vGBJUlk98FzrGtLTWSt5F9DjEnjfcFWQWzwJnTuc9Yj0Tv89tSZTGF8NEjZk6Ma2zk4WJOCCOuVCTUucYsJPh529kToxKtng$" style="mso-padding-alt: 7.5px; padding: 7.5px; border-radius: 2px; border: 1.5px solid #666666; "><strong style="font-weight: normal; color: #000000; text-decoration: none; font-family: 'Arial', sans-serif; font-size:14px; line-height: 40px; "> Report Suspicious </strong></a> </td>
</tr></table>
</td></tr></table>
</td></tr></table>
<![endif]-->
<![if !((ie)|(mso))]>
<div dir="ltr" lang="en" id="pfptBanner2rkz7yt" style="all: revert !important; display:block !important; text-align: left !important; margin:16px 0px 16px 0px !important; padding:8px 16px 8px 16px !important; border-radius: 4px !important; min-width: 200px !important; background-color: #CFD3D7 !important; background-color: #CFD3D7; border-top: 4px solid #8c8e91 !important; border-top: 4px solid #8c8e91;">
<div id="pfptBanner2rkz7yt" style="all: unset !important; float:left !important; display:block !important; margin: 0px 0px 1px 0px !important; max-width: 600px !important;">
<div id="pfptBanner2rkz7yt" style="all: unset !important; display:block !important; visibility: visible !important; background-color: #CFD3D7 !important; color:#000000 !important; color:#000000; font-family: 'Arial', sans-serif !important; font-family: 'Arial', sans-serif; font-weight:bold !important; font-weight:bold; font-size:14px !important; line-height:18px !important; line-height:18px">
This Message Is From an External Sender
</div>
<div id="pfptBanner2rkz7yt" style="all: unset !important; display:block !important; visibility: visible !important; background-color: #CFD3D7 !important; color:#000000 !important; color:#000000; font-weight:normal; font-family: 'Arial', sans-serif !important; font-family: 'Arial', sans-serif; font-size:12px !important; line-height:18px !important; line-height:18px; margin-top:2px !important;">
This message came from outside your organization.
</div>
</div>
<div id="pfptBanner2rkz7yt" style="all: unset !important; float: right !important; display: block !important; display: block; margin: 0px 0px 0px 16px !important; text-align: right !important; width: fit-content !important;">
<a id="pfptBanner2rkz7yt" href="https://us-phishalarm-ewt.proofpoint.com/EWT/v1/KGKeukY!vYQd06mLz2qhajGen88lX-izeqy6zAzfjdnZH-vGBJUlk98FzrGtLTWSt5F9DjEnjfcFWQWzwJnTuc9Yj0Tv89tSZTGF8NEjZk6Ma2zk4WJOCCOuVCTUucYsJPh529kToxKtng$"
style="all: unset !important; display: inline-block !important; text-decoration: none">
<div class="pfptPrimaryButton2rkz7yt" style="display: inline-block !important; display: inline-block; visibility: visible !important; opacity: 1 !important; color: #000000 !important; color: #000000; font-family: 'Arial', sans-serif !important; font-family: 'Arial', sans-serif; font-size: 14px !important; font-weight: normal !important; text-decoration: none !important; border-radius: 2px !important; padding: 7.5px 16px !important; margin: 3px 0 3px 16px !important; white-space: nowrap !important; width: fit-content !important;
border: 1px solid #666666">
Report Suspicious
</div>
</a>
</div>
<div style="clear: both !important; display: block !important; visibility: hidden !important; line-height: 0 !important; font-size: 0.01px !important; height: 0px"> </div>
</div>
<![endif]>
<div style="display:none !important;display:none;visibility:hidden;mso-hide:all;font-size:1px;color:#ffffff;line-height:1px;height:0px;max-height:0px;opacity:0;overflow:hidden;">ZjQcmQRYFpfptBannerEnd</div>
<!-- Email Banner : END -->
<!-- BaNnErBlUrFlE-BoDy-end -->
<div dir="ltr">Hi Hari,<div><br></div><div>Thank you,</div><div><br></div><div>It is actually rather hard to reproduce in a standalone example, in my experience. I wrote a simple standalone example with the same partitioning as the actual case, and was not able to reproduce it. We have several MPI_Gatherv calls, that operate on the same size of distributed data, and we are allocating memory to store the gathered data. </div><div><br></div><div>Interestingly, on the first data object that we gather, the result is correct, but after we allocate memory to store the result elsewhere and communicate a second data object representing the exact same data, we get an incorrect result for the gatherv.</div><div><br></div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Sep 28, 2022 at 2:41 PM Subramoni, Hari <<a href="mailto:subramoni.1@osu.edu">subramoni.1@osu.edu</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class="msg-3514060511585584934">
<div lang="EN-US" style="overflow-wrap: break-word;">
<div class="m_-3514060511585584934WordSection1">
<p class="MsoNormal">Hi, John.<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">Sorry to hear that you’re facing issues. Let us try this out internally and get back to you shortly.<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">Thx,<u></u><u></u></p>
<p class="MsoNormal">Hari.<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<div style="border-right:none;border-bottom:none;border-left:none;border-top:1pt solid rgb(225,225,225);padding:3pt 0in 0in">
<p class="MsoNormal"><b>From:</b> Mvapich-discuss <<a href="mailto:mvapich-discuss-bounces@lists.osu.edu" target="_blank">mvapich-discuss-bounces@lists.osu.edu</a>>
<b>On Behalf Of </b>John Moore via Mvapich-discuss<br>
<b>Sent:</b> Wednesday, September 28, 2022 1:38 PM<br>
<b>To:</b> <a href="mailto:mvapich-discuss@lists.osu.edu" target="_blank">mvapich-discuss@lists.osu.edu</a><br>
<b>Subject:</b> [Mvapich-discuss] Possible buffer overflow for large messages?<u></u><u></u></p>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<p class="MsoNormal"><span style="font-size:1pt;color:white">Hello, We have a code that does a large Gatherv operation, where the size of the gathered message > 4GB. It is approximately 8GB. We have noticed that the result
of the gatherv operation is incorrect for these large calls. The sizes that <u></u>
<u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:1pt;color:white"><u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal">Hello,<u></u><u></u></p>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">We have a code that does a large Gatherv operation, where the size of the gathered message > 4GB. It is approximately 8GB. We have noticed that the result of the gatherv operation is incorrect for these large calls. The sizes that we are
passing into Gatherv are all within the int limit, and we are using custom data types (MPI_Type_Contiguous) to allow for this larger message size. <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">We have also tried replacing the Gatherv call with Isend/Irecv calls, which are all within the int representation range in terms of the number of bytes communicated, with the same incorrect result.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">When we compile with OpenMPI, the result is correct. Also, when we run the operations on smaller data sets with MVAPICH2 the result is correct. <br>
<br>
This job is being run across two nodes with 16 ranks total (8 ranks each) When we place all the data on a single node, and use the same input data and number of ranks, we again get the correct result. This leads me to believe that some remote send/receive buffer
is being exceeded.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">We are running MVAPICH2-GDR-2.3.6, but these buffers are all CPU buffers, and we are running this executable with MV2_USE_CUDA=0. Perhaps there are some environmental variables to change here? Any advice would be greatly appreciated.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Thank you,<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">John<u></u><u></u></p>
</div>
</div>
</div>
</div>
</div></blockquote></div>