[Mvapich-discuss] OMB: Patch for Memory leak in validate_data

[Jin, Shi]

!-------------------------------------------------------------------|
  This Message Is From an External Sender
  This message came from outside your organization.
|-------------------------------------------------------------------!

Hi Hari,

Thank you very much. May I ask one more question? I found the for PT2PT validate_data() always allocate and free temporary buffer of message size inside the function, and the function is called in every iteration. It makes the total run time of OMB much slower when --validation is turned on. In my observation, running osu_mbw_mr (72 ranks, 36 ppn) takes 20 minutes to finish with --validation, while taking 3 minutes without --validation. Is it something we can improve in the future versions? I think we can use a persistent and large enough buffer through the test for validation, so we don't have to alloc/free in flight.


Thanks and best regards,
Shi

On 6/9/23, 9:31 AM, "Subramoni, Hari" <subramoni.1 at osu.edu <mailto:subramoni.1 at osu.edu>> wrote:


CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.






Hi, Luke/Jin.


Thanks again for providing the patch and making OMB more robust.


We have taken in the patch with an acknowledgement to you. It will be available with the next release.


Best,
Hari.


-----Original Message-----
From: Subramoni, Hari <subramoni.1 at osu.edu <mailto:subramoni.1 at osu.edu>>
Sent: Thursday, June 8, 2023 7:25 AM
To: Robison, Luke <lrbison at amazon.com <mailto:lrbison at amazon.com>>; Announcement about MVAPICH2 (MPI over InfiniBand, RoCE, Omni-Path, iWARP and EFA) Libraries developed at NBCL/OSU <mvapich-discuss at lists.osu.edu <mailto:mvapich-discuss at lists.osu.edu>>
Cc: Jin, Shi <sjina at amazon.com <mailto:sjina at amazon.com>>; Subramoni, Hari <subramoni.1 at osu.edu <mailto:subramoni.1 at osu.edu>>
Subject: RE: OMB: Patch for Memory leak in validate_data


Hi, Luke/Jin.


Thanks for reporting the issue and providing the patch. WE really appreciate it.


We will review it and take it in with an acknowledgement to you.


Best,
Hari.


-----Original Message-----
From: Mvapich-discuss <mvapich-discuss-bounces at lists.osu.edu <mailto:mvapich-discuss-bounces at lists.osu.edu>> On Behalf Of Robison, Luke via Mvapich-discuss
Sent: Tuesday, June 6, 2023 3:05 PM
To: mvapich-discuss at lists.osu.edu <mailto:mvapich-discuss at lists.osu.edu>
Cc: Jin, Shi <sjina at amazon.com <mailto:sjina at amazon.com>>
Subject: [Mvapich-discuss] OMB: Patch for Memory leak in validate_data


Hello,


My team has found an issue with the validate_data() function in osu-micro-benchmarks. Initially it was presented as a crash during the execution of osu_multi_lat, but we find it may affect more tests than just that one. The problem is that "expected_buffer" is not freed during validation, and eventually memory leaks exhaust system memory.


The patch is below. Please consider crediting Shi Jin for this fix. The patch was generated originally on 7.0.1, but found to also be present in 7.1.1.


Thank you,
Luke Robison
Amazon AWS


---


Author: Shi Jin <sjina at amazon.com <mailto:sjina at amazon.com>>
Date: Fri Jun 2 22:35:05 2023 +0000


c/util: Fix a memory leak in validate_data()


In validate_data(), expected_buffer is not freed
after being malloced, which causes a memory leak
and could make the benchmark crash due to out of
memory. This patch fixes this issue.


diff --git a/c/util/osu_util_mpi.c b/c/util/osu_util_mpi.c index 99243b7..2e98272 100644
--- a/c/util/osu_util_mpi.c
+++ b/c/util/osu_util_mpi.c
@@ -1766,8 +1766,10 @@ uint8_t validate_data(void* r_buf, size_t size, int num_procs,
} elseif if (memcmp(temp_char_r_buf, expected_buffer, num_elements)) {
free(temp_r_buf);
+ free(expected_buffer);
return 1;
}
free(temp_r_buf);
+ free(expected_buffer);
return 0;
}
break;
_______________________________________________
Mvapich-discuss mailing list
Mvapich-discuss at lists.osu.edu <mailto:Mvapich-discuss at lists.osu.edu>
https://lists.osu.edu/mailman/listinfo/mvapich-discuss <https://lists.osu.edu/mailman/listinfo/mvapich-discuss>