<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Hello Dr. Panda,</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Thank you so much for your response. We are seeing the same issues with MVAPICH 2.3.6 as well. We suspect the problem is with our IB network configuration, but it would be very helpful if we could run MVAPICH with some verbose logging turned on, to have more
visibility into what communication is attempted and failing/hanging. Is this possible?</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Thanks again,</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Shan-Ho Tsai<br>
</div>
<div>
<div id="appendonsend"></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<hr tabindex="-1" style="display:inline-block; width:98%">
<div id="divRplyFwdMsg" dir="ltr"><font style="font-size:11pt" face="Calibri, sans-serif" color="#000000"><b>From:</b> Panda, Dhabaleswar <panda@cse.ohio-state.edu><br>
<b>Sent:</b> Tuesday, February 15, 2022 3:16 PM<br>
<b>To:</b> Mvapich-discuss@lists.osu.edu <Mvapich-discuss@lists.osu.edu>; Shan-ho Tsai <shtsai@uga.edu><br>
<b>Subject:</b> Re: Enable verbose logging</font>
<div> </div>
</div>
<div class="BodyFragment"><font size="2"><span style="font-size:11pt">
<div class="PlainText">[EXTERNAL SENDER - PROCEED CAUTIOUSLY]<br>
<br>
<br>
Thanks for your note. Sorry to know that you are experiencing this issue. Can you please update the MVAPICH version to the latest one (2.3.6) and see whether the issue persists? The 2.3.4 version is already more than one and a half years old.<br>
<br>
Thanks,<br>
<br>
DK<br>
<br>
________________________________________<br>
From: Mvapich-discuss <mvapich-discuss-bounces+panda.2=osu.edu@lists.osu.edu> on behalf of Shan-ho Tsai via Mvapich-discuss <mvapich-discuss@lists.osu.edu><br>
Sent: Tuesday, February 15, 2022 2:59 PM<br>
To: Mvapich-discuss@lists.osu.edu<br>
Subject: [Mvapich-discuss] Enable verbose logging<br>
<br>
<br>
Greetings, MVAPICH team,<br>
<br>
We are running an application with MVAPICH 2.3.4 on multiple nodes (e.g. 16) and occasionally the application will get stuck with what appears to be communication issues between MPI processes. Is there a way to turn on verbose logging when using mpirun_rsh
or the Slurm srun to launch the MPI processes, to help us troubleshoot our MPI communication issues on the IB network?<br>
<br>
Thank you very much in advance.<br>
<br>
Best regards,<br>
<br>
Shan-Ho Tsai<br>
University of Georgia<br>
<br>
<br>
<br>
</div>
</span></font></div>
</div>
</body>
</html>