[Mvapich-discuss] Enable verbose logging

[Shan-ho Tsai]

Hello Dr. Panda,

Thank you so much for your response. We are seeing the same issues with MVAPICH 2.3.6 as well. We suspect the problem is with our IB network configuration, but it would be very helpful if we could run MVAPICH with some verbose logging turned on, to have more visibility into what communication is attempted and failing/hanging. Is this possible?

Thanks again,
Shan-Ho Tsai

________________________________
From: Panda, Dhabaleswar <panda at cse.ohio-state.edu>
Sent: Tuesday, February 15, 2022 3:16 PM
To: Mvapich-discuss at lists.osu.edu <Mvapich-discuss at lists.osu.edu>; Shan-ho Tsai <shtsai at uga.edu>
Subject: Re: Enable verbose logging

[EXTERNAL SENDER - PROCEED CAUTIOUSLY]


Thanks for your note. Sorry to know that you are experiencing this issue. Can you please update the MVAPICH version to the latest one (2.3.6) and see whether the issue persists? The 2.3.4 version is already more than one and a half years old.

Thanks,

DK

________________________________________
From: Mvapich-discuss <mvapich-discuss-bounces+panda.2=osu.edu at lists.osu.edu> on behalf of Shan-ho Tsai via Mvapich-discuss <mvapich-discuss at lists.osu.edu>
Sent: Tuesday, February 15, 2022 2:59 PM
To: Mvapich-discuss at lists.osu.edu
Subject: [Mvapich-discuss] Enable verbose logging


Greetings, MVAPICH team,

We are running an application with MVAPICH 2.3.4 on multiple nodes (e.g. 16) and occasionally the application will get stuck with what appears to be communication issues between MPI processes. Is there a way to turn on verbose logging when using mpirun_rsh or the Slurm srun to launch the MPI processes, to help us troubleshoot our MPI communication issues on the IB network?

Thank you very much in advance.

Best regards,

Shan-Ho Tsai
University of Georgia



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osu.edu/pipermail/mvapich-discuss/attachments/20220215/6304a82c/attachment-0020.html>