<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
Hi Christof, <br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
Thanks for the detailed bug report. The odd process count looks like an issue in our CPU binding. I will take a look at that and see where the error is occurring, that one should be fairly straightforward to fix. Was this a PMI1 or PMI2 build of MVAPICH?</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
For the hanging issue, let me take a look and see if I can reproduce this on our system. There may be a conflict with our shared memory implementation and the underlying OFI shared memory support. Can you please try running with the environment variable
<code>MVP_USE_SHARED_MEM=0</code> set? This will disable our enhanced shared memory designs so that we can check if they are somehow conflicting with PSM2/OPX. Am I correct in assuming that anything less than 58 ppn is working?
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
Thanks,</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
Nat<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<br>
</div>
<div id="appendonsend"></div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> Mvapich-discuss <mvapich-discuss-bounces@lists.osu.edu> on behalf of christof.koehler--- via Mvapich-discuss <mvapich-discuss@lists.osu.edu><br>
<b>Sent:</b> Sunday, May 28, 2023 08:56<br>
<b>To:</b> mvapich-discuss@lists.osu.edu <mvapich-discuss@lists.osu.edu><br>
<b>Subject:</b> Re: [Mvapich-discuss] mvapich 3.0b, hangs depending on number of ranks</font>
<div> </div>
</div>
<div class="BodyFragment"><font size="2"><span style="font-size:11pt;">
<div class="PlainText">Addendum:<br>
<br>
While the stack traces note the mpi library locations as<br>
mvapich2/3.0a-libfabric16 and mvapich2/3.0a I can assure you that the<br>
mvapich 3.0 library source used is in fact 3.0b and not 3.0a !<br>
<br>
On Sun, May 28, 2023 at 02:47:27PM +0200, christof.koehler--- via Mvapich-discuss wrote:<br>
> Hello everybody,<br>
> <br>
> first, my apologies for the very lengthy email.<br>
> <br>
> I have been testing the mvapich 3.0b on our cluster (Rocky Linux 9.1,<br>
> OmniPath interconnect, 64 cores per node) a bit more. I see a mpi hello<br>
> world program hanging depending on the number of ranks used, with some<br>
> slight variations in pstack traces depending on the libfabric version.<br>
> I will also report in this email a, possibly unrelated, problem with <br>
> launching an odd number of ranks (tasks) on single and multiple nodes <br>
> when using mvapich2 2.3.7 and mvapich 3.0b.<br>
> <br>
> I did not test this number of ranks before. Actually, before assembling<br>
> the data for the email I thought it was a problem when trying to<br>
> run on more than one node. I discovered this when changing hfi1 module<br>
> parameters for openmpi and re-testing everything.<br>
> <br>
> Note that we have the hfi1 module parameter num_user_contexts=128 set <br>
> to accomodate openmpi. Launcher is always srun --mpi=pmi2 ... with<br>
> CpuBind=cores set in the partition definition.<br>
> <br>
> The same program appears to work fine with mpich 4.1.1 (ofi) and<br>
> openmpi 4.1.5 (apparently ofi) in all situations on one and two nodes.<br>
> <br>
> PART 1, launching odd number of ranks<br>
> -------------------------------------<br>
> When trying to launch 57 ranks (--ntasks-per-node=57) the launchers of<br>
> mvapich2 and mapich 3.0b fail with these error messages. Note that<br>
> mvapich2 (!, not 3.0b) works when using an even number of ranks up to 64 <br>
> on a single node, I did not test the odd ones (59, 61, 63) in between. I<br>
> assume mvapich2 will work for any even number of ranks.<br>
> <br>
> mvapich2<br>
> Error parsing CPU mapping string<br>
> INTERNAL ERROR: invalid error code ffffffff (Ring Index out of range) in<br>
> smpi_setaffinity:2741<br>
> Fatal error in MPI_Init: Other MPI error, error stack:<br>
> MPIR_Init_thread(493)........: <br>
> MPID_Init(400)...............: <br>
> MPIDI_CH3I_set_affinity(3594): <br>
> smpi_setaffinity(2741).......: Error parsing CPU mapping string<br>
> srun: Job step aborted: Waiting up to 32 seconds for job step to finish.<br>
> slurmstepd: error: *** STEP 931.1 ON node013 CANCELLED AT<br>
> 2023-05-28T13:38:46 ***<br>
> srun: error: node013: tasks 0-55: Killed<br>
> srun: error: node013: task 56: Exited with exit code 1<br>
> <br>
> mvapich 3.0b<br>
> <br>
> Error parsing CPU mapping string<br>
> Invalid error code (-1) (error ring index 127 invalid)<br>
> INTERNAL ERROR: invalid error code ffffffff (Ring Index out of range) in<br>
> smpi_setaffinity:2789<br>
> Abort(2141583) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init:<br>
> Other MPI error, error stack:<br>
> MPIR_Init_thread(175)...........: <br>
> MPID_Init(597)..................: <br>
> srun: Job step aborted: Waiting up to 32 seconds for job step to finish.<br>
> MPIDI_MVP_mpi_init_hook(268)....: <br>
> MPIDI_MVP_CH4_set_affinity(3745): <br>
> smpi_setaffinity(2789)..........: Error parsing CPU mapping string<br>
> In: PMI_Abort(2141583, Fatal error in PMPI_Init: Other MPI error, error<br>
> stack:<br>
> MPIR_Init_thread(175)...........: <br>
> MPID_Init(597)..................: <br>
> MPIDI_MVP_mpi_init_hook(268)....: <br>
> MPIDI_MVP_CH4_set_affinity(3745): <br>
> smpi_setaffinity(2789)..........: Error parsing CPU mapping string)<br>
> slurmstepd: error: *** STEP 932.0 ON node013 CANCELLED AT<br>
> 2023-05-28T13:45:25 ***<br>
> srun: error: node013: tasks 0-55: Killed<br>
> srun: error: node013: task 56: Exited with exit code 143<br>
> <br>
> <br>
> PART 2, hanging mpi hello world with mvapich 3.0b<br>
> -------------------------------------------------<br>
> Launching mvapich 3.0b via srun with --ntasks-per-node=58 or more even<br>
> ranks just hangs (Cornelis provided libfabric 1.16.1 and self-compiled 1.18.0).
<br>
> I did not test odd ranks. I see the mpi processes being spawned, but the <br>
> program never finishes or progresses. I add some pstack traces which might be <br>
> helpful. Please advise me how to obtain better debugging information if needed.<br>
> <br>
> <br>
> libfabric 1.16.1 (appears to use psm2 or ofi psm2 provider under the hood?)<br>
> Thread 2 (Thread 0x7f402c460640 (LWP 24222) "mpi_hello_world"):<br>
> #0 0x00007f409739771f in poll () from target:/lib64/libc.so.6<br>
> #1 0x00007f409657e245 in ips_ptl_pollintr () from target:/lib64/libpsm2.so.2
<br>
> #2 0x00007f40972f4802 in start_thread () from target:/lib64/libc.so.6
<br>
> #3 0x00007f4097294450 in clone3 () from target:/lib64/libc.so.6<br>
> Thread 1 (Thread 0x7f40961cbac0 (LWP 24118) "mpi_hello_world"):<br>
> #0 0x00007f409657cabd in ips_ptl_poll () from target:/lib64/libpsm2.so.2
<br>
> #1 0x00007f409657a96f in psmi_poll_internal () from target:/lib64/libpsm2.so.2
<br>
> #2 0x00007f409657482d in psm2_mq_ipeek () from target:/lib64/libpsm2.so.2
<br>
> #3 0x00007f4096dd9459 in psmx2_cq_poll_mq (cq=cq@entry=0x697d00, trx_ctxt=0x6856d0, event_in=event_in@entry=0x7ffd119acdc0,<br>
> count=count@entry=8, src_addr=src_addr@entry=0x0) at prov/psm2/src/psmx2_cq.c:1086<br>
> #4 0x00007f4096ddc295 in psmx2_cq_readfrom (cq=0x697d00, buf=0x7ffd119acdc0, count=8, src_addr=0x0) at prov/psm2/src/psmx2_cq.c:1591
<br>
> #5 0x00007f4097bf22e7 in MPIDI_OFI_progress () from target:/cluster/mpi/mvapich2/3.0a-libfabric16/gcc11.3.1/lib/libmpi.so.12
<br>
> #6 0x00007f4097c19bfd in MPIDI_MVP_progress () from target:/cluster/mpi/mvapich2/3.0a-libfabric16/gcc11.3.1/lib/libmpi.so.12
<br>
> #7 0x00007f4097bcb3c2 in progress_test () from target:/cluster/mpi/mvapich2/3.0a-libfabric16/gcc11.3.1/lib/libmpi.so.12
<br>
> #8 0x00007f4097bcb793 in MPID_Progress_wait () from target:/cluster/mpi/mvapich2/3.0a-libfabric16/gcc11.3.1/lib/libmpi.so.12
<br>
> #9 0x00007f4097684681 in MPIR_Wait_state () from target:/cluster/mpi/mvapich2/3.0a-libfabric16/gcc11.3.1/lib/libmpi.so.12
<br>
> #10 0x00007f4097b4d479 in MPIC_Wait () from target:/cluster/mpi/mvapich2/3.0a-libfabric16/gcc11.3.1/lib/libmpi.so.12
<br>
> #11 0x00007f4097b4d9ee in MPIC_Recv () from target:/cluster/mpi/mvapich2/3.0a-libfabric16/gcc11.3.1/lib/libmpi.so.12
<br>
> #12 0x00007f40976fd73f in MPIR_Allreduce_intra_recursive_doubling () from target:/cluster/mpi/mvapich2/3.0a-libfabric16/gcc11.3.1/lib/libmpi.so.12<br>
> #13 0x00007f4097609ca2 in MPIR_Allreduce_impl () from target:/cluster/mpi/mvapich2/3.0a-libfabric16/gcc11.3.1/lib/libmpi.so.12
<br>
> #14 0x00007f4097b523de in create_2level_comm () from target:/cluster/mpi/mvapich2/3.0a-libfabric16/gcc11.3.1/lib/libmpi.so.12
<br>
> #15 0x00007f4097c07a38 in MPIDI_MVP_post_init () from target:/cluster/mpi/mvapich2/3.0a-libfabric16/gcc11.3.1/lib/libmpi.so.12
<br>
> #16 0x00007f4097bc8b40 in MPID_InitCompleted () from target:/cluster/mpi/mvapich2/3.0a-libfabric16/gcc11.3.1/lib/libmpi.so.12
<br>
> #17 0x00007f409766dd4f in MPIR_Init_thread () from target:/cluster/mpi/mvapich2/3.0a-libfabric16/gcc11.3.1/lib/libmpi.so.12
<br>
> #18 0x00007f409766dae2 in PMPI_Init () from target:/cluster/mpi/mvapich2/3.0a-libfabric16/gcc11.3.1/lib/libmpi.so.12
<br>
> #19 0x000000000040119d in main ()<br>
> <br>
> libfabric 1.18.1 (this appears to be the fi_opx provider now)<br>
> #0 0x00007fd43d31b087 in fi_opx_shm_poll_many.constprop () from target:/cluster/libraries/libfabric/1.18.0/lib/libfabric.so.1<br>
> #1 0x00007fd43d300367 in fi_opx_cq_read_FI_CQ_FORMAT_TAGGED_0_OFI_RELIABILITY_KIND_ONLOAD_FI_OPX_HDRQ_MASK_2048_0x0018000000000000ull () from target:/cluster/libraries/libfabric/1.18.0/lib/libfabric.so.1<br>
> #2 0x00007fd43e2392e7 in MPIDI_OFI_progress () from target:/cluster/mpi/mvapich2/3.0a/gcc11.3.1/lib/libmpi.so.12<br>
> #3 0x00007fd43e260bfd in MPIDI_MVP_progress () from target:/cluster/mpi/mvapich2/3.0a/gcc11.3.1/lib/libmpi.so.12<br>
> #4 0x00007fd43e2123c2 in progress_test () from target:/cluster/mpi/mvapich2/3.0a/gcc11.3.1/lib/libmpi.so.12<br>
> #5 0x00007fd43e212793 in MPID_Progress_wait () from target:/cluster/mpi/mvapich2/3.0a/gcc11.3.1/lib/libmpi.so.12<br>
> #6 0x00007fd43dccb681 in MPIR_Wait_state () from target:/cluster/mpi/mvapich2/3.0a/gcc11.3.1/lib/libmpi.so.12<br>
> #7 0x00007fd43e194479 in MPIC_Wait () from target:/cluster/mpi/mvapich2/3.0a/gcc11.3.1/lib/libmpi.so.12<br>
> #8 0x00007fd43e1949ee in MPIC_Recv () from target:/cluster/mpi/mvapich2/3.0a/gcc11.3.1/lib/libmpi.so.12<br>
> #9 0x00007fd43dd4473f in MPIR_Allreduce_intra_recursive_doubling () from target:/cluster/mpi/mvapich2/3.0a/gcc11.3.1/lib/libmpi.so.12<br>
> #10 0x00007fd43dc50ca2 in MPIR_Allreduce_impl () from target:/cluster/mpi/mvapich2/3.0a/gcc11.3.1/lib/libmpi.so.12<br>
> #11 0x00007fd43e1993de in create_2level_comm () from target:/cluster/mpi/mvapich2/3.0a/gcc11.3.1/lib/libmpi.so.12<br>
> #12 0x00007fd43e24ea38 in MPIDI_MVP_post_init () from target:/cluster/mpi/mvapich2/3.0a/gcc11.3.1/lib/libmpi.so.12<br>
> #13 0x00007fd43e20fb40 in MPID_InitCompleted () from target:/cluster/mpi/mvapich2/3.0a/gcc11.3.1/lib/libmpi.so.12<br>
> #14 0x00007fd43dcb4d4f in MPIR_Init_thread () from target:/cluster/mpi/mvapich2/3.0a/gcc11.3.1/lib/libmpi.so.12<br>
> #15 0x00007fd43dcb4ae2 in PMPI_Init () from target:/cluster/mpi/mvapich2/3.0a/gcc11.3.1/lib/libmpi.so.12<br>
> #16 0x000000000040119d in main ()<br>
> <br>
> #0 0x00007fd43d0140e0 in __errno_location@plt () from target:/cluster/libraries/libfabric/1.18.0/lib/libfabric.so.1<br>
> #1 0x00007fd43d300dd8 in fi_opx_cq_read_FI_CQ_FORMAT_TAGGED_0_OFI_RELIABILITY_KIND_ONLOAD_FI_OPX_HDRQ_MASK_2048_0x0018000000000000ull () from target:/cluster/libraries/libfabric/1.18.0/lib/libfabric.so.1<br>
> #2 0x00007fd43e2392e7 in MPIDI_OFI_progress () from target:/cluster/mpi/mvapich2/3.0a/gcc11.3.1/lib/libmpi.so.12<br>
> #3 0x00007fd43e260bfd in MPIDI_MVP_progress () from target:/cluster/mpi/mvapich2/3.0a/gcc11.3.1/lib/libmpi.so.12<br>
> #4 0x00007fd43e2123c2 in progress_test () from target:/cluster/mpi/mvapich2/3.0a/gcc11.3.1/lib/libmpi.so.12<br>
> #5 0x00007fd43e212793 in MPID_Progress_wait () from target:/cluster/mpi/mvapich2/3.0a/gcc11.3.1/lib/libmpi.so.12<br>
> #6 0x00007fd43dccb681 in MPIR_Wait_state () from target:/cluster/mpi/mvapich2/3.0a/gcc11.3.1/lib/libmpi.so.12<br>
> #7 0x00007fd43e194479 in MPIC_Wait () from target:/cluster/mpi/mvapich2/3.0a/gcc11.3.1/lib/libmpi.so.12<br>
> #8 0x00007fd43e1949ee in MPIC_Recv () from target:/cluster/mpi/mvapich2/3.0a/gcc11.3.1/lib/libmpi.so.12<br>
> #9 0x00007fd43dd4473f in MPIR_Allreduce_intra_recursive_doubling () from target:/cluster/mpi/mvapich2/3.0a/gcc11.3.1/lib/libmpi.so.12<br>
> #10 0x00007fd43dc50ca2 in MPIR_Allreduce_impl () from target:/cluster/mpi/mvapich2/3.0a/gcc11.3.1/lib/libmpi.so.12<br>
> #11 0x00007fd43e1993de in create_2level_comm () from target:/cluster/mpi/mvapich2/3.0a/gcc11.3.1/lib/libmpi.so.12<br>
> #12 0x00007fd43e24ea38 in MPIDI_MVP_post_init () from target:/cluster/mpi/mvapich2/3.0a/gcc11.3.1/lib/libmpi.so.12<br>
> #13 0x00007fd43e20fb40 in MPID_InitCompleted () from target:/cluster/mpi/mvapich2/3.0a/gcc11.3.1/lib/libmpi.so.12<br>
> #14 0x00007fd43dcb4d4f in MPIR_Init_thread () from target:/cluster/mpi/mvapich2/3.0a/gcc11.3.1/lib/libmpi.so.12<br>
> #15 0x00007fd43dcb4ae2 in PMPI_Init () from target:/cluster/mpi/mvapich2/3.0a/gcc11.3.1/lib/libmpi.so.12<br>
> #16 0x000000000040119d in main ()<br>
> <br>
> Best Regards<br>
> <br>
> Christof<br>
> <br>
> <br>
> -- <br>
> Dr. rer. nat. Christof Köhler email: c.koehler@uni-bremen.de<br>
> Universitaet Bremen/FB1/BCCMS phone: +49-(0)421-218-62334<br>
> Am Fallturm 1/ TAB/ Raum 3.06 fax: +49-(0)421-218-62770<br>
> 28359 Bremen <br>
<br>
<br>
<br>
> _______________________________________________<br>
> Mvapich-discuss mailing list<br>
> Mvapich-discuss@lists.osu.edu<br>
> <a href="https://lists.osu.edu/mailman/listinfo/mvapich-discuss">https://lists.osu.edu/mailman/listinfo/mvapich-discuss</a><br>
<br>
<br>
-- <br>
Dr. rer. nat. Christof Köhler email: c.koehler@uni-bremen.de<br>
Universitaet Bremen/FB1/BCCMS phone: +49-(0)421-218-62334<br>
Am Fallturm 1/ TAB/ Raum 3.06 fax: +49-(0)421-218-62770<br>
28359 Bremen <br>
</div>
</span></font></div>
</body>
</html>