<div dir="ltr"><div>As far as I can tell, all the IB ports have IPoIB addresses assigned. I will try with USE_RDMA_CM=0. A rerun without change change got a different error, I think I was seeing this kind of error last summer and fall, but it went away.<br></div><div><br></div><div>mlx5: host9: got completion with error:<br>00000000 00000000 00000000 00000000<br>00000000 00000000 00000000 00000000<br>00000006 00000000 00000000 00000000<br>00000000 12006802 000039a6 0210c3d2<br>[host9:mpi_rank_6][handle_cqe] Send desc error in msg to 10, wc_opcode=0<br>[host9:mpi_rank_6][handle_cqe] Msg from 10: wc.status=2 (local QP operation error), wc.wr_id=0xc21fcc0, wc.opcode=0, vbuf->phead->type=32 = MPIDI_CH3_PKT_RNDV_REQ_TO_SEND<br>[host9:mpi_rank_6][mv2_print_wc_status_error] IBV_WC_LOC_QP_OP_ERR: This event is generated when a QP error occurs. For example, it may be generated if a) user neglects to specify responder_resources and initiator_depth values in struct rdma_conn_param before calling rdma_connect() on the client side and rdma_accept() on the server side, b) a Work Request that was posted in a local Send Queue of a UD QP contains an Address Handle that is associated with a Protection Domain to a QP which is associated with a different Protection Domain, or c) an opcode which is not supported by the transport type of the QP is not supported (for example: RDMA Write over a UD QP).<br>[host9:mpi_rank_6][handle_cqe] src/mpid/ch3/channels/mrail/src/gen2/ibv_channel_manager.c:499: [] Got completion with error 2, vendor code=0x68, dest rank=10<br></div><div><br></div><div><div><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><br>.. Lana (<a href="mailto:lana.deere@gmail.com" target="_blank">lana.deere@gmail.com</a>)<br><br><br></div></div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Jul 28, 2021 at 7:46 AM Subramoni, Hari <<a href="mailto:subramoni.1@osu.edu">subramoni.1@osu.edu</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div style="overflow-wrap: break-word;" lang="EN-US">
<div class="gmail-m_-314040701339726314WordSection1">
<p class="MsoNormal">Hi, Lana.<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">It looks like IP addresses were not assigned to all the IB ports.<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">As a workaround, can you please set MV2_USE_RDMA_CM=0 and try?<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">Thx,<u></u><u></u></p>
<p class="MsoNormal">Hari.<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">PS: Please try and move to MVAPICH2 2.3.6. It has a lot of fixes and performance enhancements compared to the 2.3.5 release.<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<div style="border-color:rgb(225,225,225) currentcolor currentcolor;border-style:solid none none;border-width:1pt medium medium;padding:3pt 0in 0in">
<p class="MsoNormal"><b>From:</b> Mvapich-discuss <<a href="mailto:mvapich-discuss-bounces@lists.osu.edu" target="_blank">mvapich-discuss-bounces@lists.osu.edu</a>>
<b>On Behalf Of </b>Lana Deere via Mvapich-discuss<br>
<b>Sent:</b> Tuesday, July 27, 2021 6:24 PM<br>
<b>To:</b> <a href="mailto:mvapich-discuss@lists.osu.edu" target="_blank">mvapich-discuss@lists.osu.edu</a><br>
<b>Subject:</b> [Mvapich-discuss] RDMA CM messages<u></u><u></u></p>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<div>
<p class="MsoNormal">I'm using mvapich2.3.5 on CentOS 7.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">I've got an MPI job which is failing intermittently. One of the failure symptoms is a hang in MPI_InitThread, with this traceback:<u></u><u></u></p>
</div>
<div style="margin-left:30pt">
<p class="MsoNormal"><span style="font-family:"Courier New"">/lib64/libpthread.so.0 read<br>
libmpi.so.12 PMIU_readline<br>
libmpi.so.12 <br>
libmpi.so.12 UPMI_BARRIER<br>
libmpi.so.12 rdma_cm_exchange_hostid<br>
libmpi.so.12 MPIDI_CH3I_RDMA_CM_Init<br>
libmpi.so.12 MPIDI_CH3_Init<br>
libmpi.so.12 MPID_Init<br>
libmpi.so.12 MPIR_Init_thread<br>
libmpi.so.12 MPI_Init_thread</span><u></u><u></u></p>
</div>
<div>
<div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">A run which didn't fail produced this warning:<u></u><u></u></p>
</div>
<div style="margin-left:30pt">
<p class="MsoNormal"><span style="font-family:"Courier New"">Warning: RDMA CM Initialization failed. Continuing without RDMA CM support. Please set MV2_USE_RDMA_CM=0 to disable RDMA CM.</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Does anyone have advice on tracking this down? Does it suggest a software issue? An infiniband hardware issue?<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Thanks.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12pt"><br>
.. Lana (<a href="mailto:lana.deere@gmail.com" target="_blank">lana.deere@gmail.com</a>)<br>
<br>
<u></u><u></u></p>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote></div>