<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:"Avenir Next LT Pro";
panose-1:2 11 5 4 2 2 2 2 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
font-size:12.0pt;
font-family:"Calibri",sans-serif;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:12.0pt;
font-family:"Calibri",sans-serif;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style>
</head>
<body lang="EN-CA" link="#0563C1" vlink="#954F72" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt">The problem starts when testing the system and configuring cluster Ethernet MTU to 1500, we start getting the MPICM_Init_UD_CM error as seen below. I did revert the cluster MTU back to 9000 and
confirm everything is working just fine as before. Then I did try controlling the default MTU (as stated in the message below) to 4000 and expecting the effective RDMA MTU been 2048 but I still get this error even if the specified MTU is way above the limit.
It seems we do specify the default MTU it will not work properly.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt">Is my understanding of the “MV2_DEFAULT_MTU” correct as this will dictate the maximum System MTU and the rdma_default_mtu should be the maximum RDMA MTU fitting within the default MTU.
<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt">[user@dell-s13-h1 ~]$ export MV2_DEFAULT_MTU=4000<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt">[user@dell-s13-h1 ~]$ /opt/bm/hpc/mvapich2-2.3.7pre-rockportqos/bin/mpiexec -np 1504 -f /rockshare/ngagnon/dummy/intel-mpi-2021-09-30-1635/i_1024/rpn_32/syseng_48/mvapich-host.cfg -env MV2_HOMOGENEOUS_CLUSTER=1
-env MV2_HYBRID_ENABLE_THRESHOLD=102400 -env MV2_NDREG_ENTRIES_MAX=100000 -env MV2_NDREG_ENTRIES=50000 /opt/bm/hpc/mvapich2-2.3.7pre-rockportqos/libexec/osu-micro-benchmarks/mpi/collective/osu_alltoall -f -i 100 -m:4<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt">[dell-s13-h2:mpi_rank_0][MPICM_Init_UD_CM] sizeof cm_msg (1296) >= rdma_default_mtu (1024).<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt">[dell-s13-h2:mpi_rank_0][MPICM_Init_UD_CM] Try increasing the MV2_DEFAULT_MTU or reduce MAX_NUM_HCAS, or MAX_NUM_QP_PER_PORT in ibv_param.h.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt">[cli_0]: aborting job:<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt">Fatal error in MPI_Init:<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt">Other MPI error, error stack:<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt">MPIR_Init_thread(493)...:
<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt">MPID_Init(419)..........: channel initialization failed<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt">MPIDI_CH3_Init(581).....:
<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt">MPIDI_CH3I_CM_Init(2054):
<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt">MPICM_Init_UD_CM(2092)..:
<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt">(unknown)(): Other MPI error<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><b><span style="font-size:8.0pt;font-family:"Avenir Next LT Pro",sans-serif">Nicolas Gagnon<o:p></o:p></span></b></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Avenir Next LT Pro",sans-serif">Principal Designer/Architect, Engineering<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Avenir Next LT Pro",sans-serif"><a href="mailto:ngagnon@rockportnetworks.com"><span style="color:#0563C1">ngagnon@rockportnetworks.com</span></a>
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Avenir Next LT Pro",sans-serif">Rockport | Simplify the Network<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Avenir Next LT Pro",sans-serif"><o:p> </o:p></span></p>
<div>
<p class="MsoNormal"><span lang="EN-US"><a href="https://urldefense.com/v3/__https://rockportnetworks.com/__;!!KGKeukY!kkq30nk5jPD0xGrOHiBz5eNLHzw1gY3tmocSpShpof2vjrMd6nC1lWhO4Tk2Pi-0rtaZTlJLEw$"><span lang="EN-US" style="font-size:11.0pt;color:windowtext;text-decoration:none"><img border="0" width="135" height="29" style="width:1.4062in;height:.302in" id="Picture_x0020_1" src="cid:image001.png@01D7BC2E.771A5190" alt="signature_1389006283"></span></a></span><span lang="EN-US" style="font-size:11.0pt"><o:p></o:p></span></p>
</div>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
</div>
</body>
</html>