<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:x="urn:schemas-microsoft-com:office:excel" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=us-ascii"><meta name=Generator content="Microsoft Word 15 (filtered medium)"><!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:"Arial Black";
panose-1:2 11 10 4 2 1 2 2 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.EmailStyle18
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:windowtext;}
span.EmailStyle20
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></head><body lang=EN-US link="#0563C1" vlink="#954F72"><div class=WordSection1><p class=MsoNormal>Dear Hari,<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>I noticed that state message to and thought it was odd as I was able to do pings and other network traffic over the interface, but I looked more into it and did realize that I hadn’t fully enabled PVRDMA on the hypervisor. I went ahead and added a vmkernel and firewall rules per VMware’s guidance and then restarted the interface in the VM and now the state shows PORT_ACTIVE (4). I also saw more details now in the output:<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'>hca_id: vmw_pvrdma0<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> transport: InfiniBand (0)<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> fw_ver: 3.0.000<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> node_guid: 0050:5600:009b:f2fe<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> sys_image_guid: 0000:0000:0000:0000<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> vendor_id: 0x15ad<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> vendor_part_id: 2080<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> hw_ver: 0x1<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> board_id: 1<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> phys_port_cnt: 1<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> max_mr_size: 0xffffffff<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> page_size_cap: 0xc<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> max_qp: 32768<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> max_qp_wr: 1024<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> device_cap_flags: 0x00201400<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> PORT_ACTIVE_EVENT<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> RC_RNR_NAK_GEN<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> MEM_MGT_EXTENSIONS<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> max_sge: 16<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> max_sge_rd: 16<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> max_cq: 4096<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> max_cqe: 262144<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> max_mr: 262144<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> max_pd: 4096<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> max_qp_rd_atom: 16<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> max_ee_rd_atom: 0<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> max_res_rd_atom: 0<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> max_qp_init_rd_atom: 128<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> max_ee_init_rd_atom: 0<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> atomic_cap: ATOMIC_NONE (0)<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> max_ee: 0<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> max_rdd: 0<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> max_mw: 0<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> max_raw_ipv6_qp: 0<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> max_raw_ethy_qp: 0<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> max_mcast_grp: 0<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> max_mcast_qp_attach: 0<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> max_total_mcast_qp_attach: 0<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> max_ah: 1048576<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> max_fmr: 0<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> max_srq: 4096<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> max_srq_wr: 1024<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> max_srq_sge: 16<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> max_pkeys: 128<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> local_ca_ack_delay: 5<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> general_odp_caps:<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> rc_odp_caps:<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> NO SUPPORT<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> uc_odp_caps:<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> NO SUPPORT<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> ud_odp_caps:<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> NO SUPPORT<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> xrc_odp_caps:<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> NO SUPPORT<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> completion_timestamp_mask not supported<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> core clock not supported<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> device_cap_flags_ex: 0x201400<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> tso_caps:<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> max_tso: 0<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> rss_caps:<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> max_rwq_indirection_tables: 0<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> max_rwq_indirection_table_size: 0<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> rx_hash_function: 0x0<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> rx_hash_fields_mask: 0x0<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> max_wq_type_rq: 0<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> packet_pacing_caps:<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> qp_rate_limit_min: 0kbps<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> qp_rate_limit_max: 0kbps<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> tag matching not supported<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> num_comp_vectors: 1<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> port: 1<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> state: PORT_ACTIVE (4)<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> max_mtu: 4096 (5)<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> active_mtu: 4096 (5)<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> sm_lid: 0<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> port_lid: 0<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> port_lmc: 0x00<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> link_layer: Ethernet<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> max_msg_sz: 0x7fffffff<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> port_cap_flags: 0x04010000<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> port_cap_flags2: 0x0000<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> max_vl_num: 1 (1)<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> bad_pkey_cntr: 0x0<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> qkey_viol_cntr: 0x0<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> sm_sl: 0<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> pkey_tbl_len: 1<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> gid_tbl_len: 6<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> subnet_timeout: 0<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> init_type_reply: 0<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> active_width: 1X (1)<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> active_speed: 2.5 Gbps (1)<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> phys_state: LINK_UP (5)<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> GID[ 0]: fe80::250:56ff:fe9b:f2fe, RoCE v2<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> GID[ 1]: ::ffff:192.168.0.148, RoCE v2<o:p></o:p></span></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>The MVAPICH2 error appears to be the same though:<o:p></o:p></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'>[host:mpi_rank_0][mv2_get_hca_type] **********************WARNING***********************<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'>[host:mpi_rank_0][mv2_get_hca_type] Failed to automatically detect the HCA architecture.<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'>[host:mpi_rank_0][mv2_get_hca_type] This may lead to subpar communication performance.<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'>[host:mpi_rank_0][mv2_get_hca_type] ****************************************************<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'>[host:mpi_rank_0][rdma_open_hca] Unknown HCA type: this build of MVAPICH2 does notfully support the HCA found on the system (try with other build options)<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'>[cli_1]: aborting job:<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'>Fatal error in MPI_Init:<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'>Other MPI error, error stack:<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'>MPIR_Init_thread(493)............:<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'>MPID_Init(419)...................: channel initialization failed<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'>MPIDI_CH3_Init(470)..............: rdma_get_control_parameters<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'>rdma_get_control_parameters(1925): rdma_open_hca<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'>rdma_open_hca(1080)..............: Failed to open HCA: No such file or directory<o:p></o:p></span></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Sincerely,<o:p></o:p></p><div><p class=MsoNormal><i><span style='font-size:12.0pt;font-family:"Arial",sans-serif;color:gray'><img width=82 height=32 style='width:.8541in;height:.3333in' id="_x0000_i1027" src="cid:image001.jpg@01D7E6CF.3F6B89F0" alt="Description: LL"></span></i><span style='font-size:12.0pt;font-family:"Times New Roman",serif;color:#1F497D'><o:p></o:p></span></p><p class=MsoNormal><i><span style='font-size:12.0pt;font-family:"Arial Black",sans-serif;color:gray'>Nicholas Clark</span></i><span style='font-size:12.0pt;font-family:"Times New Roman",serif;color:#1F497D'><o:p></o:p></span></p><p class=MsoNormal><span style='font-size:8.0pt;font-family:"Arial",sans-serif;color:gray'>MIT Lincoln Laboratory <o:p></o:p></span></p><p class=MsoNormal><span style='font-size:8.0pt;font-family:"Arial",sans-serif;color:gray'>ISR and Tactical Systems Division<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:8.0pt;font-family:"Arial",sans-serif;color:gray'>Embedded and Open Systems Group<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:8.0pt;font-family:"Arial",sans-serif;color:gray'>Systems Administration</span><span style='font-size:8.0pt;font-family:"Times New Roman",serif;color:#1F497D'><o:p></o:p></span></p><p class=MsoNormal><span style='font-size:8.0pt;font-family:"Arial",sans-serif;color:gray'>244 Wood St., S3-487</span><i><span style='font-family:"Arial",sans-serif;color:#1F497D'><o:p></o:p></span></i></p><p class=MsoNormal><span style='font-size:8.0pt;font-family:"Arial",sans-serif;color:gray'>Lexington, MA 02421-6426</span><span style='font-size:8.0pt;font-family:"Times New Roman",serif;color:#1F497D'><o:p></o:p></span></p><p class=MsoNormal><span style='font-size:8.0pt;font-family:"Arial",sans-serif;color:gray'>(O): 781-981-9342</span><span style='font-size:8.0pt;font-family:"Times New Roman",serif;color:#1F497D'><o:p></o:p></span></p><p class=MsoNormal><span style='font-size:8.0pt;font-family:"Arial",sans-serif;color:#1F497D'><a href="mailto:nicholas.clark@ll.mit.edu"><span style='color:blue'>nicholas.clark@ll.mit.edu</span></a></span><span style='font-size:8.0pt;font-family:"Times New Roman",serif;color:#1F497D'><o:p></o:p></span></p></div><p class=MsoNormal><o:p> </o:p></p><div><div style='border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in'><p class=MsoNormal><b>From:</b> Subramoni, Hari <subramoni.1@osu.edu> <br><b>Sent:</b> Wednesday, December 1, 2021 12:42 PM<br><b>To:</b> Clark, Nicholas - 1002 - MITLL <Nicholas.Clark@ll.mit.edu><br><b>Cc:</b> mvapich-discuss@lists.osu.edu; Subramoni, Hari <subramoni.1@osu.edu><br><b>Subject:</b> RE: [Mvapich-discuss] MVAPICH 2.3.6 HCA Support for VMXNET3 or PVRDMA VMware Network Adapters<o:p></o:p></p></div></div><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Hi, Nicholas.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>According the output, the port is down. That is probably why MVAPICH2 is complaining that I cannot find an appropriate HCA. Can you please bring it up and try again? <o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'>state: PORT_DOWN<o:p></o:p></span></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Thx,<o:p></o:p></p><p class=MsoNormal>Hari.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><div><div style='border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in'><p class=MsoNormal><b>From:</b> Clark, Nicholas - 1002 - MITLL <<a href="mailto:Nicholas.Clark@ll.mit.edu">Nicholas.Clark@ll.mit.edu</a>> <br><b>Sent:</b> Wednesday, December 1, 2021 11:19 AM<br><b>To:</b> Subramoni, Hari <<a href="mailto:subramoni.1@osu.edu">subramoni.1@osu.edu</a>><br><b>Cc:</b> <a href="mailto:mvapich-discuss@lists.osu.edu">mvapich-discuss@lists.osu.edu</a><br><b>Subject:</b> RE: [Mvapich-discuss] MVAPICH 2.3.6 HCA Support for VMXNET3 or PVRDMA VMware Network Adapters<o:p></o:p></p></div></div><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Dear Hari,<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>These are the results of ibv_devinfo:<o:p></o:p></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'>hca_id: vmw_pvrdma0<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> transport: InfiniBand (0)<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> fw_ver: 3.0.000<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> node_guid: 0050:5600:009b:71e5<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> sys_image_guid: 0000:0000:0000:0000<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> vendor_id: 0x15ad<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> vendor_part_id: 2080<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> hw_ver: 0x1<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> board_id: 1<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> phys_port_cnt: 1<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> max_mr_size: 0xffffffff<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> page_size_cap: 0xc<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> max_qp: 32768<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> max_qp_wr: 1024<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> device_cap_flags: 0x00201400<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> PORT_ACTIVE_EVENT<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> RC_RNR_NAK_GEN<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> MEM_MGT_EXTENSIONS<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> max_sge: 16<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> max_sge_rd: 16<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> max_cq: 4096<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> max_cqe: 262144<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> max_mr: 262144<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> max_pd: 4096<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> max_qp_rd_atom: 16<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> max_ee_rd_atom: 0<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> max_res_rd_atom: 0<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> max_qp_init_rd_atom: 128<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> max_ee_init_rd_atom: 0<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> atomic_cap: ATOMIC_NONE (0)<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> max_ee: 0<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> max_rdd: 0<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> max_mw: 0<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> max_raw_ipv6_qp: 0<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> max_raw_ethy_qp: 0<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> max_mcast_grp: 0<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> max_mcast_qp_attach: 0<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> max_total_mcast_qp_attach: 0<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> max_ah: 1048576<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> max_fmr: 0<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> max_srq: 4096<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> max_srq_wr: 1024<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> max_srq_sge: 16<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> max_pkeys: 128<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> local_ca_ack_delay: 5<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> general_odp_caps:<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> rc_odp_caps:<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> NO SUPPORT<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> uc_odp_caps:<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> NO SUPPORT<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> ud_odp_caps:<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> NO SUPPORT<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> xrc_odp_caps:<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> NO SUPPORT<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> completion_timestamp_mask not supported<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> core clock not supported<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> device_cap_flags_ex: 0x201400<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> tso_caps:<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> max_tso: 0<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> rss_caps:<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> max_rwq_indirection_tables: 0<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> max_rwq_indirection_table_size: 0<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> rx_hash_function: 0x0<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> rx_hash_fields_mask: 0x0<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> max_wq_type_rq: 0<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> packet_pacing_caps:<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> qp_rate_limit_min: 0kbps<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> qp_rate_limit_max: 0kbps<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> tag matching not supported<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> num_comp_vectors: 1<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> port: 1<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> state: PORT_DOWN (1)<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> max_mtu: 4096 (5)<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> active_mtu: 4096 (5)<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> sm_lid: 0<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> port_lid: 0<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> port_lmc: 0x00<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> link_layer: Ethernet<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> max_msg_sz: 0x7fffffff<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> port_cap_flags: 0x04010000<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> port_cap_flags2: 0x0000<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> max_vl_num: 1 (1)<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> bad_pkey_cntr: 0x0<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> qkey_viol_cntr: 0x0<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> sm_sl: 0<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> pkey_tbl_len: 1<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> gid_tbl_len: 6<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> subnet_timeout: 0<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> init_type_reply: 0<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> active_width: 1X (1)<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> active_speed: 2.5 Gbps (1)<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> phys_state: LINK_UP (5)<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'> GID[ 0]: fe80::250:56ff:fe9b:71e5, RoCE v2<o:p></o:p></span></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>I am not able to grant remote access to the VM.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Sincerely,<o:p></o:p></p><div><p class=MsoNormal><i><span style='font-size:12.0pt;font-family:"Arial",sans-serif;color:gray'><img border=0 width=82 height=32 style='width:.8541in;height:.3333in' id="_x0000_i1025" src="cid:image001.jpg@01D7E6CF.3F6B89F0" alt="Description: LL"></span></i><span style='font-size:12.0pt;font-family:"Times New Roman",serif;color:#1F497D'><o:p></o:p></span></p><p class=MsoNormal><i><span style='font-size:12.0pt;font-family:"Arial Black",sans-serif;color:gray'>Nicholas Clark</span></i><span style='font-size:12.0pt;font-family:"Times New Roman",serif;color:#1F497D'><o:p></o:p></span></p><p class=MsoNormal><span style='font-size:8.0pt;font-family:"Arial",sans-serif;color:gray'>MIT Lincoln Laboratory <o:p></o:p></span></p><p class=MsoNormal><span style='font-size:8.0pt;font-family:"Arial",sans-serif;color:gray'>ISR and Tactical Systems Division<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:8.0pt;font-family:"Arial",sans-serif;color:gray'>Embedded and Open Systems Group<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:8.0pt;font-family:"Arial",sans-serif;color:gray'>Systems Administration</span><span style='font-size:8.0pt;font-family:"Times New Roman",serif;color:#1F497D'><o:p></o:p></span></p><p class=MsoNormal><span style='font-size:8.0pt;font-family:"Arial",sans-serif;color:gray'>244 Wood St., S3-487</span><i><span style='font-family:"Arial",sans-serif;color:gray'><o:p></o:p></span></i></p><p class=MsoNormal><span style='font-size:8.0pt;font-family:"Arial",sans-serif;color:gray'>Lexington, MA 02421-6426</span><span style='font-size:8.0pt;font-family:"Times New Roman",serif;color:#1F497D'><o:p></o:p></span></p><p class=MsoNormal><span style='font-size:8.0pt;font-family:"Arial",sans-serif;color:gray'>(O): 781-981-9342</span><span style='font-size:8.0pt;font-family:"Times New Roman",serif;color:#1F497D'><o:p></o:p></span></p><p class=MsoNormal><span style='font-size:8.0pt;font-family:"Arial",sans-serif;color:#1F497D'><a href="mailto:nicholas.clark@ll.mit.edu"><span style='color:blue'>nicholas.clark@ll.mit.edu</span></a></span><span style='font-size:8.0pt;font-family:"Times New Roman",serif;color:#1F497D'><o:p></o:p></span></p></div><p class=MsoNormal><o:p> </o:p></p><div><div style='border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in'><p class=MsoNormal><b>From:</b> Subramoni, Hari <<a href="mailto:subramoni.1@osu.edu">subramoni.1@osu.edu</a>> <br><b>Sent:</b> Wednesday, December 1, 2021 11:12 AM<br><b>To:</b> Clark, Nicholas - 1002 - MITLL <<a href="mailto:Nicholas.Clark@ll.mit.edu">Nicholas.Clark@ll.mit.edu</a>><br><b>Cc:</b> <a href="mailto:mvapich-discuss@lists.osu.edu">mvapich-discuss@lists.osu.edu</a>; Subramoni, Hari <<a href="mailto:subramoni.1@osu.edu">subramoni.1@osu.edu</a>><br><b>Subject:</b> RE: [Mvapich-discuss] MVAPICH 2.3.6 HCA Support for VMXNET3 or PVRDMA VMware Network Adapters<o:p></o:p></p></div></div><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Hi, Nicholas.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>MVAPICH2 has support for running in VMs.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Could you please send us the output of ibv_devinfo -v on the VM? Do you think it is possible to get temporary remote access to debug the problem ourselves?<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Best,<o:p></o:p></p><p class=MsoNormal>Hari.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><div><div style='border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in'><p class=MsoNormal><b>From:</b> Mvapich-discuss <<a href="mailto:mvapich-discuss-bounces@lists.osu.edu">mvapich-discuss-bounces@lists.osu.edu</a>> <b>On Behalf Of </b>Clark, Nicholas - 1002 - MITLL via Mvapich-discuss<br><b>Sent:</b> Tuesday, November 30, 2021 4:49 PM<br><b>To:</b> <a href="mailto:mvapich-discuss@lists.osu.edu">mvapich-discuss@lists.osu.edu</a><br><b>Subject:</b> [Mvapich-discuss] MVAPICH 2.3.6 HCA Support for VMXNET3 or PVRDMA VMware Network Adapters<o:p></o:p></p></div></div><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Does MVPAICH2 have support for running in VMs with VMXNET3 or the PVRDMA network adapter that supports RoCE v1/v2?<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Currently with the default build parameters and native rdma-core libraries on RHEL 8.5, I am seeing this message about unknown HCA on both VMXNET3 and PVRDMA:<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>[rdma_open_hca] Unknown HCA type: this build of MVAPICH2 does notfully support the HCA found on the system (try with other build options)<o:p></o:p></p><p class=MsoNormal>[cli_1]: aborting job:<o:p></o:p></p><p class=MsoNormal>Fatal error in MPI_Init:<o:p></o:p></p><p class=MsoNormal>Other MPI error, error stack:<o:p></o:p></p><p class=MsoNormal>MPIR_Init_thread(493)............:<o:p></o:p></p><p class=MsoNormal>MPID_Init(419)...................: channel initialization failed<o:p></o:p></p><p class=MsoNormal>MPIDI_CH3_Init(470)..............: rdma_get_control_parameters<o:p></o:p></p><p class=MsoNormal>rdma_get_control_parameters(1925): rdma_open_hca<o:p></o:p></p><p class=MsoNormal>rdma_open_hca(1080)..............: Failed to open HCA: No such file or directory<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Sincerely,<o:p></o:p></p><p class=MsoNormal><i><span style='font-size:12.0pt;font-family:"Arial",sans-serif;color:gray'><img border=0 width=82 height=32 style='width:.8541in;height:.3333in' id="Picture_x0020_1" src="cid:image001.jpg@01D7E6CF.3F6B89F0" alt="Description: LL"></span></i><span style='font-size:12.0pt;font-family:"Times New Roman",serif;color:#1F497D'><o:p></o:p></span></p><p class=MsoNormal><i><span style='font-size:12.0pt;font-family:"Arial Black",sans-serif;color:gray'>Nicholas Clark</span></i><span style='font-size:12.0pt;font-family:"Times New Roman",serif;color:#1F497D'><o:p></o:p></span></p><p class=MsoNormal><span style='font-size:8.0pt;font-family:"Arial",sans-serif;color:gray'>MIT Lincoln Laboratory <o:p></o:p></span></p><p class=MsoNormal><span style='font-size:8.0pt;font-family:"Arial",sans-serif;color:gray'>ISR and Tactical Systems Division<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:8.0pt;font-family:"Arial",sans-serif;color:gray'>Embedded and Open Systems Group<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:8.0pt;font-family:"Arial",sans-serif;color:gray'>Systems Administration</span><span style='font-size:8.0pt;font-family:"Times New Roman",serif;color:#1F497D'><o:p></o:p></span></p><p class=MsoNormal><span style='font-size:8.0pt;font-family:"Arial",sans-serif;color:gray'>244 Wood St., S3-487</span><i><span style='font-family:"Arial",sans-serif;color:gray'><o:p></o:p></span></i></p><p class=MsoNormal><span style='font-size:8.0pt;font-family:"Arial",sans-serif;color:gray'>Lexington, MA 02421-6426</span><span style='font-size:8.0pt;font-family:"Times New Roman",serif;color:#1F497D'><o:p></o:p></span></p><p class=MsoNormal><span style='font-size:8.0pt;font-family:"Arial",sans-serif;color:gray'>(O): 781-981-9342</span><span style='font-size:8.0pt;font-family:"Times New Roman",serif;color:#1F497D'><o:p></o:p></span></p><p class=MsoNormal><span style='font-size:8.0pt;font-family:"Arial",sans-serif;color:#1F497D'><a href="mailto:nicholas.clark@ll.mit.edu"><span style='color:blue'>nicholas.clark@ll.mit.edu</span></a></span><span style='font-size:8.0pt;font-family:"Times New Roman",serif;color:#1F497D'><o:p></o:p></span></p><p class=MsoNormal><o:p> </o:p></p></div></body></html>