[mvapich-discuss] problems with MVAPICH2 over 10GbE
Jonathan Perkins
perkinjo at cse.ohio-state.edu
Tue Sep 27 22:47:58 EDT 2011
This message is to report the Jeff's problem has been resolved offline
by a firmware upgrade for his HCAs in combination with a patch that
has been included in MVAPICH2-1.7rc2.
On Fri, Aug 19, 2011 at 10:04 AM, Konz, Jeffrey (SSA Solution Centers)
<jeffrey.konz at hp.com> wrote:
> Jonathan,
>
> Unfortunately no success with the patch. IB works fine but 10G fails.
>
> I get this error:
>
> Fatal error in MPI_Init:
> Internal MPI error!
>
> [atl3-11:mpispawn_0][readline] Unexpected End-Of-File on file descriptor 6. MPI process died?
> [atl3-11:mpispawn_0][mtpmi_processops] Error while reading PMI socket. MPI process died?
> [atl3-11:mpispawn_0][child_handler] MPI process (rank: 0, pid: 31650) exited with status 1
> [atl3-11:mpirun_rsh][process_mpispawn_connection] mpispawn_0 from node atl3-11 aborted: Error while reading a PMI socket (4)
> Fatal error in MPI_Init:
> Internal MPI error!
>
> [atl3-12:mpispawn_1][readline] Unexpected End-Of-File on file descriptor 5. MPI process died?
> [atl3-12:mpispawn_1][mtpmi_processops] Error while reading PMI socket. MPI process died?
> [atl3-12:mpispawn_1][child_handler] MPI process (rank: 1, pid: 30919) exited with status 1
>
> My command is:
> mpirun_rsh -ssh -np 2 -hostfile hosts MV2_USE_RDMAOE=1 MV2_DEFAULT_PORT=2 ./osu_bw
>
>
> -Jeff
>
>> -----Original Message-----
>> From: Jonathan Perkins [mailto:perkinjo at cse.ohio-state.edu]
>> Sent: Friday, August 19, 2011 9:04 AM
>> To: Konz, Jeffrey (SSA Solution Centers)
>> Cc: mvapich-discuss at cse.ohio-state.edu
>> Subject: Re: [mvapich-discuss] problems with MVAPICH2 over 10GbE
>>
>> This is fine, you actually have explicitly specified a few options
>> that are enabled by
>> default. This list of options could be shortened to `--enable-shared
>> --enable-xrc'.
>>
>> On Fri, Aug 19, 2011 at 8:59 AM, Konz, Jeffrey (SSA Solution Centers)
>> <jeffrey.konz at hp.com> wrote:
>> > Jonathan,
>> >
>> > Got the patch. I want to verify the build options.
>> >
>> > My complete list of options is
>> > "--enable-shared --enable-sharedlibs=gcc --enable-romio --enable-f77
>> --enable-fc --with-device=ch3:mrail --with-rdma=gen2 --enable-xrc".
>> >
>> > Are these correct? Anything else I should enable?
>> >
>> > Thanks,
>> >
>> > -Jeff
>> >
>> >
>> >
>> >> -----Original Message-----
>> >> From: Jonathan Perkins [mailto:perkinjo at cse.ohio-state.edu]
>> >> Sent: Thursday, August 18, 2011 9:11 PM
>> >> To: Konz, Jeffrey (SSA Solution Centers)
>> >> Cc: mvapich-discuss at cse.ohio-state.edu
>> >> Subject: Re: [mvapich-discuss] problems with MVAPICH2 over 10GbE
>> >>
>> >> Hello Jeffrey, we looked into this a bit more. After applying the
>> >> attached patch and rebuilding mvapich2 you can use a variable called
>> >> MV2_DEFAULT_PORT to control which port to use.
>> >>
>> >> First apply the patch using `patch -p0 < multi_port_roce_patch.txt'
>> at
>> >> the top level of the mvapich2 source code and rebuild the library.
>> To
>> >> run in RDMAoE mode on the 2nd port do something like:
>> >>
>> >> mpirun_rsh -np 2 node1 node2 MV2_USE_RDMAOE=1 MV2_DEFAULT_PORT=2
>> ./prog
>> >>
>> >> Please let us know if this works for you.
>> >>
>> >> On Wed, Aug 17, 2011 at 4:48 PM, Konz, Jeffrey (SSA Solution
>> Centers)
>> >> <jeffrey.konz at hp.com> wrote:
>> >> > Jonathan,
>> >> >
>> >> > An issue is selecting the right port on the Mellanox NIC, it has
>> two
>> >> ports: 1 IB, 1 10GigE.
>> >> > Not sure how to do that.
>> >> >
>> >> > #ibstat
>> >> > CA 'mlx4_0'
>> >> > CA type: MT26438
>> >> > Number of ports: 2
>> >> > Firmware version: 2.7.9100
>> >> > Hardware version: b0
>> >> > Node GUID: 0x78e7d10300214bbc
>> >> > System image GUID: 0x78e7d10300214bbf
>> >> > Port 1:
>> >> > State: Active
>> >> > Physical state: LinkUp
>> >> > Rate: 40
>> >> > Base lid: 13
>> >> > LMC: 0
>> >> > SM lid: 1
>> >> > Capability mask: 0x02510868
>> >> > Port GUID: 0x78e7d10300214bbd
>> >> > Link layer: IB
>> >> > Port 2:
>> >> > State: Active
>> >> > Physical state: LinkUp
>> >> > Rate: 40
>> >> > Base lid: 0
>> >> > LMC: 0
>> >> > SM lid: 0
>> >> > Capability mask: 0x00010000
>> >> > Port GUID: 0x7ae7d1fffe214bbd
>> >> > Link layer: Ethernet
>> >> >
>> >> >
>> >> > #ibv_devinfo -v
>> >> > hca_id: mlx4_0
>> >> > transport: InfiniBand (0)
>> >> > fw_ver: 2.7.9100
>> >> > node_guid: 78e7:d103:0021:4bbc
>> >> > sys_image_guid: 78e7:d103:0021:4bbf
>> >> > vendor_id: 0x02c9
>> >> > vendor_part_id: 26438
>> >> > hw_ver: 0xB0
>> >> > board_id: HP_0200000003
>> >> > phys_port_cnt: 2
>> >> > max_mr_size: 0xffffffffffffffff
>> >> > page_size_cap: 0xfffffe00
>> >> > max_qp: 260032
>> >> > max_qp_wr: 16351
>> >> > device_cap_flags: 0x007c9c76
>> >> > max_sge: 32
>> >> > max_sge_rd: 0
>> >> > max_cq: 65408
>> >> > max_cqe: 4194303
>> >> > max_mr: 524272
>> >> > max_pd: 32764
>> >> > max_qp_rd_atom: 16
>> >> > max_ee_rd_atom: 0
>> >> > max_res_rd_atom: 4160512
>> >> > max_qp_init_rd_atom: 128
>> >> > max_ee_init_rd_atom: 0
>> >> > atomic_cap: ATOMIC_HCA (1)
>> >> > max_ee: 0
>> >> > max_rdd: 0
>> >> > max_mw: 0
>> >> > max_raw_ipv6_qp: 0
>> >> > max_raw_ethy_qp: 2
>> >> > max_mcast_grp: 8192
>> >> > max_mcast_qp_attach: 56
>> >> > max_total_mcast_qp_attach: 458752
>> >> > max_ah: 0
>> >> > max_fmr: 0
>> >> > max_srq: 65472
>> >> > max_srq_wr: 16383
>> >> > max_srq_sge: 31
>> >> > max_pkeys: 128
>> >> > local_ca_ack_delay: 15
>> >> > port: 1
>> >> > state: PORT_ACTIVE (4)
>> >> > max_mtu: 2048 (4)
>> >> > active_mtu: 2048 (4)
>> >> > sm_lid: 1
>> >> > port_lid: 13
>> >> > port_lmc: 0x00
>> >> > link_layer: IB
>> >> > max_msg_sz: 0x40000000
>> >> > port_cap_flags: 0x02510868
>> >> > max_vl_num: 8 (4)
>> >> > bad_pkey_cntr: 0x0
>> >> > qkey_viol_cntr: 0x0
>> >> > sm_sl: 0
>> >> > pkey_tbl_len: 128
>> >> > gid_tbl_len: 128
>> >> > subnet_timeout: 18
>> >> > init_type_reply: 0
>> >> > active_width: 4X (2)
>> >> > active_speed: 10.0 Gbps (4)
>> >> > phys_state: LINK_UP (5)
>> >> > GID[ 0]:
>> >> > fe80:0000:0000:0000:78e7:d103:0021:4bbd
>> >> >
>> >> > port: 2
>> >> > state: PORT_ACTIVE (4)
>> >> > max_mtu: 2048 (4)
>> >> > active_mtu: 1024 (3)
>> >> > sm_lid: 0
>> >> > port_lid: 0
>> >> > port_lmc: 0x00
>> >> > link_layer: Ethernet
>> >> > max_msg_sz: 0x40000000
>> >> > port_cap_flags: 0x00010000
>> >> > max_vl_num: 8 (4)
>> >> > bad_pkey_cntr: 0x0
>> >> > qkey_viol_cntr: 0x0
>> >> > sm_sl: 0
>> >> > pkey_tbl_len: 1
>> >> > gid_tbl_len: 128
>> >> > subnet_timeout: 0
>> >> > init_type_reply: 0
>> >> > active_width: 4X (2)
>> >> > active_speed: 10.0 Gbps (4)
>> >> > phys_state: LINK_UP (5)
>> >> > GID[ 0]:
>> >> > fe80:0000:0000:0000:7ae7:d1ff:fe21:4bbd
>> >> >
>> >> >
>> >> > -Jeff
>> >> >
>> >> >
>> >> >> -----Original Message-----
>> >> >> From: Jonathan Perkins [mailto:perkinjo at cse.ohio-state.edu]
>> >> >> Sent: Wednesday, August 17, 2011 12:25 PM
>> >> >> To: Konz, Jeffrey (SSA Solution Centers)
>> >> >> Cc: mvapich-discuss at cse.ohio-state.edu
>> >> >> Subject: Re: [mvapich-discuss] problems with MVAPICH2 over 10GbE
>> >> >>
>> >> >> Thanks for your report, I'm checking with some of the other
>> >> >> developers to verify the way this should work. I believe that
>> you
>> >> do
>> >> >> not need to use the IP addresses of the RDMAoE port but instead
>> >> >> specify the the HCA name using MV2_IBA_HCA in addition to the
>> >> MV2_USE_RDMAOE=1 option.
>> >> >>
>> >> >> The name of the HCA can be found by using the ibstat command and
>> >> >> should look something like mlx4_...
>> >> >>
>> >> >> On Wed, Aug 17, 2011 at 11:17 AM, Konz, Jeffrey (SSA Solution
>> >> >> Centers) <jeffrey.konz at hp.com> wrote:
>> >> >> > I am running on a cluster with the Mellanox LOM that supports
>> both
>> >> >> > IB
>> >> >> and 10 GbE.
>> >> >> > Both ports on the interface are active, one is on IB network
>> the
>> >> >> other on 10 GbE network.
>> >> >> >
>> >> >> > I built mvapich2-1.7rc1 with these options :
>> >> >> > "--with-device=ch3:mrail
>> >> >> --with-rdma=gen2"
>> >> >> >
>> >> >> > Running over IB works fine.
>> >> >> >
>> >> >> > When I try to run over the 10GbE network with the
>> >> "MV2_USE_RDMAOE=1"
>> >> >> option I get this error:
>> >> >> >
>> >> >> > Fatal error in MPI_Init:
>> >> >> > Internal MPI error!
>> >> >> >
>> >> >> > [atl3-13:mpispawn_0][readline] Unexpected End-Of-File on file
>> >> >> descriptor 5. MPI process died?
>> >> >> > [atl3-13:mpispawn_0][mtpmi_processops] Error while reading PMI
>> >> >> socket. MPI process died?
>> >> >> > [atl3-13:mpispawn_0][child_handler] MPI process (rank: 0, pid:
>> >> >> > 23500)
>> >> >> exited with status 1
>> >> >> > [atl3-13:mpirun_rsh][process_mpispawn_connection] mpispawn_0
>> from
>> >> >> node 10.10.0.149 aborted: Error while reading a PMI socket (4)
>> >> >> >
>> >> >> > In the hostfile I specified the IP addresses of the 10 GbE
>> ports.
>> >> >> >
>> >> >> > I am running incorrectly or have I not built mvapich with the
>> >> >> > correct
>> >> >> options?
>> >> >> >
>> >> >> > Thanks,
>> >> >> >
>> >> >> > -Jeff
>> >> >> >
>> >> >> >
>> >> >> > _______________________________________________
>> >> >> > mvapich-discuss mailing list
>> >> >> > mvapich-discuss at cse.ohio-state.edu
>> >> >> > http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss
>> >> >> >
>> >> >> >
>> >> >>
>> >> >>
>> >> >>
>> >> >> --
>> >> >> Jonathan Perkins
>> >> >> http://www.cse.ohio-state.edu/~perkinjo
>> >> >
>> >> >
>> >>
>> >>
>> >>
>> >> --
>> >> Jonathan Perkins
>> >> http://www.cse.ohio-state.edu/~perkinjo
>> >
>> >
>>
>>
>>
>> --
>> Jonathan Perkins
>> http://www.cse.ohio-state.edu/~perkinjo
>
>
--
Jonathan Perkins
http://www.cse.ohio-state.edu/~perkinjo
More information about the mvapich-discuss
mailing list