[mvapich-discuss] problems with MVAPICH2 over 10GbE
Konz, Jeffrey (SSA Solution Centers)
jeffrey.konz at hp.com
Fri Aug 19 10:04:52 EDT 2011
Jonathan,
Unfortunately no success with the patch. IB works fine but 10G fails.
I get this error:
Fatal error in MPI_Init:
Internal MPI error!
[atl3-11:mpispawn_0][readline] Unexpected End-Of-File on file descriptor 6. MPI process died?
[atl3-11:mpispawn_0][mtpmi_processops] Error while reading PMI socket. MPI process died?
[atl3-11:mpispawn_0][child_handler] MPI process (rank: 0, pid: 31650) exited with status 1
[atl3-11:mpirun_rsh][process_mpispawn_connection] mpispawn_0 from node atl3-11 aborted: Error while reading a PMI socket (4)
Fatal error in MPI_Init:
Internal MPI error!
[atl3-12:mpispawn_1][readline] Unexpected End-Of-File on file descriptor 5. MPI process died?
[atl3-12:mpispawn_1][mtpmi_processops] Error while reading PMI socket. MPI process died?
[atl3-12:mpispawn_1][child_handler] MPI process (rank: 1, pid: 30919) exited with status 1
My command is:
mpirun_rsh -ssh -np 2 -hostfile hosts MV2_USE_RDMAOE=1 MV2_DEFAULT_PORT=2 ./osu_bw
-Jeff
> -----Original Message-----
> From: Jonathan Perkins [mailto:perkinjo at cse.ohio-state.edu]
> Sent: Friday, August 19, 2011 9:04 AM
> To: Konz, Jeffrey (SSA Solution Centers)
> Cc: mvapich-discuss at cse.ohio-state.edu
> Subject: Re: [mvapich-discuss] problems with MVAPICH2 over 10GbE
>
> This is fine, you actually have explicitly specified a few options
> that are enabled by
> default. This list of options could be shortened to `--enable-shared
> --enable-xrc'.
>
> On Fri, Aug 19, 2011 at 8:59 AM, Konz, Jeffrey (SSA Solution Centers)
> <jeffrey.konz at hp.com> wrote:
> > Jonathan,
> >
> > Got the patch. I want to verify the build options.
> >
> > My complete list of options is
> > "--enable-shared --enable-sharedlibs=gcc --enable-romio --enable-f77
> --enable-fc --with-device=ch3:mrail --with-rdma=gen2 --enable-xrc".
> >
> > Are these correct? Anything else I should enable?
> >
> > Thanks,
> >
> > -Jeff
> >
> >
> >
> >> -----Original Message-----
> >> From: Jonathan Perkins [mailto:perkinjo at cse.ohio-state.edu]
> >> Sent: Thursday, August 18, 2011 9:11 PM
> >> To: Konz, Jeffrey (SSA Solution Centers)
> >> Cc: mvapich-discuss at cse.ohio-state.edu
> >> Subject: Re: [mvapich-discuss] problems with MVAPICH2 over 10GbE
> >>
> >> Hello Jeffrey, we looked into this a bit more. After applying the
> >> attached patch and rebuilding mvapich2 you can use a variable called
> >> MV2_DEFAULT_PORT to control which port to use.
> >>
> >> First apply the patch using `patch -p0 < multi_port_roce_patch.txt'
> at
> >> the top level of the mvapich2 source code and rebuild the library.
> To
> >> run in RDMAoE mode on the 2nd port do something like:
> >>
> >> mpirun_rsh -np 2 node1 node2 MV2_USE_RDMAOE=1 MV2_DEFAULT_PORT=2
> ./prog
> >>
> >> Please let us know if this works for you.
> >>
> >> On Wed, Aug 17, 2011 at 4:48 PM, Konz, Jeffrey (SSA Solution
> Centers)
> >> <jeffrey.konz at hp.com> wrote:
> >> > Jonathan,
> >> >
> >> > An issue is selecting the right port on the Mellanox NIC, it has
> two
> >> ports: 1 IB, 1 10GigE.
> >> > Not sure how to do that.
> >> >
> >> > #ibstat
> >> > CA 'mlx4_0'
> >> > CA type: MT26438
> >> > Number of ports: 2
> >> > Firmware version: 2.7.9100
> >> > Hardware version: b0
> >> > Node GUID: 0x78e7d10300214bbc
> >> > System image GUID: 0x78e7d10300214bbf
> >> > Port 1:
> >> > State: Active
> >> > Physical state: LinkUp
> >> > Rate: 40
> >> > Base lid: 13
> >> > LMC: 0
> >> > SM lid: 1
> >> > Capability mask: 0x02510868
> >> > Port GUID: 0x78e7d10300214bbd
> >> > Link layer: IB
> >> > Port 2:
> >> > State: Active
> >> > Physical state: LinkUp
> >> > Rate: 40
> >> > Base lid: 0
> >> > LMC: 0
> >> > SM lid: 0
> >> > Capability mask: 0x00010000
> >> > Port GUID: 0x7ae7d1fffe214bbd
> >> > Link layer: Ethernet
> >> >
> >> >
> >> > #ibv_devinfo -v
> >> > hca_id: mlx4_0
> >> > transport: InfiniBand (0)
> >> > fw_ver: 2.7.9100
> >> > node_guid: 78e7:d103:0021:4bbc
> >> > sys_image_guid: 78e7:d103:0021:4bbf
> >> > vendor_id: 0x02c9
> >> > vendor_part_id: 26438
> >> > hw_ver: 0xB0
> >> > board_id: HP_0200000003
> >> > phys_port_cnt: 2
> >> > max_mr_size: 0xffffffffffffffff
> >> > page_size_cap: 0xfffffe00
> >> > max_qp: 260032
> >> > max_qp_wr: 16351
> >> > device_cap_flags: 0x007c9c76
> >> > max_sge: 32
> >> > max_sge_rd: 0
> >> > max_cq: 65408
> >> > max_cqe: 4194303
> >> > max_mr: 524272
> >> > max_pd: 32764
> >> > max_qp_rd_atom: 16
> >> > max_ee_rd_atom: 0
> >> > max_res_rd_atom: 4160512
> >> > max_qp_init_rd_atom: 128
> >> > max_ee_init_rd_atom: 0
> >> > atomic_cap: ATOMIC_HCA (1)
> >> > max_ee: 0
> >> > max_rdd: 0
> >> > max_mw: 0
> >> > max_raw_ipv6_qp: 0
> >> > max_raw_ethy_qp: 2
> >> > max_mcast_grp: 8192
> >> > max_mcast_qp_attach: 56
> >> > max_total_mcast_qp_attach: 458752
> >> > max_ah: 0
> >> > max_fmr: 0
> >> > max_srq: 65472
> >> > max_srq_wr: 16383
> >> > max_srq_sge: 31
> >> > max_pkeys: 128
> >> > local_ca_ack_delay: 15
> >> > port: 1
> >> > state: PORT_ACTIVE (4)
> >> > max_mtu: 2048 (4)
> >> > active_mtu: 2048 (4)
> >> > sm_lid: 1
> >> > port_lid: 13
> >> > port_lmc: 0x00
> >> > link_layer: IB
> >> > max_msg_sz: 0x40000000
> >> > port_cap_flags: 0x02510868
> >> > max_vl_num: 8 (4)
> >> > bad_pkey_cntr: 0x0
> >> > qkey_viol_cntr: 0x0
> >> > sm_sl: 0
> >> > pkey_tbl_len: 128
> >> > gid_tbl_len: 128
> >> > subnet_timeout: 18
> >> > init_type_reply: 0
> >> > active_width: 4X (2)
> >> > active_speed: 10.0 Gbps (4)
> >> > phys_state: LINK_UP (5)
> >> > GID[ 0]:
> >> > fe80:0000:0000:0000:78e7:d103:0021:4bbd
> >> >
> >> > port: 2
> >> > state: PORT_ACTIVE (4)
> >> > max_mtu: 2048 (4)
> >> > active_mtu: 1024 (3)
> >> > sm_lid: 0
> >> > port_lid: 0
> >> > port_lmc: 0x00
> >> > link_layer: Ethernet
> >> > max_msg_sz: 0x40000000
> >> > port_cap_flags: 0x00010000
> >> > max_vl_num: 8 (4)
> >> > bad_pkey_cntr: 0x0
> >> > qkey_viol_cntr: 0x0
> >> > sm_sl: 0
> >> > pkey_tbl_len: 1
> >> > gid_tbl_len: 128
> >> > subnet_timeout: 0
> >> > init_type_reply: 0
> >> > active_width: 4X (2)
> >> > active_speed: 10.0 Gbps (4)
> >> > phys_state: LINK_UP (5)
> >> > GID[ 0]:
> >> > fe80:0000:0000:0000:7ae7:d1ff:fe21:4bbd
> >> >
> >> >
> >> > -Jeff
> >> >
> >> >
> >> >> -----Original Message-----
> >> >> From: Jonathan Perkins [mailto:perkinjo at cse.ohio-state.edu]
> >> >> Sent: Wednesday, August 17, 2011 12:25 PM
> >> >> To: Konz, Jeffrey (SSA Solution Centers)
> >> >> Cc: mvapich-discuss at cse.ohio-state.edu
> >> >> Subject: Re: [mvapich-discuss] problems with MVAPICH2 over 10GbE
> >> >>
> >> >> Thanks for your report, I'm checking with some of the other
> >> >> developers to verify the way this should work. I believe that
> you
> >> do
> >> >> not need to use the IP addresses of the RDMAoE port but instead
> >> >> specify the the HCA name using MV2_IBA_HCA in addition to the
> >> MV2_USE_RDMAOE=1 option.
> >> >>
> >> >> The name of the HCA can be found by using the ibstat command and
> >> >> should look something like mlx4_...
> >> >>
> >> >> On Wed, Aug 17, 2011 at 11:17 AM, Konz, Jeffrey (SSA Solution
> >> >> Centers) <jeffrey.konz at hp.com> wrote:
> >> >> > I am running on a cluster with the Mellanox LOM that supports
> both
> >> >> > IB
> >> >> and 10 GbE.
> >> >> > Both ports on the interface are active, one is on IB network
> the
> >> >> other on 10 GbE network.
> >> >> >
> >> >> > I built mvapich2-1.7rc1 with these options :
> >> >> > "--with-device=ch3:mrail
> >> >> --with-rdma=gen2"
> >> >> >
> >> >> > Running over IB works fine.
> >> >> >
> >> >> > When I try to run over the 10GbE network with the
> >> "MV2_USE_RDMAOE=1"
> >> >> option I get this error:
> >> >> >
> >> >> > Fatal error in MPI_Init:
> >> >> > Internal MPI error!
> >> >> >
> >> >> > [atl3-13:mpispawn_0][readline] Unexpected End-Of-File on file
> >> >> descriptor 5. MPI process died?
> >> >> > [atl3-13:mpispawn_0][mtpmi_processops] Error while reading PMI
> >> >> socket. MPI process died?
> >> >> > [atl3-13:mpispawn_0][child_handler] MPI process (rank: 0, pid:
> >> >> > 23500)
> >> >> exited with status 1
> >> >> > [atl3-13:mpirun_rsh][process_mpispawn_connection] mpispawn_0
> from
> >> >> node 10.10.0.149 aborted: Error while reading a PMI socket (4)
> >> >> >
> >> >> > In the hostfile I specified the IP addresses of the 10 GbE
> ports.
> >> >> >
> >> >> > I am running incorrectly or have I not built mvapich with the
> >> >> > correct
> >> >> options?
> >> >> >
> >> >> > Thanks,
> >> >> >
> >> >> > -Jeff
> >> >> >
> >> >> >
> >> >> > _______________________________________________
> >> >> > mvapich-discuss mailing list
> >> >> > mvapich-discuss at cse.ohio-state.edu
> >> >> > http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss
> >> >> >
> >> >> >
> >> >>
> >> >>
> >> >>
> >> >> --
> >> >> Jonathan Perkins
> >> >> http://www.cse.ohio-state.edu/~perkinjo
> >> >
> >> >
> >>
> >>
> >>
> >> --
> >> Jonathan Perkins
> >> http://www.cse.ohio-state.edu/~perkinjo
> >
> >
>
>
>
> --
> Jonathan Perkins
> http://www.cse.ohio-state.edu/~perkinjo
More information about the mvapich-discuss
mailing list