[mvapich-discuss] Network selection with MVAPICH 2.1, Slurm and RoCE

[Hari Subramoni]

Dear Davide,

Apologies for the delay in getting back to you.

With the configuration you have mentioned MVAPICH2 will always select an
RDMA enabled network interface. It will never select the non-RDMA enabled 1Gb/s
Ethernet interface. It will fail if its not able to find an RDMA enabled
network interface. Could you please let us know why you think its trying to
connect through the 1 Gb/s link?

Could you please send us the output of ibv_devinfo command on the nodes on
which you're trying to run the program?

Have you had a chance to try out the second method mentioned in the
following section of the userguide that deals with running MVAPICH2 in RoCE
mode?

http://mvapich.cse.ohio-state.edu/static/media/mvapich/mvapich2-2.2b-userguide.html#x1-380005.2.7

Thanks for pointing out the inconsistency in the userguide. The
"MV2_USE_RDMA_CM"
is also applicable for the OFA-RoCE interface. We will take care of fixing
this soon.

Regards,
Hari.

On Tue, Feb 2, 2016 at 12:13 PM, Davide Vanzo <vanzod at accre.vanderbilt.edu>
wrote:

> Hi all,
> Our cluster is interconnected with a standard 1Gb/s Ethernet network. A
> subset of nodes share a 56Gb/s RoCE switch which is dedicated to such
> nodes, i.e. no uplink to the rest of the cluster. In this way each RoCE
> node has two interfaces: one for RoCE and another for Ethernet. We use
> Slurm as scheduler and I compiled MVAPICH 2.1 with the following
> configuration flags:
>
> --with-device=ch3:mrail
> --with-rdma=gen2
> --with-ib-include=/usr/include/infiniband
> --with-ib-libpath=/usr/lib64
> --enable-hwloc
> --with-pmi=pmi2
> --with-pm=slurm
> --with-slurm=/usr/scheduler/slurm
> --enable-fortran=yes
> --enable-cxx
>
> I tried to run the OSU benchmark tests on the RoCE interface but I can't
> still figure out how to tell MVAPICH which network it should use when
> invoking it via srun. I checked in the documentation and I tried with
> adding the two IP addresses corresponding to the RoCE interfaces in
> /etc/mv2.conf and using the following flags:
>
> export MV2_USE_RoCE=1
> export MV2_USE_RDMA_CM=1
> srun ./osu_bw
>
> but it still attempts to connect on the Ethernet interface and the return
> status is 1.
>
> I also want to point out a thing about the MV2_USE_RDMA_CM variable. In
> the documentation at section 5.2.7 it explicitly says to set such variable
> to 1 in order to use the private VLAN. However on section 11.85, it says
> that it can be applied only to OFA-IB-CH3 and OFA-iWARP-CH3 interfaces, not
> with OFA-RoCE. Am I missing something?
>
> Thank you in advance,
> Davide
>
> _______________________________________________
> mvapich-discuss mailing list
> mvapich-discuss at cse.ohio-state.edu
> http://mailman.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.cse.ohio-state.edu/pipermail/mvapich-discuss/attachments/20160204/380edcdd/attachment-0001.html>