[mvapich-discuss] How to select specific out of multiple interfaces for communication and support for heterogeneous fabrics

[Devendar Bureddy]

Hi Michael

Please see inline answers for your questions.

On Fri, Jul 5, 2013 at 5:13 PM, Michael Thomadakis <drmichaelt7777 at gmail.com
> wrote:

> Hello mvapich2.
>
> We area seriously considering deploying MVAPICH2 on HPC clusters which
> consists of nodes with different types of interfaces.
>
>
> 1) Interface selection
>
> We are using MVAPICH2 1.9 and was wondering how one would go about
> selecting* at run time* which networking interface to use for MPI
> communications in case that both IB, 10GigE and 1 GigE are present.
>
> This issues arises in a cluster with nodes that are equipped with
> different types of interfaces:
>
> *Some *have both IB-QDR or FDR and 10- and 1-GigE. Others *only* have
> 10-GigE and 1-GigE and simply others only 1-GigE.
>

The run-time selection of network interface should be possible with Nemesis
interface. please see user guide section for more details :
http://mvapich.cse.ohio-state.edu/support/user_guide_mvapich2-1.9.html#x1-200004.12


Please note that CH3-Gen2 interface still provides more features,
performance and scalability compared to the Nemesis interface for pure
InfiniBand.


>
>
> 2) MVAPICH2 level of support for Heterogeneous Fabric
>
> Can MVAPICH2 support running an MPI application using a mix of nodes with
> all of the above networking interface combinations ?
>
>   2.a) Can the same MPI code (SPMD or MPMD) have a subset of its ranks run
> on nodes with QDR IB and another subset on FDR IB simultaneously? These are
> Mellanox QDR and FDR HCAs.
>

Yes, this will work


>
> Mellanox mentioned to us that they support both QDR and FDR HCAs attached
> to the same IB subnet. Do you think MVAPICH2 will have any issue with this?
>
> 2.b) Can the same MPI code (SPMD or MPMD) have a subset of its ranks run
> on nodes with IB and another subset over 10GiGE simultaneously?
>
> That is imagine nodes I1, I2, ..., IN having say QDR HCAs and nodes G1,
> G2, GM having only 10GigE interfaces. Could we have the same MPI
> application run across both types of nodes?
>

No. combination of IB and 10G will not work.


- Devendar


>
> Or should there be say 2 communicators with one of them explicitly
> overlaid on a IB only subnet and the other on a 10GigE only subnet?
>
>
> Please let me know if the above are not very clear.
>
> Thank you much
> Michael
>
> _______________________________________________
> mvapich-discuss mailing list
> mvapich-discuss at cse.ohio-state.edu
> http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss
>
>


-- 
Devendar
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.cse.ohio-state.edu/pipermail/mvapich-discuss/attachments/20130705/d87dfa6d/attachment.html