[mvapich-discuss] How to select specific out of multiple interfaces for communication and support for heterogeneous fabrics

[Michael Thomadakis]

I was wondering if the same MVAPICH2 1.9 library built on a system with
RDMA enabled NICs (IB or RoCE) can then be used also from nodes that have
NO RDMA NICs. Or should I built two versions of MVAPICH2?

thanks
Michael


On Fri, Jul 5, 2013 at 9:32 PM, Michael Thomadakis <drmichaelt7777 at gmail.com
> wrote:

> Hi Devendar,
>
> thanks for the clear answers...  The sub-clusters grew independently and
> now it occurred to the owners to unify them all under one fabric some with
> QDR some with FDR.
>
> take care
> Michael
>
>
> On Fri, Jul 5, 2013 at 6:39 PM, Devendar Bureddy <
> bureddy at cse.ohio-state.edu> wrote:
>
>> Hi Michael
>>
>> Please see inline answers for your questions.
>>
>> On Fri, Jul 5, 2013 at 5:13 PM, Michael Thomadakis <
>> drmichaelt7777 at gmail.com> wrote:
>>
>>> Hello mvapich2.
>>>
>>> We area seriously considering deploying MVAPICH2 on HPC clusters which
>>> consists of nodes with different types of interfaces.
>>>
>>>
>>> 1) Interface selection
>>>
>>> We are using MVAPICH2 1.9 and was wondering how one would go about
>>> selecting* at run time* which networking interface to use for MPI
>>> communications in case that both IB, 10GigE and 1 GigE are present.
>>>
>>> This issues arises in a cluster with nodes that are equipped with
>>> different types of interfaces:
>>>
>>> *Some *have both IB-QDR or FDR and 10- and 1-GigE. Others *only* have
>>> 10-GigE and 1-GigE and simply others only 1-GigE.
>>>
>>
>> The run-time selection of network interface should be possible with
>> Nemesis interface. please see user guide section for more details :
>> http://mvapich.cse.ohio-state.edu/support/user_guide_mvapich2-1.9.html#x1-200004.12
>>
>>
>> Please note that CH3-Gen2 interface still provides more features,
>> performance and scalability compared to the Nemesis interface for pure
>> InfiniBand.
>>
>>
>>>
>>>
>>> 2) MVAPICH2 level of support for Heterogeneous Fabric
>>>
>>> Can MVAPICH2 support running an MPI application using a mix of nodes
>>> with all of the above networking interface combinations ?
>>>
>>>   2.a) Can the same MPI code (SPMD or MPMD) have a subset of its ranks
>>> run on nodes with QDR IB and another subset on FDR IB simultaneously? These
>>> are Mellanox QDR and FDR HCAs.
>>>
>>
>> Yes, this will work
>>
>>
>>>
>>> Mellanox mentioned to us that they support both QDR and FDR HCAs
>>> attached to the same IB subnet. Do you think MVAPICH2 will have any issue
>>> with this?
>>>
>>> 2.b) Can the same MPI code (SPMD or MPMD) have a subset of its ranks run
>>> on nodes with IB and another subset over 10GiGE simultaneously?
>>>
>>> That is imagine nodes I1, I2, ..., IN having say QDR HCAs and nodes G1,
>>> G2, GM having only 10GigE interfaces. Could we have the same MPI
>>> application run across both types of nodes?
>>>
>>
>> No. combination of IB and 10G will not work.
>>
>>
>> - Devendar
>>
>>
>>>
>>> Or should there be say 2 communicators with one of them explicitly
>>> overlaid on a IB only subnet and the other on a 10GigE only subnet?
>>>
>>>
>>> Please let me know if the above are not very clear.
>>>
>>> Thank you much
>>> Michael
>>>
>>> _______________________________________________
>>> mvapich-discuss mailing list
>>> mvapich-discuss at cse.ohio-state.edu
>>> http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss
>>>
>>>
>>
>>
>> --
>> Devendar
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.cse.ohio-state.edu/pipermail/mvapich-discuss/attachments/20130724/0b9fd943/attachment-0001.html