[mvapich-discuss] How to select specific out of multiple interfaces for communication and support for heterogeneous fabrics

[Devendar Bureddy]

Hi Michael

On Wed, Jul 24, 2013 at 6:17 PM, Michael Thomadakis <
drmichaelt7777 at gmail.com> wrote:

> I was wondering if the same MVAPICH2 1.9 library built on a system with
> RDMA enabled NICs (IB or RoCE) can then be used also from nodes that have
> NO RDMA NICs. Or should I built two versions of MVAPICH2?
>

MVAPICH2 built(default configuration) for RDMA NICs should work for IB,RoCEand
iWARP. This build also work on single node that do not have RDMA NIC.  This
build do not work for multi-node (NO RDMA NICs) MPI job.

-Devendar


>
> thanks
> Michael
>
>
> On Fri, Jul 5, 2013 at 9:32 PM, Michael Thomadakis <
> drmichaelt7777 at gmail.com> wrote:
>
>> Hi Devendar,
>>
>> thanks for the clear answers...  The sub-clusters grew independently and
>> now it occurred to the owners to unify them all under one fabric some with
>> QDR some with FDR.
>>
>> take care
>> Michael
>>
>>
>> On Fri, Jul 5, 2013 at 6:39 PM, Devendar Bureddy <
>> bureddy at cse.ohio-state.edu> wrote:
>>
>>> Hi Michael
>>>
>>> Please see inline answers for your questions.
>>>
>>> On Fri, Jul 5, 2013 at 5:13 PM, Michael Thomadakis <
>>> drmichaelt7777 at gmail.com> wrote:
>>>
>>>> Hello mvapich2.
>>>>
>>>> We area seriously considering deploying MVAPICH2 on HPC clusters which
>>>> consists of nodes with different types of interfaces.
>>>>
>>>>
>>>> 1) Interface selection
>>>>
>>>> We are using MVAPICH2 1.9 and was wondering how one would go about
>>>> selecting* at run time* which networking interface to use for MPI
>>>> communications in case that both IB, 10GigE and 1 GigE are present.
>>>>
>>>> This issues arises in a cluster with nodes that are equipped with
>>>> different types of interfaces:
>>>>
>>>> *Some *have both IB-QDR or FDR and 10- and 1-GigE. Others *only* have
>>>> 10-GigE and 1-GigE and simply others only 1-GigE.
>>>>
>>>
>>> The run-time selection of network interface should be possible with
>>> Nemesis interface. please see user guide section for more details :
>>> http://mvapich.cse.ohio-state.edu/support/user_guide_mvapich2-1.9.html#x1-200004.12
>>>
>>>
>>> Please note that CH3-Gen2 interface still provides more features,
>>> performance and scalability compared to the Nemesis interface for pure
>>> InfiniBand.
>>>
>>>
>>>>
>>>>
>>>> 2) MVAPICH2 level of support for Heterogeneous Fabric
>>>>
>>>> Can MVAPICH2 support running an MPI application using a mix of nodes
>>>> with all of the above networking interface combinations ?
>>>>
>>>>   2.a) Can the same MPI code (SPMD or MPMD) have a subset of its ranks
>>>> run on nodes with QDR IB and another subset on FDR IB simultaneously? These
>>>> are Mellanox QDR and FDR HCAs.
>>>>
>>>
>>> Yes, this will work
>>>
>>>
>>>>
>>>> Mellanox mentioned to us that they support both QDR and FDR HCAs
>>>> attached to the same IB subnet. Do you think MVAPICH2 will have any issue
>>>> with this?
>>>>
>>>> 2.b) Can the same MPI code (SPMD or MPMD) have a subset of its ranks
>>>> run on nodes with IB and another subset over 10GiGE simultaneously?
>>>>
>>>> That is imagine nodes I1, I2, ..., IN having say QDR HCAs and nodes G1,
>>>> G2, GM having only 10GigE interfaces. Could we have the same MPI
>>>> application run across both types of nodes?
>>>>
>>>
>>> No. combination of IB and 10G will not work.
>>>
>>>
>>> - Devendar
>>>
>>>
>>>>
>>>> Or should there be say 2 communicators with one of them explicitly
>>>> overlaid on a IB only subnet and the other on a 10GigE only subnet?
>>>>
>>>>
>>>> Please let me know if the above are not very clear.
>>>>
>>>> Thank you much
>>>> Michael
>>>>
>>>> _______________________________________________
>>>> mvapich-discuss mailing list
>>>> mvapich-discuss at cse.ohio-state.edu
>>>> http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss
>>>>
>>>>
>>>
>>>
>>> --
>>> Devendar
>>>
>>
>>
>


-- 
Devendar
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.cse.ohio-state.edu/pipermail/mvapich-discuss/attachments/20130724/c8aa1f2a/attachment.html