[mvapich-discuss] avoiding the QPI bus

[khaled hamidouche]

Dear Enrico,

Assuming you do not have any Host-Host communication, then to achieve your
goal just disable shared memory. MV2_USE_SHARED_MEM=0. This will ensure
that all the communication goes through HCA and not through QPI.

But just to clarify, MV2-GDR does use automatic selection of HCA  to avoid
the usage of GPUDirect RDMA code path through QPI at all, So if need to use
QPI, we first copy to the host and then use QPI (which is not a big
limitation)

Thanks

On Fri, Apr 15, 2016 at 5:43 AM, Enrico Calore <enrico.calore at fe.infn.it>
wrote:

> Hi all,
> in our cluster nodes we have several NVIDIA GPUs and one Mellanox IB
> card for each PCIe root complex of the two sockets.
> Since we know about the bad performances of MPI transfers through the
> QPI bus, we would like to test the performances of our multi-GPU codes
> in different scenarios.
>
> In general, in our codes, each MPI process running on a CPU core of a
> specific socket, controls one GPU attached to the PCIe root complex
> connected to the same socket.
> Kernels running on GPUs need to exchange data with other GPUs (connected
> to the same PCIe root complex, connected to the PCIe root complex of the
> other socket or in another compute node), using GPUDirect-RDMA.
>
> We would like to test performances in a couple of scenarios:
>
> Scenario A) GPUs attached to different PCIe root complexes performs
> intra-node-iter-socket communications through the QPI bus.
>
> Scenario B) GPUs attached to different PCIe root complexes performs
> intra-node-inter-socket communications through the Infiniband Fabrics
> (as shown in this pic:
>
> http://www.cirrascale.com/blog/wp-content/uploads/2014/08/PCIe-Block-Diagram-Typical-8-GPU-System-720x485.png
> )
>
>
> Is there a way to control the use of the QPI bus and a way to report if
> it is used or not, when using the MVAPICH2-GDR MPI library?
>
>
> Thanks in advance and Best Regards,
>
> Enrico
>
>
>
>
> _______________________________________________
> mvapich-discuss mailing list
> mvapich-discuss at cse.ohio-state.edu
> http://mailman.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.cse.ohio-state.edu/pipermail/mvapich-discuss/attachments/20160415/b6faf421/attachment-0001.html>