[mvapich-discuss] intra node communication between GPUs

[sreeram potluri]

Ye,

Can you tell us the configuration in which the two GPUs are connected to a
node? Are they on the same I/O Hub or socket? 'lspci -tv' should give the
information we are looking for.

Sreeram Potluri

On Mon, Jun 10, 2013 at 10:30 AM, Ye Wang <wang1351 at purdue.edu> wrote:

> Hi,
>
> I am using mvapich2-1.9 to do communications between GPUs. When I tried to
> test the bandwidth using the osu_bw in the osu_benchmarks package between
> GPUs on a GPU cluster, I found that the bandwidth between two GPUs on a
> same PCIe bus is smaller than the bandwidth between two GPUs on different
> nodes. I can not figure out the reason. I think the communication between
> two GPUs on a same PCIe bus should be through PCIe bus directly with the
> support of GPUDirect v2. Why is it slower than communication between two
> GPUs on separate nodes?
>
> The following is the bandwidth between two GPUs on same PCIe bus:
>
> # Send Buffer on DEVICE (D) and Receive Buffer on DEVICE (D)
> # Size        Bandwidth (MB/s)
> 1                         0.13
> 2                         0.25
> 4                         0.51
> 8                         1.01
> 16                        2.03
> 32                        4.04
> 64                        8.08
> 128                      16.25
> 256                      34.22
> 512                      67.96
> 1024                    137.11
> 2048                    272.85
> 4096                    540.70
> 8192                   1092.38
> 16384                  2125.43
> 32768                  3134.74
> 65536                  4022.73
> 131072                 4836.31
> 262144                 4944.00
> 524288                 5009.88
> 1048576                5019.20
> 2097152                5052.10
> 4194304                5067.23
>
> And this is the bandwidth between GPUs on two nodes:
>
> # Send Buffer on DEVICE (D) and Receive Buffer on DEVICE (D)
> # Size        Bandwidth (MB/s)
> 1                         0.09
> 2                         0.17
> 4                         0.34
> 8                         0.69
> 16                        1.35
> 32                        2.68
> 64                        5.46
> 128                      10.88
> 256                      21.89
> 512                      43.00
> 1024                     83.51
> 2048                    160.64
> 4096                    297.13
> 8192                    524.53
> 16384                  1547.57
> 32768                  2525.94
> 65536                  3607.36
> 131072                 4556.56
> 262144                 5174.55
> 524288                 5318.21
> 1048576                5346.60
> 2097152                5373.96
> 4194304                5405.89
>
> Thanks,
>
> Ye
> _______________________________________________
> mvapich-discuss mailing list
> mvapich-discuss at cse.ohio-state.edu
> http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.cse.ohio-state.edu/pipermail/mvapich-discuss/attachments/20130610/d9455fd1/attachment-0001.html