[mvapich-discuss] How to select the infiniband

[Ji Wan]

Dear Jonathan,

Thanks for your reply.
As suggested by the user guide, I have re-built MVAPICH2 with the following
options:

./configure \
  LDFLAGS='-lstdc++ -L/cm/shared/apps/slurm/2.5.7/lib64
-L/cm/shared/apps/cuda50/toolkit/current/lib64' \
  CPPFLAGS='-I/usr/local/cuda/include' \
  --prefix=$MYROOT \
  --disable-f77 --disable-fc  \
  --enable-g=dbg --disable-fast \
  --enable-cuda --with-cuda=/cm/shared/apps/cuda50/toolkit/current \
  --with-pm=no --with-pmi=slurm --with-slurm=/cm/shared/apps/slurm/current \
  --enable-threads=multiple \
  --with-device=ch3:mrail --with-rdma=gen2


I tried to use the attached program to test the time consumed by transfer
10GB data between two nodes by MPI_Send/MPI_Recv.
The compile and execute commands are list below:

mpic++ speedtest.cpp -O3 -o speedtest
srun -p gpu -n 2 --ntasks-per-node=1 ./speedtest

And the outputs is :

[1] <- [0]: begin
[0] -> [1]: begin
[1] <- [0]: end (0, 0, 0, time: 9521.510ms)
[0] -> [1]: end (0, 0, 0, time: 8574.974ms)

which indicates that it took ~9.5s for sending and ~8.5s for receving.

Since I have three network interface between the two nodes (*eth0*: 1Gbps,
*eth2*: 10Gbps, *ib0*: 40Gbps), does these mean that the program actually
using *eth2* rather than *ib0*?

By the way, is the source code of MVAPICH2-GDR available? We want to use it
on RHEL 6.0 with CUDA 5.0, but cannot find an appropriate RPM package.




*--Best regards,Wan Ji*


On Wed, Jun 25, 2014 at 11:42 PM, Jonathan Perkins <
perkinjo at cse.ohio-state.edu> wrote:

> Yes, please see section 4.4 of our userguide.
>
>
> http://mvapich.cse.ohio-state.edu/static/media/mvapich/mvapich2-2.0-userguide.html#x1-110004.4
>
>
> On Wed, Jun 25, 2014 at 11:39 AM, Ji Wan <wanjime at gmail.com> wrote:
>
>> Dear developers,
>>
>> I am developing on a clusters with 4 nodes which are connected by both
>> regular Ethernet (eth0: 1Gbps) and infiniband (ib0: 40Gbps).
>> Is there any way to choose inifniband other than Ethernet?
>>
>>
>>
>> *--Best regards,Wan Ji*
>>
>> _______________________________________________
>> mvapich-discuss mailing list
>> mvapich-discuss at cse.ohio-state.edu
>> http://mailman.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss
>>
>>
>
>
> --
> Jonathan Perkins
> http://www.cse.ohio-state.edu/~perkinjo
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.cse.ohio-state.edu/pipermail/mvapich-discuss/attachments/20140626/d2676881/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: speedtest.cpp
Type: text/x-c++src
Size: 1593 bytes
Desc: not available
URL: <http://mailman.cse.ohio-state.edu/pipermail/mvapich-discuss/attachments/20140626/d2676881/attachment.bin>