[mvapich-discuss]Infiniband - how do I know that it is working?

[Devendar Bureddy]

On Thu, May 31, 2012 at 9:46 AM, Ju JiaJia <jujj603 at gmail.com> wrote:

> I am trying OpenMPI, but the result is the same. So i'm testing HPL with a
> large dataset, still running. 20000 is a little small.
>
> ldd shows :
> [Thu May 31 21:17:26]user39 at m1:Linux_x86_64_CBLAS_openmpi$ ldd
> xhpl_mvapich2
>         libpthread.so.0 => /lib64/libpthread.so.0 (0x0000003ac7400000)
>         librdmacm.so.1 => /usr/lib64/librdmacm.so.1 (0x0000003165a00000)
>         libibverbs.so.1 => /usr/lib64/libibverbs.so.1 (0x0000003165600000)
>         libibumad.so.3 => /usr/lib64/libibumad.so.3 (0x00002b5942650000)
>         libdl.so.2 => /lib64/libdl.so.2 (0x0000003ac6c00000)
>         librt.so.1 => /lib64/librt.so.1 (0x0000003ac8000000)
>         libgfortran.so.1 => /usr/lib64/libgfortran.so.1
> (0x00002b5942857000)
>         libm.so.6 => /lib64/libm.so.6 (0x0000003ac7000000)
>         libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x0000003acac00000)
>         libc.so.6 => /lib64/libc.so.6 (0x0000003ac6800000)
>         /lib64/ld-linux-x86-64.so.2 (0x0000003ac5800000)
>
> mfile is a node list :
> [Thu May 31 21:20:28]user39 at m1:Linux_x86_64_CBLAS_openmpi$ cat mfile
> c04b19
> c04b20
>
> I already send osu_latency and osu_bw result in the mail.
> And when i run mvapich2 built HPL, i can't see big RX/TX change of
> ifconfig's output. ifconfig shows 1.1G of network data flow with  MPICH2
> built HPL. Why ?
>

ifconfig RX/TX do not reflect the infiniband native verbs communication.
These counters will change only if you run TCP/IP over infiniband network.

The communication might not be bottleneck for not observing much difference
in your experiments.

-Devendar





>
> Thanks for you reply, very appreciated.
>
>
> On Thu, May 31, 2012 at 8:50 PM, Jonathan Perkins <
> perkinjo at cse.ohio-state.edu> wrote:
>
>> Hello, can you try running the osu_latency test (the
>> osu-micro-benchmarks also now contains collective benchmarks if you'd
>> like to try those out)?  It may be easier to tell which network it is
>> using that way.  You can also try
>>
>>    ldd /path/to/application
>>
>> to make sure it is linked to the correct library (assuming you built
>> with shared libraries).
>>
>> What are the contents of mfile?
>>
>>
>> On Thu, May 31, 2012 at 12:16:30PM +0800, Ju JiaJia wrote:
>> > I have run osu_latency and osu_bw with both mvapich2 and mpich2 on two
>> > nodes, here is my test result
>> >
>> >
>> MVAPICH2(Infiniband）MPICH2（Ethernet）01.4515.1611.4915.2521.4915.1541.4915.15
>> >
>> 81.515.17161.5315.17321.5716.4641.6616.881281.8317.82562.920.395123.1426.06
>> >
>> 1K3.847.52K4.89193.974K5.99228.978K7.85233.1216K12.51241.3332K18.08432.3164K
>> >
>> 28.7675.25128K50.021429.21256K92.572582.83512K177.654693.171M347.759110.472M
>> > 688.1217942.074M1373.935609.21
>> >
>> > Here  is HPL results on two nodes:
>> > HPL.out.ethernet.211:   WR00L2C2       20000   128     2     1
>>    *
>> > 1686.40*              3.163e+00
>> > HPL.out.ib.211:            WR00L2C2       20000   128     2     1
>> > * 1639.99 *             3.252e+00
>> >
>> > run script :
>> > export MV2_ENABLE_AFFINITY=0
>> > PATH=/gos4/user39/jujj/program-files/mvapich2/bin:$PATH
>> >
>> > #
>> > mpiexec -f mfile -n 2 taskset -c 0 ./xhpl > HPL.out.ib.211 2>&1
>> > mpiexec -f mfile -n 2 taskset -c 0 ./xhpl_ethernet >
>> HPL.out.ethernet.211
>> > 2>&1
>> >
>> > As you can see, no big difference. I tested NAMD also, no big
>> difference.
>> > Is there any tools like netstat over infiniband ? So i can see whether
>> > connections is built or which network is being used. Or MVAPICH2 support
>> > some ways to do this? Log or something?
>> >
>> > On Thu, May 31, 2012 at 11:43 AM, Dhabaleswar Panda <
>> > panda at cse.ohio-state.edu> wrote:
>> >
>> > > After installing MVAPICH2, you can run OSU MPI Micro-Benchmarks to
>> verify
>> > > that your installtion is correct. You can verify your performance
>> with the
>> > > performance numbers/graphs available at the MVAPICH site. After that
>> you
>> > > can carry out your applications-level study.
>> > >
>> > > DK
>> > >
>> > > On Thu, 31 May 2012, Ju JiaJia wrote:
>> > >
>> > > > Hi all:
>> > > >
>> > > > I am currently running HPL built using mvapich2 which use
>> infiniband,
>> > > > but it shows no apparently difference, compares in performance to a
>> > > > build using mpich2 which uses Ethenet.
>> > > >
>> > > > There should be some improvement in performance. I doubt whether
>> > > > infiniband is working. Anyone knows how to check whether infiniband
>> is
>> > > > working ?
>> > > >
>> > > >
>> > > > Ju JiaJia
>> > > >
>> > >
>> > >
>>
>> > _______________________________________________
>> > mvapich-discuss mailing list
>> > mvapich-discuss at cse.ohio-state.edu
>> > http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss
>>
>>
>> --
>> Jonathan Perkins
>> http://www.cse.ohio-state.edu/~perkinjo
>>
>
>
> _______________________________________________
> mvapich-discuss mailing list
> mvapich-discuss at cse.ohio-state.edu
> http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss
>
>


-- 
Devendar
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.cse.ohio-state.edu/pipermail/mvapich-discuss/attachments/20120531/13fdda80/attachment-0001.html