[mvapich-discuss]Infiniband - how do I know that it is working?

[Ju JiaJia]

I am trying OpenMPI, but the result is the same. So i'm testing HPL with a
large dataset, still running. 20000 is a little small.

ldd shows :
[Thu May 31 21:17:26]user39 at m1:Linux_x86_64_CBLAS_openmpi$ ldd
xhpl_mvapich2
        libpthread.so.0 => /lib64/libpthread.so.0 (0x0000003ac7400000)
        librdmacm.so.1 => /usr/lib64/librdmacm.so.1 (0x0000003165a00000)
        libibverbs.so.1 => /usr/lib64/libibverbs.so.1 (0x0000003165600000)
        libibumad.so.3 => /usr/lib64/libibumad.so.3 (0x00002b5942650000)
        libdl.so.2 => /lib64/libdl.so.2 (0x0000003ac6c00000)
        librt.so.1 => /lib64/librt.so.1 (0x0000003ac8000000)
        libgfortran.so.1 => /usr/lib64/libgfortran.so.1 (0x00002b5942857000)
        libm.so.6 => /lib64/libm.so.6 (0x0000003ac7000000)
        libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x0000003acac00000)
        libc.so.6 => /lib64/libc.so.6 (0x0000003ac6800000)
        /lib64/ld-linux-x86-64.so.2 (0x0000003ac5800000)

mfile is a node list :
[Thu May 31 21:20:28]user39 at m1:Linux_x86_64_CBLAS_openmpi$ cat mfile
c04b19
c04b20

I already send osu_latency and osu_bw result in the mail.
And when i run mvapich2 built HPL, i can't see big RX/TX change of
ifconfig's output. ifconfig shows 1.1G of network data flow with  MPICH2
built HPL. Why ?

Thanks for you reply, very appreciated.


On Thu, May 31, 2012 at 8:50 PM, Jonathan Perkins <
perkinjo at cse.ohio-state.edu> wrote:

> Hello, can you try running the osu_latency test (the
> osu-micro-benchmarks also now contains collective benchmarks if you'd
> like to try those out)?  It may be easier to tell which network it is
> using that way.  You can also try
>
>    ldd /path/to/application
>
> to make sure it is linked to the correct library (assuming you built
> with shared libraries).
>
> What are the contents of mfile?
>
>
> On Thu, May 31, 2012 at 12:16:30PM +0800, Ju JiaJia wrote:
> > I have run osu_latency and osu_bw with both mvapich2 and mpich2 on two
> > nodes, here is my test result
> >
> >
> MVAPICH2(Infiniband）MPICH2（Ethernet）01.4515.1611.4915.2521.4915.1541.4915.15
> >
> 81.515.17161.5315.17321.5716.4641.6616.881281.8317.82562.920.395123.1426.06
> >
> 1K3.847.52K4.89193.974K5.99228.978K7.85233.1216K12.51241.3332K18.08432.3164K
> >
> 28.7675.25128K50.021429.21256K92.572582.83512K177.654693.171M347.759110.472M
> > 688.1217942.074M1373.935609.21
> >
> > Here  is HPL results on two nodes:
> > HPL.out.ethernet.211:   WR00L2C2       20000   128     2     1
>  *
> > 1686.40*              3.163e+00
> > HPL.out.ib.211:            WR00L2C2       20000   128     2     1
> > * 1639.99 *             3.252e+00
> >
> > run script :
> > export MV2_ENABLE_AFFINITY=0
> > PATH=/gos4/user39/jujj/program-files/mvapich2/bin:$PATH
> >
> > #
> > mpiexec -f mfile -n 2 taskset -c 0 ./xhpl > HPL.out.ib.211 2>&1
> > mpiexec -f mfile -n 2 taskset -c 0 ./xhpl_ethernet > HPL.out.ethernet.211
> > 2>&1
> >
> > As you can see, no big difference. I tested NAMD also, no big difference.
> > Is there any tools like netstat over infiniband ? So i can see whether
> > connections is built or which network is being used. Or MVAPICH2 support
> > some ways to do this? Log or something?
> >
> > On Thu, May 31, 2012 at 11:43 AM, Dhabaleswar Panda <
> > panda at cse.ohio-state.edu> wrote:
> >
> > > After installing MVAPICH2, you can run OSU MPI Micro-Benchmarks to
> verify
> > > that your installtion is correct. You can verify your performance with
> the
> > > performance numbers/graphs available at the MVAPICH site. After that
> you
> > > can carry out your applications-level study.
> > >
> > > DK
> > >
> > > On Thu, 31 May 2012, Ju JiaJia wrote:
> > >
> > > > Hi all:
> > > >
> > > > I am currently running HPL built using mvapich2 which use infiniband,
> > > > but it shows no apparently difference, compares in performance to a
> > > > build using mpich2 which uses Ethenet.
> > > >
> > > > There should be some improvement in performance. I doubt whether
> > > > infiniband is working. Anyone knows how to check whether infiniband
> is
> > > > working ?
> > > >
> > > >
> > > > Ju JiaJia
> > > >
> > >
> > >
>
> > _______________________________________________
> > mvapich-discuss mailing list
> > mvapich-discuss at cse.ohio-state.edu
> > http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss
>
>
> --
> Jonathan Perkins
> http://www.cse.ohio-state.edu/~perkinjo
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.cse.ohio-state.edu/pipermail/mvapich-discuss/attachments/20120531/65a113be/attachment.html