[mvapich-discuss]Infiniband - how do I know that it is working?

Thu May 31 09:52:46 EDT 2012

On Thu, May 31, 2012 at 09:46:50PM +0800, Ju JiaJia wrote:
> I am trying OpenMPI, but the result is the same. So i'm testing HPL with a
> large dataset, still running. 20000 is a little small.
> 
> ldd shows :
> [Thu May 31 21:17:26]user39 at m1:Linux_x86_64_CBLAS_openmpi$ ldd
> xhpl_mvapich2
>         libpthread.so.0 => /lib64/libpthread.so.0 (0x0000003ac7400000)
>         librdmacm.so.1 => /usr/lib64/librdmacm.so.1 (0x0000003165a00000)
>         libibverbs.so.1 => /usr/lib64/libibverbs.so.1 (0x0000003165600000)
>         libibumad.so.3 => /usr/lib64/libibumad.so.3 (0x00002b5942650000)
>         libdl.so.2 => /lib64/libdl.so.2 (0x0000003ac6c00000)
>         librt.so.1 => /lib64/librt.so.1 (0x0000003ac8000000)
>         libgfortran.so.1 => /usr/lib64/libgfortran.so.1 (0x00002b5942857000)
>         libm.so.6 => /lib64/libm.so.6 (0x0000003ac7000000)
>         libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x0000003acac00000)
>         libc.so.6 => /lib64/libc.so.6 (0x0000003ac6800000)
>         /lib64/ld-linux-x86-64.so.2 (0x0000003ac5800000)

Looks fine, infiniband libraries are linked.

> 
> mfile is a node list :
> [Thu May 31 21:20:28]user39 at m1:Linux_x86_64_CBLAS_openmpi$ cat mfile
> c04b19
> c04b20

Looks fine.

> 
> I already send osu_latency and osu_bw result in the mail.
> And when i run mvapich2 built HPL, i can't see big RX/TX change of
> ifconfig's output. ifconfig shows 1.1G of network data flow with  MPICH2
> built HPL. Why ?

I noticed this after sending the initial email, the results look
correct.  When using MPICH2 (ethernet) you should see much more RX/TX
change in ifconfig compared to MVAPICH2 (infiniband) because ifconfig
shows the ethernet devices.

> 
> Thanks for you reply, very appreciated.

No problem.

> 
> 
> On Thu, May 31, 2012 at 8:50 PM, Jonathan Perkins <
> perkinjo at cse.ohio-state.edu> wrote:
> 
> > Hello, can you try running the osu_latency test (the
> > osu-micro-benchmarks also now contains collective benchmarks if you'd
> > like to try those out)?  It may be easier to tell which network it is
> > using that way.  You can also try
> >
> >    ldd /path/to/application
> >
> > to make sure it is linked to the correct library (assuming you built
> > with shared libraries).
> >
> > What are the contents of mfile?
> >
> >
> > On Thu, May 31, 2012 at 12:16:30PM +0800, Ju JiaJia wrote:
> > > I have run osu_latency and osu_bw with both mvapich2 and mpich2 on two
> > > nodes, here is my test result
> > >
> > >
> > MVAPICH2(Infiniband）MPICH2（Ethernet）01.4515.1611.4915.2521.4915.1541.4915.15
> > >
> > 81.515.17161.5315.17321.5716.4641.6616.881281.8317.82562.920.395123.1426.06
> > >
> > 1K3.847.52K4.89193.974K5.99228.978K7.85233.1216K12.51241.3332K18.08432.3164K
> > >
> > 28.7675.25128K50.021429.21256K92.572582.83512K177.654693.171M347.759110.472M
> > > 688.1217942.074M1373.935609.21
> > >
> > > Here  is HPL results on two nodes:
> > > HPL.out.ethernet.211:   WR00L2C2       20000   128     2     1
> >  *
> > > 1686.40*              3.163e+00
> > > HPL.out.ib.211:            WR00L2C2       20000   128     2     1
> > > * 1639.99 *             3.252e+00
> > >
> > > run script :
> > > export MV2_ENABLE_AFFINITY=0
> > > PATH=/gos4/user39/jujj/program-files/mvapich2/bin:$PATH
> > >
> > > #
> > > mpiexec -f mfile -n 2 taskset -c 0 ./xhpl > HPL.out.ib.211 2>&1
> > > mpiexec -f mfile -n 2 taskset -c 0 ./xhpl_ethernet > HPL.out.ethernet.211
> > > 2>&1
> > >
> > > As you can see, no big difference. I tested NAMD also, no big difference.
> > > Is there any tools like netstat over infiniband ? So i can see whether
> > > connections is built or which network is being used. Or MVAPICH2 support
> > > some ways to do this? Log or something?
> > >
> > > On Thu, May 31, 2012 at 11:43 AM, Dhabaleswar Panda <
> > > panda at cse.ohio-state.edu> wrote:
> > >
> > > > After installing MVAPICH2, you can run OSU MPI Micro-Benchmarks to
> > verify
> > > > that your installtion is correct. You can verify your performance with
> > the
> > > > performance numbers/graphs available at the MVAPICH site. After that
> > you
> > > > can carry out your applications-level study.
> > > >
> > > > DK
> > > >
> > > > On Thu, 31 May 2012, Ju JiaJia wrote:
> > > >
> > > > > Hi all:
> > > > >
> > > > > I am currently running HPL built using mvapich2 which use infiniband,
> > > > > but it shows no apparently difference, compares in performance to a
> > > > > build using mpich2 which uses Ethenet.
> > > > >
> > > > > There should be some improvement in performance. I doubt whether
> > > > > infiniband is working. Anyone knows how to check whether infiniband
> > is
> > > > > working ?
> > > > >
> > > > >
> > > > > Ju JiaJia
> > > > >
> > > >
> > > >
> >
> > > _______________________________________________
> > > mvapich-discuss mailing list
> > > mvapich-discuss at cse.ohio-state.edu
> > > http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss
> >
> >
> > --
> > Jonathan Perkins
> > http://www.cse.ohio-state.edu/~perkinjo
> >

-- 
Jonathan Perkins
http://www.cse.ohio-state.edu/~perkinjo