[mvapich-discuss]Infiniband - how do I know that it is working?

Thu May 31 10:09:07 EDT 2012

Ok, Thanks for your reply, very appreciated.

On Thu, May 31, 2012 at 9:52 PM, Jonathan Perkins <
perkinjo at cse.ohio-state.edu> wrote:

> On Thu, May 31, 2012 at 09:46:50PM +0800, Ju JiaJia wrote:
> > I am trying OpenMPI, but the result is the same. So i'm testing HPL with
> a
> > large dataset, still running. 20000 is a little small.
> >
> > ldd shows :
> > [Thu May 31 21:17:26]user39 at m1:Linux_x86_64_CBLAS_openmpi$ ldd
> > xhpl_mvapich2
> >         libpthread.so.0 => /lib64/libpthread.so.0 (0x0000003ac7400000)
> >         librdmacm.so.1 => /usr/lib64/librdmacm.so.1 (0x0000003165a00000)
> >         libibverbs.so.1 => /usr/lib64/libibverbs.so.1
> (0x0000003165600000)
> >         libibumad.so.3 => /usr/lib64/libibumad.so.3 (0x00002b5942650000)
> >         libdl.so.2 => /lib64/libdl.so.2 (0x0000003ac6c00000)
> >         librt.so.1 => /lib64/librt.so.1 (0x0000003ac8000000)
> >         libgfortran.so.1 => /usr/lib64/libgfortran.so.1
> (0x00002b5942857000)
> >         libm.so.6 => /lib64/libm.so.6 (0x0000003ac7000000)
> >         libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x0000003acac00000)
> >         libc.so.6 => /lib64/libc.so.6 (0x0000003ac6800000)
> >         /lib64/ld-linux-x86-64.so.2 (0x0000003ac5800000)
>
> Looks fine, infiniband libraries are linked.
>
> >
> > mfile is a node list :
> > [Thu May 31 21:20:28]user39 at m1:Linux_x86_64_CBLAS_openmpi$ cat mfile
> > c04b19
> > c04b20
>
> Looks fine.
>
> >
> > I already send osu_latency and osu_bw result in the mail.
> > And when i run mvapich2 built HPL, i can't see big RX/TX change of
> > ifconfig's output. ifconfig shows 1.1G of network data flow with  MPICH2
> > built HPL. Why ?
>
> I noticed this after sending the initial email, the results look
> correct.  When using MPICH2 (ethernet) you should see much more RX/TX
> change in ifconfig compared to MVAPICH2 (infiniband) because ifconfig
> shows the ethernet devices.
>
> >
> > Thanks for you reply, very appreciated.
>
> No problem.
>
> >
> >
> > On Thu, May 31, 2012 at 8:50 PM, Jonathan Perkins <
> > perkinjo at cse.ohio-state.edu> wrote:
> >
> > > Hello, can you try running the osu_latency test (the
> > > osu-micro-benchmarks also now contains collective benchmarks if you'd
> > > like to try those out)?  It may be easier to tell which network it is
> > > using that way.  You can also try
> > >
> > >    ldd /path/to/application
> > >
> > > to make sure it is linked to the correct library (assuming you built
> > > with shared libraries).
> > >
> > > What are the contents of mfile?
> > >
> > >
> > > On Thu, May 31, 2012 at 12:16:30PM +0800, Ju JiaJia wrote:
> > > > I have run osu_latency and osu_bw with both mvapich2 and mpich2 on
> two
> > > > nodes, here is my test result
> > > >
> > > >
> > >
> MVAPICH2(Infiniband）MPICH2（Ethernet）01.4515.1611.4915.2521.4915.1541.4915.15
> > > >
> > >
> 81.515.17161.5315.17321.5716.4641.6616.881281.8317.82562.920.395123.1426.06
> > > >
> > >
> 1K3.847.52K4.89193.974K5.99228.978K7.85233.1216K12.51241.3332K18.08432.3164K
> > > >
> > >
> 28.7675.25128K50.021429.21256K92.572582.83512K177.654693.171M347.759110.472M
> > > > 688.1217942.074M1373.935609.21
> > > >
> > > > Here  is HPL results on two nodes:
> > > > HPL.out.ethernet.211:   WR00L2C2       20000   128     2     1
> > >  *
> > > > 1686.40*              3.163e+00
> > > > HPL.out.ib.211:            WR00L2C2       20000   128     2     1
> > > > * 1639.99 *             3.252e+00
> > > >
> > > > run script :
> > > > export MV2_ENABLE_AFFINITY=0
> > > > PATH=/gos4/user39/jujj/program-files/mvapich2/bin:$PATH
> > > >
> > > > #
> > > > mpiexec -f mfile -n 2 taskset -c 0 ./xhpl > HPL.out.ib.211 2>&1
> > > > mpiexec -f mfile -n 2 taskset -c 0 ./xhpl_ethernet >
> HPL.out.ethernet.211
> > > > 2>&1
> > > >
> > > > As you can see, no big difference. I tested NAMD also, no big
> difference.
> > > > Is there any tools like netstat over infiniband ? So i can see
> whether
> > > > connections is built or which network is being used. Or MVAPICH2
> support
> > > > some ways to do this? Log or something?
> > > >
> > > > On Thu, May 31, 2012 at 11:43 AM, Dhabaleswar Panda <
> > > > panda at cse.ohio-state.edu> wrote:
> > > >
> > > > > After installing MVAPICH2, you can run OSU MPI Micro-Benchmarks to
> > > verify
> > > > > that your installtion is correct. You can verify your performance
> with
> > > the
> > > > > performance numbers/graphs available at the MVAPICH site. After
> that
> > > you
> > > > > can carry out your applications-level study.
> > > > >
> > > > > DK
> > > > >
> > > > > On Thu, 31 May 2012, Ju JiaJia wrote:
> > > > >
> > > > > > Hi all:
> > > > > >
> > > > > > I am currently running HPL built using mvapich2 which use
> infiniband,
> > > > > > but it shows no apparently difference, compares in performance
> to a
> > > > > > build using mpich2 which uses Ethenet.
> > > > > >
> > > > > > There should be some improvement in performance. I doubt whether
> > > > > > infiniband is working. Anyone knows how to check whether
> infiniband
> > > is
> > > > > > working ?
> > > > > >
> > > > > >
> > > > > > Ju JiaJia
> > > > > >
> > > > >
> > > > >
> > >
> > > > _______________________________________________
> > > > mvapich-discuss mailing list
> > > > mvapich-discuss at cse.ohio-state.edu
> > > > http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss
> > >
> > >
> > > --
> > > Jonathan Perkins
> > > http://www.cse.ohio-state.edu/~perkinjo
> > >
>
> --
> Jonathan Perkins
> http://www.cse.ohio-state.edu/~perkinjo
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.cse.ohio-state.edu/pipermail/mvapich-discuss/attachments/20120531/d9c21290/attachment.html