[mvapich-discuss] Verify the application is really running
over IB
wgy at altair.com.cn
wgy at altair.com.cn
Sat Sep 1 01:48:57 EDT 2007
Hello, Jeff and Dr, Panda:
I get back with the following resluts and further ques:
1)latency test reslut:
the latency.c I downloaded from mvapich website can be complied in the
cluster with errors:
[radioss at hpc-node-01 job1]$ /usr/local/topspin/mpi/mpich/bin/mpicc
latency.c -o lat
latency.c:66: error: syntax error before "for"
latency.c:68: error: `i' undeclared here (not in a function)
latency.c:68: warning: data definition has no type or storage class
latency.c:69: error: syntax error before '}' token
latency.c:73: error: `skip_large' undeclared here (not in a function)
latency.c:73: warning: data definition has no type or storage class
latency.c:74: error: syntax error before '}' token
latency.c:76: warning: parameter names (without types) in function
declaration
latency.c:76: warning: data definition has no type or storage class
latency.c:78: error: syntax error before "if"
latency.c:82: error: syntax error before numeric constant
latency.c:82: warning: data definition has no type or storage class
latency.c:83: error: syntax error before numeric constant
latency.c:84: warning: data definition has no type or storage class
latency.c:86: error: initializer element is not constant
latency.c:86: warning: data definition has no type or storage class
latency.c:88: error: syntax error before '}' token
latency.c:92: error: syntax error before numeric constant
latency.c:92: warning: data definition has no type or storage class
latency.c:98: error: `t_start' undeclared here (not in a function)
latency.c:98: error: `loop' undeclared here (not in a function)
latency.c:98: warning: data definition has no type or storage class
latency.c:99: error: syntax error before string constant
latency.c:99: warning: conflicting types for built-in function 'fprintf'
latency.c:99: warning: data definition has no type or storage class
latency.c:104: warning: data definition has no type or storage class
latency.c:105: error: syntax error before "return"
latency.c:107:2: warning: no newline at end of file
latency.c:68: error: storage size of `r_buf' isn't know
I had to use mpi_latency.c shipped with mvapich in the cluster and got the
following latency test results.
[radioss at hpc-node-01 job1]$ /usr/local/topspin/mpi/mpich/bin/mpirun_rsh
-np 2 -hostfile appfile ./lat 10000 1
1 6.288650
[radioss at hpc-node-01 job1]$ /usr/local/topspin/mpi/mpich/bin/mpirun_rsh
-np 2 -hostfile appfile ./lat 10000 4
4 6.410350
while Topspin's Host-Side Drivers User Guide for Linux Release 3.1.0 gives
the following latency test figure as an example:
[root at qa-bc1-blade2 root]# /usr/local/topspin/mpi/mpich/bin/mpirun_ssh -np
2 qabc1-
blade2 qa-bc1-blade3 /usr/local/topspin/mpi/mpich/bin/mpi_latency 10000 1
1 6.684000
2) Jeff Squyres once asked me:
>> I have 4-cores nodes here..
>> I would expect to run it as:
>> /usr/local/topspin/mpi/mpich/bin/mpirun_ssh -np 2 -hostfile hosts
>^^ Is that the right path? Or is it "mvapich"? Regardless, I think
wherever you find mpirun_ssh under /usr/local/topspin/mpi is probably
the right one.
the path is right, and I am pretty sure it is mavapich because:
i)rpm -qf /usr/local/topspin/mpi/mpich/bin/mpirun_ssh gives:
topspin-ib-mpi-rhel4-3.2.0-118
ii)[radioss at hpc-node-01 local]$
/usr/local/topspin/mpi/mpich/bin/mpirun_rsh -v
OSU MVAPICH VERSION 0.9.5-SingleRail
3)when I try to use hp mpi 2.2.5 over the IB network I got the following:
[radioss at hpc-node-01 job1]$ /opt/hpmpi/bin/mpirun -stdio=i0
-cpu_bind=cyclic -VAPI -f appfile < PFTANKD01
dlopen test for MPI_ICLIB_VAPI__VAPI_MAIN could not open libs in list
libmtl_common.so libmpga.so libmosal.so libvapi.so:
/usr/local/topspin/lib64/libmosal.so: undefined symbol: pthread_create
dlopen test for MPI_ICLIB_VAPI__VAPI_CISCO could not open libs in list
libpthread.so libmosal.so libvapi.so: /usr/lib64/libpthread.so:
invalid ELF header
mpid: MPI BUG: VAPI requested but not available
what does it probably indicate? anything is wrong with the IB configuration?
RPM packages installed there:
[radioss at hpc-node-01 job1]$ rpm -qa|grep topspin
topspin-ib-rhel4-3.2.0-118
topspin-ib-mpi-rhel4-3.2.0-118
topspin-ib-mod-rhel4-2.6.9-42.ELsmp-3.2.0-118
4)You guys suggested me to use HP MPI (no native mvapich) and OFED IB
stack if possible.
now I have some questiosn hope you can have a quick comment or refer me
some website link so that I can read through:
i)how to verify which IB stack is used here, OFED or Cisco/Topspin IB
stack? what's the advantages of OFED IB stack over Cisco/Topspin IB stack?
ii)what's the advatages of HP HPI over "native mvapich"? what means by
"native mvapich"? the one shipped with Cisco/Topspin? is it enough to
upgrade mvapich to the latest one which is availabel on mvapich website?
Thanks a lot for all of you for your kindly help!
Henry, Wu.
| On Aug 29, 2007, at 12:54 PM, wgy at altair.com.cn wrote:
|
|> Yes, I think I used mavapich shipped with Topspin, but I am not sure
|> unless I know how to verify it.
|
| If it's in the /usr/local/topspin directory, it's the Topspin (later
| Cisco) MVAPICH.
|
|> about latency test, I downloaded
|> https://mvapich.cse.ohio-state.edu/svn/mpi/mvapich/trunk/
|> osu_benchmarks/osu_latency.c
|> and will compile it to run a benchmark. Can you please tell me how
|> should
|> I run it? how many nodes should be used and how many cpus should be
|> involved?
|
| You typically run it with 2 MPI processes; one on each host. It
| measures the MPI network latency between those two hosts.
|
|> I have 4-cores nodes here..
|> I would expect to run it as:
|> /usr/local/topspin/mpi/mpich/bin/mpirun_ssh -np 2 -hostfile hosts
|
| ^^ Is that the right path? Or is it "mvapich"? Regardless, I think
| wherever you find mpirun_ssh under /usr/local/topspin/mpi is probably
| the right one.
|
|> osu_latency.o
|
| Is your executable really named osu_latency.o? That's uncommon.
| Regardless, run the executable that you got when you compiled
| osu_latency.c with mpicc.
|
|> and include the following in the hosts file
|> hpc-node-01
|> hpc-node-02
|
| Sounds right. I'm not an MVAPICH expert, though -- so I defer to the
| maintainers here on this list for the finer details...
|
|> Is it right?
|> Thanks a lot, I am really a newbie with Infiniband....
|
| If this is your own system, I do want to stress that OFED is really
| the way to go with HPC InfiniBand installations these days. The
| MPI's that are included are much more recent, and all new development
| work is happening in the OFED arena.
|
| I recommend that you upgrade if you can.
|
|
|> Henry, Wu
|>
|>
|> | On Aug 29, 2007, at 12:25 PM, wgy at altair.com.cn wrote:
|> |
|> |> Hello, Jeff:
|> |> The mvapich version is OSU mvapich0.95.
|> |> does it mean that it is Cisco IB stack and therefor the application
|> |> I run
|> |> with mvapich is really running over IB network?
|> |
|> | The version of MVAPICH, by itself, does not mean that it is or is
|> not
|> | running over IB.
|> |
|> | What *implies* that you are running over IB is:
|> |
|> | - You implied that you are using the MVAPICH shipped with the
|> Topspin
|> | IB stack (which is not OFED). Is that correct?
|> | - I *believe* that the Topspin MVAPICH did not have TCP support
|> | compiled into it (Topspin was before my time, but I am pretty sure
|> | that the Cisco MVAPICH shipped with the Cisco IB stack does not)
|> |
|> | What would *prove* that you are using IB (vs. gige) is:
|> |
|> | - Run a simple latency test, as Dr. Panda suggested. Your latency
|> | should be single-digit microseconds (exact numbers depend on your
|> | hardware -- this might be all older stuff since you mentioned
|> | "Topspin", not "Cisco"; Topspin was acquired by Cisco quite a while
|> | ago...). If your latency is much higher than that (e.g., 50 us),
|> | you're using gige.
|> |
|> |
|> |
|> |> Thanks.
|> |>
|> |> Henry, Wu.
|> |> | In addition to what Dr. Panda said, Cisco recommends that all HPC
|> |> | customers upgrade to the OFED IB driver stack if possible (some
|> |> | customers cannot upgrade for various reasons). FWIW: all new
|> HPC/
|> |> MPI
|> |> | work is occurring in the OFED arena.
|> |> |
|> |> | I bring this up because you specifically mention Topspin
|> Infiniband,
|> |> | which I'm *assuming* is the Cisco IB stack (not the OFED IB
|> stack),
|> |> | and is therefore shipping with a somewhat older version of
|> MVAPICH
|> |> | that was derived from the OSU MVAPICH. The Cisco MVAPICH should
|> |> only
|> |> | be compiled with IB support enabled; a simple latency test should
|> |> | prove that you're running over IB and not ethernet.
|> |> |
|> |> | Much more recent versions of MPI implementations are included
|> with
|> |> | the OFED stack (Cisco provides binary distributions of OFED on
|> |> | www.cisco.com).
|> |> |
|> |> |
|> |> | On Aug 29, 2007, at 11:44 AM, Dhabaleswar Panda wrote:
|> |> |
|> |> |>
|> |> |>
|> |> |> On Wed, 29 Aug 2007 wgy at altair.com.cn wrote:
|> |> |>
|> |> |>> Hello, list:
|> |> |>> It might be a silly questions but I wonder how to verify run
|> with
|> |> |>> mvapich
|> |> |>> (come with Topspin Infiniband) over Infiniband, NOT Gigabite
|> |> network.
|> |> |>> Is there an option to force mvapich to use IB network otherwise
|> |> |>> just exits?
|> |> |>
|> |> |> MVAPICH has several underlying interfaces: Gen2, uDAPL, VAPI,
|> TCP/
|> |> |> IP and
|> |> |> shared memory. Please take a look at the user guide
|> (available from
|> |> |> mvapich project page) to see the differences and capabilities of
|> |> these
|> |> |> interfaces. Gen2 interface (corresponding to OFED) will give you
|> |> |> the best performance and scalability. If you have OFED stack
|> |> |> installed,
|> |> |> you should be able to configure mvapich to run over Gen2
|> interface
|> |> |> (as per the instructions indicated in the user guide). During
|> OFED
|> |> |> installation, you can also select mvapich from the package.
|> |> |>
|> |> |> On your existing installation, you can also run OSU benchmarks
|> |> (such
|> |> |> as OSU latency). If you get latency number in the range of 2~4
|> |> |> microsec
|> |> |> for short messages (say 4 bytes), it is already running over the
|> |> |> native
|> |> |> IB.
|> |> |>
|> |> |> Hope this helps.
|> |> |>
|> |> |> DK
|> |> |>
|> |> |>> Thanks for your suggestion.
|> |> |>> Rdgs.
|> |> |>> Henry, Wu
|> |> |>>
|> |> |>> _______________________________________________
|> |> |>> mvapich-discuss mailing list
|> |> |>> mvapich-discuss at cse.ohio-state.edu
|> |> |>> http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss
|> |> |>>
|> |> |>
|> |> |> _______________________________________________
|> |> |> mvapich-discuss mailing list
|> |> |> mvapich-discuss at cse.ohio-state.edu
|> |> |> http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss
|> |> |
|> |> |
|> |> | --
|> |> | Jeff Squyres
|> |> | Cisco Systems
|> |> |
|> |> |
|> |
|> |
|> | --
|> | Jeff Squyres
|> | Cisco Systems
|> |
|> |
|
|
| --
| Jeff Squyres
| Cisco Systems
|
|
More information about the mvapich-discuss
mailing list