[mvapich-discuss] Verify the application is really running over IB

[wgy at altair.com.cn]

Hello, Jeff:
thanks for your help!
Yes, I think I used mavapich shipped with Topspin, but I am not sure
unless I  know how to verify it.
about latency test, I downloaded
https://mvapich.cse.ohio-state.edu/svn/mpi/mvapich/trunk/osu_benchmarks/osu_latency.c
and will compile it to run a benchmark. Can you please tell me how should
I run it? how many nodes should be used and how many cpus should be
involved?
I have 4-cores nodes here..
I would expect to run it as:
/usr/local/topspin/mpi/mpich/bin/mpirun_ssh -np 2 -hostfile hosts
osu_latency.o
and include the following in the hosts file
hpc-node-01
hpc-node-02
Is it right?
Thanks a  lot, I am really a newbie with Infiniband....
Henry, Wu


| On Aug 29, 2007, at 12:25 PM, wgy at altair.com.cn wrote:
|
|> Hello, Jeff:
|> The mvapich version is OSU mvapich0.95.
|> does it mean that it is Cisco IB stack and therefor the application
|> I run
|> with mvapich is really running over IB network?
|
| The version of MVAPICH, by itself, does not mean that it is or is not
| running over IB.
|
| What *implies* that you are running over IB is:
|
| - You implied that you are using the MVAPICH shipped with the Topspin
| IB stack (which is not OFED).  Is that correct?
| - I *believe* that the Topspin MVAPICH did not have TCP support
| compiled into it (Topspin was before my time, but I am pretty sure
| that the Cisco MVAPICH shipped with the Cisco IB stack does not)
|
| What would *prove* that you are using IB (vs. gige) is:
|
| - Run a simple latency test, as Dr. Panda suggested.  Your latency
| should be single-digit microseconds (exact numbers depend on your
| hardware -- this might be all older stuff since you mentioned
| "Topspin", not "Cisco"; Topspin was acquired by Cisco quite a while
| ago...).  If your latency is much higher than that (e.g., 50 us),
| you're using gige.
|
|
|
|> Thanks.
|>
|> Henry, Wu.
|> | In addition to what Dr. Panda said, Cisco recommends that all HPC
|> | customers upgrade to the OFED IB driver stack if possible (some
|> | customers cannot upgrade for various reasons).  FWIW: all new HPC/
|> MPI
|> | work is occurring in the OFED arena.
|> |
|> | I bring this up because you specifically mention Topspin Infiniband,
|> | which I'm *assuming* is the Cisco IB stack (not the OFED IB stack),
|> | and is therefore shipping with a somewhat older version of MVAPICH
|> | that was derived from the OSU MVAPICH.  The Cisco MVAPICH should
|> only
|> | be compiled with IB support enabled; a simple latency test should
|> | prove that you're running over IB and not ethernet.
|> |
|> | Much more recent versions of MPI implementations are included with
|> | the OFED stack (Cisco provides binary distributions of OFED on
|> | www.cisco.com).
|> |
|> |
|> | On Aug 29, 2007, at 11:44 AM, Dhabaleswar Panda wrote:
|> |
|> |>
|> |>
|> |> On Wed, 29 Aug 2007 wgy at altair.com.cn wrote:
|> |>
|> |>> Hello, list:
|> |>> It might be a silly questions but I wonder how to verify run with
|> |>> mvapich
|> |>> (come with Topspin Infiniband) over Infiniband, NOT Gigabite
|> network.
|> |>> Is there an option to force mvapich to use IB network otherwise
|> |>> just exits?
|> |>
|> |> MVAPICH has several underlying interfaces: Gen2, uDAPL, VAPI, TCP/
|> |> IP and
|> |> shared memory. Please take a look at the user guide (available from
|> |> mvapich project page) to see the differences and capabilities of
|> these
|> |> interfaces. Gen2 interface (corresponding to OFED) will give you
|> |> the best performance and scalability. If you have OFED stack
|> |> installed,
|> |> you should be able to configure mvapich to run over Gen2 interface
|> |> (as per the instructions indicated in the user guide). During OFED
|> |> installation, you can also select mvapich from the package.
|> |>
|> |> On your existing installation, you can also run OSU benchmarks
|> (such
|> |> as OSU latency). If you get latency number in the range of 2~4
|> |> microsec
|> |> for short messages (say 4 bytes), it is already running over the
|> |> native
|> |> IB.
|> |>
|> |> Hope this helps.
|> |>
|> |> DK
|> |>
|> |>> Thanks for your suggestion.
|> |>> Rdgs.
|> |>> Henry, Wu
|> |>>
|> |>> _______________________________________________
|> |>> mvapich-discuss mailing list
|> |>> mvapich-discuss at cse.ohio-state.edu
|> |>> http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss
|> |>>
|> |>
|> |> _______________________________________________
|> |> mvapich-discuss mailing list
|> |> mvapich-discuss at cse.ohio-state.edu
|> |> http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss
|> |
|> |
|> | --
|> | Jeff Squyres
|> | Cisco Systems
|> |
|> |
|
|
| --
| Jeff Squyres
| Cisco Systems
|
|