[mvapich-discuss] Verify the application is really running over IB

[Jeff Squyres]

On Aug 29, 2007, at 12:54 PM, wgy at altair.com.cn wrote:

> Yes, I think I used mavapich shipped with Topspin, but I am not sure
> unless I  know how to verify it.

If it's in the /usr/local/topspin directory, it's the Topspin (later  
Cisco) MVAPICH.

> about latency test, I downloaded
> https://mvapich.cse.ohio-state.edu/svn/mpi/mvapich/trunk/ 
> osu_benchmarks/osu_latency.c
> and will compile it to run a benchmark. Can you please tell me how  
> should
> I run it? how many nodes should be used and how many cpus should be
> involved?

You typically run it with 2 MPI processes; one on each host.  It  
measures the MPI network latency between those two hosts.

> I have 4-cores nodes here..
> I would expect to run it as:
> /usr/local/topspin/mpi/mpich/bin/mpirun_ssh -np 2 -hostfile hosts

^^ Is that the right path?  Or is it "mvapich"?  Regardless, I think  
wherever you find mpirun_ssh under /usr/local/topspin/mpi is probably  
the right one.

> osu_latency.o

Is your executable really named osu_latency.o?  That's uncommon.   
Regardless, run the executable that you got when you compiled  
osu_latency.c with mpicc.

> and include the following in the hosts file
> hpc-node-01
> hpc-node-02

Sounds right.  I'm not an MVAPICH expert, though -- so I defer to the  
maintainers here on this list for the finer details...

> Is it right?
> Thanks a  lot, I am really a newbie with Infiniband....

If this is your own system, I do want to stress that OFED is really  
the way to go with HPC InfiniBand installations these days.  The  
MPI's that are included are much more recent, and all new development  
work is happening in the OFED arena.

I recommend that you upgrade if you can.


> Henry, Wu
>
>
> | On Aug 29, 2007, at 12:25 PM, wgy at altair.com.cn wrote:
> |
> |> Hello, Jeff:
> |> The mvapich version is OSU mvapich0.95.
> |> does it mean that it is Cisco IB stack and therefor the application
> |> I run
> |> with mvapich is really running over IB network?
> |
> | The version of MVAPICH, by itself, does not mean that it is or is  
> not
> | running over IB.
> |
> | What *implies* that you are running over IB is:
> |
> | - You implied that you are using the MVAPICH shipped with the  
> Topspin
> | IB stack (which is not OFED).  Is that correct?
> | - I *believe* that the Topspin MVAPICH did not have TCP support
> | compiled into it (Topspin was before my time, but I am pretty sure
> | that the Cisco MVAPICH shipped with the Cisco IB stack does not)
> |
> | What would *prove* that you are using IB (vs. gige) is:
> |
> | - Run a simple latency test, as Dr. Panda suggested.  Your latency
> | should be single-digit microseconds (exact numbers depend on your
> | hardware -- this might be all older stuff since you mentioned
> | "Topspin", not "Cisco"; Topspin was acquired by Cisco quite a while
> | ago...).  If your latency is much higher than that (e.g., 50 us),
> | you're using gige.
> |
> |
> |
> |> Thanks.
> |>
> |> Henry, Wu.
> |> | In addition to what Dr. Panda said, Cisco recommends that all HPC
> |> | customers upgrade to the OFED IB driver stack if possible (some
> |> | customers cannot upgrade for various reasons).  FWIW: all new  
> HPC/
> |> MPI
> |> | work is occurring in the OFED arena.
> |> |
> |> | I bring this up because you specifically mention Topspin  
> Infiniband,
> |> | which I'm *assuming* is the Cisco IB stack (not the OFED IB  
> stack),
> |> | and is therefore shipping with a somewhat older version of  
> MVAPICH
> |> | that was derived from the OSU MVAPICH.  The Cisco MVAPICH should
> |> only
> |> | be compiled with IB support enabled; a simple latency test should
> |> | prove that you're running over IB and not ethernet.
> |> |
> |> | Much more recent versions of MPI implementations are included  
> with
> |> | the OFED stack (Cisco provides binary distributions of OFED on
> |> | www.cisco.com).
> |> |
> |> |
> |> | On Aug 29, 2007, at 11:44 AM, Dhabaleswar Panda wrote:
> |> |
> |> |>
> |> |>
> |> |> On Wed, 29 Aug 2007 wgy at altair.com.cn wrote:
> |> |>
> |> |>> Hello, list:
> |> |>> It might be a silly questions but I wonder how to verify run  
> with
> |> |>> mvapich
> |> |>> (come with Topspin Infiniband) over Infiniband, NOT Gigabite
> |> network.
> |> |>> Is there an option to force mvapich to use IB network otherwise
> |> |>> just exits?
> |> |>
> |> |> MVAPICH has several underlying interfaces: Gen2, uDAPL, VAPI,  
> TCP/
> |> |> IP and
> |> |> shared memory. Please take a look at the user guide  
> (available from
> |> |> mvapich project page) to see the differences and capabilities of
> |> these
> |> |> interfaces. Gen2 interface (corresponding to OFED) will give you
> |> |> the best performance and scalability. If you have OFED stack
> |> |> installed,
> |> |> you should be able to configure mvapich to run over Gen2  
> interface
> |> |> (as per the instructions indicated in the user guide). During  
> OFED
> |> |> installation, you can also select mvapich from the package.
> |> |>
> |> |> On your existing installation, you can also run OSU benchmarks
> |> (such
> |> |> as OSU latency). If you get latency number in the range of 2~4
> |> |> microsec
> |> |> for short messages (say 4 bytes), it is already running over the
> |> |> native
> |> |> IB.
> |> |>
> |> |> Hope this helps.
> |> |>
> |> |> DK
> |> |>
> |> |>> Thanks for your suggestion.
> |> |>> Rdgs.
> |> |>> Henry, Wu
> |> |>>
> |> |>> _______________________________________________
> |> |>> mvapich-discuss mailing list
> |> |>> mvapich-discuss at cse.ohio-state.edu
> |> |>> http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss
> |> |>>
> |> |>
> |> |> _______________________________________________
> |> |> mvapich-discuss mailing list
> |> |> mvapich-discuss at cse.ohio-state.edu
> |> |> http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss
> |> |
> |> |
> |> | --
> |> | Jeff Squyres
> |> | Cisco Systems
> |> |
> |> |
> |
> |
> | --
> | Jeff Squyres
> | Cisco Systems
> |
> |


-- 
Jeff Squyres
Cisco Systems