[mvapich-discuss] Verify the application is really running over IB

[Jeff Squyres]

On Aug 29, 2007, at 12:25 PM, wgy at altair.com.cn wrote:

> Hello, Jeff:
> The mvapich version is OSU mvapich0.95.
> does it mean that it is Cisco IB stack and therefor the application  
> I run
> with mvapich is really running over IB network?

The version of MVAPICH, by itself, does not mean that it is or is not  
running over IB.

What *implies* that you are running over IB is:

- You implied that you are using the MVAPICH shipped with the Topspin  
IB stack (which is not OFED).  Is that correct?
- I *believe* that the Topspin MVAPICH did not have TCP support  
compiled into it (Topspin was before my time, but I am pretty sure  
that the Cisco MVAPICH shipped with the Cisco IB stack does not)

What would *prove* that you are using IB (vs. gige) is:

- Run a simple latency test, as Dr. Panda suggested.  Your latency  
should be single-digit microseconds (exact numbers depend on your  
hardware -- this might be all older stuff since you mentioned  
"Topspin", not "Cisco"; Topspin was acquired by Cisco quite a while  
ago...).  If your latency is much higher than that (e.g., 50 us),  
you're using gige.



> Thanks.
>
> Henry, Wu.
> | In addition to what Dr. Panda said, Cisco recommends that all HPC
> | customers upgrade to the OFED IB driver stack if possible (some
> | customers cannot upgrade for various reasons).  FWIW: all new HPC/ 
> MPI
> | work is occurring in the OFED arena.
> |
> | I bring this up because you specifically mention Topspin Infiniband,
> | which I'm *assuming* is the Cisco IB stack (not the OFED IB stack),
> | and is therefore shipping with a somewhat older version of MVAPICH
> | that was derived from the OSU MVAPICH.  The Cisco MVAPICH should  
> only
> | be compiled with IB support enabled; a simple latency test should
> | prove that you're running over IB and not ethernet.
> |
> | Much more recent versions of MPI implementations are included with
> | the OFED stack (Cisco provides binary distributions of OFED on
> | www.cisco.com).
> |
> |
> | On Aug 29, 2007, at 11:44 AM, Dhabaleswar Panda wrote:
> |
> |>
> |>
> |> On Wed, 29 Aug 2007 wgy at altair.com.cn wrote:
> |>
> |>> Hello, list:
> |>> It might be a silly questions but I wonder how to verify run with
> |>> mvapich
> |>> (come with Topspin Infiniband) over Infiniband, NOT Gigabite  
> network.
> |>> Is there an option to force mvapich to use IB network otherwise
> |>> just exits?
> |>
> |> MVAPICH has several underlying interfaces: Gen2, uDAPL, VAPI, TCP/
> |> IP and
> |> shared memory. Please take a look at the user guide (available from
> |> mvapich project page) to see the differences and capabilities of  
> these
> |> interfaces. Gen2 interface (corresponding to OFED) will give you
> |> the best performance and scalability. If you have OFED stack
> |> installed,
> |> you should be able to configure mvapich to run over Gen2 interface
> |> (as per the instructions indicated in the user guide). During OFED
> |> installation, you can also select mvapich from the package.
> |>
> |> On your existing installation, you can also run OSU benchmarks  
> (such
> |> as OSU latency). If you get latency number in the range of 2~4
> |> microsec
> |> for short messages (say 4 bytes), it is already running over the
> |> native
> |> IB.
> |>
> |> Hope this helps.
> |>
> |> DK
> |>
> |>> Thanks for your suggestion.
> |>> Rdgs.
> |>> Henry, Wu
> |>>
> |>> _______________________________________________
> |>> mvapich-discuss mailing list
> |>> mvapich-discuss at cse.ohio-state.edu
> |>> http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss
> |>>
> |>
> |> _______________________________________________
> |> mvapich-discuss mailing list
> |> mvapich-discuss at cse.ohio-state.edu
> |> http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss
> |
> |
> | --
> | Jeff Squyres
> | Cisco Systems
> |
> |


-- 
Jeff Squyres
Cisco Systems