[mvapich-discuss] mvapich: using IPoIB or direct IB?

[Michael Zebrowski]

Thanks very much Sayantan. I will give those commands a try.
By the way we are using mpirun_rsh...

Regards,
Michael

-----Original Message-----
From: Sayantan Sur [mailto:surs at cse.ohio-state.edu] 
Sent: Wednesday, July 11, 2007 3:57 PM
To: Michael Zebrowski
Cc: mvapich-discuss at cse.ohio-state.edu
Subject: Re: [mvapich-discuss] mvapich: using IPoIB or direct IB?

Hi Michael,

Michael Zebrowski wrote:
>
> Does anyone know a good way to determine whether mvapich is utilizing 
> IPoIB or direct IB during its run? I have validated that the IB packet

> counters increase significantly (using OFED's perfquery) on nodes that

> are referenced during the mpi run (and do not increase when they are 
> not referenced), but I wanted to confirm that this is an accurate 
> approach in determining this.
>
>  
>
> Also, what exactly does the 'tdevice' executable report?
>

There are multiple ways to determine this:

1) $MVAPICH/bin/mpirun_rsh -v

For the latest MVAPICH version, it should print "OSU MVAPICH VERSION 
0.9.9-SingleRail". If you use `mpirun_rsh' to launch any MPI program, 
then you can be sure that it will use direct IB for the run.

2) Build and execute the OSU benchmarks. For modern servers equipped 
with Mellanox DDR cards, you should get results around 1400 MB/s from 
osu_bw.c and around 2500 for osu_bibw.c. These numbers are possible only

using direct IB.

The perf counters are unfortunately not a good way to determine whether 
it is using direct IB or not, since in both modes (IPoIB and direct IB),

packets are sent over the IB device and counters may change.

Currently the 'tdevice' script just prints ch_p4 (which is the TCP/IP 
support from MPICH).

Please let us know if you have questions.

Thanks,
Sayantan.

>  
>
> - Michael
>
>  
>
>
> NOTICE:
> This message may contain privileged or otherwise confidential
information.
> If you are not the intended recipient, please immediately advise the
sender
> by reply email and delete the message and any attachments without
using,
> copying or disclosing the contents.
>   
>
------------------------------------------------------------------------
>
> _______________________________________________
> mvapich-discuss mailing list
> mvapich-discuss at cse.ohio-state.edu
> http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss
>   


-- 
http://www.cse.ohio-state.edu/~surs


NOTICE:
This message may contain privileged or otherwise confidential information.
If you are not the intended recipient, please immediately advise the sender
by reply email and delete the message and any attachments without using,
copying or disclosing the contents.