[mvapich-discuss] theoretical limit to pt2pt communication

[Panda, Dhabaleswar]

Hi Peter, 

Glad to know that the OSU micro-benchmark suite has been beneficial for you. 

Many different parameters within an MPI stack can be tuned to obtain best results 
for any particular benchmark or application. Please take a look at the slides of MVAPICH2 
tutorial presented at last year's MVAPICH User Group Meeting (MUG). The tutorial 
was presented during Wednesday, August 19th, afternoon.

http://mug.mvapich.cse.ohio-state.edu/mug/15/

We will be happy to work with you and provide such tuning guidelines as long as we know your 
exact requirements. If you are interested, we can have an off-line discussion with you on this. 

Also, please note that the standard QDR 40Gbps InfiniBand uses  8:10 encoding scheme. So at the 
payload level, you should be expecting to get around 32 Gbps. 

Thanks for your suggestion of adding the micro-benchmark name next to the performance graphs
on the MVAPICH website. Since the OSU micro-benchmarks have been in the community for many
years and people know about these, we have not done these to avoid redundancy. We will consider 
your request and see how to improve these performance graph pages. 

Thanks, 

DK


________________________________________
From: mvapich-discuss-bounces at cse.ohio-state.edu on behalf of Peter Steinbach [steinbach at scionics.de]
Sent: Thursday, March 17, 2016 4:33 AM
To: mvapich-discuss at cse.ohio-state.edu
Subject: [mvapich-discuss] theoretical limit to pt2pt communication

Hi to all,

our cluster is having more and more MPI users. For this, we recently had
to profile and debug the performance of our mpi installation. The OSU
benchmark suite from here:
http://mvapich.cse.ohio-state.edu/benchmarks/
was very instrumental for this process. So thank you very much for
providing it.

As a performance engineer, I however wonder what the theoretical limit
of say osu_mbr_mw is? In other words, if I get some performance X from
osu_mbr_mw, when should/must I stop researching the MPI/IB documentation
where to tweak?

To be more concrete, on our local Qlogic QDR 40Gbps network
infrastructure, osu_mbw_mr levels off at 3342.28 MBps with mvapich 2.1.
Apparently, this is the same value that is reported here:
http://mvapich.cse.ohio-state.edu/performance/pt_to_pt/
If I run the same benchmark linking to openmpi 1.10.2, osu_mbw_mr levels
off at 3961.54 MBps.

As it is very hard to correlate both numbers to the expected 5GBps that
I expect from QDR 40Gbps, I am a bit lost.

I'd appreciate your feedback!
Peter


PS. It would be nice if the plots on e.g.
http://mvapich.cse.ohio-state.edu/performance/pt_to_pt/
would connect somehow to which command in the OSU benchmarks they
correspond.
_______________________________________________
mvapich-discuss mailing list
mvapich-discuss at cse.ohio-state.edu
http://mailman.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss
-------------- next part --------------
A non-text attachment was scrubbed...
Name: winmail.dat
Type: application/ms-tnef
Size: 4855 bytes
Desc: not available
URL: <http://mailman.cse.ohio-state.edu/pipermail/mvapich-discuss/attachments/20160317/1278f744/attachment.bin>