[mvapich-discuss] theoretical limit to pt2pt communication

[Peter Steinbach]

Hi to all,

our cluster is having more and more MPI users. For this, we recently had 
to profile and debug the performance of our mpi installation. The OSU 
benchmark suite from here:
http://mvapich.cse.ohio-state.edu/benchmarks/
was very instrumental for this process. So thank you very much for 
providing it.

As a performance engineer, I however wonder what the theoretical limit 
of say osu_mbr_mw is? In other words, if I get some performance X from 
osu_mbr_mw, when should/must I stop researching the MPI/IB documentation 
where to tweak?

To be more concrete, on our local Qlogic QDR 40Gbps network 
infrastructure, osu_mbw_mr levels off at 3342.28 MBps with mvapich 2.1. 
Apparently, this is the same value that is reported here:
http://mvapich.cse.ohio-state.edu/performance/pt_to_pt/
If I run the same benchmark linking to openmpi 1.10.2, osu_mbw_mr levels 
off at 3961.54 MBps.

As it is very hard to correlate both numbers to the expected 5GBps that 
I expect from QDR 40Gbps, I am a bit lost.

I'd appreciate your feedback!
Peter


PS. It would be nice if the plots on e.g.
http://mvapich.cse.ohio-state.edu/performance/pt_to_pt/
would connect somehow to which command in the OSU benchmarks they 
correspond.