[mvapich-discuss] OSU benchmarks interpretation

[Peter Kjellström]

On Wednesday, March 02, 2011 08:03:42 am Nikita Andreev wrote:
> I'm benchmarking bandwidth between two compute nodes equipped with Mellanox
> ConnectX DDR InfiniBand two-port HCAs. I run benchmarks under OpenMPI which
> supports dual-rail configurations.
> 
> Results for message size 4194304:
> 
> osu_bw               4917.75 MB/s
> osu_bibw           5007.49 MB/s
> osu_put_bw     3489.35 MB/s
> osu_put_bibw  3876.96 MB/s
> osu_get_bw      3482.18 MB/s

This is way too fast for a single DDR ConnectX you're probably running the 
test using shared memory on one node.

Expected DDR performance (one port) is roughly:
 unidir PCIe 2.5GT: 1400 MB/s
 unidir PCIe 5.0GT: 1950 MB/s
 bidir: ~2x unidir

Using both ports will on 2.5GT PCIe be pointless (can't even push one port) 
and on 5.0GT I'm guessing you'd max out at ~3000 MB/s unidir but have not 
tried it myself.

=> Multirail for performace pretty much needs two HCAs.

/Peter
 
> 
> 
> I have several questions:
> 
> 
> 
> 1. As far as I understand DDR IB has 16Gb/s data rate. Hence dual-rail has
> 32Gb/s or 4GB/s theoretical peak throughput. But osu_bw shows data rate
> higher than theoretical. How is that possible?
> 
> 
> 
> 2. osu_bw is unidirectional and osu_bibw is bidirectional test. So I
> suppose it should have two times higher throughput but it's almost the
> same as unidirectional.
> 
> 
> 
> 3. RDMA put/get do not involve target node in operation and should be
> faster than ordinary send/recv. Why are they slower?
> 
> 
> 
> Regards,
> 
> Nikita

-- 
-= Peter Kjellström
-= National Supercomputer Centre
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: This is a digitally signed message part.
Url : http://mail.cse.ohio-state.edu/pipermail/mvapich-discuss/attachments/20110302/cda18633/attachment.bin