[mvapich-discuss] OSU benchmarks interpretation

[Nikita Andreev]

I'm benchmarking bandwidth between two compute nodes equipped with Mellanox
ConnectX DDR InfiniBand two-port HCAs. I run benchmarks under OpenMPI which
supports dual-rail configurations.

 

Results for message size 4194304:

 

osu_bw               4917.75 MB/s

osu_bibw           5007.49 MB/s

osu_put_bw     3489.35 MB/s

osu_put_bibw  3876.96 MB/s

osu_get_bw      3482.18 MB/s

 

I have several questions:

 

1. As far as I understand DDR IB has 16Gb/s data rate. Hence dual-rail has
32Gb/s or 4GB/s theoretical peak throughput. But osu_bw shows data rate
higher than theoretical. How is that possible?

 

2. osu_bw is unidirectional and osu_bibw is bidirectional test. So I suppose
it should have two times higher throughput but it's almost the same as
unidirectional.

 

3. RDMA put/get do not involve target node in operation and should be faster
than ordinary send/recv. Why are they slower?

 

Regards,

Nikita

 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.cse.ohio-state.edu/pipermail/mvapich-discuss/attachments/20110302/2b61a1ab/attachment.html