[mvapich-discuss] Slow MPI_Put for short msgs on Infiniband

[Hajime Fujita]

Hi,

I'm currently using MVAPICH2-2.0a on Midway cluster [1] at UChicago.
I found that MPI_Put performance for small message size was terribly bad 
on Infiniband.

When I run the attached benchmark program with 2 nodes, I got the 
following result. The first number in each line is access size (in 
bytes) and the second number is time (in seconds) to send 1MB of buffer.
When I launch 2 processes in a single node, MPI_Put performance is 
almost similar to send/recv.

I'd like to know if this is a natual (anavoidable) behavior, or if there 
is any way to avoid/mitigate this performance penalty (e.g. by tweaking 
some build-time/runtime parameter).


Message-based send/recv
4, 0.248301
8, 0.118962
16, 0.213744
32, 0.378181
64, 0.045802
128, 0.016429
256, 0.013882
512, 0.006235
1024, 0.002326
2048, 0.001569
4096, 0.000832
8192, 0.000414
16384, 0.001361
32768, 0.000745
65536, 0.000486
131072, 0.000365
262144, 0.000305
524288, 0.000272
1048576, 0.000260
RMA-based put
16, 18.282146
32, 4.329981
64, 1.085714
128, 0.273277
256, 0.070170
512, 0.017509
1024, 0.004376
2048, 0.001390
4096, 0.000537
8192, 0.000314
16384, 0.000525
32768, 0.000360
65536, 0.000278
131072, 0.000240
262144, 0.000230
524288, 0.000228
1048576, 0.000228


MVAPICH version and configuration information are as follows:
[hfujita at midway-login1 mpimbench]$ mpichversion
MVAPICH2 Version:     	2.0a
MVAPICH2 Release date:	unreleased development copy
MVAPICH2 Device:      	ch3:mrail
MVAPICH2 configure:   	--prefix=/software/mvapich2-2.0-el6-x86_64 
--enable-shared
MVAPICH2 CC:  	cc    -DNDEBUG -DNVALGRIND -O2
MVAPICH2 CXX: 	c++   -DNDEBUG -DNVALGRIND -O2
MVAPICH2 F77: 	gfortran -L/lib -L/lib   -O2
MVAPICH2 FC:  	gfortran   -O2

Please let me know if you need more information about the environment.


[1]: http://rcc.uchicago.edu/resources/midway_specs.html


Thanks,
Hajime

-------------- next part --------------
A non-text attachment was scrubbed...
Name: mpimbench.c
Type: text/x-csrc
Size: 3227 bytes
Desc: not available
URL: <http://mailman.cse.ohio-state.edu/pipermail/mvapich-discuss/attachments/20131023/fa0dabf5/attachment.bin>