[mvapich-discuss] Slow MPI_Put for short msgs on Infiniband

[Mingzhe Li]

Hi Fujita,

I took a look at your benchmark and found that the number of short messages
issued is much larger than that of large messages. The overhead of sending
1024 1Byte small messages back to back is much higher than sending one 1KB
message. That's why the latency is high for small messages in the one sided
operation.

One question I have is what's the use case for this benchmark? What kind of
application use this pattern? Or you are just trying do some comparison
between two sided send/recv and one sided?

Thanks,
Mingzhe

> Hi,
>
> I'm currently using MVAPICH2-2.0a on Midway cluster [1] at UChicago.
> I found that MPI_Put performance for small message size was terribly bad
> on Infiniband.
>
> When I run the attached benchmark program with 2 nodes, I got the
> following result. The first number in each line is access size (in bytes)
> and the second number is time (in seconds) to send 1MB of buffer.
> When I launch 2 processes in a single node, MPI_Put performance is almost
> similar to send/recv.
>
> I'd like to know if this is a natual (anavoidable) behavior, or if there
> is any way to avoid/mitigate this performance penalty (e.g. by tweaking
> some build-time/runtime parameter).
>
>
> Message-based send/recv
> 4, 0.248301
> 8, 0.118962
> 16, 0.213744
> 32, 0.378181
> 64, 0.045802
> 128, 0.016429
> 256, 0.013882
> 512, 0.006235
> 1024, 0.002326
> 2048, 0.001569
> 4096, 0.000832
> 8192, 0.000414
> 16384, 0.001361
> 32768, 0.000745
> 65536, 0.000486
> 131072, 0.000365
> 262144, 0.000305
> 524288, 0.000272
> 1048576, 0.000260
> RMA-based put
> 16, 18.282146
> 32, 4.329981
> 64, 1.085714
> 128, 0.273277
> 256, 0.070170
> 512, 0.017509
> 1024, 0.004376
> 2048, 0.001390
> 4096, 0.000537
> 8192, 0.000314
> 16384, 0.000525
> 32768, 0.000360
> 65536, 0.000278
> 131072, 0.000240
> 262144, 0.000230
> 524288, 0.000228
> 1048576, 0.000228
>
>
> MVAPICH version and configuration information are as follows:
> [hfujita at midway-login1 mpimbench]$ mpichversion
> MVAPICH2 Version:      2.0a
> MVAPICH2 Release date: unreleased development copy
> MVAPICH2 Device:       ch3:mrail
> MVAPICH2 configure:    --prefix=/software/mvapich2-2.0-el6-x86_64
> --enable-shared
> MVAPICH2 CC:   cc    -DNDEBUG -DNVALGRIND -O2
> MVAPICH2 CXX: c++   -DNDEBUG -DNVALGRIND -O2
> MVAPICH2 F77: gfortran -L/lib -L/lib   -O2
> MVAPICH2 FC:   gfortran   -O2
>
> Please let me know if you need more information about the environment.
>
>
> [1]: http://rcc.uchicago.edu/resources/midway_specs.html
>
>
> Thanks,
> Hajime
>
>     _______________________________________________
> mvapich-discuss mailing list
> mvapich-discuss at cse.ohio-state.edu
> http://mailman.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.cse.ohio-state.edu/pipermail/mvapich-discuss/attachments/20131024/8604ed5b/attachment.html>