[mvapich-discuss] Slow MPI_Put for short msgs on Infiniband

[Jeff Hammond]

It's a communication benchmark and this is what communication
benchmarks do.  Have you ever looked at the OSU MPI Benchmarks?

Jeff

On Thu, Oct 24, 2013 at 12:54 PM, Mingzhe Li
<li.2192 at buckeyemail.osu.edu> wrote:
> Hi Fujita,
>
> I took a look at your benchmark and found that the number of short messages
> issued is much larger than that of large messages. The overhead of sending
> 1024 1Byte small messages back to back is much higher than sending one 1KB
> message. That's why the latency is high for small messages in the one sided
> operation.
>
> One question I have is what's the use case for this benchmark? What kind of
> application use this pattern? Or you are just trying do some comparison
> between two sided send/recv and one sided?
>
> Thanks,
> Mingzhe
>>
>> Hi,
>>
>> I'm currently using MVAPICH2-2.0a on Midway cluster [1] at UChicago.
>> I found that MPI_Put performance for small message size was terribly bad
>> on Infiniband.
>>
>> When I run the attached benchmark program with 2 nodes, I got the
>> following result. The first number in each line is access size (in bytes)
>> and the second number is time (in seconds) to send 1MB of buffer.
>> When I launch 2 processes in a single node, MPI_Put performance is almost
>> similar to send/recv.
>>
>> I'd like to know if this is a natual (anavoidable) behavior, or if there
>> is any way to avoid/mitigate this performance penalty (e.g. by tweaking some
>> build-time/runtime parameter).
>>
>>
>> Message-based send/recv
>> 4, 0.248301
>> 8, 0.118962
>> 16, 0.213744
>> 32, 0.378181
>> 64, 0.045802
>> 128, 0.016429
>> 256, 0.013882
>> 512, 0.006235
>> 1024, 0.002326
>> 2048, 0.001569
>> 4096, 0.000832
>> 8192, 0.000414
>> 16384, 0.001361
>> 32768, 0.000745
>> 65536, 0.000486
>> 131072, 0.000365
>> 262144, 0.000305
>> 524288, 0.000272
>> 1048576, 0.000260
>> RMA-based put
>> 16, 18.282146
>> 32, 4.329981
>> 64, 1.085714
>> 128, 0.273277
>> 256, 0.070170
>> 512, 0.017509
>> 1024, 0.004376
>> 2048, 0.001390
>> 4096, 0.000537
>> 8192, 0.000314
>> 16384, 0.000525
>> 32768, 0.000360
>> 65536, 0.000278
>> 131072, 0.000240
>> 262144, 0.000230
>> 524288, 0.000228
>> 1048576, 0.000228
>>
>>
>> MVAPICH version and configuration information are as follows:
>> [hfujita at midway-login1 mpimbench]$ mpichversion
>> MVAPICH2 Version:      2.0a
>> MVAPICH2 Release date: unreleased development copy
>> MVAPICH2 Device:       ch3:mrail
>> MVAPICH2 configure:    --prefix=/software/mvapich2-2.0-el6-x86_64
>> --enable-shared
>> MVAPICH2 CC:   cc    -DNDEBUG -DNVALGRIND -O2
>> MVAPICH2 CXX: c++   -DNDEBUG -DNVALGRIND -O2
>> MVAPICH2 F77: gfortran -L/lib -L/lib   -O2
>> MVAPICH2 FC:   gfortran   -O2
>>
>> Please let me know if you need more information about the environment.
>>
>>
>> [1]: http://rcc.uchicago.edu/resources/midway_specs.html
>>
>>
>> Thanks,
>> Hajime
>>
>> _______________________________________________
>> mvapich-discuss mailing list
>> mvapich-discuss at cse.ohio-state.edu
>> http://mailman.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss
>>
>>
>
>
> _______________________________________________
> mvapich-discuss mailing list
> mvapich-discuss at cse.ohio-state.edu
> http://mailman.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss
>



-- 
Jeff Hammond
jeff.science at gmail.com