[mvapich-discuss] Announcing the release of MVAPICH2 0.9.5 with
SRQ, intergrated multi-rail and TotalView support
Sayantan Sur
surs at cse.ohio-state.edu
Wed Sep 6 17:49:08 EDT 2006
Pasha,
Pavel Shamis (Pasha) wrote:
> You measurements are absolutely correct the difference in IB send/recv
> latency is much bigger. But in most cases in mvapich small messages
> will be send via fast_path that doesn't effected by SRQ performance,
> is not it ?
That is correct, small messages will be sent over RDMA but only as long
as RDMA buffers are available. I don't expect to see any impact in the
latency numbers (since its ping-pong), but the OSU bandwidth numbers
will be adversely affected. This is because, the OSU bandwidth test
window is larger than the number of available RDMA buffers. Increasing
the number of RDMA buffers per connection is not an option since it will
hurt scalability. This is the reason we do not recommend the use of SRQ
on PCI-X HCAs.
I am posting my results on the same platform at the MPI level. Hopefully
this will make things clear ... If you are seeing different numbers than
these, please let us know.
Thanks,
Sayantan.
# OSU MPI Bandwidth Test (Version 2.2)
# Size SRQ Send/Recv %worse
1 0.117846 0.255632 53.910
2 0.237887 0.536440 55.660
4 0.477906 1.073195 55.470
8 0.953037 2.153523 55.750
16 1.896051 4.294762 55.860
32 3.809666 8.591325 55.660
64 7.627845 17.238332 55.760
128 15.363553 35.628235 56.880
256 30.677627 72.093637 57.450
512 63.313689 167.576966 62.220
1024 125.406150 347.542027 63.920
2048 226.807406 510.902358 55.610
4096 334.948380 551.765944 39.300
8192 436.473830 576.432043 24.290
16384 425.040940 646.770085 34.290
32768 594.262397 748.541752 20.620
65536 735.887993 812.637004 9.4500
131072 808.396382 848.289538 4.7100
262144 847.907978 866.419436 2.1400
524288 685.423589 691.632904 .9000
1048576 660.902806 663.661603 .4200
2097152 661.124665 662.326883 .1900
4194304 662.446692 663.010765 .0900
# OSU MPI Bidirectional Bandwidth Test (Version 2.2)
# Size SRQ Send/Recv %worse
1 0.150586 0.249202 39.580
2 0.295947 0.501528 41.000
4 0.597914 1.007537 40.660
8 1.144198 1.947434 41.250
16 2.506916 4.083260 38.610
32 4.698972 7.998438 41.260
64 9.957578 16.557188 39.860
128 18.561022 33.342152 44.340
256 38.699467 67.938298 43.040
512 75.074174 144.690246 48.120
1024 155.202956 264.466012 41.320
2048 283.579797 417.127854 32.020
4096 448.992036 489.033570 8.1900
8192 573.236680 597.162757 4.0100
16384 497.970271 670.209325 25.700
32768 661.311806 778.619045 15.070
65536 777.511273 851.073708 8.6500
131072 853.118544 888.188573 3.9500
262144 887.744448 919.956078 3.5100
524288 872.973892 886.560527 1.5400
1048576 832.791313 871.912553 4.4900
2097152 745.726338 749.759917 .5400
4194304 742.594247 742.950993 .0500
# OSU MPI Latency Test (Version 2.2)
# Size SRQ Send/Recv %worse
0 4.81 4.78 .6300
1 4.85 4.83 .4200
2 4.85 4.83 .4200
4 4.85 4.83 .4200
8 4.93 4.91 .4100
16 4.94 4.91 .6100
32 5.06 5.01 .9900
64 5.08 5.04 .7900
128 6.17 6.13 .6500
256 6.86 6.78 1.1700
512 7.76 7.65 1.4200
1024 9.67 9.51 1.6600
2048 12.37 12.16 1.7000
4096 15.90 15.46 2.7700
8192 27.42 26.47 3.4700
16384 71.51 51.67 27.7500
32768 92.89 70.70 23.8900
65536 134.55 111.19 17.3700
131072 216.94 192.48 11.2800
262144 384.51 362.96 5.6100
524288 800.62 764.50 4.5200
1048576 1575.04 1505.64 4.4100
2097152 3099.57 2979.01 3.8900
4194304 6144.59 5925.67 3.5700
--
http://www.cse.ohio-state.edu/~surs
More information about the mvapich-discuss
mailing list