[mvapich-discuss] Bad performance of IVSHMEM with OSU benchmarks

[Maksym Planeta]

Hello,

I'm testing MVAPICH-Virt with ivshmem and get following results:
       
/opt/mvapich2/virt/2.2/libexec/osu-micro-benchmarks/mpi/pt2pt/osu_latency

|    Size | MV2_VIRT_USE_IVSHMEM=1 | MV2_VIRT_USE_IVSHMEM=0 | Different hosts |
|---------+------------------------+------------------------+-----------------|
|       0 |                   1.66 |                   2.24 |            2.37 |
|       1 |                   1.85 |                   2.28 |            2.36 |
|       2 |                   1.84 |                   2.28 |            2.35 |
|       4 |                   1.81 |                   2.28 |            2.35 |
|       8 |                   1.81 |                   2.29 |            2.35 |
|      16 |                   2.08 |                   2.29 |            2.35 |
|      32 |                   2.11 |                   2.33 |            2.38 |
|      64 |                   2.55 |                   2.41 |            2.47 |
|     128 |                   3.38 |                   2.61 |            2.66 |
|     256 |                   4.98 |                   3.75 |            3.79 |
|     512 |                   8.24 |                   4.22 |            4.20 |
|    1024 |                  14.76 |                   5.10 |            5.03 |
|    2048 |                  28.16 |                   5.92 |            5.78 |
|    4096 |                  55.00 |                   8.01 |            7.72 |
|    8192 |                 111.31 |                  11.81 |           11.08 |
|   16384 |                 226.82 |                  21.75 |           20.48 |
|   32768 |                 405.21 |                  34.28 |           31.35 |
|   65536 |                 836.55 |                  59.46 |           52.90 |
|  131072 |                1370.20 |                 109.42 |           96.21 |
|  262144 |                2452.60 |                 209.56 |          182.24 |
|  524288 |                4607.32 |                 409.25 |          354.29 |
| 1048576 |                8931.82 |                 808.76 |          698.10 |
| 2097152 |               17628.58 |                1619.73 |         1385.93 |
| 4194304 |               35174.61 |                3428.74 |         2769.21 |


/opt/mvapich2/2.3a/libexec/osu-micro-benchmarks/mpi/pt2pt/osu_bw

|    Size | MV2_VIRT_USE_IVSHMEM=1 | MV2_VIRT_USE_IVSHMEM=0 | Different hosts |
|---------+------------------------+------------------------+-----------------|
|       1 |                   0.76 |                   1.38 |            1.44 |
|       2 |                   1.53 |                   2.81 |            2.85 |
|       4 |                   3.08 |                   5.71 |            5.93 |
|       8 |                   6.25 |                  11.61 |           11.52 |
|      16 |                  11.96 |                  23.11 |           23.69 |
|      32 |                  21.74 |                  45.55 |           46.60 |
|      64 |                  36.88 |                  87.51 |           94.12 |
|     128 |                  56.82 |                 176.55 |          177.99 |
|     256 |                  77.78 |                 260.54 |          308.03 |
|     512 |                  94.98 |                 436.69 |          522.30 |
|    1024 |                 106.81 |                 656.44 |          783.26 |
|    2048 |                 113.17 |                 921.42 |          381.99 |
|    4096 |                 117.82 |                1157.44 |         1342.62 |
|    8192 |                 119.60 |                1219.63 |         1408.23 |
|   16384 |                 120.19 |                1200.70 |         1398.67 |
|   32768 |                 120.66 |                1254.64 |         1457.09 |
|   65536 |                 120.14 |                1282.70 |         1488.76 |
|  131072 |                 120.57 |                1297.82 |         1506.97 |
|  262144 |                 120.99 |                1305.74 |         1514.10 |
|  524288 |                 121.15 |                1308.78 |         1521.31 |
| 1048576 |                 121.32 |                1310.92 |         1523.65 |
| 2097152 |                 121.41 |                1311.11 |         1524.68 |
| 4194304 |                 121.06 |                1307.22 |         1525.95 |


Please don't look at absolute numbers to closely, because I had to plug the IB boards into PCIe 4x slot (and it still underperforms). The point is that the relative difference is dramatic.

And this is how I configured ivshmem:

  <qemu:commandline>
    <qemu:arg value='-device'/>
    <qemu:arg value='ivshmem-plain,memdev=hostmem'/>
    <qemu:arg value='-object'/>
    <qemu:arg value='memory-backend-file,size=256M,share,mem-path=/dev/shm/ivshmem,id=hostmem'/>
  </qemu:commandline>


What would you recommend to configure ivshmem for better performance?

-- 
Regards,
Maksym Planeta

-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 5174 bytes
Desc: S/MIME Cryptographic Signature
URL: <http://mailman.cse.ohio-state.edu/pipermail/mvapich-discuss/attachments/20170711/30629274/attachment.p7s>