[mvapich-discuss] Bad performance of IVSHMEM with OSU benchmarks

[Xiaoyi Lu]

Hi, Maksym,

Thanks for your feedback.

Can you please let us know your system configurations? Like KVM, QEMU versions, cpu info, memory size, HCA, etc. How many VMs and number of processes per VM are run on your system?

Thanks,
Xiaoyi

> On Jul 11, 2017, at 2:59 PM, Maksym Planeta <mplaneta at os.inf.tu-dresden.de> wrote:
> 
> Hello,
> 
> I'm testing MVAPICH-Virt with ivshmem and get following results:
> 
> /opt/mvapich2/virt/2.2/libexec/osu-micro-benchmarks/mpi/pt2pt/osu_latency
> 
> |    Size | MV2_VIRT_USE_IVSHMEM=1 | MV2_VIRT_USE_IVSHMEM=0 | Different hosts |
> |---------+------------------------+------------------------+-----------------|
> |       0 |                   1.66 |                   2.24 |            2.37 |
> |       1 |                   1.85 |                   2.28 |            2.36 |
> |       2 |                   1.84 |                   2.28 |            2.35 |
> |       4 |                   1.81 |                   2.28 |            2.35 |
> |       8 |                   1.81 |                   2.29 |            2.35 |
> |      16 |                   2.08 |                   2.29 |            2.35 |
> |      32 |                   2.11 |                   2.33 |            2.38 |
> |      64 |                   2.55 |                   2.41 |            2.47 |
> |     128 |                   3.38 |                   2.61 |            2.66 |
> |     256 |                   4.98 |                   3.75 |            3.79 |
> |     512 |                   8.24 |                   4.22 |            4.20 |
> |    1024 |                  14.76 |                   5.10 |            5.03 |
> |    2048 |                  28.16 |                   5.92 |            5.78 |
> |    4096 |                  55.00 |                   8.01 |            7.72 |
> |    8192 |                 111.31 |                  11.81 |           11.08 |
> |   16384 |                 226.82 |                  21.75 |           20.48 |
> |   32768 |                 405.21 |                  34.28 |           31.35 |
> |   65536 |                 836.55 |                  59.46 |           52.90 |
> |  131072 |                1370.20 |                 109.42 |           96.21 |
> |  262144 |                2452.60 |                 209.56 |          182.24 |
> |  524288 |                4607.32 |                 409.25 |          354.29 |
> | 1048576 |                8931.82 |                 808.76 |          698.10 |
> | 2097152 |               17628.58 |                1619.73 |         1385.93 |
> | 4194304 |               35174.61 |                3428.74 |         2769.21 |
> 
> 
> /opt/mvapich2/2.3a/libexec/osu-micro-benchmarks/mpi/pt2pt/osu_bw
> 
> |    Size | MV2_VIRT_USE_IVSHMEM=1 | MV2_VIRT_USE_IVSHMEM=0 | Different hosts |
> |---------+------------------------+------------------------+-----------------|
> |       1 |                   0.76 |                   1.38 |            1.44 |
> |       2 |                   1.53 |                   2.81 |            2.85 |
> |       4 |                   3.08 |                   5.71 |            5.93 |
> |       8 |                   6.25 |                  11.61 |           11.52 |
> |      16 |                  11.96 |                  23.11 |           23.69 |
> |      32 |                  21.74 |                  45.55 |           46.60 |
> |      64 |                  36.88 |                  87.51 |           94.12 |
> |     128 |                  56.82 |                 176.55 |          177.99 |
> |     256 |                  77.78 |                 260.54 |          308.03 |
> |     512 |                  94.98 |                 436.69 |          522.30 |
> |    1024 |                 106.81 |                 656.44 |          783.26 |
> |    2048 |                 113.17 |                 921.42 |          381.99 |
> |    4096 |                 117.82 |                1157.44 |         1342.62 |
> |    8192 |                 119.60 |                1219.63 |         1408.23 |
> |   16384 |                 120.19 |                1200.70 |         1398.67 |
> |   32768 |                 120.66 |                1254.64 |         1457.09 |
> |   65536 |                 120.14 |                1282.70 |         1488.76 |
> |  131072 |                 120.57 |                1297.82 |         1506.97 |
> |  262144 |                 120.99 |                1305.74 |         1514.10 |
> |  524288 |                 121.15 |                1308.78 |         1521.31 |
> | 1048576 |                 121.32 |                1310.92 |         1523.65 |
> | 2097152 |                 121.41 |                1311.11 |         1524.68 |
> | 4194304 |                 121.06 |                1307.22 |         1525.95 |
> 
> 
> Please don't look at absolute numbers to closely, because I had to plug the IB boards into PCIe 4x slot (and it still underperforms). The point is that the relative difference is dramatic.
> 
> And this is how I configured ivshmem:
> 
>  <qemu:commandline>
>    <qemu:arg value='-device'/>
>    <qemu:arg value='ivshmem-plain,memdev=hostmem'/>
>    <qemu:arg value='-object'/>
>    <qemu:arg value='memory-backend-file,size=256M,share,mem-path=/dev/shm/ivshmem,id=hostmem'/>
>  </qemu:commandline>
> 
> 
> What would you recommend to configure ivshmem for better performance?
> 
> -- 
> Regards,
> Maksym Planeta
> 
> _______________________________________________
> mvapich-discuss mailing list
> mvapich-discuss at cse.ohio-state.edu
> http://mailman.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss