[Mvapich-discuss] mvapich2-3.0a and slingshot 11 - performance questions

[Ben Kirk]

Hi, I've recently installed mvapich2-3.0a with the hopes of testing
slingshot 11 support.

When ./configuring with the recommended

--with-device=ch4:ofi --with-libfabric=/opt/cray/libfabric/1.15.2.0

I am able to build the library, but find even simple pt2pt performance
exceptionally slow.  Comparing intra (inside 1 node) to inter (across 2
nodes):

#********* Intra-Node-CPU (Bare Metal) *****************

#/glade/work/benkirk/codes/dev_stack/install/osu-micro-benchmarks-6.2-mvapich2-3.0a-cray_libfabric-derecho-gcc-12.2.0/libexec/osu-micro-benchmarks/mpi/pt2pt/osu_bw

# OSU MPI Bandwidth Test v6.2

# Size      Bandwidth (MB/s)

1                       2.90

2                       6.10

4                      11.44

8                      24.28

16                     46.73

32                     93.20

64                    194.77

128                   346.13

256                   759.24

512                  1389.39

1024                 2655.62

2048                 5258.23

4096                 9112.13

8192                12012.65

16384               11235.18

32768               10769.48

65536               19743.95

131072              29157.27

262144              27715.51

524288              27497.12

1048576             13662.50

2097152             10807.83

4194304             10830.14

#********* Inter-Node-CPU (Bare Metal) *****************

#/glade/work/benkirk/codes/dev_stack/install/osu-micro-benchmarks-6.2-mvapich2-3.0a-cray_libfabric-derecho-gcc-12.2.0/libexec/osu-micro-benchmarks/mpi/pt2pt/osu_bw

# OSU MPI Bandwidth Test v6.2

# Size      Bandwidth (MB/s)

1                       0.30

2                       0.60

4                       1.20

8                       2.40

16                      4.78

32                      9.52

64                     17.50

128                    34.09

256                    54.02

512                   101.24

1024                  196.80

2048                  195.56

4096                  262.10

8192                  316.01

16384                 291.34

32768                 289.64

65536                 290.30

131072                260.94

262144                298.70

524288                342.67

1048576               295.31

2097152               351.07

4194304               334.36

Are there any debugging variables or other tricks I should be aware of?

Thanks!!

--

Ben Kirk

NCAR Computational & Information Systems Laboratory

Consulting Services Group Head
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osu.edu/pipermail/mvapich-discuss/attachments/20230503/d2555490/attachment-0006.html>