[mvapich-discuss] CPU affinity - cores being assigned twice

[Steve Rakowski]

Hello,

I am trying to set up MVAPICH2 on a small cluster of Ubuntu 14.04 servers.
Each has 2x Xeon E5-2697 v3 processors for a total of 28 cores.
HyperThreading is disabled. They are connected via Mellanox ConnectX-3 HCAs
to a Mellanox FDR switch. MVAPICH2 is 2.2b.

Tests with few cores run correctly, but attempting to use a large number of
cores on each node results in extremely poor performance and/or failure.
Some operations just hang. After some troubleshooting I may have tracked
this down to CPU affinity. The SHOW_CPU_BINDING option indicates it is
using some cores twice. (see output 1 below) If it means anything, this
system finds cores on the first physical processor as 0, 2, 4, ... and on
the second 1, 3, 5, ...

The manual states that "bunch" is the default CPU binding policy. Oddly
enough, manually setting MV2_CPU_BINDING_POLICY to "bunch" seems to work.
The operations no longer fail. However, the bundled OSU benchmarks show
"jumpy" latency measurements. (output 2 below)

The IB fabric appears to be functioning normally. I was able to connect
some older Ivy Bridge servers (also with ConnectX-3) to the same Infiniband
switch. The same benchmark works fine without the jumpy measurements.

Any thoughts on either the affinity issue or the jumpy performance problem?

Thanks.
Steve

Output 1 - failure
node1$ mpiexec -env MV2_SHOW_CPU_BINDING=1 -ppn 28 -hosts node1,node2
/opt/mpi/mvapich2/libexec/mvapich2/osu_allgather
-------------CPU AFFINITY-------------
RANK:0  CPU_SET:   0
RANK:1  CPU_SET:   2
RANK:2  CPU_SET:   4
RANK:3  CPU_SET:   6
RANK:4  CPU_SET:   8
RANK:5  CPU_SET:  10
RANK:6  CPU_SET:  12
RANK:7  CPU_SET:   0
RANK:8  CPU_SET:   2
RANK:9  CPU_SET:   4
RANK:10  CPU_SET:   6
RANK:11  CPU_SET:   8
RANK:12  CPU_SET:  10
RANK:13  CPU_SET:  12
RANK:14  CPU_SET:  15
RANK:15  CPU_SET:  17
RANK:16  CPU_SET:  19
RANK:17  CPU_SET:  21
RANK:18  CPU_SET:  23
RANK:19  CPU_SET:  25
RANK:20  CPU_SET:  27
RANK:21  CPU_SET:  15
RANK:22  CPU_SET:  17
RANK:23  CPU_SET:  19
RANK:24  CPU_SET:  21
RANK:25  CPU_SET:  23
RANK:26  CPU_SET:  25
RANK:27  CPU_SET:  27
-------------------------------------

# OSU MPI Allgather Latency Test
# Size       Avg Latency(us)
<hang>
^C

Output 2 - Correct binding, but jumpy performance
node1$ mpiexec -env MV2_SHOW_CPU_BINDING=1 -env
MV2_CPU_BINDING_POLICY=bunch -ppn 28 -hosts node1,node2
/opt/mpi/mvapich2/libexec/mvapich2/osu_allgather
-------------CPU AFFINITY-------------
RANK:0  CPU_SET:   0
RANK:1  CPU_SET:   2
RANK:2  CPU_SET:   4
RANK:3  CPU_SET:   6
RANK:4  CPU_SET:   8
RANK:5  CPU_SET:  10
RANK:6  CPU_SET:  12
RANK:7  CPU_SET:  14
RANK:8  CPU_SET:  16
RANK:9  CPU_SET:  18
RANK:10  CPU_SET:  20
RANK:11  CPU_SET:  22
RANK:12  CPU_SET:  24
RANK:13  CPU_SET:  26
RANK:14  CPU_SET:   1
RANK:15  CPU_SET:   3
RANK:16  CPU_SET:   5
RANK:17  CPU_SET:   7
RANK:18  CPU_SET:   9
RANK:19  CPU_SET:  11
RANK:20  CPU_SET:  13
RANK:21  CPU_SET:  15
RANK:22  CPU_SET:  17
RANK:23  CPU_SET:  19
RANK:24  CPU_SET:  21
RANK:25  CPU_SET:  23
RANK:26  CPU_SET:  25
RANK:27  CPU_SET:  27
-------------------------------------

# OSU MPI Allgather Latency Test
# Size       Avg Latency(us)
1                      11.74
2                      64.38
4                       8.19
8                       8.45
16                     15.80
32                     17.49
64                     12.17
128                    30.06
256                    86.95
512                    37.33
1024                  160.15
2048                  442.24
4096                  702.30
8192                 1754.90
16384                3699.49
32768                8189.64
65536               16935.42
131072               6609.63
262144              12731.29
524288              25628.73
1048576             59931.01
node1$
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.cse.ohio-state.edu/pipermail/mvapich-discuss/attachments/20151123/18037edf/attachment.html>