[mvapich-discuss] cpu affinity with mvapich2-1.4

[Bernd Kallies]

Dear mvapich2 dev team,

I played a bit with task-to-core-binding as implemented in
mvapich2-1.4.0.

After browsing the sources, it turned out that there is a very crude
scheme implemented in mvapich2 to detect the cpu topology and guess a
proper value of MV2_CPU_MAPPING at run time, if it is not set by the
user. Basically the code parses /proc/cpuinfo, determines which
processor belongs to which socket, compares the found pattern to
something known, and sets MV2_CPU_MAPPING to what is thought to be
useful for this pattern. By doing so, neither the cache hierarchy of the
particular architecture nor strange core numbering schemes nor
not-implemented architectures can be handled.

I attach a patch for
src/mpid/ch3/channels/mrail/src/rdma/ch3_smp_progress.c
which tries to do this in a more sophisticated way based on the hwloc
API (http://www.open-mpi.org/projects/hwloc/), which is cited in the
mvapich2 user manual but obviously not used so far. In addition, the
patch includes tiny fixes to the original scheme based on /proc/cpuinfo.

The implemented topology-based guess of MV2_CPU_MAPPING yields
scattering of MPI tasks on available (logical) processors by means of a
max-distance approach regarding shared resources (caches, sockets,
cores). Other schemes may be implemented in the future (like the
opposite of "scatter") and selected by the user via some environment
variable.

The hwloc-based code is switched on with cpp flag _SMP_HWLOC_. For
distribution of the code, one should define this as well as the hwloc
include and lib paths via configure. I simply configured with

CPPFLAGS="-I/sw/local/packages/hwloc-0.9.2/include -D_SMP_HWLOC_"
LDFLAGS=-L/sw/local/packages/hwloc-0.9.2/lib LIBS=-lhwloc

I tested the code successfully on our Intel Harpertown and Intel Nehalem
architectures with various strange core numbering schemes provided by
our hardware vendor. I did not check on other Intel architectures, nor
on AMD or other.

Hope this helps somehow to improve mvapich2.

Sincerely, BK

-- 
Dr. Bernd Kallies
Konrad-Zuse-Zentrum für Informationstechnik Berlin
Takustr. 7
14195 Berlin
Tel: +49-30-84185-270
Fax: +49-30-84185-311
e-mail: kallies at zib.de

-------------- next part --------------
A non-text attachment was scrubbed...
Name: get_cpu_mapping.patch
Type: text/x-patch
Size: 10719 bytes
Desc: not available
Url : http://mail.cse.ohio-state.edu/pipermail/mvapich-discuss/attachments/20091204/372e51c6/get_cpu_mapping-0001.bin