[mvapich-discuss] cpu affinity with mvapich2-1.4

[Krishna Chaitanya Kandalla]

Dr. Kallies,
                Thanks for sending us this patch. We have taken a look 
at the hwloc interface and what it can do, but we haven't yet moved into 
incorporating it into our code. This will surely give us a jump-start. 
We will apply your patch and give it in a spin on our Nehalem, 
Clovertown and Harpertown clusters.
                

Thanks,
Krishna

Bernd Kallies wrote:
> Dear mvapich2 dev team,
>
> I played a bit with task-to-core-binding as implemented in
> mvapich2-1.4.0.
>
> After browsing the sources, it turned out that there is a very crude
> scheme implemented in mvapich2 to detect the cpu topology and guess a
> proper value of MV2_CPU_MAPPING at run time, if it is not set by the
> user. Basically the code parses /proc/cpuinfo, determines which
> processor belongs to which socket, compares the found pattern to
> something known, and sets MV2_CPU_MAPPING to what is thought to be
> useful for this pattern. By doing so, neither the cache hierarchy of the
> particular architecture nor strange core numbering schemes nor
> not-implemented architectures can be handled.
>
> I attach a patch for
> src/mpid/ch3/channels/mrail/src/rdma/ch3_smp_progress.c
> which tries to do this in a more sophisticated way based on the hwloc
> API (http://www.open-mpi.org/projects/hwloc/), which is cited in the
> mvapich2 user manual but obviously not used so far. In addition, the
> patch includes tiny fixes to the original scheme based on /proc/cpuinfo.
>
> The implemented topology-based guess of MV2_CPU_MAPPING yields
> scattering of MPI tasks on available (logical) processors by means of a
> max-distance approach regarding shared resources (caches, sockets,
> cores). Other schemes may be implemented in the future (like the
> opposite of "scatter") and selected by the user via some environment
> variable.
>
> The hwloc-based code is switched on with cpp flag _SMP_HWLOC_. For
> distribution of the code, one should define this as well as the hwloc
> include and lib paths via configure. I simply configured with
>
> CPPFLAGS="-I/sw/local/packages/hwloc-0.9.2/include -D_SMP_HWLOC_"
> LDFLAGS=-L/sw/local/packages/hwloc-0.9.2/lib LIBS=-lhwloc
>
> I tested the code successfully on our Intel Harpertown and Intel Nehalem
> architectures with various strange core numbering schemes provided by
> our hardware vendor. I did not check on other Intel architectures, nor
> on AMD or other.
>
> Hope this helps somehow to improve mvapich2.
>
> Sincerely, BK
>
>   
> ------------------------------------------------------------------------
>
> _______________________________________________
> mvapich-discuss mailing list
> mvapich-discuss at cse.ohio-state.edu
> http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss