[mvapich-discuss] Affinity problem

[Uday K Bondhugula]

On 01/18/2012 04:34 PM, Vsevolod Nikonorov wrote:
> Good afternoon.
>
> I have a problem with process pinning when I execute hybrid parallel
> tasks (MPI + OpenMP): after starting my task with mpirun on a cluster
> having something like this:
>
> #pragma omp parallel { printf ("%d\n", omp_get_num_threads ()); }
>
> (mpirun -np 10 -hostfile hostfile a.out) I see that only one thread
> is started; if I specify the number threads manually like this:
>
> omp_set_num_threads (10);
>
> I see 10 threads, but all of them are started on the same processor
> core, making performance really low. I suspect there is some default
> pinning - all threads are pinned to parent core (core which has
> started the parent process), and automatic discovering of other cores
> by OpenMP is unable for the same reason.
>
> So my question is: how can I change the default pinning
> configuratoion? There is of course some information on that in the
> user guide, which consists of the following recomendations:

I experienced the same issue when I had started using MVAPICH2 several 
months ago. I was able to solve it with

MV2_ENABLE_AFFINITY=0

i.e.,

mpirun_rsh  -np <nprocs> -hostfile ... MV2_ENABLE_AFFINITY=0 
OMP_NUM_THREADS=4 ./exec

http://mvapich.cse.ohio-state.edu/support/user_guide_mvapich2-1.8_alpha1_p1.html#x1-710006.14

In addition, if you are using ICC as the underlying compiler, with 
KMP_AFFINITY=verbose, you'll be able to exactly see what's going on. For 
eg.,

[uday at ukr ~/tmp/]₹ KMP_AFFINITY=verbose ./par
OMP: Info #204: KMP_AFFINITY: decoding x2APIC ids. 
 

OMP: Info #202: KMP_AFFINITY: Affinity capable, using global cpuid leaf 
11 info 

OMP: Info #154: KMP_AFFINITY: Initial OS proc set respected: 
{0,1,2,3,4,5,6,7} 

OMP: Info #156: KMP_AFFINITY: 8 available OS procs
OMP: Info #157: KMP_AFFINITY: Uniform topology
OMP: Info #179: KMP_AFFINITY: 1 packages x 4 cores/pkg x 2 threads/core 
(4 total cores)
OMP: Info #147: KMP_AFFINITY: Internal thread 0 bound to OS proc set 
{0,1,2,3,4,5,6,7}
OMP: Info #147: KMP_AFFINITY: Internal thread 1 bound to OS proc set 
{0,1,2,3,4,5,6,7}
OMP: Info #147: KMP_AFFINITY: Internal thread 2 bound to OS proc set 
{0,1,2,3,4,5,6,7}
OMP: Info #147: KMP_AFFINITY: Internal thread 4 bound to OS proc set 
{0,1,2,3,4,5,6,7}
OMP: Info #147: KMP_AFFINITY: Internal thread 3 bound to OS proc set 
{0,1,2,3,4,5,6,7}
OMP: Info #147: KMP_AFFINITY: Internal thread 5 bound to OS proc set 
{0,1,2,3,4,5,6,7}
OMP: Info #147: KMP_AFFINITY: Internal thread 6 bound to OS proc set 
{0,1,2,3,4,5,6,7}
OMP: Info #147: KMP_AFFINITY: Internal thread 7 bound to OS proc set 
{0,1,2,3,4,5,6,7}


-Uday


>
> 1. set VIADEV_USE_AFFINITY environment variable to 0; 2. set
> MV_USE_AFFINITY variable to 0.
>
> There is a comment also saying that those variables will not take
> effect while _AFFINITY_ is not set. I tried to find some information
> about it in the user guide and Google, but all I have found is some
> citation from some header files (#ifdef _AFFINITY_ in stdio.h).
>
> Could you help me solving this problem?
>
> Thanks in advance!
>