[mvapich-discuss] process binding with mvapich2-1.7rc2

[Dan Kokron]

I am using mvapich2-1.7rc2 configure as follows

./configure CC=icc CXX=icpc F77=ifort FC=ifort CFLAGS=-fpic
CXXFLAGS=-fpic FFLAGS=-fpic FCFLAGS=-fpic
--prefix=/home/dkokron/play/mvapich2-1.7rc2/install --enable-f77
--enable-fc --enable-cxx --enable-romio --enable-threads=default
--with-device=ch3:mrail --with-rdma=gen2 --with-hwloc
-disable-multi-aliases -enable-xrc=yes -enable-hybrid

I specified a process binding of 1,7,3,9 on the command line as follows

mvapich2-1.7rc2/install/bin/mpirun   -prepend-rank
-launcher-exec /usr/bin/sshmpi -binding user:1,7,3,9 -topolib hwloc -f
$PBS_NODEFILE -np 256

'top' shows a different binding (0,1,2,3)

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+   P COMMAND        
 9106 dkokron   20   0 1764m 586m  30m R  100  2.4   9:58.24  0 GSIsa.x         
 9107 dkokron   20   0 1692m 506m  18m R  100  2.1  10:05.06  1 GSIsa.x         
 9108 dkokron   20   0 1691m 506m  18m R  100  2.1  10:03.42  2 GSIsa.x         
 9109 dkokron   20   0 1692m 507m  18m R  100  2.1  10:04.35  3 GSIsa.x

lstopo from hwloc-1.3 shows the following for each compute node

Machine (24GB)
  NUMANode L#0 (P#0 12GB) + Socket L#0 + L3 L#0 (12MB)
    L2 L#0 (256KB) + L1 L#0 (32KB) + Core L#0 + PU L#0 (P#0)
    L2 L#1 (256KB) + L1 L#1 (32KB) + Core L#1 + PU L#1 (P#1)
    L2 L#2 (256KB) + L1 L#2 (32KB) + Core L#2 + PU L#2 (P#2)
    L2 L#3 (256KB) + L1 L#3 (32KB) + Core L#3 + PU L#3 (P#3)
    L2 L#4 (256KB) + L1 L#4 (32KB) + Core L#4 + PU L#4 (P#4)
    L2 L#5 (256KB) + L1 L#5 (32KB) + Core L#5 + PU L#5 (P#5)
  NUMANode L#1 (P#1 12GB) + Socket L#1 + L3 L#1 (12MB)
    L2 L#6 (256KB) + L1 L#6 (32KB) + Core L#6 + PU L#6 (P#6)
    L2 L#7 (256KB) + L1 L#7 (32KB) + Core L#7 + PU L#7 (P#7)
    L2 L#8 (256KB) + L1 L#8 (32KB) + Core L#8 + PU L#8 (P#8)
    L2 L#9 (256KB) + L1 L#9 (32KB) + Core L#9 + PU L#9 (P#9)
    L2 L#10 (256KB) + L1 L#10 (32KB) + Core L#10 + PU L#10 (P#10)
    L2 L#11 (256KB) + L1 L#11 (32KB) + Core L#11 + PU L#11 (P#11)
-- 
Dan Kokron
Global Modeling and Assimilation Office, Code 610.1
NASA Goddard Space Flight Center
Greenbelt, MD 20771 USA
Daniel.S.Kokron at nasa.gov
Phone: (301) 614-5192
Fax:   (301) 614-5304