[mvapich-discuss] FW: Only half CPU usage in multi-node MPI run

[Hari Subramoni]

Hello Mike,

Can you please re-run the application after setting MV2_SHOW_ENV_INFO=2
and MV2_SHOW_CPU_BINDING=2 and send us the output? Could you also send us
the output of mpiname -a?

Please refer to the following sections of the userguide for more
information on these parameters

http://mvapich.cse.ohio-state.edu/static/media/mvapich/mvapich2-2.2b-userguide.html#x1-15700010.6
http://mvapich.cse.ohio-state.edu/static/media/mvapich/mvapich2-2.2b-userguide.html#x1-15800010.7

Thx,
Hari.

> ------------------------------
> *From:* Mike Chen [mike.scchen at gmail.com]
> *Sent:* Thursday, March 10, 2016 6:37 AM
> *To:* mvapich-discuss at cse.ohio-state.edu
> *Subject:* Only half CPU usage in multi-node MPI run
>
> Hi~
> I'm really confused and looking for help :p
>
> The configutaion:
> 1 master node + 2 computing nodes
> Computing nodes: dual E5-2630 v3, 16 cores / node
> Mellanox ConnectX-3 56Gbps IB
>
> CentOS 6.7, kernel 2.6.32-573.el6.x86-64
> Mellanox OFED 3.1-1.0.3
> PGI compiler (C, C++, Fortran) v14.10
> MVAPICH2 2.0.1 and 2.2b, both configured with:
> --with-device=ch3:mrail --with-rdma=gen2 CC=pgcc CXX=pgCC FC=pgfortran
> F77=pgfortran
> Compiled on master node, with numactl, libpciaccess devel packages from OS
> repository installed.
>
> Torque 4.2.9, with the following edit in the pbs_mom init script:
> ulimit -l unlimited
> ulimit -s unlimited
>
> The symptom:
> Tested with both versions of MVAPICH2, both Intel / PGI compiler, and code
> of:
> 1. HPL 2.1
> 2. WRF 3.6.1
> When these codes run on one node, the CPU cores are fully occupied.
> But when run on multiple nodes, the CPU usage of each MPI process is ~50%
> only.
> Running with mpirun directly or submit with the Torque shows the same
> behavior.
>
> The HPL code built with PGI and OpenMPI v1.8.4 run normally, with each MPI
> process ~100% CPU usage.
> Have not tried other code / MPI combinations.
>
> Did I miss something the MVAPICH2 setup process?
> Please provide some suggestions ;)
>
> Mike Chen
> Research Assistant
> Dept. of Atmospheric Science, National Taiwan University
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.cse.ohio-state.edu/pipermail/mvapich-discuss/attachments/20160310/8770d3a3/attachment-0001.html>