[mvapich-discuss] Faster is Slower

[Taylor, Norm R.]

Hi,
  I'm wondering if you can advise me on an issue I'm encountering with
MPI+Infiniband, using MVAPICH. I'm finding that the high rate at which
collective operations - e.g. MPI_GATHER - poll to determine if all nodes
have entered the operation steals too many CPU cycles from other
processes, slowing down overall performance. Is there a way I can tune
these operations to be more CPU-efficient? I actually improve
performance by adding a few microseconds of sleep time to the data
transfer processes (these are the ones using MPI+Infiniband) to give
more CPU cycles to the computational processes. This tuning is very
specific to the problem at hand and the number of nodes in use. Tuning
at the process level seems still inefficient - it would be better if the
sleep time was applied inside the collective operations. Is there a way
I can set a parameter somewhere to make that happen?

Note: I don't seem to have this problem using TCPIP as the data
transport because the polling rate is forced to be slower by the slower
protocol.

Thanks,
  Norm Taylor

Norm Taylor
Zeta Associates Inc.
taylor-norm at zai.com