[mvapich-discuss] MPI_Recv inconsistent performance

[David Winslow]

All,
We are experiencing dramatic differences in performance of MPI_Recv between calls. With a cluster of 4 nodes (all on same box), we use MPI_Recv to receive a message. We timed the receive function and we see times from 2ms to greater than 200ms to receive the exact same message (a 4 byte integer).  We don’t understand what could cause this significant disparity and unpredictability between the calls.

Pseudo code

Get_time()
MPI_Probe()
MPI_Recv()
Get_time()

Is there a way to control or configure MVAPICH2 to make the behavior of MPI_Probe/MPI_Recv more predictable/consistent? 

output of mpiname -a

MVAPICH2 2.2b Mon Nov 12 20:00:00 EST 2015 ch3:psm

Compilation
CC: gcc    -DNDEBUG -DNVALGRIND -O2
CXX: g++   -DNDEBUG -DNVALGRIND -O2
F77:
FC:

Configuration
--prefix=/opt/mvapich2-2.2b-install-psm --with-device=ch3:psm --disable-fortran