[mvapich-discuss] Performance issue

[Masahiro Nakao]

Dear all,

I use mvapich2-1.7a.
I tried to measure a troughtput performance,
but the value of performance I don't understand.

The source code is as below.
---
double tmp[SIZE];
  :
MPI_Barrier(MPI_COMM_WORLD);
t1 = gettimeofday_sec();
if(rank==0)   MPI_Send(tmp, SIZE, MPI_DOUBLE, 1, 999, MPI_COMM_WORLD );
else  MPI_Recv(tmp, SIZE, MPI_DOUBLE, 0, 999, MPI_COMM_WORLD, &status );
MPI_Barrier(MPI_COMM_WORLD);
t2 = gettimeofday_sec();

printf("%d Byte:%.1f MByte/s\n", sizeof(tmp), sizeof(tmp)/(t2-t1)/1000000);
---
This program was run on 2 nodes.

Results are as below. SIZE = 1, 2, 4, ... , 4096
---
    8 Byte:  0.6 MByte/s
   16 Byte:  1.1 MByte/s
   32 Byte:  2.1 MByte/s
   64 Byte:  4.5 MByte/s
  128 Byte:  8.0 MByte/s
  256 Byte: 15.1 MByte/s
  512 Byte: 30.2 MByte/s
 1024 Byte: 68.2 MByte/s
 2048 Byte:102.3 MByte/s
 4096 Byte:157.6 MByte/s
 8192 Byte:195.2 MByte/s
16384 Byte: 80.8 MByte/s
32768 Byte:142.4 MByte/s
---

Why is the troughtput down, when the value of transfar size is 16384 ?

My environment is as below.
---
- Quad-Core AMD Opteron(tm) Processor 8356
- DDR Infiniband (Mellanox ConnectX)
---

Regard,
-- 
Masahiro NAKAO
Email : mnakao at ccs.tsukuba.ac.jp
Researcher
Center for Computational Sciences
UNIVERSITY OF TSUKUBA