[mvapich-discuss] HPL Test results

[Michael Zebrowski]

We are currently running HPL tests on a 38 node cluster, each node consisting of Xeon 3.4 Ghz (dual core), 4G RAM, Ethernet and IB interface. For IB we are using Mellanox Cards: Dual 4X IB Port MT25208 Infinihost III Ex. (we are using only 1 of the 2 available IB ports on each node)

We are using mvapich 0.9.7 for the IB mpi runs, and mpich 1.2.7 for the mpi runs over ethernet.

Separate HPL binaries were compiled for both IB and Ethernet interconnects using the appropriate mpicc's found in their respective bin folders. (ie. Mvapich/bin/mpicc , mpich/bin/mpicc). Note that HPL was compiled using GotoBLAS rather than the CBLAS.

RPeak = Clock * #Cores * Flops/Cycle/Core
	= 3.4 * 2 * 2

HPL:

Ns=SQRT(0.8*NodeBytes*Nodes/8)
Nb=160

Below are the initial results of the tests: 

Proc	Ns	GFLOPS-IB	GFLOPS-ETH	Rpeak	 IB % Efficiency	Eth % Efficiency
1   20353   5.87        11.30       6.8         86.3          166.17
2   20353   10.09       9.95        13.6        74.19         73.14
4   28784   22.33       20.11       27.2        82.1          73.93
8   40707   40.35       34.59       54.4        74.17         63.58
16  57568   65.01       63.34       108.8       59.75         58.22
32  81414   74.57       68.52       217.6       34.27         31.49
64  115137  129.80      131.10      435.2       29.83         30.13

Does anyone have any ideas as to possible reasons for the above results?
Suggestions of avenues that should be investigated?

Questions:

1. On a single processor run, ethernet surpasses the theoretical maximum (rpeak). How is this possible? I was under the impression that interconnects are not being utilized for single proc runs, so how is it that the IB and Ethernet results are so drastically different? Also notice that Ethernet beats out IB on the 64 proc run.

2. How is it that IB is only slightly better than Ethernet for proc runs: 2,4,8,16,32?


- Michael

NOTICE:
This message may contain privileged or otherwise confidential information.
If you are not the intended recipient, please immediately advise the sender
by reply email and delete the message and any attachments without using,
copying or disclosing the contents.