[mvapich-discuss] Odd behavior for Pallas Send/Receive Benchmark in MVAPICH 1.1

[Mike Heinz]

Interesting.

For this test, we're using a couple of AMD opterons, running at 2.4 ghz, and RHEL 4u6, a pair of Mellanox DDR HCAs and a Qlogic 9xxx switch.

We took the default when installing OFED and, looking at the build log, it appears that OFED used OPTIMIZATION_FLAG='-O3 -fno-strict-aliasing' when compiling mvapich. No optimization was chosen when compiling Pallas.


--
Michael Heinz
Principal Engineer, Qlogic Corporation
King of Prussia, Pennsylvania

-----Original Message-----
From: Dhabaleswar Panda [mailto:panda at cse.ohio-state.edu] 
Sent: Monday, June 01, 2009 3:03 PM
To: Mike Heinz
Cc: mvapich-discuss at cse.ohio-state.edu; mwheinz at me.com; John Russo; Todd Rimmer
Subject: Re: [mvapich-discuss] Odd behavior for Pallas Send/Receive Benchmark in MVAPICH 1.1

Hi Mike,

Thanks for your report. We tried running PMB (as well IMB, the latest one)
on both the released version of MVAPICH 1.1.0 and the branch version. We
are getting the peak bandwidth to be in the range of 2400-2600 MB/s
consistently. The experiments were done with Mellanox-IB cards, DDR switch
and Intel Colvertown platforms. We are not able to reproduce the problem
you are mentioning.

Could you please provide more details on the platform, adapter, switch,
etc. Also, let us know if you are using any specific optimization level.

Thanks,

DK

On Mon, 1 Jun 2009, Mike Heinz wrote:

> We had a customer report what they thought was a hardware problem, and I was assigned to investigate. Basically, they were claiming odd variations in performance during PALLAS runs to test their Infiniband fabric.
>
> What I discovered, however, was a much more interesting problem could be duplicated on any fabric, as long as I was using MVAPICH 1.1.0.
>
> Basically, what I saw was that, given two hosts and a switch, the Pallas Send Receive benchmark compiled with MVAPICH 1.1.0 would report a performance of EITHER about 2600 MB/S OR 1850 MB/S with little variation otherwise. Moreover, this behavior is unique to MVAPICH 1.1.0 - switching to MVAPICH 2 eliminated the variation. I've attached a chart so you can see what I mean.
>
> [cid:image002.png at 01C9E2A9.4A349440]
>
> I realize that, looking at the chart, your first instinct is to announce "clearly there was other traffic on the fabric that was interfering with the benchmark" - but I assure you that was not the case. Moreover, using the same nodes and same switch, but compiling with MVAPICH2, shows a complete elimination of the effect:
>
> [cid:image005.png at 01C9E2A9.4A349440]
>
> Does anyone have any ideas what's going on? If anyone wants to replicate this test, all I did was to perform 100 runs of
>
> ./PMB2.2.1/SRC_PMB/PMB-MPI1 Sendrecv
>
> I only used the 4 meg message size for these charts, but that is just for clarity. The issue appears to affect shorter messages as well.
>
> --
> Michael Heinz
> Principal Engineer, Qlogic Corporation
> King of Prussia, Pennsylvania
>