[mvapich-discuss] Odd behavior for Pallas Send/Receive Benchmark in MVAPICH 1.1

[Krishna Chaitanya]

Mike,
          We have run tests on  Intel Clovertown and AMD Barcelona machines
with MVAPICH-1.1 and we have compiled the library with the flags that you
mentioned in your previous mail. Unfortunately, we are not able to reproduce
the issue. We have run the complete sendrecv pallas benchmark about a 100
times in a loop and we see that the peak bandiwdth is in the 2400 - 2600
MB/s range consistently.
           Could you try running the benchmark on two nodes connected back
to back? This will eliminate any network or switch issues.

Thanks,
Krishna

On Mon, Jun 1, 2009 at 3:15 PM, Mike Heinz <michael.heinz at qlogic.com> wrote:

> Interesting.
>
> For this test, we're using a couple of AMD opterons, running at 2.4 ghz,
> and RHEL 4u6, a pair of Mellanox DDR HCAs and a Qlogic 9xxx switch.
>
> We took the default when installing OFED and, looking at the build log, it
> appears that OFED used OPTIMIZATION_FLAG='-O3 -fno-strict-aliasing' when
> compiling mvapich. No optimization was chosen when compiling Pallas.
>
>
> --
> Michael Heinz
> Principal Engineer, Qlogic Corporation
> King of Prussia, Pennsylvania
>
> -----Original Message-----
> From: Dhabaleswar Panda [mailto:panda at cse.ohio-state.edu]
> Sent: Monday, June 01, 2009 3:03 PM
> To: Mike Heinz
> Cc: mvapich-discuss at cse.ohio-state.edu; mwheinz at me.com; John Russo; Todd
> Rimmer
> Subject: Re: [mvapich-discuss] Odd behavior for Pallas Send/Receive
> Benchmark in MVAPICH 1.1
>
> Hi Mike,
>
> Thanks for your report. We tried running PMB (as well IMB, the latest one)
> on both the released version of MVAPICH 1.1.0 and the branch version. We
> are getting the peak bandwidth to be in the range of 2400-2600 MB/s
> consistently. The experiments were done with Mellanox-IB cards, DDR switch
> and Intel Colvertown platforms. We are not able to reproduce the problem
> you are mentioning.
>
> Could you please provide more details on the platform, adapter, switch,
> etc. Also, let us know if you are using any specific optimization level.
>
> Thanks,
>
> DK
>
> On Mon, 1 Jun 2009, Mike Heinz wrote:
>
> > We had a customer report what they thought was a hardware problem, and I
> was assigned to investigate. Basically, they were claiming odd variations in
> performance during PALLAS runs to test their Infiniband fabric.
> >
> > What I discovered, however, was a much more interesting problem could be
> duplicated on any fabric, as long as I was using MVAPICH 1.1.0.
> >
> > Basically, what I saw was that, given two hosts and a switch, the Pallas
> Send Receive benchmark compiled with MVAPICH 1.1.0 would report a
> performance of EITHER about 2600 MB/S OR 1850 MB/S with little variation
> otherwise. Moreover, this behavior is unique to MVAPICH 1.1.0 - switching to
> MVAPICH 2 eliminated the variation. I've attached a chart so you can see
> what I mean.
> >
> > [cid:image002.png at 01C9E2A9.4A349440]
> >
> > I realize that, looking at the chart, your first instinct is to announce
> "clearly there was other traffic on the fabric that was interfering with the
> benchmark" - but I assure you that was not the case. Moreover, using the
> same nodes and same switch, but compiling with MVAPICH2, shows a complete
> elimination of the effect:
> >
> > [cid:image005.png at 01C9E2A9.4A349440]
> >
> > Does anyone have any ideas what's going on? If anyone wants to replicate
> this test, all I did was to perform 100 runs of
> >
> > ./PMB2.2.1/SRC_PMB/PMB-MPI1 Sendrecv
> >
> > I only used the 4 meg message size for these charts, but that is just for
> clarity. The issue appears to affect shorter messages as well.
> >
> > --
> > Michael Heinz
> > Principal Engineer, Qlogic Corporation
> > King of Prussia, Pennsylvania
> >
>
>
> _______________________________________________
> mvapich-discuss mailing list
> mvapich-discuss at cse.ohio-state.edu
> http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.cse.ohio-state.edu/pipermail/mvapich-discuss/attachments/20090602/c42e431c/attachment.html