[mvapich-discuss] FW: [mpich-discuss] Odd differences between runs on two clusters / Lost messages?

Mon Sep 22 17:32:07 EDT 2008

Hi Nicolas,

Thanks for your reply and confirming that you are running MVAPICH2. Does
your cluster run MVAPICH2 1.0.0 or the latest bugfix version of MVAPICH2
which is 1.0.3. Please note that when we make our release, we go through
extensive testing. However, it is not possible to test it against all
possible MPI programs in the world. As we get feedbacks from users about
problems, we continuously fix bugs and make bugfix releases periodically.
In addition, subsequent releases also add new features. On the MVAPICH2
front, new releases also get `sync'ed' up with the latest MPICH2 releases
from ANL. For example, MVAPICH2 1.0 series is based on MPICH2 1.0.5p4
where as MVAPICH2 1.2 series is based on MPICH2 1.0.7. Thus, if your
application works well with MPICH2 1.0.7, it is not guranteed that it will
work with MVAPICH2 1.0 series (because it is based on MPICH2 1.0.5p4).
That's why I had suggested you in my earlier e-mail to test it against
MVAPICH2 1.2 series because it is based on MPICH2 1.0.7.

Based on the above description, let me suggest you to proceed as follows:

1. Since your cluster has MVAPICH2 1.0 installed, please test your
application using MPICH2 1.0.5p4. If it does not work with MPICH2 1.0.5p4,
it will not work with MVAPICH2 1.0.

2. If your program works with MPICH2 1.0.5p4, but not with MVAPICH2 1.0.0,
please tell your cluster administrator to update your installation to the
latest bugfix release of 1.0 series (MVAPICH2 1.0.3.).  Such bugfix
updates are normal for clusters and your cluster adiministrator should be
doing this periodically. Details of the changes in the mvapich2 1.0 series
can be obtained from the following URL:

http://mvapich.cse.ohio-state.edu/download/mvapich2/changes.shtml

3. If your application still fails with MVAPICH2 1.0.3, you may try to
install MVAPICH2 1.2Rc2 on your directoy and try it out. MVAPICH2 is a
user-level library and you should be able to install it in your home
directory and try it out.

4. If it still does not work out, let us know and we will take a look
at the issue in more depth. If you can send us a sample of your
application, it will be much faster for us to debug the problem
and come up with a fix in MVAPICH2 library.

Thanks,

DK

On Mon, 22 Sep 2008, Nicolas Rosner wrote:

> Hello, thanks for your reply!
>
>
> >  Please note that we have two versions [...]
>
> Sorry, my bad for omitting the 2 after MVAPICH.  Yes, the production
> cluster IS using MVAPICH2 version 1.0.
>
>
> >  If your application runs well with MPICH2 1.0.7, you should use
> >  MVAPICH2 1.2 (this is based on MPICH2 1.0.7). If this version does not
> >  work, please let us know.
>
> As I said in my previous email, I'd love to do so, but the second
> cluster's administration is beyond my control. I have already asked
> the admins to do the upgrade; I guess they might do it sooner or
> later, or not at all, depending on the kind of impact that would have
> on other users of the cluster.
>
> Was 1.0 a particularly flawed or quirky version of MVAPICH2?  It seems
> to be working fine for the rest of the cluster users.  Then again, the
> admin said I'm the only one doing actual MPI development right now
> (rest of users tend to use "canned" MPI programs and feed them their
> datasets).  On the other hand, how different can 1.2 be from 1.0,
> considering I'm just using three or four primitives?  (Ssend, Send,
> Iprobe, Recv).
>
> I still think there must be some error on my code, rather than the
> base libraries.
>
> If there's anything you think I should try while waiting for the
> update, I'd be very interested in hearing about it.
>
> Thanks again for your time,
> Nicolás
>
>
>
> >  On Mon, 22 Sep 2008, Rajeev Thakur wrote:
> >
> >  > I am forwarding your note to the MVAPICH mailing list.
> >  >
> >  > Rajeev
> >  >
> >  >
> >  > -----Original Message-----
> >  > From: owner-mpich-discuss at mcs.anl.gov
> >  > [mailto:owner-mpich-discuss at mcs.anl.gov] On Behalf Of Nicolas Rosner
> >  > Sent: Monday, September 22, 2008 6:29 AM
> >  > To: mpich-discuss at mcs.anl.gov
> >  > Subject: [mpich-discuss] Odd differences between runs on two clusters / Lost
> >  > messages?
> >  >
> >  > Hello, hope this is the right place to ask about this.
> >  >
> >  > I'm developing an MPI app using a "task pool" approach:  there is a
> >  > "pool" process that essentially monitors a queue, allowing (safely
> >  > synchronized, hopefully) PUSH and POP messages on this data structure,
> >  > and there are N agents that:
> >  >
> >  > 1) POP a new task from the pool
> >  > 2) try to solve it for a while
> >  > 3) either declare it solved and go back to step 1, or
> >  > 4) declare it "too hard", split it up into subtasks, and
> >  > 5) PUSH each generated subtask back into the pool
> >  > 6) go back to step 1.
> >  >
> >  > The tasks have hierarchical IDs, the root task being "1".  Thus, an
> >  > agent could obtain task 1.23.15 and, after massaging it for a while,
> >  > decide to partition it into 1.23.15.1, 1.23.15.2, 1.23.15.3, etc, all
> >  > of which would be queued at the central pool, waiting to be obtained
> >  > by other agents, and so on.
> >  >
> >  > My program seems to run fine on my test cluster, which consists of 3
> >  > dual-core PCs in my office running MPICH2 1.0.7.  But it's not working
> >  > well on the production one, consisting of 50 quad-core nodes, on an
> >  > InfiniBand network, running MVAPICH 1.0.
> >  >
> >  > I have already asked the cluster admins whether it would be possible
> >  > to upgrade MPI on the cluster to the latest MVAPICH release (which
> >  > seems to be based on the same MPICH2 1.0.7 that is installed on the
> >  > test cluster). But the problem seems basic enough, and I'd be
> >  > surprised if the rather old MVAPICH version was to blame. (Meaning, my
> >  > guess is that I probably have some bug that shows itself quite easily
> >  > on the one platform, remains asymptomatic on the other one, yet is
> >  > still a bug).
> >  >
> >  > You can see an example of a (very verbose) logfile showing the
> >  > unwanted behavior here:
> >  >
> >  > http://neurus.dc.uba.ar/rsat/logs/publ/112/knine.o122
> >  >
> >  > The three lines where we last hear about agents 2, 3 and 4 are
> >  >
> >  > 8.74 s -- Agent 3 sending PUSH message to pool for task 1.1.7
> >  > 9.15 s -- Agent 4 sending PUSH message to pool for task 1.2.7
> >  > 29.75 s -- Agent 2 sending PUSH message to pool for task 1.3.7
> >  >
> >  > The agents are using fully synchronous Ssend()s for the PUSH messages,
> >  > and the pool process is using Iprobe() to test whether there is a new
> >  > message, and if that returns true, Recv() to get it.
> >  >
> >  > Notice how several PUSHes in a row succeed fine, then suddenly one of
> >  > them gets lost somehow. The pool process doesn't seem to ever get the
> >  > message (i.e. Iprobe() never returns true for it) so, naturally, the
> >  > sending agent blocks forever on its Ssend() call. Once this happens to
> >  > all agents, progress stops.
> >  >
> >  > If I try to run that same test, with same inputs, same parameters,
> >  > same PRNG seed, same number of agents, etc, but on the test cluster in
> >  > my office, it runs fine; messages are not lost, progress never stops
> >  > and the process eventually ends normally.
> >  >
> >  > Any input, comments or suggestions would be greatly appreciated.  I
> >  > can provide source code, more logs and further details upon request to
> >  > anyone interested in helping me out.
> >  >
> >  > Thanks a lot in advance!
> >  >
> >  > Nicolás
> >  >
> >  >
> >  >
> >
> > > _______________________________________________
> >  > mvapich-discuss mailing list
> >  > mvapich-discuss at cse.ohio-state.edu
> >  > http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss
> >  >
> >
> >
>
> _______________________________________________
> mvapich-discuss mailing list
> mvapich-discuss at cse.ohio-state.edu
> http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss
>