[mvapich-discuss] Bug: deadlock between ibv_destroy_srq and async_thread

[Matthew Koop]

We've done some offline discussion and the patch appears to have resolved
the issue.  This is now checked into our SVN and will be in future
MVAPICH2 releases.

Matt

On Wed, 28 May 2008 David_Kewley at Dell.com wrote:

> Matt,
>
> Thank you very much.  Glancing at the patch, looks good to me.  I'll
> apply and test it ASAP, should be able to report the results tomorrow or
> Friday.
>
> David
>
> > -----Original Message-----
> > From: Matthew Koop [mailto:koop at cse.ohio-state.edu]
> > Sent: Wednesday, May 28, 2008 11:35 AM
> > To: Kewley, David
> > Cc: mvapich-discuss at cse.ohio-state.edu
> > Subject: Re: [mvapich-discuss] Bug: deadlock between ibv_destroy_srq
> and
> > async_thread
> >
> > David,
> >
> > Can you try the attached patch and let us know if it solves your
> problem?
> > This adds some synchronization between the threads.
> >
> > Thanks,
> >
> > Matt
> >
> > On Fri, 23 May 2008 David_Kewley at Dell.com wrote:
> >
> > > I have a user running a 192-way job using MVAPICH2 1.0.1 and OFED
> > > 1.2.5.5,
> > > where MPI_Finalize() does not return.  In the two example jobs I've
> > > examined,
> > > 189 processes exited, but the other three hung.  The ranks that hung
> > > were
> > > different in the two examples, so I don't think the "3" is
> significant.
> > >
> > > All processes I've looked at appear to be stuck in the same way.  In
> > > normal
> > > running, each process has four threads.  When the process gets
> stuck,
> > > only the
> > > original thread remains.  Here is a gdb backtrace from one:
> > >
> > > #0  0x00000036b2608b3a in pthread_cond_wait@@GLIBC_2.3.2 () from
> > > /lib64/tls/libpthread.so.0
> > > #1  0x0000002a9595405b in ibv_cmd_destroy_srq (srq=0x82b370) at
> > > src/cmd.c:582
> > > #2  0x0000002a962b5419 in mthca_destroy_srq (srq=0x82b3bc) at
> > > src/verbs.c:475
> > > #3  0x0000002a9564878e in MPIDI_CH3I_CM_Finalize () from
> > > /opt/mvapich2/1.0.1/intel/10.1.015/lib/libmpich.so
> > > #4  0x0000002a955c053b in MPIDI_CH3_Finalize () from
> > > /opt/mvapich2/1.0.1/intel/10.1.015/lib/libmpich.so
> > > #5  0x0000002a95626202 in MPID_Finalize () from
> > > /opt/mvapich2/1.0.1/intel/10.1.015/lib/libmpich.so
> > > #6  0x0000002a955f7fee in PMPI_Finalize () from
> > > /opt/mvapich2/1.0.1/intel/10.1.015/lib/libmpich.so
> > > #7  0x0000002a955f7eae in pmpi_finalize_ () from
> > > /opt/mvapich2/1.0.1/intel/10.1.015/lib/libmpich.so
> > > #8  0x0000000000459ff8 in stoprog_ ()
> > > #9  0x000000000047afa6 in MAIN__ ()
> > > #10 0x0000000000405d62 in main ()
> > >
> > > After hours of opportunity to study the MVAPICH2 code :), I think I
> > > tracked it
> > > down to lines 1302-1306 in rdma_iba_init.c:
> > >
> > >             if (MPIDI_CH3I_RDMA_Process.has_srq) {
> > >
> pthread_cancel(MPIDI_CH3I_RDMA_Process.async_thread[i]);
> > >
> pthread_join(MPIDI_CH3I_RDMA_Process.async_thread[i],
> > > NULL);
> > >
> ibv_destroy_srq(MPIDI_CH3I_RDMA_Process.srq_hndl[i]);
> > >             }
> > >
> > > Consider what would happen if async_thread() were processing a
> > > IBV_EVENT_SRQ_LIMIT_REACHED event when pthread_cancel() was called
> on
> > > async_thread().  async_thread() has already called
> ibv_get_async_event()
> > > for this event, but it has not yet called ibv_ack_async_event().
> The
> > > result would be the observed deadlock in this part of
> > > ibv_cmd_destroy_srq():
> > >
> > >         pthread_mutex_lock(&srq->mutex);
> > >         while (srq->events_completed != resp.events_reported)
> > >                 pthread_cond_wait(&srq->cond, &srq->mutex);
> > >         pthread_mutex_unlock(&srq->mutex);
> > >
> > > That is, events_completed == events_reported-1 at this point.  The
> > > pthread_cond_signal() would be called, and events_completed could be
> > > made
> > > equal to events_reported, only by by calling ibv_ack_async_event()
> on
> > > this
> > > event.  But that will never happen because async_thread() is the
> only
> > > code
> > > that would have done that, and it's already been pthread_cancel()'d
> and
> > > pthread_join()'d before ibv_destroy_srq() is called.
> > >
> > > I think the fix is to add some sort of synchronization between
> > > async_thread() and the code that calls the pthread_cancel() on it.
> To
> > > the
> > > MVAPICH developers: Do you think you can work up a fix soon, and
> forward
> > > the patch for testing?
> > >
> > > Thanks,
> > > David
> > >
> > >
> > > David Kewley
> > > Dell Infrastructure Consulting Services
> > > Onsite Engineer at the Maui HPC Center
> > > Cell: 602-460-7617
> > > David_Kewley at Dell.com
> > >
> > > Dell Services: http://www.dell.com/services/
> > > How am I doing? Email my manager Russell_Kelly at Dell.com with any
> > > feedback.
> > >
> > > _______________________________________________
> > > mvapich-discuss mailing list
> > > mvapich-discuss at cse.ohio-state.edu
> > > http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss
> > >
>