[mvapich-discuss] Races with MPI_THREAD_MULTI (fwd)

Tue Jul 22 14:21:08 EDT 2008

Hi Roberto,

> MPI_Send() to MPI_Ssend() everything
> seems working well with mpich2 v1.0.7. My application doesn't race
> anymore at least after dispatching
> 50.000 jobs across 4 nodes, but trying to execute the same application
> against the last mvapich2 1.2rc1
> I'm still getting the same problem as shown below.

Good to know that it works with mpich2. How do you configure/compile
mvapich2-1.2rc1? Are you using the ofed interface (default configuration),
or using the TCP/IP stack (adding --with-device=ch3:sock in the configure
option)? The TCP/IP stack should be the same as mpich2, in other words, it
should work with your application. The implementation on ofed interface
does not support spawn yet, thus there will be issues.

> I've another question, since this multithreaded application has to run
> into a cluster with 1024 nodes equiped
> with Mellanox IB card, I really like to know if the OpenFabrics-IB
> interface does support the MPI_THREAD_MULTIPLE
> initialization and also the MPI_Comm_spawn() implementation.

mvapich2 on the ofed (IB) interface supports MPI_THREAD_MULTIPLE.
MPI_Comm_spawn() is not yet supported, however. We are working on this
support and it will be available in our next release.

Thanks.

-- Wei

> >> On Fri, 18 Jul 2008, Roberto Fichera wrote:
> >>
> >>
> >>> Hi All on the list,
> >>>
> >>> I'm trying to use mvapich2 v1.2rc1 in a multithreaded application,
> >>> initialize using MPI_THREAD_MULTI.
> >>> I've the master application doing the following thing, start several
> >>> thread depending by the assigned nodes,
> >>> on each node a slave application is spawned using the MPI_Comm_spawn().
> >>> Before to call the
> >>> MPI_Comm_spawn() I prepare the given MPI_Info struct, one for each
> >>> thread, in order to set the all keys
> >>> (host and wdir) for addressing the wanted behaviour. So, as sooner as
> >>> the master application starts, it races
> >>> immediately with 4 nodes, 1 master and 3 slaves. Below you can see the
> >>> status of the master application at race
> >>> time. It seems stuck on the PMIU_readline() which never returns so the
> >>> global lock is never relesead. MVAPICH2
> >>> is compiled with:
> >>>
> >>> PKG_PATH=/HRI/External/mvapich2/1.2rc1
> >>>
> >>> ./configure --prefix=$PKG_PATH \
> >>>             --bindir=$PKG_PATH/bin/linux-x86_64-gcc-glibc2.3.4 \
> >>>             --sbindir=$PKG_PATH/bin/linux-x86_64-gcc-glibc2.3.4 \
> >>>             --libdir=$PKG_PATH/lib/linux-x86_64-gcc-glibc2.3.4 \
> >>>             --enable-sharedlibs=gcc \
> >>>             --enable-f90 \
> >>>             --enable-threads=multiple \
> >>>             --enable-g=-ggdb \
> >>>             --enable-debuginfo \
> >>>             --with-device=ch3:sock \
> >>>             --datadir=$PKG_PATH/data  \
> >>>             --with-htmldir=$PKG_PATH/doc/html \
> >>>             --with-docdir=$PKG_PATH/doc \
> >>>             LDFLAGS='-Wl,-z,noexecstack'
> >>>
> >>> so I'm using the ch3:sock device.
> >>>
> >>> -----Thread 2
> >>> [Switching to thread 2 (Thread 1115699536 (LWP 29479))]#0
> >>> 0x00000033ca40cef4 in __lll_lock_wait () from /lib64/libpthread.so.0
> >>> (gdb) bt
> >>> #0  0x00000033ca40cef4 in __lll_lock_wait () from /lib64/libpthread.so.0
> >>> #1  0x00000033ca408915 in _L_lock_102 () from /lib64/libpthread.so.0
> >>> --->>#2  0x00000033ca408390 in pthread_mutex_lock () from
> >>> /lib64/libpthread.so.0
> >>> --->>#3  0x00002aaaab382654 in PMPI_Info_set () from
> >>> /home/roberto/.HRI/Proxy/HRI/External/mvapich2/1.2/lib/linux-x86_64-gcc-glibc2.3.4/libmpich.so.1.1
> >>> #4  0x0000000000417627 in ParallelWorker_setSlaveInfo (self=<value
> >>> optimized out>, key=0x0, value=0x33ca40ff58
> >>> "!\204ï¿½ï¿½\r\206ï¿½ï¿½\030\204ï¿½ï¿½3\206ï¿½ï¿½\n\204ï¿½ï¿½\n\204ï¿½ï¿½\n\204ï¿½ï¿½\n\204ï¿½ï¿½\n\204ï¿½ï¿½\n\204ï¿½ï¿½\n\204ï¿½ï¿½\n\204ï¿½ï¿½\n\204ï¿½ï¿½\n\204ï¿½ï¿½\n\204ï¿½ï¿½\n\204ï¿½ï¿½\177\205ï¿½ï¿½\177\205ï¿½ï¿½\177\205ï¿½ï¿½\177\205ï¿½ï¿½\n\204ï¿½ï¿½\n\204ï¿½ï¿½\n\204ï¿½ï¿½\n\204ï¿½ï¿½\n\204ï¿½ï¿½\n\204ï¿½ï¿½\n\204ï¿½ï¿½\n\204ï¿½ï¿½\n\204ï¿½ï¿½\n\204ï¿½ï¿½\n\204ï¿½ï¿½\n\204ï¿½ï¿½\033\205ï¿½ï¿½\033\205ï¿½ï¿½\033\205ï¿½ï¿½\033\205ï¿½ï¿½\n\204ï¿½ï¿½\n\204ï¿½ï¿½\n\204ï¿½ï¿½\n\204ï¿½ï¿½\n\204ï¿½ï¿½\n\204ï¿½ï¿½\n\204ï¿½ï¿½\n\204ï¿½ï¿½\n\204ï¿½ï¿½\n\204ï¿½ï¿½\n\204ï¿½ï¿½\n\204ï¿½ï¿½\033\205ï¿½ï¿½\033\205ï¿½ï¿½"...)
> >>> at ParallelWorker.c:664
> >>> #5  0x0000000000418905 in ParallelWorker_handleParallel (self=0x62ff50)
> >>> at ParallelWorker.c:719
> >>> #6  0x000000000041b39e in ParallelWorker_threadMain (arg=0x62ff50) at
> >>> ParallelWorker.c:504
> >>> #7  0x00000033ca406407 in start_thread () from /lib64/libpthread.so.0
> >>> #8  0x00000033c94d4b0d in clone () from /lib64/libc.so.6
> >>>
> >>> -----Thread 3
> >>> [Switching to thread 3 (Thread 1105209680 (LWP 29478))]#0
> >>> 0x00000033ca40cef4 in __lll_lock_wait () from /lib64/libpthread.so.0
> >>> (gdb) bt
> >>> #0  0x00000033ca40cef4 in __lll_lock_wait () from /lib64/libpthread.so.0
> >>> #1  0x00000033ca408915 in _L_lock_102 () from /lib64/libpthread.so.0
> >>> --->>#2  0x00000033ca408390 in pthread_mutex_lock () from
> >>> /lib64/libpthread.so.0
> >>> --->>#3  0x00002aaaab382654 in PMPI_Info_set () from
> >>> /home/roberto/.HRI/Proxy/HRI/External/mvapich2/1.2/lib/linux-x86_64-gcc-glibc2.3.4/libmpich.so.1.1
> >>> #4  0x0000000000417627 in ParallelWorker_setSlaveInfo (self=<value
> >>> optimized out>, key=0x0, value=0x33ca40ff58
> >>> "!\204ï¿½ï¿½\r\206ï¿½ï¿½\030\204ï¿½ï¿½3\206ï¿½ï¿½\n\204ï¿½ï¿½\n\204ï¿½ï¿½\n\204ï¿½ï¿½\n\204ï¿½ï¿½\n\204ï¿½ï¿½\n\204ï¿½ï¿½\n\204ï¿½ï¿½\n\204ï¿½ï¿½\n\204ï¿½ï¿½\n\204ï¿½ï¿½\n\204ï¿½ï¿½\n\204ï¿½ï¿½\177\205ï¿½ï¿½\177\205ï¿½ï¿½\177\205ï¿½ï¿½\177\205ï¿½ï¿½\n\204ï¿½ï¿½\n\204ï¿½ï¿½\n\204ï¿½ï¿½\n\204ï¿½ï¿½\n\204ï¿½ï¿½\n\204ï¿½ï¿½\n\204ï¿½ï¿½\n\204ï¿½ï¿½\n\204ï¿½ï¿½\n\204ï¿½ï¿½\n\204ï¿½ï¿½\n\204ï¿½ï¿½\033\205ï¿½ï¿½\033\205ï¿½ï¿½\033\205ï¿½ï¿½\033\205ï¿½ï¿½\n\204ï¿½ï¿½\n\204ï¿½ï¿½\n\204ï¿½ï¿½\n\204ï¿½ï¿½\n\204ï¿½ï¿½\n\204ï¿½ï¿½\n\204ï¿½ï¿½\n\204ï¿½ï¿½\n\204ï¿½ï¿½\n\204ï¿½ï¿½\n\204ï¿½ï¿½\n\204ï¿½ï¿½\033\205ï¿½ï¿½\033\205ï¿½ï¿½"...)
> >>> at ParallelWorker.c:664
> >>> #5  0x0000000000418905 in ParallelWorker_handleParallel (self=0x62f270)
> >>> at ParallelWorker.c:719
> >>> #6  0x000000000041b39e in ParallelWorker_threadMain (arg=0x62f270) at
> >>> ParallelWorker.c:504
> >>> #7  0x00000033ca406407 in start_thread () from /lib64/libpthread.so.0
> >>> #8  0x00000033c94d4b0d in clone () from /lib64/libc.so.6
> >>>
> >>> -----Thread 4
> >>> [Switching to thread 4 (Thread 1094719824 (LWP 29477))]#0
> >>> 0x00000033ca40d34b in read () from /lib64/libpthread.so.0
> >>> (gdb) bt
> >>> #0  0x00000033ca40d34b in read () from /lib64/libpthread.so.0
> >>> --->>#1  0x00002aaaab3db84a in PMIU_readline () from
> >>> /home/roberto/.HRI/Proxy/HRI/External/mvapich2/1.2/lib/linux-x86_64-gcc-glibc2.3.4/libmpich.so.1.1
> >>> --->>#2  0x00002aaaab3d9d37 in PMI_Spawn_multiple () from
> >>> /home/roberto/.HRI/Proxy/HRI/External/mvapich2/1.2/lib/linux-x86_64-gcc-glibc2.3.4/libmpich.so.1.1
> >>> #3  0x00002aaaab333893 in MPIDI_Comm_spawn_multiple () from
> >>> /home/roberto/.HRI/Proxy/HRI/External/mvapich2/1.2/lib/linux-x86_64-gcc-glibc2.3.4/libmpich.so.1.1
> >>> #4  0x00002aaaab38bcf6 in MPID_Comm_spawn_multiple () from
> >>> /home/roberto/.HRI/Proxy/HRI/External/mvapich2/1.2/lib/linux-x86_64-gcc-glibc2.3.4/libmpich.so.1.1
> >>> #5  0x00002aaaab355a10 in PMPI_Comm_spawn () from
> >>> /home/roberto/.HRI/Proxy/HRI/External/mvapich2/1.2/lib/linux-x86_64-gcc-glibc2.3.4/libmpich.so.1.1
> >>> #6  0x00000000004189d8 in ParallelWorker_handleParallel (self=0x62ad40)
> >>> at ParallelWorker.c:754
> >>> #7  0x000000000041b39e in ParallelWorker_threadMain (arg=0x62ad40) at
> >>> ParallelWorker.c:504
> >>> #8  0x00000033ca406407 in start_thread () from /lib64/libpthread.so.0
> >>> #9  0x00000033c94d4b0d in clone () from /lib64/libc.so.6
> >>>
> >>> I also tried to run against MPICH2 v1.0.7, but here I got a similar
> >>> scenery which show up after between 1 - 2 hours of execution,
> >>> see below:
> >>>
> >>> ----- thread 2
> >>> [Switching to thread 2 (Thread 1094719824 (LWP 1279))]#0  0x00000033c94cbd66 in poll () from /lib64/libc.so.6
> >>> (gdb) bt
> >>> #0  0x00000033c94cbd66 in poll () from /lib64/libc.so.6
> >>> #1  0x00002aaaab5a3d2f in MPIDU_Sock_wait () from /home/roberto/.HRI/Proxy/HRI/External/mpich2/1.0.7/lib/linux-x86_64-gcc-glibc2.3.4/libmpich.so.1.1
> >>> #2  0x00002aaaab52bdc7 in MPIDI_CH3I_Progress () from /home/roberto/.HRI/Proxy/HRI/External/mpich2/1.0.7/lib/linux-x86_64-gcc-glibc2.3.4/libmpich.so.1.1
> >>> #3  0x00002aaaab5301a7 in MPIDI_CH3U_VC_WaitForClose () from /home/roberto/.HRI/Proxy/HRI/External/mpich2/1.0.7/lib/linux-x86_64-gcc-glibc2.3.4/libmpich.so.1.1
> >>> #4  0x00002aaaab56f162 in MPID_Comm_disconnect () from /home/roberto/.HRI/Proxy/HRI/External/mpich2/1.0.7/lib/linux-x86_64-gcc-glibc2.3.4/libmpich.so.1.1
> >>> #5  0x00002aaaab5417ec in PMPI_Comm_disconnect () from /home/roberto/.HRI/Proxy/HRI/External/mpich2/1.0.7/lib/linux-x86_64-gcc-glibc2.3.4/libmpich.so.1.1
> >>> #6  0x00002aaaabda5a99 in ParallelWorker_destroySlave (self=0x6358e0) at ParallelWorker.c:819
> >>> #7  0x00002aaaabda6223 in ParallelWorker_threadMain (arg=0x6358e0) at ParallelWorker.c:515
> >>> #8  0x00000033ca406407 in start_thread () from /lib64/libpthread.so.0
> >>> #9  0x00000033c94d4b0d in clone () from /lib64/libc.so.6
> >>>
> >>> ----- thread 3
> >>> [Switching to thread 3 (Thread 1084229968 (LWP 1278))]#0  0x00000033ca40a8f9 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
> >>> (gdb) bt
> >>> #0  0x00000033ca40a8f9 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
> >>> #1  0x00002aaaab52bec7 in MPIDI_CH3I_Progress () from /home/roberto/.HRI/Proxy/HRI/External/mpich2/1.0.7/lib/linux-x86_64-gcc-glibc2.3.4/libmpich.so.1.1
> >>> #2  0x00002aaaab5301a7 in MPIDI_CH3U_VC_WaitForClose () from /home/roberto/.HRI/Proxy/HRI/External/mpich2/1.0.7/lib/linux-x86_64-gcc-glibc2.3.4/libmpich.so.1.1
> >>> #3  0x00002aaaab56f162 in MPID_Comm_disconnect () from /home/roberto/.HRI/Proxy/HRI/External/mpich2/1.0.7/lib/linux-x86_64-gcc-glibc2.3.4/libmpich.so.1.1
> >>> #4  0x00002aaaab5417ec in PMPI_Comm_disconnect () from /home/roberto/.HRI/Proxy/HRI/External/mpich2/1.0.7/lib/linux-x86_64-gcc-glibc2.3.4/libmpich.so.1.1
> >>> #5  0x00002aaaabda5a99 in ParallelWorker_destroySlave (self=0x634d20) at ParallelWorker.c:819
> >>> #6  0x00002aaaabda6223 in ParallelWorker_threadMain (arg=0x634d20) at ParallelWorker.c:515
> >>> #7  0x00000033ca406407 in start_thread () from /lib64/libpthread.so.0
> >>> #8  0x00000033c94d4b0d in clone () from /lib64/libc.so.6
> >>>
> >>>
> >>> ----- thread 4
> >>> [Switching to thread 4 (Thread 1115699536 (LWP 1277))]#0  0x00000033ca40a8f9 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
> >>> (gdb) bt
> >>> #0  0x00000033ca40a8f9 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
> >>> #1  0x00002aaaab52bec7 in MPIDI_CH3I_Progress () from /home/roberto/.HRI/Proxy/HRI/External/mpich2/1.0.7/lib/linux-x86_64-gcc-glibc2.3.4/libmpich.so.1.1
> >>> #2  0x00002aaaab5301a7 in MPIDI_CH3U_VC_WaitForClose () from /home/roberto/.HRI/Proxy/HRI/External/mpich2/1.0.7/lib/linux-x86_64-gcc-glibc2.3.4/libmpich.so.1.1
> >>> #3  0x00002aaaab56f162 in MPID_Comm_disconnect () from /home/roberto/.HRI/Proxy/HRI/External/mpich2/1.0.7/lib/linux-x86_64-gcc-glibc2.3.4/libmpich.so.1.1
> >>> #4  0x00002aaaab5417ec in PMPI_Comm_disconnect () from /home/roberto/.HRI/Proxy/HRI/External/mpich2/1.0.7/lib/linux-x86_64-gcc-glibc2.3.4/libmpich.so.1.1
> >>> #5  0x00002aaaabda5a99 in ParallelWorker_destroySlave (self=0x6341a0) at ParallelWorker.c:819
> >>> #6  0x00002aaaabda6223 in ParallelWorker_threadMain (arg=0x6341a0) at ParallelWorker.c:515
> >>> #7  0x00000033ca406407 in start_thread () from /lib64/libpthread.so.0
> >>> #8  0x00000033c94d4b0d in clone () from /lib64/libc.so.6
> >>>
> >>> where the thread 2 is poll()ing never never returns, so never signals
> >>> the poll() completion and than all the others
> >>> waiters in the MPIDI_CH3I_Progress() condition will never wake up.
> >>>
> >>> Does anyone is having the same problem?
> >>>
> >>> Thanks in advance,
> >>> Roberto Fichera.
> >>>
> >>>
> >>
> >>
> >>
> >
> > ------------------------------------------------------------------------
> >
> > _______________________________________________
> > mvapich-discuss mailing list
> > mvapich-discuss at cse.ohio-state.edu
> > http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss
> >
>
>