[mvapich-discuss] Races with MPI_THREAD_MULTI (fwd)

Roberto Fichera kernel at tekno-soft.it
Tue Jul 22 14:27:50 EDT 2008 

wei huang ha scritto:
> Hi Roberto,
>
>   
>> MPI_Send() to MPI_Ssend() everything
>> seems working well with mpich2 v1.0.7. My application doesn't race
>> anymore at least after dispatching
>> 50.000 jobs across 4 nodes, but trying to execute the same application
>> against the last mvapich2 1.2rc1
>> I'm still getting the same problem as shown below.
>>     
>
> Good to know that it works with mpich2. How do you configure/compile
> mvapich2-1.2rc1? Are you using the ofed interface (default configuration),
> or using the TCP/IP stack (adding --with-device=ch3:sock in the configure
> option)? The TCP/IP stack should be the same as mpich2, in other words, it
> should work with your application. The implementation on ofed interface
> does not support spawn yet, thus there will be issues.
>   
Here is how I configured it, which is basically the same as for the mpich2

PKG_PATH=/HRI/External/mvapich2/1.2rc1

./configure --prefix=$PKG_PATH \
            --bindir=$PKG_PATH/bin/${MAKEFILE_PLATFORM} \
            --sbindir=$PKG_PATH/bin/${MAKEFILE_PLATFORM} \
            --libdir=$PKG_PATH/lib/${MAKEFILE_PLATFORM} \
            --enable-sharedlibs=gcc \
            --enable-f90 \
            --enable-cxx \
            --enable-romio \
            --enable-threads=multiple \
            --enable-g=gdb \
            --enable-debuginfo \
            --with-device=ch3:sock \
            --datadir=$PKG_PATH/data  \
            --with-htmldir=$PKG_PATH/doc/html \
            --with-docdir=$PKG_PATH/doc \
            LDFLAGS='-Wl,-z,noexecstack'

-----------------------------------------------------------
PKG_PATH=/HRI/External/mpich2/1.0.7


./configure --prefix=$PKG_PATH \
            --bindir=$PKG_PATH/bin/$MAKEFILE_PLATFORM \
            --sbindir=$PKG_PATH/bin/$MAKEFILE_PLATFORM \
            --libdir=$PKG_PATH/lib/$MAKEFILE_PLATFORM \
            --enable-sharedlibs=gcc \
            --enable-f90 \
            --enable-cxx \
            --enable-romio \
            --enable-mpe \
            --enable-threads=multiple \
            --enable-debuginfo \
            --enable-g=gdb \
            --with-device=ch3:sock \
            --datadir=$PKG_PATH/data  \
            --with-htmldir=$PKG_PATH/doc/html \
            --with-docdir=$PKG_PATH/doc \
            LDFLAGS='-Wl,-z,noexecstack'

>   
>> I've another question, since this multithreaded application has to run
>> into a cluster with 1024 nodes equiped
>> with Mellanox IB card, I really like to know if the OpenFabrics-IB
>> interface does support the MPI_THREAD_MULTIPLE
>> initialization and also the MPI_Comm_spawn() implementation.
>>     
>
> mvapich2 on the ofed (IB) interface supports MPI_THREAD_MULTIPLE.
> MPI_Comm_spawn() is not yet supported, however. We are working on this
> support and it will be available in our next release.
>   
So the question now become, when do you plan to release the ofed version 
with MPI_Comm_spawn() support ;-)?
> Thanks.
>
> -- Wei
>
>
>   
>>>> On Fri, 18 Jul 2008, Roberto Fichera wrote:
>>>>
>>>>
>>>>         
>>>>> Hi All on the list,
>>>>>
>>>>> I'm trying to use mvapich2 v1.2rc1 in a multithreaded application,
>>>>> initialize using MPI_THREAD_MULTI.
>>>>> I've the master application doing the following thing, start several
>>>>> thread depending by the assigned nodes,
>>>>> on each node a slave application is spawned using the MPI_Comm_spawn().
>>>>> Before to call the
>>>>> MPI_Comm_spawn() I prepare the given MPI_Info struct, one for each
>>>>> thread, in order to set the all keys
>>>>> (host and wdir) for addressing the wanted behaviour. So, as sooner as
>>>>> the master application starts, it races
>>>>> immediately with 4 nodes, 1 master and 3 slaves. Below you can see the
>>>>> status of the master application at race
>>>>> time. It seems stuck on the PMIU_readline() which never returns so the
>>>>> global lock is never relesead. MVAPICH2
>>>>> is compiled with:
>>>>>
>>>>> PKG_PATH=/HRI/External/mvapich2/1.2rc1
>>>>>
>>>>> ./configure --prefix=$PKG_PATH \
>>>>>             --bindir=$PKG_PATH/bin/linux-x86_64-gcc-glibc2.3.4 \
>>>>>             --sbindir=$PKG_PATH/bin/linux-x86_64-gcc-glibc2.3.4 \
>>>>>             --libdir=$PKG_PATH/lib/linux-x86_64-gcc-glibc2.3.4 \
>>>>>             --enable-sharedlibs=gcc \
>>>>>             --enable-f90 \
>>>>>             --enable-threads=multiple \
>>>>>             --enable-g=-ggdb \
>>>>>             --enable-debuginfo \
>>>>>             --with-device=ch3:sock \
>>>>>             --datadir=$PKG_PATH/data  \
>>>>>             --with-htmldir=$PKG_PATH/doc/html \
>>>>>             --with-docdir=$PKG_PATH/doc \
>>>>>             LDFLAGS='-Wl,-z,noexecstack'
>>>>>
>>>>> so I'm using the ch3:sock device.
>>>>>
>>>>> -----Thread 2
>>>>> [Switching to thread 2 (Thread 1115699536 (LWP 29479))]#0
>>>>> 0x00000033ca40cef4 in __lll_lock_wait () from /lib64/libpthread.so.0
>>>>> (gdb) bt
>>>>> #0  0x00000033ca40cef4 in __lll_lock_wait () from /lib64/libpthread.so.0
>>>>> #1  0x00000033ca408915 in _L_lock_102 () from /lib64/libpthread.so.0
>>>>> --->>#2  0x00000033ca408390 in pthread_mutex_lock () from
>>>>> /lib64/libpthread.so.0
>>>>> --->>#3  0x00002aaaab382654 in PMPI_Info_set () from
>>>>> /home/roberto/.HRI/Proxy/HRI/External/mvapich2/1.2/lib/linux-x86_64-gcc-glibc2.3.4/libmpich.so.1.1
>>>>> #4  0x0000000000417627 in ParallelWorker_setSlaveInfo (self=<value
>>>>> optimized out>, key=0x0, value=0x33ca40ff58
>>>>> "!\204��\r\206��\030\204��3\206��\n\204��\n\204��\n\204��\n\204��\n\204��\n\204��\n\204��\n\204��\n\204��\n\204��\n\204��\n\204��\177\205��\177\205��\177\205��\177\205��\n\204��\n\204��\n\204��\n\204��\n\204��\n\204��\n\204��\n\204��\n\204��\n\204��\n\204��\n\204��\033\205��\033\205��\033\205��\033\205��\n\204��\n\204��\n\204��\n\204��\n\204��\n\204��\n\204��\n\204��\n\204��\n\204��\n\204��\n\204��\033\205��\033\205��"...)
>>>>> at ParallelWorker.c:664
>>>>> #5  0x0000000000418905 in ParallelWorker_handleParallel (self=0x62ff50)
>>>>> at ParallelWorker.c:719
>>>>> #6  0x000000000041b39e in ParallelWorker_threadMain (arg=0x62ff50) at
>>>>> ParallelWorker.c:504
>>>>> #7  0x00000033ca406407 in start_thread () from /lib64/libpthread.so.0
>>>>> #8  0x00000033c94d4b0d in clone () from /lib64/libc.so.6
>>>>>
>>>>> -----Thread 3
>>>>> [Switching to thread 3 (Thread 1105209680 (LWP 29478))]#0
>>>>> 0x00000033ca40cef4 in __lll_lock_wait () from /lib64/libpthread.so.0
>>>>> (gdb) bt
>>>>> #0  0x00000033ca40cef4 in __lll_lock_wait () from /lib64/libpthread.so.0
>>>>> #1  0x00000033ca408915 in _L_lock_102 () from /lib64/libpthread.so.0
>>>>> --->>#2  0x00000033ca408390 in pthread_mutex_lock () from
>>>>> /lib64/libpthread.so.0
>>>>> --->>#3  0x00002aaaab382654 in PMPI_Info_set () from
>>>>> /home/roberto/.HRI/Proxy/HRI/External/mvapich2/1.2/lib/linux-x86_64-gcc-glibc2.3.4/libmpich.so.1.1
>>>>> #4  0x0000000000417627 in ParallelWorker_setSlaveInfo (self=<value
>>>>> optimized out>, key=0x0, value=0x33ca40ff58
>>>>> "!\204��\r\206��\030\204��3\206��\n\204��\n\204��\n\204��\n\204��\n\204��\n\204��\n\204��\n\204��\n\204��\n\204��\n\204��\n\204��\177\205��\177\205��\177\205��\177\205��\n\204��\n\204��\n\204��\n\204��\n\204��\n\204��\n\204��\n\204��\n\204��\n\204��\n\204��\n\204��\033\205��\033\205��\033\205��\033\205��\n\204��\n\204��\n\204��\n\204��\n\204��\n\204��\n\204��\n\204��\n\204��\n\204��\n\204��\n\204��\033\205��\033\205��"...)
>>>>> at ParallelWorker.c:664
>>>>> #5  0x0000000000418905 in ParallelWorker_handleParallel (self=0x62f270)
>>>>> at ParallelWorker.c:719
>>>>> #6  0x000000000041b39e in ParallelWorker_threadMain (arg=0x62f270) at
>>>>> ParallelWorker.c:504
>>>>> #7  0x00000033ca406407 in start_thread () from /lib64/libpthread.so.0
>>>>> #8  0x00000033c94d4b0d in clone () from /lib64/libc.so.6
>>>>>
>>>>> -----Thread 4
>>>>> [Switching to thread 4 (Thread 1094719824 (LWP 29477))]#0
>>>>> 0x00000033ca40d34b in read () from /lib64/libpthread.so.0
>>>>> (gdb) bt
>>>>> #0  0x00000033ca40d34b in read () from /lib64/libpthread.so.0
>>>>> --->>#1  0x00002aaaab3db84a in PMIU_readline () from
>>>>> /home/roberto/.HRI/Proxy/HRI/External/mvapich2/1.2/lib/linux-x86_64-gcc-glibc2.3.4/libmpich.so.1.1
>>>>> --->>#2  0x00002aaaab3d9d37 in PMI_Spawn_multiple () from
>>>>> /home/roberto/.HRI/Proxy/HRI/External/mvapich2/1.2/lib/linux-x86_64-gcc-glibc2.3.4/libmpich.so.1.1
>>>>> #3  0x00002aaaab333893 in MPIDI_Comm_spawn_multiple () from
>>>>> /home/roberto/.HRI/Proxy/HRI/External/mvapich2/1.2/lib/linux-x86_64-gcc-glibc2.3.4/libmpich.so.1.1
>>>>> #4  0x00002aaaab38bcf6 in MPID_Comm_spawn_multiple () from
>>>>> /home/roberto/.HRI/Proxy/HRI/External/mvapich2/1.2/lib/linux-x86_64-gcc-glibc2.3.4/libmpich.so.1.1
>>>>> #5  0x00002aaaab355a10 in PMPI_Comm_spawn () from
>>>>> /home/roberto/.HRI/Proxy/HRI/External/mvapich2/1.2/lib/linux-x86_64-gcc-glibc2.3.4/libmpich.so.1.1
>>>>> #6  0x00000000004189d8 in ParallelWorker_handleParallel (self=0x62ad40)
>>>>> at ParallelWorker.c:754
>>>>> #7  0x000000000041b39e in ParallelWorker_threadMain (arg=0x62ad40) at
>>>>> ParallelWorker.c:504
>>>>> #8  0x00000033ca406407 in start_thread () from /lib64/libpthread.so.0
>>>>> #9  0x00000033c94d4b0d in clone () from /lib64/libc.so.6
>>>>>
>>>>> I also tried to run against MPICH2 v1.0.7, but here I got a similar
>>>>> scenery which show up after between 1 - 2 hours of execution,
>>>>> see below:
>>>>>
>>>>> ----- thread 2
>>>>> [Switching to thread 2 (Thread 1094719824 (LWP 1279))]#0  0x00000033c94cbd66 in poll () from /lib64/libc.so.6
>>>>> (gdb) bt
>>>>> #0  0x00000033c94cbd66 in poll () from /lib64/libc.so.6
>>>>> #1  0x00002aaaab5a3d2f in MPIDU_Sock_wait () from /home/roberto/.HRI/Proxy/HRI/External/mpich2/1.0.7/lib/linux-x86_64-gcc-glibc2.3.4/libmpich.so.1.1
>>>>> #2  0x00002aaaab52bdc7 in MPIDI_CH3I_Progress () from /home/roberto/.HRI/Proxy/HRI/External/mpich2/1.0.7/lib/linux-x86_64-gcc-glibc2.3.4/libmpich.so.1.1
>>>>> #3  0x00002aaaab5301a7 in MPIDI_CH3U_VC_WaitForClose () from /home/roberto/.HRI/Proxy/HRI/External/mpich2/1.0.7/lib/linux-x86_64-gcc-glibc2.3.4/libmpich.so.1.1
>>>>> #4  0x00002aaaab56f162 in MPID_Comm_disconnect () from /home/roberto/.HRI/Proxy/HRI/External/mpich2/1.0.7/lib/linux-x86_64-gcc-glibc2.3.4/libmpich.so.1.1
>>>>> #5  0x00002aaaab5417ec in PMPI_Comm_disconnect () from /home/roberto/.HRI/Proxy/HRI/External/mpich2/1.0.7/lib/linux-x86_64-gcc-glibc2.3.4/libmpich.so.1.1
>>>>> #6  0x00002aaaabda5a99 in ParallelWorker_destroySlave (self=0x6358e0) at ParallelWorker.c:819
>>>>> #7  0x00002aaaabda6223 in ParallelWorker_threadMain (arg=0x6358e0) at ParallelWorker.c:515
>>>>> #8  0x00000033ca406407 in start_thread () from /lib64/libpthread.so.0
>>>>> #9  0x00000033c94d4b0d in clone () from /lib64/libc.so.6
>>>>>
>>>>> ----- thread 3
>>>>> [Switching to thread 3 (Thread 1084229968 (LWP 1278))]#0  0x00000033ca40a8f9 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
>>>>> (gdb) bt
>>>>> #0  0x00000033ca40a8f9 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
>>>>> #1  0x00002aaaab52bec7 in MPIDI_CH3I_Progress () from /home/roberto/.HRI/Proxy/HRI/External/mpich2/1.0.7/lib/linux-x86_64-gcc-glibc2.3.4/libmpich.so.1.1
>>>>> #2  0x00002aaaab5301a7 in MPIDI_CH3U_VC_WaitForClose () from /home/roberto/.HRI/Proxy/HRI/External/mpich2/1.0.7/lib/linux-x86_64-gcc-glibc2.3.4/libmpich.so.1.1
>>>>> #3  0x00002aaaab56f162 in MPID_Comm_disconnect () from /home/roberto/.HRI/Proxy/HRI/External/mpich2/1.0.7/lib/linux-x86_64-gcc-glibc2.3.4/libmpich.so.1.1
>>>>> #4  0x00002aaaab5417ec in PMPI_Comm_disconnect () from /home/roberto/.HRI/Proxy/HRI/External/mpich2/1.0.7/lib/linux-x86_64-gcc-glibc2.3.4/libmpich.so.1.1
>>>>> #5  0x00002aaaabda5a99 in ParallelWorker_destroySlave (self=0x634d20) at ParallelWorker.c:819
>>>>> #6  0x00002aaaabda6223 in ParallelWorker_threadMain (arg=0x634d20) at ParallelWorker.c:515
>>>>> #7  0x00000033ca406407 in start_thread () from /lib64/libpthread.so.0
>>>>> #8  0x00000033c94d4b0d in clone () from /lib64/libc.so.6
>>>>>
>>>>>
>>>>> ----- thread 4
>>>>> [Switching to thread 4 (Thread 1115699536 (LWP 1277))]#0  0x00000033ca40a8f9 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
>>>>> (gdb) bt
>>>>> #0  0x00000033ca40a8f9 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
>>>>> #1  0x00002aaaab52bec7 in MPIDI_CH3I_Progress () from /home/roberto/.HRI/Proxy/HRI/External/mpich2/1.0.7/lib/linux-x86_64-gcc-glibc2.3.4/libmpich.so.1.1
>>>>> #2  0x00002aaaab5301a7 in MPIDI_CH3U_VC_WaitForClose () from /home/roberto/.HRI/Proxy/HRI/External/mpich2/1.0.7/lib/linux-x86_64-gcc-glibc2.3.4/libmpich.so.1.1
>>>>> #3  0x00002aaaab56f162 in MPID_Comm_disconnect () from /home/roberto/.HRI/Proxy/HRI/External/mpich2/1.0.7/lib/linux-x86_64-gcc-glibc2.3.4/libmpich.so.1.1
>>>>> #4  0x00002aaaab5417ec in PMPI_Comm_disconnect () from /home/roberto/.HRI/Proxy/HRI/External/mpich2/1.0.7/lib/linux-x86_64-gcc-glibc2.3.4/libmpich.so.1.1
>>>>> #5  0x00002aaaabda5a99 in ParallelWorker_destroySlave (self=0x6341a0) at ParallelWorker.c:819
>>>>> #6  0x00002aaaabda6223 in ParallelWorker_threadMain (arg=0x6341a0) at ParallelWorker.c:515
>>>>> #7  0x00000033ca406407 in start_thread () from /lib64/libpthread.so.0
>>>>> #8  0x00000033c94d4b0d in clone () from /lib64/libc.so.6
>>>>>
>>>>> where the thread 2 is poll()ing never never returns, so never signals
>>>>> the poll() completion and than all the others
>>>>> waiters in the MPIDI_CH3I_Progress() condition will never wake up.
>>>>>
>>>>> Does anyone is having the same problem?
>>>>>
>>>>> Thanks in advance,
>>>>> Roberto Fichera.
>>>>>
>>>>>
>>>>>           
>>>>
>>>>         
>>> ------------------------------------------------------------------------
>>>
>>> _______________________________________________
>>> mvapich-discuss mailing list
>>> mvapich-discuss at cse.ohio-state.edu
>>> http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss
>>>
>>>       
>>     
>
>
>   

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.cse.ohio-state.edu/pipermail/mvapich-discuss/attachments/20080722/ccca9f3d/attachment-0001.html

More information about the mvapich-discuss mailing list