[mvapich-discuss] mvapich 2.3 and slurm with pmi2

Fri Jul 27 01:31:03 EDT 2018

Hello Sourav,

thanks for the reply. These two parameters indeed fix my issue and result
in the expected output of

"OSU MPI Hello World Test v5.4.3
This is a test with 112 processes"

wich mvapich 2.3.
Indeed it is enough to set "export MV2_USE_RDMA_CM=0" to "fix" the startup
problem.
What surprises me is, that the command works as expected till a certain
number of processes.
Is this a configuration problem on the HPC or a problem with mvapich?

Thanks for the support again!

Greetings
Mathias

Am Do., 26. Juli 2018 um 19:58 Uhr schrieb Sourav Chakraborty <
chakraborty.52 at buckeyemail.osu.edu>:

> Hi Matthias,
>
> Thanks for providing the information. Can you try setting these two
> parameters and see if it fixes the issue?
>
> export MV2_USE_RDMA_CM=0
> export MV2_USE_RING_STARTUP=0
>
> Thanks,
> Sourav
>
>
> On Wed, Jul 25, 2018 at 8:01 AM Mathias Anselmann <
> mathias.anselmann at gmail.com> wrote:
>
>> Hello Sourav,
>> I try to answer your questions as good as I can:
>>
>> 1.)
>> I downloaded OMB and ran the "osu_hello" program. Results:
>> MVAPICH 2.2:
>> No problem, if I salloc with 224 tasks and run "srun -n112 ./osu_hello"
>> the output is:
>>
>> # OSU MPI Hello World Test v5.4.3
>> This is a test with 112 processes
>>
>> MVAPICH 2.3:
>> If I do the same with mvapich 2.3 it just stucks, as my own program.
>> The output of pstack <pid> for osu_hello is:
>>
>> $ pstack 13692
>> Thread 5 (Thread 0x2b6c88d27700 (LWP 13696)):
>> #0  0x00002b6c88ff6411 in sigwait () from /usr/lib64/libpthread.so.0
>> #1  0x00000000005a6ed8 in _srun_signal_mgr (job_ptr=0x151ae10) at
>> srun_job.c:1422
>> #2  0x00002b6c88feedd5 in start_thread () from /usr/lib64/libpthread.so.0
>> #3  0x00002b6c89301b3d in clone () from /usr/lib64/libc.so.6
>> Thread 4 (Thread 0x2b6c8a639700 (LWP 13697)):
>> #0  0x00002b6c892f6e9d in poll () from /usr/lib64/libc.so.6
>> #1  0x000000000042a96a in _poll_internal (pfds=0x2b6c8c0009e0, nfds=2,
>> shutdown_time=0) at eio.c:362
>> #2  0x000000000042a737 in eio_handle_mainloop (eio=0x2b6c8c0008d0) at
>> eio.c:326
>> #3  0x00002b6c89c0eb6c in _agent (unused=0x0) at agent.c:327
>> #4  0x00002b6c88feedd5 in start_thread () from /usr/lib64/libpthread.so.0
>> #5  0x00002b6c89301b3d in clone () from /usr/lib64/libc.so.6
>> Thread 3 (Thread 0x2b6c8a73a700 (LWP 13698)):
>> #0  0x00002b6c892f6e9d in poll () from /usr/lib64/libc.so.6
>> #1  0x000000000042a96a in _poll_internal (pfds=0x2b6c900008d0, nfds=3,
>> shutdown_time=0) at eio.c:362
>> #2  0x000000000042a737 in eio_handle_mainloop (eio=0x1520880) at eio.c:326
>> #3  0x00000000005969b4 in _msg_thr_internal (arg=0x151b200) at
>> step_launch.c:1053
>> #4  0x00002b6c88feedd5 in start_thread () from /usr/lib64/libpthread.so.0
>> #5  0x00002b6c89301b3d in clone () from /usr/lib64/libc.so.6
>> Thread 2 (Thread 0x2b6c8a83b700 (LWP 13699)):
>> #0  0x00002b6c892f6e9d in poll () from /usr/lib64/libc.so.6
>> #1  0x000000000042a96a in _poll_internal (pfds=0x2b6c94000dc0, nfds=6,
>> shutdown_time=0) at eio.c:362
>> #2  0x000000000042a737 in eio_handle_mainloop (eio=0x1524010) at eio.c:326
>> #3  0x0000000000592161 in _io_thr_internal (cio_arg=0x1523d90) at
>> step_io.c:810
>> #4  0x00002b6c88feedd5 in start_thread () from /usr/lib64/libpthread.so.0
>> #5  0x00002b6c89301b3d in clone () from /usr/lib64/libc.so.6
>> Thread 1 (Thread 0x2b6c88c24bc0 (LWP 13692)):
>> #0  0x00002b6c88ff2945 in pthread_cond_wait@@GLIBC_2.3.2 () from
>> /usr/lib64/libpthread.so.0
>> #1  0x00000000005954f3 in slurm_step_launch_wait_finish (ctx=0x151b1b0)
>> at step_launch.c:627
>> #2  0x00002b6c89e282b5 in launch_p_step_wait (job=0x151ae10,
>> got_alloc=false) at launch_slurm.c:698
>> #3  0x000000000059c763 in launch_g_step_wait (job=0x151ae10,
>> got_alloc=false) at launch.c:521
>> #4  0x000000000042921a in srun (ac=5, av=0x7ffc6b4b7148) at srun.c:262
>> #5  0x0000000000429cfe in main (argc=5, argv=0x7ffc6b4b7148) at
>> srun.wrapper.c:17
>>
>>
>> 2.) The system is hyperthreaded. The application doesn't support
>> threading. If I start the app with as many processes as tasks every
>> processor gets full load on the "real" core and the "virtual core". To
>> overcome this issue I use n with half the number of the tasks, so every
>> processor gets one process which results in full load on  the physical
>> processor.
>>
>> 3.) No, the configuration is homogenious.
>>
>> I hope that helps debugging.
>>
>> Greetings,
>>
>> Mathias
>>
>> Am Di., 24. Juli 2018 um 20:06 Uhr schrieb Sourav Chakraborty <
>> chakraborty.52 at buckeyemail.osu.edu>:
>>
>>> Hi Mathias,
>>>
>>> Thanks for your report. We were not able to reproduce this issue
>>> locally. Can you please provide the following information so that we could
>>> debug this further?
>>>
>>> 1. Does the hang happen only for this particular application or any
>>> application (e.g. OMB?) Can you provide a stacktrace of the application by
>>> running pstack <pid>?
>>>
>>> 2. What did you mean by "disable threading"? Is this a hyperthreaded
>>> system? Is the application multi-threaded?
>>>
>>> 3. Does the system have multiple HCAs per node or a heterogeneous
>>> configuration (Different CPU/HCA across nodes)?
>>>
>>> Thanks,
>>> Sourav
>>>
>>>
>>>
>>> On Tue, Jul 24, 2018 at 9:52 AM Mathias Anselmann <
>>> mathias.anselmann at gmail.com> wrote:
>>>
>>>> Hello,
>>>> I just wanted to try out the latest version of mvapich on our HPC,
>>>> running RedHat 7.3 with slurm.
>>>> On the machine I have installed locally a GCC 7.3 and compiled mvapich
>>>> (locally) with:
>>>>
>>>> --enable-shared --with-pm=slurm --with-pmi=pmi2
>>>>
>>>> For testing purposes I start the my program via srun (after allocating
>>>> some nodes via "salloc").
>>>> So, e.g.:
>>>>
>>>> salloc --ntasks=224 --exclusive bash
>>>>
>>>> and then:
>>>>
>>>> srun -n 112 --mpi=pmi2 ./ProgramBinary
>>>>
>>>> In srun I use half of the --ntasks argument to disable threading.
>>>>
>>>> With mvapich 2.2 this works like a charm and I tested it up to a
>>>> --ntasks=560 and -n 280. My program starts within seconds and runs as
>>>> expected.
>>>> I used the same configure flags for mvapich 2.2 as for 2.3 now.
>>>>
>>>> With 2.3 I have the following issue:
>>>> srun works fine, but if I increase the nodes it won't start my program.
>>>> With --ntasks=168 and -n 84 srun works like a charm, but if I go for
>>>> --ntasks=224 and -n 112 srun seems to not start the program. There comes no
>>>> output and I manually cancelled the job after 15 minutes of waiting.
>>>>
>>>> Does anybody have a clue what's going on here?
>>>> If more infos are needed, I can provide them of course.
>>>>
>>>> Greetings,
>>>>
>>>> Mathias
>>>> _______________________________________________
>>>> mvapich-discuss mailing list
>>>> mvapich-discuss at cse.ohio-state.edu
>>>> http://mailman.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss
>>>>
>>> _______________________________________________
>> mvapich-discuss mailing list
>> mvapich-discuss at cse.ohio-state.edu
>> http://mailman.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.cse.ohio-state.edu/pipermail/mvapich-discuss/attachments/20180727/29305f99/attachment-0001.html>