[mvapich-discuss] MVAPICH 2.3a-GDR on POWER8: Run Problems

[Ammar Ahmad Awan]

Hi Andreas,

Glad to know that you are able to run the benchmarks now.

Jiri's suggestion to disable it automatically is on our roadmap. We plan to
do this for the next release.

Please note that there shall be no side effects of disabling GPUDirect
support using MV2_USE_GPUDIRECT=0.

Regards,
Ammar


On Tue, Dec 19, 2017 at 8:56 AM, Herten, Andreas <a.herten at fz-juelich.de>
wrote:

> Hi Sourav,
>
> Indeed, setting `MV2_USE_GPUDIRECT=0` will lead to working OSU benchmarks.
> Thanks for the hint!
>
> I’d like to reiterate Jiri Kraus’ suggestion: It would be great if MVAPICH
> would detect the need to disable the variable automatically when run on a
> POWER system.
>
> Does `MV2_USE_GPUDIRECT=0` have further side effects? For instance, does
> MVAPICH still use GPUDirect P2P when this variable is set?
>
> Thanks for your support,
>
> -Andreas
>
> —
> NVIDIA Application Lab
> Jülich Supercomputing Centre
> Forschungszentrum Jülich, Germany
> +49 2461 61 1825 <+49%202461%20611825>
>
> Am 06.12.2017 um 21:25 schrieb Sourav Chakraborty <
> chakraborty.52 at buckeyemail.osu.edu>:
>
> Hi Andreas,
>
> GPUDIRECT currently does not work on OpenPOWER systems. Can you please set
> run the test with MV2_USE_GPUDIRECT=0?.
>
> Thanks,
> Sourav
>
>
> On Tue, Dec 5, 2017 at 8:26 AM, Herten, Andreas <a.herten at fz-juelich.de>
> wrote:
>
>> Dear Sourav, Dear all,
>>
>> After running quite a lot of jobs to try to determine where the problems
>> arise from, let me finally come back to you with some insights. (In the
>> following log files, the file / parameters used for job submission are
>> printed towards the very end of the file.)
>>
>> • mpirun, -np 2, -hostfile: Tried this, it did not change anything – see
>> file mpirun-np-hostfile.log. I noticed here that run times greatly
>> fluctuate. There are rare occurrences, in which the benchmark runs
>> flawlessly. But in the majority, it is stuck as reported. That is, it
>> *looks* like being stuck. By chance I kept some jobs running (as with
>> the current log file) and found out that after durations of up to 10
>> minutes, the benchmark still completes. I was unable to determine any
>> systematics with respect to very-slow benchmarks vs. flawless benchmark. My
>> feeling was, that the small message sizes were particularly slow, while
>> larger message sizes would run faster.
>> • mpirun_rsh, -export, -np 2, -hostfile: mpirun_rsh does not seem to
>> work. See attached log file mpirun_rsh-hostfile.log.
>>
>> In our quest to understand the underlying behaviour, we had a look at
>> environment variables with which to tune MVAPICH, particularly the
>> GPU-aware part. We set »MV2_GPUDIRECT_LIMIT=0« (default: 8192) in the
>> environment – and with this, the benchmarks seem not to be stuck anymore!
>> Find attached the log file mpirun-gpudirect.log. There is still some
>> curious bandwidth fluctuations which do not appear to be systematic.
>> In mpirun-gpudirect-recv.log we also set »MV2_USE_GPUDIRECT_RECEIVE_LIMIT=0«,
>> apparently reducing the fluctuations.
>>
>> The topology of the Minsky POWER8 system as given by `nvidia-smi topo -m`
>> is the following. Maybe this is information you find useful:
>> GPU0 GPU1 GPU2 GPU3 mlx5_0 CPU Affinity
>> GPU0  X  NV2 SOC SOC SOC 0-79
>> GPU1 NV2  X  SOC SOC SOC 0-79
>> GPU2 SOC SOC  X  NV2 SOC 80-159
>> GPU3 SOC SOC NV2  X  SOC 80-159
>> mlx5_0 SOC SOC SOC SOC  X
>>
>> We surely would like to use MVAPICH’s GPUDirect RDMA feature in our
>> system! How can we?
>>
>> Best,
>>
>> -Andreas
>> EDIT: Again with log files.
>> —
>> NVIDIA Application Lab
>> Jülich Supercomputing Centre
>> Forschungszentrum Jülich, Germany
>> +49 2461 61 1825 <+49%202461%20611825>
>>
>>
>>
>>
>>
>> Am 25.11.2017 um 15:23 schrieb Sourav Chakraborty <
>> chakraborty.52 at buckeyemail.osu.edu>:
>>
>> Hi Dr. Herten,
>>
>> Can you please try specifying the number of processes and the hostfile?
>> Example:
>>
>> mpirun -np 2 -hostfile $LSB_DJOB_HOSTFILE ./get_local_rank ./osu_bibw D D
>>
>> mpirun_rsh can also be used in place of mpirun.
>>
>> mpirun -export -np 2 -hostfile $LSB_DJOB_HOSTFILE ./get_local_rank
>> ./osu_bibw D D
>>
>> (By default mpirun_rsh does not export the environment variables and
>> specifying -export option is required.)
>>
>> Thank you,
>> Sourav
>>
>> On Thu, Nov 23, 2017 at 7:45 AM, Herten, Andreas <a.herten at fz-juelich.de>
>> wrote:
>>
>>> Dear all,
>>>
>>> I’m involved in operation of a POWER8+NVIDIA cluster, JURON, at Jülich
>>> Supercomputing Centre.
>>>
>>> I downloaded the GDR built for POWER but am unable to run CUDA-aware MPI
>>> applications.
>>>
>>> Testing with the packaged OSU microbenchmarks, the program is stuck
>>> right at the beginning. See below.
>>>
>>> JURON runs CentOS with GCC 4.8.5, but we also tested loading GCC 5.4.0.
>>> We use an EDR InfiniBand interconnect. Our batch system is LSF – is MVAPICH
>>> built with LSF support?
>>> Please let me know if you need further information for debugging.
>>>
>>> > $ env | grep MV2
>>> > MV2_ENABLE_AFFINITY=0
>>> > MV2_USE_CUDA=1
>>> > MV2_USE_GPUDIRECT_GDRCOPY=0
>>>
>>> > $ bsub -n 2 -I -R "rusage[ngpus_shared=20]" mpirun
>>> /gpfs/software/opt/mvapich2/2.3a-GDR-cuda_8/libexec/osu-micro-benchmarks/get_local_rank
>>> /gpfs/software/opt/mvapich2/2.3a-GDR-cuda_8/libexec/osu-micr
>>> o-benchmarks/mpi/pt2pt/osu_bibw D D
>>> > Job <9706> is submitted to default queue <normal.i>.
>>> > <<Waiting for dispatch ...>>
>>> > <<Starting on juronc05>>
>>> > # OSU MPI-CUDA Bi-Directional Bandwidth Test v5.4.0
>>> > # Send Buffer on DEVICE (D) and Receive Buffer on DEVICE (D)
>>> > # Size      Bandwidth (MB/s)
>>>
>>> -Andreas
>>> —
>>> Dr. Andreas Herten
>>> NVIDIA Application Lab
>>> Jülich Supercomputing Centre
>>> Forschungszentrum Jülich, Germany
>>> +49 2461 61 1825 <+49%202461%20611825>
>>>
>>>
>>> _______________________________________________
>>> mvapich-discuss mailing list
>>> mvapich-discuss at cse.ohio-state.edu
>>> http://mailman.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss
>>>
>>>
>>
>>
>>
>
>
> _______________________________________________
> mvapich-discuss mailing list
> mvapich-discuss at cse.ohio-state.edu
> http://mailman.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.cse.ohio-state.edu/pipermail/mvapich-discuss/attachments/20171219/e0bbfb8d/attachment-0001.html>