[mvapich-discuss] MVAPICH 2.3a-GDR on POWER8: Run Problems

[Sourav Chakraborty]

Hi Andreas,

GPUDIRECT currently does not work on OpenPOWER systems. Can you please set
run the test with MV2_USE_GPUDIRECT=0?.

Thanks,
Sourav


On Tue, Dec 5, 2017 at 8:26 AM, Herten, Andreas <a.herten at fz-juelich.de>
wrote:

> Dear Sourav, Dear all,
>
> After running quite a lot of jobs to try to determine where the problems
> arise from, let me finally come back to you with some insights. (In the
> following log files, the file / parameters used for job submission are
> printed towards the very end of the file.)
>
> • mpirun, -np 2, -hostfile: Tried this, it did not change anything – see
> file mpirun-np-hostfile.log. I noticed here that run times greatly
> fluctuate. There are rare occurrences, in which the benchmark runs
> flawlessly. But in the majority, it is stuck as reported. That is, it
> *looks* like being stuck. By chance I kept some jobs running (as with the
> current log file) and found out that after durations of up to 10 minutes,
> the benchmark still completes. I was unable to determine any systematics
> with respect to very-slow benchmarks vs. flawless benchmark. My feeling
> was, that the small message sizes were particularly slow, while larger
> message sizes would run faster.
> • mpirun_rsh, -export, -np 2, -hostfile: mpirun_rsh does not seem to work.
> See attached log file mpirun_rsh-hostfile.log.
>
> In our quest to understand the underlying behaviour, we had a look at
> environment variables with which to tune MVAPICH, particularly the
> GPU-aware part. We set »MV2_GPUDIRECT_LIMIT=0« (default: 8192) in the
> environment – and with this, the benchmarks seem not to be stuck anymore!
> Find attached the log file mpirun-gpudirect.log. There is still some
> curious bandwidth fluctuations which do not appear to be systematic.
> In mpirun-gpudirect-recv.log we also set »MV2_USE_GPUDIRECT_RECEIVE_LIMIT=0«,
> apparently reducing the fluctuations.
>
> The topology of the Minsky POWER8 system as given by `nvidia-smi topo -m`
> is the following. Maybe this is information you find useful:
> GPU0 GPU1 GPU2 GPU3 mlx5_0 CPU Affinity
> GPU0  X  NV2 SOC SOC SOC 0-79
> GPU1 NV2  X  SOC SOC SOC 0-79
> GPU2 SOC SOC  X  NV2 SOC 80-159
> GPU3 SOC SOC NV2  X  SOC 80-159
> mlx5_0 SOC SOC SOC SOC  X
>
> We surely would like to use MVAPICH’s GPUDirect RDMA feature in our
> system! How can we?
>
> Best,
>
> -Andreas
> EDIT: Again with log files.
> —
> NVIDIA Application Lab
> Jülich Supercomputing Centre
> Forschungszentrum Jülich, Germany
> +49 2461 61 1825 <+49%202461%20611825>
>
>
>
>
>
> Am 25.11.2017 um 15:23 schrieb Sourav Chakraborty <
> chakraborty.52 at buckeyemail.osu.edu>:
>
> Hi Dr. Herten,
>
> Can you please try specifying the number of processes and the hostfile?
> Example:
>
> mpirun -np 2 -hostfile $LSB_DJOB_HOSTFILE ./get_local_rank ./osu_bibw D D
>
> mpirun_rsh can also be used in place of mpirun.
>
> mpirun -export -np 2 -hostfile $LSB_DJOB_HOSTFILE ./get_local_rank
> ./osu_bibw D D
>
> (By default mpirun_rsh does not export the environment variables and
> specifying -export option is required.)
>
> Thank you,
> Sourav
>
> On Thu, Nov 23, 2017 at 7:45 AM, Herten, Andreas <a.herten at fz-juelich.de>
> wrote:
>
>> Dear all,
>>
>> I’m involved in operation of a POWER8+NVIDIA cluster, JURON, at Jülich
>> Supercomputing Centre.
>>
>> I downloaded the GDR built for POWER but am unable to run CUDA-aware MPI
>> applications.
>>
>> Testing with the packaged OSU microbenchmarks, the program is stuck right
>> at the beginning. See below.
>>
>> JURON runs CentOS with GCC 4.8.5, but we also tested loading GCC 5.4.0.
>> We use an EDR InfiniBand interconnect. Our batch system is LSF – is MVAPICH
>> built with LSF support?
>> Please let me know if you need further information for debugging.
>>
>> > $ env | grep MV2
>> > MV2_ENABLE_AFFINITY=0
>> > MV2_USE_CUDA=1
>> > MV2_USE_GPUDIRECT_GDRCOPY=0
>>
>> > $ bsub -n 2 -I -R "rusage[ngpus_shared=20]" mpirun
>> /gpfs/software/opt/mvapich2/2.3a-GDR-cuda_8/libexec/osu-micro-benchmarks/get_local_rank
>> /gpfs/software/opt/mvapich2/2.3a-GDR-cuda_8/libexec/osu-micr
>> o-benchmarks/mpi/pt2pt/osu_bibw D D
>> > Job <9706> is submitted to default queue <normal.i>.
>> > <<Waiting for dispatch ...>>
>> > <<Starting on juronc05>>
>> > # OSU MPI-CUDA Bi-Directional Bandwidth Test v5.4.0
>> > # Send Buffer on DEVICE (D) and Receive Buffer on DEVICE (D)
>> > # Size      Bandwidth (MB/s)
>>
>> -Andreas
>> —
>> Dr. Andreas Herten
>> NVIDIA Application Lab
>> Jülich Supercomputing Centre
>> Forschungszentrum Jülich, Germany
>> +49 2461 61 1825 <+49%202461%20611825>
>>
>>
>> _______________________________________________
>> mvapich-discuss mailing list
>> mvapich-discuss at cse.ohio-state.edu
>> http://mailman.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss
>>
>>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.cse.ohio-state.edu/pipermail/mvapich-discuss/attachments/20171206/5581a6a8/attachment.html>