[mvapich-discuss] MVAPICH 2.3a-GDR on POWER8: Run Problems

Herten, Andreas a.herten at fz-juelich.de
Tue Dec 19 08:56:26 EST 2017 

Hi Sourav,

Indeed, setting `MV2_USE_GPUDIRECT=0` will lead to working OSU benchmarks. Thanks for the hint!

I’d like to reiterate Jiri Kraus’ suggestion: It would be great if MVAPICH would detect the need to disable the variable automatically when run on a POWER system.

Does `MV2_USE_GPUDIRECT=0` have further side effects? For instance, does MVAPICH still use GPUDirect P2P when this variable is set?

Thanks for your support,

-Andreas

—
NVIDIA Application Lab
Jülich Supercomputing Centre
Forschungszentrum Jülich, Germany
+49 2461 61 1825

> Am 06.12.2017 um 21:25 schrieb Sourav Chakraborty <chakraborty.52 at buckeyemail.osu.edu>:
> 
> Hi Andreas,
> 
> GPUDIRECT currently does not work on OpenPOWER systems. Can you please set run the test with MV2_USE_GPUDIRECT=0?.
> 
> Thanks,
> Sourav
> 
> 
> On Tue, Dec 5, 2017 at 8:26 AM, Herten, Andreas <a.herten at fz-juelich.de <mailto:a.herten at fz-juelich.de>> wrote:
> Dear Sourav, Dear all,
> 
> After running quite a lot of jobs to try to determine where the problems arise from, let me finally come back to you with some insights. (In the following log files, the file / parameters used for job submission are printed towards the very end of the file.)
> 
> • mpirun, -np 2, -hostfile: Tried this, it did not change anything – see file mpirun-np-hostfile.log. I noticed here that run times greatly fluctuate. There are rare occurrences, in which the benchmark runs flawlessly. But in the majority, it is stuck as reported. That is, it looks like being stuck. By chance I kept some jobs running (as with the current log file) and found out that after durations of up to 10 minutes, the benchmark still completes. I was unable to determine any systematics with respect to very-slow benchmarks vs. flawless benchmark. My feeling was, that the small message sizes were particularly slow, while larger message sizes would run faster.
> • mpirun_rsh, -export, -np 2, -hostfile: mpirun_rsh does not seem to work. See attached log file mpirun_rsh-hostfile.log.
> 
> In our quest to understand the underlying behaviour, we had a look at environment variables with which to tune MVAPICH, particularly the GPU-aware part. We set »MV2_GPUDIRECT_LIMIT=0« (default: 8192) in the environment – and with this, the benchmarks seem not to be stuck anymore!
> Find attached the log file mpirun-gpudirect.log. There is still some curious bandwidth fluctuations which do not appear to be systematic. In mpirun-gpudirect-recv.log we also set »MV2_USE_GPUDIRECT_RECEIVE_LIMIT=0«, apparently reducing the fluctuations.
> 
> The topology of the Minsky POWER8 system as given by `nvidia-smi topo -m` is the following. Maybe this is information you find useful:
> 	GPU0	GPU1	GPU2	GPU3	mlx5_0	CPU Affinity
> GPU0	 X 	NV2	SOC	SOC	SOC	0-79
> GPU1	NV2	 X 	SOC	SOC	SOC	0-79
> GPU2	SOC	SOC	 X 	NV2	SOC	80-159
> GPU3	SOC	SOC	NV2	 X 	SOC	80-159
> mlx5_0	SOC	SOC	SOC	SOC	 X
> 
> We surely would like to use MVAPICH’s GPUDirect RDMA feature in our system! How can we?
> 
> Best,
> 
> -Andreas
> EDIT: Again with log files.
> —
> NVIDIA Application Lab
> Jülich Supercomputing Centre
> Forschungszentrum Jülich, Germany
> +49 2461 61 1825 <tel:+49%202461%20611825>
> 
> 
> 
> 
> 
>> Am 25.11.2017 um 15:23 schrieb Sourav Chakraborty <chakraborty.52 at buckeyemail.osu.edu <mailto:chakraborty.52 at buckeyemail.osu.edu>>:
>> 
>> Hi Dr. Herten,
>> 
>> Can you please try specifying the number of processes and the hostfile? Example:
>> 
>> mpirun -np 2 -hostfile $LSB_DJOB_HOSTFILE ./get_local_rank ./osu_bibw D D
>> 
>> mpirun_rsh can also be used in place of mpirun. 
>> 
>> mpirun -export -np 2 -hostfile $LSB_DJOB_HOSTFILE ./get_local_rank ./osu_bibw D D
>> 
>> (By default mpirun_rsh does not export the environment variables and specifying -export option is required.)
>> 
>> Thank you,
>> Sourav
>> 
>> On Thu, Nov 23, 2017 at 7:45 AM, Herten, Andreas <a.herten at fz-juelich.de <mailto:a.herten at fz-juelich.de>> wrote:
>> Dear all,
>> 
>> I’m involved in operation of a POWER8+NVIDIA cluster, JURON, at Jülich Supercomputing Centre.
>> 
>> I downloaded the GDR built for POWER but am unable to run CUDA-aware MPI applications.
>> 
>> Testing with the packaged OSU microbenchmarks, the program is stuck right at the beginning. See below.
>> 
>> JURON runs CentOS with GCC 4.8.5, but we also tested loading GCC 5.4.0. We use an EDR InfiniBand interconnect. Our batch system is LSF – is MVAPICH built with LSF support?
>> Please let me know if you need further information for debugging.
>> 
>> > $ env | grep MV2
>> > MV2_ENABLE_AFFINITY=0
>> > MV2_USE_CUDA=1
>> > MV2_USE_GPUDIRECT_GDRCOPY=0
>> 
>> > $ bsub -n 2 -I -R "rusage[ngpus_shared=20]" mpirun /gpfs/software/opt/mvapich2/2.3a-GDR-cuda_8/libexec/osu-micro-benchmarks/get_local_rank /gpfs/software/opt/mvapich2/2.3a-GDR-cuda_8/libexec/osu-micro-benchmarks/mpi/pt2pt/osu_bibw D D
>> > Job <9706> is submitted to default queue <normal.i>.
>> > <<Waiting for dispatch ...>>
>> > <<Starting on juronc05>>
>> > # OSU MPI-CUDA Bi-Directional Bandwidth Test v5.4.0
>> > # Send Buffer on DEVICE (D) and Receive Buffer on DEVICE (D)
>> > # Size      Bandwidth (MB/s)
>> 
>> -Andreas
>> —
>> Dr. Andreas Herten
>> NVIDIA Application Lab
>> Jülich Supercomputing Centre
>> Forschungszentrum Jülich, Germany
>> +49 2461 61 1825 <tel:+49%202461%20611825>
>> 
>> 
>> _______________________________________________
>> mvapich-discuss mailing list
>> mvapich-discuss at cse.ohio-state.edu <mailto:mvapich-discuss at cse.ohio-state.edu>
>> http://mailman.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss <http://mailman.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss>
>> 
>> 
> 
> 
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.cse.ohio-state.edu/pipermail/mvapich-discuss/attachments/20171219/c239f267/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 5209 bytes
Desc: not available
URL: <http://mailman.cse.ohio-state.edu/pipermail/mvapich-discuss/attachments/20171219/c239f267/attachment-0001.p7s>

More information about the mvapich-discuss mailing list