[mvapich-discuss] mvapich2-gdr

[Thomas Geenen]

hi DK,

yes thats much better.
seems that i will have to do some tweaking to get similar results compared to openmpi for medium sized messages.
that will be something for tomorrow


cheers
Thomas Geenen


| Senior adviseur Supercomputing | SURFsara | Science Park 140 | 1098 XG Amsterdam | T 06 16923022 | thomas.geenen at surfsara.nl | www.surfsara.nl |

Aanwezig op di | woe | do | vr |

[SURF_SARA_fc]
________________________________
From: Panda, Dhabaleswar [panda at cse.ohio-state.edu]
Sent: Wednesday, August 27, 2014 5:52 PM
To: Thomas Geenen; mvapich-discuss at cse.ohio-state.edu
Cc: Panda, Dhabaleswar
Subject: RE: mvapich2-gdr

Hi Thomas,

MVAPICH2-GDR (with GPU Direct support) is a different package than the standard MVAPICH2 release.
The standard MVAPICH2 release does not have GPU Direct RDMA support.

Please download the latest MVAPICH2-GDR 2.0 package from MVAPICH->Download
page and use it according to the README file available inside the package (or from the
following URL) to use the GPU Direct RDMA functionality:

http://mvapich.cse.ohio-state.edu/static/media/mvapich/MV2-GDR-README.txt

Hope this helps.

Thanks,

DK

________________________________
From: mvapich-discuss-bounces at cse.ohio-state.edu on behalf of Thomas Geenen [thomas.geenen at surfsara.nl]
Sent: Wednesday, August 27, 2014 11:39 AM
To: mvapich-discuss at cse.ohio-state.edu
Subject: [mvapich-discuss] mvapich2-gdr

i try to use mvapich2-gdr on our cluster.

we have it setup with
ofed 2.1
cuda 6.5
nvidia driver 340.29
nv_peer_mem loaded

with openmpi 1.8.1 we get the expected bandwidth an latency values runing the osu_bw and osu_latency with -d cuda D D

with mvapich we get the values we get when running openmpi without gpudirect
i get the sources from
https://scm.mvapich.cse.ohio-state.edu/svn/mpi/mvapich2
i configure with
./configure --prefix=/home/thomasge/build/mvapich-tag-2.0 --enable-cuda --with-cuda-inclu
de=/hpc/sw/cuda/6.5.14/include --with-cuda-libpath=/hpc/sw/cuda/6.5.14/lib64

then i run as follows
export CUDA_VISIBLE_DEVICES=0
export MV2_USE_CUDA=1
export MV2_USE_GPUDIRECT=1
export MV2_RAIL_SHARING_POLICY=FIXED_MAPPING
export MV2_PROCESS_TO_RAIL_MAPPING=mlx4_0
export MV2_CPU_BINDING_LEVEL=SOCKET
export MV2_CPU_BINDING_POLICY=SCATTER

ulimit -l unlimited
mpirun -np 2 ./osu_latency -d cuda D D > osu_latency.out 2>&1

obviously i am overlooking something

best
Thomas Geenen


| Senior adviseur Supercomputing | SURFsara | Science Park 140 | 1098 XG Amsterdam | T 06 16923022 | thomas.geenen at surfsara.nl | www.surfsara.nl |

Aanwezig op di | woe | do | vr |

SURFsara heeft een nieuw algemeen telefoonnummer: 020 800 1300

[SURF_SARA_fc]
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.cse.ohio-state.edu/pipermail/mvapich-discuss/attachments/20140827/1aa245ef/attachment.html>