[mvapich-discuss] mvapich2-gdr

[Thomas Geenen]

i try to use mvapich2-gdr on our cluster.

we have it setup with
ofed 2.1
cuda 6.5
nvidia driver 340.29
nv_peer_mem loaded

with openmpi 1.8.1 we get the expected bandwidth an latency values runing the osu_bw and osu_latency with -d cuda D D

with mvapich we get the values we get when running openmpi without gpudirect
i get the sources from
https://scm.mvapich.cse.ohio-state.edu/svn/mpi/mvapich2
i configure with
./configure --prefix=/home/thomasge/build/mvapich-tag-2.0 --enable-cuda --with-cuda-inclu
de=/hpc/sw/cuda/6.5.14/include --with-cuda-libpath=/hpc/sw/cuda/6.5.14/lib64

then i run as follows
export CUDA_VISIBLE_DEVICES=0
export MV2_USE_CUDA=1
export MV2_USE_GPUDIRECT=1
export MV2_RAIL_SHARING_POLICY=FIXED_MAPPING
export MV2_PROCESS_TO_RAIL_MAPPING=mlx4_0
export MV2_CPU_BINDING_LEVEL=SOCKET
export MV2_CPU_BINDING_POLICY=SCATTER

ulimit -l unlimited
mpirun -np 2 ./osu_latency -d cuda D D > osu_latency.out 2>&1

obviously i am overlooking something

best
Thomas Geenen


| Senior adviseur Supercomputing | SURFsara | Science Park 140 | 1098 XG Amsterdam | T 06 16923022 | thomas.geenen at surfsara.nl | www.surfsara.nl |

Aanwezig op di | woe | do | vr |

SURFsara heeft een nieuw algemeen telefoonnummer: 020 800 1300

[SURF_SARA_fc]
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.cse.ohio-state.edu/pipermail/mvapich-discuss/attachments/20140827/a95e4c9c/attachment.html>