[mvapich-discuss] Infiniband-less Single Node with Multiple GPUs

[sreeram potluri]

Hi Brody,

Thanks for reporting the issue. From the error it appears that the
host/device buffer detection is failing in the code.

Which version of CUDA runtime/driver are you using?

Can you run the test with this runtime parameter MV2_CUDA_USE_NAIVE=0?

Best
Sreeram Potluri

On Wed, Aug 29, 2012 at 9:54 PM, Brody Huval <brodyh at stanford.edu> wrote:

> Hi,
>
> I am trying to set up MVAPICH2 on a single node with 8 GPUs and without
> infiniband. I tried testing it out with the osu micro benchmarks but am
> currently getting an error. I configured and ran as follows:
>
>
> *brodyh at watts0:/scr/brodyh/local/libexec/osu-micro-benchmarks$* mpiname -a
> MVAPICH2 1.8 Mon Apr 30 14:56:40 EDT 2012 ch3:mrail
>
> Compilation
> CC: gcc    -DNDEBUG -DNVALGRIND -O2
> CXX: c++   -DNDEBUG -DNVALGRIND -O2
> F77: gfortran   -O2
> FC: gfortran   -O2
>
> Configuration
> --prefix=/scr/brodyh/local --enable-cuda --with-cuda=/usr/local/cuda
>
>
>
> *brodyh at watts0:/scr/brodyh/local/libexec/osu-micro-benchmarks$*mpirun_rsh -np 2 watts0 watts0 MV2_USE_CUDA=1 MV2_USE_SHARED_MEM=1
> MV2_SMP_SEND_BUF_SIZE=262144 get_local_rank ./osu_bw D D
> [watts0.Stanford.EDU:mpi_rank_1][cuda_stage_free] cudaMemcpy failed with
> 11 at 1261
> [watts0.Stanford.EDU:mpi_rank_0][cuda_stage_free] cudaMemcpy failed with
> 11 at 1261
> [watts0.Stanford.EDU:mpispawn_0][readline] Unexpected End-Of-File on file
> descriptor 5. MPI process died?
> [watts0.Stanford.EDU:mpispawn_0][mtpmi_processops] Error while reading
> PMI socket. MPI process died?
> [watts0.Stanford.EDU:mpispawn_0][child_handler] MPI process (rank: 1,
> pid: 7480) exited with status 255
> [watts0.Stanford.EDU:mpispawn_0][child_handler] MPI process (rank: 0,
> pid: 7479) exited with status 255
>
>
>
>
> Any idea what could be causing this? Thank you very much for your time.
>
>
> Best,
> Brody Huval
>
> _______________________________________________
> mvapich-discuss mailing list
> mvapich-discuss at cse.ohio-state.edu
> http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.cse.ohio-state.edu/pipermail/mvapich-discuss/attachments/20120830/f6a0559d/attachment.html