[mvapich-discuss] problems with MPI + GPU

[Brian Budge]

Hi all -

Sorry for all the traffic, but I'm getting very close to being able to
reliably run my application with mvapich2.

The problem I am having now is with GPUs.   I am running an application
which uses GPUs and the CUDA programming environment to accelerate
computation.  It's exciting stuff, and depending on the problem, I see 2 to
6x speedup (I am running a ray tracing type application).  Everything works
if I run without MPI, but if I run with mvapich2, my GPU initialization
fails about 75% of the time, making my runs quite unreliable.  In the 25%
when the device initializes, everything else works fine.

Now, I'm not sure what could possibly cause this, and I could see this
problem cropping up due to any of the following factors:

1) bug in mvapich2
2) bug in CUDA
3) bug in OFED IB stuff

Does anyone have any ideas how to even begin tracking this down?  Could it
be something like infiniband device initialization walking into NVIDIA's
memory space?
  I'm grasping at straws here ;)

Thanks,
  Brian
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.cse.ohio-state.edu/pipermail/mvapich-discuss/attachments/20080108/91425665/attachment.html