[EXTERNAL] Re: [mvapich-discuss] MVA1.9a --enable-cuda with ch3:socks compile errors

Fri Oct 26 16:57:05 EDT 2012

Hi

I found the issue with the device setting which was caused by another third library which was doing an implicid context creation. Getting rid of it didn't fix the issue with the memCopy error though. But at least now multi GPU execution with MVAPICH worked properly with the CUDA_NAIVE flag. I give the fix you attached a try on monday when I am back at work. I was not using CUDA5 though. I was using CUDA4.2 for this thing with a 304.47 driver.

Christian

________________________________
From: sreeram.chowdary at gmail.com [sreeram.chowdary at gmail.com] on behalf of sreeram potluri [potluri at cse.ohio-state.edu]
Sent: Thursday, October 25, 2012 8:20 PM
To: Trott, Christian Robert (-EXP)
Cc: mvapich-discuss at cse.ohio-state.edu
Subject: Re: [EXTERNAL] Re: [mvapich-discuss] MVA1.9a --enable-cuda with ch3:socks compile errors

Hi Christian,

[perseus.sandia.gov:mpi_rank_0][cuda_stage_free] cudaMemcpy failed with 11 at 1564

I assume you are using the CUDA 5.0 toolkit. There has been a change in behavior of device pointer detection in the driver and we had a patch to handle this in our MVAPICH2 1.8.1 release. The attached patch should fix this in the 1.9a version your using.

MPI_Init always attaches my processes to GPU 0 only. No matter whether or not I set the device to something else via cudaSetDevice or not before calling MPI_Init. Even adding a cudaThreadSynchronize before MPI_init doesn't change that. For example then running this way with two processes and telling them to get GPU 0 and 1 respectively, nvidia-smi reports three contexts: two on GPU 0 and one on GPU 1. So I assume for rank 1 I got two contexts one chosen by me and one chosen by MPI (which defaults to GPU 0). But when MPI and I choose different GPUs the code crashes. It does that even if just running 1 process (in which case no actual communication using GPU buffers takes place).

MVAPICH2 should respect your device selection if it is done before MPI_Init. I have tried the test case below and I see that it is working as expected. I have pasted the corresponding output from nvidia-smi. Is this similar to what you are trying to do? If you can share your test case with which you are seeing issues, that will be great.

-------------------------------------------------------
    mydev = 1;
    cudaSetDevice(mydev);

    MPI_Init(&c, &v);

    fprintf(stderr, "After MPI_Init \n");
    sleep(30);

    MPI_Finalize();
-------------------------------------------------------
nvidia-smi output

+-----------------------------------------------------------------------------+
| Compute processes:                                               GPU Memory |
|  GPU       PID  Process name                                     Usage      |
|=============================================================================|
|    1     28448  ./test_event_destroy                                  51MB  |
+-----------------------------------------------------------------------------+
--------------------------------------------------------

Best
Sreeram Potluri

On 10/24/2012 09:06 AM, sreeram potluri wrote:
You should be able to use MVAPICH2 within a node as long as libibverbs is
available. I am assuming you are using 1.9a for this test too.

Can you try using these options when you conifgure: --disable-rdmacm
--disable-mcast

If that does not work, can you give us more details on the issue you are
facing?

The designs for Internode GPU communication in MVAPICH2 take advantage of
features offered by InfiniBand. There are no plans to move these to
CH3-sock at this point.

Sreeram Potluri

On Wed, Oct 24, 2012 at 10:50 AM, Christian Trott<crtrott at sandia.gov<mailto:crtrott at sandia.gov>>wrote:

Thanks

thats what I thought. Do you know if I can compile for that interface on
my local workstation which does not have Infiniband? And if yes do you have
a link to a list of stuff I need to install (just adding libibverbs via yum
didn't seem to be sufficient. Also is support for the CH3:socks interface
for GPU to GPU transfer planned?
I am currently in the process of deciding whether or not I rely for direct
CUDA support within MPI for a number of projects (currently just evaluation
but potentially that would include Trilinos and LAMMPS from Sandia) instead
of writing my own data shuffling stuff. My current status is that it seems
that we got support on Infininband clusters by both MVAPICH2 and OpenMPI,
Cray seems to have something in release soon for their network, and OpenMPI
seems to work on my local machine as well.

Cheers
Christian

On 10/24/2012 08:40 AM, sreeram potluri wrote:

Hi Christian

GPU support is only available with the InfiniBand Gen2 (OFA-IB-CH3)
Interface.

Please refer to this sections of our userguide on how to build and run

http://mvapich.cse.ohio-state.**edu/support/user_guide_**
mvapich2-1.9a.html#x1-140004.5<http://mvapich.cse.ohio-state.edu/support/user_guide_mvapich2-1.9a.html#x1-140004.5>
http://mvapich.cse.ohio-state.**edu/support/user_guide_**
mvapich2-1.9a.html#x1-780006.**18<http://mvapich.cse.ohio-state.edu/support/user_guide_mvapich2-1.9a.html#x1-780006.18>

Best
Sreeram

On Wed, Oct 24, 2012 at 10:19 AM, Christian Trott<crtrott at sandia.gov<mailto:crtrott at sandia.gov>>**
wrote:

  Hi all
is it possible to use the GPU support with the CH3:socks interface? When
I
try to compile the 1.9a release with
./configure --enable-cuda --with-cuda=/opt/nvidia/cuda/****5.0.36/
--with-device=ch3:sock --prefix=/opt/mpi/mvapich2-1.***
*9/intel-12.1/cuda5036
CC=/opt/intel/composer_xe_****2011_sp1.9.293/bin/intel64/icc

I run into these errors:

CC              ch3_isend.c
ch3_isend.c(20): error: a value of type "MPIDI_CH3_PktGeneric_t" cannot
be
assigned to an entity of type "void *"
        sreq->dev.pending_pkt = *(MPIDI_CH3_PktGeneric_t *) hdr;

    CC              ch3_isendv.c
ch3_isendv.c(28): error: a value of type "MPIDI_CH3_PktGeneric_t" cannot
be assigned to an entity of type "void *"
        sreq->dev.pending_pkt = *(MPIDI_CH3_PktGeneric_t *)
iov[0].MPID_IOV_BUF;

Thanks for your help
Christian

______________________________****_________________
mvapich-discuss mailing list
mvapich-discuss at cse.ohio-**sta**te.edu<http://te.edu><http://state.edu><
mvapich-discuss at cse.**ohio-state.edu<http://ohio-state.edu><mvapich-discuss at cse.ohio-state.edu<mailto:mvapich-discuss at cse.ohio-state.edu>>
http://mail.cse.ohio-state.****edu/mailman/listinfo/mvapich-****discuss<
http://mail.cse.ohio-**state.edu/mailman/listinfo/**mvapich-discuss<http://state.edu/mailman/listinfo/**mvapich-discuss><http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.cse.ohio-state.edu/pipermail/mvapich-discuss/attachments/20121026/aa50a6f4/attachment-0001.html