[mvapich-discuss] Lot of errors when running with cuda-memcheck

[Akshay Venkatesh]

Hi Nuno,

We looked at this and it's unexpected. Can we get a reproducer for this?
The output of mpiname -a and the set of runtime parameters used for the
reproducer will also be helpful.

Thanks.
On Apr 22, 2015 9:33 PM, "Nuno Cardoso" <nmrcardoso at gmail.com> wrote:

> Hi,
>
> I'm getting a lot of errors:
> ========= Program hit CUDA_ERROR_INVALID_VALUE (error 1) due to "invalid
> argument" on CUDA API call to cuPointerGetAttribute.
> =========     Saved host backtrace up to driver entry point at error
> =========     Host Frame:/usr/lib64/nvidia/libcuda.so.1
> (cuPointerGetAttribute + 0x16c) [0x148a0c]
> =========     Host Frame:/home/ncardoso/.mpi/lib/libmpi.so.12
> (is_device_buffer + 0x6f) [0x38c70f]
> =========     Host Frame:/home/ncardoso/.mpi/lib/libmpi.so.12
> (MPIR_Localcopy + 0x8b8) [0x2d9c58]
> =========     Host Frame:/home/ncardoso/.mpi/lib/libmpi.so.12
> (MPIR_Allreduce_reduce_shmem_MV2 + 0x35c) [0xcb4ac]
> =========     Host Frame:/home/ncardoso/.mpi/lib/libmpi.so.12
> (MPIR_Allreduce_two_level_MV2 + 0x2d8) [0xce808]
> =========     Host Frame:/home/ncardoso/.mpi/lib/libmpi.so.12
> (MPIR_Allreduce_index_tuned_intra_MV2 + 0x916) [0xcf316]
> =========     Host Frame:/home/ncardoso/.mpi/lib/libmpi.so.12
> (MPIR_Allreduce_impl + 0x24) [0x6c704]
> =========     Host Frame:/home/ncardoso/.mpi/lib/libmpi.so.12
> (PMPI_Allreduce + 0x54e) [0x6ccfe]
>
> This happens if the env MV2_USE_CUDA is active and without using device
> pointers in MPI functions.
> This is bad in the sense that I cannot use cuda-memcheck if a make my
> program  mpi cuda-aware.
>
> There is a solution to avoid this?
>
>
> Best regards,
> Nuno Cardoso
>
>
> _______________________________________________
> mvapich-discuss mailing list
> mvapich-discuss at cse.ohio-state.edu
> http://mailman.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.cse.ohio-state.edu/pipermail/mvapich-discuss/attachments/20150422/c8329a80/attachment.html>