[mvapich-discuss] Segmentation fault in MPI_Allreduce with data on GPU memory

Mayuko Ishii m-ishii at dr.jp.nec.com
Wed Nov 13 23:21:22 EST 2019 

Hi Ching-Hsiang Chu, 

I sent you an email, but it seems like it didn't arrive, so I'm sending it again.

Thank you for your replay.

On adding LD_PRELOAD to the environment,
The program runs, but the result of test allreduce from CPU to GPU is incorrect.
When using mvapich2-gdr-2.3a-pgi_17.1, the result is correct.

- mvapich2-gdr-2.3.2-pgi19.1_cuda10.1
  nprocs =            2 , size =           32
  make data by Allreduce on CPUs.
rank  0:   0.0000000000000000E+00   0.1280000000000000E+03   0.0000000000000000E+00
  test allreduce on GPU buffers.
rank  1:   0.0000000000000000E+00   0.1280000000000000E+03   0.0000000000000000E+00
rank  0:   0.0000000000000000E+00   0.1280000000000000E+03   0.0000000000000000E+00
  test allreduce on GPU twice.
rank  0:   0.0000000000000000E+00   0.1280000000000000E+03   0.0000000000000000E+00
  test allreduce from GPU to CPU.
rank  1:   0.0000000000000000E+00   0.1280000000000000E+03   0.0000000000000000E+00
rank  1:   0.0000000000000000E+00   0.1280000000000000E+03   0.0000000000000000E+00
rank  0:   0.0000000000000000E+00   0.1280000000000000E+03   0.0000000000000000E+00
  test allreduce from CPU to GPU.
rank  1:   0.0000000000000000E+00   0.1280000000000000E+03   0.0000000000000000E+00
rank  0:   0.2064512000000000E+07   0.2097152000000000E+07   0.9844360351562500E+00 #incorrect
rank  1:   0.2064512000000000E+07   0.2097152000000000E+07   0.9844360351562500E+00 #incorrect


- mvapich2-gdr-2.3a-pgi_17.10
  nprocs =            2 , size =           32
  make data by Allreduce on CPUs.
rank  0:   0.0000000000000000E+00   0.1280000000000000E+03   0.0000000000000000E+00
  test allreduce on GPU buffers.
rank  1:   0.0000000000000000E+00   0.1280000000000000E+03   0.0000000000000000E+00
rank  0:   0.0000000000000000E+00   0.1280000000000000E+03   0.0000000000000000E+00
  test allreduce on GPU twice.
rank  0:   0.0000000000000000E+00   0.1280000000000000E+03   0.0000000000000000E+00
rank  1:   0.0000000000000000E+00   0.1280000000000000E+03   0.0000000000000000E+00
  test allreduce from GPU to CPU.
rank  0:   0.0000000000000000E+00   0.1280000000000000E+03   0.0000000000000000E+00
  test allreduce from CPU to GPU.
rank  1:   0.0000000000000000E+00   0.1280000000000000E+03   0.0000000000000000E+00
rank  0:   0.0000000000000000E+00   0.1280000000000000E+03   0.0000000000000000E+00
rank  1:   0.0000000000000000E+00   0.1280000000000000E+03   0.0000000000000000E+00 #correct
rank  1:   0.0000000000000000E+00   0.1280000000000000E+03   0.0000000000000000E+00 #correct

Could you investigate the place where the result is incorrect?

By the way, 
is it OK to change the position of “-lmpi” of mpif90 file as follows, without adding LD_PRELOAD ?

$ diff mpif90 mpif90.org
392c392
<         $Show $FC $PROFILE_INCPATHS ${final_fcflags} $rpath_flags -lmpi ${final_ldflags} "${allargs[@]}" $FCINCDIRS $FCMODDIRS $ITAC_OPTIONS -L$libdir -lmpifort $PROFILE_PRELIB $PROFILE_FOO $PROFILE_POSTLIB ${final_libs}
---
>         $Show $FC $PROFILE_INCPATHS ${final_fcflags} ${final_ldflags} "${allargs[@]}" $FCINCDIRS $FCMODDIRS $ITAC_OPTIONS -L$libdir -lmpifort $PROFILE_PRELIB $PROFILE_FOO $rpath_flags -lmpi  $PROFILE_POSTLIB ${final_libs}


Thanks.


Sent: Friday, November 1, 2019 1:33 AM
Subject: Re: [mvapich-discuss] Segmentation fault in MPI_Allreduce with data on GPU memory

Hi Mayuko,

It looks like the MVAPICH2-GDR runtime is not able to detect GPU memory correctly. Could you please try to add LD_PRELOAD to the environment?

E.g., 
export LD_PRELOAD=/system/apps/mvapich2-gdr/2.3.2/cuda10.1.mofed4.4.pgi19.1/lib64/libmpi.so:$LD_PRELOAD

More information can be found in the MVAPICH2-GDR userguide (http://mvapich.cse.ohio-state.edu/userguide/gdr/#_example_use_of_ld_preload). 

Please let us know if this resolves the issue.
Thanks,

Ching-Hsiang Chu

________________________________________
From: mvapich-discuss <mvapich-discuss-bounces at cse.ohio-state.edu> on behalf of Subramoni, Hari <subramoni.1 at osu.edu>
Sent: Thursday, October 31, 2019 8:01 AM
To: Mayuko Ishii <m-ishii at dr.jp.nec.com>; mvapich-discuss at cse.ohio-state.edu <mvapich-discuss at mailman.cse.ohio-state.edu>
Subject: Re: [mvapich-discuss] Segmentation fault in MPI_Allreduce with data on GPU memory 
 
Dear, Mayuko.

Sorry to hear that you're facing issues. We will take a look at this and get back to you.

Best,
Hari.

-----Original Message-----
From: mvapich-discuss-bounces at cse.ohio-state.edu <mvapich-discuss-bounces at mailman.cse.ohio-state.edu> On Behalf Of Mayuko Ishii
Sent: Thursday, October 31, 2019 12:36 AM
To: mvapich-discuss at cse.ohio-state.edu <mvapich-discuss at mailman.cse.ohio-state.edu>
Subject: [mvapich-discuss] Segmentation fault in MPI_Allreduce with data on GPU memory

Hi MVAPICH Team:

Using MVAPICH2-GDR 2.3.2 PGI (mvapich2-gdr-mcast.cuda10.1.mofed4.4.pgi19.1-2.3.2-1.el7.x86_64.rpm),
The program crash on Segmentation fault in MPI_Allreduce with data on GPU memory.

-- Error Output 

[gnode19:mpi_rank_0][error_sighandler] Caught error: Segmentation fault (signal 11)
rank  1:   0.0000000000000000E+00   0.1280000000000000E+03   0.0000000000000000E+00
[gnode22:mpi_rank_1][error_sighandler] Caught error: Segmentation fault (signal 11)
[gnode19:mpi_rank_0][print_backtrace]   0: /system/apps/mvapich2-gdr/2.3.2/cuda10.1.mofed4.4.pgi19.1/lib64/libmpi.so.12(print_backtrace+0x26) [0x2ad5e1671e66]
[gnode19:mpi_rank_0][print_backtrace]   1: /system/apps/mvapich2-gdr/2.3.2/cuda10.1.mofed4.4.pgi19.1/lib64/libmpi.so.12(error_sighandler+0x9f) [0x2ad5e167206f]
[gnode19:mpi_rank_0][print_backtrace]   2: /usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../../lib64/libpthread.so.0(+0xf5d0) [0x2ad5f4c5a5d0]
[gnode19:mpi_rank_0][print_backtrace]   3: /usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../../lib64/libc.so.6(+0x153d69) [0x2ad5f5a23d69]
[gnode19:mpi_rank_0][print_backtrace]   4: /system/apps/mvapich2-gdr/2.3.2/cuda10.1.mofed4.4.pgi19.1/lib64/libmpi.so.12(MPIR_Localcopy+0x5fa) [0x2ad5e145a11a]
[gnode19:mpi_rank_0][print_backtrace]   5: /system/apps/mvapich2-gdr/2.3.2/cuda10.1.mofed4.4.pgi19.1/lib64/libmpi.so.12(MPIR_Allreduce_two_level_MV2+0x28d) [0x2ad5e0e6d87d]
[gnode19:mpi_rank_0][print_backtrace]   6: /system/apps/mvapich2-gdr/2.3.2/cuda10.1.mofed4.4.pgi19.1/lib64/libmpi.so.12(MPIR_Allreduce_index_tuned_intra_MV2+0x12d4) [0x2ad5e0e72c94]
[gnode19:mpi_rank_0][print_backtrace]   7: /system/apps/mvapich2-gdr/2.3.2/cuda10.1.mofed4.4.pgi19.1/lib64/libmpi.so.12(MPIR_Allreduce_MV2+0xba) [0x2ad5e0e745da]
[gnode19:mpi_rank_0][print_backtrace]   8: /system/apps/mvapich2-gdr/2.3.2/cuda10.1.mofed4.4.pgi19.1/lib64/libmpi.so.12(MPIR_Allreduce_impl+0x86) [0x2ad5e0db6f56]
[gnode19:mpi_rank_0][print_backtrace]   9: /system/apps/mvapich2-gdr/2.3.2/cuda10.1.mofed4.4.pgi19.1/lib64/libmpi.so.12(PMPI_Allreduce+0x1342) [0x2ad5e0db8402]
[gnode19:mpi_rank_0][print_backtrace]  10: /system/apps/mvapich2-gdr/2.3.2/cuda10.1.mofed4.4.pgi19.1/lib64/libmpifort.so.12(mpi_allreduce_+0xb0) [0x2ad5e09f69b0] [gnode19:mpi_rank_0][print_backtrace]  11: ./a.out() [0x402cc8] [gnode19:mpi_rank_0][print_backtrace]  12: ./a.out() [0x401dc6] [gnode19:mpi_rank_0][print_backtrace]  13: /usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../../lib64/libc.so.6(__libc_start_main+0xf5) [0x2ad5f58f23d5] [gnode19:mpi_rank_0][print_backtrace]  14: ./a.out() [0x401c99]
-----

- How to reproduce

I have attached a sample program.

export CUDA_HOME=/usr/local/cuda-10.1
export CUDA_PATH=/usr/local/cuda-10.1
export MV2_PATH=/system/apps/mvapich2-gdr/2.3.2/cuda10.1.mofed4.4.pgi19.1
export PATH=/system/apps/mvapich2-gdr/2.3.2/cuda10.1.mofed4.4.pgi19.1/bin:/usr/local/cuda-10.1/bin:/system/apps/pgi/19.1/linux86-64/19.1/bin:$PATH
export LD_LIBRARY_PATH=/system/apps/mvapich2-gdr/2.3.2/cuda10.1.mofed4.4.pgi19.1/lib64:/usr/local/cuda-10.1/lib64:/system/apps/pgi/19.1/linux86-64/19.1/lib:$LD_LIBRARY_PATH
export LIBRARY_PATH=/system/apps/mvapich2-gdr/2.3.2/cuda10.1.mofed4.4.pgi19.1/lib64:/usr/local/cuda-10.1/lib64:$LIBRARY_PATH
export CPATH=/system/apps/mvapich2-gdr/2.3.2/cuda10.1.mofed4.4.pgi19.1/include:/usr/local/cuda-10.1/include:$CPATH
export MV2_USE_CUDA=1
export MV2_GPUDIRECT_GDRCOPY_LIB=/opt/gdrcopy/libgdrapi.so
export MV2_DEBUG_SHOW_BACKTRACE=1

mpif90 -Mcuda=cc70 ./mpi_reduction_test.CUF mpirun_rsh -np 2 -hostfile hostfile ./a.out 0 32

- System information
--HW (per node)
  Intel Xeon Gold 6126 x2
  NVIDIA Tesla V100 x4
  Infiniband HDR100 x4

--SW
  CentOS Linux release 7.6.1810 (3.10.0-957.5.1.el7.x86_64)
  MLNX_OFED_LINUX-4.6-1.0.1.1 (OFED-4.6-1.0.1):
  CUDA10.1


Thanks.

_______________________________________________
mvapich-discuss mailing list
mvapich-discuss at cse.ohio-state.edu
http://mailman.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss

More information about the mvapich-discuss mailing list