[mvapich-discuss] [EXTERNAL] RE: Seg-fault in MPI_Init with GCC's address sanitizer

[Subramoni, Hari]

Hello, Michael.

Thanks. Let us try out this driver.

Regards,
Hari.

From: mvapich-discuss-bounces at cse.ohio-state.edu On Behalf Of Deakin, Michael
Sent: Tuesday, December 12, 2017 11:54 AM
To: Subramoni, Hari <subramoni.1 at osu.edu>; mvapich-discuss at cse.ohio-state.edu <mvapich-discuss at mailman.cse.ohio-state.edu>
Subject: Re: [mvapich-discuss] [EXTERNAL] RE: Seg-fault in MPI_Init with GCC's address sanitizer


Yes, I should be able to try the newest MVAPICH. I'll let you know how that goes.

The code that's failing is available here: https://github.com/ESMCI/cime/tree/master/src/drivers/mct; it's a driver for climate simulation components, though the ones I need to reproduce the error are unfortunately not publicly available.



Thanks,

Michael



________________________________
From: Subramoni, Hari <subramoni.1 at osu.edu<mailto:subramoni.1 at osu.edu>>
Sent: Monday, December 11, 2017 3:57 PM
To: Deakin, Michael; mvapich-discuss at cse.ohio-state.edu<mailto:mvapich-discuss at cse.ohio-state.edu>
Cc: Subramoni, Hari
Subject: [EXTERNAL] RE: Seg-fault in MPI_Init with GCC's address sanitizer

Hello, Michael.

Sorry to hear that you are facing issue. We had fixed similar issues with the MVAPICH2 2.3 series. Would it be possible to try your program with latest MVAPICH2 2.3b release (http://mvapich.cse.ohio-state.edu/downloads/) and see if the issue persists?

If the code is available and you can point us to it, we can also try it locally to see if we are able to reproduce the issue.

Regards,
Hari.

From: mvapich-discuss-bounces at cse.ohio-state.edu<mailto:mvapich-discuss-bounces at cse.ohio-state.edu> On Behalf Of Deakin, Michael
Sent: Monday, December 11, 2017 3:40 PM
To: mvapich-discuss at cse.ohio-state.edu<mailto:mvapich-discuss at cse.ohio-state.edu> <mvapich-discuss at mailman.cse.ohio-state.edu<mailto:mvapich-discuss at mailman.cse.ohio-state.edu>>
Subject: [mvapich-discuss] Seg-fault in MPI_Init with GCC's address sanitizer


Hi,



I'm trying to debug a program (E3SM) using address sanitizer, and am getting seg-faults in mpi-init. It looks like it's trying to read from or write to a null pointer. in the log I've attached.

This issue seems to be dependent on other environment settings that I don't understand, as I'm able to run the same program with the address sanitizer and different settings without seg-faulting.

Finally, I'm reasonably certain this isn't an issue in E3SM as I'm able to run to completion with openmpi and the address sanitizer.

Would anyone here have ideas on why this is occurring? If you need more information to debug this and hopefully fix it in a future version, please let me know.


$ mpiname -a
MVAPICH2 2.2b Mon Nov 12 20:00:00 EST 2015 ch3:mrail

Compilation
CC: /blues/gpfs/home/software/spack/lib/spack/env/gcc/gcc    -DNDEBUG -DNVALGRIND -O2
CXX: /blues/gpfs/home/software/spack/lib/spack/env/gcc/g++   -DNDEBUG -DNVALGRIND -O2
F77: /blues/gpfs/home/software/spack/lib/spack/env/gcc/gfortran -L/lib -L/lib   -O2
FC: /blues/gpfs/home/software/spack/lib/spack/env/gcc/gfortran   -O2

Configuration
--prefix=/blues/gpfs/home/software/spack/opt/spack/linux-x86_64/gcc-5.3.0/mvapich2-2.2b-yzk3up5jlbhoyyfgu7tuezfbh2dhee6f --enable-shared --enable-romio --disable-silent-rules --enable-fortran=all --enable-fast=all --with-device=ch3:mrail --with-rdma=gen2



Launch command:

$ mpiexec  -l  -n 144  /lcrc/group/acme/mdeakin/acme_scratch/f_sanitize_test/bld/e3sm.exe  >> e3sm.log



Environment variables (should be the same in both runs):

MV2_ENABLE_AFFINITY=1
MV2_USE_SHARED_MEM=1
MV2_SMP_USE_CMA=1​

MV2_SHOW_CPU_BINDING=1



Thanks,

Michael Deakin
-------------- next part --------------
A non-text attachment was scrubbed...
Name: winmail.dat
Type: application/ms-tnef
Size: 18224 bytes
Desc: not available
URL: <http://mailman.cse.ohio-state.edu/pipermail/mvapich-discuss/attachments/20171212/40fb1293/attachment-0001.bin>