[mvapich-discuss] Segfault when building HDF5 with MVAPICH2 2.1rc1 on SLES11 SP3
Jonathan Perkins
perkinjo at cse.ohio-state.edu
Tue Feb 10 16:17:15 EST 2015
That's a great bit of debugging there. Thanks for sharing this. As far
as I can tell this seems related to an issue the Mozilla folks
encountered and created a fix for a few years back.
I don't see a quick work around without sacrificing performance right
now so I hope that you're able to get things running without needing to
set this variable permanently. Please keep us posted.
On Tue, Feb 10, 2015 at 02:35:09PM -0500, Thompson, Matt (GSFC-610.1)[SCIENCE SYSTEMS AND APPLICATIONS INC] wrote:
> All right, here we go. A Linux guru far better than I (Aaron Knister of
> NCCS) investigated this and found:
>
> >I've done a bunch more digging and here's what's happening:
> >
> >1. symbol relocation processing occurs for libc
> >2. references to malloc() within libc get bound to mvapich2's malloc
> >from libmpi
> >3. ...stuff happens...
> >4. getpwuid is called, which is bound to getpwuid in glibc
> >5. getpwuid does a dlopen of libnss_sss
> >6. getpwuid calls _nss_sss_getpwuid_r from libnss_sss
> >7. Deep within _nss_sss_getpwuid_r asprintf is called
> >8. asprintf is bound to libc and internally calls malloc, which from
> >step 2 is bound to mvapich2's malloc
> >
> >note that what we have here is a pointer to memory that has been
> >allocated by mvapich2's malloc implementation
> >
> >9. libnss_sss then calls free() to free the memory allocated by
> >mvapich2's libmpi
> >10. the dynamic linker then binds free() to glibc
> >11. segfault occurs because we tried to free() memory using a
> >different memory management library than the one that was used to
> >allocate it
> >
> >In step 10, the library search order used by the dynamic linker/loader
> >is different then in all previous symbol relocation processing.
>
> Then he discovered:
>
> >Ah! I just found this:
> >
> >If you set the environment variable RTLD_DEEPBIND to 0 then the
> >segfault disappears. I'm trying to better understand this, but works
> >by altering the behaviour of glibc when loading NSS modules. The
> >symbol resolution order seems to change with it set (for the better
> >in this case).
>
> And his final word:
>
> >I think I've figured out who is in the "wrong here". This is a result of
> >a patch SuSE applied to glibc to dlopen NSS libraries with the
> >RTLD_DEEPBIND flag which puts the symbols inside the library ahead of
> >the global scope. This results in the behaviour we saw earlier where the
> >free() called by the dlopen'd libnss_sss was resolved to the glibc lib
> >required by libnss_sss instead of the global scope which resolved and
> >bound free() to mvapich2's libmpi.
> >
> >The environment variable I found works by disabling this behaviour and
> >making it the glibc default, where symbols in dlopen()'d libraries
> >aren't put ahead of the global scope.
> >
> >SuSE has a legitimate reason for their patch, but the solution to this
> >issue is to set the RTLD_DEEPBIND environment variable to "0" to resolve
> >the segfaults.
>
> So, apparently there is a way around the build fail.
>
> I suppose the next question is whether this is just a build setting or
> something that would need to be permanently enabled.
>
> Matt
>
> On 02/06/2015 09:50 AM, Jonathan Perkins wrote:
> >Hi Matt. I'm not very sure what is going on but as a first step can you
> >share the output of `mpiname -a' from your MVAPICH2 build? It may also
> >to add the -show option to the Makefile command that builds the
> >executable that is failing with the segmentation fault and send that
> >output as well.
> >
> >On Fri, Feb 06, 2015 at 08:53:04AM -0500, Thompson, Matt (GSFC-610.1)[SCIENCE SYSTEMS AND APPLICATIONS INC] wrote:
> >>MVAPICH Discuss,
> >>
> >>I have an issue that is weirdly specific. When I try to build either
> >>HDF5-1.8.12 or the latest stable HDF5-1.8.14 (with --enable-parallel) with
> >>MVAPICH2 2.1rc1 on SLES 11 SP3, the HDF5 build fails with a segfault:
> >>
> >>>libtool: link: mpicc -std=c99 -O3 -fPIC -o H5make_libsettings H5make_libsettings.o -L/discover/swdev/USER/Baselibs/TmpBaselibs/GMAO-Baselibs-4_0_6-FixHDF5/x86_64-unknown-linux-gnu/ifort/Linux/lib /discover/swdev/USER/Baselibs/TmpBaselibs/GMAO-Baselibs-4_0_6-FixHDF5/x86_64-unknown-linux-gnu/ifort/Linux/lib/libsz.a -lz -ldl -lm
> >>>LD_LIBRARY_PATH="$LD_LIBRARY_PATH`echo -lm | \
> >>> sed -e 's/-L/:/g' -e 's/ //g'`" \
> >>> ./H5make_libsettings > H5lib_settings.c || \
> >>> (test $HDF5_Make_Ignore && echo "*** Error ignored") || \
> >>> (rm -f H5lib_settings.c ; exit 1)
> >>>/bin/sh: line 4: 1838 Segmentation fault (core dumped) LD_LIBRARY_PATH="$LD_LIBRARY_PATH`echo -lm | sed -e 's/-L/:/g' -e 's/ //g'`" ./H5make_libsettings > H5lib_settings.c
> >>
> >>I can be fairly certain with that specificity because I've tried the
> >>following things (all with Intel 15.0.0.090):
> >>
> >> MVAPICH2 2.1rc1 on SLES 11 SP1: Works
> >> MVAPICH2 2.1rc1 on SLES 11 SP3: FAIL
> >> Intel MPI 5.0.1.135 on SLES 11 SP1: Works
> >> Intel MPI 5.0.1.135 on SLES 11 SP3: Works
> >> MPT 2.11 on SLES 11 SP3: Works
> >>
> >>I've also tried without --enable-parallel:
> >>
> >> No Parallel HDF5 on SLES 11 SP3: Works
> >>
> >>though in that case, the C compiler would be gcc not icc (since it's not
> >>calling mpicc which points to icc).
> >>
> >>Other than that, everything else is the same in each environment.
> >>
> >>I also tried compiling with -O0 -g -traceback and got the same failure.
> >>Looking at the core in gdb:
> >>
> >>>(gdb) backtrace
> >>>#0 0x00002aaaabfe0802 in _int_free () from /lib64/libc.so.6
> >>>#1 0x00002aaaabfe3b5c in free () from /lib64/libc.so.6
> >>>#2 0x00002aaaaf70c35d in ?? () from /lib64/libnss_sss.so.2
> >>>#3 0x00002aaaaf70c6f0 in ?? () from /lib64/libnss_sss.so.2
> >>>#4 0x00002aaaaf70a275 in _nss_sss_getpwuid_r () from /lib64/libnss_sss.so.2
> >>>#5 0x00002aaaac00fb2c in getpwuid_r@@GLIBC_2.2.5 () from /lib64/libc.so.6
> >>>#6 0x00002aaaac00f37f in getpwuid () from /lib64/libc.so.6
> >>>#7 0x0000000000401993 in print_header () at H5make_libsettings.c:185
> >>>#8 0x0000000000401d3a in main () at H5make_libsettings.c:290
> >>
> >> From this testing it seems like it isn't the compiler, it isn't *just* the
> >>operating system, and it isn't *just* the MPI stack, but rather the
> >>combination of MVAPICH2 2.1rc1 and SLES 11 SP3. This has cropped up because
> >>part of the supercomputer I work on has transitioned to SLES 11 SP3. And in
> >>attempting to rebuild some libraries to diagnose some issues, this came up.
> >>
> >>Now I have asked better computer engineers than I here to try to figure this
> >>out as well, but I was wondering if anyone here might know why one would
> >>fail while others succeed? That is, if you've seen something similar?
> >>
> >>Matt
> >>--
> >>Matt Thompson SSAI, Sr Software Test Engr
> >>NASA GSFC, Global Modeling and Assimilation Office
> >>Code 610.1, 8800 Greenbelt Rd, Greenbelt, MD 20771
> >>Phone: 301-614-6712 Fax: 301-614-6246
> >>_______________________________________________
> >>mvapich-discuss mailing list
> >>mvapich-discuss at cse.ohio-state.edu
> >>http://mailman.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss
> >
>
>
> --
> Matt Thompson SSAI, Sr Software Test Engr
> NASA GSFC, Global Modeling and Assimilation Office
> Code 610.1, 8800 Greenbelt Rd, Greenbelt, MD 20771
> Phone: 301-614-6712 Fax: 301-614-6246
--
Jonathan Perkins
More information about the mvapich-discuss
mailing list