[mvapich-discuss] Re: [openfabrics-ewg] Announcing the release of MVAPICH2 0.9.8 with Checkpoint/Restart, iWARP, RDMA CM-based connection manageme

[Shaun Rowland]

david elsen wrote:
> Shaun,
> 
> Please see below:
> 
> [root at ammasso1 ~]# strings /etc/ld.so.cache | grep librdmacm librdmacm.so
> grep: librdmacm.so: No such file or directory
> [root at ammasso1 ~]# strings /etc/ld.so.cache | grep librdmacm
> [root at ammasso1 ~]# strings /etc/ld.so.cache | grep librdmacm.so
> [root at ammasso1 ~]#

OK. Just to be clear, the library does not have to be in the ld.so.cache
for it to work. As long as ld.so can find the library, it should be
fine. The library could be anywhere, and usually you'd set
LD_LIBRARY_PATH if it were in some strange location. Again, I can easily
demonstrate that by moving libraries around or making my own shared library.

If you want to try building a test shared library and see if your system
can find it, you could grab this test library code I wrote a long time
ago here:

http://www.cse.ohio-state.edu/~rowland/libtest.tar.gz

This should only depend on $PWD being set in your environment. It should
be. If it is not, you can substitute the path of where you untar that
file on the command line examples below and in the Makefile. This will
at least show you that it is possible to link to a shared library that
is not in a standard location, not in the ld.so.cache or in the default
search path configuration, nor has been prelinked. Here are two ways to
use the above test code if you are interested in trying it on this
particular machine:

Simple Method
-------------

This creates a simple shared library with no specified soname. The
soname is the DT_SONAME field of the shared library file. As I was
mentioning earlier, a shared library usually has an soname of something
like "libname.so.1" and the following files exist where it is installed
(--> is the symlinking)

libname.so   --> libname.so.1        [or --> libname.so.1.0 directly]
libname.so.1 --> libname.so.1.0
libname.so.1.0

This way, when you build against the library, your binary will be "told"
to look for the soname version of the library file. If you do "ldd", it
would show "libname.so.1" normally (the soname) and not the libname.so
or the actual specific library file itself. This allows you to do things
like a minor library upgrade. This is built in my second example. In
this example, I do not set soname and it just uses the library's simple
name. This seems similar to your librdmacm.so situation perhaps:

[rowland at k30-oib ~]$ tar -vxzf libtest.tar.gz
libtest/
libtest/test.h
libtest/test.c
libtest/test-program.c
libtest/Makefile

[rowland at k30-oib ~]$ cd libtest

[rowland at k30-oib libtest]$ ls
Makefile  test.c  test.h  test-program.c

[rowland at k30-oib libtest]$ pwd
/home/7/rowland/libtest

[rowland at k30-oib libtest]$ make simple
gcc -c -fPIC test.c
gcc -shared -o libtest.so test.o
gcc    -c -o test-program.o test-program.c
gcc -o test-program test-program.o -L/home/7/rowland/libtest -ltest

[rowland at k30-oib libtest]$ objdump -x test-program |grep NEEDED
   NEEDED      libtest.so
   NEEDED      libc.so.6

[rowland at k30-oib libtest]$ ldd test-program
         libtest.so => not found
         libc.so.6 => /lib/tls/libc.so.6 (0x00292000)
         /lib/ld-linux.so.2 (0x00279000)

[rowland at k30-oib libtest]$ export LD_LIBRARY_PATH=$PWD

[rowland at k30-oib libtest]$ ldd test-program
         libtest.so => /home/7/rowland/libtest/libtest.so (0x009e3000)
         libc.so.6 => /lib/tls/libc.so.6 (0x00292000)
         /lib/ld-linux.so.2 (0x00279000)

[rowland at k30-oib libtest]$ ./test-program
In shared library function...

You should be able to do the same thing and have the library found once
LD_LIBRARY_PATH is set.


Normal Method
-------------

Here I continue from where I left off above and create a more "standard"
shared library with soname set, etc.

[rowland at k30-oib libtest]$ unset LD_LIBRARY_PATH

[rowland at k30-oib libtest]$ make clean
rm -f libtest.so* test-program *.o *~

[rowland at k30-oib libtest]$ ls
Makefile  test.c  test.h  test-program.c

[rowland at k30-oib libtest]$ make normal
gcc -c -fPIC test.c
gcc -shared -Wl,-soname,libtest.so.1 -o libtest.so.1.0 test.o
ln -s libtest.so.1.0 libtest.so.1
ln -s libtest.so.1 libtest.so
gcc    -c -o test-program.o test-program.c
gcc -o test-program test-program.o -L/home/7/rowland/libtest -ltest

[rowland at k30-oib libtest]$ objdump -x test-program |grep NEEDED
   NEEDED      libtest.so.1
   NEEDED      libc.so.6

[rowland at k30-oib libtest]$ ldd test-program
         libtest.so.1 => not found
         libc.so.6 => /lib/tls/libc.so.6 (0x00292000)
         /lib/ld-linux.so.2 (0x00279000)

[rowland at k30-oib libtest]$ export LD_LIBRARY_PATH=$PWD

[rowland at k30-oib libtest]$ ldd test-program
         libtest.so.1 => /home/7/rowland/libtest/libtest.so.1 (0x00af1000)
         libc.so.6 => /lib/tls/libc.so.6 (0x00292000)
         /lib/ld-linux.so.2 (0x00279000)

[rowland at k30-oib libtest]$ ./test-program
In shared library function...

Again, this should work for you. It would be interesting if it did not
work for some reason. In any case, this is about all I know about shared
libraries at the moment...
-- 
Shaun Rowland	rowland at cse.ohio-state.edu
http://www.cse.ohio-state.edu/~rowland/