<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof">
Hi Ryan, <br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof">
This is interesting. You are correct that we no longer link directly to the ib libraries, this is to allow MVAPICH2 to run on SMP only machines without needing to install the ib libraries. Instead, we try to dynamically open them as needed from within the library
at runtime. The environment variable provided in the error message is there only as a fallback and should only be necessary if the library is not available on the standard
<code>LD_LIBRARY_PATH</code>. However, it looks like yours should be available on there from
<code>/usr/lib64</code>; is there any chance that the library is on a different path on the compute nodes than it is on the head node? I will try to reproduce this and see if I can figure out why it would be failing to open your libibmad.so.
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof">
To go back to using the older linking process, please try adding <code>--disable-ibv-dlopen</code> to your configure line. Can you try that let us know if it works for you?
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof">
Thanks, <br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof">
Nat<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof">
<br>
</div>
<div id="appendonsend"></div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> Mvapich-discuss <mvapich-discuss-bounces+shineman.5=osu.edu@lists.osu.edu> on behalf of Ryan Novosielski via Mvapich-discuss <mvapich-discuss@lists.osu.edu><br>
<b>Sent:</b> Monday, July 11, 2022 11:36<br>
<b>To:</b> mvapich-discuss@lists.osu.edu <mvapich-discuss@lists.osu.edu><br>
<b>Subject:</b> [Mvapich-discuss] MVAPICH2 2.3.7-1 (and 2.3.6) "mv2_mad_dlopen_init" re: "Error opening libibmad.so: libibmad.so", GCC 10.4, CentOS 7.x</font>
<div> </div>
</div>
<div class="BodyFragment"><font size="2"><span style="font-size:11pt;">
<div class="PlainText">!-------------------------------------------------------------------|<br>
This Message Is From an External Sender<br>
This message came from outside your organization.<br>
|-------------------------------------------------------------------!<br>
<br>
Hi there,<br>
<br>
I'm getting error messages when running an MPI job with SLURM (18.08)<br>
using MVAPICH2, I assume, post 2.3.5, when the following change was made:<br>
<br>
NEW Remove dependency on underlying libibverbs, libibmad, libibumad, and<br>
librdmacm libraries using dlopen<br>
<br>
Here's what I'm seeing:<br>
<br>
[novosirj@amarel-test2 mpihello]$ srun --mpi=pmi2 -n 4<br>
./mpi_hello_world.gcc-10.4.mvapich2-2.3.7-1<br>
srun: job 20824691 queued and waiting for resources<br>
srun: job 20824691 has been allocated resources<br>
Error opening libibmad.so: libibmad.so: cannot open shared object file:<br>
No such file or directory.<br>
mv2_mad_dlopen_init returned -1<br>
Error opening libibmad.so: libibmad.so: cannot open shared object file:<br>
No such file or directory.<br>
mv2_mad_dlopen_init returned -1<br>
Error opening libibmad.so: libibmad.so: cannot open shared object file:<br>
No such file or directory.<br>
mv2_mad_dlopen_init returned -1<br>
Error opening libibmad.so: libibmad.so: cannot open shared object file:<br>
No such file or directory.<br>
mv2_mad_dlopen_init returned -1<br>
Hello world from processor slepner021.amarel.rutgers.edu, rank 1 out of<br>
4 processors<br>
Hello world from processor slepner021.amarel.rutgers.edu, rank 2 out of<br>
4 processors<br>
Hello world from processor slepner021.amarel.rutgers.edu, rank 3 out of<br>
4 processors<br>
Hello world from processor slepner009.amarel.rutgers.edu, rank 0 out of<br>
4 processors<br>
<br>
I don't see this on 2.3. MPI seems to be working, but I assume it's not<br>
using Infiniband?<br>
<br>
The libraries do exist:<br>
<br>
[novosirj@amarel-test2 mpihello]$ rpm -ql infiniband-diags | grep mad<br>
/usr/lib64/libibmad.so.5<br>
/usr/lib64/libibmad.so.5.5.0<br>
<br>
And while I assume it's normal to not see libibmad/libibumad in ldd -v<br>
output anymore post 2.3.5 (and I don't), here's what I see on 2.3, just<br>
to give you an idea of how it used to work:<br>
<br>
[novosirj@amarel-test2 mpihello]$ ldd -v<br>
mpi_hello_world.gcc-10.4.mvapich2-2.3.7-1 | head -50<br>
<br>
linux-vdso.so.1 => (0x00007fff07b0c000)<br>
<br>
libmpi.so.12 =><br>
/opt/sw/packages/gcc-4_8/mvapich2/2.3/lib/libmpi.so.12 (0x00007f36e87d9000)<br>
libc.so.6 => /lib64/libc.so.6 (0x00007f36e840b000)<br>
libgfortran.so.3 => /lib64/libgfortran.so.3 (0x00007f36e80e9000)<br>
libm.so.6 => /lib64/libm.so.6 (0x00007f36e7de7000)<br>
libnuma.so.1 => /lib64/libnuma.so.1 (0x00007f36e7bdb000)<br>
libxml2.so.2 => /lib64/libxml2.so.2 (0x00007f36e7871000)<br>
libibmad.so.5 => /lib64/libibmad.so.5 (0x00007f36e7656000)<br>
librdmacm.so.1 => /lib64/librdmacm.so.1 (0x00007f36e743f000)<br>
libibumad.so.3 => /lib64/libibumad.so.3 (0x00007f36e7236000)<br>
libibverbs.so.1 => /lib64/libibverbs.so.1 (0x00007f36e701d000)<br>
libdl.so.2 => /lib64/libdl.so.2 (0x00007f36e6e19000)<br>
librt.so.1 => /lib64/librt.so.1 (0x00007f36e6c11000)<br>
libpmi2.so.0 => /lib64/libpmi2.so.0 (0x00007f36e69f9000)<br>
libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f36e67dd000)<br>
libgcc_s.so.1 => /opt/sw/packages/gcc/10.4/lib64/libgcc_s.so.1<br>
(0x00007f36e65c5000)<br>
libquadmath.so.0 =><br>
/opt/sw/packages/gcc/10.4/lib64/libquadmath.so.0 (0x00007f36e637e000)<br>
/lib64/ld-linux-x86-64.so.2 (0x00007f36e8f40000)<br>
libz.so.1 => /lib64/libz.so.1 (0x00007f36e6168000)<br>
liblzma.so.5 => /lib64/liblzma.so.5 (0x00007f36e5f42000)<br>
libosmcomp.so.4 => /lib64/libosmcomp.so.4 (0x00007f36e5d33000)<br>
libnl-route-3.so.200 => /lib64/libnl-route-3.so.200<br>
(0x00007f36e5ac6000)<br>
libnl-3.so.200 => /lib64/libnl-3.so.200 (0x00007f36e58a5000)<br>
<br>
What can/should I do about this?<br>
Sometimes I see (not clear what conditions trigger it, but I have at <br>
least one set of output running one of the OSU benchmarks):<br>
<br>
Please retry with MV2_LIBIBMAD_PATH=<path/to/libibmad.so><br>
<br>
It seems like what's suggested in the error message is not a great <br>
idea/this should be dealt with at compile time.<br>
<br>
This is my build script; relatively uncomplicated:<br>
<br>
[novosirj@amarel-test2 build]$ more <br>
~/src/build-mvapich2-2.3.7-1-gcc-10.4.sh #!/bin/sh<br>
<br>
module purge<br>
module load gcc/10.4<br>
module list<br>
<br>
export FFLAGS="-fallow-argument-mismatch"<br>
<br>
../mvapich2-2.3.7-1/configure --with-pmi=pmi2 --with-pm=slurm <br>
--prefix=/opt/sw/packages/gcc-10/mvapich2/2.3.7-1 && \<br>
make -j32 && make check && make install<br>
<br>
And the configure process doesn't seem to point out anything amiss:<br>
<br>
checking for the InfiniBand includes path... default<br>
checking for the InfiniBand library path... default<br>
checking for library containing shm_open... -lrt<br>
checking infiniband/verbs.h usability... yes<br>
checking infiniband/verbs.h presence... yes<br>
checking for infiniband/verbs.h... yes<br>
configure: checking checking for InfiniBand umad installation...<br>
checking infiniband/umad.h usability... yes<br>
checking infiniband/umad.h presence... yes<br>
checking for infiniband/umad.h... yes<br>
configure: InfiniBand libumad found<br>
checking whether to enable hybrid communication channel... yes<br>
configure: checking for RDMA CM support...<br>
checking rdma/rdma_cma.h usability... yes<br>
checking rdma/rdma_cma.h presence... yes<br>
checking for rdma/rdma_cma.h... yes<br>
configure: RDMA CM support enabled<br>
configure: checking for hardware multicast support...<br>
checking infiniband/mad.h usability... yes<br>
checking infiniband/mad.h presence... yes<br>
checking for infiniband/mad.h... yes<br>
<br>
Thanks!<br>
<br>
-- <br>
#BlackLivesMatter<br>
____<br>
|| \\UTGERS, |----------------------*O*------------------------<br>
||_// the State | Ryan Novosielski - novosirj@rutgers.edu<br>
|| \\ University | Sr. Technologist - 973/972.0922 ~*~ RBHS Campus<br>
|| \\ of NJ | Office of Advanced Res. Comp. - MSB C630, Newark<br>
_______________________________________________<br>
Mvapich-discuss mailing list<br>
Mvapich-discuss@lists.osu.edu<br>
<a href="https://lists.osu.edu/mailman/listinfo/mvapich-discuss">https://lists.osu.edu/mailman/listinfo/mvapich-discuss</a><br>
</div>
</span></font></div>
</body>
</html>