From kumarmohil07 at gmail.com  Tue Oct  1 08:51:29 2024
From: kumarmohil07 at gmail.com (Mohil Kumar)
Date: Tue, 1 Oct 2024 18:21:29 +0530
Subject: [Mvapich-discuss] Regarding how topology awareness works in mvapich
Message-ID: <CAMVkEeWpjyOR9uD_xusW5JsO53R1s9U6YAp18nUxvzQHEQiwAQ@mail.gmail.com>

Hello,
Myself Mohil, Currently I have been studying how topology awareness in
mvapich works. I get to know about *physical* topology and *virtual*
topology and how the virtual topology is mapped to physical topology(like
various *mapping* *algorithms*).
But before mapping virtual topology on physical topology we have to detect
the physical topology, which involves detecting both hardware(intranode)
and network topology(internode). Hardware topology is detected using *hwloc*
library.

But I am confused how the network topology detection works. Will it be done
by a job scheduler like *SLURM* or *mvapich* to get this info from the
underlying network interface somehow.

If anyone is able to help me I will be very thankful to you. Also, if I
make any mistakes in my interpretation of the whole thing, I am open to any
suggestions.

With regards
MOHIL
kumarmohil07 at gmail.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osu.edu/pipermail/mvapich-discuss/attachments/20241001/e176f93b/attachment.html>

From e.schnetzer at ll.mit.edu  Wed Oct  2 16:41:14 2024
From: e.schnetzer at ll.mit.edu (Schnetzer, Everett - 1002 - MITLL)
Date: Wed, 2 Oct 2024 20:41:14 +0000
Subject: [Mvapich-discuss] MPI_ANY_SOURCE not receiving message on MVAPICH
	3.0
Message-ID: <BN1P110MB07073EFBDF05D2C8B2C78C90DF70A@BN1P110MB0707.NAMP110.PROD.OUTLOOK.COM>

?Hello, 

We had an application break when moving up to v3.0, and the error was a failure to receive a message when the receiver sets the source to MPI_ANY_SOURCE. Below is code for a simple test case, which is run as a single process. On MVAPICH2 v2.3.7, the code completes normally, and on v3.0 the receiver thread never receives the second message and the main program hangs on the join. Our build is on Red Hat Enterprise Linux 9.4, using gcc-13.2.0. 

Sincerely, 
Everett Schnetzer 
MIT Lincoln Laboratory 

Test code: 

#include <iostream> 
#include <stdlib.h> 
#include <mpi.h> 
#include <assert.h> 
#include <thread> 
#include <unistd.h> 

void receiverTest() 
{ 
std::cout << "thread calling receive from rank 0" << std::endl; 
MPI_Status status; 
uint64_t parameters[2]; 
int retval = MPI_Recv(parameters, 2, MPI_UINT64_T, 0, 0, MPI_COMM_WORLD, &status); 
assert(retval == MPI_SUCCESS); 

std::cout << "thread calling receive from any rank" << std::endl; 
retval = MPI_Recv(parameters, 2, MPI_UINT64_T, MPI_ANY_SOURCE, 0, MPI_COMM_WORLD, &status); 
assert(retval == MPI_SUCCESS); 

std::cout << "receive done" << std::endl; 
} 

int main(int argc, char** argv) 
{ 
int provided; 
MPI_Init_thread(nullptr, nullptr, MPI_THREAD_MULTIPLE, &provided); 
assert(provided == MPI_THREAD_MULTIPLE); 

std::thread rcvThread = std::thread(&receiverTest); 

sleep(1); 

std::cout << "first send to thread" << std::endl; 
uint64_t parameters[2]; 
int retval = MPI_Send(parameters, 2, MPI_UINT64_T, 0, 0, MPI_COMM_WORLD); 
assert(retval == MPI_SUCCESS); 

sleep(1); 

std::cout << "second send to thread" << std::endl; 
retval = MPI_Send(parameters, 2, MPI_UINT64_T, 0, 0, MPI_COMM_WORLD); 
assert(retval == MPI_SUCCESS); 

assert(rcvThread.joinable()); 
rcvThread.join(); 

MPI_Finalize(); 

return 0; 
} 


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osu.edu/pipermail/mvapich-discuss/attachments/20241002/6114f507/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/x-pkcs7-signature
Size: 8122 bytes
Desc: not available
URL: <http://lists.osu.edu/pipermail/mvapich-discuss/attachments/20241002/6114f507/attachment-0001.bin>

From shineman.5 at osu.edu  Thu Oct  3 10:31:50 2024
From: shineman.5 at osu.edu (Shineman, Nat)
Date: Thu, 3 Oct 2024 14:31:50 +0000
Subject: [Mvapich-discuss] MPI_ANY_SOURCE not receiving message on
	MVAPICH	3.0
In-Reply-To: <BN1P110MB07073EFBDF05D2C8B2C78C90DF70A@BN1P110MB0707.NAMP110.PROD.OUTLOOK.COM>
References: <BN1P110MB07073EFBDF05D2C8B2C78C90DF70A@BN1P110MB0707.NAMP110.PROD.OUTLOOK.COM>
Message-ID: <PH0PR01MB7382BD348B7504320C05E670C9712@PH0PR01MB7382.prod.exchangelabs.com>

Hi Everett,

Are you able to try this test case with MVAPICH-Plus 4.0, the Beta version is available on our website as RPMs for various systems. This issue should be resolved in the latest version.

Thanks,
Nat
________________________________
From: Mvapich-discuss <mvapich-discuss-bounces at lists.osu.edu> on behalf of Schnetzer, Everett - 1002 - MITLL via Mvapich-discuss <mvapich-discuss at lists.osu.edu>
Sent: Wednesday, October 2, 2024 16:41
To: mvapich-discuss at lists.osu.edu <mvapich-discuss at lists.osu.edu>
Subject: [Mvapich-discuss] MPI_ANY_SOURCE not receiving message on MVAPICH 3.0


Hello,



We had an application break when moving up to v3.0, and the error was a failure to receive a message when the receiver sets the source to MPI_ANY_SOURCE. Below is code for a simple test case, which is run as a single process. On MVAPICH2 v2.3.7, the code completes normally, and on v3.0 the receiver thread never receives the second message and the main program hangs on the join. Our build is on Red Hat Enterprise Linux 9.4, using gcc-13.2.0.



Sincerely,

Everett Schnetzer

MIT Lincoln Laboratory



Test code:



#include <iostream>

#include <stdlib.h>

#include <mpi.h>

#include <assert.h>

#include <thread>

#include <unistd.h>



void receiverTest()

{

    std::cout << "thread calling receive from rank 0" << std::endl;

    MPI_Status status;

    uint64_t parameters[2];

    int retval = MPI_Recv(parameters, 2, MPI_UINT64_T, 0, 0, MPI_COMM_WORLD, &status);

    assert(retval == MPI_SUCCESS);



    std::cout << "thread calling receive from any rank" << std::endl;

    retval = MPI_Recv(parameters, 2, MPI_UINT64_T, MPI_ANY_SOURCE, 0, MPI_COMM_WORLD, &status);

    assert(retval == MPI_SUCCESS);



    std::cout << "receive done" << std::endl;

}



int main(int argc, char** argv)

{

    int provided;

    MPI_Init_thread(nullptr, nullptr, MPI_THREAD_MULTIPLE, &provided);

    assert(provided == MPI_THREAD_MULTIPLE);



    std::thread rcvThread = std::thread(&receiverTest);



    sleep(1);



    std::cout << "first send to thread" << std::endl;

    uint64_t parameters[2];

    int retval = MPI_Send(parameters, 2, MPI_UINT64_T, 0, 0, MPI_COMM_WORLD);

    assert(retval == MPI_SUCCESS);



    sleep(1);



    std::cout << "second send to thread" << std::endl;

    retval = MPI_Send(parameters, 2, MPI_UINT64_T, 0, 0, MPI_COMM_WORLD);

    assert(retval == MPI_SUCCESS);



    assert(rcvThread.joinable());

    rcvThread.join();



    MPI_Finalize();



    return 0;

}


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osu.edu/pipermail/mvapich-discuss/attachments/20241003/842dca1b/attachment.html>

From novosirj at rutgers.edu  Thu Oct 10 12:20:23 2024
From: novosirj at rutgers.edu (Ryan Novosielski)
Date: Thu, 10 Oct 2024 16:20:23 +0000
Subject: [Mvapich-discuss] MVAPICH2 2.3.7-1 (and 2.3.6)
 "mv2_mad_dlopen_init" re: "Error opening libibmad.so: libibmad.so", GCC 10.4,
 CentOS 7.x
In-Reply-To: <3fe5a02e-7e7a-45fc-79bd-c55f91149a09@rutgers.edu>
References: <45bd4aa0-c3d0-b0f3-2ae0-4e5d956eba63@rutgers.edu>
	<BY5PR01MB5812DB3F68A954157764F3C8C9879@BY5PR01MB5812.prod.exchangelabs.com>
	<3fe5a02e-7e7a-45fc-79bd-c55f91149a09@rutgers.edu>
Message-ID: <AEC71D60-0B50-40DD-B864-2D7ACAAC9C02@rutgers.edu>

Nat,

Was anything ever learned about this situation? We?re in the process of trying to figure out what to do about it, and while adding that package does work (we?ve not yet tested the alternative ? to use the old build method), we?re reluctant to make a change when it?s not expected behavior that we?re trying to correct.

It seems like the situation, to refresh your (and my) memory is that something about the runtime requires libibmad.so and that libibmad.so.5 or libibmad.so.5.5.0 do not satisfy that requirement, and I think that was unexpected.

We have not attempted to use 3.x at all yet, which I assume will come with its own things to deal with, and it does seem like 2.3.7 remains the latest version there.

Thanks!

--
#BlackLivesMatter
____
|| \\UTGERS,     |---------------------------*O*---------------------------
||_// the State  |         Ryan Novosielski - novosirj at rutgers.edu
|| \\ University | Sr. Technologist - 973/972.0922 (2x0922) ~*~ RBHS Campus
||  \\    of NJ  | Office of Advanced Research Computing - MSB A555B, Newark
     `'

On Jul 11, 2022, at 17:55, Ryan Novosielski via Mvapich-discuss <mvapich-discuss at lists.osu.edu> wrote:

Aha!

[root at amarel1 ~]# find /usr/lib64 -xdev -name "libibmad*"
/usr/lib64/libibmad.so<https://urldefense.com/v3/__http://libibmad.so/__;!!KGKeukY!wjfyR0zEO9tXBgv2wtkssZ22t74-mLgDHMwsr6Cr3YiT3YCLZpv7_uNG7KYTRFf92cuhOn8mqyVKm9XSMNaFNo27FSQg9a4$ >.5.5.0
/usr/lib64/libibmad.so<https://urldefense.com/v3/__http://libibmad.so/__;!!KGKeukY!wjfyR0zEO9tXBgv2wtkssZ22t74-mLgDHMwsr6Cr3YiT3YCLZpv7_uNG7KYTRFf92cuhOn8mqyVKm9XSMNaFNo27FSQg9a4$ >.5
/usr/lib64/libibmad.so<https://urldefense.com/v3/__http://libibmad.so/__;!!KGKeukY!wjfyR0zEO9tXBgv2wtkssZ22t74-mLgDHMwsr6Cr3YiT3YCLZpv7_uNG7KYTRFf92cuhOn8mqyVKm9XSMNaFNo27FSQg9a4$ >

[root at amarel1 ~]# ssh slepner021 find /usr/lib64 -xdev -name "libibmad*"
/usr/lib64/libibmad.so<https://urldefense.com/v3/__http://libibmad.so/__;!!KGKeukY!wjfyR0zEO9tXBgv2wtkssZ22t74-mLgDHMwsr6Cr3YiT3YCLZpv7_uNG7KYTRFf92cuhOn8mqyVKm9XSMNaFNo27FSQg9a4$ >.5.5.0
/usr/lib64/libibmad.so<https://urldefense.com/v3/__http://libibmad.so/__;!!KGKeukY!wjfyR0zEO9tXBgv2wtkssZ22t74-mLgDHMwsr6Cr3YiT3YCLZpv7_uNG7KYTRFf92cuhOn8mqyVKm9XSMNaFNo27FSQg9a4$ >.5


[root at amarel1 ~]# rpm -qa | grep infiniband-diags

infiniband-diags-devel-2.1.0-1.el7.x86_64

infiniband-diags-2.1.0-1.el7.x86_64


[root at amarel1 ~]# ssh slepner021 rpm -qa | grep infiniband-diags

infiniband-diags-2.1.0-1.el7.x86_64


This is a pretty common thing we do -- have a more complete set of development libraries, etc., on our login nodes. I'm kind of surprised to see that this sort of packaging is normal -- that the -devel package contains a link from *.so to the library the non-devel package, but without the version number. I never noticed. I've never hit this problem before though with other software.

On 7/11/22 16:32, Shineman, Nat wrote:
Hi Ryan,
This is interesting. You are correct that we no longer link directly to the ib libraries, this is to allow MVAPICH2 to run on SMP only machines without needing to install the ib libraries. Instead, we try to dynamically open them as needed from within the library at runtime. The environment variable provided in the error message is there only as a fallback and should only be necessary if the library is not available on the standard |LD_LIBRARY_PATH|?. However, it looks like yours should be available on there from |/usr/lib64|?; is there any chance that the library is on a different path on the compute nodes than it is on the head node? I will try to reproduce this and see if I can figure out why it would be failing to open your libibmad.so.
To go back to using the older linking process, please try adding |--disable-ibv-dlopen|? to your configure line. Can you try that let us know if it works for you?
Thanks,
Nat
------------------------------------------------------------------------
*From:* Mvapich-discuss <mvapich-discuss-bounces+shineman.5=osu.edu at lists.osu.edu> on behalf of Ryan Novosielski via Mvapich-discuss <mvapich-discuss at lists.osu.edu>
*Sent:* Monday, July 11, 2022 11:36
*To:* mvapich-discuss at lists.osu.edu <mvapich-discuss at lists.osu.edu>
*Subject:* [Mvapich-discuss] MVAPICH2 2.3.7-1 (and 2.3.6) "mv2_mad_dlopen_init" re: "Error opening libibmad.so: libibmad.so", GCC 10.4, CentOS 7.x
Hi there,
I'm getting error messages when running an MPI job with SLURM (18.08)
using MVAPICH2, I assume, post 2.3.5, when the following change was made:
NEW Remove dependency on underlying libibverbs, libibmad, libibumad, and
librdmacm libraries using dlopen
Here's what I'm seeing:
[novosirj at amarel-test2 mpihello]$ srun --mpi=pmi2 -n 4
./mpi_hello_world.gcc-10.4.mvapich2-2.3.7-1
srun: job 20824691 queued and waiting for resources
srun: job 20824691 has been allocated resources
Error opening libibmad.so: libibmad.so: cannot open shared object file:
No such file or directory.
mv2_mad_dlopen_init returned -1
Error opening libibmad.so: libibmad.so: cannot open shared object file:
No such file or directory.
mv2_mad_dlopen_init returned -1
Error opening libibmad.so: libibmad.so: cannot open shared object file:
No such file or directory.
mv2_mad_dlopen_init returned -1
Error opening libibmad.so: libibmad.so: cannot open shared object file:
No such file or directory.
mv2_mad_dlopen_init returned -1
Hello world from processor slepner021.amarel.rutgers.edu, rank 1 out of
4 processors
Hello world from processor slepner021.amarel.rutgers.edu, rank 2 out of
4 processors
Hello world from processor slepner021.amarel.rutgers.edu, rank 3 out of
4 processors
Hello world from processor slepner009.amarel.rutgers.edu, rank 0 out of
4 processors
I don't see this on 2.3. MPI seems to be working, but I assume it's not
using Infiniband?
The libraries do exist:
[novosirj at amarel-test2 mpihello]$ rpm -ql infiniband-diags | grep mad
/usr/lib64/libibmad.so.5
/usr/lib64/libibmad.so.5.5.0
And while I assume it's normal to not see libibmad/libibumad in ldd -v
output anymore post 2.3.5 (and I don't), here's what I see on 2.3, just
to give you an idea of how it used to work:
[novosirj at amarel-test2 mpihello]$ ldd -v
mpi_hello_world.gcc-10.4.mvapich2-2.3.7-1 | head -50
          linux-vdso.so.1 =>  (0x00007fff07b0c000)
          libmpi.so.12 =>
/opt/sw/packages/gcc-4_8/mvapich2/2.3/lib/libmpi.so.12 (0x00007f36e87d9000)
          libc.so.6 => /lib64/libc.so.6 (0x00007f36e840b000)
          libgfortran.so.3 => /lib64/libgfortran.so.3 (0x00007f36e80e9000)
          libm.so.6 => /lib64/libm.so.6 (0x00007f36e7de7000)
          libnuma.so.1 => /lib64/libnuma.so.1 (0x00007f36e7bdb000)
          libxml2.so.2 => /lib64/libxml2.so.2 (0x00007f36e7871000)
          libibmad.so.5 => /lib64/libibmad.so.5 (0x00007f36e7656000)
          librdmacm.so.1 => /lib64/librdmacm.so.1 (0x00007f36e743f000)
          libibumad.so.3 => /lib64/libibumad.so.3 (0x00007f36e7236000)
          libibverbs.so.1 => /lib64/libibverbs.so.1 (0x00007f36e701d000)
          libdl.so.2 => /lib64/libdl.so.2 (0x00007f36e6e19000)
          librt.so.1 => /lib64/librt.so.1 (0x00007f36e6c11000)
          libpmi2.so.0 => /lib64/libpmi2.so.0 (0x00007f36e69f9000)
          libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f36e67dd000)
          libgcc_s.so.1 => /opt/sw/packages/gcc/10.4/lib64/libgcc_s.so.1
(0x00007f36e65c5000)
          libquadmath.so.0 =>
/opt/sw/packages/gcc/10.4/lib64/libquadmath.so.0 (0x00007f36e637e000)
          /lib64/ld-linux-x86-64.so.2 (0x00007f36e8f40000)
          libz.so.1 => /lib64/libz.so.1 (0x00007f36e6168000)
          liblzma.so.5 => /lib64/liblzma.so.5 (0x00007f36e5f42000)
          libosmcomp.so.4 => /lib64/libosmcomp.so.4 (0x00007f36e5d33000)
          libnl-route-3.so.200 => /lib64/libnl-route-3.so.200
(0x00007f36e5ac6000)
          libnl-3.so.200 => /lib64/libnl-3.so.200 (0x00007f36e58a5000)
What can/should I do about this?
Sometimes I see (not clear what conditions trigger it, but I have at
least one set of output running one of the OSU benchmarks):
Please retry with MV2_LIBIBMAD_PATH=<path/to/libibmad.so>
It seems like what's suggested in the error message is not a great
idea/this should be dealt with at compile time.
This is my build script; relatively uncomplicated:
[novosirj at amarel-test2 build]$ more
~/src/build-mvapich2-2.3.7-1-gcc-10.4.sh #!/bin/sh
module purge
module load gcc/10.4
module list
export FFLAGS="-fallow-argument-mismatch"
../mvapich2-2.3.7-1/configure --with-pmi=pmi2 --with-pm=slurm
--prefix=/opt/sw/packages/gcc-10/mvapich2/2.3.7-1 && \
         make -j32 && make check && make install
And the configure process doesn't seem to point out anything amiss:
checking for the InfiniBand includes path... default
checking for the InfiniBand library path... default
checking for library containing shm_open... -lrt
checking infiniband/verbs.h usability... yes
checking infiniband/verbs.h presence... yes
checking for infiniband/verbs.h... yes
configure: checking checking for InfiniBand umad installation...
checking infiniband/umad.h usability... yes
checking infiniband/umad.h presence... yes
checking for infiniband/umad.h... yes
configure: InfiniBand libumad found
checking whether to enable hybrid communication channel... yes
configure: checking for RDMA CM support...
checking rdma/rdma_cma.h usability... yes
checking rdma/rdma_cma.h presence... yes
checking for rdma/rdma_cma.h... yes
configure: RDMA CM support enabled
configure: checking for hardware multicast support...
checking infiniband/mad.h usability... yes
checking infiniband/mad.h presence... yes
checking for infiniband/mad.h... yes
Thanks!
--
#BlackLivesMatter
____
  || \\UTGERS,     |----------------------*O*------------------------
  ||_// the State  |    Ryan Novosielski - novosirj at rutgers.edu
  || \\ University | Sr. Technologist - 973/972.0922 ~*~ RBHS Campus
  ||  \\    of NJ  | Office of Advanced Res. Comp. - MSB C630, Newark
_______________________________________________
Mvapich-discuss mailing list
Mvapich-discuss at lists.osu.edu
https://lists.osu.edu/mailman/listinfo/mvapich-discuss<https://lists.osu.edu/mailman/listinfo/mvapich-discuss>

--
#BlackLivesMatter
____
|| \\UTGERS,     |----------------------*O*------------------------
||_// the State  |    Ryan Novosielski - novosirj at rutgers.edu<mailto:novosirj at rutgers.edu>
|| \\ University | Sr. Technologist - 973/972.0922 ~*~ RBHS Campus
||  \\    of NJ  | Office of Advanced Res. Comp. - MSB C630, Newark
     `'
_______________________________________________
Mvapich-discuss mailing list
Mvapich-discuss at lists.osu.edu<mailto:Mvapich-discuss at lists.osu.edu>
https://lists.osu.edu/mailman/listinfo/mvapich-discuss

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osu.edu/pipermail/mvapich-discuss/attachments/20241010/4b7ab988/attachment-0001.html>

From bo18resch11002 at iith.ac.in  Mon Oct 14 01:41:41 2024
From: bo18resch11002 at iith.ac.in (Nitin Kulhar)
Date: Mon, 14 Oct 2024 11:11:41 +0530
Subject: [Mvapich-discuss] MVAPICH2-GDR-2.3.7: Wrapper Script Changes After
	'rpm2cpio' Extraction
Message-ID: <CAP1fmNzN8ejqDt_oo9=QSP5GQWYZcdWhtATtDw_bAc3deVWGuw@mail.gmail.com>

Dear all

I am installing mvapich2-gdr-2.3.7
in the user-space of our institute's
HPC facility from the rpm:
*mvapich2-gdr-cuda11.4.mofed5.4.gnu11.2.0-2.3.7-1.el8.x86_64.rpm*
How should I change the wrapper
scripts (mpicc, mpicxx, mpif77,
mpif90, mpifc) to render them fit
to compile Amber24 executables
with CUDA and/or MPI support?

I can load CUDA-11.4 from a
module available on the HPC,
and, I have a local installation
of gcc (11.2.0) in the user-space
under the $HOME tree.

I would be grateful to receive any
help in this regard.

Thanks and regards
Nitin

-- 


Disclaimer:- This?footer text is to convey that this email is sent by one 
of the?users of IITH. So, do not mark it as SPAM.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osu.edu/pipermail/mvapich-discuss/attachments/20241014/f5ca65d7/attachment.html>

From mohil at cdac.in  Mon Oct 14 02:17:47 2024
From: mohil at cdac.in (mohil)
Date: Mon, 14 Oct 2024 11:47:47 +0530 (IST)
Subject: [Mvapich-discuss] Regarding Topology awareness in MVAPICH
Message-ID: <666329362.21946.1728886667184.JavaMail.open-xchange@webmail.cdac.in>

hello all,

Recently i am studying about topology awareness in mvapich-3.0, so hardware
topology detection part is clear that is managed using the hwloc library. But I
don`t get how the network topology detection work, as I get to know mvapich-3.0
doesn't use netloc for topology discovery. So, I want to know. If network
topology awareness is present in recent mvapich-3.0?
If yes, how does the network topology awareness work in mvapich-3.0 ? Where in
source code i can found more about the network topology detection and
optimisation done using that info?

I will be extremely thankful to anyone able to help me with this information.

With Regards
MOHIL
mohil at cdac.in
------------------------------------------------------------------------------------------------------------
[ C-DAC is on Social-Media too. Kindly follow us at:
Facebook: https://urldefense.com/v3/__https://www.facebook.com/CDACINDIA__;!!KGKeukY!wgpGvVROip1ylhtv2uDAGJaYcmB0ASt4-pHd3-FUaPu2zef-csw-4b6NfUPXA2HPCjj1HHEMGqLTUubTFhlOSDtW$  & Twitter: @cdacindia ]

This e-mail is for the sole use of the intended recipient(s) and may
contain confidential and privileged information. If you are not the
intended recipient, please contact the sender by reply e-mail and destroy
all copies and the original message. Any unauthorized review, use,
disclosure, dissemination, forwarding, printing or copying of this email
is strictly prohibited and appropriate legal action will be taken.
------------------------------------------------------------------------------------------------------------

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osu.edu/pipermail/mvapich-discuss/attachments/20241014/be7ec6e9/attachment-0001.html>

From bo18resch11002 at iith.ac.in  Mon Oct 14 10:52:14 2024
From: bo18resch11002 at iith.ac.in (Nitin Kulhar)
Date: Mon, 14 Oct 2024 20:22:14 +0530
Subject: [Mvapich-discuss] MVAPICH2-GDR-2.3.7: Wrapper Script Changes
	After 'rpm2cpio' Extraction
In-Reply-To: <CAP1fmNzN8ejqDt_oo9=QSP5GQWYZcdWhtATtDw_bAc3deVWGuw@mail.gmail.com>
References: <CAP1fmNzN8ejqDt_oo9=QSP5GQWYZcdWhtATtDw_bAc3deVWGuw@mail.gmail.com>
Message-ID: <CAP1fmNxVo4yNb++DWfGxY3HJsu818b3eL=MxX0HvFUjQqpZJAQ@mail.gmail.com>

Dear concerned
This is to bring to your attention
that there is a Subject/Body
mix-up between two threads on
the list. I am seeing the Body
of a different thread under the
Subject Line of my message.

Link to my message:
https://lists.osu.edu/pipermail/mvapich-discuss/2024-October/011769.html

On Mon, Oct 14, 2024 at 11:11?AM Nitin Kulhar <bo18resch11002 at iith.ac.in>
wrote:

> Dear all
>
> I am installing mvapich2-gdr-2.3.7
> in the user-space of our institute's
> HPC facility from the rpm:
> *mvapich2-gdr-cuda11.4.mofed5.4.gnu11.2.0-2.3.7-1.el8.x86_64.rpm*
> How should I change the wrapper
> scripts (mpicc, mpicxx, mpif77,
> mpif90, mpifc) to render them fit
> to compile Amber24 executables
> with CUDA and/or MPI support?
>
> I can load CUDA-11.4 from a
> module available on the HPC,
> and, I have a local installation
> of gcc (11.2.0) in the user-space
> under the $HOME tree.
>
> I would be grateful to receive any
> help in this regard.
>
> Thanks and regards
> Nitin
>

-- 


Disclaimer:- This?footer text is to convey that this email is sent by one 
of the?users of IITH. So, do not mark it as SPAM.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osu.edu/pipermail/mvapich-discuss/attachments/20241014/df782cca/attachment.html>

From zyou at osc.edu  Wed Oct 16 11:50:19 2024
From: zyou at osc.edu (You, Zhi-Qiang)
Date: Wed, 16 Oct 2024 15:50:19 +0000
Subject: [Mvapich-discuss] Handling MVAPICH 3.0 full subscription warning
Message-ID: <BL0PR01MB50106BCFBFF1ED776D635375C6462@BL0PR01MB5010.prod.exchangelabs.com>

Hi,

We encountered the following warning message while running a full-node MPI job with MVAPICH 3.0:

[][mvp_generate_implicit_cpu_mapping] WARNING: You appear to be running at full subscription for this job. UCX spawns an additional thread for each process which may result in oversubscribed cores and poor performance. Please consider reserving at least 2 cores per node for the additional threads, enabling SMT, or setting MVP_THREADS_PER_PROCESS=2 to ensure that sufficient resources are available.

The suggestion to set MVP_THREADS_PER_PROCESS=2 not only fails to improve performance but actually degrades it. Can this warning message be safely ignored, or is there any action I need to take to address it?

Best,
ZQ

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osu.edu/pipermail/mvapich-discuss/attachments/20241016/16123335/attachment.html>

From shineman.5 at osu.edu  Wed Oct 16 14:16:10 2024
From: shineman.5 at osu.edu (Shineman, Nat)
Date: Wed, 16 Oct 2024 18:16:10 +0000
Subject: [Mvapich-discuss] Handling MVAPICH 3.0 full subscription warning
In-Reply-To: <BL0PR01MB50106BCFBFF1ED776D635375C6462@BL0PR01MB5010.prod.exchangelabs.com>
References: <BL0PR01MB50106BCFBFF1ED776D635375C6462@BL0PR01MB5010.prod.exchangelabs.com>
Message-ID: <PH0PR01MB7382827442A6443DCCCB7A12C9462@PH0PR01MB7382.prod.exchangelabs.com>

Hi ZQ,

You are probably seeing degraded performance because you are still running the application at full subscription and requesting that MVAPICH reserve 2 cores per process. The warning should probably more accurately state that you should cap your runs at 1/2 subscription and set the listed environment variable. This would prevent you from oversubscribing cores.

However, if you are seeing satisfactory performance with oversubscribed cores in full subscription, please feel free to ignore the warning.

Thanks,
Nat
________________________________
From: Mvapich-discuss <mvapich-discuss-bounces at lists.osu.edu> on behalf of You, Zhi-Qiang via Mvapich-discuss <mvapich-discuss at lists.osu.edu>
Sent: Wednesday, October 16, 2024 11:50
To: mvapich-discuss at lists.osu.edu <mvapich-discuss at lists.osu.edu>
Subject: [Mvapich-discuss] Handling MVAPICH 3.0 full subscription warning


Hi,



We encountered the following warning message while running a full-node MPI job with MVAPICH 3.0:

[][mvp_generate_implicit_cpu_mapping] WARNING: You appear to be running at full subscription for this job. UCX spawns an additional thread for each process which may result in oversubscribed cores and poor performance. Please consider reserving at least 2 cores per node for the additional threads, enabling SMT, or setting MVP_THREADS_PER_PROCESS=2 to ensure that sufficient resources are available.

The suggestion to set MVP_THREADS_PER_PROCESS=2 not only fails to improve performance but actually degrades it. Can this warning message be safely ignored, or is there any action I need to take to address it?



Best,

ZQ


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osu.edu/pipermail/mvapich-discuss/attachments/20241016/0ed6b931/attachment.html>

From simone at fnal.gov  Thu Oct 17 12:07:23 2024
From: simone at fnal.gov (James N Simone)
Date: Thu, 17 Oct 2024 16:07:23 +0000
Subject: [Mvapich-discuss] <no subject>
Message-ID: <09AC6A77-F2AA-4E28-981B-C3C2A71F5025@contoso.com>

!-------------------------------------------------------------------|
  This Message Is From an External Sender
  This message came from outside your organization.
|-------------------------------------------------------------------!

UNSUBSCRIBE






From zyou at osc.edu  Thu Oct 24 10:54:23 2024
From: zyou at osc.edu (You, Zhi-Qiang)
Date: Thu, 24 Oct 2024 14:54:23 +0000
Subject: [Mvapich-discuss] Inquiry Regarding IB Performance on New Cluster
 with NDR 400 and MVAPICH
Message-ID: <BL0PR01MB50107E49C536D6F5DE9FA074C64E2@BL0PR01MB5010.prod.exchangelabs.com>

Dear MVAPICH developers,

We recently tested IB performance on our new cluster at OSC. On other clusters, we are able to achieve the theoretical limits of the IB devices using omb_bibw. For example:


  *   NDR 200: 48 GB/s
  *   EDR 100: 24.5 GB/s

However, on the new cluster with NDR 400, the best performance I am able to achieve is around 67 GB/s. Does this number seem reasonable? I have tested with MVAPICH 3.0 and MVAPICH 2.3.7-1, both built with Intel 2021.10.0. The OMB versions used are 5.9 for MVAPICH 2.3.7-1 and 7.3 for MVAPICH 3.0. The parameters used for omb_bibw were ?-m 4194304:4194304 -x 1000 -i 10000?.

Best regards,
ZQ
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osu.edu/pipermail/mvapich-discuss/attachments/20241024/8ef3f284/attachment.html>

From panda at cse.ohio-state.edu  Thu Oct 24 23:57:03 2024
From: panda at cse.ohio-state.edu (Panda, Dhabaleswar)
Date: Fri, 25 Oct 2024 03:57:03 +0000
Subject: [Mvapich-discuss] The MVAPICH Group is looking for a talented
 full-time software developer to join the team
Message-ID: <DM6PR01MB431583DFC4726D723712BE16DA4F2@DM6PR01MB4315.prod.exchangelabs.com>

More details on this position are available from:

https://osu.wd1.myworkdayjobs.com/OSUCareers/job/Columbus-Campus/Applications-Developer_R116828-1

If you are interested in this position and have authorization to work in the USA, please apply ASAP. 

Thanks, 

DK Panda
 


From zyou at osc.edu  Thu Oct 31 12:11:04 2024
From: zyou at osc.edu (You, Zhi-Qiang)
Date: Thu, 31 Oct 2024 16:11:04 +0000
Subject: [Mvapich-discuss] Handling MVAPICH 3.0 full subscription warning
In-Reply-To: <PH0PR01MB7382827442A6443DCCCB7A12C9462@PH0PR01MB7382.prod.exchangelabs.com>
References: <BL0PR01MB50106BCFBFF1ED776D635375C6462@BL0PR01MB5010.prod.exchangelabs.com>
	<PH0PR01MB7382827442A6443DCCCB7A12C9462@PH0PR01MB7382.prod.exchangelabs.com>
Message-ID: <BL0PR01MB501045D37526D8346BF24FAAC6552@BL0PR01MB5010.prod.exchangelabs.com>

Hi Nat,

Thank you for the suggestion. I have a few questions:


  1.  Does this message indicate that oversubscription is occurring, or is it simply a warning that appears every time a full-node job is run? In one user?s case, I did not observe any oversubscription, although the warning was present.
  2.  UCX is also the default for OpenMPI, but I did not see a similar warning when running a full-node job with OpenMPI. Why does this only happen with MVAPICH?

Thank you,
ZQ

From: Shineman, Nat <shineman.5 at osu.edu>
Date: Wednesday, October 16, 2024 at 2:16?PM
To: mvapich-discuss at lists.osu.edu <mvapich-discuss at lists.osu.edu>, You, Zhi-Qiang <zyou at osc.edu>
Subject: Re: Handling MVAPICH 3.0 full subscription warning
Hi ZQ,

You are probably seeing degraded performance because you are still running the application at full subscription and requesting that MVAPICH reserve 2 cores per process. The warning should probably more accurately state that you should cap your runs at 1/2 subscription and set the listed environment variable. This would prevent you from oversubscribing cores.

However, if you are seeing satisfactory performance with oversubscribed cores in full subscription, please feel free to ignore the warning.

Thanks,
Nat
________________________________
From: Mvapich-discuss <mvapich-discuss-bounces at lists.osu.edu> on behalf of You, Zhi-Qiang via Mvapich-discuss <mvapich-discuss at lists.osu.edu>
Sent: Wednesday, October 16, 2024 11:50
To: mvapich-discuss at lists.osu.edu <mvapich-discuss at lists.osu.edu>
Subject: [Mvapich-discuss] Handling MVAPICH 3.0 full subscription warning


Hi,



We encountered the following warning message while running a full-node MPI job with MVAPICH 3.0:

[][mvp_generate_implicit_cpu_mapping] WARNING: You appear to be running at full subscription for this job. UCX spawns an additional thread for each process which may result in oversubscribed cores and poor performance. Please consider reserving at least 2 cores per node for the additional threads, enabling SMT, or setting MVP_THREADS_PER_PROCESS=2 to ensure that sufficient resources are available.

The suggestion to set MVP_THREADS_PER_PROCESS=2 not only fails to improve performance but actually degrades it. Can this warning message be safely ignored, or is there any action I need to take to address it?



Best,

ZQ


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osu.edu/pipermail/mvapich-discuss/attachments/20241031/10492c20/attachment.html>

From shineman.5 at osu.edu  Thu Oct 31 12:55:46 2024
From: shineman.5 at osu.edu (Shineman, Nat)
Date: Thu, 31 Oct 2024 16:55:46 +0000
Subject: [Mvapich-discuss] Handling MVAPICH 3.0 full subscription warning
In-Reply-To: <BL0PR01MB501045D37526D8346BF24FAAC6552@BL0PR01MB5010.prod.exchangelabs.com>
References: <BL0PR01MB50106BCFBFF1ED776D635375C6462@BL0PR01MB5010.prod.exchangelabs.com>
	<PH0PR01MB7382827442A6443DCCCB7A12C9462@PH0PR01MB7382.prod.exchangelabs.com>
	<BL0PR01MB501045D37526D8346BF24FAAC6552@BL0PR01MB5010.prod.exchangelabs.com>
Message-ID: <PH0PR01MB73829ECF926880542E3051E8C9552@PH0PR01MB7382.prod.exchangelabs.com>

ZQ,

This warning will appear anytime you allocate a full subscription job with the UCX netmod enabled. It is based on observed behaviour of UCX, which we have seen allocated an extra progress thread for each process that calls ucx_init. I cannot guarantee this will happen in every case, especially if you are using your own UCX version, but we have observed it in most if not all of our testing with UCX. If you are not seeing any performance impacts and can observe no oversubscription, then I would encourage you to disregard the warning and trust what you are observing.

Regarding your second question, the warning is coming from MVAPICH, so that is why it does not appear in OpenMPI. I cannot speak to how OpenMPI handles these additional UCX progress threads or what if anything they might do to notify a user.

Thanks,
Nat
________________________________
From: You, Zhi-Qiang <zyou at osc.edu>
Sent: Thursday, October 31, 2024 12:11
To: Shineman, Nat <shineman.5 at osu.edu>; mvapich-discuss at lists.osu.edu <mvapich-discuss at lists.osu.edu>
Subject: Re: Handling MVAPICH 3.0 full subscription warning


Hi Nat,



Thank you for the suggestion. I have a few questions:



  1.  Does this message indicate that oversubscription is occurring, or is it simply a warning that appears every time a full-node job is run? In one user?s case, I did not observe any oversubscription, although the warning was present.
  2.  UCX is also the default for OpenMPI, but I did not see a similar warning when running a full-node job with OpenMPI. Why does this only happen with MVAPICH?



Thank you,

ZQ



From: Shineman, Nat <shineman.5 at osu.edu>
Date: Wednesday, October 16, 2024 at 2:16?PM
To: mvapich-discuss at lists.osu.edu <mvapich-discuss at lists.osu.edu>, You, Zhi-Qiang <zyou at osc.edu>
Subject: Re: Handling MVAPICH 3.0 full subscription warning

Hi ZQ,



You are probably seeing degraded performance because you are still running the application at full subscription and requesting that MVAPICH reserve 2 cores per process. The warning should probably more accurately state that you should cap your runs at 1/2 subscription and set the listed environment variable. This would prevent you from oversubscribing cores.



However, if you are seeing satisfactory performance with oversubscribed cores in full subscription, please feel free to ignore the warning.



Thanks,

Nat

________________________________

From: Mvapich-discuss <mvapich-discuss-bounces at lists.osu.edu> on behalf of You, Zhi-Qiang via Mvapich-discuss <mvapich-discuss at lists.osu.edu>
Sent: Wednesday, October 16, 2024 11:50
To: mvapich-discuss at lists.osu.edu <mvapich-discuss at lists.osu.edu>
Subject: [Mvapich-discuss] Handling MVAPICH 3.0 full subscription warning



Hi,



We encountered the following warning message while running a full-node MPI job with MVAPICH 3.0:

[][mvp_generate_implicit_cpu_mapping] WARNING: You appear to be running at full subscription for this job. UCX spawns an additional thread for each process which may result in oversubscribed cores and poor performance. Please consider reserving at least 2 cores per node for the additional threads, enabling SMT, or setting MVP_THREADS_PER_PROCESS=2 to ensure that sufficient resources are available.

The suggestion to set MVP_THREADS_PER_PROCESS=2 not only fails to improve performance but actually degrades it. Can this warning message be safely ignored, or is there any action I need to take to address it?



Best,

ZQ


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osu.edu/pipermail/mvapich-discuss/attachments/20241031/3ffc943f/attachment-0001.html>

From kumarmohil07 at gmail.com  Tue Oct  1 08:51:29 2024
From: kumarmohil07 at gmail.com (Mohil Kumar)
Date: Tue, 1 Oct 2024 18:21:29 +0530
Subject: [Mvapich-discuss] Regarding how topology awareness works in mvapich
Message-ID: <CAMVkEeWpjyOR9uD_xusW5JsO53R1s9U6YAp18nUxvzQHEQiwAQ@mail.gmail.com>

Hello,
Myself Mohil, Currently I have been studying how topology awareness in
mvapich works. I get to know about *physical* topology and *virtual*
topology and how the virtual topology is mapped to physical topology(like
various *mapping* *algorithms*).
But before mapping virtual topology on physical topology we have to detect
the physical topology, which involves detecting both hardware(intranode)
and network topology(internode). Hardware topology is detected using *hwloc*
library.

But I am confused how the network topology detection works. Will it be done
by a job scheduler like *SLURM* or *mvapich* to get this info from the
underlying network interface somehow.

If anyone is able to help me I will be very thankful to you. Also, if I
make any mistakes in my interpretation of the whole thing, I am open to any
suggestions.

With regards
MOHIL
kumarmohil07 at gmail.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osu.edu/pipermail/mvapich-discuss/attachments/20241001/e176f93b/attachment-0002.html>

From e.schnetzer at ll.mit.edu  Wed Oct  2 16:41:14 2024
From: e.schnetzer at ll.mit.edu (Schnetzer, Everett - 1002 - MITLL)
Date: Wed, 2 Oct 2024 20:41:14 +0000
Subject: [Mvapich-discuss] MPI_ANY_SOURCE not receiving message on MVAPICH
	3.0
Message-ID: <BN1P110MB07073EFBDF05D2C8B2C78C90DF70A@BN1P110MB0707.NAMP110.PROD.OUTLOOK.COM>

?Hello, 

We had an application break when moving up to v3.0, and the error was a failure to receive a message when the receiver sets the source to MPI_ANY_SOURCE. Below is code for a simple test case, which is run as a single process. On MVAPICH2 v2.3.7, the code completes normally, and on v3.0 the receiver thread never receives the second message and the main program hangs on the join. Our build is on Red Hat Enterprise Linux 9.4, using gcc-13.2.0. 

Sincerely, 
Everett Schnetzer 
MIT Lincoln Laboratory 

Test code: 

#include <iostream> 
#include <stdlib.h> 
#include <mpi.h> 
#include <assert.h> 
#include <thread> 
#include <unistd.h> 

void receiverTest() 
{ 
std::cout << "thread calling receive from rank 0" << std::endl; 
MPI_Status status; 
uint64_t parameters[2]; 
int retval = MPI_Recv(parameters, 2, MPI_UINT64_T, 0, 0, MPI_COMM_WORLD, &status); 
assert(retval == MPI_SUCCESS); 

std::cout << "thread calling receive from any rank" << std::endl; 
retval = MPI_Recv(parameters, 2, MPI_UINT64_T, MPI_ANY_SOURCE, 0, MPI_COMM_WORLD, &status); 
assert(retval == MPI_SUCCESS); 

std::cout << "receive done" << std::endl; 
} 

int main(int argc, char** argv) 
{ 
int provided; 
MPI_Init_thread(nullptr, nullptr, MPI_THREAD_MULTIPLE, &provided); 
assert(provided == MPI_THREAD_MULTIPLE); 

std::thread rcvThread = std::thread(&receiverTest); 

sleep(1); 

std::cout << "first send to thread" << std::endl; 
uint64_t parameters[2]; 
int retval = MPI_Send(parameters, 2, MPI_UINT64_T, 0, 0, MPI_COMM_WORLD); 
assert(retval == MPI_SUCCESS); 

sleep(1); 

std::cout << "second send to thread" << std::endl; 
retval = MPI_Send(parameters, 2, MPI_UINT64_T, 0, 0, MPI_COMM_WORLD); 
assert(retval == MPI_SUCCESS); 

assert(rcvThread.joinable()); 
rcvThread.join(); 

MPI_Finalize(); 

return 0; 
} 


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osu.edu/pipermail/mvapich-discuss/attachments/20241002/6114f507/attachment-0002.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/x-pkcs7-signature
Size: 8122 bytes
Desc: not available
URL: <http://lists.osu.edu/pipermail/mvapich-discuss/attachments/20241002/6114f507/attachment-0002.bin>

From shineman.5 at osu.edu  Thu Oct  3 10:31:50 2024
From: shineman.5 at osu.edu (Shineman, Nat)
Date: Thu, 3 Oct 2024 14:31:50 +0000
Subject: [Mvapich-discuss] MPI_ANY_SOURCE not receiving message on
	MVAPICH	3.0
In-Reply-To: <BN1P110MB07073EFBDF05D2C8B2C78C90DF70A@BN1P110MB0707.NAMP110.PROD.OUTLOOK.COM>
References: <BN1P110MB07073EFBDF05D2C8B2C78C90DF70A@BN1P110MB0707.NAMP110.PROD.OUTLOOK.COM>
Message-ID: <PH0PR01MB7382BD348B7504320C05E670C9712@PH0PR01MB7382.prod.exchangelabs.com>

Hi Everett,

Are you able to try this test case with MVAPICH-Plus 4.0, the Beta version is available on our website as RPMs for various systems. This issue should be resolved in the latest version.

Thanks,
Nat
________________________________
From: Mvapich-discuss <mvapich-discuss-bounces at lists.osu.edu> on behalf of Schnetzer, Everett - 1002 - MITLL via Mvapich-discuss <mvapich-discuss at lists.osu.edu>
Sent: Wednesday, October 2, 2024 16:41
To: mvapich-discuss at lists.osu.edu <mvapich-discuss at lists.osu.edu>
Subject: [Mvapich-discuss] MPI_ANY_SOURCE not receiving message on MVAPICH 3.0


Hello,



We had an application break when moving up to v3.0, and the error was a failure to receive a message when the receiver sets the source to MPI_ANY_SOURCE. Below is code for a simple test case, which is run as a single process. On MVAPICH2 v2.3.7, the code completes normally, and on v3.0 the receiver thread never receives the second message and the main program hangs on the join. Our build is on Red Hat Enterprise Linux 9.4, using gcc-13.2.0.



Sincerely,

Everett Schnetzer

MIT Lincoln Laboratory



Test code:



#include <iostream>

#include <stdlib.h>

#include <mpi.h>

#include <assert.h>

#include <thread>

#include <unistd.h>



void receiverTest()

{

    std::cout << "thread calling receive from rank 0" << std::endl;

    MPI_Status status;

    uint64_t parameters[2];

    int retval = MPI_Recv(parameters, 2, MPI_UINT64_T, 0, 0, MPI_COMM_WORLD, &status);

    assert(retval == MPI_SUCCESS);



    std::cout << "thread calling receive from any rank" << std::endl;

    retval = MPI_Recv(parameters, 2, MPI_UINT64_T, MPI_ANY_SOURCE, 0, MPI_COMM_WORLD, &status);

    assert(retval == MPI_SUCCESS);



    std::cout << "receive done" << std::endl;

}



int main(int argc, char** argv)

{

    int provided;

    MPI_Init_thread(nullptr, nullptr, MPI_THREAD_MULTIPLE, &provided);

    assert(provided == MPI_THREAD_MULTIPLE);



    std::thread rcvThread = std::thread(&receiverTest);



    sleep(1);



    std::cout << "first send to thread" << std::endl;

    uint64_t parameters[2];

    int retval = MPI_Send(parameters, 2, MPI_UINT64_T, 0, 0, MPI_COMM_WORLD);

    assert(retval == MPI_SUCCESS);



    sleep(1);



    std::cout << "second send to thread" << std::endl;

    retval = MPI_Send(parameters, 2, MPI_UINT64_T, 0, 0, MPI_COMM_WORLD);

    assert(retval == MPI_SUCCESS);



    assert(rcvThread.joinable());

    rcvThread.join();



    MPI_Finalize();



    return 0;

}


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osu.edu/pipermail/mvapich-discuss/attachments/20241003/842dca1b/attachment-0002.html>

From novosirj at rutgers.edu  Thu Oct 10 12:20:23 2024
From: novosirj at rutgers.edu (Ryan Novosielski)
Date: Thu, 10 Oct 2024 16:20:23 +0000
Subject: [Mvapich-discuss] MVAPICH2 2.3.7-1 (and 2.3.6)
 "mv2_mad_dlopen_init" re: "Error opening libibmad.so: libibmad.so", GCC 10.4,
 CentOS 7.x
In-Reply-To: <3fe5a02e-7e7a-45fc-79bd-c55f91149a09@rutgers.edu>
References: <45bd4aa0-c3d0-b0f3-2ae0-4e5d956eba63@rutgers.edu>
	<BY5PR01MB5812DB3F68A954157764F3C8C9879@BY5PR01MB5812.prod.exchangelabs.com>
	<3fe5a02e-7e7a-45fc-79bd-c55f91149a09@rutgers.edu>
Message-ID: <AEC71D60-0B50-40DD-B864-2D7ACAAC9C02@rutgers.edu>

Nat,

Was anything ever learned about this situation? We?re in the process of trying to figure out what to do about it, and while adding that package does work (we?ve not yet tested the alternative ? to use the old build method), we?re reluctant to make a change when it?s not expected behavior that we?re trying to correct.

It seems like the situation, to refresh your (and my) memory is that something about the runtime requires libibmad.so and that libibmad.so.5 or libibmad.so.5.5.0 do not satisfy that requirement, and I think that was unexpected.

We have not attempted to use 3.x at all yet, which I assume will come with its own things to deal with, and it does seem like 2.3.7 remains the latest version there.

Thanks!

--
#BlackLivesMatter
____
|| \\UTGERS,     |---------------------------*O*---------------------------
||_// the State  |         Ryan Novosielski - novosirj at rutgers.edu
|| \\ University | Sr. Technologist - 973/972.0922 (2x0922) ~*~ RBHS Campus
||  \\    of NJ  | Office of Advanced Research Computing - MSB A555B, Newark
     `'

On Jul 11, 2022, at 17:55, Ryan Novosielski via Mvapich-discuss <mvapich-discuss at lists.osu.edu> wrote:

Aha!

[root at amarel1 ~]# find /usr/lib64 -xdev -name "libibmad*"
/usr/lib64/libibmad.so<https://urldefense.com/v3/__http://libibmad.so/__;!!KGKeukY!wjfyR0zEO9tXBgv2wtkssZ22t74-mLgDHMwsr6Cr3YiT3YCLZpv7_uNG7KYTRFf92cuhOn8mqyVKm9XSMNaFNo27FSQg9a4$ >.5.5.0
/usr/lib64/libibmad.so<https://urldefense.com/v3/__http://libibmad.so/__;!!KGKeukY!wjfyR0zEO9tXBgv2wtkssZ22t74-mLgDHMwsr6Cr3YiT3YCLZpv7_uNG7KYTRFf92cuhOn8mqyVKm9XSMNaFNo27FSQg9a4$ >.5
/usr/lib64/libibmad.so<https://urldefense.com/v3/__http://libibmad.so/__;!!KGKeukY!wjfyR0zEO9tXBgv2wtkssZ22t74-mLgDHMwsr6Cr3YiT3YCLZpv7_uNG7KYTRFf92cuhOn8mqyVKm9XSMNaFNo27FSQg9a4$ >

[root at amarel1 ~]# ssh slepner021 find /usr/lib64 -xdev -name "libibmad*"
/usr/lib64/libibmad.so<https://urldefense.com/v3/__http://libibmad.so/__;!!KGKeukY!wjfyR0zEO9tXBgv2wtkssZ22t74-mLgDHMwsr6Cr3YiT3YCLZpv7_uNG7KYTRFf92cuhOn8mqyVKm9XSMNaFNo27FSQg9a4$ >.5.5.0
/usr/lib64/libibmad.so<https://urldefense.com/v3/__http://libibmad.so/__;!!KGKeukY!wjfyR0zEO9tXBgv2wtkssZ22t74-mLgDHMwsr6Cr3YiT3YCLZpv7_uNG7KYTRFf92cuhOn8mqyVKm9XSMNaFNo27FSQg9a4$ >.5


[root at amarel1 ~]# rpm -qa | grep infiniband-diags

infiniband-diags-devel-2.1.0-1.el7.x86_64

infiniband-diags-2.1.0-1.el7.x86_64


[root at amarel1 ~]# ssh slepner021 rpm -qa | grep infiniband-diags

infiniband-diags-2.1.0-1.el7.x86_64


This is a pretty common thing we do -- have a more complete set of development libraries, etc., on our login nodes. I'm kind of surprised to see that this sort of packaging is normal -- that the -devel package contains a link from *.so to the library the non-devel package, but without the version number. I never noticed. I've never hit this problem before though with other software.

On 7/11/22 16:32, Shineman, Nat wrote:
Hi Ryan,
This is interesting. You are correct that we no longer link directly to the ib libraries, this is to allow MVAPICH2 to run on SMP only machines without needing to install the ib libraries. Instead, we try to dynamically open them as needed from within the library at runtime. The environment variable provided in the error message is there only as a fallback and should only be necessary if the library is not available on the standard |LD_LIBRARY_PATH|?. However, it looks like yours should be available on there from |/usr/lib64|?; is there any chance that the library is on a different path on the compute nodes than it is on the head node? I will try to reproduce this and see if I can figure out why it would be failing to open your libibmad.so.
To go back to using the older linking process, please try adding |--disable-ibv-dlopen|? to your configure line. Can you try that let us know if it works for you?
Thanks,
Nat
------------------------------------------------------------------------
*From:* Mvapich-discuss <mvapich-discuss-bounces+shineman.5=osu.edu at lists.osu.edu> on behalf of Ryan Novosielski via Mvapich-discuss <mvapich-discuss at lists.osu.edu>
*Sent:* Monday, July 11, 2022 11:36
*To:* mvapich-discuss at lists.osu.edu <mvapich-discuss at lists.osu.edu>
*Subject:* [Mvapich-discuss] MVAPICH2 2.3.7-1 (and 2.3.6) "mv2_mad_dlopen_init" re: "Error opening libibmad.so: libibmad.so", GCC 10.4, CentOS 7.x
Hi there,
I'm getting error messages when running an MPI job with SLURM (18.08)
using MVAPICH2, I assume, post 2.3.5, when the following change was made:
NEW Remove dependency on underlying libibverbs, libibmad, libibumad, and
librdmacm libraries using dlopen
Here's what I'm seeing:
[novosirj at amarel-test2 mpihello]$ srun --mpi=pmi2 -n 4
./mpi_hello_world.gcc-10.4.mvapich2-2.3.7-1
srun: job 20824691 queued and waiting for resources
srun: job 20824691 has been allocated resources
Error opening libibmad.so: libibmad.so: cannot open shared object file:
No such file or directory.
mv2_mad_dlopen_init returned -1
Error opening libibmad.so: libibmad.so: cannot open shared object file:
No such file or directory.
mv2_mad_dlopen_init returned -1
Error opening libibmad.so: libibmad.so: cannot open shared object file:
No such file or directory.
mv2_mad_dlopen_init returned -1
Error opening libibmad.so: libibmad.so: cannot open shared object file:
No such file or directory.
mv2_mad_dlopen_init returned -1
Hello world from processor slepner021.amarel.rutgers.edu, rank 1 out of
4 processors
Hello world from processor slepner021.amarel.rutgers.edu, rank 2 out of
4 processors
Hello world from processor slepner021.amarel.rutgers.edu, rank 3 out of
4 processors
Hello world from processor slepner009.amarel.rutgers.edu, rank 0 out of
4 processors
I don't see this on 2.3. MPI seems to be working, but I assume it's not
using Infiniband?
The libraries do exist:
[novosirj at amarel-test2 mpihello]$ rpm -ql infiniband-diags | grep mad
/usr/lib64/libibmad.so.5
/usr/lib64/libibmad.so.5.5.0
And while I assume it's normal to not see libibmad/libibumad in ldd -v
output anymore post 2.3.5 (and I don't), here's what I see on 2.3, just
to give you an idea of how it used to work:
[novosirj at amarel-test2 mpihello]$ ldd -v
mpi_hello_world.gcc-10.4.mvapich2-2.3.7-1 | head -50
          linux-vdso.so.1 =>  (0x00007fff07b0c000)
          libmpi.so.12 =>
/opt/sw/packages/gcc-4_8/mvapich2/2.3/lib/libmpi.so.12 (0x00007f36e87d9000)
          libc.so.6 => /lib64/libc.so.6 (0x00007f36e840b000)
          libgfortran.so.3 => /lib64/libgfortran.so.3 (0x00007f36e80e9000)
          libm.so.6 => /lib64/libm.so.6 (0x00007f36e7de7000)
          libnuma.so.1 => /lib64/libnuma.so.1 (0x00007f36e7bdb000)
          libxml2.so.2 => /lib64/libxml2.so.2 (0x00007f36e7871000)
          libibmad.so.5 => /lib64/libibmad.so.5 (0x00007f36e7656000)
          librdmacm.so.1 => /lib64/librdmacm.so.1 (0x00007f36e743f000)
          libibumad.so.3 => /lib64/libibumad.so.3 (0x00007f36e7236000)
          libibverbs.so.1 => /lib64/libibverbs.so.1 (0x00007f36e701d000)
          libdl.so.2 => /lib64/libdl.so.2 (0x00007f36e6e19000)
          librt.so.1 => /lib64/librt.so.1 (0x00007f36e6c11000)
          libpmi2.so.0 => /lib64/libpmi2.so.0 (0x00007f36e69f9000)
          libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f36e67dd000)
          libgcc_s.so.1 => /opt/sw/packages/gcc/10.4/lib64/libgcc_s.so.1
(0x00007f36e65c5000)
          libquadmath.so.0 =>
/opt/sw/packages/gcc/10.4/lib64/libquadmath.so.0 (0x00007f36e637e000)
          /lib64/ld-linux-x86-64.so.2 (0x00007f36e8f40000)
          libz.so.1 => /lib64/libz.so.1 (0x00007f36e6168000)
          liblzma.so.5 => /lib64/liblzma.so.5 (0x00007f36e5f42000)
          libosmcomp.so.4 => /lib64/libosmcomp.so.4 (0x00007f36e5d33000)
          libnl-route-3.so.200 => /lib64/libnl-route-3.so.200
(0x00007f36e5ac6000)
          libnl-3.so.200 => /lib64/libnl-3.so.200 (0x00007f36e58a5000)
What can/should I do about this?
Sometimes I see (not clear what conditions trigger it, but I have at
least one set of output running one of the OSU benchmarks):
Please retry with MV2_LIBIBMAD_PATH=<path/to/libibmad.so>
It seems like what's suggested in the error message is not a great
idea/this should be dealt with at compile time.
This is my build script; relatively uncomplicated:
[novosirj at amarel-test2 build]$ more
~/src/build-mvapich2-2.3.7-1-gcc-10.4.sh #!/bin/sh
module purge
module load gcc/10.4
module list
export FFLAGS="-fallow-argument-mismatch"
../mvapich2-2.3.7-1/configure --with-pmi=pmi2 --with-pm=slurm
--prefix=/opt/sw/packages/gcc-10/mvapich2/2.3.7-1 && \
         make -j32 && make check && make install
And the configure process doesn't seem to point out anything amiss:
checking for the InfiniBand includes path... default
checking for the InfiniBand library path... default
checking for library containing shm_open... -lrt
checking infiniband/verbs.h usability... yes
checking infiniband/verbs.h presence... yes
checking for infiniband/verbs.h... yes
configure: checking checking for InfiniBand umad installation...
checking infiniband/umad.h usability... yes
checking infiniband/umad.h presence... yes
checking for infiniband/umad.h... yes
configure: InfiniBand libumad found
checking whether to enable hybrid communication channel... yes
configure: checking for RDMA CM support...
checking rdma/rdma_cma.h usability... yes
checking rdma/rdma_cma.h presence... yes
checking for rdma/rdma_cma.h... yes
configure: RDMA CM support enabled
configure: checking for hardware multicast support...
checking infiniband/mad.h usability... yes
checking infiniband/mad.h presence... yes
checking for infiniband/mad.h... yes
Thanks!
--
#BlackLivesMatter
____
  || \\UTGERS,     |----------------------*O*------------------------
  ||_// the State  |    Ryan Novosielski - novosirj at rutgers.edu
  || \\ University | Sr. Technologist - 973/972.0922 ~*~ RBHS Campus
  ||  \\    of NJ  | Office of Advanced Res. Comp. - MSB C630, Newark
_______________________________________________
Mvapich-discuss mailing list
Mvapich-discuss at lists.osu.edu
https://lists.osu.edu/mailman/listinfo/mvapich-discuss<https://lists.osu.edu/mailman/listinfo/mvapich-discuss>

--
#BlackLivesMatter
____
|| \\UTGERS,     |----------------------*O*------------------------
||_// the State  |    Ryan Novosielski - novosirj at rutgers.edu<mailto:novosirj at rutgers.edu>
|| \\ University | Sr. Technologist - 973/972.0922 ~*~ RBHS Campus
||  \\    of NJ  | Office of Advanced Res. Comp. - MSB C630, Newark
     `'
_______________________________________________
Mvapich-discuss mailing list
Mvapich-discuss at lists.osu.edu<mailto:Mvapich-discuss at lists.osu.edu>
https://lists.osu.edu/mailman/listinfo/mvapich-discuss

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osu.edu/pipermail/mvapich-discuss/attachments/20241010/4b7ab988/attachment-0002.html>

From bo18resch11002 at iith.ac.in  Mon Oct 14 01:41:41 2024
From: bo18resch11002 at iith.ac.in (Nitin Kulhar)
Date: Mon, 14 Oct 2024 11:11:41 +0530
Subject: [Mvapich-discuss] MVAPICH2-GDR-2.3.7: Wrapper Script Changes After
	'rpm2cpio' Extraction
Message-ID: <CAP1fmNzN8ejqDt_oo9=QSP5GQWYZcdWhtATtDw_bAc3deVWGuw@mail.gmail.com>

Dear all

I am installing mvapich2-gdr-2.3.7
in the user-space of our institute's
HPC facility from the rpm:
*mvapich2-gdr-cuda11.4.mofed5.4.gnu11.2.0-2.3.7-1.el8.x86_64.rpm*
How should I change the wrapper
scripts (mpicc, mpicxx, mpif77,
mpif90, mpifc) to render them fit
to compile Amber24 executables
with CUDA and/or MPI support?

I can load CUDA-11.4 from a
module available on the HPC,
and, I have a local installation
of gcc (11.2.0) in the user-space
under the $HOME tree.

I would be grateful to receive any
help in this regard.

Thanks and regards
Nitin

-- 


Disclaimer:- This?footer text is to convey that this email is sent by one 
of the?users of IITH. So, do not mark it as SPAM.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osu.edu/pipermail/mvapich-discuss/attachments/20241014/f5ca65d7/attachment-0002.html>

From mohil at cdac.in  Mon Oct 14 02:17:47 2024
From: mohil at cdac.in (mohil)
Date: Mon, 14 Oct 2024 11:47:47 +0530 (IST)
Subject: [Mvapich-discuss] Regarding Topology awareness in MVAPICH
Message-ID: <666329362.21946.1728886667184.JavaMail.open-xchange@webmail.cdac.in>

hello all,

Recently i am studying about topology awareness in mvapich-3.0, so hardware
topology detection part is clear that is managed using the hwloc library. But I
don`t get how the network topology detection work, as I get to know mvapich-3.0
doesn't use netloc for topology discovery. So, I want to know. If network
topology awareness is present in recent mvapich-3.0?
If yes, how does the network topology awareness work in mvapich-3.0 ? Where in
source code i can found more about the network topology detection and
optimisation done using that info?

I will be extremely thankful to anyone able to help me with this information.

With Regards
MOHIL
mohil at cdac.in
------------------------------------------------------------------------------------------------------------
[ C-DAC is on Social-Media too. Kindly follow us at:
Facebook: https://urldefense.com/v3/__https://www.facebook.com/CDACINDIA__;!!KGKeukY!wgpGvVROip1ylhtv2uDAGJaYcmB0ASt4-pHd3-FUaPu2zef-csw-4b6NfUPXA2HPCjj1HHEMGqLTUubTFhlOSDtW$  & Twitter: @cdacindia ]

This e-mail is for the sole use of the intended recipient(s) and may
contain confidential and privileged information. If you are not the
intended recipient, please contact the sender by reply e-mail and destroy
all copies and the original message. Any unauthorized review, use,
disclosure, dissemination, forwarding, printing or copying of this email
is strictly prohibited and appropriate legal action will be taken.
------------------------------------------------------------------------------------------------------------

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osu.edu/pipermail/mvapich-discuss/attachments/20241014/be7ec6e9/attachment-0002.html>

From bo18resch11002 at iith.ac.in  Mon Oct 14 10:52:14 2024
From: bo18resch11002 at iith.ac.in (Nitin Kulhar)
Date: Mon, 14 Oct 2024 20:22:14 +0530
Subject: [Mvapich-discuss] MVAPICH2-GDR-2.3.7: Wrapper Script Changes
	After 'rpm2cpio' Extraction
In-Reply-To: <CAP1fmNzN8ejqDt_oo9=QSP5GQWYZcdWhtATtDw_bAc3deVWGuw@mail.gmail.com>
References: <CAP1fmNzN8ejqDt_oo9=QSP5GQWYZcdWhtATtDw_bAc3deVWGuw@mail.gmail.com>
Message-ID: <CAP1fmNxVo4yNb++DWfGxY3HJsu818b3eL=MxX0HvFUjQqpZJAQ@mail.gmail.com>

Dear concerned
This is to bring to your attention
that there is a Subject/Body
mix-up between two threads on
the list. I am seeing the Body
of a different thread under the
Subject Line of my message.

Link to my message:
https://lists.osu.edu/pipermail/mvapich-discuss/2024-October/011769.html

On Mon, Oct 14, 2024 at 11:11?AM Nitin Kulhar <bo18resch11002 at iith.ac.in>
wrote:

> Dear all
>
> I am installing mvapich2-gdr-2.3.7
> in the user-space of our institute's
> HPC facility from the rpm:
> *mvapich2-gdr-cuda11.4.mofed5.4.gnu11.2.0-2.3.7-1.el8.x86_64.rpm*
> How should I change the wrapper
> scripts (mpicc, mpicxx, mpif77,
> mpif90, mpifc) to render them fit
> to compile Amber24 executables
> with CUDA and/or MPI support?
>
> I can load CUDA-11.4 from a
> module available on the HPC,
> and, I have a local installation
> of gcc (11.2.0) in the user-space
> under the $HOME tree.
>
> I would be grateful to receive any
> help in this regard.
>
> Thanks and regards
> Nitin
>

-- 


Disclaimer:- This?footer text is to convey that this email is sent by one 
of the?users of IITH. So, do not mark it as SPAM.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osu.edu/pipermail/mvapich-discuss/attachments/20241014/df782cca/attachment-0002.html>

From zyou at osc.edu  Wed Oct 16 11:50:19 2024
From: zyou at osc.edu (You, Zhi-Qiang)
Date: Wed, 16 Oct 2024 15:50:19 +0000
Subject: [Mvapich-discuss] Handling MVAPICH 3.0 full subscription warning
Message-ID: <BL0PR01MB50106BCFBFF1ED776D635375C6462@BL0PR01MB5010.prod.exchangelabs.com>

Hi,

We encountered the following warning message while running a full-node MPI job with MVAPICH 3.0:

[][mvp_generate_implicit_cpu_mapping] WARNING: You appear to be running at full subscription for this job. UCX spawns an additional thread for each process which may result in oversubscribed cores and poor performance. Please consider reserving at least 2 cores per node for the additional threads, enabling SMT, or setting MVP_THREADS_PER_PROCESS=2 to ensure that sufficient resources are available.

The suggestion to set MVP_THREADS_PER_PROCESS=2 not only fails to improve performance but actually degrades it. Can this warning message be safely ignored, or is there any action I need to take to address it?

Best,
ZQ

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osu.edu/pipermail/mvapich-discuss/attachments/20241016/16123335/attachment-0002.html>

From shineman.5 at osu.edu  Wed Oct 16 14:16:10 2024
From: shineman.5 at osu.edu (Shineman, Nat)
Date: Wed, 16 Oct 2024 18:16:10 +0000
Subject: [Mvapich-discuss] Handling MVAPICH 3.0 full subscription warning
In-Reply-To: <BL0PR01MB50106BCFBFF1ED776D635375C6462@BL0PR01MB5010.prod.exchangelabs.com>
References: <BL0PR01MB50106BCFBFF1ED776D635375C6462@BL0PR01MB5010.prod.exchangelabs.com>
Message-ID: <PH0PR01MB7382827442A6443DCCCB7A12C9462@PH0PR01MB7382.prod.exchangelabs.com>

Hi ZQ,

You are probably seeing degraded performance because you are still running the application at full subscription and requesting that MVAPICH reserve 2 cores per process. The warning should probably more accurately state that you should cap your runs at 1/2 subscription and set the listed environment variable. This would prevent you from oversubscribing cores.

However, if you are seeing satisfactory performance with oversubscribed cores in full subscription, please feel free to ignore the warning.

Thanks,
Nat
________________________________
From: Mvapich-discuss <mvapich-discuss-bounces at lists.osu.edu> on behalf of You, Zhi-Qiang via Mvapich-discuss <mvapich-discuss at lists.osu.edu>
Sent: Wednesday, October 16, 2024 11:50
To: mvapich-discuss at lists.osu.edu <mvapich-discuss at lists.osu.edu>
Subject: [Mvapich-discuss] Handling MVAPICH 3.0 full subscription warning


Hi,



We encountered the following warning message while running a full-node MPI job with MVAPICH 3.0:

[][mvp_generate_implicit_cpu_mapping] WARNING: You appear to be running at full subscription for this job. UCX spawns an additional thread for each process which may result in oversubscribed cores and poor performance. Please consider reserving at least 2 cores per node for the additional threads, enabling SMT, or setting MVP_THREADS_PER_PROCESS=2 to ensure that sufficient resources are available.

The suggestion to set MVP_THREADS_PER_PROCESS=2 not only fails to improve performance but actually degrades it. Can this warning message be safely ignored, or is there any action I need to take to address it?



Best,

ZQ


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osu.edu/pipermail/mvapich-discuss/attachments/20241016/0ed6b931/attachment-0002.html>

From simone at fnal.gov  Thu Oct 17 12:07:23 2024
From: simone at fnal.gov (James N Simone)
Date: Thu, 17 Oct 2024 16:07:23 +0000
Subject: [Mvapich-discuss] <no subject>
Message-ID: <09AC6A77-F2AA-4E28-981B-C3C2A71F5025@contoso.com>

!-------------------------------------------------------------------|
  This Message Is From an External Sender
  This message came from outside your organization.
|-------------------------------------------------------------------!

UNSUBSCRIBE







From zyou at osc.edu  Thu Oct 24 10:54:23 2024
From: zyou at osc.edu (You, Zhi-Qiang)
Date: Thu, 24 Oct 2024 14:54:23 +0000
Subject: [Mvapich-discuss] Inquiry Regarding IB Performance on New Cluster
 with NDR 400 and MVAPICH
Message-ID: <BL0PR01MB50107E49C536D6F5DE9FA074C64E2@BL0PR01MB5010.prod.exchangelabs.com>

Dear MVAPICH developers,

We recently tested IB performance on our new cluster at OSC. On other clusters, we are able to achieve the theoretical limits of the IB devices using omb_bibw. For example:


  *   NDR 200: 48 GB/s
  *   EDR 100: 24.5 GB/s

However, on the new cluster with NDR 400, the best performance I am able to achieve is around 67 GB/s. Does this number seem reasonable? I have tested with MVAPICH 3.0 and MVAPICH 2.3.7-1, both built with Intel 2021.10.0. The OMB versions used are 5.9 for MVAPICH 2.3.7-1 and 7.3 for MVAPICH 3.0. The parameters used for omb_bibw were ?-m 4194304:4194304 -x 1000 -i 10000?.

Best regards,
ZQ
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osu.edu/pipermail/mvapich-discuss/attachments/20241024/8ef3f284/attachment-0002.html>

From panda at cse.ohio-state.edu  Thu Oct 24 23:57:03 2024
From: panda at cse.ohio-state.edu (Panda, Dhabaleswar)
Date: Fri, 25 Oct 2024 03:57:03 +0000
Subject: [Mvapich-discuss] The MVAPICH Group is looking for a talented
 full-time software developer to join the team
Message-ID: <DM6PR01MB431583DFC4726D723712BE16DA4F2@DM6PR01MB4315.prod.exchangelabs.com>

More details on this position are available from:

https://osu.wd1.myworkdayjobs.com/OSUCareers/job/Columbus-Campus/Applications-Developer_R116828-1

If you are interested in this position and have authorization to work in the USA, please apply ASAP. 

Thanks, 

DK Panda
 



From zyou at osc.edu  Thu Oct 31 12:11:04 2024
From: zyou at osc.edu (You, Zhi-Qiang)
Date: Thu, 31 Oct 2024 16:11:04 +0000
Subject: [Mvapich-discuss] Handling MVAPICH 3.0 full subscription warning
In-Reply-To: <PH0PR01MB7382827442A6443DCCCB7A12C9462@PH0PR01MB7382.prod.exchangelabs.com>
References: <BL0PR01MB50106BCFBFF1ED776D635375C6462@BL0PR01MB5010.prod.exchangelabs.com>
	<PH0PR01MB7382827442A6443DCCCB7A12C9462@PH0PR01MB7382.prod.exchangelabs.com>
Message-ID: <BL0PR01MB501045D37526D8346BF24FAAC6552@BL0PR01MB5010.prod.exchangelabs.com>

Hi Nat,

Thank you for the suggestion. I have a few questions:


  1.  Does this message indicate that oversubscription is occurring, or is it simply a warning that appears every time a full-node job is run? In one user?s case, I did not observe any oversubscription, although the warning was present.
  2.  UCX is also the default for OpenMPI, but I did not see a similar warning when running a full-node job with OpenMPI. Why does this only happen with MVAPICH?

Thank you,
ZQ

From: Shineman, Nat <shineman.5 at osu.edu>
Date: Wednesday, October 16, 2024 at 2:16?PM
To: mvapich-discuss at lists.osu.edu <mvapich-discuss at lists.osu.edu>, You, Zhi-Qiang <zyou at osc.edu>
Subject: Re: Handling MVAPICH 3.0 full subscription warning
Hi ZQ,

You are probably seeing degraded performance because you are still running the application at full subscription and requesting that MVAPICH reserve 2 cores per process. The warning should probably more accurately state that you should cap your runs at 1/2 subscription and set the listed environment variable. This would prevent you from oversubscribing cores.

However, if you are seeing satisfactory performance with oversubscribed cores in full subscription, please feel free to ignore the warning.

Thanks,
Nat
________________________________
From: Mvapich-discuss <mvapich-discuss-bounces at lists.osu.edu> on behalf of You, Zhi-Qiang via Mvapich-discuss <mvapich-discuss at lists.osu.edu>
Sent: Wednesday, October 16, 2024 11:50
To: mvapich-discuss at lists.osu.edu <mvapich-discuss at lists.osu.edu>
Subject: [Mvapich-discuss] Handling MVAPICH 3.0 full subscription warning


Hi,



We encountered the following warning message while running a full-node MPI job with MVAPICH 3.0:

[][mvp_generate_implicit_cpu_mapping] WARNING: You appear to be running at full subscription for this job. UCX spawns an additional thread for each process which may result in oversubscribed cores and poor performance. Please consider reserving at least 2 cores per node for the additional threads, enabling SMT, or setting MVP_THREADS_PER_PROCESS=2 to ensure that sufficient resources are available.

The suggestion to set MVP_THREADS_PER_PROCESS=2 not only fails to improve performance but actually degrades it. Can this warning message be safely ignored, or is there any action I need to take to address it?



Best,

ZQ


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osu.edu/pipermail/mvapich-discuss/attachments/20241031/10492c20/attachment-0002.html>

From shineman.5 at osu.edu  Thu Oct 31 12:55:46 2024
From: shineman.5 at osu.edu (Shineman, Nat)
Date: Thu, 31 Oct 2024 16:55:46 +0000
Subject: [Mvapich-discuss] Handling MVAPICH 3.0 full subscription warning
In-Reply-To: <BL0PR01MB501045D37526D8346BF24FAAC6552@BL0PR01MB5010.prod.exchangelabs.com>
References: <BL0PR01MB50106BCFBFF1ED776D635375C6462@BL0PR01MB5010.prod.exchangelabs.com>
	<PH0PR01MB7382827442A6443DCCCB7A12C9462@PH0PR01MB7382.prod.exchangelabs.com>
	<BL0PR01MB501045D37526D8346BF24FAAC6552@BL0PR01MB5010.prod.exchangelabs.com>
Message-ID: <PH0PR01MB73829ECF926880542E3051E8C9552@PH0PR01MB7382.prod.exchangelabs.com>

ZQ,

This warning will appear anytime you allocate a full subscription job with the UCX netmod enabled. It is based on observed behaviour of UCX, which we have seen allocated an extra progress thread for each process that calls ucx_init. I cannot guarantee this will happen in every case, especially if you are using your own UCX version, but we have observed it in most if not all of our testing with UCX. If you are not seeing any performance impacts and can observe no oversubscription, then I would encourage you to disregard the warning and trust what you are observing.

Regarding your second question, the warning is coming from MVAPICH, so that is why it does not appear in OpenMPI. I cannot speak to how OpenMPI handles these additional UCX progress threads or what if anything they might do to notify a user.

Thanks,
Nat
________________________________
From: You, Zhi-Qiang <zyou at osc.edu>
Sent: Thursday, October 31, 2024 12:11
To: Shineman, Nat <shineman.5 at osu.edu>; mvapich-discuss at lists.osu.edu <mvapich-discuss at lists.osu.edu>
Subject: Re: Handling MVAPICH 3.0 full subscription warning


Hi Nat,



Thank you for the suggestion. I have a few questions:



  1.  Does this message indicate that oversubscription is occurring, or is it simply a warning that appears every time a full-node job is run? In one user?s case, I did not observe any oversubscription, although the warning was present.
  2.  UCX is also the default for OpenMPI, but I did not see a similar warning when running a full-node job with OpenMPI. Why does this only happen with MVAPICH?



Thank you,

ZQ



From: Shineman, Nat <shineman.5 at osu.edu>
Date: Wednesday, October 16, 2024 at 2:16?PM
To: mvapich-discuss at lists.osu.edu <mvapich-discuss at lists.osu.edu>, You, Zhi-Qiang <zyou at osc.edu>
Subject: Re: Handling MVAPICH 3.0 full subscription warning

Hi ZQ,



You are probably seeing degraded performance because you are still running the application at full subscription and requesting that MVAPICH reserve 2 cores per process. The warning should probably more accurately state that you should cap your runs at 1/2 subscription and set the listed environment variable. This would prevent you from oversubscribing cores.



However, if you are seeing satisfactory performance with oversubscribed cores in full subscription, please feel free to ignore the warning.



Thanks,

Nat

________________________________

From: Mvapich-discuss <mvapich-discuss-bounces at lists.osu.edu> on behalf of You, Zhi-Qiang via Mvapich-discuss <mvapich-discuss at lists.osu.edu>
Sent: Wednesday, October 16, 2024 11:50
To: mvapich-discuss at lists.osu.edu <mvapich-discuss at lists.osu.edu>
Subject: [Mvapich-discuss] Handling MVAPICH 3.0 full subscription warning



Hi,



We encountered the following warning message while running a full-node MPI job with MVAPICH 3.0:

[][mvp_generate_implicit_cpu_mapping] WARNING: You appear to be running at full subscription for this job. UCX spawns an additional thread for each process which may result in oversubscribed cores and poor performance. Please consider reserving at least 2 cores per node for the additional threads, enabling SMT, or setting MVP_THREADS_PER_PROCESS=2 to ensure that sufficient resources are available.

The suggestion to set MVP_THREADS_PER_PROCESS=2 not only fails to improve performance but actually degrades it. Can this warning message be safely ignored, or is there any action I need to take to address it?



Best,

ZQ


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osu.edu/pipermail/mvapich-discuss/attachments/20241031/3ffc943f/attachment-0002.html>