[Mvapich-discuss] OMB v7.1 - Build fails due to pointer-related issues in osu_util_mpi.c
Subramoni, Hari
subramoni.1 at osu.edu
Wed Jun 21 16:44:10 EDT 2023
Great 😊. Sorry for the trouble.
From: Kandes, Martin <mkandes at sdsc.edu>
Sent: Wednesday, June 21, 2023 3:06 PM
To: Subramoni, Hari <subramoni.1 at osu.edu>; Announcement about MVAPICH2 (MPI over InfiniBand, RoCE, Omni-Path, iWARP and EFA) Libraries developed at NBCL/OSU <mvapich-discuss at lists.osu.edu>; Kandes, Martin <mkandes at sdsc.edu>
Subject: Re: [Mvapich-discuss] OMB v7.1 - Build fails due to pointer-related issues in osu_util_mpi.c
Hari, Ah, yup. This is looking more promising now [3]. Marty [3] .. . make[3]: Leaving directory '/tmp/osu-micro-benchmarks-7. 1-1/c' make[2]: Leaving directory '/tmp/osu-micro-benchmarks-7. 1-1/c' make[1]: Leaving directory '/tmp/osu-micro-benchmarks-7. 1-1/c'
ZjQcmQRYFpfptBannerStart
This Message Is From an External Sender
This message came from outside your organization.
Report Suspicious <https://us-phishalarm-ewt.proofpoint.com/EWT/v1/KGKeukY!vwQd8KhtD6YA5Rdxn-4ldhGxZexbaOcEBZskZKAeKTCPmvbg4DQFpbPAErg14K9qkzCLMhOwMr7SP2v9uDyWGlsM8lDNucQczn2VSpRrxYgxYtZQuDatMuov-CHP4DVMKe-8MU7trHSHeg$>
ZjQcmQRYFpfptBannerEnd
Hari,
Ah, yup. This is looking more promising now [3].
Marty
[3]
...
make[3]: Leaving directory '/tmp/osu-micro-benchmarks-7.1-1/c'
make[2]: Leaving directory '/tmp/osu-micro-benchmarks-7.1-1/c'
make[1]: Leaving directory '/tmp/osu-micro-benchmarks-7.1-1/c'
make[1]: Entering directory '/tmp/osu-micro-benchmarks-7.1-1'
make[2]: Entering directory '/tmp/osu-micro-benchmarks-7.1-1'
make[2]: Nothing to be done for 'install-exec-am'.
make[2]: Nothing to be done for 'install-data-am'.
make[2]: Leaving directory '/tmp/osu-micro-benchmarks-7.1-1'
make[1]: Leaving directory '/tmp/osu-micro-benchmarks-7.1-1'
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
0 upgraded, 0 newly installed, 0 to remove and 3 not upgraded.
INFO: Adding labels
INFO: Adding environment to container
INFO: Adding runscript
INFO: Creating SIF file...
INFO: Build complete: omb-7.1-ubuntu-22.04-mlnx-ofed-5.8-2.0.3.0-openmpi-4.1.3.sif
ubuntu at mkandes-container-builder:~/naked-singularity/definition-files/omb$ singularity shell omb-7.1-ubuntu-22.04-mlnx-ofed-5.8-2.0.3.0-openmpi-4.1.3.sif
Singularity> ls /opt/omb/7.1-1/
collective one-sided pt2pt startup
Singularity> ls /opt/omb/7.1-1/collective/
osu_allgather osu_ialltoall osu_ireduce_scatter
osu_allgatherv osu_ialltoallv osu_iscatter
osu_allreduce osu_ialltoallw osu_iscatterv
osu_alltoall osu_ibarrier osu_neighbor_allgather
osu_alltoallv osu_ibcast osu_neighbor_allgatherv
osu_alltoallw osu_igather osu_neighbor_alltoall
osu_barrier osu_igatherv osu_neighbor_alltoallv
osu_bcast osu_ineighbor_allgather osu_neighbor_alltoallw
osu_gather osu_ineighbor_allgatherv osu_reduce
osu_gatherv osu_ineighbor_alltoall osu_reduce_scatter
osu_iallgather osu_ineighbor_alltoallv osu_scatter
osu_iallgatherv osu_ineighbor_alltoallw osu_scatterv
osu_iallreduce osu_ireduce
Singularity>
________________________________
From: Mvapich-discuss <mvapich-discuss-bounces+mkandes=sdsc.edu at lists.osu.edu<mailto:mvapich-discuss-bounces+mkandes=sdsc.edu at lists.osu.edu>> on behalf of Kandes, Martin via Mvapich-discuss <mvapich-discuss at lists.osu.edu<mailto:mvapich-discuss at lists.osu.edu>>
Sent: Wednesday, June 21, 2023 11:56 AM
To: Subramoni, Hari <subramoni.1 at osu.edu<mailto:subramoni.1 at osu.edu>>; Announcement about MVAPICH2 (MPI over InfiniBand, RoCE, Omni-Path, iWARP and EFA) Libraries developed at NBCL/OSU <mvapich-discuss at lists.osu.edu<mailto:mvapich-discuss at lists.osu.edu>>
Subject: Re: [Mvapich-discuss] OMB v7.1 - Build fails due to pointer-related issues in osu_util_mpi.c
Hi Hari, Was this a recent fix? This build issue was noticed on 6/17/23 [1] using the v7. 1 tarball. However, maybe now I see a '-1' added to the end of the tarball name [2], which was not included the name of the tarball I used. Marty [1] -rw-rw-r--
Hi Hari,
Was this a recent fix? This build issue was noticed on 6/17/23 [1] using the v7.1 tarball. However, maybe now I see a '-1' added to the end of the tarball name [2], which was not included the name of the tarball I used.
Marty
[1]
-rw-rw-r-- 1 ubuntu ubuntu 3.3K Jun 17 22:12 Singularity.omb-7.1-ubuntu-22.04-mlnx-ofed-5.8-2.0.3.0-openmpi-4.1.3
-rwxr-xr-x 1 ubuntu ubuntu 779M Jun 17 22:13 omb-7.1-ubuntu-22.04-mlnx-ofed-5.8-2.0.3.0-openmpi-4.1.3.sif
[2]
https://mvapich.cse.ohio-state.edu/download/mvapich/osu-micro-benchmarks-7.1-1.tar.gz<https://urldefense.com/v3/__https:/mvapich.cse.ohio-state.edu/download/mvapich/osu-micro-benchmarks-7.1-1.tar.gz__;!!Mih3wA!GzqfrGlwcCKwvoQPXUiOUgDWXX7CfXE5QDezjdGAYMeU5ecXpuU5RpljT1HGTKt-VtD7TX51aVqagQaMnv7IOiGdz91x3Yw$>
________________________________
From: Subramoni, Hari <subramoni.1 at osu.edu<mailto:subramoni.1 at osu.edu>>
Sent: Tuesday, June 20, 2023 5:39 AM
To: Kandes, Martin <mkandes at sdsc.edu<mailto:mkandes at sdsc.edu>>; Announcement about MVAPICH2 (MPI over InfiniBand, RoCE, Omni-Path, iWARP and EFA) Libraries developed at NBCL/OSU <mvapich-discuss at lists.osu.edu<mailto:mvapich-discuss at lists.osu.edu>>
Cc: Subramoni, Hari <subramoni.1 at osu.edu<mailto:subramoni.1 at osu.edu>>
Subject: RE: [Mvapich-discuss] OMB v7.1 - Build fails due to pointer-related issues in osu_util_mpi.c
Hi, Marty.
We have seen this issue in the past. It is because of a difference in the datatype objects being used by MPICH derivatives and OpenMPI. I believe we had fixed this issue and updated the tarball on the website with the fix.
Can you please let us know if the latest tarball fixes things for you?
Thx,
Hari.
From: Mvapich-discuss <mvapich-discuss-bounces+subramon=cse.ohio-state.edu at lists.osu.edu<mailto:mvapich-discuss-bounces+subramon=cse.ohio-state.edu at lists.osu.edu>> On Behalf Of Kandes, Martin via Mvapich-discuss
Sent: Saturday, June 17, 2023 6:35 PM
To: mvapich-discuss at lists.osu.edu<mailto:mvapich-discuss at lists.osu.edu>
Subject: [Mvapich-discuss] OMB v7.1 - Build fails due to pointer-related issues in osu_util_mpi.c
Hi all, Has anyone run into this build issue yet [1}? If not, I can post more details. No issue backing off to OMB v6. 2. Marty [1] .. . config. status: executing libtool commands Making all in c make[1]: Entering directory '/tmp/osu-micro-benchmarks-7. 1/c'
Hi all,
Has anyone run into this build issue yet [1}? If not, I can post more details. No issue backing off to OMB v6.2.
Marty
[1]
...
config.status: executing libtool commands
Making all in c
make[1]: Entering directory '/tmp/osu-micro-benchmarks-7.1/c'
Making all in mpi
make[2]: Entering directory '/tmp/osu-micro-benchmarks-7.1/c/mpi'
Making all in pt2pt
make[3]: Entering directory '/tmp/osu-micro-benchmarks-7.1/c/mpi/pt2pt'
Making all in standard
make[4]: Entering directory '/tmp/osu-micro-benchmarks-7.1/c/mpi/pt2pt/standard'
depbase=`echo osu_bibw.o | sed 's|[^/]*$|.deps/&|;s|\.o$||'`;\
mpicc -DPACKAGE_NAME=\"OSU-Micro-Benchmarks\" -DPACKAGE_TARNAME=\"osu-micro-benchmarks\" -DPACKAGE_VERSION=\"7.1\" -DPACKAGE_STRING=\"OSU-Micro-Benchmarks\ 7.1\" -DPACKAGE_BUGREPORT=\"mvapich-discuss at cse.ohio-state.edu\<mailto:mvapich-discuss at cse.ohio-state.edu\>" -DPACKAGE_URL=\"\" -DPACKAGE=\"osu-micro-benchmarks\" -DVERSION=\"7.1\" -DSTDC_HEADERS=1 -DHAVE_SYS_TYPES_H=1 -DHAVE_SYS_STAT_H=1 -DHAVE_STDLIB_H=1 -DHAVE_STRING_H=1 -DHAVE_MEMORY_H=1 -DHAVE_STRINGS_H=1 -DHAVE_INTTYPES_H=1 -DHAVE_STDINT_H=1 -DHAVE_UNISTD_H=1 -DHAVE_DLFCN_H=1 -DLT_OBJDIR=\".libs/\" -DHAVE_STDLIB_H=1 -DHAVE_STRING_H=1 -DHAVE_SYS_TIME_H=1 -DHAVE_UNISTD_H=1 -DHAVE_MATH_H=1 -DHAVE_GETPAGESIZE=1 -DHAVE_GETTIMEOFDAY=1 -DHAVE_MEMSET=1 -DHAVE_SQRT=1 -DFIELD_WIDTH=18 -DFLOAT_PRECISION=2 -I. -I../../../../c/util -g -O2 -MT osu_bibw.o -MD -MP -MF $depbase.Tpo -c -o osu_bibw.o osu_bibw.c &&\
mv -f $depbase.Tpo $depbase.Po
depbase=`echo ../../../util/osu_util.o | sed 's|[^/]*$|.deps/&|;s|\.o$||'`;\
mpicc -DPACKAGE_NAME=\"OSU-Micro-Benchmarks\" -DPACKAGE_TARNAME=\"osu-micro-benchmarks\" -DPACKAGE_VERSION=\"7.1\" -DPACKAGE_STRING=\"OSU-Micro-Benchmarks\ 7.1\" -DPACKAGE_BUGREPORT=\"mvapich-discuss at cse.ohio-state.edu\<mailto:mvapich-discuss at cse.ohio-state.edu\>" -DPACKAGE_URL=\"\" -DPACKAGE=\"osu-micro-benchmarks\" -DVERSION=\"7.1\" -DSTDC_HEADERS=1 -DHAVE_SYS_TYPES_H=1 -DHAVE_SYS_STAT_H=1 -DHAVE_STDLIB_H=1 -DHAVE_STRING_H=1 -DHAVE_MEMORY_H=1 -DHAVE_STRINGS_H=1 -DHAVE_INTTYPES_H=1 -DHAVE_STDINT_H=1 -DHAVE_UNISTD_H=1 -DHAVE_DLFCN_H=1 -DLT_OBJDIR=\".libs/\" -DHAVE_STDLIB_H=1 -DHAVE_STRING_H=1 -DHAVE_SYS_TIME_H=1 -DHAVE_UNISTD_H=1 -DHAVE_MATH_H=1 -DHAVE_GETPAGESIZE=1 -DHAVE_GETTIMEOFDAY=1 -DHAVE_MEMSET=1 -DHAVE_SQRT=1 -DFIELD_WIDTH=18 -DFLOAT_PRECISION=2 -I. -I../../../../c/util -g -O2 -MT ../../../util/osu_util.o -MD -MP -MF $depbase.Tpo -c -o ../../../util/osu_util.o ../../../util/osu_util.c &&\
mv -f $depbase.Tpo $depbase.Po
depbase=`echo ../../../util/osu_util_mpi.o | sed 's|[^/]*$|.deps/&|;s|\.o$||'`;\
mpicc -DPACKAGE_NAME=\"OSU-Micro-Benchmarks\" -DPACKAGE_TARNAME=\"osu-micro-benchmarks\" -DPACKAGE_VERSION=\"7.1\" -DPACKAGE_STRING=\"OSU-Micro-Benchmarks\ 7.1\" -DPACKAGE_BUGREPORT=\"mvapich-discuss at cse.ohio-state.edu\<mailto:mvapich-discuss at cse.ohio-state.edu\>" -DPACKAGE_URL=\"\" -DPACKAGE=\"osu-micro-benchmarks\" -DVERSION=\"7.1\" -DSTDC_HEADERS=1 -DHAVE_SYS_TYPES_H=1 -DHAVE_SYS_STAT_H=1 -DHAVE_STDLIB_H=1 -DHAVE_STRING_H=1 -DHAVE_MEMORY_H=1 -DHAVE_STRINGS_H=1 -DHAVE_INTTYPES_H=1 -DHAVE_STDINT_H=1 -DHAVE_UNISTD_H=1 -DHAVE_DLFCN_H=1 -DLT_OBJDIR=\".libs/\" -DHAVE_STDLIB_H=1 -DHAVE_STRING_H=1 -DHAVE_SYS_TIME_H=1 -DHAVE_UNISTD_H=1 -DHAVE_MATH_H=1 -DHAVE_GETPAGESIZE=1 -DHAVE_GETTIMEOFDAY=1 -DHAVE_MEMSET=1 -DHAVE_SQRT=1 -DFIELD_WIDTH=18 -DFLOAT_PRECISION=2 -I. -I../../../../c/util -g -O2 -MT ../../../util/osu_util_mpi.o -MD -MP -MF $depbase.Tpo -c -o ../../../util/osu_util_mpi.o ../../../util/osu_util_mpi.c &&\
mv -f $depbase.Tpo $depbase.Po
../../../util/osu_util_mpi.c: In function 'omb_assign_to_type':
../../../util/osu_util_mpi.c:1411:13: error: switch quantity not an integer
1411 | switch (dtype) {
| ^~~~~
../../../util/osu_util_mpi.c:1412:9: error: pointers are not permitted as case values
1412 | case MPI_CHAR:
| ^~~~
../../../util/osu_util_mpi.c:1416:9: error: pointers are not permitted as case values
1416 | case MPI_INT:
| ^~~~
../../../util/osu_util_mpi.c:1419:9: error: pointers are not permitted as case values
1419 | case MPI_FLOAT:
| ^~~~
../../../util/osu_util_mpi.c: In function 'omb_get_num_elements':
../../../util/osu_util_mpi.c:1430:13: error: switch quantity not an integer
1430 | switch (dtype) {
| ^~~~~
../../../util/osu_util_mpi.c:1431:9: error: pointers are not permitted as case values
1431 | case MPI_CHAR:
| ^~~~
../../../util/osu_util_mpi.c:1434:9: error: pointers are not permitted as case values
1434 | case MPI_FLOAT:
| ^~~~
../../../util/osu_util_mpi.c:1437:9: error: pointers are not permitted as case values
1437 | case MPI_INT:
| ^~~~
../../../util/osu_util_mpi.c: In function 'omb_validate_neighbour_col':
../../../util/osu_util_mpi.c:1970:21: error: switch quantity not an integer
1970 | switch (dtype) {
| ^~~~~
../../../util/osu_util_mpi.c:1971:17: error: pointers are not permitted as case values
1971 | case MPI_CHAR:
| ^~~~
../../../util/osu_util_mpi.c:1980:17: error: pointers are not permitted as case values
1980 | case MPI_INT:
| ^~~~
../../../util/osu_util_mpi.c:1988:17: error: pointers are not permitted as case values
1988 | case MPI_FLOAT:
| ^~~~
../../../util/osu_util_mpi.c: In function 'validate_reduce_scatter':
../../../util/osu_util_mpi.c:2039:17: error: switch quantity not an integer
2039 | switch (dtype) {
| ^~~~~
../../../util/osu_util_mpi.c:2040:13: error: pointers are not permitted as case values
2040 | case MPI_CHAR:
| ^~~~
../../../util/osu_util_mpi.c:2046:13: error: pointers are not permitted as case values
2046 | case MPI_INT:
| ^~~~
../../../util/osu_util_mpi.c:2052:13: error: pointers are not permitted as case values
2052 | case MPI_FLOAT:
| ^~~~
make[4]: *** [Makefile:582: ../../../util/osu_util_mpi.o] Error 1
make[4]: Leaving directory '/tmp/osu-micro-benchmarks-7.1/c/mpi/pt2pt/standard'
make[3]: *** [Makefile:353: all-recursive] Error 1
make[3]: Leaving directory '/tmp/osu-micro-benchmarks-7.1/c/mpi/pt2pt'
make[2]: *** [Makefile:354: all-recursive] Error 1
make[2]: Leaving directory '/tmp/osu-micro-benchmarks-7.1/c/mpi'
make[1]: *** [Makefile:428: all-recursive] Error 1
make[1]: Leaving directory '/tmp/osu-micro-benchmarks-7.1/c'
make: *** [Makefile:382: all-recursive] Error 1
Making install in c
make[1]: Entering directory '/tmp/osu-micro-benchmarks-7.1/c'
Making install in mpi
make[2]: Entering directory '/tmp/osu-micro-benchmarks-7.1/c/mpi'
Making install in pt2pt
make[3]: Entering directory '/tmp/osu-micro-benchmarks-7.1/c/mpi/pt2pt'
Making install in standard
make[4]: Entering directory '/tmp/osu-micro-benchmarks-7.1/c/mpi/pt2pt/standard'
depbase=`echo ../../../util/osu_util_mpi.o | sed 's|[^/]*$|.deps/&|;s|\.o$||'`;\
mpicc -DPACKAGE_NAME=\"OSU-Micro-Benchmarks\" -DPACKAGE_TARNAME=\"osu-micro-benchmarks\" -DPACKAGE_VERSION=\"7.1\" -DPACKAGE_STRING=\"OSU-Micro-Benchmarks\ 7.1\" -DPACKAGE_BUGREPORT=\"mvapich-discuss at cse.ohio-state.edu\<mailto:mvapich-discuss at cse.ohio-state.edu\>" -DPACKAGE_URL=\"\" -DPACKAGE=\"osu-micro-benchmarks\" -DVERSION=\"7.1\" -DSTDC_HEADERS=1 -DHAVE_SYS_TYPES_H=1 -DHAVE_SYS_STAT_H=1 -DHAVE_STDLIB_H=1 -DHAVE_STRING_H=1 -DHAVE_MEMORY_H=1 -DHAVE_STRINGS_H=1 -DHAVE_INTTYPES_H=1 -DHAVE_STDINT_H=1 -DHAVE_UNISTD_H=1 -DHAVE_DLFCN_H=1 -DLT_OBJDIR=\".libs/\" -DHAVE_STDLIB_H=1 -DHAVE_STRING_H=1 -DHAVE_SYS_TIME_H=1 -DHAVE_UNISTD_H=1 -DHAVE_MATH_H=1 -DHAVE_GETPAGESIZE=1 -DHAVE_GETTIMEOFDAY=1 -DHAVE_MEMSET=1 -DHAVE_SQRT=1 -DFIELD_WIDTH=18 -DFLOAT_PRECISION=2 -I. -I../../../../c/util -g -O2 -MT ../../../util/osu_util_mpi.o -MD -MP -MF $depbase.Tpo -c -o ../../../util/osu_util_mpi.o ../../../util/osu_util_mpi.c &&\
mv -f $depbase.Tpo $depbase.Po
../../../util/osu_util_mpi.c: In function 'omb_assign_to_type':
../../../util/osu_util_mpi.c:1411:13: error: switch quantity not an integer
1411 | switch (dtype) {
| ^~~~~
../../../util/osu_util_mpi.c:1412:9: error: pointers are not permitted as case values
1412 | case MPI_CHAR:
| ^~~~
../../../util/osu_util_mpi.c:1416:9: error: pointers are not permitted as case values
1416 | case MPI_INT:
| ^~~~
../../../util/osu_util_mpi.c:1419:9: error: pointers are not permitted as case values
1419 | case MPI_FLOAT:
| ^~~~
../../../util/osu_util_mpi.c: In function 'omb_get_num_elements':
../../../util/osu_util_mpi.c:1430:13: error: switch quantity not an integer
1430 | switch (dtype) {
| ^~~~~
../../../util/osu_util_mpi.c:1431:9: error: pointers are not permitted as case values
1431 | case MPI_CHAR:
| ^~~~
../../../util/osu_util_mpi.c:1434:9: error: pointers are not permitted as case values
1434 | case MPI_FLOAT:
| ^~~~
../../../util/osu_util_mpi.c:1437:9: error: pointers are not permitted as case values
1437 | case MPI_INT:
| ^~~~
../../../util/osu_util_mpi.c: In function 'omb_validate_neighbour_col':
../../../util/osu_util_mpi.c:1970:21: error: switch quantity not an integer
1970 | switch (dtype) {
| ^~~~~
../../../util/osu_util_mpi.c:1971:17: error: pointers are not permitted as case values
1971 | case MPI_CHAR:
| ^~~~
../../../util/osu_util_mpi.c:1980:17: error: pointers are not permitted as case values
1980 | case MPI_INT:
| ^~~~
../../../util/osu_util_mpi.c:1988:17: error: pointers are not permitted as case values
1988 | case MPI_FLOAT:
| ^~~~
../../../util/osu_util_mpi.c: In function 'validate_reduce_scatter':
../../../util/osu_util_mpi.c:2039:17: error: switch quantity not an integer
2039 | switch (dtype) {
| ^~~~~
../../../util/osu_util_mpi.c:2040:13: error: pointers are not permitted as case values
2040 | case MPI_CHAR:
| ^~~~
../../../util/osu_util_mpi.c:2046:13: error: pointers are not permitted as case values
2046 | case MPI_INT:
| ^~~~
../../../util/osu_util_mpi.c:2052:13: error: pointers are not permitted as case values
2052 | case MPI_FLOAT:
| ^~~~
make[4]: *** [Makefile:582: ../../../util/osu_util_mpi.o] Error 1
make[4]: Leaving directory '/tmp/osu-micro-benchmarks-7.1/c/mpi/pt2pt/standard'
make[3]: *** [Makefile:353: install-recursive] Error 1
make[3]: Leaving directory '/tmp/osu-micro-benchmarks-7.1/c/mpi/pt2pt'
make[2]: *** [Makefile:354: install-recursive] Error 1
make[2]: Leaving directory '/tmp/osu-micro-benchmarks-7.1/c/mpi'
make[1]: *** [Makefile:428: install-recursive] Error 1
make[1]: Leaving directory '/tmp/osu-micro-benchmarks-7.1/c'
make: *** [Makefile:382: install-recursive] Error 1
/.post.script: line 50: cd: /opt/omb/7.1: No such file or directory
mv: cannot stat 'libexec/osu-micro-benchmarks/mpi/collective': No such file or directory
mv: cannot stat 'libexec/osu-micro-benchmarks/mpi/one-sided': No such file or directory
mv: cannot stat 'libexec/osu-micro-benchmarks/mpi/pt2pt': No such file or directory
mv: cannot stat 'libexec/osu-micro-benchmarks/mpi/startup': No such file or directory
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
0 upgraded, 0 newly installed, 0 to remove and 3 not upgraded.
INFO: Adding labels
INFO: Adding environment to container
INFO: Adding runscript
INFO: Creating SIF file...
INFO: Build complete: omb-7.1-ubuntu-22.04-openmpi-4.1.3.sif
ubuntu at mkandes-container-builder:~/naked-singularity/definition-files/omb$
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osu.edu/pipermail/mvapich-discuss/attachments/20230621/8cb21afe/attachment-0005.html>
More information about the Mvapich-discuss
mailing list