[Mvapich-discuss] Why are all RANK=0 when I'm using MVAPICH?

Fri Nov 14 13:53:10 EST 2025

Greeting community,

I'm building mvapich with spack and trying to use it in production use. The build is fairly straightforward:

Concretized
--------------------------------
[+]  mvapich at 3.0%gcc at 12.3.0~alloca~cuda~debug+regcache+wrapperrpath<mailto:mvapich at 3.0%25gcc at 12.3.0~alloca~cuda~debug+regcache+wrapperrpath> build_system=autotools ch3_rank_bits=32 file_systems=auto netmod=ucx pmi_version=simple process_managers=slurm threads=multiple arch=linux-ubuntu22.04-zen2
[e]      ^bison at 3.8.2%gcc at 12.3.0~color<mailto:%5ebison at 3.8.2%25gcc at 12.3.0~color> build_system=autotools arch=linux-ubuntu22.04-zen2
[e]      ^findutils at 4.8.0%gcc at 12.3.0<mailto:%5efindutils at 4.8.0%25gcc at 12.3.0> build_system=autotools patches=440b954 arch=linux-ubuntu22.04-zen2
[+]      ^gcc-runtime at 12.3.0%gcc at 12.3.0<mailto:%5egcc-runtime at 12.3.0%25gcc at 12.3.0> build_system=generic arch=linux-ubuntu22.04-zen2
[e]      ^glibc at 2.35%gcc at 12.3.0<mailto:%5eglibc at 2.35%25gcc at 12.3.0> build_system=autotools arch=linux-ubuntu22.04-zen2
[e]      ^gmake at 4.3%gcc at 12.3.0~guile<mailto:%5egmake at 4.3%25gcc at 12.3.0~guile> build_system=generic patches=599f134 arch=linux-ubuntu22.04-zen2
[+]      ^libpciaccess at 0.17%gcc at 12.3.0<mailto:%5elibpciaccess at 0.17%25gcc at 12.3.0> build_system=autotools arch=linux-ubuntu22.04-zen2
[e]          ^libtool at 2.4.6%gcc at 12.3.0<mailto:%5elibtool at 2.4.6%25gcc at 12.3.0> build_system=autotools arch=linux-ubuntu22.04-zen2
[+]          ^util-macros at 1.19.3%gcc at 12.3.0<mailto:%5eutil-macros at 1.19.3%25gcc at 12.3.0> build_system=autotools arch=linux-ubuntu22.04-zen2
[+]      ^libxml2 at 2.10.3%gcc at 12.3.0+pic~python+shared<mailto:%5elibxml2 at 2.10.3%25gcc at 12.3.0+pic~python+shared> build_system=autotools arch=linux-ubuntu22.04-zen2
[+]          ^xz at 5.4.6%gcc at 12.3.0~pic<mailto:%5exz at 5.4.6%25gcc at 12.3.0~pic> build_system=autotools libs=shared,static arch=linux-ubuntu22.04-zen2
[+]      ^pkgconf at 2.2.0%gcc at 12.3.0<mailto:%5epkgconf at 2.2.0%25gcc at 12.3.0> build_system=autotools arch=linux-ubuntu22.04-zen2
[e]      ^slurm at 24.05.7%gcc at 12.3.0~cgroup~gtk~hdf5~hwloc~mariadb~nvml~pam~pmix+readline~restd~rsmi<mailto:%5eslurm at 24.05.7%25gcc at 12.3.0~cgroup~gtk~hdf5~hwloc~mariadb~nvml~pam~pmix+readline~restd~rsmi> build_system=autotools sysconfdir=PREFIX/etc arch=linux-ubuntu22.04-zen2
[e]      ^ucx at 1.18%gcc at 12.3.0~assertions~backtrace_detail~cma~cuda~dc~debug~dm+examples~gdrcopy~gtest~ib_hw_tm~java~knem~logging~mlx5_dv+openmp+optimizations~parameter_checking+pic~rc~rdmacm~rocm~thread_multiple~ucg~ud~verbs~vfs~xpmem<mailto:%5eucx at 1.18%25gcc at 12.3.0~assertions~backtrace_detail~cma~cuda~dc~debug~dm+examples~gdrcopy~gtest~ib_hw_tm~java~knem~logging~mlx5_dv+openmp+optimizations~parameter_checking+pic~rc~rdmacm~rocm~thread_multiple~ucg~ud~verbs~vfs~xpmem> build_system=autotools libs=shared,static opt=3 simd=auto arch=linux-ubuntu22.04-zen2
[+]      ^zlib-ng at 2.1.6%gcc at 12.3.0+compat+new_strategies+opt+pic+shared<mailto:%5ezlib-ng at 2.1.6%25gcc at 12.3.0+compat+new_strategies+opt+pic+shared> build_system=autotools arch=linux-ubuntu22.04-zen2

And the application program is a small test case:
#include <stdio.h>
#include <mpi.h>
#include <zlib.h>
#include <unistd.h>
#include <errno.h>
#include <string.h>

int main(int argc, char **argv) {
  int rank;
  char hostname[256];

  MPI_Init(&argc, &argv);

  MPI_Comm_rank(MPI_COMM_WORLD, &rank);

  gethostname(hostname, sizeof(hostname));
  printf("Hello world from rank %d on %s\n", rank,hostname);

  if (rank == 0) {
    printf("zlib version: %s\n", ZLIB_VERSION);
/*    printf("zlib-ng version: %s\n", ZLIBNG_VERSION); */
  }

  MPI_Finalize();
}

The usual answer is that I might be using 2 different mpi packages (i.e. openmpi + mvapich), checked that and it does not appear to be the case:

chris at augie:~$ ldd ./a.out
        linux-vdso.so.1 (0x00007ffe0e28d000)
        libmpi.so.12 => /mnt/beegfs/home/spack/spack/opt/spack/linux-ubuntu22.04-zen2/gcc-12.3.0/mvapich-3.0-cosed2xwznawlvn3khfemwdzbcjozp3d/lib/libmpi.so.12 (0x00001491c9f59000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00001491c9d19000)
        libpciaccess.so.0 => /mnt/beegfs/home/spack/spack/opt/spack/linux-ubuntu22.04-zen2/gcc-12.3.0/libpciaccess-0.17-fqdfrz6bxnslv5prv6iok55bz3i4oupu/lib/libpciaccess.so.0 (0x00001491c9d0b000)
        libxml2.so.2 => /mnt/beegfs/home/spack/spack/opt/spack/linux-ubuntu22.04-zen2/gcc-12.3.0/libxml2-2.10.3-u6g3r33et3pxq62z4jjyypv5goormddr/lib/libxml2.so.2 (0x00001491c9b9a000)
        libucp.so.0 => /usr/lib/libucp.so.0 (0x00001491c9ac5000)
        libucs.so.0 => /usr/lib/libucs.so.0 (0x00001491c9a62000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00001491c997b000)
        /lib64/ld-linux-x86-64.so.2 (0x00001491cb068000)
        libz.so.1 => /mnt/beegfs/home/spack/spack/opt/spack/linux-ubuntu22.04-zen2/gcc-12.3.0/zlib-ng-2.1.6-lzzj5kvgu4zn72nruzsbnod554swe4rb/lib/libz.so.1 (0x00001491c9952000)
        liblzma.so.5 => /mnt/beegfs/home/spack/spack/opt/spack/linux-ubuntu22.04-zen2/gcc-12.3.0/xz-5.4.6-eusbwoeb7he7a3lyizkglevxnotl6d3x/lib/liblzma.so.5 (0x00001491c9922000)
        libuct.so.0 => /usr/lib/libuct.so.0 (0x00001491c98e5000)
        libucm.so.0 => /usr/lib/libucm.so.0 (0x00001491c98ca000)

chris at augie:~$ ldd /mnt/beegfs/home/spack/spack/opt/spack/linux-ubuntu22.04-zen2/gcc-12.3.0/mvapich-3.0-cosed2xwznawlvn3khfemwdzbcjozp3d/bin/mpiexec
        linux-vdso.so.1 (0x00007ffd9010c000)
        liblz4.so.1 => /lib/x86_64-linux-gnu/liblz4.so.1 (0x000014ecd44ee000)
        libslurmfull.so => /usr/lib/x86_64-linux-gnu/slurm/libslurmfull.so (0x000014ecd42c8000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x000014ecd409f000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x000014ecd3fb8000)
        libresolv.so.2 => /lib/x86_64-linux-gnu/libresolv.so.2 (0x000014ecd3fa4000)
        /lib64/ld-linux-x86-64.so.2 (0x000014ecd4550000)

So what am I missing?

Christopher A. Washburn
Villanova University
Research Computing Administrator
Villanova Research
800 Lancaster Avenue
Villanova, Pennsylvania 19085
Phone: 610 519-4711
Cell: 484 431-6619
Christopher.washburn at villanova.edu

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osu.edu/pipermail/mvapich-discuss/attachments/20251114/b79c32c9/attachment-0001.html>