From subramoni.1 at osu.edu Mon Jul 10 18:34:27 2023 From: subramoni.1 at osu.edu (Subramoni, Hari) Date: Mon, 10 Jul 2023 22:34:27 +0000 Subject: [Mvapich] Announcing the release of OSU Micro-Benchmarks (OMB) 7.2 Message-ID: The MVAPICH team is pleased to announce the release of OSU Micro-Benchmarks (OMB) 7.2. This release introduces support for MPI Sessions for various benchmarks. It also adds support for MPI_IN_PLACE and an option to rotate the root for various collective benchmarks. Please note that OMB is also available through the Spack package manager. Now the system administrators and users of OSU Micro-Benchmarks (OMB) will be able to install these libraries on their systems using Spack. The new features, enhancements, and bug fixes for OSU Micro-Benchmarks (OMB) 7.2 are listed here: * New Features & Enhancements (since 7.1) - Add MPI-4 sessions based initialization support to following benchmarks * Point-to-point benchmarks supported * osu_bibw, osu_bw, osu_mbw_mr, osu_latency, osu_multi_lat, * osu_latency_mp, osu_latency_mt, osu_bw_persistent, * osu_bibw_persistent, osu_latency_persistent * Blocking benchmarks supported * osu_allgather, osu_allgatherv, osu_alltoall, osu_allreduce, * osu_alltoallv, osu_alltoallw, osu_bcast, osu_barrier, osu_gather, * osu_gatherv, osu_reduce, osu_reduce_scatter, osu_scatter, * osu_scatterv * Non-Blocking benchmarks supported * osu_iallgather, osu_iallgatherv, osu_iallreduce, osu_ialltoall, * osu_ialltoallv, osu_ialltoallw, osu_ibcast, osu_ibarrier, * osu_igather, osu_igatherv, osu_ireduce, osu_iscatter, * osu_iscatterv, osu_ireduce_scatter * Neighborhood benchmarks * osu_neighbor_allgather, osu_neighbor_allgatherv, * osu_neighbor_alltoall,osu_neighbor_alltoallv, * osu_neighbor_alltoallw, osu_ineighbor_allgatherv, * osu_ineighbor_allgatherv, osu_ineighbor_alltoall, * osu_ineighbor_alltoallv, osu_ineighbor_alltoallw * Startup benchmarks * osu_init - Add MPI_IN_PLACE support for following blocking and non-blocking collectives * Blocking benchmarks supported * osu_allgather, osu_allgatherv, osu_alltoall, osu_allreduce, * osu_alltoallv, osu_alltoallw, osu_gather, osu_gatherv, osu_reduce, * osu_reduce_scatter, osu_scatter, osu_scatterv * Non-Blocking benchmarks supported * osu_iallgather, osu_iallgatherv, osu_iallreduce, osu_ialltoall, * osu_ialltoallv, osu_ialltoallw, osu_igather, osu_igatherv, * osu_ireduce, osu_iscatter, osu_iscatterv, osu_ireduce_scatter - Add an option to set root rank for rooted blocking and non-blocking collectives * Blocking benchmarks supported * osu_gather, osu_gatherv, osu_reduce, osu_scatter, * osu_scatterv * Non-Blocking benchmarks supported * osu_igather, osu_igatherv, osu_ireduce, osu_iscatter, * osu_iscatterv * Bug Fixes - Fixed memory leak in point-to-point benchmarks when validation is enabled. * Thanks to Shi Jin @Amazon for report and patch. - Fixed missing '#' formatting bug in osu_ibarrier header. * Thanks to Nick Hagerty @ORNL for report. For downloading OMB 7.2 and associated README instructions, please visit the following URL: http://mvapich.cse.ohio-state.edu All questions, feedback, bug reports, hints for performance tuning, patches, and enhancements are welcome. Please post it to the mvapich-discuss mailing list (mvapich-discuss at lists.osu.edu). Thanks, The MVAPICH Team PS: We are also happy to inform you that the number of organizations using MVAPICH2 libraries (and registered at the MVAPICH site) has crossed 3,325 worldwide (in 90 countries). The number of downloads from the MVAPICH site has crossed 1,689,000 (1.68 million). The MVAPICH team would like to thank all its users and organizations!! -------------- next part -------------- An HTML attachment was scrubbed... URL: From rlei at vols.utk.edu Tue Jul 11 10:00:36 2023 From: rlei at vols.utk.edu (Lei, Ryan) Date: Tue, 11 Jul 2023 10:00:36 -0400 Subject: [Mvapich] Fwd: Mvapich2-2.3.7 issue In-Reply-To: References: Message-ID: ---------- Forwarded message --------- From: Lei, Ryan Date: Mon, Jul 10, 2023 at 9:53?AM Subject: Mvapich2-2.3.7 issue To: mvapich-discuss at lists.osu.edu Hello Sir/Madam, I am facing some mpirun error when I try to run the code based on my own cluster. I have 2 jetson nano as slave node and one jetson orin as master node, each have installed the software correctly. But there is an issue that when I do "mpirun -n # -hosts ip(master),ip(slave)./code" , everything works successfully and correctly. However, when I try only work on the slave node, take out the master ip in the -hosts, everything will fail and have error shows below: [image: error.jpg] Even if I try to fix the firewalls based on the command, it still is not working. MayI ask why and how do I fix it? Ps: more detailed output provide by .txt. Hope to hear from you soon. sincerely, Ryan Lei -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: error.jpg Type: image/jpeg Size: 261935 bytes Desc: not available URL: -------------- next part -------------- host: 192.168.2.10 host: 192.168.2.11 ================================================================================================== mpiexec options: ---------------- Base path: /usr/local/bin/ Launcher: (null) Debug level: 1 Enable X: -1 Global environment: ------------------- SHELL=/bin/bash SESSION_MANAGER=local/reuOrin-desktop:@/tmp/.ICE-unix/3043,unix/reuOrin-desktop:/tmp/.ICE-unix/3043 QT_ACCESSIBILITY=1 COLORTERM=truecolor XDG_CONFIG_DIRS=/etc/xdg/xdg-ubuntu:/etc/xdg XDG_MENU_PREFIX=gnome- GNOME_DESKTOP_SESSION_ID=this-is-deprecated GNOME_SHELL_SESSION_MODE=ubuntu SSH_AUTH_SOCK=/run/user/1000/keyring/ssh XMODIFIERS=@im=ibus DESKTOP_SESSION=ubuntu SSH_AGENT_PID=3007 GTK_MODULES=gail:atk-bridge PWD=/home/reu LOGNAME=reu XDG_SESSION_DESKTOP=ubuntu XDG_SESSION_TYPE=x11 GPG_AGENT_INFO=/run/user/1000/gnupg/S.gpg-agent:0:1 XAUTHORITY=/run/user/1000/gdm/Xauthority GJS_DEBUG_TOPICS=JS ERROR;JS LOG WINDOWPATH=2 HOME=/home/reu USERNAME=reu IM_CONFIG_PHASE=1 LANG=en_US.UTF-8 LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=00:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.zst=01;31:*.tzst=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.wim=01;31:*.swm=01;31:*.dwm=01;31:*.esd=01;31:*.jpg=01;35:*.jpeg=01;35:*.mjpg=01;35:*.mjpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.oga=00;36:*.opus=00;36:*.spx=00;36:*.xspf=00;36: XDG_CURRENT_DESKTOP=ubuntu:GNOME VTE_VERSION=6003 GNOME_TERMINAL_SCREEN=/org/gnome/Terminal/screen/59a28a14_f606_4ca7_bd10_beeeddc46a21 INVOCATION_ID=877fc76f797f4911acd1380fe960344b MANAGERPID=2830 GJS_DEBUG_OUTPUT=stderr LESSCLOSE=/usr/bin/lesspipe %s %s XDG_SESSION_CLASS=user TERM=xterm-256color LESSOPEN=| /usr/bin/lesspipe %s USER=reu GNOME_TERMINAL_SERVICE=:1.102 SHLVL=1 QT_IM_MODULE=ibus LD_LIBRARY_PATH=/usr/local/cuda/lib64: XDG_RUNTIME_DIR=/run/user/1000 CUDA_HOME=/usr/local/cuda JOURNAL_STREAM=7:58083 XDG_DATA_DIRS=/usr/share/ubuntu:/usr/local/share/:/usr/share/:/var/lib/snapd/desktop PATH=/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin GDMSESSION=ubuntu DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1000/bus _=/usr/local/bin/mpirun Hydra internal environment: --------------------------- GFORTRAN_UNBUFFERED_PRECONNECTED=y Proxy information: ********************* [1] proxy: 192.168.2.10 (1 cores) Exec list: ./hello (1 processes); ./hello (1 processes); ./hello (1 processes); ./hello (1 processes); ./hello (1 processes); [2] proxy: 192.168.2.11 (1 cores) Exec list: ./hello (1 processes); ./hello (1 processes); ./hello (1 processes); ./hello (1 processes); ./hello (1 processes); ================================================================================================== [mpiexec at reuOrin-desktop] Timeout set to -1 (-1 means infinite) [mpiexec at reuOrin-desktop] Got a control port string of reuOrin-desktop:40197 Proxy launch args: /usr/local/bin/hydra_pmi_proxy --control-port reuOrin-desktop:40197 --debug --rmk user --launcher ssh --demux poll --pgid 0 --retries 10 --usize -2 --proxy-id Arguments being passed to proxy 0: --version 3.2.1 --iface-ip-env-name MPIR_CVAR_CH3_INTERFACE_HOSTNAME --hostname 192.168.2.10 --global-core-map 0,1,2 --pmi-id-map 0,0 --global-process-count 10 --auto-cleanup 1 --pmi-kvsname kvs_25221_0 --pmi-process-mapping (vector,(0,2,1)) --ckpoint-num -1 --global-inherited-env 49 'SHELL=/bin/bash' 'SESSION_MANAGER=local/reuOrin-desktop:@/tmp/.ICE-unix/3043,unix/reuOrin-desktop:/tmp/.ICE-unix/3043' 'QT_ACCESSIBILITY=1' 'COLORTERM=truecolor' 'XDG_CONFIG_DIRS=/etc/xdg/xdg-ubuntu:/etc/xdg' 'XDG_MENU_PREFIX=gnome-' 'GNOME_DESKTOP_SESSION_ID=this-is-deprecated' 'GNOME_SHELL_SESSION_MODE=ubuntu' 'SSH_AUTH_SOCK=/run/user/1000/keyring/ssh' 'XMODIFIERS=@im=ibus' 'DESKTOP_SESSION=ubuntu' 'SSH_AGENT_PID=3007' 'GTK_MODULES=gail:atk-bridge' 'PWD=/home/reu' 'LOGNAME=reu' 'XDG_SESSION_DESKTOP=ubuntu' 'XDG_SESSION_TYPE=x11' 'GPG_AGENT_INFO=/run/user/1000/gnupg/S.gpg-agent:0:1' 'XAUTHORITY=/run/user/1000/gdm/Xauthority' 'GJS_DEBUG_TOPICS=JS ERROR;JS LOG' 'WINDOWPATH=2' 'HOME=/home/reu' 'USERNAME=reu' 'IM_CONFIG_PHASE=1' 'LANG=en_US.UTF-8' 'LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=00:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.zst=01;31:*.tzst=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.wim=01;31:*.swm=01;31:*.dwm=01;31:*.esd=01;31:*.jpg=01;35:*.jpeg=01;35:*.mjpg=01;35:*.mjpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.oga=00;36:*.opus=00;36:*.spx=00;36:*.xspf=00;36:' 'XDG_CURRENT_DESKTOP=ubuntu:GNOME' 'VTE_VERSION=6003' 'GNOME_TERMINAL_SCREEN=/org/gnome/Terminal/screen/59a28a14_f606_4ca7_bd10_beeeddc46a21' 'INVOCATION_ID=877fc76f797f4911acd1380fe960344b' 'MANAGERPID=2830' 'GJS_DEBUG_OUTPUT=stderr' 'LESSCLOSE=/usr/bin/lesspipe %s %s' 'XDG_SESSION_CLASS=user' 'TERM=xterm-256color' 'LESSOPEN=| /usr/bin/lesspipe %s' 'USER=reu' 'GNOME_TERMINAL_SERVICE=:1.102' 'SHLVL=1' 'QT_IM_MODULE=ibus' 'LD_LIBRARY_PATH=/usr/local/cuda/lib64:' 'XDG_RUNTIME_DIR=/run/user/1000' 'CUDA_HOME=/usr/local/cuda' 'JOURNAL_STREAM=7:58083' 'XDG_DATA_DIRS=/usr/share/ubuntu:/usr/local/share/:/usr/share/:/var/lib/snapd/desktop' 'PATH=/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin' 'GDMSESSION=ubuntu' 'DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1000/bus' '_=/usr/local/bin/mpirun' --global-user-env 0 --global-system-env 1 'GFORTRAN_UNBUFFERED_PRECONNECTED=y' --proxy-core-count 1 --exec --exec-appnum 0 --exec-proc-count 1 --exec-local-env 0 --exec-wdir /home/reu --exec-args 1 ./hello --exec --exec-appnum 0 --exec-proc-count 1 --exec-local-env 0 --exec-wdir /home/reu --exec-args 1 ./hello --exec --exec-appnum 0 --exec-proc-count 1 --exec-local-env 0 --exec-wdir /home/reu --exec-args 1 ./hello --exec --exec-appnum 0 --exec-proc-count 1 --exec-local-env 0 --exec-wdir /home/reu --exec-args 1 ./hello --exec --exec-appnum 0 --exec-proc-count 1 --exec-local-env 0 --exec-wdir /home/reu --exec-args 1 ./hello Arguments being passed to proxy 1: --version 3.2.1 --iface-ip-env-name MPIR_CVAR_CH3_INTERFACE_HOSTNAME --hostname 192.168.2.11 --global-core-map 0,1,2 --pmi-id-map 0,1 --global-process-count 10 --auto-cleanup 1 --pmi-kvsname kvs_25221_0 --pmi-process-mapping (vector,(0,2,1)) --ckpoint-num -1 --global-inherited-env 49 'SHELL=/bin/bash' 'SESSION_MANAGER=local/reuOrin-desktop:@/tmp/.ICE-unix/3043,unix/reuOrin-desktop:/tmp/.ICE-unix/3043' 'QT_ACCESSIBILITY=1' 'COLORTERM=truecolor' 'XDG_CONFIG_DIRS=/etc/xdg/xdg-ubuntu:/etc/xdg' 'XDG_MENU_PREFIX=gnome-' 'GNOME_DESKTOP_SESSION_ID=this-is-deprecated' 'GNOME_SHELL_SESSION_MODE=ubuntu' 'SSH_AUTH_SOCK=/run/user/1000/keyring/ssh' 'XMODIFIERS=@im=ibus' 'DESKTOP_SESSION=ubuntu' 'SSH_AGENT_PID=3007' 'GTK_MODULES=gail:atk-bridge' 'PWD=/home/reu' 'LOGNAME=reu' 'XDG_SESSION_DESKTOP=ubuntu' 'XDG_SESSION_TYPE=x11' 'GPG_AGENT_INFO=/run/user/1000/gnupg/S.gpg-agent:0:1' 'XAUTHORITY=/run/user/1000/gdm/Xauthority' 'GJS_DEBUG_TOPICS=JS ERROR;JS LOG' 'WINDOWPATH=2' 'HOME=/home/reu' 'USERNAME=reu' 'IM_CONFIG_PHASE=1' 'LANG=en_US.UTF-8' 'LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=00:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.zst=01;31:*.tzst=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.wim=01;31:*.swm=01;31:*.dwm=01;31:*.esd=01;31:*.jpg=01;35:*.jpeg=01;35:*.mjpg=01;35:*.mjpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.oga=00;36:*.opus=00;36:*.spx=00;36:*.xspf=00;36:' 'XDG_CURRENT_DESKTOP=ubuntu:GNOME' 'VTE_VERSION=6003' 'GNOME_TERMINAL_SCREEN=/org/gnome/Terminal/screen/59a28a14_f606_4ca7_bd10_beeeddc46a21' 'INVOCATION_ID=877fc76f797f4911acd1380fe960344b' 'MANAGERPID=2830' 'GJS_DEBUG_OUTPUT=stderr' 'LESSCLOSE=/usr/bin/lesspipe %s %s' 'XDG_SESSION_CLASS=user' 'TERM=xterm-256color' 'LESSOPEN=| /usr/bin/lesspipe %s' 'USER=reu' 'GNOME_TERMINAL_SERVICE=:1.102' 'SHLVL=1' 'QT_IM_MODULE=ibus' 'LD_LIBRARY_PATH=/usr/local/cuda/lib64:' 'XDG_RUNTIME_DIR=/run/user/1000' 'CUDA_HOME=/usr/local/cuda' 'JOURNAL_STREAM=7:58083' 'XDG_DATA_DIRS=/usr/share/ubuntu:/usr/local/share/:/usr/share/:/var/lib/snapd/desktop' 'PATH=/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin' 'GDMSESSION=ubuntu' 'DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1000/bus' '_=/usr/local/bin/mpirun' --global-user-env 0 --global-system-env 1 'GFORTRAN_UNBUFFERED_PRECONNECTED=y' --proxy-core-count 1 --exec --exec-appnum 0 --exec-proc-count 1 --exec-local-env 0 --exec-wdir /home/reu --exec-args 1 ./hello --exec --exec-appnum 0 --exec-proc-count 1 --exec-local-env 0 --exec-wdir /home/reu --exec-args 1 ./hello --exec --exec-appnum 0 --exec-proc-count 1 --exec-local-env 0 --exec-wdir /home/reu --exec-args 1 ./hello --exec --exec-appnum 0 --exec-proc-count 1 --exec-local-env 0 --exec-wdir /home/reu --exec-args 1 ./hello --exec --exec-appnum 0 --exec-proc-count 1 --exec-local-env 0 --exec-wdir /home/reu --exec-args 1 ./hello [mpiexec at reuOrin-desktop] Launch arguments: /usr/bin/ssh -x 192.168.2.10 "/usr/local/bin/hydra_pmi_proxy" --control-port reuOrin-desktop:40197 --debug --rmk user --launcher ssh --demux poll --pgid 0 --retries 10 --usize -2 --proxy-id 0 [mpiexec at reuOrin-desktop] Launch arguments: /usr/bin/ssh -x 192.168.2.11 "/usr/local/bin/hydra_pmi_proxy" --control-port reuOrin-desktop:40197 --debug --rmk user --launcher ssh --demux poll --pgid 0 --retries 10 --usize -2 --proxy-id 1 -------------- next part -------------- host: 192.168.2.20 host: 192.168.2.10 host: 192.168.2.11 ================================================================================================== mpiexec options: ---------------- Base path: /usr/local/bin/ Launcher: (null) Debug level: 1 Enable X: -1 Global environment: ------------------- SHELL=/bin/bash SESSION_MANAGER=local/reuOrin-desktop:@/tmp/.ICE-unix/3043,unix/reuOrin-desktop:/tmp/.ICE-unix/3043 QT_ACCESSIBILITY=1 COLORTERM=truecolor XDG_CONFIG_DIRS=/etc/xdg/xdg-ubuntu:/etc/xdg XDG_MENU_PREFIX=gnome- GNOME_DESKTOP_SESSION_ID=this-is-deprecated GNOME_SHELL_SESSION_MODE=ubuntu SSH_AUTH_SOCK=/run/user/1000/keyring/ssh XMODIFIERS=@im=ibus DESKTOP_SESSION=ubuntu SSH_AGENT_PID=3007 GTK_MODULES=gail:atk-bridge PWD=/home/reu LOGNAME=reu XDG_SESSION_DESKTOP=ubuntu XDG_SESSION_TYPE=x11 GPG_AGENT_INFO=/run/user/1000/gnupg/S.gpg-agent:0:1 XAUTHORITY=/run/user/1000/gdm/Xauthority GJS_DEBUG_TOPICS=JS ERROR;JS LOG WINDOWPATH=2 HOME=/home/reu USERNAME=reu IM_CONFIG_PHASE=1 LANG=en_US.UTF-8 LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=00:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.zst=01;31:*.tzst=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.wim=01;31:*.swm=01;31:*.dwm=01;31:*.esd=01;31:*.jpg=01;35:*.jpeg=01;35:*.mjpg=01;35:*.mjpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.oga=00;36:*.opus=00;36:*.spx=00;36:*.xspf=00;36: XDG_CURRENT_DESKTOP=ubuntu:GNOME VTE_VERSION=6003 GNOME_TERMINAL_SCREEN=/org/gnome/Terminal/screen/b34ccce0_720f_4f1a_9535_78b2ab4c39be INVOCATION_ID=877fc76f797f4911acd1380fe960344b MANAGERPID=2830 GJS_DEBUG_OUTPUT=stderr LESSCLOSE=/usr/bin/lesspipe %s %s XDG_SESSION_CLASS=user TERM=xterm-256color LESSOPEN=| /usr/bin/lesspipe %s USER=reu GNOME_TERMINAL_SERVICE=:1.102 SHLVL=1 QT_IM_MODULE=ibus LD_LIBRARY_PATH=/usr/local/cuda/lib64: XDG_RUNTIME_DIR=/run/user/1000 CUDA_HOME=/usr/local/cuda JOURNAL_STREAM=7:58083 XDG_DATA_DIRS=/usr/share/ubuntu:/usr/local/share/:/usr/share/:/var/lib/snapd/desktop PATH=/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin GDMSESSION=ubuntu DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1000/bus _=/usr/local/bin/mpirun Hydra internal environment: --------------------------- GFORTRAN_UNBUFFERED_PRECONNECTED=y Proxy information: ********************* [1] proxy: 192.168.2.20 (1 cores) Exec list: ./hello (1 processes); ./hello (1 processes); ./hello (1 processes); ./hello (1 processes); [2] proxy: 192.168.2.10 (1 cores) Exec list: ./hello (1 processes); ./hello (1 processes); ./hello (1 processes); [3] proxy: 192.168.2.11 (1 cores) Exec list: ./hello (1 processes); ./hello (1 processes); ./hello (1 processes); ================================================================================================== [mpiexec at reuOrin-desktop] Timeout set to -1 (-1 means infinite) [mpiexec at reuOrin-desktop] Got a control port string of 192.168.2.20:33223 Proxy launch args: /usr/local/bin/hydra_pmi_proxy --control-port 192.168.2.20:33223 --debug --rmk user --launcher ssh --demux poll --pgid 0 --retries 10 --usize -2 --proxy-id Arguments being passed to proxy 0: --version 3.2.1 --iface-ip-env-name MPIR_CVAR_CH3_INTERFACE_HOSTNAME --hostname 192.168.2.20 --global-core-map 0,1,3 --pmi-id-map 0,0 --global-process-count 10 --auto-cleanup 1 --pmi-kvsname kvs_23478_0 --pmi-process-mapping (vector,(0,3,1)) --ckpoint-num -1 --global-inherited-env 49 'SHELL=/bin/bash' 'SESSION_MANAGER=local/reuOrin-desktop:@/tmp/.ICE-unix/3043,unix/reuOrin-desktop:/tmp/.ICE-unix/3043' 'QT_ACCESSIBILITY=1' 'COLORTERM=truecolor' 'XDG_CONFIG_DIRS=/etc/xdg/xdg-ubuntu:/etc/xdg' 'XDG_MENU_PREFIX=gnome-' 'GNOME_DESKTOP_SESSION_ID=this-is-deprecated' 'GNOME_SHELL_SESSION_MODE=ubuntu' 'SSH_AUTH_SOCK=/run/user/1000/keyring/ssh' 'XMODIFIERS=@im=ibus' 'DESKTOP_SESSION=ubuntu' 'SSH_AGENT_PID=3007' 'GTK_MODULES=gail:atk-bridge' 'PWD=/home/reu' 'LOGNAME=reu' 'XDG_SESSION_DESKTOP=ubuntu' 'XDG_SESSION_TYPE=x11' 'GPG_AGENT_INFO=/run/user/1000/gnupg/S.gpg-agent:0:1' 'XAUTHORITY=/run/user/1000/gdm/Xauthority' 'GJS_DEBUG_TOPICS=JS ERROR;JS LOG' 'WINDOWPATH=2' 'HOME=/home/reu' 'USERNAME=reu' 'IM_CONFIG_PHASE=1' 'LANG=en_US.UTF-8' 'LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=00:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.zst=01;31:*.tzst=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.wim=01;31:*.swm=01;31:*.dwm=01;31:*.esd=01;31:*.jpg=01;35:*.jpeg=01;35:*.mjpg=01;35:*.mjpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.oga=00;36:*.opus=00;36:*.spx=00;36:*.xspf=00;36:' 'XDG_CURRENT_DESKTOP=ubuntu:GNOME' 'VTE_VERSION=6003' 'GNOME_TERMINAL_SCREEN=/org/gnome/Terminal/screen/b34ccce0_720f_4f1a_9535_78b2ab4c39be' 'INVOCATION_ID=877fc76f797f4911acd1380fe960344b' 'MANAGERPID=2830' 'GJS_DEBUG_OUTPUT=stderr' 'LESSCLOSE=/usr/bin/lesspipe %s %s' 'XDG_SESSION_CLASS=user' 'TERM=xterm-256color' 'LESSOPEN=| /usr/bin/lesspipe %s' 'USER=reu' 'GNOME_TERMINAL_SERVICE=:1.102' 'SHLVL=1' 'QT_IM_MODULE=ibus' 'LD_LIBRARY_PATH=/usr/local/cuda/lib64:' 'XDG_RUNTIME_DIR=/run/user/1000' 'CUDA_HOME=/usr/local/cuda' 'JOURNAL_STREAM=7:58083' 'XDG_DATA_DIRS=/usr/share/ubuntu:/usr/local/share/:/usr/share/:/var/lib/snapd/desktop' 'PATH=/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin' 'GDMSESSION=ubuntu' 'DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1000/bus' '_=/usr/local/bin/mpirun' --global-user-env 0 --global-system-env 1 'GFORTRAN_UNBUFFERED_PRECONNECTED=y' --proxy-core-count 1 --exec --exec-appnum 0 --exec-proc-count 1 --exec-local-env 0 --exec-wdir /home/reu --exec-args 1 ./hello --exec --exec-appnum 0 --exec-proc-count 1 --exec-local-env 0 --exec-wdir /home/reu --exec-args 1 ./hello --exec --exec-appnum 0 --exec-proc-count 1 --exec-local-env 0 --exec-wdir /home/reu --exec-args 1 ./hello --exec --exec-appnum 0 --exec-proc-count 1 --exec-local-env 0 --exec-wdir /home/reu --exec-args 1 ./hello Arguments being passed to proxy 1: --version 3.2.1 --iface-ip-env-name MPIR_CVAR_CH3_INTERFACE_HOSTNAME --hostname 192.168.2.10 --global-core-map 0,1,3 --pmi-id-map 0,1 --global-process-count 10 --auto-cleanup 1 --pmi-kvsname kvs_23478_0 --pmi-process-mapping (vector,(0,3,1)) --ckpoint-num -1 --global-inherited-env 49 'SHELL=/bin/bash' 'SESSION_MANAGER=local/reuOrin-desktop:@/tmp/.ICE-unix/3043,unix/reuOrin-desktop:/tmp/.ICE-unix/3043' 'QT_ACCESSIBILITY=1' 'COLORTERM=truecolor' 'XDG_CONFIG_DIRS=/etc/xdg/xdg-ubuntu:/etc/xdg' 'XDG_MENU_PREFIX=gnome-' 'GNOME_DESKTOP_SESSION_ID=this-is-deprecated' 'GNOME_SHELL_SESSION_MODE=ubuntu' 'SSH_AUTH_SOCK=/run/user/1000/keyring/ssh' 'XMODIFIERS=@im=ibus' 'DESKTOP_SESSION=ubuntu' 'SSH_AGENT_PID=3007' 'GTK_MODULES=gail:atk-bridge' 'PWD=/home/reu' 'LOGNAME=reu' 'XDG_SESSION_DESKTOP=ubuntu' 'XDG_SESSION_TYPE=x11' 'GPG_AGENT_INFO=/run/user/1000/gnupg/S.gpg-agent:0:1' 'XAUTHORITY=/run/user/1000/gdm/Xauthority' 'GJS_DEBUG_TOPICS=JS ERROR;JS LOG' 'WINDOWPATH=2' 'HOME=/home/reu' 'USERNAME=reu' 'IM_CONFIG_PHASE=1' 'LANG=en_US.UTF-8' 'LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=00:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.zst=01;31:*.tzst=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.wim=01;31:*.swm=01;31:*.dwm=01;31:*.esd=01;31:*.jpg=01;35:*.jpeg=01;35:*.mjpg=01;35:*.mjpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.oga=00;36:*.opus=00;36:*.spx=00;36:*.xspf=00;36:' 'XDG_CURRENT_DESKTOP=ubuntu:GNOME' 'VTE_VERSION=6003' 'GNOME_TERMINAL_SCREEN=/org/gnome/Terminal/screen/b34ccce0_720f_4f1a_9535_78b2ab4c39be' 'INVOCATION_ID=877fc76f797f4911acd1380fe960344b' 'MANAGERPID=2830' 'GJS_DEBUG_OUTPUT=stderr' 'LESSCLOSE=/usr/bin/lesspipe %s %s' 'XDG_SESSION_CLASS=user' 'TERM=xterm-256color' 'LESSOPEN=| /usr/bin/lesspipe %s' 'USER=reu' 'GNOME_TERMINAL_SERVICE=:1.102' 'SHLVL=1' 'QT_IM_MODULE=ibus' 'LD_LIBRARY_PATH=/usr/local/cuda/lib64:' 'XDG_RUNTIME_DIR=/run/user/1000' 'CUDA_HOME=/usr/local/cuda' 'JOURNAL_STREAM=7:58083' 'XDG_DATA_DIRS=/usr/share/ubuntu:/usr/local/share/:/usr/share/:/var/lib/snapd/desktop' 'PATH=/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin' 'GDMSESSION=ubuntu' 'DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1000/bus' '_=/usr/local/bin/mpirun' --global-user-env 0 --global-system-env 1 'GFORTRAN_UNBUFFERED_PRECONNECTED=y' --proxy-core-count 1 --exec --exec-appnum 0 --exec-proc-count 1 --exec-local-env 0 --exec-wdir /home/reu --exec-args 1 ./hello --exec --exec-appnum 0 --exec-proc-count 1 --exec-local-env 0 --exec-wdir /home/reu --exec-args 1 ./hello --exec --exec-appnum 0 --exec-proc-count 1 --exec-local-env 0 --exec-wdir /home/reu --exec-args 1 ./hello Arguments being passed to proxy 2: --version 3.2.1 --iface-ip-env-name MPIR_CVAR_CH3_INTERFACE_HOSTNAME --hostname 192.168.2.11 --global-core-map 0,1,3 --pmi-id-map 0,2 --global-process-count 10 --auto-cleanup 1 --pmi-kvsname kvs_23478_0 --pmi-process-mapping (vector,(0,3,1)) --ckpoint-num -1 --global-inherited-env 49 'SHELL=/bin/bash' 'SESSION_MANAGER=local/reuOrin-desktop:@/tmp/.ICE-unix/3043,unix/reuOrin-desktop:/tmp/.ICE-unix/3043' 'QT_ACCESSIBILITY=1' 'COLORTERM=truecolor' 'XDG_CONFIG_DIRS=/etc/xdg/xdg-ubuntu:/etc/xdg' 'XDG_MENU_PREFIX=gnome-' 'GNOME_DESKTOP_SESSION_ID=this-is-deprecated' 'GNOME_SHELL_SESSION_MODE=ubuntu' 'SSH_AUTH_SOCK=/run/user/1000/keyring/ssh' 'XMODIFIERS=@im=ibus' 'DESKTOP_SESSION=ubuntu' 'SSH_AGENT_PID=3007' 'GTK_MODULES=gail:atk-bridge' 'PWD=/home/reu' 'LOGNAME=reu' 'XDG_SESSION_DESKTOP=ubuntu' 'XDG_SESSION_TYPE=x11' 'GPG_AGENT_INFO=/run/user/1000/gnupg/S.gpg-agent:0:1' 'XAUTHORITY=/run/user/1000/gdm/Xauthority' 'GJS_DEBUG_TOPICS=JS ERROR;JS LOG' 'WINDOWPATH=2' 'HOME=/home/reu' 'USERNAME=reu' 'IM_CONFIG_PHASE=1' 'LANG=en_US.UTF-8' 'LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=00:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.zst=01;31:*.tzst=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.wim=01;31:*.swm=01;31:*.dwm=01;31:*.esd=01;31:*.jpg=01;35:*.jpeg=01;35:*.mjpg=01;35:*.mjpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.oga=00;36:*.opus=00;36:*.spx=00;36:*.xspf=00;36:' 'XDG_CURRENT_DESKTOP=ubuntu:GNOME' 'VTE_VERSION=6003' 'GNOME_TERMINAL_SCREEN=/org/gnome/Terminal/screen/b34ccce0_720f_4f1a_9535_78b2ab4c39be' 'INVOCATION_ID=877fc76f797f4911acd1380fe960344b' 'MANAGERPID=2830' 'GJS_DEBUG_OUTPUT=stderr' 'LESSCLOSE=/usr/bin/lesspipe %s %s' 'XDG_SESSION_CLASS=user' 'TERM=xterm-256color' 'LESSOPEN=| /usr/bin/lesspipe %s' 'USER=reu' 'GNOME_TERMINAL_SERVICE=:1.102' 'SHLVL=1' 'QT_IM_MODULE=ibus' 'LD_LIBRARY_PATH=/usr/local/cuda/lib64:' 'XDG_RUNTIME_DIR=/run/user/1000' 'CUDA_HOME=/usr/local/cuda' 'JOURNAL_STREAM=7:58083' 'XDG_DATA_DIRS=/usr/share/ubuntu:/usr/local/share/:/usr/share/:/var/lib/snapd/desktop' 'PATH=/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin' 'GDMSESSION=ubuntu' 'DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1000/bus' '_=/usr/local/bin/mpirun' --global-user-env 0 --global-system-env 1 'GFORTRAN_UNBUFFERED_PRECONNECTED=y' --proxy-core-count 1 --exec --exec-appnum 0 --exec-proc-count 1 --exec-local-env 0 --exec-wdir /home/reu --exec-args 1 ./hello --exec --exec-appnum 0 --exec-proc-count 1 --exec-local-env 0 --exec-wdir /home/reu --exec-args 1 ./hello --exec --exec-appnum 0 --exec-proc-count 1 --exec-local-env 0 --exec-wdir /home/reu --exec-args 1 ./hello [mpiexec at reuOrin-desktop] Launch arguments: /usr/local/bin/hydra_pmi_proxy --control-port 192.168.2.20:33223 --debug --rmk user --launcher ssh --demux poll --pgid 0 --retries 10 --usize -2 --proxy-id 0 [mpiexec at reuOrin-desktop] Launch arguments: /usr/bin/ssh -x 192.168.2.10 "/usr/local/bin/hydra_pmi_proxy" --control-port 192.168.2.20:33223 --debug --rmk user --launcher ssh --demux poll --pgid 0 --retries 10 --usize -2 --proxy-id 1 [mpiexec at reuOrin-desktop] Launch arguments: /usr/bin/ssh -x 192.168.2.11 "/usr/local/bin/hydra_pmi_proxy" --control-port 192.168.2.20:33223 --debug --rmk user --launcher ssh --demux poll --pgid 0 --retries 10 --usize -2 --proxy-id 2 [proxy:0:0 at reuOrin-desktop] got pmi command (from 0): init pmi_version=1 pmi_subversion=1 [proxy:0:0 at reuOrin-desktop] PMI response: cmd=response_to_init pmi_version=1 pmi_subversion=1 rc=0 [proxy:0:0 at reuOrin-desktop] got pmi command (from 0): get_maxes [proxy:0:0 at reuOrin-desktop] PMI response: cmd=maxes kvsname_max=256 keylen_max=64 vallen_max=1024 [proxy:0:0 at reuOrin-desktop] got pmi command (from 6): init pmi_version=1 pmi_subversion=1 [proxy:0:0 at reuOrin-desktop] PMI response: cmd=response_to_init pmi_version=1 pmi_subversion=1 rc=0 [proxy:0:0 at reuOrin-desktop] got pmi command (from 8): init pmi_version=1 pmi_subversion=1 [proxy:0:0 at reuOrin-desktop] PMI response: cmd=response_to_init pmi_version=1 pmi_subversion=1 rc=0 [proxy:0:0 at reuOrin-desktop] got pmi command (from 13): init pmi_version=1 pmi_subversion=1 [proxy:0:0 at reuOrin-desktop] PMI response: cmd=response_to_init pmi_version=1 pmi_subversion=1 rc=0 [proxy:0:0 at reuOrin-desktop] got pmi command (from 0): get_appnum [proxy:0:0 at reuOrin-desktop] PMI response: cmd=appnum appnum=0 [proxy:0:0 at reuOrin-desktop] got pmi command (from 6): get_maxes [proxy:0:0 at reuOrin-desktop] PMI response: cmd=maxes kvsname_max=256 keylen_max=64 vallen_max=1024 [proxy:0:0 at reuOrin-desktop] got pmi command (from 8): get_maxes [proxy:0:0 at reuOrin-desktop] PMI response: cmd=maxes kvsname_max=256 keylen_max=64 vallen_max=1024 [proxy:0:0 at reuOrin-desktop] got pmi command (from 0): get_my_kvsname [proxy:0:0 at reuOrin-desktop] PMI response: cmd=my_kvsname kvsname=kvs_23478_0 [proxy:0:0 at reuOrin-desktop] got pmi command (from 6): get_appnum [proxy:0:0 at reuOrin-desktop] PMI response: cmd=appnum appnum=0 [proxy:0:0 at reuOrin-desktop] got pmi command (from 13): get_maxes [proxy:0:0 at reuOrin-desktop] PMI response: cmd=maxes kvsname_max=256 keylen_max=64 vallen_max=1024 [proxy:0:0 at reuOrin-desktop] got pmi command (from 0): get_my_kvsname [proxy:0:0 at reuOrin-desktop] PMI response: cmd=my_kvsname kvsname=kvs_23478_0 [proxy:0:0 at reuOrin-desktop] got pmi command (from 6): get_my_kvsname [proxy:0:0 at reuOrin-desktop] PMI response: cmd=my_kvsname kvsname=kvs_23478_0 [proxy:0:0 at reuOrin-desktop] got pmi command (from 8): get_appnum [proxy:0:0 at reuOrin-desktop] PMI response: cmd=appnum appnum=0 [proxy:0:0 at reuOrin-desktop] got pmi command (from 0): get kvsname=kvs_23478_0 key=PMI_process_mapping [proxy:0:0 at reuOrin-desktop] PMI response: cmd=get_result rc=0 msg=success value=(vector,(0,3,1)) [proxy:0:0 at reuOrin-desktop] got pmi command (from 6): get_my_kvsname [proxy:0:0 at reuOrin-desktop] PMI response: cmd=my_kvsname kvsname=kvs_23478_0 [proxy:0:0 at reuOrin-desktop] got pmi command (from 13): get_appnum [proxy:0:0 at reuOrin-desktop] PMI response: cmd=appnum appnum=0 [proxy:0:0 at reuOrin-desktop] got pmi command (from 0): put kvsname=kvs_23478_0 key=sharedFilename-0 value=/dev/shm/mpich_shar_tmpvgZseM [proxy:0:0 at reuOrin-desktop] cached command: sharedFilename-0=/dev/shm/mpich_shar_tmpvgZseM [proxy:0:0 at reuOrin-desktop] PMI response: cmd=put_result rc=0 msg=success [proxy:0:0 at reuOrin-desktop] got pmi command (from 6): get kvsname=kvs_23478_0 key=PMI_process_mapping [proxy:0:0 at reuOrin-desktop] PMI response: cmd=get_result rc=0 msg=success value=(vector,(0,3,1)) [proxy:0:0 at reuOrin-desktop] got pmi command (from 8): get_my_kvsname [proxy:0:0 at reuOrin-desktop] PMI response: cmd=my_kvsname kvsname=kvs_23478_0 [proxy:0:0 at reuOrin-desktop] got pmi command (from 0): barrier_in [proxy:0:0 at reuOrin-desktop] got pmi command (from 6): barrier_in [proxy:0:0 at reuOrin-desktop] got pmi command (from 13): get_my_kvsname [proxy:0:0 at reuOrin-desktop] PMI response: cmd=my_kvsname kvsname=kvs_23478_0 [proxy:0:0 at reuOrin-desktop] got pmi command (from 8): get_my_kvsname [proxy:0:0 at reuOrin-desktop] PMI response: cmd=my_kvsname kvsname=kvs_23478_0 [proxy:0:0 at reuOrin-desktop] got pmi command (from 13): get_my_kvsname [proxy:0:0 at reuOrin-desktop] PMI response: cmd=my_kvsname kvsname=kvs_23478_0 [proxy:0:0 at reuOrin-desktop] got pmi command (from 8): get kvsname=kvs_23478_0 key=PMI_process_mapping [proxy:0:0 at reuOrin-desktop] PMI response: cmd=get_result rc=0 msg=success value=(vector,(0,3,1)) [proxy:0:0 at reuOrin-desktop] got pmi command (from 13): get kvsname=kvs_23478_0 key=PMI_process_mapping [proxy:0:0 at reuOrin-desktop] PMI response: cmd=get_result rc=0 msg=success value=(vector,(0,3,1)) [proxy:0:0 at reuOrin-desktop] got pmi command (from 8): barrier_in [proxy:0:0 at reuOrin-desktop] got pmi command (from 13): barrier_in [proxy:0:0 at reuOrin-desktop] flushing 1 put command(s) out [proxy:0:0 at reuOrin-desktop] [mpiexec at reuOrin-desktop] [pgid: 0] got PMI command: cmd=put sharedFilename-0=/dev/shm/mpich_shar_tmpvgZseM forwarding command (cmd=put sharedFilename-0=/dev/shm/mpich_shar_tmpvgZseM) upstream [proxy:0:0 at reuOrin-desktop] forwarding command (cmd=barrier_in) upstream [mpiexec at reuOrin-desktop] [pgid: 0] got PMI command: cmd=barrier_in [proxy:0:1 at reu-desktop] got pmi command (from 4): init pmi_version=1 pmi_subversion=1 [proxy:0:1 at reu-desktop] PMI response: cmd=response_to_init pmi_version=1 pmi_subversion=1 rc=0 [proxy:0:1 at reu-desktop] got pmi command (from 5): init pmi_version=1 pmi_subversion=1 [proxy:0:1 at reu-desktop] PMI response: cmd=response_to_init pmi_version=1 pmi_subversion=1 rc=0 [proxy:0:1 at reu-desktop] got pmi command (from 4): get_maxes [proxy:0:1 at reu-desktop] PMI response: cmd=maxes kvsname_max=256 keylen_max=64 vallen_max=1024 [proxy:0:1 at reu-desktop] got pmi command (from 6): init pmi_version=1 pmi_subversion=1 [proxy:0:1 at reu-desktop] PMI response: cmd=response_to_init pmi_version=1 pmi_subversion=1 rc=0 [proxy:0:1 at reu-desktop] got pmi command (from 4): get_appnum [proxy:0:1 at reu-desktop] PMI response: cmd=appnum appnum=0 [proxy:0:1 at reu-desktop] got pmi command (from 5): get_maxes [proxy:0:1 at reu-desktop] PMI response: cmd=maxes kvsname_max=256 keylen_max=64 vallen_max=1024 [proxy:0:1 at reu-desktop] got pmi command (from 4): get_my_kvsname [proxy:0:1 at reu-desktop] PMI response: cmd=my_kvsname kvsname=kvs_23478_0 [proxy:0:1 at reu-desktop] got pmi command (from 6): get_maxes [proxy:0:1 at reu-desktop] PMI response: cmd=maxes kvsname_max=256 keylen_max=64 vallen_max=1024 [proxy:0:1 at reu-desktop] got pmi command (from 4): get_my_kvsname [proxy:0:1 at reu-desktop] PMI response: cmd=my_kvsname kvsname=kvs_23478_0 [proxy:0:1 at reu-desktop] got pmi command (from 5): get_appnum [proxy:0:1 at reu-desktop] PMI response: cmd=appnum appnum=0 [proxy:0:1 at reu-desktop] got pmi command (from 4): get kvsname=kvs_23478_0 key=PMI_process_mapping [proxy:0:1 at reu-desktop] PMI response: cmd=get_result rc=0 msg=success value=(vector,(0,3,1)) [proxy:0:1 at reu-desktop] got pmi command (from 5): get_my_kvsname [proxy:0:1 at reu-desktop] PMI response: cmd=my_kvsname kvsname=kvs_23478_0 [proxy:0:1 at reu-desktop] got pmi command (from 6): get_appnum [proxy:0:1 at reu-desktop] PMI response: cmd=appnum appnum=0 [proxy:0:1 at reu-desktop] got pmi command (from 4): put kvsname=kvs_23478_0 key=sharedFilename-1 value=/dev/shm/mpich_shar_tmpmhgotU [proxy:0:1 at reu-desktop] cached command: sharedFilename-1=/dev/shm/mpich_shar_tmpmhgotU [proxy:0:1 at reu-desktop] PMI response: cmd=put_result rc=0 msg=success [proxy:0:1 at reu-desktop] got pmi command (from 5): get_my_kvsname [proxy:0:1 at reu-desktop] PMI response: cmd=my_kvsname kvsname=kvs_23478_0 [proxy:0:1 at reu-desktop] got pmi command (from 4): barrier_in [proxy:0:1 at reu-desktop] got pmi command (from 6): get_my_kvsname [proxy:0:1 at reu-desktop] PMI response: cmd=my_kvsname kvsname=kvs_23478_0 [proxy:0:1 at reu-desktop] got pmi command (from 5): get kvsname=kvs_23478_0 key=PMI_process_mapping [proxy:0:1 at reu-desktop] PMI response: cmd=get_result rc=0 msg=success value=(vector,(0,3,1)) [proxy:0:1 at reu-desktop] got pmi command (from 6): get_my_kvsname [proxy:0:1 at reu-desktop] PMI response: cmd=my_kvsname kvsname=kvs_23478_0 [proxy:0:1 at reu-desktop] got pmi command (from 5): barrier_in [proxy:0:1 at reu-desktop] got pmi command (from 6): get kvsname=kvs_23478_0 key=PMI_process_mapping [proxy:0:1 at reu-desktop] PMI response: cmd=get_result rc=0 msg=success value=(vector,(0,3,1)) [mpiexec at reuOrin-desktop] [pgid: 0] got PMI command: cmd=put sharedFilename-1=/dev/shm/mpich_shar_tmpmhgotU [proxy:0:1 at reu-desktop] got pmi command (from 6): barrier_in [proxy:0:1 at reu-desktop] flushing 1 put command(s) out [proxy:0:1 at reu-desktop] forwarding command (cmd=put sharedFilename-1=/dev/shm/mpich_shar_tmpmhgotU) upstream [proxy:0:1 at reu-desktop] forwarding command (cmd=barrier_in) upstream [mpiexec at reuOrin-desktop] [pgid: 0] got PMI command: cmd=barrier_in [proxy:0:2 at reuN2-desktop] got pmi command (from 4): init pmi_version=1 pmi_subversion=1 [proxy:0:2 at reuN2-desktop] PMI response: cmd=response_to_init pmi_version=1 pmi_subversion=1 rc=0 [proxy:0:2 at reuN2-desktop] got pmi command (from 4): get_maxes [proxy:0:2 at reuN2-desktop] PMI response: cmd=maxes kvsname_max=256 keylen_max=64 vallen_max=1024 [proxy:0:2 at reuN2-desktop] got pmi command (from 5): init pmi_version=1 pmi_subversion=1 [proxy:0:2 at reuN2-desktop] PMI response: cmd=response_to_init pmi_version=1 pmi_subversion=1 rc=0 [proxy:0:2 at reuN2-desktop] got pmi command (from 6): init pmi_version=1 pmi_subversion=1 [proxy:0:2 at reuN2-desktop] PMI response: cmd=response_to_init pmi_version=1 pmi_subversion=1 rc=0 [proxy:0:2 at reuN2-desktop] got pmi command (from 4): get_appnum [proxy:0:2 at reuN2-desktop] PMI response: cmd=appnum appnum=0 [proxy:0:2 at reuN2-desktop] got pmi command (from 5): get_maxes [proxy:0:2 at reuN2-desktop] PMI response: cmd=maxes kvsname_max=256 keylen_max=64 vallen_max=1024 [proxy:0:2 at reuN2-desktop] got pmi command (from 4): get_my_kvsname [proxy:0:2 at reuN2-desktop] PMI response: cmd=my_kvsname kvsname=kvs_23478_0 [proxy:0:2 at reuN2-desktop] got pmi command (from 6): get_maxes [proxy:0:2 at reuN2-desktop] PMI response: cmd=maxes kvsname_max=256 keylen_max=64 vallen_max=1024 [proxy:0:2 at reuN2-desktop] got pmi command (from 4): get_my_kvsname [proxy:0:2 at reuN2-desktop] PMI response: cmd=my_kvsname kvsname=kvs_23478_0 [proxy:0:2 at reuN2-desktop] got pmi command (from 5): get_appnum [proxy:0:2 at reuN2-desktop] PMI response: cmd=appnum appnum=0 [proxy:0:2 at reuN2-desktop] got pmi command (from 4): get kvsname=kvs_23478_0 key=PMI_process_mapping [proxy:0:2 at reuN2-desktop] PMI response: cmd=get_result rc=0 msg=success value=(vector,(0,3,1)) [proxy:0:2 at reuN2-desktop] got pmi command (from 5): get_my_kvsname [proxy:0:2 at reuN2-desktop] PMI response: cmd=my_kvsname kvsname=kvs_23478_0 [proxy:0:2 at reuN2-desktop] got pmi command (from 6): get_appnum [proxy:0:2 at reuN2-desktop] PMI response: cmd=appnum appnum=0 [proxy:0:2 at reuN2-desktop] got pmi command (from 4): put kvsname=kvs_23478_0 key=sharedFilename-2 value=/dev/shm/mpich_shar_tmpwPuw7j [proxy:0:2 at reuN2-desktop] cached command: sharedFilename-2=/dev/shm/mpich_shar_tmpwPuw7j [proxy:0:2 at reuN2-desktop] PMI response: cmd=put_result rc=0 msg=success [proxy:0:2 at reuN2-desktop] got pmi command (from 5): get_my_kvsname [proxy:0:2 at reuN2-desktop] PMI response: cmd=my_kvsname kvsname=kvs_23478_0 [mpiexec at reuOrin-desktop] [pgid: 0] got PMI command: cmd=put sharedFilename-2=/dev/shm/mpich_shar_tmpwPuw7j [proxy:0:2 at reuN2-desktop] got pmi command (from 4): barrier_in [proxy:0:2 at reuN2-desktop] got pmi command (from 6): get_my_kvsname [proxy:0:2 at reuN2-desktop] PMI response: cmd=my_kvsname kvsname=kvs_23478_0 [proxy:0:2 at reuN2-desktop] got pmi command (from 5): get kvsname=kvs_23478_0 key=PMI_process_mapping [proxy:0:2 at reuN2-desktop] PMI response: cmd=get_result rc=0 msg=success value=(vector,(0,3,1)) [proxy:0:2 at reuN2-desktop] got pmi command (from 6): get_my_kvsname [proxy:0:2 at reuN2-desktop] PMI response: cmd=my_kvsname kvsname=kvs_23478_0 [proxy:0:2 at reuN2-desktop] got pmi command (from 5): barrier_in [proxy:0:2 at reuN2-desktop] got pmi command (from 6): get kvsname=kvs_23478_0 key=PMI_process_mapping [proxy:0:2 at reuN2-desktop] PMI response: cmd=get_result rc=0 msg=success value=(vector,(0,3,1)) [proxy:0:2 at reuN2-desktop] got pmi command (from 6): barrier_in [proxy:0:2 at reuN2-desktop] flushing 1 put command(s) out [proxy:0:2 at reuN2-desktop] forwarding command (cmd=put sharedFilename-2=/dev/shm/mpich_shar_tmpwPuw7j) upstream [proxy:0:2 at reuN2-desktop] forwarding command (cmd=barrier_in) upstream [mpiexec at reuOrin-desktop] [pgid: 0] got PMI command: cmd=barrier_in [mpiexec at reuOrin-desktop] PMI response to fd 6 pid 6: cmd=keyval_cache sharedFilename-0=/dev/shm/mpich_shar_tmpvgZseM sharedFilename-1=/dev/shm/mpich_shar_tmpmhgotU sharedFilename-2=/dev/shm/mpich_shar_tmpwPuw7j [mpiexec at reuOrin-desktop] PMI response to fd 7 pid 6: cmd=keyval_cache sharedFilename-0=/dev/shm/mpich_shar_tmpvgZseM sharedFilename-1=/dev/shm/mpich_shar_tmpmhgotU sharedFilename-2=/dev/shm/mpich_shar_tmpwPuw7j [mpiexec at reuOrin-desktop] PMI response to fd 15 pid 6: cmd=keyval_cache sharedFilename-0=/dev/shm/mpich_shar_tmpvgZseM sharedFilename-1=/dev/shm/mpich_shar_tmpmhgotU sharedFilename-2=/dev/shm/mpich_shar_tmpwPuw7j [mpiexec at reuOrin-desktop] PMI response to fd 6 pid 6: cmd=barrier_out [mpiexec at reuOrin-desktop] PMI response to fd 7 pid 6: cmd=barrier_out [mpiexec at reuOrin-desktop] PMI response to fd 15 pid 6: cmd=barrier_out [proxy:0:0 at reuOrin-desktop] PMI response: cmd=barrier_out [proxy:0:0 at reuOrin-desktop] PMI response: cmd=barrier_out [proxy:0:0 at reuOrin-desktop] PMI response: cmd=barrier_out [proxy:0:0 at reuOrin-desktop] PMI response: cmd=barrier_out [proxy:0:0 at reuOrin-desktop] got pmi command (from 6): get kvsname=kvs_23478_0 key=sharedFilename-0 [proxy:0:0 at reuOrin-desktop] PMI response: cmd=get_result rc=0 msg=success value=/dev/shm/mpich_shar_tmpvgZseM [proxy:0:0 at reuOrin-desktop] got pmi command (from 8): get kvsname=kvs_23478_0 key=sharedFilename-0 [proxy:0:0 at reuOrin-desktop] PMI response: cmd=get_result rc=0 msg=success value=/dev/shm/mpich_shar_tmpvgZseM [proxy:0:0 at reuOrin-desktop] got pmi command (from 13): get kvsname=kvs_23478_0 key=sharedFilename-0 [proxy:0:0 at reuOrin-desktop] PMI response: cmd=get_result rc=0 msg=success value=/dev/shm/mpich_shar_tmpvgZseM [proxy:0:2 at reuN2-desktop] PMI response: cmd=barrier_out [proxy:0:2 at reuN2-desktop] PMI response: cmd=barrier_out [proxy:0:2 at reuN2-desktop] PMI response: cmd=barrier_out [proxy:0:2 at reuN2-desktop] got pmi command (from 5): get kvsname=kvs_23478_0 key=sharedFilename-2 [proxy:0:2 at reuN2-desktop] PMI response: cmd=get_result rc=0 msg=success value=/dev/shm/mpich_shar_tmpwPuw7j [proxy:0:2 at reuN2-desktop] got pmi command (from 6): get kvsname=kvs_23478_0 key=sharedFilename-2 [proxy:0:2 at reuN2-desktop] PMI response: cmd=get_result rc=0 msg=success value=/dev/shm/mpich_shar_tmpwPuw7j [proxy:0:0 at reuOrin-desktop] got pmi command (from 0): put kvsname=kvs_23478_0 key=Pbusinesscard-0 value=description#192.168.2.20$port#56375$ifname#192.168.2.20$ [proxy:0:0 at reuOrin-desktop] cached command: Pbusinesscard-0=description#192.168.2.20$port#56375$ifname#192.168.2.20$ [proxy:0:0 at reuOrin-desktop] PMI response: cmd=put_result rc=0 msg=success [proxy:0:0 at reuOrin-desktop] got pmi command (from 6): put kvsname=kvs_23478_0 key=Pbusinesscard-3 value=description#192.168.2.20$port#42255$ifname#192.168.2.20$ [proxy:0:0 at reuOrin-desktop] cached command: Pbusinesscard-3=description#192.168.2.20$port#42255$ifname#192.168.2.20$ [proxy:0:0 at reuOrin-desktop] PMI response: cmd=put_result rc=0 msg=success [proxy:0:0 at reuOrin-desktop] got pmi command (from 8): put kvsname=kvs_23478_0 key=Pbusinesscard-6 value=description#192.168.2.20$port#54961$ifname#192.168.2.20$ [proxy:0:0 at reuOrin-desktop] cached command: Pbusinesscard-6=description#192.168.2.20$port#54961$ifname#192.168.2.20$ [proxy:0:0 at reuOrin-desktop] PMI response: cmd=put_result rc=0 msg=success [proxy:0:0 at reuOrin-desktop] got pmi command (from 13): put kvsname=kvs_23478_0 key=Pbusinesscard-9 value=description#192.168.2.20$port#55841$ifname#192.168.2.20$ [proxy:0:0 at reuOrin-desktop] cached command: Pbusinesscard-9=description#192.168.2.20$port#55841$ifname#192.168.2.20$ [proxy:0:0 at reuOrin-desktop] PMI response: cmd=put_result rc=0 msg=success [proxy:0:0 at reuOrin-desktop] got pmi command (from 0): barrier_in [proxy:0:0 at reuOrin-desktop] got pmi command (from 6): barrier_in [proxy:0:0 at reuOrin-desktop] got pmi command (from 8): barrier_in [proxy:0:2 at reuN2-desktop] got pmi command (from 4): put kvsname=kvs_23478_0 key=Pbusinesscard-2 [proxy:0:0 at reuOrin-desktop] got pmi command (from 13): barrier_in [proxy:0:0 at reuOrin-desktop] flushing 4 put command(s) out [proxy:0:0 at reuOrin-desktop] forwarding command (cmd=put Pbusinesscard-0=description#192.168.2.20$port#56375$ifname#192.168.2.20$ Pbusinesscard-3=description#192.168.2.20$port#42255$ifname#192.168.2.20$ Pbusinesscard-6=description#192.168.2.20$port#54961$ifname#192.168.2.20$ Pbusinesscard-9=description#192.168.2.20$port#55841$ifname#192.168.2.20$) upstream [mpiexec at reuOrin-desktop] [pgid: 0] got PMI command: cmd=put Pbusinesscard-0=description#192.168.2.20$port#56375$ifname#192.168.2.20$ Pbusinesscard-3=description#192.168.2.20$port#42255$ifname#192.168.2.20$ Pbusinesscard-6=description#192.168.2.20$port#54961$ifname#192.168.2.20$ Pbusinesscard-9=description#192.168.2.20$port#55841$ifname#192.168.2.20$ [proxy:0:0 at reuOrin-desktop] forwarding command (cmd=barrier_in) upstream value=description#192.168.2.11$port#44085$ifname#192.168.2.11$ [proxy:0:2 at reuN2-desktop] cached command: Pbusinesscard-2=description#192.168.2.11$port#44085$ifname#192.168.2.11$ [proxy:0:2 at reuN2-desktop] PMI response: cmd=put_result rc=0 msg=success [proxy:0:2 at reuN2-desktop] [mpiexec at reuOrin-desktop] [pgid: 0] got PMI command: cmd=barrier_in got pmi command (from 5): put kvsname=kvs_23478_0 key=Pbusinesscard-5 value=description#192.168.2.11$port#56233$ifname#192.168.2.11$ [proxy:0:2 at reuN2-desktop] cached command: Pbusinesscard-5=description#192.168.2.11$port#56233$ifname#192.168.2.11$ [proxy:0:2 at reuN2-desktop] PMI response: cmd=put_result rc=0 msg=success [proxy:0:2 at reuN2-desktop] got pmi command (from 6): put kvsname=kvs_23478_0 key=Pbusinesscard-8 value=description#192.168.2.11$port#39723$ifname#192.168.2.11$ [proxy:0:2 at reuN2-desktop] cached command: Pbusinesscard-8=description#192.168.2.11$port#39723$ifname#192.168.2.11$ [proxy:0:2 at reuN2-desktop] PMI response: cmd=put_result rc=0 msg=success [proxy:0:2 at reuN2-desktop] got pmi command (from 4): barrier_in [proxy:0:2 at reuN2-desktop] got pmi command (from 5): barrier_in [mpiexec at reuOrin-desktop] [pgid: 0] got PMI command: cmd=put Pbusinesscard-2=description#192.168.2.11$port#44085$ifname#192.168.2.11$ Pbusinesscard-5=description#192.168.2.11$port#56233$ifname#192.168.2.11$ Pbusinesscard-8=description#192.168.2.11$port#39723$ifname#192.168.2.11$ [proxy:0:2 at reuN2-desktop] got pmi command (from 6): barrier_in [proxy:0:2 at reuN2-desktop] flushing 3 put command(s) out [proxy:0:2 at reuN2-desktop] forwarding command (cmd=put Pbusinesscard-2=description#192.168.2.11$port#44085$ifname#192.168.2.11$ Pbusinesscard-5=description#192.168.2.11$port#56233$ifname#192.168.2.11$ Pbusinesscard-8=description#192.168.2.11$port#39723$ifname#192.168.2.11$) upstream [proxy:0:2 at reuN2-desktop] forwarding command (cmd=barrier_in) upstream [mpiexec at reuOrin-desktop] [pgid: 0] got PMI command: cmd=barrier_in [proxy:0:1 at reu-desktop] PMI response: cmd=barrier_out [proxy:0:1 at reu-desktop] PMI response: cmd=barrier_out [proxy:0:1 at reu-desktop] PMI response: cmd=barrier_out [proxy:0:1 at reu-desktop] got pmi command (from 5): get kvsname=kvs_23478_0 key=sharedFilename-1 [proxy:0:1 at reu-desktop] PMI response: cmd=get_result rc=0 msg=success value=/dev/shm/mpich_shar_tmpmhgotU [proxy:0:1 at reu-desktop] got pmi command (from 6): get kvsname=kvs_23478_0 key=sharedFilename-1 [proxy:0:1 at reu-desktop] PMI response: cmd=get_result rc=0 msg=success value=/dev/shm/mpich_shar_tmpmhgotU [proxy:0:1 at reu-desktop] got pmi command (from 4): put kvsname=kvs_23478_0 key=Pbusinesscard-1 value=description#192.168.2.10$port#57871$ifname#192.168.2.10$ [proxy:0:1 at reu-desktop] cached command: Pbusinesscard-1=description#192.168.2.10$port#57871$ifname#192.168.2.10$ [proxy:0:1 at reu-desktop] PMI response: cmd=put_result rc=0 msg=success [proxy:0:1 at reu-desktop] got pmi command (from 5): put kvsname=kvs_23478_0 key=Pbusinesscard-4 value=description#192.168.2.10$port#56219$ifname#192.168.2.10$ [proxy:0:1 at reu-desktop] cached command: Pbusinesscard-4=description#192.168.2.10$port#56219$ifname#192.168.2.10$ [proxy:0:1 at reu-desktop] PMI response: cmd=put_result rc=0 msg=success [mpiexec at reuOrin-desktop] [pgid: 0] got PMI command: cmd=put Pbusinesscard-1=description#192.168.2.10$port#57871$ifname#192.168.2.10$ Pbusinesscard-4=description#192.168.2.10$port#56219$ifname#192.168.2.10$ Pbusinesscard-7=description#192.168.2.10$port#52015$ifname#192.168.2.10$ [proxy:0:1 at reu-desktop] got pmi command (from 6): put kvsname=kvs_23478_0 key=Pbusinesscard-7 value=description#192.168.2.10$port#52015$ifname#192.168.2.10$ [proxy:0:1 at reu-desktop] cached command: Pbusinesscard-7=description#192.168.2.10$port#52015$ifname#192.168.2.10$ [proxy:0:1 at reu-desktop] PMI response: cmd=put_result rc=0 msg=success [proxy:0:1 at reu-desktop] got pmi command (from 5): barrier_in [proxy:0:1 at reu-desktop] got pmi command (from 4): barrier_in [proxy:0:1 at reu-desktop] got pmi command (from 6): barrier_in [proxy:0:1 at reu-desktop] flushing 3 put command(s) out [proxy:0:1 at reu-desktop] forwarding command (cmd=put Pbusinesscard-1=description#192.168.2.10$port#57871$ifname#192.168.2.10$ Pbusinesscard-4=description#192.168.2.10$port#56219$ifname#192.168.2.10$ Pbusinesscard-7=description#192.168.2.10$port#52015$ifname#192.168.2.10$) upstream [proxy:0:1 at reu-desktop] forwarding command (cmd=barrier_in) upstream [mpiexec at reuOrin-desktop] [pgid: 0] got PMI command: cmd=barrier_in [mpiexec at reuOrin-desktop] PMI response to fd 6 pid 6: cmd=keyval_cache Pbusinesscard-0=description#192.168.2.20$port#56375$ifname#192.168.2.20$ Pbusinesscard-3=description#192.168.2.20$port#42255$ifname#192.168.2.20$ Pbusinesscard-6=description#192.168.2.20$port#54961$ifname#192.168.2.20$ Pbusinesscard-9=description#192.168.2.20$port#55841$ifname#192.168.2.20$ Pbusinesscard-2=description#192.168.2.11$port#44085$ifname#192.168.2.11$ Pbusinesscard-5=description#192.168.2.11$port#56233$ifname#192.168.2.11$ Pbusinesscard-8=description#192.168.2.11$port#39723$ifname#192.168.2.11$ Pbusinesscard-1=description#192.168.2.10$port#57871$ifname#192.168.2.10$ Pbusinesscard-4=description#192.168.2.10$port#56219$ifname#192.168.2.10$ Pbusinesscard-7=description#192.168.2.10$port#52015$ifname#192.168.2.10$ [mpiexec at reuOrin-desktop] PMI response to fd 7 pid 6: cmd=keyval_cache Pbusinesscard-0=description#192.168.2.20$port#56375$ifname#192.168.2.20$ Pbusinesscard-3=description#192.168.2.20$port#42255$ifname#192.168.2.20$ Pbusinesscard-6=description#192.168.2.20$port#54961$ifname#192.168.2.20$ Pbusinesscard-9=description#192.168.2.20$port#55841$ifname#192.168.2.20$ Pbusinesscard-2=description#192.168.2.11$port#44085$ifname#192.168.2.11$ Pbusinesscard-5=description#192.168.2.11$port#56233$ifname#192.168.2.11$ Pbusinesscard-8=description#192.168.2.11$port#39723$ifname#192.168.2.11$ Pbusinesscard-1=description#192.168.2.10$port#57871$ifname#192.168.2.10$ Pbusinesscard-4=description#192.168.2.10$port#56219$ifname#192.168.2.10$ Pbusinesscard-7=description#192.168.2.10$port#52015$ifname#192.168.2.10$ [mpiexec at reuOrin-desktop] PMI response to fd 15 pid 6: cmd=keyval_cache Pbusinesscard-0=description#192.168.2.20$port#56375$ifname#192.168.2.20$ Pbusinesscard-3=description#192.168.2.20$port#42255$ifname#192.168.2.20$ Pbusinesscard-6=description#192.168.2.20$port#54961$ifname#192.168.2.20$ Pbusinesscard-9=description#192.168.2.20$port#55841$ifname#192.168.2.20$ Pbusinesscard-2=description#192.168.2.11$port#44085$ifname#192.168.2.11$ Pbusinesscard-5=description#192.168.2.11$port#56233$ifname#192.168.2.11$ Pbusinesscard-8=description#192.168.2.11$port#39723$ifname#192.168.2.11$ Pbusinesscard-1=description#192.168.2.10$port#57871$ifname#192.168.2.10$ Pbusinesscard-4=description#192.168.2.10$port#56219$ifname#192.168.2.10$ Pbusinesscard-7=description#192.168.2.10$port#52015$ifname#192.168.2.10$ [mpiexec at reuOrin-desktop] PMI response to fd 6 pid 6: cmd=barrier_out [mpiexec at reuOrin-desktop] PMI response to fd 7 pid 6: cmd=barrier_out [mpiexec at reuOrin-desktop] PMI response to fd 15 pid 6: cmd=barrier_out [proxy:0:0 at reuOrin-desktop] PMI response: cmd=barrier_out [proxy:0:0 at reuOrin-desktop] PMI response: cmd=barrier_out [proxy:0:0 at reuOrin-desktop] PMI response: cmd=barrier_out [proxy:0:0 at reuOrin-desktop] PMI response: cmd=barrier_out [proxy:0:1 at reu-desktop] PMI response: cmd=barrier_out [proxy:0:1 at reu-desktop] PMI response: cmd=barrier_out [proxy:0:2 at reuN2-desktop] PMI response: cmd=barrier_out [proxy:0:2 at reuN2-desktop] PMI response: cmd=barrier_out [proxy:0:2 at reuN2-desktop] PMI response: cmd=barrier_out [proxy:0:1 at reu-desktop] PMI response: cmd=barrier_out Hello World from Process 9 of 10 Hello World from Process 0 of 10 Hello World from Process 3 of 10 Hello World from Process 6 of 10 [proxy:0:0 at reuOrin-desktop] got pmi command (from 0): finalize [proxy:0:0 at reuOrin-desktop] PMI response: cmd=finalize_ack [proxy:0:0 at reuOrin-desktop] got pmi command (from 8): finalize [proxy:0:0 at reuOrin-desktop] PMI response: cmd=finalize_ack [proxy:0:0 at reuOrin-desktop] got pmi command (from 13): finalize [proxy:0:0 at reuOrin-desktop] PMI response: cmd=finalize_ack [proxy:0:0 at reuOrin-desktop] got pmi command (from 6): finalize [proxy:0:0 at reuOrin-desktop] PMI response: cmd=finalize_ack Hello World from Process 4 of 10 Hello World from Process 2 of 10 Hello World from Process 8 of 10 Hello World from Process 1 of 10 Hello World from Process 7 of 10 [proxy:0:1 at reu-desktop] got pmi command (from 4): finalize [proxy:0:1 at reu-desktop] PMI response: cmd=finalize_ack [proxy:0:1 at reu-desktop] got pmi command (from 6): finalize [proxy:0:1 at reu-desktop] PMI response: cmd=finalize_ack [proxy:0:1 at reu-desktop] got pmi command (from 5): finalize [proxy:0:1 at reu-desktop] PMI response: cmd=finalize_ack Hello World from Process 5 of 10 [proxy:0:2 at reuN2-desktop] got pmi command (from 5): finalize [proxy:0:2 at reuN2-desktop] PMI response: cmd=finalize_ack [proxy:0:2 at reuN2-desktop] got pmi command (from 6): finalize [proxy:0:2 at reuN2-desktop] PMI response: cmd=finalize_ack [proxy:0:2 at reuN2-desktop] got pmi command (from 4): finalize [proxy:0:2 at reuN2-desktop] PMI response: cmd=finalize_ack From panda at cse.ohio-state.edu Sat Jul 15 09:57:49 2023 From: panda at cse.ohio-state.edu (Panda, Dhabaleswar) Date: Sat, 15 Jul 2023 13:57:49 +0000 Subject: [Mvapich] Join the MVAPICH team for multiple events at PEARC '23 In-Reply-To: References: Message-ID: The MVAPICH team members will be participating in multiple events at the PEARC '23 conference, to be held in Oregon, USA, during July 23-27, 2023. More details of the events are provided at: https://mvapich.cse.ohio-state.edu/conference/937/talks/ Join us for these presentations and interact with the project team members!! Thanks, The MVAPICH Team From panda at cse.ohio-state.edu Wed Jul 19 20:36:24 2023 From: panda at cse.ohio-state.edu (Panda, Dhabaleswar) Date: Thu, 20 Jul 2023 00:36:24 +0000 Subject: [Mvapich] Announcing the release of MVAPICH-Plus 3.0a2 Message-ID: The MVAPICH team is pleased to announce the release of MVAPICH-Plus 3.0a2. The new MVAPICH-Plus series is an advanced version of the MVAPICH MPI library. It is targeted to support unified MVAPICH2-GDR and MVAPICH2-X features. It is also targeted to provide optimized support for modern platforms (CPU, GPU, and interconnects) for HPC, Deep Learning, Machine Learning, Big Data and Data Science applications. The major features and enhancements available in MVAPICH-Plus 3.0a2 are as follows: - Based on MVAPICH 3.0 - Support for various high-performance communication fabrics - InfiniBand, Slingshot-10/11, Omni-Path, OPX, RoCE, and Ethernet - Support naive CPU staging approach for collectives for small messages - Tune naive limits for the following systems - Pitzer at OSC, Owens at OSC, Ascend at OSC, Frontera at TACC, Lonestar6 at TACC, ThetaGPU at ALCF, Polaris at ALCF, Tioga at LLNL - Initial support for blocking collectives on NVIDIA and AMD GPUs - Reduce_local, Reduce_scatter_block - Initial support for blocking collectives on NVIDIA and AMD GPUs - Allgather, Allgatherv, Allreduce, Alltoall, Alltoallv, Bcast, Gather, Gatherv, Reduce, Reduce_scatter, Scatter, Scatterv - Initial support for non-blocking GPU collectives on NVIDIA and AMD GPUs - Iallgather, Iallgatherv, Iallreduce, Ialltoall, Ialltoallv, Ibcast, Igather, Igatherv, Ireduce, Ireduce_scatter, Iscatter, Iscatterv - Initial support for blocking GPU to GPU point-to-point operations on NVIDIA and AMD GPUs - Send, Recv - Alpha support for non-blocking GPU to GPU point-to-point operations on NVIDIA and AMD GPUs - Isend, Irecv - Tested with - Various HPC applications, mini-applications, and benchmarks - MPI4cuML (a custom cuML package with MPI support) - Tested with CUDA <= 11.6 and CUDA 12.0 - Tested with ROCM <= 5.6.0 For downloading MVAPICH-Plus 3.0a2 library and associated user guide, please visit the following URL: http://mvapich.cse.ohio-state.edu All questions, feedback, bug reports, hints for performance tuning, patches, and enhancements are welcome. Please post it to the mvapich-discuss mailing list (mvapich-discuss at lists.osu.edu). Thanks, The MVAPICH Team PS: We are also happy to inform that the number of organizations using MVAPICH2 libraries (and registered at the MVAPICH site) has crossed 3,325 worldwide (in 90 countries). The number of downloads from the MVAPICH site has crossed 1,691,000 (1.69 million). The MVAPICH team would like to thank all its users and organizations!! From panda at cse.ohio-state.edu Thu Jul 27 23:22:08 2023 From: panda at cse.ohio-state.edu (Panda, Dhabaleswar) Date: Fri, 28 Jul 2023 03:22:08 +0000 Subject: [Mvapich] Online attendance for the MUG '23 conference is now free Message-ID: The MVAPICH User Group (MUG) conference organizers have put together an excellent program for the 11th annual MUG '23 conference. The conference will be held during August 21-23, 2023. It will be a hybrid event. The preliminary program is available from http://mug.mvapich.cse.ohio-state.edu/program/ Thanks to multiple organizations (Broadcom, Cornelis Networks, US-National Science Foundation (NSF), NVIDIA, Ohio Supercomputer Center, Ohio State University, Ohio State University/Translational Data Analytics Institute, ParaTools, and X-ScaleSolutions) for extending sponsorships to this conference!! These sponsorships have helped us to waive the registration fee for all online attendees (non-speakers). All interested parties (faculty, students, engineers, software developers, managers, etc.) can now attend the conference online for free. Please register for the conference using the registration link available from http://mug.mvapich.cse.ohio-state.edu/registration/ The Zoom link will be sent to all registered attendees a few days before the conference starts. If you have any questions, please send a note to mug at cse.ohio-state.edu. Thanks, The MUG '23 Organizers