[Mvapich] Fwd: Mvapich2-2.3.7 issue
Lei, Ryan
rlei at vols.utk.edu
Tue Jul 11 10:00:36 EDT 2023
---------- Forwarded message ---------
From: Lei, Ryan <rlei at vols.utk.edu>
Date: Mon, Jul 10, 2023 at 9:53 AM
Subject: Mvapich2-2.3.7 issue
To: mvapich-discuss at lists.osu.edu <mvapich-discuss at lists.osu.edu>
Hello Sir/Madam,
I am facing some mpirun error when I try to run the code based on my own
cluster.
I have 2 jetson nano as slave node and one jetson orin as master node, each
have installed the software correctly. But there is an issue that when I do
"mpirun -n # -hosts ip(master),ip(slave)./code" , everything works
successfully and correctly. However, when I try only work on the slave
node, take out the master ip in the -hosts, everything will fail and have
error shows below:
[image: error.jpg]
Even if I try to fix the firewalls based on the command, it still is not
working. MayI ask why and how do I fix it? Ps: more detailed output provide
by .txt.
Hope to hear from you soon.
sincerely,
Ryan Lei
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osu.edu/pipermail/mvapich/attachments/20230711/ae80d3e1/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: error.jpg
Type: image/jpeg
Size: 261935 bytes
Desc: not available
URL: <http://lists.osu.edu/pipermail/mvapich/attachments/20230711/ae80d3e1/attachment-0001.jpg>
-------------- next part --------------
host: 192.168.2.10
host: 192.168.2.11
==================================================================================================
mpiexec options:
----------------
Base path: /usr/local/bin/
Launcher: (null)
Debug level: 1
Enable X: -1
Global environment:
-------------------
SHELL=/bin/bash
SESSION_MANAGER=local/reuOrin-desktop:@/tmp/.ICE-unix/3043,unix/reuOrin-desktop:/tmp/.ICE-unix/3043
QT_ACCESSIBILITY=1
COLORTERM=truecolor
XDG_CONFIG_DIRS=/etc/xdg/xdg-ubuntu:/etc/xdg
XDG_MENU_PREFIX=gnome-
GNOME_DESKTOP_SESSION_ID=this-is-deprecated
GNOME_SHELL_SESSION_MODE=ubuntu
SSH_AUTH_SOCK=/run/user/1000/keyring/ssh
XMODIFIERS=@im=ibus
DESKTOP_SESSION=ubuntu
SSH_AGENT_PID=3007
GTK_MODULES=gail:atk-bridge
PWD=/home/reu
LOGNAME=reu
XDG_SESSION_DESKTOP=ubuntu
XDG_SESSION_TYPE=x11
GPG_AGENT_INFO=/run/user/1000/gnupg/S.gpg-agent:0:1
XAUTHORITY=/run/user/1000/gdm/Xauthority
GJS_DEBUG_TOPICS=JS ERROR;JS LOG
WINDOWPATH=2
HOME=/home/reu
USERNAME=reu
IM_CONFIG_PHASE=1
LANG=en_US.UTF-8
LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=00:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.zst=01;31:*.tzst=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.wim=01;31:*.swm=01;31:*.dwm=01;31:*.esd=01;31:*.jpg=01;35:*.jpeg=01;35:*.mjpg=01;35:*.mjpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.oga=00;36:*.opus=00;36:*.spx=00;36:*.xspf=00;36:
XDG_CURRENT_DESKTOP=ubuntu:GNOME
VTE_VERSION=6003
GNOME_TERMINAL_SCREEN=/org/gnome/Terminal/screen/59a28a14_f606_4ca7_bd10_beeeddc46a21
INVOCATION_ID=877fc76f797f4911acd1380fe960344b
MANAGERPID=2830
GJS_DEBUG_OUTPUT=stderr
LESSCLOSE=/usr/bin/lesspipe %s %s
XDG_SESSION_CLASS=user
TERM=xterm-256color
LESSOPEN=| /usr/bin/lesspipe %s
USER=reu
GNOME_TERMINAL_SERVICE=:1.102
SHLVL=1
QT_IM_MODULE=ibus
LD_LIBRARY_PATH=/usr/local/cuda/lib64:
XDG_RUNTIME_DIR=/run/user/1000
CUDA_HOME=/usr/local/cuda
JOURNAL_STREAM=7:58083
XDG_DATA_DIRS=/usr/share/ubuntu:/usr/local/share/:/usr/share/:/var/lib/snapd/desktop
PATH=/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
GDMSESSION=ubuntu
DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1000/bus
_=/usr/local/bin/mpirun
Hydra internal environment:
---------------------------
GFORTRAN_UNBUFFERED_PRECONNECTED=y
Proxy information:
*********************
[1] proxy: 192.168.2.10 (1 cores)
Exec list: ./hello (1 processes); ./hello (1 processes); ./hello (1 processes); ./hello (1 processes); ./hello (1 processes);
[2] proxy: 192.168.2.11 (1 cores)
Exec list: ./hello (1 processes); ./hello (1 processes); ./hello (1 processes); ./hello (1 processes); ./hello (1 processes);
==================================================================================================
[mpiexec at reuOrin-desktop] Timeout set to -1 (-1 means infinite)
[mpiexec at reuOrin-desktop] Got a control port string of reuOrin-desktop:40197
Proxy launch args: /usr/local/bin/hydra_pmi_proxy --control-port reuOrin-desktop:40197 --debug --rmk user --launcher ssh --demux poll --pgid 0 --retries 10 --usize -2 --proxy-id
Arguments being passed to proxy 0:
--version 3.2.1 --iface-ip-env-name MPIR_CVAR_CH3_INTERFACE_HOSTNAME --hostname 192.168.2.10 --global-core-map 0,1,2 --pmi-id-map 0,0 --global-process-count 10 --auto-cleanup 1 --pmi-kvsname kvs_25221_0 --pmi-process-mapping (vector,(0,2,1)) --ckpoint-num -1 --global-inherited-env 49 'SHELL=/bin/bash' 'SESSION_MANAGER=local/reuOrin-desktop:@/tmp/.ICE-unix/3043,unix/reuOrin-desktop:/tmp/.ICE-unix/3043' 'QT_ACCESSIBILITY=1' 'COLORTERM=truecolor' 'XDG_CONFIG_DIRS=/etc/xdg/xdg-ubuntu:/etc/xdg' 'XDG_MENU_PREFIX=gnome-' 'GNOME_DESKTOP_SESSION_ID=this-is-deprecated' 'GNOME_SHELL_SESSION_MODE=ubuntu' 'SSH_AUTH_SOCK=/run/user/1000/keyring/ssh' 'XMODIFIERS=@im=ibus' 'DESKTOP_SESSION=ubuntu' 'SSH_AGENT_PID=3007' 'GTK_MODULES=gail:atk-bridge' 'PWD=/home/reu' 'LOGNAME=reu' 'XDG_SESSION_DESKTOP=ubuntu' 'XDG_SESSION_TYPE=x11' 'GPG_AGENT_INFO=/run/user/1000/gnupg/S.gpg-agent:0:1' 'XAUTHORITY=/run/user/1000/gdm/Xauthority' 'GJS_DEBUG_TOPICS=JS ERROR;JS LOG' 'WINDOWPATH=2' 'HOME=/home/reu' 'USERNAME=reu' 'IM_CONFIG_PHASE=1' 'LANG=en_US.UTF-8' 'LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=00:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.zst=01;31:*.tzst=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.wim=01;31:*.swm=01;31:*.dwm=01;31:*.esd=01;31:*.jpg=01;35:*.jpeg=01;35:*.mjpg=01;35:*.mjpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.oga=00;36:*.opus=00;36:*.spx=00;36:*.xspf=00;36:' 'XDG_CURRENT_DESKTOP=ubuntu:GNOME' 'VTE_VERSION=6003' 'GNOME_TERMINAL_SCREEN=/org/gnome/Terminal/screen/59a28a14_f606_4ca7_bd10_beeeddc46a21' 'INVOCATION_ID=877fc76f797f4911acd1380fe960344b' 'MANAGERPID=2830' 'GJS_DEBUG_OUTPUT=stderr' 'LESSCLOSE=/usr/bin/lesspipe %s %s' 'XDG_SESSION_CLASS=user' 'TERM=xterm-256color' 'LESSOPEN=| /usr/bin/lesspipe %s' 'USER=reu' 'GNOME_TERMINAL_SERVICE=:1.102' 'SHLVL=1' 'QT_IM_MODULE=ibus' 'LD_LIBRARY_PATH=/usr/local/cuda/lib64:' 'XDG_RUNTIME_DIR=/run/user/1000' 'CUDA_HOME=/usr/local/cuda' 'JOURNAL_STREAM=7:58083' 'XDG_DATA_DIRS=/usr/share/ubuntu:/usr/local/share/:/usr/share/:/var/lib/snapd/desktop' 'PATH=/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin' 'GDMSESSION=ubuntu' 'DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1000/bus' '_=/usr/local/bin/mpirun' --global-user-env 0 --global-system-env 1 'GFORTRAN_UNBUFFERED_PRECONNECTED=y' --proxy-core-count 1 --exec --exec-appnum 0 --exec-proc-count 1 --exec-local-env 0 --exec-wdir /home/reu --exec-args 1 ./hello --exec --exec-appnum 0 --exec-proc-count 1 --exec-local-env 0 --exec-wdir /home/reu --exec-args 1 ./hello --exec --exec-appnum 0 --exec-proc-count 1 --exec-local-env 0 --exec-wdir /home/reu --exec-args 1 ./hello --exec --exec-appnum 0 --exec-proc-count 1 --exec-local-env 0 --exec-wdir /home/reu --exec-args 1 ./hello --exec --exec-appnum 0 --exec-proc-count 1 --exec-local-env 0 --exec-wdir /home/reu --exec-args 1 ./hello
Arguments being passed to proxy 1:
--version 3.2.1 --iface-ip-env-name MPIR_CVAR_CH3_INTERFACE_HOSTNAME --hostname 192.168.2.11 --global-core-map 0,1,2 --pmi-id-map 0,1 --global-process-count 10 --auto-cleanup 1 --pmi-kvsname kvs_25221_0 --pmi-process-mapping (vector,(0,2,1)) --ckpoint-num -1 --global-inherited-env 49 'SHELL=/bin/bash' 'SESSION_MANAGER=local/reuOrin-desktop:@/tmp/.ICE-unix/3043,unix/reuOrin-desktop:/tmp/.ICE-unix/3043' 'QT_ACCESSIBILITY=1' 'COLORTERM=truecolor' 'XDG_CONFIG_DIRS=/etc/xdg/xdg-ubuntu:/etc/xdg' 'XDG_MENU_PREFIX=gnome-' 'GNOME_DESKTOP_SESSION_ID=this-is-deprecated' 'GNOME_SHELL_SESSION_MODE=ubuntu' 'SSH_AUTH_SOCK=/run/user/1000/keyring/ssh' 'XMODIFIERS=@im=ibus' 'DESKTOP_SESSION=ubuntu' 'SSH_AGENT_PID=3007' 'GTK_MODULES=gail:atk-bridge' 'PWD=/home/reu' 'LOGNAME=reu' 'XDG_SESSION_DESKTOP=ubuntu' 'XDG_SESSION_TYPE=x11' 'GPG_AGENT_INFO=/run/user/1000/gnupg/S.gpg-agent:0:1' 'XAUTHORITY=/run/user/1000/gdm/Xauthority' 'GJS_DEBUG_TOPICS=JS ERROR;JS LOG' 'WINDOWPATH=2' 'HOME=/home/reu' 'USERNAME=reu' 'IM_CONFIG_PHASE=1' 'LANG=en_US.UTF-8' 'LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=00:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.zst=01;31:*.tzst=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.wim=01;31:*.swm=01;31:*.dwm=01;31:*.esd=01;31:*.jpg=01;35:*.jpeg=01;35:*.mjpg=01;35:*.mjpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.oga=00;36:*.opus=00;36:*.spx=00;36:*.xspf=00;36:' 'XDG_CURRENT_DESKTOP=ubuntu:GNOME' 'VTE_VERSION=6003' 'GNOME_TERMINAL_SCREEN=/org/gnome/Terminal/screen/59a28a14_f606_4ca7_bd10_beeeddc46a21' 'INVOCATION_ID=877fc76f797f4911acd1380fe960344b' 'MANAGERPID=2830' 'GJS_DEBUG_OUTPUT=stderr' 'LESSCLOSE=/usr/bin/lesspipe %s %s' 'XDG_SESSION_CLASS=user' 'TERM=xterm-256color' 'LESSOPEN=| /usr/bin/lesspipe %s' 'USER=reu' 'GNOME_TERMINAL_SERVICE=:1.102' 'SHLVL=1' 'QT_IM_MODULE=ibus' 'LD_LIBRARY_PATH=/usr/local/cuda/lib64:' 'XDG_RUNTIME_DIR=/run/user/1000' 'CUDA_HOME=/usr/local/cuda' 'JOURNAL_STREAM=7:58083' 'XDG_DATA_DIRS=/usr/share/ubuntu:/usr/local/share/:/usr/share/:/var/lib/snapd/desktop' 'PATH=/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin' 'GDMSESSION=ubuntu' 'DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1000/bus' '_=/usr/local/bin/mpirun' --global-user-env 0 --global-system-env 1 'GFORTRAN_UNBUFFERED_PRECONNECTED=y' --proxy-core-count 1 --exec --exec-appnum 0 --exec-proc-count 1 --exec-local-env 0 --exec-wdir /home/reu --exec-args 1 ./hello --exec --exec-appnum 0 --exec-proc-count 1 --exec-local-env 0 --exec-wdir /home/reu --exec-args 1 ./hello --exec --exec-appnum 0 --exec-proc-count 1 --exec-local-env 0 --exec-wdir /home/reu --exec-args 1 ./hello --exec --exec-appnum 0 --exec-proc-count 1 --exec-local-env 0 --exec-wdir /home/reu --exec-args 1 ./hello --exec --exec-appnum 0 --exec-proc-count 1 --exec-local-env 0 --exec-wdir /home/reu --exec-args 1 ./hello
[mpiexec at reuOrin-desktop] Launch arguments: /usr/bin/ssh -x 192.168.2.10 "/usr/local/bin/hydra_pmi_proxy" --control-port reuOrin-desktop:40197 --debug --rmk user --launcher ssh --demux poll --pgid 0 --retries 10 --usize -2 --proxy-id 0
[mpiexec at reuOrin-desktop] Launch arguments: /usr/bin/ssh -x 192.168.2.11 "/usr/local/bin/hydra_pmi_proxy" --control-port reuOrin-desktop:40197 --debug --rmk user --launcher ssh --demux poll --pgid 0 --retries 10 --usize -2 --proxy-id 1
-------------- next part --------------
host: 192.168.2.20
host: 192.168.2.10
host: 192.168.2.11
==================================================================================================
mpiexec options:
----------------
Base path: /usr/local/bin/
Launcher: (null)
Debug level: 1
Enable X: -1
Global environment:
-------------------
SHELL=/bin/bash
SESSION_MANAGER=local/reuOrin-desktop:@/tmp/.ICE-unix/3043,unix/reuOrin-desktop:/tmp/.ICE-unix/3043
QT_ACCESSIBILITY=1
COLORTERM=truecolor
XDG_CONFIG_DIRS=/etc/xdg/xdg-ubuntu:/etc/xdg
XDG_MENU_PREFIX=gnome-
GNOME_DESKTOP_SESSION_ID=this-is-deprecated
GNOME_SHELL_SESSION_MODE=ubuntu
SSH_AUTH_SOCK=/run/user/1000/keyring/ssh
XMODIFIERS=@im=ibus
DESKTOP_SESSION=ubuntu
SSH_AGENT_PID=3007
GTK_MODULES=gail:atk-bridge
PWD=/home/reu
LOGNAME=reu
XDG_SESSION_DESKTOP=ubuntu
XDG_SESSION_TYPE=x11
GPG_AGENT_INFO=/run/user/1000/gnupg/S.gpg-agent:0:1
XAUTHORITY=/run/user/1000/gdm/Xauthority
GJS_DEBUG_TOPICS=JS ERROR;JS LOG
WINDOWPATH=2
HOME=/home/reu
USERNAME=reu
IM_CONFIG_PHASE=1
LANG=en_US.UTF-8
LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=00:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.zst=01;31:*.tzst=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.wim=01;31:*.swm=01;31:*.dwm=01;31:*.esd=01;31:*.jpg=01;35:*.jpeg=01;35:*.mjpg=01;35:*.mjpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.oga=00;36:*.opus=00;36:*.spx=00;36:*.xspf=00;36:
XDG_CURRENT_DESKTOP=ubuntu:GNOME
VTE_VERSION=6003
GNOME_TERMINAL_SCREEN=/org/gnome/Terminal/screen/b34ccce0_720f_4f1a_9535_78b2ab4c39be
INVOCATION_ID=877fc76f797f4911acd1380fe960344b
MANAGERPID=2830
GJS_DEBUG_OUTPUT=stderr
LESSCLOSE=/usr/bin/lesspipe %s %s
XDG_SESSION_CLASS=user
TERM=xterm-256color
LESSOPEN=| /usr/bin/lesspipe %s
USER=reu
GNOME_TERMINAL_SERVICE=:1.102
SHLVL=1
QT_IM_MODULE=ibus
LD_LIBRARY_PATH=/usr/local/cuda/lib64:
XDG_RUNTIME_DIR=/run/user/1000
CUDA_HOME=/usr/local/cuda
JOURNAL_STREAM=7:58083
XDG_DATA_DIRS=/usr/share/ubuntu:/usr/local/share/:/usr/share/:/var/lib/snapd/desktop
PATH=/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
GDMSESSION=ubuntu
DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1000/bus
_=/usr/local/bin/mpirun
Hydra internal environment:
---------------------------
GFORTRAN_UNBUFFERED_PRECONNECTED=y
Proxy information:
*********************
[1] proxy: 192.168.2.20 (1 cores)
Exec list: ./hello (1 processes); ./hello (1 processes); ./hello (1 processes); ./hello (1 processes);
[2] proxy: 192.168.2.10 (1 cores)
Exec list: ./hello (1 processes); ./hello (1 processes); ./hello (1 processes);
[3] proxy: 192.168.2.11 (1 cores)
Exec list: ./hello (1 processes); ./hello (1 processes); ./hello (1 processes);
==================================================================================================
[mpiexec at reuOrin-desktop] Timeout set to -1 (-1 means infinite)
[mpiexec at reuOrin-desktop] Got a control port string of 192.168.2.20:33223
Proxy launch args: /usr/local/bin/hydra_pmi_proxy --control-port 192.168.2.20:33223 --debug --rmk user --launcher ssh --demux poll --pgid 0 --retries 10 --usize -2 --proxy-id
Arguments being passed to proxy 0:
--version 3.2.1 --iface-ip-env-name MPIR_CVAR_CH3_INTERFACE_HOSTNAME --hostname 192.168.2.20 --global-core-map 0,1,3 --pmi-id-map 0,0 --global-process-count 10 --auto-cleanup 1 --pmi-kvsname kvs_23478_0 --pmi-process-mapping (vector,(0,3,1)) --ckpoint-num -1 --global-inherited-env 49 'SHELL=/bin/bash' 'SESSION_MANAGER=local/reuOrin-desktop:@/tmp/.ICE-unix/3043,unix/reuOrin-desktop:/tmp/.ICE-unix/3043' 'QT_ACCESSIBILITY=1' 'COLORTERM=truecolor' 'XDG_CONFIG_DIRS=/etc/xdg/xdg-ubuntu:/etc/xdg' 'XDG_MENU_PREFIX=gnome-' 'GNOME_DESKTOP_SESSION_ID=this-is-deprecated' 'GNOME_SHELL_SESSION_MODE=ubuntu' 'SSH_AUTH_SOCK=/run/user/1000/keyring/ssh' 'XMODIFIERS=@im=ibus' 'DESKTOP_SESSION=ubuntu' 'SSH_AGENT_PID=3007' 'GTK_MODULES=gail:atk-bridge' 'PWD=/home/reu' 'LOGNAME=reu' 'XDG_SESSION_DESKTOP=ubuntu' 'XDG_SESSION_TYPE=x11' 'GPG_AGENT_INFO=/run/user/1000/gnupg/S.gpg-agent:0:1' 'XAUTHORITY=/run/user/1000/gdm/Xauthority' 'GJS_DEBUG_TOPICS=JS ERROR;JS LOG' 'WINDOWPATH=2' 'HOME=/home/reu' 'USERNAME=reu' 'IM_CONFIG_PHASE=1' 'LANG=en_US.UTF-8' 'LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=00:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.zst=01;31:*.tzst=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.wim=01;31:*.swm=01;31:*.dwm=01;31:*.esd=01;31:*.jpg=01;35:*.jpeg=01;35:*.mjpg=01;35:*.mjpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.oga=00;36:*.opus=00;36:*.spx=00;36:*.xspf=00;36:' 'XDG_CURRENT_DESKTOP=ubuntu:GNOME' 'VTE_VERSION=6003' 'GNOME_TERMINAL_SCREEN=/org/gnome/Terminal/screen/b34ccce0_720f_4f1a_9535_78b2ab4c39be' 'INVOCATION_ID=877fc76f797f4911acd1380fe960344b' 'MANAGERPID=2830' 'GJS_DEBUG_OUTPUT=stderr' 'LESSCLOSE=/usr/bin/lesspipe %s %s' 'XDG_SESSION_CLASS=user' 'TERM=xterm-256color' 'LESSOPEN=| /usr/bin/lesspipe %s' 'USER=reu' 'GNOME_TERMINAL_SERVICE=:1.102' 'SHLVL=1' 'QT_IM_MODULE=ibus' 'LD_LIBRARY_PATH=/usr/local/cuda/lib64:' 'XDG_RUNTIME_DIR=/run/user/1000' 'CUDA_HOME=/usr/local/cuda' 'JOURNAL_STREAM=7:58083' 'XDG_DATA_DIRS=/usr/share/ubuntu:/usr/local/share/:/usr/share/:/var/lib/snapd/desktop' 'PATH=/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin' 'GDMSESSION=ubuntu' 'DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1000/bus' '_=/usr/local/bin/mpirun' --global-user-env 0 --global-system-env 1 'GFORTRAN_UNBUFFERED_PRECONNECTED=y' --proxy-core-count 1 --exec --exec-appnum 0 --exec-proc-count 1 --exec-local-env 0 --exec-wdir /home/reu --exec-args 1 ./hello --exec --exec-appnum 0 --exec-proc-count 1 --exec-local-env 0 --exec-wdir /home/reu --exec-args 1 ./hello --exec --exec-appnum 0 --exec-proc-count 1 --exec-local-env 0 --exec-wdir /home/reu --exec-args 1 ./hello --exec --exec-appnum 0 --exec-proc-count 1 --exec-local-env 0 --exec-wdir /home/reu --exec-args 1 ./hello
Arguments being passed to proxy 1:
--version 3.2.1 --iface-ip-env-name MPIR_CVAR_CH3_INTERFACE_HOSTNAME --hostname 192.168.2.10 --global-core-map 0,1,3 --pmi-id-map 0,1 --global-process-count 10 --auto-cleanup 1 --pmi-kvsname kvs_23478_0 --pmi-process-mapping (vector,(0,3,1)) --ckpoint-num -1 --global-inherited-env 49 'SHELL=/bin/bash' 'SESSION_MANAGER=local/reuOrin-desktop:@/tmp/.ICE-unix/3043,unix/reuOrin-desktop:/tmp/.ICE-unix/3043' 'QT_ACCESSIBILITY=1' 'COLORTERM=truecolor' 'XDG_CONFIG_DIRS=/etc/xdg/xdg-ubuntu:/etc/xdg' 'XDG_MENU_PREFIX=gnome-' 'GNOME_DESKTOP_SESSION_ID=this-is-deprecated' 'GNOME_SHELL_SESSION_MODE=ubuntu' 'SSH_AUTH_SOCK=/run/user/1000/keyring/ssh' 'XMODIFIERS=@im=ibus' 'DESKTOP_SESSION=ubuntu' 'SSH_AGENT_PID=3007' 'GTK_MODULES=gail:atk-bridge' 'PWD=/home/reu' 'LOGNAME=reu' 'XDG_SESSION_DESKTOP=ubuntu' 'XDG_SESSION_TYPE=x11' 'GPG_AGENT_INFO=/run/user/1000/gnupg/S.gpg-agent:0:1' 'XAUTHORITY=/run/user/1000/gdm/Xauthority' 'GJS_DEBUG_TOPICS=JS ERROR;JS LOG' 'WINDOWPATH=2' 'HOME=/home/reu' 'USERNAME=reu' 'IM_CONFIG_PHASE=1' 'LANG=en_US.UTF-8' 'LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=00:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.zst=01;31:*.tzst=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.wim=01;31:*.swm=01;31:*.dwm=01;31:*.esd=01;31:*.jpg=01;35:*.jpeg=01;35:*.mjpg=01;35:*.mjpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.oga=00;36:*.opus=00;36:*.spx=00;36:*.xspf=00;36:' 'XDG_CURRENT_DESKTOP=ubuntu:GNOME' 'VTE_VERSION=6003' 'GNOME_TERMINAL_SCREEN=/org/gnome/Terminal/screen/b34ccce0_720f_4f1a_9535_78b2ab4c39be' 'INVOCATION_ID=877fc76f797f4911acd1380fe960344b' 'MANAGERPID=2830' 'GJS_DEBUG_OUTPUT=stderr' 'LESSCLOSE=/usr/bin/lesspipe %s %s' 'XDG_SESSION_CLASS=user' 'TERM=xterm-256color' 'LESSOPEN=| /usr/bin/lesspipe %s' 'USER=reu' 'GNOME_TERMINAL_SERVICE=:1.102' 'SHLVL=1' 'QT_IM_MODULE=ibus' 'LD_LIBRARY_PATH=/usr/local/cuda/lib64:' 'XDG_RUNTIME_DIR=/run/user/1000' 'CUDA_HOME=/usr/local/cuda' 'JOURNAL_STREAM=7:58083' 'XDG_DATA_DIRS=/usr/share/ubuntu:/usr/local/share/:/usr/share/:/var/lib/snapd/desktop' 'PATH=/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin' 'GDMSESSION=ubuntu' 'DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1000/bus' '_=/usr/local/bin/mpirun' --global-user-env 0 --global-system-env 1 'GFORTRAN_UNBUFFERED_PRECONNECTED=y' --proxy-core-count 1 --exec --exec-appnum 0 --exec-proc-count 1 --exec-local-env 0 --exec-wdir /home/reu --exec-args 1 ./hello --exec --exec-appnum 0 --exec-proc-count 1 --exec-local-env 0 --exec-wdir /home/reu --exec-args 1 ./hello --exec --exec-appnum 0 --exec-proc-count 1 --exec-local-env 0 --exec-wdir /home/reu --exec-args 1 ./hello
Arguments being passed to proxy 2:
--version 3.2.1 --iface-ip-env-name MPIR_CVAR_CH3_INTERFACE_HOSTNAME --hostname 192.168.2.11 --global-core-map 0,1,3 --pmi-id-map 0,2 --global-process-count 10 --auto-cleanup 1 --pmi-kvsname kvs_23478_0 --pmi-process-mapping (vector,(0,3,1)) --ckpoint-num -1 --global-inherited-env 49 'SHELL=/bin/bash' 'SESSION_MANAGER=local/reuOrin-desktop:@/tmp/.ICE-unix/3043,unix/reuOrin-desktop:/tmp/.ICE-unix/3043' 'QT_ACCESSIBILITY=1' 'COLORTERM=truecolor' 'XDG_CONFIG_DIRS=/etc/xdg/xdg-ubuntu:/etc/xdg' 'XDG_MENU_PREFIX=gnome-' 'GNOME_DESKTOP_SESSION_ID=this-is-deprecated' 'GNOME_SHELL_SESSION_MODE=ubuntu' 'SSH_AUTH_SOCK=/run/user/1000/keyring/ssh' 'XMODIFIERS=@im=ibus' 'DESKTOP_SESSION=ubuntu' 'SSH_AGENT_PID=3007' 'GTK_MODULES=gail:atk-bridge' 'PWD=/home/reu' 'LOGNAME=reu' 'XDG_SESSION_DESKTOP=ubuntu' 'XDG_SESSION_TYPE=x11' 'GPG_AGENT_INFO=/run/user/1000/gnupg/S.gpg-agent:0:1' 'XAUTHORITY=/run/user/1000/gdm/Xauthority' 'GJS_DEBUG_TOPICS=JS ERROR;JS LOG' 'WINDOWPATH=2' 'HOME=/home/reu' 'USERNAME=reu' 'IM_CONFIG_PHASE=1' 'LANG=en_US.UTF-8' 'LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=00:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.zst=01;31:*.tzst=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.wim=01;31:*.swm=01;31:*.dwm=01;31:*.esd=01;31:*.jpg=01;35:*.jpeg=01;35:*.mjpg=01;35:*.mjpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.oga=00;36:*.opus=00;36:*.spx=00;36:*.xspf=00;36:' 'XDG_CURRENT_DESKTOP=ubuntu:GNOME' 'VTE_VERSION=6003' 'GNOME_TERMINAL_SCREEN=/org/gnome/Terminal/screen/b34ccce0_720f_4f1a_9535_78b2ab4c39be' 'INVOCATION_ID=877fc76f797f4911acd1380fe960344b' 'MANAGERPID=2830' 'GJS_DEBUG_OUTPUT=stderr' 'LESSCLOSE=/usr/bin/lesspipe %s %s' 'XDG_SESSION_CLASS=user' 'TERM=xterm-256color' 'LESSOPEN=| /usr/bin/lesspipe %s' 'USER=reu' 'GNOME_TERMINAL_SERVICE=:1.102' 'SHLVL=1' 'QT_IM_MODULE=ibus' 'LD_LIBRARY_PATH=/usr/local/cuda/lib64:' 'XDG_RUNTIME_DIR=/run/user/1000' 'CUDA_HOME=/usr/local/cuda' 'JOURNAL_STREAM=7:58083' 'XDG_DATA_DIRS=/usr/share/ubuntu:/usr/local/share/:/usr/share/:/var/lib/snapd/desktop' 'PATH=/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin' 'GDMSESSION=ubuntu' 'DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1000/bus' '_=/usr/local/bin/mpirun' --global-user-env 0 --global-system-env 1 'GFORTRAN_UNBUFFERED_PRECONNECTED=y' --proxy-core-count 1 --exec --exec-appnum 0 --exec-proc-count 1 --exec-local-env 0 --exec-wdir /home/reu --exec-args 1 ./hello --exec --exec-appnum 0 --exec-proc-count 1 --exec-local-env 0 --exec-wdir /home/reu --exec-args 1 ./hello --exec --exec-appnum 0 --exec-proc-count 1 --exec-local-env 0 --exec-wdir /home/reu --exec-args 1 ./hello
[mpiexec at reuOrin-desktop] Launch arguments: /usr/local/bin/hydra_pmi_proxy --control-port 192.168.2.20:33223 --debug --rmk user --launcher ssh --demux poll --pgid 0 --retries 10 --usize -2 --proxy-id 0
[mpiexec at reuOrin-desktop] Launch arguments: /usr/bin/ssh -x 192.168.2.10 "/usr/local/bin/hydra_pmi_proxy" --control-port 192.168.2.20:33223 --debug --rmk user --launcher ssh --demux poll --pgid 0 --retries 10 --usize -2 --proxy-id 1
[mpiexec at reuOrin-desktop] Launch arguments: /usr/bin/ssh -x 192.168.2.11 "/usr/local/bin/hydra_pmi_proxy" --control-port 192.168.2.20:33223 --debug --rmk user --launcher ssh --demux poll --pgid 0 --retries 10 --usize -2 --proxy-id 2
[proxy:0:0 at reuOrin-desktop] got pmi command (from 0): init
pmi_version=1 pmi_subversion=1
[proxy:0:0 at reuOrin-desktop] PMI response: cmd=response_to_init pmi_version=1 pmi_subversion=1 rc=0
[proxy:0:0 at reuOrin-desktop] got pmi command (from 0): get_maxes
[proxy:0:0 at reuOrin-desktop] PMI response: cmd=maxes kvsname_max=256 keylen_max=64 vallen_max=1024
[proxy:0:0 at reuOrin-desktop] got pmi command (from 6): init
pmi_version=1 pmi_subversion=1
[proxy:0:0 at reuOrin-desktop] PMI response: cmd=response_to_init pmi_version=1 pmi_subversion=1 rc=0
[proxy:0:0 at reuOrin-desktop] got pmi command (from 8): init
pmi_version=1 pmi_subversion=1
[proxy:0:0 at reuOrin-desktop] PMI response: cmd=response_to_init pmi_version=1 pmi_subversion=1 rc=0
[proxy:0:0 at reuOrin-desktop] got pmi command (from 13): init
pmi_version=1 pmi_subversion=1
[proxy:0:0 at reuOrin-desktop] PMI response: cmd=response_to_init pmi_version=1 pmi_subversion=1 rc=0
[proxy:0:0 at reuOrin-desktop] got pmi command (from 0): get_appnum
[proxy:0:0 at reuOrin-desktop] PMI response: cmd=appnum appnum=0
[proxy:0:0 at reuOrin-desktop] got pmi command (from 6): get_maxes
[proxy:0:0 at reuOrin-desktop] PMI response: cmd=maxes kvsname_max=256 keylen_max=64 vallen_max=1024
[proxy:0:0 at reuOrin-desktop] got pmi command (from 8): get_maxes
[proxy:0:0 at reuOrin-desktop] PMI response: cmd=maxes kvsname_max=256 keylen_max=64 vallen_max=1024
[proxy:0:0 at reuOrin-desktop] got pmi command (from 0): get_my_kvsname
[proxy:0:0 at reuOrin-desktop] PMI response: cmd=my_kvsname kvsname=kvs_23478_0
[proxy:0:0 at reuOrin-desktop] got pmi command (from 6): get_appnum
[proxy:0:0 at reuOrin-desktop] PMI response: cmd=appnum appnum=0
[proxy:0:0 at reuOrin-desktop] got pmi command (from 13): get_maxes
[proxy:0:0 at reuOrin-desktop] PMI response: cmd=maxes kvsname_max=256 keylen_max=64 vallen_max=1024
[proxy:0:0 at reuOrin-desktop] got pmi command (from 0): get_my_kvsname
[proxy:0:0 at reuOrin-desktop] PMI response: cmd=my_kvsname kvsname=kvs_23478_0
[proxy:0:0 at reuOrin-desktop] got pmi command (from 6): get_my_kvsname
[proxy:0:0 at reuOrin-desktop] PMI response: cmd=my_kvsname kvsname=kvs_23478_0
[proxy:0:0 at reuOrin-desktop] got pmi command (from 8): get_appnum
[proxy:0:0 at reuOrin-desktop] PMI response: cmd=appnum appnum=0
[proxy:0:0 at reuOrin-desktop] got pmi command (from 0): get
kvsname=kvs_23478_0 key=PMI_process_mapping
[proxy:0:0 at reuOrin-desktop] PMI response: cmd=get_result rc=0 msg=success value=(vector,(0,3,1))
[proxy:0:0 at reuOrin-desktop] got pmi command (from 6): get_my_kvsname
[proxy:0:0 at reuOrin-desktop] PMI response: cmd=my_kvsname kvsname=kvs_23478_0
[proxy:0:0 at reuOrin-desktop] got pmi command (from 13): get_appnum
[proxy:0:0 at reuOrin-desktop] PMI response: cmd=appnum appnum=0
[proxy:0:0 at reuOrin-desktop] got pmi command (from 0): put
kvsname=kvs_23478_0 key=sharedFilename-0 value=/dev/shm/mpich_shar_tmpvgZseM
[proxy:0:0 at reuOrin-desktop] cached command: sharedFilename-0=/dev/shm/mpich_shar_tmpvgZseM
[proxy:0:0 at reuOrin-desktop] PMI response: cmd=put_result rc=0 msg=success
[proxy:0:0 at reuOrin-desktop] got pmi command (from 6): get
kvsname=kvs_23478_0 key=PMI_process_mapping
[proxy:0:0 at reuOrin-desktop] PMI response: cmd=get_result rc=0 msg=success value=(vector,(0,3,1))
[proxy:0:0 at reuOrin-desktop] got pmi command (from 8): get_my_kvsname
[proxy:0:0 at reuOrin-desktop] PMI response: cmd=my_kvsname kvsname=kvs_23478_0
[proxy:0:0 at reuOrin-desktop] got pmi command (from 0): barrier_in
[proxy:0:0 at reuOrin-desktop] got pmi command (from 6): barrier_in
[proxy:0:0 at reuOrin-desktop] got pmi command (from 13): get_my_kvsname
[proxy:0:0 at reuOrin-desktop] PMI response: cmd=my_kvsname kvsname=kvs_23478_0
[proxy:0:0 at reuOrin-desktop] got pmi command (from 8): get_my_kvsname
[proxy:0:0 at reuOrin-desktop] PMI response: cmd=my_kvsname kvsname=kvs_23478_0
[proxy:0:0 at reuOrin-desktop] got pmi command (from 13): get_my_kvsname
[proxy:0:0 at reuOrin-desktop] PMI response: cmd=my_kvsname kvsname=kvs_23478_0
[proxy:0:0 at reuOrin-desktop] got pmi command (from 8): get
kvsname=kvs_23478_0 key=PMI_process_mapping
[proxy:0:0 at reuOrin-desktop] PMI response: cmd=get_result rc=0 msg=success value=(vector,(0,3,1))
[proxy:0:0 at reuOrin-desktop] got pmi command (from 13): get
kvsname=kvs_23478_0 key=PMI_process_mapping
[proxy:0:0 at reuOrin-desktop] PMI response: cmd=get_result rc=0 msg=success value=(vector,(0,3,1))
[proxy:0:0 at reuOrin-desktop] got pmi command (from 8): barrier_in
[proxy:0:0 at reuOrin-desktop] got pmi command (from 13): barrier_in
[proxy:0:0 at reuOrin-desktop] flushing 1 put command(s) out
[proxy:0:0 at reuOrin-desktop] [mpiexec at reuOrin-desktop] [pgid: 0] got PMI command: cmd=put sharedFilename-0=/dev/shm/mpich_shar_tmpvgZseM
forwarding command (cmd=put sharedFilename-0=/dev/shm/mpich_shar_tmpvgZseM) upstream
[proxy:0:0 at reuOrin-desktop] forwarding command (cmd=barrier_in) upstream
[mpiexec at reuOrin-desktop] [pgid: 0] got PMI command: cmd=barrier_in
[proxy:0:1 at reu-desktop] got pmi command (from 4): init
pmi_version=1 pmi_subversion=1
[proxy:0:1 at reu-desktop] PMI response: cmd=response_to_init pmi_version=1 pmi_subversion=1 rc=0
[proxy:0:1 at reu-desktop] got pmi command (from 5): init
pmi_version=1 pmi_subversion=1
[proxy:0:1 at reu-desktop] PMI response: cmd=response_to_init pmi_version=1 pmi_subversion=1 rc=0
[proxy:0:1 at reu-desktop] got pmi command (from 4): get_maxes
[proxy:0:1 at reu-desktop] PMI response: cmd=maxes kvsname_max=256 keylen_max=64 vallen_max=1024
[proxy:0:1 at reu-desktop] got pmi command (from 6): init
pmi_version=1 pmi_subversion=1
[proxy:0:1 at reu-desktop] PMI response: cmd=response_to_init pmi_version=1 pmi_subversion=1 rc=0
[proxy:0:1 at reu-desktop] got pmi command (from 4): get_appnum
[proxy:0:1 at reu-desktop] PMI response: cmd=appnum appnum=0
[proxy:0:1 at reu-desktop] got pmi command (from 5): get_maxes
[proxy:0:1 at reu-desktop] PMI response: cmd=maxes kvsname_max=256 keylen_max=64 vallen_max=1024
[proxy:0:1 at reu-desktop] got pmi command (from 4): get_my_kvsname
[proxy:0:1 at reu-desktop] PMI response: cmd=my_kvsname kvsname=kvs_23478_0
[proxy:0:1 at reu-desktop] got pmi command (from 6): get_maxes
[proxy:0:1 at reu-desktop] PMI response: cmd=maxes kvsname_max=256 keylen_max=64 vallen_max=1024
[proxy:0:1 at reu-desktop] got pmi command (from 4): get_my_kvsname
[proxy:0:1 at reu-desktop] PMI response: cmd=my_kvsname kvsname=kvs_23478_0
[proxy:0:1 at reu-desktop] got pmi command (from 5): get_appnum
[proxy:0:1 at reu-desktop] PMI response: cmd=appnum appnum=0
[proxy:0:1 at reu-desktop] got pmi command (from 4): get
kvsname=kvs_23478_0 key=PMI_process_mapping
[proxy:0:1 at reu-desktop] PMI response: cmd=get_result rc=0 msg=success value=(vector,(0,3,1))
[proxy:0:1 at reu-desktop] got pmi command (from 5): get_my_kvsname
[proxy:0:1 at reu-desktop] PMI response: cmd=my_kvsname kvsname=kvs_23478_0
[proxy:0:1 at reu-desktop] got pmi command (from 6): get_appnum
[proxy:0:1 at reu-desktop] PMI response: cmd=appnum appnum=0
[proxy:0:1 at reu-desktop] got pmi command (from 4): put
kvsname=kvs_23478_0 key=sharedFilename-1 value=/dev/shm/mpich_shar_tmpmhgotU
[proxy:0:1 at reu-desktop] cached command: sharedFilename-1=/dev/shm/mpich_shar_tmpmhgotU
[proxy:0:1 at reu-desktop] PMI response: cmd=put_result rc=0 msg=success
[proxy:0:1 at reu-desktop] got pmi command (from 5): get_my_kvsname
[proxy:0:1 at reu-desktop] PMI response: cmd=my_kvsname kvsname=kvs_23478_0
[proxy:0:1 at reu-desktop] got pmi command (from 4): barrier_in
[proxy:0:1 at reu-desktop] got pmi command (from 6): get_my_kvsname
[proxy:0:1 at reu-desktop] PMI response: cmd=my_kvsname kvsname=kvs_23478_0
[proxy:0:1 at reu-desktop] got pmi command (from 5): get
kvsname=kvs_23478_0 key=PMI_process_mapping
[proxy:0:1 at reu-desktop] PMI response: cmd=get_result rc=0 msg=success value=(vector,(0,3,1))
[proxy:0:1 at reu-desktop] got pmi command (from 6): get_my_kvsname
[proxy:0:1 at reu-desktop] PMI response: cmd=my_kvsname kvsname=kvs_23478_0
[proxy:0:1 at reu-desktop] got pmi command (from 5): barrier_in
[proxy:0:1 at reu-desktop] got pmi command (from 6): get
kvsname=kvs_23478_0 key=PMI_process_mapping
[proxy:0:1 at reu-desktop] PMI response: cmd=get_result rc=0 msg=success value=(vector,(0,3,1))
[mpiexec at reuOrin-desktop] [pgid: 0] got PMI command: cmd=put sharedFilename-1=/dev/shm/mpich_shar_tmpmhgotU
[proxy:0:1 at reu-desktop] got pmi command (from 6): barrier_in
[proxy:0:1 at reu-desktop] flushing 1 put command(s) out
[proxy:0:1 at reu-desktop] forwarding command (cmd=put sharedFilename-1=/dev/shm/mpich_shar_tmpmhgotU) upstream
[proxy:0:1 at reu-desktop] forwarding command (cmd=barrier_in) upstream
[mpiexec at reuOrin-desktop] [pgid: 0] got PMI command: cmd=barrier_in
[proxy:0:2 at reuN2-desktop] got pmi command (from 4): init
pmi_version=1 pmi_subversion=1
[proxy:0:2 at reuN2-desktop] PMI response: cmd=response_to_init pmi_version=1 pmi_subversion=1 rc=0
[proxy:0:2 at reuN2-desktop] got pmi command (from 4): get_maxes
[proxy:0:2 at reuN2-desktop] PMI response: cmd=maxes kvsname_max=256 keylen_max=64 vallen_max=1024
[proxy:0:2 at reuN2-desktop] got pmi command (from 5): init
pmi_version=1 pmi_subversion=1
[proxy:0:2 at reuN2-desktop] PMI response: cmd=response_to_init pmi_version=1 pmi_subversion=1 rc=0
[proxy:0:2 at reuN2-desktop] got pmi command (from 6): init
pmi_version=1 pmi_subversion=1
[proxy:0:2 at reuN2-desktop] PMI response: cmd=response_to_init pmi_version=1 pmi_subversion=1 rc=0
[proxy:0:2 at reuN2-desktop] got pmi command (from 4): get_appnum
[proxy:0:2 at reuN2-desktop] PMI response: cmd=appnum appnum=0
[proxy:0:2 at reuN2-desktop] got pmi command (from 5): get_maxes
[proxy:0:2 at reuN2-desktop] PMI response: cmd=maxes kvsname_max=256 keylen_max=64 vallen_max=1024
[proxy:0:2 at reuN2-desktop] got pmi command (from 4): get_my_kvsname
[proxy:0:2 at reuN2-desktop] PMI response: cmd=my_kvsname kvsname=kvs_23478_0
[proxy:0:2 at reuN2-desktop] got pmi command (from 6): get_maxes
[proxy:0:2 at reuN2-desktop] PMI response: cmd=maxes kvsname_max=256 keylen_max=64 vallen_max=1024
[proxy:0:2 at reuN2-desktop] got pmi command (from 4): get_my_kvsname
[proxy:0:2 at reuN2-desktop] PMI response: cmd=my_kvsname kvsname=kvs_23478_0
[proxy:0:2 at reuN2-desktop] got pmi command (from 5): get_appnum
[proxy:0:2 at reuN2-desktop] PMI response: cmd=appnum appnum=0
[proxy:0:2 at reuN2-desktop] got pmi command (from 4): get
kvsname=kvs_23478_0 key=PMI_process_mapping
[proxy:0:2 at reuN2-desktop] PMI response: cmd=get_result rc=0 msg=success value=(vector,(0,3,1))
[proxy:0:2 at reuN2-desktop] got pmi command (from 5): get_my_kvsname
[proxy:0:2 at reuN2-desktop] PMI response: cmd=my_kvsname kvsname=kvs_23478_0
[proxy:0:2 at reuN2-desktop] got pmi command (from 6): get_appnum
[proxy:0:2 at reuN2-desktop] PMI response: cmd=appnum appnum=0
[proxy:0:2 at reuN2-desktop] got pmi command (from 4): put
kvsname=kvs_23478_0 key=sharedFilename-2 value=/dev/shm/mpich_shar_tmpwPuw7j
[proxy:0:2 at reuN2-desktop] cached command: sharedFilename-2=/dev/shm/mpich_shar_tmpwPuw7j
[proxy:0:2 at reuN2-desktop] PMI response: cmd=put_result rc=0 msg=success
[proxy:0:2 at reuN2-desktop] got pmi command (from 5): get_my_kvsname
[proxy:0:2 at reuN2-desktop] PMI response: cmd=my_kvsname kvsname=kvs_23478_0
[mpiexec at reuOrin-desktop] [pgid: 0] got PMI command: cmd=put sharedFilename-2=/dev/shm/mpich_shar_tmpwPuw7j
[proxy:0:2 at reuN2-desktop] got pmi command (from 4): barrier_in
[proxy:0:2 at reuN2-desktop] got pmi command (from 6): get_my_kvsname
[proxy:0:2 at reuN2-desktop] PMI response: cmd=my_kvsname kvsname=kvs_23478_0
[proxy:0:2 at reuN2-desktop] got pmi command (from 5): get
kvsname=kvs_23478_0 key=PMI_process_mapping
[proxy:0:2 at reuN2-desktop] PMI response: cmd=get_result rc=0 msg=success value=(vector,(0,3,1))
[proxy:0:2 at reuN2-desktop] got pmi command (from 6): get_my_kvsname
[proxy:0:2 at reuN2-desktop] PMI response: cmd=my_kvsname kvsname=kvs_23478_0
[proxy:0:2 at reuN2-desktop] got pmi command (from 5): barrier_in
[proxy:0:2 at reuN2-desktop] got pmi command (from 6): get
kvsname=kvs_23478_0 key=PMI_process_mapping
[proxy:0:2 at reuN2-desktop] PMI response: cmd=get_result rc=0 msg=success value=(vector,(0,3,1))
[proxy:0:2 at reuN2-desktop] got pmi command (from 6): barrier_in
[proxy:0:2 at reuN2-desktop] flushing 1 put command(s) out
[proxy:0:2 at reuN2-desktop] forwarding command (cmd=put sharedFilename-2=/dev/shm/mpich_shar_tmpwPuw7j) upstream
[proxy:0:2 at reuN2-desktop] forwarding command (cmd=barrier_in) upstream
[mpiexec at reuOrin-desktop] [pgid: 0] got PMI command: cmd=barrier_in
[mpiexec at reuOrin-desktop] PMI response to fd 6 pid 6: cmd=keyval_cache sharedFilename-0=/dev/shm/mpich_shar_tmpvgZseM sharedFilename-1=/dev/shm/mpich_shar_tmpmhgotU sharedFilename-2=/dev/shm/mpich_shar_tmpwPuw7j
[mpiexec at reuOrin-desktop] PMI response to fd 7 pid 6: cmd=keyval_cache sharedFilename-0=/dev/shm/mpich_shar_tmpvgZseM sharedFilename-1=/dev/shm/mpich_shar_tmpmhgotU sharedFilename-2=/dev/shm/mpich_shar_tmpwPuw7j
[mpiexec at reuOrin-desktop] PMI response to fd 15 pid 6: cmd=keyval_cache sharedFilename-0=/dev/shm/mpich_shar_tmpvgZseM sharedFilename-1=/dev/shm/mpich_shar_tmpmhgotU sharedFilename-2=/dev/shm/mpich_shar_tmpwPuw7j
[mpiexec at reuOrin-desktop] PMI response to fd 6 pid 6: cmd=barrier_out
[mpiexec at reuOrin-desktop] PMI response to fd 7 pid 6: cmd=barrier_out
[mpiexec at reuOrin-desktop] PMI response to fd 15 pid 6: cmd=barrier_out
[proxy:0:0 at reuOrin-desktop] PMI response: cmd=barrier_out
[proxy:0:0 at reuOrin-desktop] PMI response: cmd=barrier_out
[proxy:0:0 at reuOrin-desktop] PMI response: cmd=barrier_out
[proxy:0:0 at reuOrin-desktop] PMI response: cmd=barrier_out
[proxy:0:0 at reuOrin-desktop] got pmi command (from 6): get
kvsname=kvs_23478_0 key=sharedFilename-0
[proxy:0:0 at reuOrin-desktop] PMI response: cmd=get_result rc=0 msg=success value=/dev/shm/mpich_shar_tmpvgZseM
[proxy:0:0 at reuOrin-desktop] got pmi command (from 8): get
kvsname=kvs_23478_0 key=sharedFilename-0
[proxy:0:0 at reuOrin-desktop] PMI response: cmd=get_result rc=0 msg=success value=/dev/shm/mpich_shar_tmpvgZseM
[proxy:0:0 at reuOrin-desktop] got pmi command (from 13): get
kvsname=kvs_23478_0 key=sharedFilename-0
[proxy:0:0 at reuOrin-desktop] PMI response: cmd=get_result rc=0 msg=success value=/dev/shm/mpich_shar_tmpvgZseM
[proxy:0:2 at reuN2-desktop] PMI response: cmd=barrier_out
[proxy:0:2 at reuN2-desktop] PMI response: cmd=barrier_out
[proxy:0:2 at reuN2-desktop] PMI response: cmd=barrier_out
[proxy:0:2 at reuN2-desktop] got pmi command (from 5): get
kvsname=kvs_23478_0 key=sharedFilename-2
[proxy:0:2 at reuN2-desktop] PMI response: cmd=get_result rc=0 msg=success value=/dev/shm/mpich_shar_tmpwPuw7j
[proxy:0:2 at reuN2-desktop] got pmi command (from 6): get
kvsname=kvs_23478_0 key=sharedFilename-2
[proxy:0:2 at reuN2-desktop] PMI response: cmd=get_result rc=0 msg=success value=/dev/shm/mpich_shar_tmpwPuw7j
[proxy:0:0 at reuOrin-desktop] got pmi command (from 0): put
kvsname=kvs_23478_0 key=Pbusinesscard-0 value=description#192.168.2.20$port#56375$ifname#192.168.2.20$
[proxy:0:0 at reuOrin-desktop] cached command: Pbusinesscard-0=description#192.168.2.20$port#56375$ifname#192.168.2.20$
[proxy:0:0 at reuOrin-desktop] PMI response: cmd=put_result rc=0 msg=success
[proxy:0:0 at reuOrin-desktop] got pmi command (from 6): put
kvsname=kvs_23478_0 key=Pbusinesscard-3 value=description#192.168.2.20$port#42255$ifname#192.168.2.20$
[proxy:0:0 at reuOrin-desktop] cached command: Pbusinesscard-3=description#192.168.2.20$port#42255$ifname#192.168.2.20$
[proxy:0:0 at reuOrin-desktop] PMI response: cmd=put_result rc=0 msg=success
[proxy:0:0 at reuOrin-desktop] got pmi command (from 8): put
kvsname=kvs_23478_0 key=Pbusinesscard-6 value=description#192.168.2.20$port#54961$ifname#192.168.2.20$
[proxy:0:0 at reuOrin-desktop] cached command: Pbusinesscard-6=description#192.168.2.20$port#54961$ifname#192.168.2.20$
[proxy:0:0 at reuOrin-desktop] PMI response: cmd=put_result rc=0 msg=success
[proxy:0:0 at reuOrin-desktop] got pmi command (from 13): put
kvsname=kvs_23478_0 key=Pbusinesscard-9 value=description#192.168.2.20$port#55841$ifname#192.168.2.20$
[proxy:0:0 at reuOrin-desktop] cached command: Pbusinesscard-9=description#192.168.2.20$port#55841$ifname#192.168.2.20$
[proxy:0:0 at reuOrin-desktop] PMI response: cmd=put_result rc=0 msg=success
[proxy:0:0 at reuOrin-desktop] got pmi command (from 0): barrier_in
[proxy:0:0 at reuOrin-desktop] got pmi command (from 6): barrier_in
[proxy:0:0 at reuOrin-desktop] got pmi command (from 8): barrier_in
[proxy:0:2 at reuN2-desktop] got pmi command (from 4): put
kvsname=kvs_23478_0 key=Pbusinesscard-2 [proxy:0:0 at reuOrin-desktop] got pmi command (from 13): barrier_in
[proxy:0:0 at reuOrin-desktop] flushing 4 put command(s) out
[proxy:0:0 at reuOrin-desktop] forwarding command (cmd=put Pbusinesscard-0=description#192.168.2.20$port#56375$ifname#192.168.2.20$ Pbusinesscard-3=description#192.168.2.20$port#42255$ifname#192.168.2.20$ Pbusinesscard-6=description#192.168.2.20$port#54961$ifname#192.168.2.20$ Pbusinesscard-9=description#192.168.2.20$port#55841$ifname#192.168.2.20$) upstream
[mpiexec at reuOrin-desktop] [pgid: 0] got PMI command: cmd=put Pbusinesscard-0=description#192.168.2.20$port#56375$ifname#192.168.2.20$ Pbusinesscard-3=description#192.168.2.20$port#42255$ifname#192.168.2.20$ Pbusinesscard-6=description#192.168.2.20$port#54961$ifname#192.168.2.20$ Pbusinesscard-9=description#192.168.2.20$port#55841$ifname#192.168.2.20$
[proxy:0:0 at reuOrin-desktop] forwarding command (cmd=barrier_in) upstream
value=description#192.168.2.11$port#44085$ifname#192.168.2.11$
[proxy:0:2 at reuN2-desktop] cached command: Pbusinesscard-2=description#192.168.2.11$port#44085$ifname#192.168.2.11$
[proxy:0:2 at reuN2-desktop] PMI response: cmd=put_result rc=0 msg=success
[proxy:0:2 at reuN2-desktop] [mpiexec at reuOrin-desktop] [pgid: 0] got PMI command: cmd=barrier_in
got pmi command (from 5): put
kvsname=kvs_23478_0 key=Pbusinesscard-5 value=description#192.168.2.11$port#56233$ifname#192.168.2.11$
[proxy:0:2 at reuN2-desktop] cached command: Pbusinesscard-5=description#192.168.2.11$port#56233$ifname#192.168.2.11$
[proxy:0:2 at reuN2-desktop] PMI response: cmd=put_result rc=0 msg=success
[proxy:0:2 at reuN2-desktop] got pmi command (from 6): put
kvsname=kvs_23478_0 key=Pbusinesscard-8 value=description#192.168.2.11$port#39723$ifname#192.168.2.11$
[proxy:0:2 at reuN2-desktop] cached command: Pbusinesscard-8=description#192.168.2.11$port#39723$ifname#192.168.2.11$
[proxy:0:2 at reuN2-desktop] PMI response: cmd=put_result rc=0 msg=success
[proxy:0:2 at reuN2-desktop] got pmi command (from 4): barrier_in
[proxy:0:2 at reuN2-desktop] got pmi command (from 5): barrier_in
[mpiexec at reuOrin-desktop] [pgid: 0] got PMI command: cmd=put Pbusinesscard-2=description#192.168.2.11$port#44085$ifname#192.168.2.11$ Pbusinesscard-5=description#192.168.2.11$port#56233$ifname#192.168.2.11$ Pbusinesscard-8=description#192.168.2.11$port#39723$ifname#192.168.2.11$
[proxy:0:2 at reuN2-desktop] got pmi command (from 6): barrier_in
[proxy:0:2 at reuN2-desktop] flushing 3 put command(s) out
[proxy:0:2 at reuN2-desktop] forwarding command (cmd=put Pbusinesscard-2=description#192.168.2.11$port#44085$ifname#192.168.2.11$ Pbusinesscard-5=description#192.168.2.11$port#56233$ifname#192.168.2.11$ Pbusinesscard-8=description#192.168.2.11$port#39723$ifname#192.168.2.11$) upstream
[proxy:0:2 at reuN2-desktop] forwarding command (cmd=barrier_in) upstream
[mpiexec at reuOrin-desktop] [pgid: 0] got PMI command: cmd=barrier_in
[proxy:0:1 at reu-desktop] PMI response: cmd=barrier_out
[proxy:0:1 at reu-desktop] PMI response: cmd=barrier_out
[proxy:0:1 at reu-desktop] PMI response: cmd=barrier_out
[proxy:0:1 at reu-desktop] got pmi command (from 5): get
kvsname=kvs_23478_0 key=sharedFilename-1
[proxy:0:1 at reu-desktop] PMI response: cmd=get_result rc=0 msg=success value=/dev/shm/mpich_shar_tmpmhgotU
[proxy:0:1 at reu-desktop] got pmi command (from 6): get
kvsname=kvs_23478_0 key=sharedFilename-1
[proxy:0:1 at reu-desktop] PMI response: cmd=get_result rc=0 msg=success value=/dev/shm/mpich_shar_tmpmhgotU
[proxy:0:1 at reu-desktop] got pmi command (from 4): put
kvsname=kvs_23478_0 key=Pbusinesscard-1 value=description#192.168.2.10$port#57871$ifname#192.168.2.10$
[proxy:0:1 at reu-desktop] cached command: Pbusinesscard-1=description#192.168.2.10$port#57871$ifname#192.168.2.10$
[proxy:0:1 at reu-desktop] PMI response: cmd=put_result rc=0 msg=success
[proxy:0:1 at reu-desktop] got pmi command (from 5): put
kvsname=kvs_23478_0 key=Pbusinesscard-4 value=description#192.168.2.10$port#56219$ifname#192.168.2.10$
[proxy:0:1 at reu-desktop] cached command: Pbusinesscard-4=description#192.168.2.10$port#56219$ifname#192.168.2.10$
[proxy:0:1 at reu-desktop] PMI response: cmd=put_result rc=0 msg=success
[mpiexec at reuOrin-desktop] [pgid: 0] got PMI command: cmd=put Pbusinesscard-1=description#192.168.2.10$port#57871$ifname#192.168.2.10$ Pbusinesscard-4=description#192.168.2.10$port#56219$ifname#192.168.2.10$ Pbusinesscard-7=description#192.168.2.10$port#52015$ifname#192.168.2.10$
[proxy:0:1 at reu-desktop] got pmi command (from 6): put
kvsname=kvs_23478_0 key=Pbusinesscard-7 value=description#192.168.2.10$port#52015$ifname#192.168.2.10$
[proxy:0:1 at reu-desktop] cached command: Pbusinesscard-7=description#192.168.2.10$port#52015$ifname#192.168.2.10$
[proxy:0:1 at reu-desktop] PMI response: cmd=put_result rc=0 msg=success
[proxy:0:1 at reu-desktop] got pmi command (from 5): barrier_in
[proxy:0:1 at reu-desktop] got pmi command (from 4): barrier_in
[proxy:0:1 at reu-desktop] got pmi command (from 6): barrier_in
[proxy:0:1 at reu-desktop] flushing 3 put command(s) out
[proxy:0:1 at reu-desktop] forwarding command (cmd=put Pbusinesscard-1=description#192.168.2.10$port#57871$ifname#192.168.2.10$ Pbusinesscard-4=description#192.168.2.10$port#56219$ifname#192.168.2.10$ Pbusinesscard-7=description#192.168.2.10$port#52015$ifname#192.168.2.10$) upstream
[proxy:0:1 at reu-desktop] forwarding command (cmd=barrier_in) upstream
[mpiexec at reuOrin-desktop] [pgid: 0] got PMI command: cmd=barrier_in
[mpiexec at reuOrin-desktop] PMI response to fd 6 pid 6: cmd=keyval_cache Pbusinesscard-0=description#192.168.2.20$port#56375$ifname#192.168.2.20$ Pbusinesscard-3=description#192.168.2.20$port#42255$ifname#192.168.2.20$ Pbusinesscard-6=description#192.168.2.20$port#54961$ifname#192.168.2.20$ Pbusinesscard-9=description#192.168.2.20$port#55841$ifname#192.168.2.20$ Pbusinesscard-2=description#192.168.2.11$port#44085$ifname#192.168.2.11$ Pbusinesscard-5=description#192.168.2.11$port#56233$ifname#192.168.2.11$ Pbusinesscard-8=description#192.168.2.11$port#39723$ifname#192.168.2.11$ Pbusinesscard-1=description#192.168.2.10$port#57871$ifname#192.168.2.10$ Pbusinesscard-4=description#192.168.2.10$port#56219$ifname#192.168.2.10$ Pbusinesscard-7=description#192.168.2.10$port#52015$ifname#192.168.2.10$
[mpiexec at reuOrin-desktop] PMI response to fd 7 pid 6: cmd=keyval_cache Pbusinesscard-0=description#192.168.2.20$port#56375$ifname#192.168.2.20$ Pbusinesscard-3=description#192.168.2.20$port#42255$ifname#192.168.2.20$ Pbusinesscard-6=description#192.168.2.20$port#54961$ifname#192.168.2.20$ Pbusinesscard-9=description#192.168.2.20$port#55841$ifname#192.168.2.20$ Pbusinesscard-2=description#192.168.2.11$port#44085$ifname#192.168.2.11$ Pbusinesscard-5=description#192.168.2.11$port#56233$ifname#192.168.2.11$ Pbusinesscard-8=description#192.168.2.11$port#39723$ifname#192.168.2.11$ Pbusinesscard-1=description#192.168.2.10$port#57871$ifname#192.168.2.10$ Pbusinesscard-4=description#192.168.2.10$port#56219$ifname#192.168.2.10$ Pbusinesscard-7=description#192.168.2.10$port#52015$ifname#192.168.2.10$
[mpiexec at reuOrin-desktop] PMI response to fd 15 pid 6: cmd=keyval_cache Pbusinesscard-0=description#192.168.2.20$port#56375$ifname#192.168.2.20$ Pbusinesscard-3=description#192.168.2.20$port#42255$ifname#192.168.2.20$ Pbusinesscard-6=description#192.168.2.20$port#54961$ifname#192.168.2.20$ Pbusinesscard-9=description#192.168.2.20$port#55841$ifname#192.168.2.20$ Pbusinesscard-2=description#192.168.2.11$port#44085$ifname#192.168.2.11$ Pbusinesscard-5=description#192.168.2.11$port#56233$ifname#192.168.2.11$ Pbusinesscard-8=description#192.168.2.11$port#39723$ifname#192.168.2.11$ Pbusinesscard-1=description#192.168.2.10$port#57871$ifname#192.168.2.10$ Pbusinesscard-4=description#192.168.2.10$port#56219$ifname#192.168.2.10$ Pbusinesscard-7=description#192.168.2.10$port#52015$ifname#192.168.2.10$
[mpiexec at reuOrin-desktop] PMI response to fd 6 pid 6: cmd=barrier_out
[mpiexec at reuOrin-desktop] PMI response to fd 7 pid 6: cmd=barrier_out
[mpiexec at reuOrin-desktop] PMI response to fd 15 pid 6: cmd=barrier_out
[proxy:0:0 at reuOrin-desktop] PMI response: cmd=barrier_out
[proxy:0:0 at reuOrin-desktop] PMI response: cmd=barrier_out
[proxy:0:0 at reuOrin-desktop] PMI response: cmd=barrier_out
[proxy:0:0 at reuOrin-desktop] PMI response: cmd=barrier_out
[proxy:0:1 at reu-desktop] PMI response: cmd=barrier_out
[proxy:0:1 at reu-desktop] PMI response: cmd=barrier_out
[proxy:0:2 at reuN2-desktop] PMI response: cmd=barrier_out
[proxy:0:2 at reuN2-desktop] PMI response: cmd=barrier_out
[proxy:0:2 at reuN2-desktop] PMI response: cmd=barrier_out
[proxy:0:1 at reu-desktop] PMI response: cmd=barrier_out
Hello World from Process 9 of 10
Hello World from Process 0 of 10
Hello World from Process 3 of 10
Hello World from Process 6 of 10
[proxy:0:0 at reuOrin-desktop] got pmi command (from 0): finalize
[proxy:0:0 at reuOrin-desktop] PMI response: cmd=finalize_ack
[proxy:0:0 at reuOrin-desktop] got pmi command (from 8): finalize
[proxy:0:0 at reuOrin-desktop] PMI response: cmd=finalize_ack
[proxy:0:0 at reuOrin-desktop] got pmi command (from 13): finalize
[proxy:0:0 at reuOrin-desktop] PMI response: cmd=finalize_ack
[proxy:0:0 at reuOrin-desktop] got pmi command (from 6): finalize
[proxy:0:0 at reuOrin-desktop] PMI response: cmd=finalize_ack
Hello World from Process 4 of 10
Hello World from Process 2 of 10
Hello World from Process 8 of 10
Hello World from Process 1 of 10
Hello World from Process 7 of 10
[proxy:0:1 at reu-desktop] got pmi command (from 4): finalize
[proxy:0:1 at reu-desktop] PMI response: cmd=finalize_ack
[proxy:0:1 at reu-desktop] got pmi command (from 6): finalize
[proxy:0:1 at reu-desktop] PMI response: cmd=finalize_ack
[proxy:0:1 at reu-desktop] got pmi command (from 5): finalize
[proxy:0:1 at reu-desktop] PMI response: cmd=finalize_ack
Hello World from Process 5 of 10
[proxy:0:2 at reuN2-desktop] got pmi command (from 5): finalize
[proxy:0:2 at reuN2-desktop] PMI response: cmd=finalize_ack
[proxy:0:2 at reuN2-desktop] got pmi command (from 6): finalize
[proxy:0:2 at reuN2-desktop] PMI response: cmd=finalize_ack
[proxy:0:2 at reuN2-desktop] got pmi command (from 4): finalize
[proxy:0:2 at reuN2-desktop] PMI response: cmd=finalize_ack
More information about the Mvapich
mailing list