[mvapich-discuss] mpi init error
Hoot Thompson
hoot at ptpnow.com
Fri Jun 15 08:16:29 EDT 2012
Hello,
I'm working on a fresh intstall (CentOS 6.2) on two new Sandy Bridge
servers with Mellanox FDR cards. Everything compiles just fine and seems
to come up but when I try simple benchmark tests, I get MPI init errors.
It works fine if I stay on the initiating server or the remote server
but not between the two. Results follow.....
***************** Running between two servers
[hoot at sandyhp1 osu_benchmarks]$ mpiexec -verbose -n 2 -hosts
10.0.0.1,10.0.0.2 /home/hoot/mvapich2-1.8-r5435/osu_benchmarks/osu_bw
host: 10.0.0.1
host: 10.0.0.2
==================================================================================================
mpiexec options:
----------------
Base path: /usr/local/bin/
Launcher: (null)
Debug level: 1
Enable X: -1
Global environment:
-------------------
HOSTNAME=sandyhp1
SELINUX_ROLE_REQUESTED=
TERM=xterm
SHELL=/bin/bash
HISTSIZE=1000
SSH_CLIENT=169.154.148.10 42095 22
SELINUX_USE_CURRENT_RANGE=
QTDIR=/usr/lib64/qt-3.3
QTINC=/usr/lib64/qt-3.3/include
SSH_TTY=/dev/pts/0
USER=hoot
LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=01;05;37;41:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arj=01;31:*.taz=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.lz=01;31:*.xz=01;31:*.bz2=01;31:*.tbz=01;31:*.tbz2=01;31:*.bz=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.rar=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.jpg=01;35:*.jpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.axv=01;35:*.anx=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=01;36:*.au=01;36:*.flac=01;36:*.mid=01;36:*.midi=01;36:*.mka=01;36:*.mp3=01;36:*.mpc=01;36:*.ogg=01;36:*.ra=01;36:*.wav=01;36:*.axa=01;36:*.oga=01;36:*.spx=01;36:*.xspf=01;36:
MAIL=/var/spool/mail/hoot
PATH=/usr/lib64/qt-3.3/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hoot/bin
PWD=/home/hoot/mvapich2-1.8-r5435/osu_benchmarks
LANG=en_US.UTF-8
SELINUX_LEVEL_REQUESTED=
SSH_ASKPASS=/usr/libexec/openssh/gnome-ssh-askpass
HISTCONTROL=ignoredups
SHLVL=1
HOME=/home/hoot
LOGNAME=hoot
QTLIB=/usr/lib64/qt-3.3/lib
CVS_RSH=ssh
SSH_CONNECTION=169.154.148.10 42095 169.154.148.45 22
LESSOPEN=|/usr/bin/lesspipe.sh %s
G_BROKEN_FILENAMES=1
_=/usr/local/bin/mpiexec
OLDPWD=/home/hoot/mvapich2-1.8-r5435
Hydra internal environment:
---------------------------
GFORTRAN_UNBUFFERED_PRECONNECTED=y
Proxy information:
*********************
[1] proxy: 10.0.0.1 (1 cores)
Exec list: /home/hoot/mvapich2-1.8-r5435/osu_benchmarks/osu_bw (1
processes);
[2] proxy: 10.0.0.2 (1 cores)
Exec list: /home/hoot/mvapich2-1.8-r5435/osu_benchmarks/osu_bw (1
processes);
==================================================================================================
[mpiexec at sandyhp1] Timeout set to -1 (-1 means infinite)
[mpiexec at sandyhp1] Got a control port string of 10.0.0.1:50740
Proxy launch args: /usr/local/bin/hydra_pmi_proxy --control-port
10.0.0.1:50740 --debug --rmk user --launcher ssh --demux poll --pgid 0
--retries 10 --proxy-id
[mpiexec at sandyhp1] PMI FD: (null); PMI PORT: (null); PMI ID/RANK: -1
Arguments being passed to proxy 0:
--version 1.4.1p1 --iface-ip-env-name MPICH_INTERFACE_HOSTNAME
--hostname 10.0.0.1 --global-core-map 0,1,1 --filler-process-map 0,1,1
--global-process-count 2 --auto-cleanup 1 --pmi-rank -1 --pmi-kvsname
kvs_7644_0 --pmi-process-mapping (vector,(0,2,1)) --ckpoint-num -1
--global-inherited-env 29 'HOSTNAME=sandyhp1' 'SELINUX_ROLE_REQUESTED='
'TERM=xterm' 'SHELL=/bin/bash' 'HISTSIZE=1000'
'SSH_CLIENT=169.154.148.10 42095 22' 'SELINUX_USE_CURRENT_RANGE='
'QTDIR=/usr/lib64/qt-3.3' 'QTINC=/usr/lib64/qt-3.3/include'
'SSH_TTY=/dev/pts/0' 'USER=hoot'
'LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=01;05;37;41:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arj=01;31:*.taz=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.lz=01;31:*.xz=01;31:*.bz2=01;31:*.tbz=01;31:*.tbz2=01;31:*.bz=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.rar=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.jpg=01;35:*.jpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.axv=01;35:*.anx=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=01;36:*.au=01;36:*.flac=01;36:*.mid=01;36:*.midi=01;36:*.mka=01;36:*.mp3=01;36:*.mpc=01;36:*.ogg=01;36:*.ra=01;36:*.wav=01;36:*.axa=01;36:*.oga=01;36:*.spx=01;36:*.xspf=01;36:'
'MAIL=/var/spool/mail/hoot'
'PATH=/usr/lib64/qt-3.3/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hoot/bin'
'PWD=/home/hoot/mvapich2-1.8-r5435/osu_benchmarks' 'LANG=en_US.UTF-8'
'SELINUX_LEVEL_REQUESTED='
'SSH_ASKPASS=/usr/libexec/openssh/gnome-ssh-askpass'
'HISTCONTROL=ignoredups' 'SHLVL=1' 'HOME=/home/hoot' 'LOGNAME=hoot'
'QTLIB=/usr/lib64/qt-3.3/lib' 'CVS_RSH=ssh'
'SSH_CONNECTION=169.154.148.10 42095 169.154.148.45 22'
'LESSOPEN=|/usr/bin/lesspipe.sh %s' 'G_BROKEN_FILENAMES=1'
'_=/usr/local/bin/mpiexec' 'OLDPWD=/home/hoot/mvapich2-1.8-r5435'
--global-user-env 0 --global-system-env 1
'GFORTRAN_UNBUFFERED_PRECONNECTED=y' --proxy-core-count 1 --exec
--exec-appnum 0 --exec-proc-count 1 --exec-local-env 0 --exec-wdir
/home/hoot/mvapich2-1.8-r5435/osu_benchmarks --exec-args 1
/home/hoot/mvapich2-1.8-r5435/osu_benchmarks/osu_bw
[mpiexec at sandyhp1] PMI FD: (null); PMI PORT: (null); PMI ID/RANK: -1
Arguments being passed to proxy 1:
--version 1.4.1p1 --iface-ip-env-name MPICH_INTERFACE_HOSTNAME
--hostname 10.0.0.2 --global-core-map 1,1,0 --filler-process-map 1,1,0
--global-process-count 2 --auto-cleanup 1 --pmi-rank -1 --pmi-kvsname
kvs_7644_0 --pmi-process-mapping (vector,(0,2,1)) --ckpoint-num -1
--global-inherited-env 29 'HOSTNAME=sandyhp1' 'SELINUX_ROLE_REQUESTED='
'TERM=xterm' 'SHELL=/bin/bash' 'HISTSIZE=1000'
'SSH_CLIENT=169.154.148.10 42095 22' 'SELINUX_USE_CURRENT_RANGE='
'QTDIR=/usr/lib64/qt-3.3' 'QTINC=/usr/lib64/qt-3.3/include'
'SSH_TTY=/dev/pts/0' 'USER=hoot'
'LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=01;05;37;41:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arj=01;31:*.taz=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.lz=01;31:*.xz=01;31:*.bz2=01;31:*.tbz=01;31:*.tbz2=01;31:*.bz=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.rar=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.jpg=01;35:*.jpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.axv=01;35:*.anx=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=01;36:*.au=01;36:*.flac=01;36:*.mid=01;36:*.midi=01;36:*.mka=01;36:*.mp3=01;36:*.mpc=01;36:*.ogg=01;36:*.ra=01;36:*.wav=01;36:*.axa=01;36:*.oga=01;36:*.spx=01;36:*.xspf=01;36:'
'MAIL=/var/spool/mail/hoot'
'PATH=/usr/lib64/qt-3.3/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hoot/bin'
'PWD=/home/hoot/mvapich2-1.8-r5435/osu_benchmarks' 'LANG=en_US.UTF-8'
'SELINUX_LEVEL_REQUESTED='
'SSH_ASKPASS=/usr/libexec/openssh/gnome-ssh-askpass'
'HISTCONTROL=ignoredups' 'SHLVL=1' 'HOME=/home/hoot' 'LOGNAME=hoot'
'QTLIB=/usr/lib64/qt-3.3/lib' 'CVS_RSH=ssh'
'SSH_CONNECTION=169.154.148.10 42095 169.154.148.45 22'
'LESSOPEN=|/usr/bin/lesspipe.sh %s' 'G_BROKEN_FILENAMES=1'
'_=/usr/local/bin/mpiexec' 'OLDPWD=/home/hoot/mvapich2-1.8-r5435'
--global-user-env 0 --global-system-env 1
'GFORTRAN_UNBUFFERED_PRECONNECTED=y' --proxy-core-count 1 --exec
--exec-appnum 0 --exec-proc-count 1 --exec-local-env 0 --exec-wdir
/home/hoot/mvapich2-1.8-r5435/osu_benchmarks --exec-args 1
/home/hoot/mvapich2-1.8-r5435/osu_benchmarks/osu_bw
[mpiexec at sandyhp1] Launch arguments: /usr/local/bin/hydra_pmi_proxy
--control-port 10.0.0.1:50740 --debug --rmk user --launcher ssh --demux
poll --pgid 0 --retries 10 --proxy-id 0
[mpiexec at sandyhp1] Launch arguments: /usr/bin/ssh -x 10.0.0.2
"/usr/local/bin/hydra_pmi_proxy" --control-port 10.0.0.1:50740 --debug
--rmk user --launcher ssh --demux poll --pgid 0 --retries 10 --proxy-id 1
[proxy:0:0 at sandyhp1] got pmi command (from 0): init
pmi_version=1 pmi_subversion=1
[proxy:0:0 at sandyhp1] PMI response: cmd=response_to_init pmi_version=1
pmi_subversion=1 rc=0
[proxy:0:0 at sandyhp1] got pmi command (from 0): get_maxes
[proxy:0:0 at sandyhp1] PMI response: cmd=maxes kvsname_max=256
keylen_max=64 vallen_max=1024
[proxy:0:0 at sandyhp1] got pmi command (from 0): get_appnum
[proxy:0:0 at sandyhp1] PMI response: cmd=appnum appnum=0
[proxy:0:0 at sandyhp1] got pmi command (from 0): get_my_kvsname
[proxy:0:0 at sandyhp1] PMI response: cmd=my_kvsname kvsname=kvs_7644_0
[proxy:0:0 at sandyhp1] got pmi command (from 0): get_my_kvsname
[proxy:0:0 at sandyhp1] PMI response: cmd=my_kvsname kvsname=kvs_7644_0
[proxy:0:0 at sandyhp1] got pmi command (from 0): get
kvsname=kvs_7644_0 key=PMI_process_mapping
[proxy:0:0 at sandyhp1] PMI response: cmd=get_result rc=0 msg=success
value=(vector,(0,2,1))
[cli_0]: aborting job:
Fatal error in MPI_Init:
Other MPI error
=====================================================================================
= BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
= EXIT CODE: 256
= CLEANING UP REMAINING PROCESSES
= YOU CAN IGNORE THE BELOW CLEANUP MESSAGES
=====================================================================================
[proxy:0:1 at sandyhp2] got pmi command (from 4): init
pmi_version=1 pmi_subversion=1
[proxy:0:1 at sandyhp2] PMI response: cmd=response_to_init pmi_version=1
pmi_subversion=1 rc=0
[proxy:0:1 at sandyhp2] got pmi command (from 4): get_maxes
[proxy:0:1 at sandyhp2] PMI response: cmd=maxes kvsname_max=256
keylen_max=64 vallen_max=1024
[proxy:0:1 at sandyhp2] got pmi command (from 4): get_appnum
[proxy:0:1 at sandyhp2] PMI response: cmd=appnum appnum=0
[proxy:0:1 at sandyhp2] got pmi command (from 4): get_my_kvsname
[proxy:0:1 at sandyhp2] PMI response: cmd=my_kvsname kvsname=kvs_7644_0
[proxy:0:1 at sandyhp2] got pmi command (from 4): get_my_kvsname
[proxy:0:1 at sandyhp2] PMI response: cmd=my_kvsname kvsname=kvs_7644_0
[proxy:0:1 at sandyhp2] got pmi command (from 4): get
kvsname=kvs_7644_0 key=PMI_process_mapping
[proxy:0:1 at sandyhp2] PMI response: cmd=get_result rc=0 msg=success
value=(vector,(0,2,1))
[cli_1]: aborting job:
Fatal error in MPI_Init:
Other MPI error
=====================================================================================
= BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
= EXIT CODE: 256
= CLEANING UP REMAINING PROCESSES
= YOU CAN IGNORE THE BELOW CLEANUP MESSAGES
=====================================================================================
***************** Running on the initiating server
[hoot at sandyhp1 osu_benchmarks]$ mpiexec -verbose -hosts
10.0.0.1,10.0.0.1 /home/hoot/mvapich2-1.8-r5435/osu_benchmarks/osu_bw
host: 10.0.0.1
==================================================================================================
mpiexec options:
----------------
Base path: /usr/local/bin/
Launcher: (null)
Debug level: 1
Enable X: -1
Global environment:
-------------------
HOSTNAME=sandyhp1
SELINUX_ROLE_REQUESTED=
TERM=xterm
SHELL=/bin/bash
HISTSIZE=1000
SSH_CLIENT=169.154.148.10 42095 22
SELINUX_USE_CURRENT_RANGE=
QTDIR=/usr/lib64/qt-3.3
QTINC=/usr/lib64/qt-3.3/include
SSH_TTY=/dev/pts/0
USER=hoot
LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=01;05;37;41:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arj=01;31:*.taz=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.lz=01;31:*.xz=01;31:*.bz2=01;31:*.tbz=01;31:*.tbz2=01;31:*.bz=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.rar=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.jpg=01;35:*.jpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.axv=01;35:*.anx=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=01;36:*.au=01;36:*.flac=01;36:*.mid=01;36:*.midi=01;36:*.mka=01;36:*.mp3=01;36:*.mpc=01;36:*.ogg=01;36:*.ra=01;36:*.wav=01;36:*.axa=01;36:*.oga=01;36:*.spx=01;36:*.xspf=01;36:
MAIL=/var/spool/mail/hoot
PATH=/usr/lib64/qt-3.3/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hoot/bin
PWD=/home/hoot/mvapich2-1.8-r5435/osu_benchmarks
LANG=en_US.UTF-8
SELINUX_LEVEL_REQUESTED=
SSH_ASKPASS=/usr/libexec/openssh/gnome-ssh-askpass
HISTCONTROL=ignoredups
SHLVL=1
HOME=/home/hoot
LOGNAME=hoot
QTLIB=/usr/lib64/qt-3.3/lib
CVS_RSH=ssh
SSH_CONNECTION=169.154.148.10 42095 169.154.148.45 22
LESSOPEN=|/usr/bin/lesspipe.sh %s
G_BROKEN_FILENAMES=1
_=/usr/local/bin/mpiexec
OLDPWD=/home/hoot/mvapich2-1.8-r5435
Hydra internal environment:
---------------------------
GFORTRAN_UNBUFFERED_PRECONNECTED=y
Proxy information:
*********************
[1] proxy: 10.0.0.1 (2 cores)
Exec list: /home/hoot/mvapich2-1.8-r5435/osu_benchmarks/osu_bw (2
processes);
==================================================================================================
[mpiexec at sandyhp1] Timeout set to -1 (-1 means infinite)
[mpiexec at sandyhp1] Got a control port string of 10.0.0.1:56066
Proxy launch args: /usr/local/bin/hydra_pmi_proxy --control-port
10.0.0.1:56066 --debug --rmk user --launcher ssh --demux poll --pgid 0
--retries 10 --proxy-id
[mpiexec at sandyhp1] PMI FD: (null); PMI PORT: (null); PMI ID/RANK: -1
Arguments being passed to proxy 0:
--version 1.4.1p1 --iface-ip-env-name MPICH_INTERFACE_HOSTNAME
--hostname 10.0.0.1 --global-core-map 0,2,0 --filler-process-map 0,2,0
--global-process-count 2 --auto-cleanup 1 --pmi-rank -1 --pmi-kvsname
kvs_7484_0 --pmi-process-mapping (vector,(0,1,2)) --ckpoint-num -1
--global-inherited-env 29 'HOSTNAME=sandyhp1' 'SELINUX_ROLE_REQUESTED='
'TERM=xterm' 'SHELL=/bin/bash' 'HISTSIZE=1000'
'SSH_CLIENT=169.154.148.10 42095 22' 'SELINUX_USE_CURRENT_RANGE='
'QTDIR=/usr/lib64/qt-3.3' 'QTINC=/usr/lib64/qt-3.3/include'
'SSH_TTY=/dev/pts/0' 'USER=hoot'
'LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=01;05;37;41:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arj=01;31:*.taz=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.lz=01;31:*.xz=01;31:*.bz2=01;31:*.tbz=01;31:*.tbz2=01;31:*.bz=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.rar=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.jpg=01;35:*.jpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.axv=01;35:*.anx=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=01;36:*.au=01;36:*.flac=01;36:*.mid=01;36:*.midi=01;36:*.mka=01;36:*.mp3=01;36:*.mpc=01;36:*.ogg=01;36:*.ra=01;36:*.wav=01;36:*.axa=01;36:*.oga=01;36:*.spx=01;36:*.xspf=01;36:'
'MAIL=/var/spool/mail/hoot'
'PATH=/usr/lib64/qt-3.3/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hoot/bin'
'PWD=/home/hoot/mvapich2-1.8-r5435/osu_benchmarks' 'LANG=en_US.UTF-8'
'SELINUX_LEVEL_REQUESTED='
'SSH_ASKPASS=/usr/libexec/openssh/gnome-ssh-askpass'
'HISTCONTROL=ignoredups' 'SHLVL=1' 'HOME=/home/hoot' 'LOGNAME=hoot'
'QTLIB=/usr/lib64/qt-3.3/lib' 'CVS_RSH=ssh'
'SSH_CONNECTION=169.154.148.10 42095 169.154.148.45 22'
'LESSOPEN=|/usr/bin/lesspipe.sh %s' 'G_BROKEN_FILENAMES=1'
'_=/usr/local/bin/mpiexec' 'OLDPWD=/home/hoot/mvapich2-1.8-r5435'
--global-user-env 0 --global-system-env 1
'GFORTRAN_UNBUFFERED_PRECONNECTED=y' --proxy-core-count 2 --exec
--exec-appnum 0 --exec-proc-count 2 --exec-local-env 0 --exec-wdir
/home/hoot/mvapich2-1.8-r5435/osu_benchmarks --exec-args 1
/home/hoot/mvapich2-1.8-r5435/osu_benchmarks/osu_bw
[mpiexec at sandyhp1] Launch arguments: /usr/local/bin/hydra_pmi_proxy
--control-port 10.0.0.1:56066 --debug --rmk user --launcher ssh --demux
poll --pgid 0 --retries 10 --proxy-id 0
[proxy:0:0 at sandyhp1] got pmi command (from 0): init
pmi_version=1 pmi_subversion=1
[proxy:0:0 at sandyhp1] PMI response: cmd=response_to_init pmi_version=1
pmi_subversion=1 rc=0
[proxy:0:0 at sandyhp1] got pmi command (from 6): init
pmi_version=1 pmi_subversion=1
[proxy:0:0 at sandyhp1] PMI response: cmd=response_to_init pmi_version=1
pmi_subversion=1 rc=0
[proxy:0:0 at sandyhp1] got pmi command (from 0): get_maxes
[proxy:0:0 at sandyhp1] PMI response: cmd=maxes kvsname_max=256
keylen_max=64 vallen_max=1024
[proxy:0:0 at sandyhp1] got pmi command (from 6): get_maxes
[proxy:0:0 at sandyhp1] PMI response: cmd=maxes kvsname_max=256
keylen_max=64 vallen_max=1024
[proxy:0:0 at sandyhp1] got pmi command (from 0): get_appnum
[proxy:0:0 at sandyhp1] PMI response: cmd=appnum appnum=0
[proxy:0:0 at sandyhp1] got pmi command (from 0): get_my_kvsname
[proxy:0:0 at sandyhp1] PMI response: cmd=my_kvsname kvsname=kvs_7484_0
[proxy:0:0 at sandyhp1] got pmi command (from 6): get_appnum
[proxy:0:0 at sandyhp1] PMI response: cmd=appnum appnum=0
[proxy:0:0 at sandyhp1] got pmi command (from 0): get_my_kvsname
[proxy:0:0 at sandyhp1] PMI response: cmd=my_kvsname kvsname=kvs_7484_0
[proxy:0:0 at sandyhp1] got pmi command (from 6): get_my_kvsname
[proxy:0:0 at sandyhp1] PMI response: cmd=my_kvsname kvsname=kvs_7484_0
[proxy:0:0 at sandyhp1] got pmi command (from 0): get
kvsname=kvs_7484_0 key=PMI_process_mapping
[proxy:0:0 at sandyhp1] PMI response: cmd=get_result rc=0 msg=success
value=(vector,(0,1,2))
[proxy:0:0 at sandyhp1] got pmi command (from 6): get_my_kvsname
[proxy:0:0 at sandyhp1] PMI response: cmd=my_kvsname kvsname=kvs_7484_0
[proxy:0:0 at sandyhp1] got pmi command (from 6): get
kvsname=kvs_7484_0 key=PMI_process_mapping
[proxy:0:0 at sandyhp1] PMI response: cmd=get_result rc=0 msg=success
value=(vector,(0,1,2))
# OSU MPI Bandwidth Test v3.6
# Size Bandwidth (MB/s)
1 2.76
2 5.54
4 11.09
8 22.27
16 44.48
32 87.19
64 171.52
128 331.24
256 612.25
512 1014.16
1024 1585.68
2048 2590.26
4096 3744.42
8192 4721.57
16384 4940.24
32768 4258.99
65536 4450.17
131072 4461.91
262144 4126.22
524288 4103.56
1048576 5106.87
2097152 5126.66
4194304 5161.63
[proxy:0:0 at sandyhp1] got pmi command (from 6): finalize
[proxy:0:0 at sandyhp1] PMI response: cmd=finalize_ack
[proxy:0:0 at sandyhp1] got pmi command (from 0): finalize
[proxy:0:0 at sandyhp1] PMI response: cmd=finalize_ack
**************** Running on the remote server
[hoot at sandyhp1 osu_benchmarks]$ mpiexec -verbose -hosts
10.0.0.2,10.0.0.2 /home/hoot/mvapich2-1.8-r5435/osu_benchmarks/osu_bw
host: 10.0.0.2
==================================================================================================
mpiexec options:
----------------
Base path: /usr/local/bin/
Launcher: (null)
Debug level: 1
Enable X: -1
Global environment:
-------------------
HOSTNAME=sandyhp1
SELINUX_ROLE_REQUESTED=
TERM=xterm
SHELL=/bin/bash
HISTSIZE=1000
SSH_CLIENT=169.154.148.10 42095 22
SELINUX_USE_CURRENT_RANGE=
QTDIR=/usr/lib64/qt-3.3
QTINC=/usr/lib64/qt-3.3/include
SSH_TTY=/dev/pts/0
USER=hoot
LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=01;05;37;41:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arj=01;31:*.taz=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.lz=01;31:*.xz=01;31:*.bz2=01;31:*.tbz=01;31:*.tbz2=01;31:*.bz=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.rar=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.jpg=01;35:*.jpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.axv=01;35:*.anx=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=01;36:*.au=01;36:*.flac=01;36:*.mid=01;36:*.midi=01;36:*.mka=01;36:*.mp3=01;36:*.mpc=01;36:*.ogg=01;36:*.ra=01;36:*.wav=01;36:*.axa=01;36:*.oga=01;36:*.spx=01;36:*.xspf=01;36:
MAIL=/var/spool/mail/hoot
PATH=/usr/lib64/qt-3.3/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hoot/bin
PWD=/home/hoot/mvapich2-1.8-r5435/osu_benchmarks
LANG=en_US.UTF-8
SELINUX_LEVEL_REQUESTED=
SSH_ASKPASS=/usr/libexec/openssh/gnome-ssh-askpass
HISTCONTROL=ignoredups
SHLVL=1
HOME=/home/hoot
LOGNAME=hoot
QTLIB=/usr/lib64/qt-3.3/lib
CVS_RSH=ssh
SSH_CONNECTION=169.154.148.10 42095 169.154.148.45 22
LESSOPEN=|/usr/bin/lesspipe.sh %s
G_BROKEN_FILENAMES=1
_=/usr/local/bin/mpiexec
OLDPWD=/home/hoot/mvapich2-1.8-r5435
Hydra internal environment:
---------------------------
GFORTRAN_UNBUFFERED_PRECONNECTED=y
Proxy information:
*********************
[1] proxy: 10.0.0.2 (2 cores)
Exec list: /home/hoot/mvapich2-1.8-r5435/osu_benchmarks/osu_bw (2
processes);
==================================================================================================
[mpiexec at sandyhp1] Timeout set to -1 (-1 means infinite)
[mpiexec at sandyhp1] Got a control port string of sandyhp1:39504
Proxy launch args: /usr/local/bin/hydra_pmi_proxy --control-port
sandyhp1:39504 --debug --rmk user --launcher ssh --demux poll --pgid 0
--retries 10 --proxy-id
[mpiexec at sandyhp1] PMI FD: (null); PMI PORT: (null); PMI ID/RANK: -1
Arguments being passed to proxy 0:
--version 1.4.1p1 --iface-ip-env-name MPICH_INTERFACE_HOSTNAME
--hostname 10.0.0.2 --global-core-map 0,2,0 --filler-process-map 0,2,0
--global-process-count 2 --auto-cleanup 1 --pmi-rank -1 --pmi-kvsname
kvs_7496_0 --pmi-process-mapping (vector,(0,1,2)) --ckpoint-num -1
--global-inherited-env 29 'HOSTNAME=sandyhp1' 'SELINUX_ROLE_REQUESTED='
'TERM=xterm' 'SHELL=/bin/bash' 'HISTSIZE=1000'
'SSH_CLIENT=169.154.148.10 42095 22' 'SELINUX_USE_CURRENT_RANGE='
'QTDIR=/usr/lib64/qt-3.3' 'QTINC=/usr/lib64/qt-3.3/include'
'SSH_TTY=/dev/pts/0' 'USER=hoot'
'LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=01;05;37;41:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arj=01;31:*.taz=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.lz=01;31:*.xz=01;31:*.bz2=01;31:*.tbz=01;31:*.tbz2=01;31:*.bz=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.rar=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.jpg=01;35:*.jpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.axv=01;35:*.anx=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=01;36:*.au=01;36:*.flac=01;36:*.mid=01;36:*.midi=01;36:*.mka=01;36:*.mp3=01;36:*.mpc=01;36:*.ogg=01;36:*.ra=01;36:*.wav=01;36:*.axa=01;36:*.oga=01;36:*.spx=01;36:*.xspf=01;36:'
'MAIL=/var/spool/mail/hoot'
'PATH=/usr/lib64/qt-3.3/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hoot/bin'
'PWD=/home/hoot/mvapich2-1.8-r5435/osu_benchmarks' 'LANG=en_US.UTF-8'
'SELINUX_LEVEL_REQUESTED='
'SSH_ASKPASS=/usr/libexec/openssh/gnome-ssh-askpass'
'HISTCONTROL=ignoredups' 'SHLVL=1' 'HOME=/home/hoot' 'LOGNAME=hoot'
'QTLIB=/usr/lib64/qt-3.3/lib' 'CVS_RSH=ssh'
'SSH_CONNECTION=169.154.148.10 42095 169.154.148.45 22'
'LESSOPEN=|/usr/bin/lesspipe.sh %s' 'G_BROKEN_FILENAMES=1'
'_=/usr/local/bin/mpiexec' 'OLDPWD=/home/hoot/mvapich2-1.8-r5435'
--global-user-env 0 --global-system-env 1
'GFORTRAN_UNBUFFERED_PRECONNECTED=y' --proxy-core-count 2 --exec
--exec-appnum 0 --exec-proc-count 2 --exec-local-env 0 --exec-wdir
/home/hoot/mvapich2-1.8-r5435/osu_benchmarks --exec-args 1
/home/hoot/mvapich2-1.8-r5435/osu_benchmarks/osu_bw
[mpiexec at sandyhp1] Launch arguments: /usr/bin/ssh -x 10.0.0.2
"/usr/local/bin/hydra_pmi_proxy" --control-port sandyhp1:39504 --debug
--rmk user --launcher ssh --demux poll --pgid 0 --retries 10 --proxy-id 0
[proxy:0:0 at sandyhp2] got pmi command (from 4): init
pmi_version=1 pmi_subversion=1
[proxy:0:0 at sandyhp2] PMI response: cmd=response_to_init pmi_version=1
pmi_subversion=1 rc=0
[proxy:0:0 at sandyhp2] got pmi command (from 4): get_maxes
[proxy:0:0 at sandyhp2] PMI response: cmd=maxes kvsname_max=256
keylen_max=64 vallen_max=1024
[proxy:0:0 at sandyhp2] got pmi command (from 5): init
pmi_version=1 pmi_subversion=1
[proxy:0:0 at sandyhp2] PMI response: cmd=response_to_init pmi_version=1
pmi_subversion=1 rc=0
[proxy:0:0 at sandyhp2] got pmi command (from 4): get_appnum
[proxy:0:0 at sandyhp2] PMI response: cmd=appnum appnum=0
[proxy:0:0 at sandyhp2] got pmi command (from 4): get_my_kvsname
[proxy:0:0 at sandyhp2] PMI response: cmd=my_kvsname kvsname=kvs_7496_0
[proxy:0:0 at sandyhp2] got pmi command (from 4): get_my_kvsname
[proxy:0:0 at sandyhp2] PMI response: cmd=my_kvsname kvsname=kvs_7496_0
[proxy:0:0 at sandyhp2] got pmi command (from 5): get_maxes
[proxy:0:0 at sandyhp2] PMI response: cmd=maxes kvsname_max=256
keylen_max=64 vallen_max=1024
[proxy:0:0 at sandyhp2] got pmi command (from 5): get_appnum
[proxy:0:0 at sandyhp2] PMI response: cmd=appnum appnum=0
[proxy:0:0 at sandyhp2] got pmi command (from 5): get_my_kvsname
[proxy:0:0 at sandyhp2] PMI response: cmd=my_kvsname kvsname=kvs_7496_0
[proxy:0:0 at sandyhp2] got pmi command (from 5): get_my_kvsname
[proxy:0:0 at sandyhp2] PMI response: cmd=my_kvsname kvsname=kvs_7496_0
[proxy:0:0 at sandyhp2] got pmi command (from 5): get
kvsname=kvs_7496_0 key=PMI_process_mapping
[proxy:0:0 at sandyhp2] PMI response: cmd=get_result rc=0 msg=success
value=(vector,(0,1,2))
[proxy:0:0 at sandyhp2] got pmi command (from 4): get
kvsname=kvs_7496_0 key=PMI_process_mapping
[proxy:0:0 at sandyhp2] PMI response: cmd=get_result rc=0 msg=success
value=(vector,(0,1,2))
# OSU MPI Bandwidth Test v3.6
# Size Bandwidth (MB/s)
1 2.76
2 5.55
4 11.07
8 21.98
16 44.50
32 86.66
64 170.38
128 327.30
256 607.49
512 1016.71
1024 1555.88
2048 2598.09
4096 3748.63
8192 4724.61
16384 4910.18
32768 4245.63
65536 4427.21
131072 4467.74
262144 4155.35
524288 4066.51
1048576 4351.23
2097152 5121.67
4194304 5669.35
[proxy:0:0 at sandyhp2] got pmi command (from 5): finalize
[proxy:0:0 at sandyhp2] PMI response: cmd=finalize_ack
[proxy:0:0 at sandyhp2] got pmi command (from 4): finalize
[proxy:0:0 at sandyhp2] PMI response: cmd=finalize_ack
More information about the mvapich-discuss
mailing list