[mvapich-discuss] mpi init error
Hoot Thompson
hoot at ptpnow.com
Fri Jun 15 10:55:49 EDT 2012
Here you go......
[hoot at sandyhp1 osu_benchmarks]$ mpiexec -n 2 -hosts 10.0.0.1,10.0.0.2
/home/hoot/mvapich2-1.8-r5435/osu_benchmarks/osu_bw
[cli_0]: aborting job:
Fatal error in MPI_Init:
Other MPI error, error stack:
MPIR_Init_thread(408).......:
MPID_Init(296)..............: channel initialization failed
MPIDI_CH3_Init(283).........:
MPIDI_CH3I_RDMA_init(172)...:
rdma_setup_startup_ring(434): cannot create cq
=====================================================================================
= BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
= EXIT CODE: 256
= CLEANING UP REMAINING PROCESSES
= YOU CAN IGNORE THE BELOW CLEANUP MESSAGES
=====================================================================================
[cli_1]: aborting job:
Fatal error in MPI_Init:
Other MPI error, error stack:
MPIR_Init_thread(408).......:
MPID_Init(296)..............: channel initialization failed
MPIDI_CH3_Init(283).........:
MPIDI_CH3I_RDMA_init(172)...:
rdma_setup_startup_ring(434): cannot create cq
=====================================================================================
= BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
= EXIT CODE: 256
= CLEANING UP REMAINING PROCESSES
= YOU CAN IGNORE THE BELOW CLEANUP MESSAGES
On 06/15/2012 10:47 AM, Jonathan Perkins wrote:
> To be honest, I would rather you leave the verbose switch off. That
> gives a lot of information about the launching process but the problem
> seems to occur after the processes are launched but before MPI_Init
> completes.
>
> By simply running with the debug build the library should give more
> additional output instead of simply saying there was an error in
> MPI_Init.
>
> On Fri, Jun 15, 2012 at 10:44:45AM -0400, Hoot Thompson wrote:
>> Should have asked, after the rebuild, how do I in fact get you the
>> additional info? Is the verbose switch enough?
>>
>> Hoot
>>
>> -----Original Message-----
>> From: Jonathan Perkins [mailto:perkinjo at cse.ohio-state.edu]
>> Sent: Friday, June 15, 2012 10:29 AM
>> To: Hoot Thompson
>> Cc: mvapich-discuss at cse.ohio-state.edu
>> Subject: Re: [mvapich-discuss] mpi init error
>>
>> Hello Hoot,
>> It seems that you're getting an MPI_Init error. Can you rebuild and add
>> '--disable-fast --enable-g=dbg' to your configuration options so that we can
>> get more information about this error?
>>
>> On Fri, Jun 15, 2012 at 08:16:29AM -0400, Hoot Thompson wrote:
>>> Hello,
>>>
>>> I'm working on a fresh intstall (CentOS 6.2) on two new Sandy Bridge
>>> servers with Mellanox FDR cards. Everything compiles just fine and
>>> seems to come up but when I try simple benchmark tests, I get MPI init
>>> errors. It works fine if I stay on the initiating server or the remote
>>> server but not between the two. Results follow.....
>>>
>>> ***************** Running between two servers
>>>
>>> [hoot at sandyhp1 osu_benchmarks]$ mpiexec -verbose -n 2 -hosts
>>> 10.0.0.1,10.0.0.2
>>> /home/hoot/mvapich2-1.8-r5435/osu_benchmarks/osu_bw
>>> host: 10.0.0.1
>>> host: 10.0.0.2
>>>
>>> ======================================================================
>>> ============================
>>> mpiexec options:
>>> ----------------
>>> Base path: /usr/local/bin/
>>> Launcher: (null)
>>> Debug level: 1
>>> Enable X: -1
>>>
>>> Global environment:
>>> -------------------
>>> HOSTNAME=sandyhp1
>>> SELINUX_ROLE_REQUESTED=
>>> TERM=xterm
>>> SHELL=/bin/bash
>>> HISTSIZE=1000
>>> SSH_CLIENT=169.154.148.10 42095 22
>>> SELINUX_USE_CURRENT_RANGE=
>>> QTDIR=/usr/lib64/qt-3.3
>>> QTINC=/usr/lib64/qt-3.3/include
>>> SSH_TTY=/dev/pts/0
>>> USER=hoot
>>>
>> LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;0
>> 1:cd=40;33;01:or=40;31;01:mi=01;05;37;41:su=37;41:sg=30;43:ca=30;41:tw=30;42
>> :ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arj=01;31:*.taz=01;31:
>> *.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.zip=01;31:*.z=01;31:*.Z=0
>> 1;31:*.dz=01;31:*.gz=01;31:*.lz=01;31:*.xz=01;31:*.bz2=01;31:*.tbz=01;31:*.t
>> bz2=01;31:*.bz=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.rar=01
>> ;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.jpg=01;35:*
>> .jpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tg
>> a=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=0
>> 1;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;
>> 35:*.m2v=01;35:*.mkv=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:
>> *.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb
>> =01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35
>> :!
>> *.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.axv=01;35:*.an
>> x=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=01;36:*.au=01;36:*.flac=01;36:*.mid=01
>> ;36:*.midi=01;36:*.mka=01;36:*.mp3=01;36:*.mpc=01;36:*.ogg=01;36:*.ra=01;36:
>> *.wav=01;36:*.axa=01;36:*.oga=01;36:*.spx=01;36:*.xspf=01;36:
>>> MAIL=/var/spool/mail/hoot
>>>
>> PATH=/usr/lib64/qt-3.3/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr
>> /sbin:/sbin:/home/hoot/bin
>>> PWD=/home/hoot/mvapich2-1.8-r5435/osu_benchmarks
>>> LANG=en_US.UTF-8
>>> SELINUX_LEVEL_REQUESTED=
>>> SSH_ASKPASS=/usr/libexec/openssh/gnome-ssh-askpass
>>> HISTCONTROL=ignoredups
>>> SHLVL=1
>>> HOME=/home/hoot
>>> LOGNAME=hoot
>>> QTLIB=/usr/lib64/qt-3.3/lib
>>> CVS_RSH=ssh
>>> SSH_CONNECTION=169.154.148.10 42095 169.154.148.45 22
>>> LESSOPEN=|/usr/bin/lesspipe.sh %s
>>> G_BROKEN_FILENAMES=1
>>> _=/usr/local/bin/mpiexec
>>> OLDPWD=/home/hoot/mvapich2-1.8-r5435
>>>
>>> Hydra internal environment:
>>> ---------------------------
>>> GFORTRAN_UNBUFFERED_PRECONNECTED=y
>>>
>>>
>>> Proxy information:
>>> *********************
>>> [1] proxy: 10.0.0.1 (1 cores)
>>> Exec list: /home/hoot/mvapich2-1.8-r5435/osu_benchmarks/osu_bw
>>> (1 processes);
>>>
>>> [2] proxy: 10.0.0.2 (1 cores)
>>> Exec list: /home/hoot/mvapich2-1.8-r5435/osu_benchmarks/osu_bw
>>> (1 processes);
>>>
>>>
>>> ======================================================================
>>> ============================
>>>
>>> [mpiexec at sandyhp1] Timeout set to -1 (-1 means infinite)
>>> [mpiexec at sandyhp1] Got a control port string of 10.0.0.1:50740
>>>
>>> Proxy launch args: /usr/local/bin/hydra_pmi_proxy --control-port
>>> 10.0.0.1:50740 --debug --rmk user --launcher ssh --demux poll --pgid
>>> 0 --retries 10 --proxy-id
>>>
>>> [mpiexec at sandyhp1] PMI FD: (null); PMI PORT: (null); PMI ID/RANK: -1
>>> Arguments being passed to proxy 0:
>>> --version 1.4.1p1 --iface-ip-env-name MPICH_INTERFACE_HOSTNAME
>>> --hostname 10.0.0.1 --global-core-map 0,1,1 --filler-process-map
>>> 0,1,1 --global-process-count 2 --auto-cleanup 1 --pmi-rank -1
>>> --pmi-kvsname kvs_7644_0 --pmi-process-mapping (vector,(0,2,1))
>>> --ckpoint-num -1 --global-inherited-env 29 'HOSTNAME=sandyhp1'
>>> 'SELINUX_ROLE_REQUESTED=' 'TERM=xterm' 'SHELL=/bin/bash'
>>> 'HISTSIZE=1000' 'SSH_CLIENT=169.154.148.10 42095 22'
>>> 'SELINUX_USE_CURRENT_RANGE=' 'QTDIR=/usr/lib64/qt-3.3'
>>> 'QTINC=/usr/lib64/qt-3.3/include' 'SSH_TTY=/dev/pts/0' 'USER=hoot'
>> 'LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;
>> 01:cd=40;33;01:or=40;31;01:mi=01;05;37;41:su=37;41:sg=30;43:ca=30;41:tw=30;4
>> 2:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arj=01;31:*.taz=01;31
>> :*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.zip=01;31:*.z=01;31:*.Z=
>> 01;31:*.dz=01;31:*.gz=01;31:*.lz=01;31:*.xz=01;31:*.bz2=01;31:*.tbz=01;31:*.
>> tbz2=01;31:*.bz=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.rar=0
>> 1;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.jpg=01;35:
>> *.jpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.t
>> ga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=
>> 01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01
>> ;35:*.m2v=01;35:*.mkv=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35
>> :*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmv
>> b=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;3
>> 5!
>> :*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.axv=01;35:*.a
>> nx=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=01;36:*.au=01;36:*.flac=01;36:*.mid=0
>> 1;36:*.midi=01;36:*.mka=01;36:*.mp3=01;36:*.mpc=01;36:*.ogg=01;36:*.ra=01;36
>> :*.wav=01;36:*.axa=01;36:*.oga=01;36:*.spx=01;36:*.xspf=01;36:'
>>> 'MAIL=/var/spool/mail/hoot'
>> 'PATH=/usr/lib64/qt-3.3/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/us
>> r/sbin:/sbin:/home/hoot/bin'
>>> 'PWD=/home/hoot/mvapich2-1.8-r5435/osu_benchmarks'
>>> 'LANG=en_US.UTF-8' 'SELINUX_LEVEL_REQUESTED='
>>> 'SSH_ASKPASS=/usr/libexec/openssh/gnome-ssh-askpass'
>>> 'HISTCONTROL=ignoredups' 'SHLVL=1' 'HOME=/home/hoot' 'LOGNAME=hoot'
>>> 'QTLIB=/usr/lib64/qt-3.3/lib' 'CVS_RSH=ssh'
>>> 'SSH_CONNECTION=169.154.148.10 42095 169.154.148.45 22'
>>> 'LESSOPEN=|/usr/bin/lesspipe.sh %s' 'G_BROKEN_FILENAMES=1'
>>> '_=/usr/local/bin/mpiexec' 'OLDPWD=/home/hoot/mvapich2-1.8-r5435'
>>> --global-user-env 0 --global-system-env 1
>>> 'GFORTRAN_UNBUFFERED_PRECONNECTED=y' --proxy-core-count 1 --exec
>>> --exec-appnum 0 --exec-proc-count 1 --exec-local-env 0 --exec-wdir
>>> /home/hoot/mvapich2-1.8-r5435/osu_benchmarks --exec-args 1
>>> /home/hoot/mvapich2-1.8-r5435/osu_benchmarks/osu_bw
>>>
>>> [mpiexec at sandyhp1] PMI FD: (null); PMI PORT: (null); PMI ID/RANK: -1
>>> Arguments being passed to proxy 1:
>>> --version 1.4.1p1 --iface-ip-env-name MPICH_INTERFACE_HOSTNAME
>>> --hostname 10.0.0.2 --global-core-map 1,1,0 --filler-process-map
>>> 1,1,0 --global-process-count 2 --auto-cleanup 1 --pmi-rank -1
>>> --pmi-kvsname kvs_7644_0 --pmi-process-mapping (vector,(0,2,1))
>>> --ckpoint-num -1 --global-inherited-env 29 'HOSTNAME=sandyhp1'
>>> 'SELINUX_ROLE_REQUESTED=' 'TERM=xterm' 'SHELL=/bin/bash'
>>> 'HISTSIZE=1000' 'SSH_CLIENT=169.154.148.10 42095 22'
>>> 'SELINUX_USE_CURRENT_RANGE=' 'QTDIR=/usr/lib64/qt-3.3'
>>> 'QTINC=/usr/lib64/qt-3.3/include' 'SSH_TTY=/dev/pts/0' 'USER=hoot'
>> 'LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;
>> 01:cd=40;33;01:or=40;31;01:mi=01;05;37;41:su=37;41:sg=30;43:ca=30;41:tw=30;4
>> 2:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arj=01;31:*.taz=01;31
>> :*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.zip=01;31:*.z=01;31:*.Z=
>> 01;31:*.dz=01;31:*.gz=01;31:*.lz=01;31:*.xz=01;31:*.bz2=01;31:*.tbz=01;31:*.
>> tbz2=01;31:*.bz=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.rar=0
>> 1;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.jpg=01;35:
>> *.jpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.t
>> ga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=
>> 01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01
>> ;35:*.m2v=01;35:*.mkv=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35
>> :*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmv
>> b=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;3
>> 5!
>> :*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.axv=01;35:*.a
>> nx=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=01;36:*.au=01;36:*.flac=01;36:*.mid=0
>> 1;36:*.midi=01;36:*.mka=01;36:*.mp3=01;36:*.mpc=01;36:*.ogg=01;36:*.ra=01;36
>> :*.wav=01;36:*.axa=01;36:*.oga=01;36:*.spx=01;36:*.xspf=01;36:'
>>> 'MAIL=/var/spool/mail/hoot'
>> 'PATH=/usr/lib64/qt-3.3/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/us
>> r/sbin:/sbin:/home/hoot/bin'
>>> 'PWD=/home/hoot/mvapich2-1.8-r5435/osu_benchmarks'
>>> 'LANG=en_US.UTF-8' 'SELINUX_LEVEL_REQUESTED='
>>> 'SSH_ASKPASS=/usr/libexec/openssh/gnome-ssh-askpass'
>>> 'HISTCONTROL=ignoredups' 'SHLVL=1' 'HOME=/home/hoot' 'LOGNAME=hoot'
>>> 'QTLIB=/usr/lib64/qt-3.3/lib' 'CVS_RSH=ssh'
>>> 'SSH_CONNECTION=169.154.148.10 42095 169.154.148.45 22'
>>> 'LESSOPEN=|/usr/bin/lesspipe.sh %s' 'G_BROKEN_FILENAMES=1'
>>> '_=/usr/local/bin/mpiexec' 'OLDPWD=/home/hoot/mvapich2-1.8-r5435'
>>> --global-user-env 0 --global-system-env 1
>>> 'GFORTRAN_UNBUFFERED_PRECONNECTED=y' --proxy-core-count 1 --exec
>>> --exec-appnum 0 --exec-proc-count 1 --exec-local-env 0 --exec-wdir
>>> /home/hoot/mvapich2-1.8-r5435/osu_benchmarks --exec-args 1
>>> /home/hoot/mvapich2-1.8-r5435/osu_benchmarks/osu_bw
>>>
>>> [mpiexec at sandyhp1] Launch arguments: /usr/local/bin/hydra_pmi_proxy
>>> --control-port 10.0.0.1:50740 --debug --rmk user --launcher ssh
>>> --demux poll --pgid 0 --retries 10 --proxy-id 0 [mpiexec at sandyhp1]
>>> Launch arguments: /usr/bin/ssh -x 10.0.0.2
>>> "/usr/local/bin/hydra_pmi_proxy" --control-port 10.0.0.1:50740 --debug
>>> --rmk user --launcher ssh --demux poll --pgid 0 --retries 10
>>> --proxy-id 1 [proxy:0:0 at sandyhp1] got pmi command (from 0): init
>>> pmi_version=1 pmi_subversion=1
>>> [proxy:0:0 at sandyhp1] PMI response: cmd=response_to_init
>>> pmi_version=1 pmi_subversion=1 rc=0
>>> [proxy:0:0 at sandyhp1] got pmi command (from 0): get_maxes
>>>
>>> [proxy:0:0 at sandyhp1] PMI response: cmd=maxes kvsname_max=256
>>> keylen_max=64 vallen_max=1024
>>> [proxy:0:0 at sandyhp1] got pmi command (from 0): get_appnum
>>>
>>> [proxy:0:0 at sandyhp1] PMI response: cmd=appnum appnum=0
>>> [proxy:0:0 at sandyhp1] got pmi command (from 0): get_my_kvsname
>>>
>>> [proxy:0:0 at sandyhp1] PMI response: cmd=my_kvsname kvsname=kvs_7644_0
>>> [proxy:0:0 at sandyhp1] got pmi command (from 0): get_my_kvsname
>>>
>>> [proxy:0:0 at sandyhp1] PMI response: cmd=my_kvsname kvsname=kvs_7644_0
>>> [proxy:0:0 at sandyhp1] got pmi command (from 0): get
>>> kvsname=kvs_7644_0 key=PMI_process_mapping [proxy:0:0 at sandyhp1] PMI
>>> response: cmd=get_result rc=0 msg=success
>>> value=(vector,(0,2,1))
>>> [cli_0]: aborting job:
>>> Fatal error in MPI_Init:
>>> Other MPI error
>>>
>>>
>>>
>> ============================================================================
>> =========
>>> = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
>>> = EXIT CODE: 256
>>> = CLEANING UP REMAINING PROCESSES
>>> = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES
>>> ======================================================================
>>> =============== [proxy:0:1 at sandyhp2] got pmi command (from 4): init
>>> pmi_version=1 pmi_subversion=1
>>> [proxy:0:1 at sandyhp2] PMI response: cmd=response_to_init
>>> pmi_version=1 pmi_subversion=1 rc=0
>>> [proxy:0:1 at sandyhp2] got pmi command (from 4): get_maxes
>>>
>>> [proxy:0:1 at sandyhp2] PMI response: cmd=maxes kvsname_max=256
>>> keylen_max=64 vallen_max=1024
>>> [proxy:0:1 at sandyhp2] got pmi command (from 4): get_appnum
>>>
>>> [proxy:0:1 at sandyhp2] PMI response: cmd=appnum appnum=0
>>> [proxy:0:1 at sandyhp2] got pmi command (from 4): get_my_kvsname
>>>
>>> [proxy:0:1 at sandyhp2] PMI response: cmd=my_kvsname kvsname=kvs_7644_0
>>> [proxy:0:1 at sandyhp2] got pmi command (from 4): get_my_kvsname
>>>
>>> [proxy:0:1 at sandyhp2] PMI response: cmd=my_kvsname kvsname=kvs_7644_0
>>> [proxy:0:1 at sandyhp2] got pmi command (from 4): get
>>> kvsname=kvs_7644_0 key=PMI_process_mapping [proxy:0:1 at sandyhp2] PMI
>>> response: cmd=get_result rc=0 msg=success
>>> value=(vector,(0,2,1))
>>> [cli_1]: aborting job:
>>> Fatal error in MPI_Init:
>>> Other MPI error
>>>
>>>
>>>
>> ============================================================================
>> =========
>>> = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
>>> = EXIT CODE: 256
>>> = CLEANING UP REMAINING PROCESSES
>>> = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES
>>> ======================================================================
>>> ===============
>>>
>>>
>>> ***************** Running on the initiating server
>>> [hoot at sandyhp1 osu_benchmarks]$ mpiexec -verbose -hosts
>>> 10.0.0.1,10.0.0.1
>>> /home/hoot/mvapich2-1.8-r5435/osu_benchmarks/osu_bw
>>> host: 10.0.0.1
>>>
>>> ======================================================================
>>> ============================
>>> mpiexec options:
>>> ----------------
>>> Base path: /usr/local/bin/
>>> Launcher: (null)
>>> Debug level: 1
>>> Enable X: -1
>>>
>>> Global environment:
>>> -------------------
>>> HOSTNAME=sandyhp1
>>> SELINUX_ROLE_REQUESTED=
>>> TERM=xterm
>>> SHELL=/bin/bash
>>> HISTSIZE=1000
>>> SSH_CLIENT=169.154.148.10 42095 22
>>> SELINUX_USE_CURRENT_RANGE=
>>> QTDIR=/usr/lib64/qt-3.3
>>> QTINC=/usr/lib64/qt-3.3/include
>>> SSH_TTY=/dev/pts/0
>>> USER=hoot
>>>
>> LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;0
>> 1:cd=40;33;01:or=40;31;01:mi=01;05;37;41:su=37;41:sg=30;43:ca=30;41:tw=30;42
>> :ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arj=01;31:*.taz=01;31:
>> *.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.zip=01;31:*.z=01;31:*.Z=0
>> 1;31:*.dz=01;31:*.gz=01;31:*.lz=01;31:*.xz=01;31:*.bz2=01;31:*.tbz=01;31:*.t
>> bz2=01;31:*.bz=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.rar=01
>> ;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.jpg=01;35:*
>> .jpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tg
>> a=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=0
>> 1;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;
>> 35:*.m2v=01;35:*.mkv=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:
>> *.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb
>> =01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35
>> :!
>> *.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.axv=01;35:*.an
>> x=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=01;36:*.au=01;36:*.flac=01;36:*.mid=01
>> ;36:*.midi=01;36:*.mka=01;36:*.mp3=01;36:*.mpc=01;36:*.ogg=01;36:*.ra=01;36:
>> *.wav=01;36:*.axa=01;36:*.oga=01;36:*.spx=01;36:*.xspf=01;36:
>>> MAIL=/var/spool/mail/hoot
>>>
>> PATH=/usr/lib64/qt-3.3/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr
>> /sbin:/sbin:/home/hoot/bin
>>> PWD=/home/hoot/mvapich2-1.8-r5435/osu_benchmarks
>>> LANG=en_US.UTF-8
>>> SELINUX_LEVEL_REQUESTED=
>>> SSH_ASKPASS=/usr/libexec/openssh/gnome-ssh-askpass
>>> HISTCONTROL=ignoredups
>>> SHLVL=1
>>> HOME=/home/hoot
>>> LOGNAME=hoot
>>> QTLIB=/usr/lib64/qt-3.3/lib
>>> CVS_RSH=ssh
>>> SSH_CONNECTION=169.154.148.10 42095 169.154.148.45 22
>>> LESSOPEN=|/usr/bin/lesspipe.sh %s
>>> G_BROKEN_FILENAMES=1
>>> _=/usr/local/bin/mpiexec
>>> OLDPWD=/home/hoot/mvapich2-1.8-r5435
>>>
>>> Hydra internal environment:
>>> ---------------------------
>>> GFORTRAN_UNBUFFERED_PRECONNECTED=y
>>>
>>>
>>> Proxy information:
>>> *********************
>>> [1] proxy: 10.0.0.1 (2 cores)
>>> Exec list: /home/hoot/mvapich2-1.8-r5435/osu_benchmarks/osu_bw
>>> (2 processes);
>>>
>>>
>>> ======================================================================
>>> ============================
>>>
>>> [mpiexec at sandyhp1] Timeout set to -1 (-1 means infinite)
>>> [mpiexec at sandyhp1] Got a control port string of 10.0.0.1:56066
>>>
>>> Proxy launch args: /usr/local/bin/hydra_pmi_proxy --control-port
>>> 10.0.0.1:56066 --debug --rmk user --launcher ssh --demux poll --pgid
>>> 0 --retries 10 --proxy-id
>>>
>>> [mpiexec at sandyhp1] PMI FD: (null); PMI PORT: (null); PMI ID/RANK: -1
>>> Arguments being passed to proxy 0:
>>> --version 1.4.1p1 --iface-ip-env-name MPICH_INTERFACE_HOSTNAME
>>> --hostname 10.0.0.1 --global-core-map 0,2,0 --filler-process-map
>>> 0,2,0 --global-process-count 2 --auto-cleanup 1 --pmi-rank -1
>>> --pmi-kvsname kvs_7484_0 --pmi-process-mapping (vector,(0,1,2))
>>> --ckpoint-num -1 --global-inherited-env 29 'HOSTNAME=sandyhp1'
>>> 'SELINUX_ROLE_REQUESTED=' 'TERM=xterm' 'SHELL=/bin/bash'
>>> 'HISTSIZE=1000' 'SSH_CLIENT=169.154.148.10 42095 22'
>>> 'SELINUX_USE_CURRENT_RANGE=' 'QTDIR=/usr/lib64/qt-3.3'
>>> 'QTINC=/usr/lib64/qt-3.3/include' 'SSH_TTY=/dev/pts/0' 'USER=hoot'
>> 'LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;
>> 01:cd=40;33;01:or=40;31;01:mi=01;05;37;41:su=37;41:sg=30;43:ca=30;41:tw=30;4
>> 2:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arj=01;31:*.taz=01;31
>> :*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.zip=01;31:*.z=01;31:*.Z=
>> 01;31:*.dz=01;31:*.gz=01;31:*.lz=01;31:*.xz=01;31:*.bz2=01;31:*.tbz=01;31:*.
>> tbz2=01;31:*.bz=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.rar=0
>> 1;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.jpg=01;35:
>> *.jpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.t
>> ga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=
>> 01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01
>> ;35:*.m2v=01;35:*.mkv=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35
>> :*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmv
>> b=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;3
>> 5!
>> :*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.axv=01;35:*.a
>> nx=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=01;36:*.au=01;36:*.flac=01;36:*.mid=0
>> 1;36:*.midi=01;36:*.mka=01;36:*.mp3=01;36:*.mpc=01;36:*.ogg=01;36:*.ra=01;36
>> :*.wav=01;36:*.axa=01;36:*.oga=01;36:*.spx=01;36:*.xspf=01;36:'
>>> 'MAIL=/var/spool/mail/hoot'
>> 'PATH=/usr/lib64/qt-3.3/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/us
>> r/sbin:/sbin:/home/hoot/bin'
>>> 'PWD=/home/hoot/mvapich2-1.8-r5435/osu_benchmarks'
>>> 'LANG=en_US.UTF-8' 'SELINUX_LEVEL_REQUESTED='
>>> 'SSH_ASKPASS=/usr/libexec/openssh/gnome-ssh-askpass'
>>> 'HISTCONTROL=ignoredups' 'SHLVL=1' 'HOME=/home/hoot' 'LOGNAME=hoot'
>>> 'QTLIB=/usr/lib64/qt-3.3/lib' 'CVS_RSH=ssh'
>>> 'SSH_CONNECTION=169.154.148.10 42095 169.154.148.45 22'
>>> 'LESSOPEN=|/usr/bin/lesspipe.sh %s' 'G_BROKEN_FILENAMES=1'
>>> '_=/usr/local/bin/mpiexec' 'OLDPWD=/home/hoot/mvapich2-1.8-r5435'
>>> --global-user-env 0 --global-system-env 1
>>> 'GFORTRAN_UNBUFFERED_PRECONNECTED=y' --proxy-core-count 2 --exec
>>> --exec-appnum 0 --exec-proc-count 2 --exec-local-env 0 --exec-wdir
>>> /home/hoot/mvapich2-1.8-r5435/osu_benchmarks --exec-args 1
>>> /home/hoot/mvapich2-1.8-r5435/osu_benchmarks/osu_bw
>>>
>>> [mpiexec at sandyhp1] Launch arguments: /usr/local/bin/hydra_pmi_proxy
>>> --control-port 10.0.0.1:56066 --debug --rmk user --launcher ssh
>>> --demux poll --pgid 0 --retries 10 --proxy-id 0 [proxy:0:0 at sandyhp1]
>>> got pmi command (from 0): init
>>> pmi_version=1 pmi_subversion=1
>>> [proxy:0:0 at sandyhp1] PMI response: cmd=response_to_init
>>> pmi_version=1 pmi_subversion=1 rc=0
>>> [proxy:0:0 at sandyhp1] got pmi command (from 6): init
>>> pmi_version=1 pmi_subversion=1
>>> [proxy:0:0 at sandyhp1] PMI response: cmd=response_to_init
>>> pmi_version=1 pmi_subversion=1 rc=0
>>> [proxy:0:0 at sandyhp1] got pmi command (from 0): get_maxes
>>>
>>> [proxy:0:0 at sandyhp1] PMI response: cmd=maxes kvsname_max=256
>>> keylen_max=64 vallen_max=1024
>>> [proxy:0:0 at sandyhp1] got pmi command (from 6): get_maxes
>>>
>>> [proxy:0:0 at sandyhp1] PMI response: cmd=maxes kvsname_max=256
>>> keylen_max=64 vallen_max=1024
>>> [proxy:0:0 at sandyhp1] got pmi command (from 0): get_appnum
>>>
>>> [proxy:0:0 at sandyhp1] PMI response: cmd=appnum appnum=0
>>> [proxy:0:0 at sandyhp1] got pmi command (from 0): get_my_kvsname
>>>
>>> [proxy:0:0 at sandyhp1] PMI response: cmd=my_kvsname kvsname=kvs_7484_0
>>> [proxy:0:0 at sandyhp1] got pmi command (from 6): get_appnum
>>>
>>> [proxy:0:0 at sandyhp1] PMI response: cmd=appnum appnum=0
>>> [proxy:0:0 at sandyhp1] got pmi command (from 0): get_my_kvsname
>>>
>>> [proxy:0:0 at sandyhp1] PMI response: cmd=my_kvsname kvsname=kvs_7484_0
>>> [proxy:0:0 at sandyhp1] got pmi command (from 6): get_my_kvsname
>>>
>>> [proxy:0:0 at sandyhp1] PMI response: cmd=my_kvsname kvsname=kvs_7484_0
>>> [proxy:0:0 at sandyhp1] got pmi command (from 0): get
>>> kvsname=kvs_7484_0 key=PMI_process_mapping [proxy:0:0 at sandyhp1] PMI
>>> response: cmd=get_result rc=0 msg=success
>>> value=(vector,(0,1,2))
>>> [proxy:0:0 at sandyhp1] got pmi command (from 6): get_my_kvsname
>>>
>>> [proxy:0:0 at sandyhp1] PMI response: cmd=my_kvsname kvsname=kvs_7484_0
>>> [proxy:0:0 at sandyhp1] got pmi command (from 6): get
>>> kvsname=kvs_7484_0 key=PMI_process_mapping [proxy:0:0 at sandyhp1] PMI
>>> response: cmd=get_result rc=0 msg=success
>>> value=(vector,(0,1,2))
>>> # OSU MPI Bandwidth Test v3.6
>>> # Size Bandwidth (MB/s)
>>> 1 2.76
>>> 2 5.54
>>> 4 11.09
>>> 8 22.27
>>> 16 44.48
>>> 32 87.19
>>> 64 171.52
>>> 128 331.24
>>> 256 612.25
>>> 512 1014.16
>>> 1024 1585.68
>>> 2048 2590.26
>>> 4096 3744.42
>>> 8192 4721.57
>>> 16384 4940.24
>>> 32768 4258.99
>>> 65536 4450.17
>>> 131072 4461.91
>>> 262144 4126.22
>>> 524288 4103.56
>>> 1048576 5106.87
>>> 2097152 5126.66
>>> 4194304 5161.63
>>> [proxy:0:0 at sandyhp1] got pmi command (from 6): finalize
>>>
>>> [proxy:0:0 at sandyhp1] PMI response: cmd=finalize_ack
>>> [proxy:0:0 at sandyhp1] got pmi command (from 0): finalize
>>>
>>> [proxy:0:0 at sandyhp1] PMI response: cmd=finalize_ack
>>>
>>>
>>>
>>>
>>> **************** Running on the remote server
>>>
>>> [hoot at sandyhp1 osu_benchmarks]$ mpiexec -verbose -hosts
>>> 10.0.0.2,10.0.0.2
>>> /home/hoot/mvapich2-1.8-r5435/osu_benchmarks/osu_bw
>>> host: 10.0.0.2
>>>
>>> ======================================================================
>>> ============================
>>> mpiexec options:
>>> ----------------
>>> Base path: /usr/local/bin/
>>> Launcher: (null)
>>> Debug level: 1
>>> Enable X: -1
>>>
>>> Global environment:
>>> -------------------
>>> HOSTNAME=sandyhp1
>>> SELINUX_ROLE_REQUESTED=
>>> TERM=xterm
>>> SHELL=/bin/bash
>>> HISTSIZE=1000
>>> SSH_CLIENT=169.154.148.10 42095 22
>>> SELINUX_USE_CURRENT_RANGE=
>>> QTDIR=/usr/lib64/qt-3.3
>>> QTINC=/usr/lib64/qt-3.3/include
>>> SSH_TTY=/dev/pts/0
>>> USER=hoot
>>>
>> LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;0
>> 1:cd=40;33;01:or=40;31;01:mi=01;05;37;41:su=37;41:sg=30;43:ca=30;41:tw=30;42
>> :ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arj=01;31:*.taz=01;31:
>> *.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.zip=01;31:*.z=01;31:*.Z=0
>> 1;31:*.dz=01;31:*.gz=01;31:*.lz=01;31:*.xz=01;31:*.bz2=01;31:*.tbz=01;31:*.t
>> bz2=01;31:*.bz=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.rar=01
>> ;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.jpg=01;35:*
>> .jpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tg
>> a=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=0
>> 1;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;
>> 35:*.m2v=01;35:*.mkv=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:
>> *.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb
>> =01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35
>> :!
>> *.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.axv=01;35:*.an
>> x=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=01;36:*.au=01;36:*.flac=01;36:*.mid=01
>> ;36:*.midi=01;36:*.mka=01;36:*.mp3=01;36:*.mpc=01;36:*.ogg=01;36:*.ra=01;36:
>> *.wav=01;36:*.axa=01;36:*.oga=01;36:*.spx=01;36:*.xspf=01;36:
>>> MAIL=/var/spool/mail/hoot
>>>
>> PATH=/usr/lib64/qt-3.3/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr
>> /sbin:/sbin:/home/hoot/bin
>>> PWD=/home/hoot/mvapich2-1.8-r5435/osu_benchmarks
>>> LANG=en_US.UTF-8
>>> SELINUX_LEVEL_REQUESTED=
>>> SSH_ASKPASS=/usr/libexec/openssh/gnome-ssh-askpass
>>> HISTCONTROL=ignoredups
>>> SHLVL=1
>>> HOME=/home/hoot
>>> LOGNAME=hoot
>>> QTLIB=/usr/lib64/qt-3.3/lib
>>> CVS_RSH=ssh
>>> SSH_CONNECTION=169.154.148.10 42095 169.154.148.45 22
>>> LESSOPEN=|/usr/bin/lesspipe.sh %s
>>> G_BROKEN_FILENAMES=1
>>> _=/usr/local/bin/mpiexec
>>> OLDPWD=/home/hoot/mvapich2-1.8-r5435
>>>
>>> Hydra internal environment:
>>> ---------------------------
>>> GFORTRAN_UNBUFFERED_PRECONNECTED=y
>>>
>>>
>>> Proxy information:
>>> *********************
>>> [1] proxy: 10.0.0.2 (2 cores)
>>> Exec list: /home/hoot/mvapich2-1.8-r5435/osu_benchmarks/osu_bw
>>> (2 processes);
>>>
>>>
>>> ======================================================================
>>> ============================
>>>
>>> [mpiexec at sandyhp1] Timeout set to -1 (-1 means infinite)
>>> [mpiexec at sandyhp1] Got a control port string of sandyhp1:39504
>>>
>>> Proxy launch args: /usr/local/bin/hydra_pmi_proxy --control-port
>>> sandyhp1:39504 --debug --rmk user --launcher ssh --demux poll --pgid
>>> 0 --retries 10 --proxy-id
>>>
>>> [mpiexec at sandyhp1] PMI FD: (null); PMI PORT: (null); PMI ID/RANK: -1
>>> Arguments being passed to proxy 0:
>>> --version 1.4.1p1 --iface-ip-env-name MPICH_INTERFACE_HOSTNAME
>>> --hostname 10.0.0.2 --global-core-map 0,2,0 --filler-process-map
>>> 0,2,0 --global-process-count 2 --auto-cleanup 1 --pmi-rank -1
>>> --pmi-kvsname kvs_7496_0 --pmi-process-mapping (vector,(0,1,2))
>>> --ckpoint-num -1 --global-inherited-env 29 'HOSTNAME=sandyhp1'
>>> 'SELINUX_ROLE_REQUESTED=' 'TERM=xterm' 'SHELL=/bin/bash'
>>> 'HISTSIZE=1000' 'SSH_CLIENT=169.154.148.10 42095 22'
>>> 'SELINUX_USE_CURRENT_RANGE=' 'QTDIR=/usr/lib64/qt-3.3'
>>> 'QTINC=/usr/lib64/qt-3.3/include' 'SSH_TTY=/dev/pts/0' 'USER=hoot'
>> 'LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;
>> 01:cd=40;33;01:or=40;31;01:mi=01;05;37;41:su=37;41:sg=30;43:ca=30;41:tw=30;4
>> 2:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arj=01;31:*.taz=01;31
>> :*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.zip=01;31:*.z=01;31:*.Z=
>> 01;31:*.dz=01;31:*.gz=01;31:*.lz=01;31:*.xz=01;31:*.bz2=01;31:*.tbz=01;31:*.
>> tbz2=01;31:*.bz=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.rar=0
>> 1;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.jpg=01;35:
>> *.jpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.t
>> ga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=
>> 01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01
>> ;35:*.m2v=01;35:*.mkv=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35
>> :*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmv
>> b=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;3
>> 5!
>> :*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.axv=01;35:*.a
>> nx=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=01;36:*.au=01;36:*.flac=01;36:*.mid=0
>> 1;36:*.midi=01;36:*.mka=01;36:*.mp3=01;36:*.mpc=01;36:*.ogg=01;36:*.ra=01;36
>> :*.wav=01;36:*.axa=01;36:*.oga=01;36:*.spx=01;36:*.xspf=01;36:'
>>> 'MAIL=/var/spool/mail/hoot'
>> 'PATH=/usr/lib64/qt-3.3/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/us
>> r/sbin:/sbin:/home/hoot/bin'
>>> 'PWD=/home/hoot/mvapich2-1.8-r5435/osu_benchmarks'
>>> 'LANG=en_US.UTF-8' 'SELINUX_LEVEL_REQUESTED='
>>> 'SSH_ASKPASS=/usr/libexec/openssh/gnome-ssh-askpass'
>>> 'HISTCONTROL=ignoredups' 'SHLVL=1' 'HOME=/home/hoot' 'LOGNAME=hoot'
>>> 'QTLIB=/usr/lib64/qt-3.3/lib' 'CVS_RSH=ssh'
>>> 'SSH_CONNECTION=169.154.148.10 42095 169.154.148.45 22'
>>> 'LESSOPEN=|/usr/bin/lesspipe.sh %s' 'G_BROKEN_FILENAMES=1'
>>> '_=/usr/local/bin/mpiexec' 'OLDPWD=/home/hoot/mvapich2-1.8-r5435'
>>> --global-user-env 0 --global-system-env 1
>>> 'GFORTRAN_UNBUFFERED_PRECONNECTED=y' --proxy-core-count 2 --exec
>>> --exec-appnum 0 --exec-proc-count 2 --exec-local-env 0 --exec-wdir
>>> /home/hoot/mvapich2-1.8-r5435/osu_benchmarks --exec-args 1
>>> /home/hoot/mvapich2-1.8-r5435/osu_benchmarks/osu_bw
>>>
>>> [mpiexec at sandyhp1] Launch arguments: /usr/bin/ssh -x 10.0.0.2
>>> "/usr/local/bin/hydra_pmi_proxy" --control-port sandyhp1:39504 --debug
>>> --rmk user --launcher ssh --demux poll --pgid 0 --retries 10
>>> --proxy-id 0 [proxy:0:0 at sandyhp2] got pmi command (from 4): init
>>> pmi_version=1 pmi_subversion=1
>>> [proxy:0:0 at sandyhp2] PMI response: cmd=response_to_init
>>> pmi_version=1 pmi_subversion=1 rc=0
>>> [proxy:0:0 at sandyhp2] got pmi command (from 4): get_maxes
>>>
>>> [proxy:0:0 at sandyhp2] PMI response: cmd=maxes kvsname_max=256
>>> keylen_max=64 vallen_max=1024
>>> [proxy:0:0 at sandyhp2] got pmi command (from 5): init
>>> pmi_version=1 pmi_subversion=1
>>> [proxy:0:0 at sandyhp2] PMI response: cmd=response_to_init
>>> pmi_version=1 pmi_subversion=1 rc=0
>>> [proxy:0:0 at sandyhp2] got pmi command (from 4): get_appnum
>>>
>>> [proxy:0:0 at sandyhp2] PMI response: cmd=appnum appnum=0
>>> [proxy:0:0 at sandyhp2] got pmi command (from 4): get_my_kvsname
>>>
>>> [proxy:0:0 at sandyhp2] PMI response: cmd=my_kvsname kvsname=kvs_7496_0
>>> [proxy:0:0 at sandyhp2] got pmi command (from 4): get_my_kvsname
>>>
>>> [proxy:0:0 at sandyhp2] PMI response: cmd=my_kvsname kvsname=kvs_7496_0
>>> [proxy:0:0 at sandyhp2] got pmi command (from 5): get_maxes
>>>
>>> [proxy:0:0 at sandyhp2] PMI response: cmd=maxes kvsname_max=256
>>> keylen_max=64 vallen_max=1024
>>> [proxy:0:0 at sandyhp2] got pmi command (from 5): get_appnum
>>>
>>> [proxy:0:0 at sandyhp2] PMI response: cmd=appnum appnum=0
>>> [proxy:0:0 at sandyhp2] got pmi command (from 5): get_my_kvsname
>>>
>>> [proxy:0:0 at sandyhp2] PMI response: cmd=my_kvsname kvsname=kvs_7496_0
>>> [proxy:0:0 at sandyhp2] got pmi command (from 5): get_my_kvsname
>>>
>>> [proxy:0:0 at sandyhp2] PMI response: cmd=my_kvsname kvsname=kvs_7496_0
>>> [proxy:0:0 at sandyhp2] got pmi command (from 5): get
>>> kvsname=kvs_7496_0 key=PMI_process_mapping [proxy:0:0 at sandyhp2] PMI
>>> response: cmd=get_result rc=0 msg=success
>>> value=(vector,(0,1,2))
>>> [proxy:0:0 at sandyhp2] got pmi command (from 4): get
>>> kvsname=kvs_7496_0 key=PMI_process_mapping [proxy:0:0 at sandyhp2] PMI
>>> response: cmd=get_result rc=0 msg=success
>>> value=(vector,(0,1,2))
>>> # OSU MPI Bandwidth Test v3.6
>>> # Size Bandwidth (MB/s)
>>> 1 2.76
>>> 2 5.55
>>> 4 11.07
>>> 8 21.98
>>> 16 44.50
>>> 32 86.66
>>> 64 170.38
>>> 128 327.30
>>> 256 607.49
>>> 512 1016.71
>>> 1024 1555.88
>>> 2048 2598.09
>>> 4096 3748.63
>>> 8192 4724.61
>>> 16384 4910.18
>>> 32768 4245.63
>>> 65536 4427.21
>>> 131072 4467.74
>>> 262144 4155.35
>>> 524288 4066.51
>>> 1048576 4351.23
>>> 2097152 5121.67
>>> 4194304 5669.35
>>> [proxy:0:0 at sandyhp2] got pmi command (from 5): finalize
>>>
>>> [proxy:0:0 at sandyhp2] PMI response: cmd=finalize_ack
>>> [proxy:0:0 at sandyhp2] got pmi command (from 4): finalize
>>>
>>> [proxy:0:0 at sandyhp2] PMI response: cmd=finalize_ack
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> mvapich-discuss mailing list
>>> mvapich-discuss at cse.ohio-state.edu
>>> http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss
>>>
>> --
>> Jonathan Perkins
>> http://www.cse.ohio-state.edu/~perkinjo
>>
>>
More information about the mvapich-discuss
mailing list