[mvapich-discuss] mpi init error
Jonathan Perkins
perkinjo at cse.ohio-state.edu
Fri Jun 15 13:57:28 EDT 2012
No problem, glad its working.
On Fri, Jun 15, 2012 at 11:27:40AM -0400, Hoot Thompson wrote:
> Perfect, that fixed it. Seems like I had this same problem long ago and
> obviously forgot the fix. Thanks for helping again.
>
> Hoot
>
> -----Original Message-----
> From: Jonathan Perkins [mailto:perkinjo at cse.ohio-state.edu]
> Sent: Friday, June 15, 2012 11:00 AM
> To: Hoot Thompson
> Cc: mvapich-discuss at cse.ohio-state.edu
> Subject: Re: [mvapich-discuss] mpi init error
>
> Thanks, it looks like this problem may be caused by not being able to lock
> enough memory.
>
> Can you please check ulimit -l on each node and make sure its unlimited (or
> something much larger than 64). The following link should help you in
> changing this setting if it is indeed 64.
>
> http://mvapich.cse.ohio-state.edu/support/user_guide_mvapich2-1.8.html#x1-12
> 50009.4.3
>
> On Fri, Jun 15, 2012 at 10:55:49AM -0400, Hoot Thompson wrote:
> > Here you go......
> >
> > [hoot at sandyhp1 osu_benchmarks]$ mpiexec -n 2 -hosts
> > 10.0.0.1,10.0.0.2
> > /home/hoot/mvapich2-1.8-r5435/osu_benchmarks/osu_bw
> > [cli_0]: aborting job:
> > Fatal error in MPI_Init:
> > Other MPI error, error stack:
> > MPIR_Init_thread(408).......:
> > MPID_Init(296)..............: channel initialization failed
> > MPIDI_CH3_Init(283).........:
> > MPIDI_CH3I_RDMA_init(172)...:
> > rdma_setup_startup_ring(434): cannot create cq
> >
> >
> >
> ============================================================================
> =========
> > = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
> > = EXIT CODE: 256
> > = CLEANING UP REMAINING PROCESSES
> > = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES
> > ======================================================================
> > ===============
> > [cli_1]: aborting job:
> > Fatal error in MPI_Init:
> > Other MPI error, error stack:
> > MPIR_Init_thread(408).......:
> > MPID_Init(296)..............: channel initialization failed
> > MPIDI_CH3_Init(283).........:
> > MPIDI_CH3I_RDMA_init(172)...:
> > rdma_setup_startup_ring(434): cannot create cq
> >
> >
> >
> ============================================================================
> =========
> > = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
> > = EXIT CODE: 256
> > = CLEANING UP REMAINING PROCESSES
> > = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > On 06/15/2012 10:47 AM, Jonathan Perkins wrote:
> > >To be honest, I would rather you leave the verbose switch off. That
> > >gives a lot of information about the launching process but the
> > >problem seems to occur after the processes are launched but before
> > >MPI_Init completes.
> > >
> > >By simply running with the debug build the library should give more
> > >additional output instead of simply saying there was an error in
> > >MPI_Init.
> > >
> > >On Fri, Jun 15, 2012 at 10:44:45AM -0400, Hoot Thompson wrote:
> > >>Should have asked, after the rebuild, how do I in fact get you the
> > >>additional info? Is the verbose switch enough?
> > >>
> > >>Hoot
> > >>
> > >>-----Original Message-----
> > >>From: Jonathan Perkins [mailto:perkinjo at cse.ohio-state.edu]
> > >>Sent: Friday, June 15, 2012 10:29 AM
> > >>To: Hoot Thompson
> > >>Cc: mvapich-discuss at cse.ohio-state.edu
> > >>Subject: Re: [mvapich-discuss] mpi init error
> > >>
> > >>Hello Hoot,
> > >>It seems that you're getting an MPI_Init error. Can you rebuild and
> > >>add '--disable-fast --enable-g=dbg' to your configuration options so
> > >>that we can get more information about this error?
> > >>
> > >>On Fri, Jun 15, 2012 at 08:16:29AM -0400, Hoot Thompson wrote:
> > >>>Hello,
> > >>>
> > >>>I'm working on a fresh intstall (CentOS 6.2) on two new Sandy
> > >>>Bridge servers with Mellanox FDR cards. Everything compiles just
> > >>>fine and seems to come up but when I try simple benchmark tests, I
> > >>>get MPI init errors. It works fine if I stay on the initiating
> > >>>server or the remote server but not between the two. Results
> follow.....
> > >>>
> > >>>***************** Running between two servers
> > >>>
> > >>>[hoot at sandyhp1 osu_benchmarks]$ mpiexec -verbose -n 2 -hosts
> > >>>10.0.0.1,10.0.0.2
> > >>>/home/hoot/mvapich2-1.8-r5435/osu_benchmarks/osu_bw
> > >>>host: 10.0.0.1
> > >>>host: 10.0.0.2
> > >>>
> > >>>===================================================================
> > >>>===
> > >>>============================
> > >>>mpiexec options:
> > >>>----------------
> > >>> Base path: /usr/local/bin/
> > >>> Launcher: (null)
> > >>> Debug level: 1
> > >>> Enable X: -1
> > >>>
> > >>> Global environment:
> > >>> -------------------
> > >>> HOSTNAME=sandyhp1
> > >>> SELINUX_ROLE_REQUESTED=
> > >>> TERM=xterm
> > >>> SHELL=/bin/bash
> > >>> HISTSIZE=1000
> > >>> SSH_CLIENT=169.154.148.10 42095 22
> > >>> SELINUX_USE_CURRENT_RANGE=
> > >>> QTDIR=/usr/lib64/qt-3.3
> > >>> QTINC=/usr/lib64/qt-3.3/include
> > >>> SSH_TTY=/dev/pts/0
> > >>> USER=hoot
> > >>>
> > >>LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd
> > >>=40;33;0
> > >>1:cd=40;33;01:or=40;31;01:mi=01;05;37;41:su=37;41:sg=30;43:ca=30;41:
> > >>tw=30;42
> >
> >>:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arj=01;31:*.taz=01;3
> 1:
> > >>*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.zip=01;31:*.z=01;
> > >>31:*.Z=0
> > >>1;31:*.dz=01;31:*.gz=01;31:*.lz=01;31:*.xz=01;31:*.bz2=01;31:*.tbz=0
> > >>1;31:*.t
> > >>bz2=01;31:*.bz=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:
> > >>*.rar=01
> > >>;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.jpg
> > >>=01;35:*
> > >>.jpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01
> > >>;35:*.tg
> > >>a=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35
> > >>:*.svg=0
> > >>1;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.
> > >>mpeg=01;
> >
> >>35:*.m2v=01;35:*.mkv=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;3
> 5:
> > >>*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;3
> > >>5:*.rmvb
> > >>=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.
> > >>dl=01;35
> > >>:!
> > >>*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.axv=01
> > >>;35:*.an
> > >>x=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=01;36:*.au=01;36:*.flac=01;36:
> > >>*.mid=01
> >
> >>;36:*.midi=01;36:*.mka=01;36:*.mp3=01;36:*.mpc=01;36:*.ogg=01;36:*.ra=01;3
> 6:
> > >>*.wav=01;36:*.axa=01;36:*.oga=01;36:*.spx=01;36:*.xspf=01;36:
> > >>> MAIL=/var/spool/mail/hoot
> > >>>
> > >>PATH=/usr/lib64/qt-3.3/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/s
> > >>bin:/usr
> > >>/sbin:/sbin:/home/hoot/bin
> > >>> PWD=/home/hoot/mvapich2-1.8-r5435/osu_benchmarks
> > >>> LANG=en_US.UTF-8
> > >>> SELINUX_LEVEL_REQUESTED=
> > >>> SSH_ASKPASS=/usr/libexec/openssh/gnome-ssh-askpass
> > >>> HISTCONTROL=ignoredups
> > >>> SHLVL=1
> > >>> HOME=/home/hoot
> > >>> LOGNAME=hoot
> > >>> QTLIB=/usr/lib64/qt-3.3/lib
> > >>> CVS_RSH=ssh
> > >>> SSH_CONNECTION=169.154.148.10 42095 169.154.148.45 22
> > >>> LESSOPEN=|/usr/bin/lesspipe.sh %s
> > >>> G_BROKEN_FILENAMES=1
> > >>> _=/usr/local/bin/mpiexec
> > >>> OLDPWD=/home/hoot/mvapich2-1.8-r5435
> > >>>
> > >>> Hydra internal environment:
> > >>> ---------------------------
> > >>> GFORTRAN_UNBUFFERED_PRECONNECTED=y
> > >>>
> > >>>
> > >>> Proxy information:
> > >>> *********************
> > >>> [1] proxy: 10.0.0.1 (1 cores)
> > >>> Exec list:
> > >>>/home/hoot/mvapich2-1.8-r5435/osu_benchmarks/osu_bw
> > >>>(1 processes);
> > >>>
> > >>> [2] proxy: 10.0.0.2 (1 cores)
> > >>> Exec list:
> > >>>/home/hoot/mvapich2-1.8-r5435/osu_benchmarks/osu_bw
> > >>>(1 processes);
> > >>>
> > >>>
> > >>>===================================================================
> > >>>===
> > >>>============================
> > >>>
> > >>>[mpiexec at sandyhp1] Timeout set to -1 (-1 means infinite)
> > >>>[mpiexec at sandyhp1] Got a control port string of 10.0.0.1:50740
> > >>>
> > >>>Proxy launch args: /usr/local/bin/hydra_pmi_proxy --control-port
> > >>>10.0.0.1:50740 --debug --rmk user --launcher ssh --demux poll
> > >>>--pgid
> > >>>0 --retries 10 --proxy-id
> > >>>
> > >>>[mpiexec at sandyhp1] PMI FD: (null); PMI PORT: (null); PMI ID/RANK:
> > >>>-1 Arguments being passed to proxy 0:
> > >>>--version 1.4.1p1 --iface-ip-env-name MPICH_INTERFACE_HOSTNAME
> > >>>--hostname 10.0.0.1 --global-core-map 0,1,1 --filler-process-map
> > >>>0,1,1 --global-process-count 2 --auto-cleanup 1 --pmi-rank -1
> > >>>--pmi-kvsname kvs_7644_0 --pmi-process-mapping (vector,(0,2,1))
> > >>>--ckpoint-num -1 --global-inherited-env 29 'HOSTNAME=sandyhp1'
> > >>>'SELINUX_ROLE_REQUESTED=' 'TERM=xterm' 'SHELL=/bin/bash'
> > >>>'HISTSIZE=1000' 'SSH_CLIENT=169.154.148.10 42095 22'
> > >>>'SELINUX_USE_CURRENT_RANGE=' 'QTDIR=/usr/lib64/qt-3.3'
> > >>>'QTINC=/usr/lib64/qt-3.3/include' 'SSH_TTY=/dev/pts/0' 'USER=hoot'
> > >>'LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:b
> > >>d=40;33;
> > >>01:cd=40;33;01:or=40;31;01:mi=01;05;37;41:su=37;41:sg=30;43:ca=30;41
> > >>:tw=30;4
> > >>2:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arj=01;31:*.t
> > >>az=01;31
> > >>:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.zip=01;31:*.z=01
> > >>;31:*.Z=
> >
> >>01;31:*.dz=01;31:*.gz=01;31:*.lz=01;31:*.xz=01;31:*.bz2=01;31:*.tbz=01;31:
> *.
> > >>tbz2=01;31:*.bz=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31
> > >>:*.rar=0
> >
> >>1;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.jpg=01;3
> 5:
> > >>*.jpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=0
> > >>1;35:*.t
> > >>ga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;3
> > >>5:*.svg=
> > >>01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*
> > >>.mpeg=01
> > >>;35:*.m2v=01;35:*.mkv=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp
> > >>4v=01;35
> > >>:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;
> > >>35:*.rmv
> > >>b=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*
> > >>.dl=01;3
> > >>5!
> > >>:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.axv=0
> > >>1;35:*.a
> > >>nx=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=01;36:*.au=01;36:*.flac=01;36
> > >>:*.mid=0
> > >>1;36:*.midi=01;36:*.mka=01;36:*.mp3=01;36:*.mpc=01;36:*.ogg=01;36:*.
> > >>ra=01;36
> > >>:*.wav=01;36:*.axa=01;36:*.oga=01;36:*.spx=01;36:*.xspf=01;36:'
> > >>>'MAIL=/var/spool/mail/hoot'
> > >>'PATH=/usr/lib64/qt-3.3/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/
> > >>sbin:/us
> > >>r/sbin:/sbin:/home/hoot/bin'
> > >>>'PWD=/home/hoot/mvapich2-1.8-r5435/osu_benchmarks'
> > >>>'LANG=en_US.UTF-8' 'SELINUX_LEVEL_REQUESTED='
> > >>>'SSH_ASKPASS=/usr/libexec/openssh/gnome-ssh-askpass'
> > >>>'HISTCONTROL=ignoredups' 'SHLVL=1' 'HOME=/home/hoot' 'LOGNAME=hoot'
> > >>>'QTLIB=/usr/lib64/qt-3.3/lib' 'CVS_RSH=ssh'
> > >>>'SSH_CONNECTION=169.154.148.10 42095 169.154.148.45 22'
> > >>>'LESSOPEN=|/usr/bin/lesspipe.sh %s' 'G_BROKEN_FILENAMES=1'
> > >>>'_=/usr/local/bin/mpiexec' 'OLDPWD=/home/hoot/mvapich2-1.8-r5435'
> > >>>--global-user-env 0 --global-system-env 1
> > >>>'GFORTRAN_UNBUFFERED_PRECONNECTED=y' --proxy-core-count 1 --exec
> > >>>--exec-appnum 0 --exec-proc-count 1 --exec-local-env 0 --exec-wdir
> > >>>/home/hoot/mvapich2-1.8-r5435/osu_benchmarks --exec-args 1
> > >>>/home/hoot/mvapich2-1.8-r5435/osu_benchmarks/osu_bw
> > >>>
> > >>>[mpiexec at sandyhp1] PMI FD: (null); PMI PORT: (null); PMI ID/RANK:
> > >>>-1 Arguments being passed to proxy 1:
> > >>>--version 1.4.1p1 --iface-ip-env-name MPICH_INTERFACE_HOSTNAME
> > >>>--hostname 10.0.0.2 --global-core-map 1,1,0 --filler-process-map
> > >>>1,1,0 --global-process-count 2 --auto-cleanup 1 --pmi-rank -1
> > >>>--pmi-kvsname kvs_7644_0 --pmi-process-mapping (vector,(0,2,1))
> > >>>--ckpoint-num -1 --global-inherited-env 29 'HOSTNAME=sandyhp1'
> > >>>'SELINUX_ROLE_REQUESTED=' 'TERM=xterm' 'SHELL=/bin/bash'
> > >>>'HISTSIZE=1000' 'SSH_CLIENT=169.154.148.10 42095 22'
> > >>>'SELINUX_USE_CURRENT_RANGE=' 'QTDIR=/usr/lib64/qt-3.3'
> > >>>'QTINC=/usr/lib64/qt-3.3/include' 'SSH_TTY=/dev/pts/0' 'USER=hoot'
> > >>'LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:b
> > >>d=40;33;
> > >>01:cd=40;33;01:or=40;31;01:mi=01;05;37;41:su=37;41:sg=30;43:ca=30;41
> > >>:tw=30;4
> > >>2:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arj=01;31:*.t
> > >>az=01;31
> > >>:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.zip=01;31:*.z=01
> > >>;31:*.Z=
> >
> >>01;31:*.dz=01;31:*.gz=01;31:*.lz=01;31:*.xz=01;31:*.bz2=01;31:*.tbz=01;31:
> *.
> > >>tbz2=01;31:*.bz=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31
> > >>:*.rar=0
> >
> >>1;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.jpg=01;3
> 5:
> > >>*.jpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=0
> > >>1;35:*.t
> > >>ga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;3
> > >>5:*.svg=
> > >>01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*
> > >>.mpeg=01
> > >>;35:*.m2v=01;35:*.mkv=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp
> > >>4v=01;35
> > >>:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;
> > >>35:*.rmv
> > >>b=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*
> > >>.dl=01;3
> > >>5!
> > >>:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.axv=0
> > >>1;35:*.a
> > >>nx=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=01;36:*.au=01;36:*.flac=01;36
> > >>:*.mid=0
> > >>1;36:*.midi=01;36:*.mka=01;36:*.mp3=01;36:*.mpc=01;36:*.ogg=01;36:*.
> > >>ra=01;36
> > >>:*.wav=01;36:*.axa=01;36:*.oga=01;36:*.spx=01;36:*.xspf=01;36:'
> > >>>'MAIL=/var/spool/mail/hoot'
> > >>'PATH=/usr/lib64/qt-3.3/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/
> > >>sbin:/us
> > >>r/sbin:/sbin:/home/hoot/bin'
> > >>>'PWD=/home/hoot/mvapich2-1.8-r5435/osu_benchmarks'
> > >>>'LANG=en_US.UTF-8' 'SELINUX_LEVEL_REQUESTED='
> > >>>'SSH_ASKPASS=/usr/libexec/openssh/gnome-ssh-askpass'
> > >>>'HISTCONTROL=ignoredups' 'SHLVL=1' 'HOME=/home/hoot' 'LOGNAME=hoot'
> > >>>'QTLIB=/usr/lib64/qt-3.3/lib' 'CVS_RSH=ssh'
> > >>>'SSH_CONNECTION=169.154.148.10 42095 169.154.148.45 22'
> > >>>'LESSOPEN=|/usr/bin/lesspipe.sh %s' 'G_BROKEN_FILENAMES=1'
> > >>>'_=/usr/local/bin/mpiexec' 'OLDPWD=/home/hoot/mvapich2-1.8-r5435'
> > >>>--global-user-env 0 --global-system-env 1
> > >>>'GFORTRAN_UNBUFFERED_PRECONNECTED=y' --proxy-core-count 1 --exec
> > >>>--exec-appnum 0 --exec-proc-count 1 --exec-local-env 0 --exec-wdir
> > >>>/home/hoot/mvapich2-1.8-r5435/osu_benchmarks --exec-args 1
> > >>>/home/hoot/mvapich2-1.8-r5435/osu_benchmarks/osu_bw
> > >>>
> > >>>[mpiexec at sandyhp1] Launch arguments: /usr/local/bin/hydra_pmi_proxy
> > >>>--control-port 10.0.0.1:50740 --debug --rmk user --launcher ssh
> > >>>--demux poll --pgid 0 --retries 10 --proxy-id 0 [mpiexec at sandyhp1]
> > >>>Launch arguments: /usr/bin/ssh -x 10.0.0.2
> > >>>"/usr/local/bin/hydra_pmi_proxy" --control-port 10.0.0.1:50740
> > >>>--debug --rmk user --launcher ssh --demux poll --pgid 0 --retries
> > >>>10 --proxy-id 1 [proxy:0:0 at sandyhp1] got pmi command (from 0): init
> > >>>pmi_version=1 pmi_subversion=1
> > >>>[proxy:0:0 at sandyhp1] PMI response: cmd=response_to_init
> > >>>pmi_version=1 pmi_subversion=1 rc=0 [proxy:0:0 at sandyhp1] got pmi
> > >>>command (from 0): get_maxes
> > >>>
> > >>>[proxy:0:0 at sandyhp1] PMI response: cmd=maxes kvsname_max=256
> > >>>keylen_max=64 vallen_max=1024
> > >>>[proxy:0:0 at sandyhp1] got pmi command (from 0): get_appnum
> > >>>
> > >>>[proxy:0:0 at sandyhp1] PMI response: cmd=appnum appnum=0
> > >>>[proxy:0:0 at sandyhp1] got pmi command (from 0): get_my_kvsname
> > >>>
> > >>>[proxy:0:0 at sandyhp1] PMI response: cmd=my_kvsname
> > >>>kvsname=kvs_7644_0 [proxy:0:0 at sandyhp1] got pmi command (from 0):
> > >>>get_my_kvsname
> > >>>
> > >>>[proxy:0:0 at sandyhp1] PMI response: cmd=my_kvsname
> > >>>kvsname=kvs_7644_0 [proxy:0:0 at sandyhp1] got pmi command (from 0):
> > >>>get
> > >>>kvsname=kvs_7644_0 key=PMI_process_mapping [proxy:0:0 at sandyhp1] PMI
> > >>>response: cmd=get_result rc=0 msg=success
> > >>>value=(vector,(0,2,1))
> > >>>[cli_0]: aborting job:
> > >>>Fatal error in MPI_Init:
> > >>>Other MPI error
> > >>>
> > >>>
> > >>>
> > >>====================================================================
> > >>========
> > >>=========
> > >>>= BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
> > >>>= EXIT CODE: 256
> > >>>= CLEANING UP REMAINING PROCESSES
> > >>>= YOU CAN IGNORE THE BELOW CLEANUP MESSAGES
> > >>>===================================================================
> > >>>=== =============== [proxy:0:1 at sandyhp2] got pmi command (from 4):
> > >>>init
> > >>>pmi_version=1 pmi_subversion=1
> > >>>[proxy:0:1 at sandyhp2] PMI response: cmd=response_to_init
> > >>>pmi_version=1 pmi_subversion=1 rc=0 [proxy:0:1 at sandyhp2] got pmi
> > >>>command (from 4): get_maxes
> > >>>
> > >>>[proxy:0:1 at sandyhp2] PMI response: cmd=maxes kvsname_max=256
> > >>>keylen_max=64 vallen_max=1024
> > >>>[proxy:0:1 at sandyhp2] got pmi command (from 4): get_appnum
> > >>>
> > >>>[proxy:0:1 at sandyhp2] PMI response: cmd=appnum appnum=0
> > >>>[proxy:0:1 at sandyhp2] got pmi command (from 4): get_my_kvsname
> > >>>
> > >>>[proxy:0:1 at sandyhp2] PMI response: cmd=my_kvsname
> > >>>kvsname=kvs_7644_0 [proxy:0:1 at sandyhp2] got pmi command (from 4):
> > >>>get_my_kvsname
> > >>>
> > >>>[proxy:0:1 at sandyhp2] PMI response: cmd=my_kvsname
> > >>>kvsname=kvs_7644_0 [proxy:0:1 at sandyhp2] got pmi command (from 4):
> > >>>get
> > >>>kvsname=kvs_7644_0 key=PMI_process_mapping [proxy:0:1 at sandyhp2] PMI
> > >>>response: cmd=get_result rc=0 msg=success
> > >>>value=(vector,(0,2,1))
> > >>>[cli_1]: aborting job:
> > >>>Fatal error in MPI_Init:
> > >>>Other MPI error
> > >>>
> > >>>
> > >>>
> > >>====================================================================
> > >>========
> > >>=========
> > >>>= BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
> > >>>= EXIT CODE: 256
> > >>>= CLEANING UP REMAINING PROCESSES
> > >>>= YOU CAN IGNORE THE BELOW CLEANUP MESSAGES
> > >>>===================================================================
> > >>>===
> > >>>===============
> > >>>
> > >>>
> > >>>***************** Running on the initiating server
> > >>>[hoot at sandyhp1 osu_benchmarks]$ mpiexec -verbose -hosts
> > >>>10.0.0.1,10.0.0.1
> > >>>/home/hoot/mvapich2-1.8-r5435/osu_benchmarks/osu_bw
> > >>>host: 10.0.0.1
> > >>>
> > >>>===================================================================
> > >>>===
> > >>>============================
> > >>>mpiexec options:
> > >>>----------------
> > >>> Base path: /usr/local/bin/
> > >>> Launcher: (null)
> > >>> Debug level: 1
> > >>> Enable X: -1
> > >>>
> > >>> Global environment:
> > >>> -------------------
> > >>> HOSTNAME=sandyhp1
> > >>> SELINUX_ROLE_REQUESTED=
> > >>> TERM=xterm
> > >>> SHELL=/bin/bash
> > >>> HISTSIZE=1000
> > >>> SSH_CLIENT=169.154.148.10 42095 22
> > >>> SELINUX_USE_CURRENT_RANGE=
> > >>> QTDIR=/usr/lib64/qt-3.3
> > >>> QTINC=/usr/lib64/qt-3.3/include
> > >>> SSH_TTY=/dev/pts/0
> > >>> USER=hoot
> > >>>
> > >>LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd
> > >>=40;33;0
> > >>1:cd=40;33;01:or=40;31;01:mi=01;05;37;41:su=37;41:sg=30;43:ca=30;41:
> > >>tw=30;42
> >
> >>:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arj=01;31:*.taz=01;3
> 1:
> > >>*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.zip=01;31:*.z=01;
> > >>31:*.Z=0
> > >>1;31:*.dz=01;31:*.gz=01;31:*.lz=01;31:*.xz=01;31:*.bz2=01;31:*.tbz=0
> > >>1;31:*.t
> > >>bz2=01;31:*.bz=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:
> > >>*.rar=01
> > >>;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.jpg
> > >>=01;35:*
> > >>.jpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01
> > >>;35:*.tg
> > >>a=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35
> > >>:*.svg=0
> > >>1;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.
> > >>mpeg=01;
> >
> >>35:*.m2v=01;35:*.mkv=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;3
> 5:
> > >>*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;3
> > >>5:*.rmvb
> > >>=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.
> > >>dl=01;35
> > >>:!
> > >>*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.axv=01
> > >>;35:*.an
> > >>x=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=01;36:*.au=01;36:*.flac=01;36:
> > >>*.mid=01
> >
> >>;36:*.midi=01;36:*.mka=01;36:*.mp3=01;36:*.mpc=01;36:*.ogg=01;36:*.ra=01;3
> 6:
> > >>*.wav=01;36:*.axa=01;36:*.oga=01;36:*.spx=01;36:*.xspf=01;36:
> > >>> MAIL=/var/spool/mail/hoot
> > >>>
> > >>PATH=/usr/lib64/qt-3.3/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/s
> > >>bin:/usr
> > >>/sbin:/sbin:/home/hoot/bin
> > >>> PWD=/home/hoot/mvapich2-1.8-r5435/osu_benchmarks
> > >>> LANG=en_US.UTF-8
> > >>> SELINUX_LEVEL_REQUESTED=
> > >>> SSH_ASKPASS=/usr/libexec/openssh/gnome-ssh-askpass
> > >>> HISTCONTROL=ignoredups
> > >>> SHLVL=1
> > >>> HOME=/home/hoot
> > >>> LOGNAME=hoot
> > >>> QTLIB=/usr/lib64/qt-3.3/lib
> > >>> CVS_RSH=ssh
> > >>> SSH_CONNECTION=169.154.148.10 42095 169.154.148.45 22
> > >>> LESSOPEN=|/usr/bin/lesspipe.sh %s
> > >>> G_BROKEN_FILENAMES=1
> > >>> _=/usr/local/bin/mpiexec
> > >>> OLDPWD=/home/hoot/mvapich2-1.8-r5435
> > >>>
> > >>> Hydra internal environment:
> > >>> ---------------------------
> > >>> GFORTRAN_UNBUFFERED_PRECONNECTED=y
> > >>>
> > >>>
> > >>> Proxy information:
> > >>> *********************
> > >>> [1] proxy: 10.0.0.1 (2 cores)
> > >>> Exec list:
> > >>>/home/hoot/mvapich2-1.8-r5435/osu_benchmarks/osu_bw
> > >>>(2 processes);
> > >>>
> > >>>
> > >>>===================================================================
> > >>>===
> > >>>============================
> > >>>
> > >>>[mpiexec at sandyhp1] Timeout set to -1 (-1 means infinite)
> > >>>[mpiexec at sandyhp1] Got a control port string of 10.0.0.1:56066
> > >>>
> > >>>Proxy launch args: /usr/local/bin/hydra_pmi_proxy --control-port
> > >>>10.0.0.1:56066 --debug --rmk user --launcher ssh --demux poll
> > >>>--pgid
> > >>>0 --retries 10 --proxy-id
> > >>>
> > >>>[mpiexec at sandyhp1] PMI FD: (null); PMI PORT: (null); PMI ID/RANK:
> > >>>-1 Arguments being passed to proxy 0:
> > >>>--version 1.4.1p1 --iface-ip-env-name MPICH_INTERFACE_HOSTNAME
> > >>>--hostname 10.0.0.1 --global-core-map 0,2,0 --filler-process-map
> > >>>0,2,0 --global-process-count 2 --auto-cleanup 1 --pmi-rank -1
> > >>>--pmi-kvsname kvs_7484_0 --pmi-process-mapping (vector,(0,1,2))
> > >>>--ckpoint-num -1 --global-inherited-env 29 'HOSTNAME=sandyhp1'
> > >>>'SELINUX_ROLE_REQUESTED=' 'TERM=xterm' 'SHELL=/bin/bash'
> > >>>'HISTSIZE=1000' 'SSH_CLIENT=169.154.148.10 42095 22'
> > >>>'SELINUX_USE_CURRENT_RANGE=' 'QTDIR=/usr/lib64/qt-3.3'
> > >>>'QTINC=/usr/lib64/qt-3.3/include' 'SSH_TTY=/dev/pts/0' 'USER=hoot'
> > >>'LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:b
> > >>d=40;33;
> > >>01:cd=40;33;01:or=40;31;01:mi=01;05;37;41:su=37;41:sg=30;43:ca=30;41
> > >>:tw=30;4
> > >>2:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arj=01;31:*.t
> > >>az=01;31
> > >>:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.zip=01;31:*.z=01
> > >>;31:*.Z=
> >
> >>01;31:*.dz=01;31:*.gz=01;31:*.lz=01;31:*.xz=01;31:*.bz2=01;31:*.tbz=01;31:
> *.
> > >>tbz2=01;31:*.bz=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31
> > >>:*.rar=0
> >
> >>1;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.jpg=01;3
> 5:
> > >>*.jpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=0
> > >>1;35:*.t
> > >>ga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;3
> > >>5:*.svg=
> > >>01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*
> > >>.mpeg=01
> > >>;35:*.m2v=01;35:*.mkv=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp
> > >>4v=01;35
> > >>:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;
> > >>35:*.rmv
> > >>b=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*
> > >>.dl=01;3
> > >>5!
> > >>:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.axv=0
> > >>1;35:*.a
> > >>nx=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=01;36:*.au=01;36:*.flac=01;36
> > >>:*.mid=0
> > >>1;36:*.midi=01;36:*.mka=01;36:*.mp3=01;36:*.mpc=01;36:*.ogg=01;36:*.
> > >>ra=01;36
> > >>:*.wav=01;36:*.axa=01;36:*.oga=01;36:*.spx=01;36:*.xspf=01;36:'
> > >>>'MAIL=/var/spool/mail/hoot'
> > >>'PATH=/usr/lib64/qt-3.3/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/
> > >>sbin:/us
> > >>r/sbin:/sbin:/home/hoot/bin'
> > >>>'PWD=/home/hoot/mvapich2-1.8-r5435/osu_benchmarks'
> > >>>'LANG=en_US.UTF-8' 'SELINUX_LEVEL_REQUESTED='
> > >>>'SSH_ASKPASS=/usr/libexec/openssh/gnome-ssh-askpass'
> > >>>'HISTCONTROL=ignoredups' 'SHLVL=1' 'HOME=/home/hoot' 'LOGNAME=hoot'
> > >>>'QTLIB=/usr/lib64/qt-3.3/lib' 'CVS_RSH=ssh'
> > >>>'SSH_CONNECTION=169.154.148.10 42095 169.154.148.45 22'
> > >>>'LESSOPEN=|/usr/bin/lesspipe.sh %s' 'G_BROKEN_FILENAMES=1'
> > >>>'_=/usr/local/bin/mpiexec' 'OLDPWD=/home/hoot/mvapich2-1.8-r5435'
> > >>>--global-user-env 0 --global-system-env 1
> > >>>'GFORTRAN_UNBUFFERED_PRECONNECTED=y' --proxy-core-count 2 --exec
> > >>>--exec-appnum 0 --exec-proc-count 2 --exec-local-env 0 --exec-wdir
> > >>>/home/hoot/mvapich2-1.8-r5435/osu_benchmarks --exec-args 1
> > >>>/home/hoot/mvapich2-1.8-r5435/osu_benchmarks/osu_bw
> > >>>
> > >>>[mpiexec at sandyhp1] Launch arguments: /usr/local/bin/hydra_pmi_proxy
> > >>>--control-port 10.0.0.1:56066 --debug --rmk user --launcher ssh
> > >>>--demux poll --pgid 0 --retries 10 --proxy-id 0
> > >>>[proxy:0:0 at sandyhp1] got pmi command (from 0): init
> > >>>pmi_version=1 pmi_subversion=1
> > >>>[proxy:0:0 at sandyhp1] PMI response: cmd=response_to_init
> > >>>pmi_version=1 pmi_subversion=1 rc=0 [proxy:0:0 at sandyhp1] got pmi
> > >>>command (from 6): init
> > >>>pmi_version=1 pmi_subversion=1
> > >>>[proxy:0:0 at sandyhp1] PMI response: cmd=response_to_init
> > >>>pmi_version=1 pmi_subversion=1 rc=0 [proxy:0:0 at sandyhp1] got pmi
> > >>>command (from 0): get_maxes
> > >>>
> > >>>[proxy:0:0 at sandyhp1] PMI response: cmd=maxes kvsname_max=256
> > >>>keylen_max=64 vallen_max=1024
> > >>>[proxy:0:0 at sandyhp1] got pmi command (from 6): get_maxes
> > >>>
> > >>>[proxy:0:0 at sandyhp1] PMI response: cmd=maxes kvsname_max=256
> > >>>keylen_max=64 vallen_max=1024
> > >>>[proxy:0:0 at sandyhp1] got pmi command (from 0): get_appnum
> > >>>
> > >>>[proxy:0:0 at sandyhp1] PMI response: cmd=appnum appnum=0
> > >>>[proxy:0:0 at sandyhp1] got pmi command (from 0): get_my_kvsname
> > >>>
> > >>>[proxy:0:0 at sandyhp1] PMI response: cmd=my_kvsname
> > >>>kvsname=kvs_7484_0 [proxy:0:0 at sandyhp1] got pmi command (from 6):
> > >>>get_appnum
> > >>>
> > >>>[proxy:0:0 at sandyhp1] PMI response: cmd=appnum appnum=0
> > >>>[proxy:0:0 at sandyhp1] got pmi command (from 0): get_my_kvsname
> > >>>
> > >>>[proxy:0:0 at sandyhp1] PMI response: cmd=my_kvsname
> > >>>kvsname=kvs_7484_0 [proxy:0:0 at sandyhp1] got pmi command (from 6):
> > >>>get_my_kvsname
> > >>>
> > >>>[proxy:0:0 at sandyhp1] PMI response: cmd=my_kvsname
> > >>>kvsname=kvs_7484_0 [proxy:0:0 at sandyhp1] got pmi command (from 0):
> > >>>get
> > >>>kvsname=kvs_7484_0 key=PMI_process_mapping [proxy:0:0 at sandyhp1] PMI
> > >>>response: cmd=get_result rc=0 msg=success
> > >>>value=(vector,(0,1,2))
> > >>>[proxy:0:0 at sandyhp1] got pmi command (from 6): get_my_kvsname
> > >>>
> > >>>[proxy:0:0 at sandyhp1] PMI response: cmd=my_kvsname
> > >>>kvsname=kvs_7484_0 [proxy:0:0 at sandyhp1] got pmi command (from 6):
> > >>>get
> > >>>kvsname=kvs_7484_0 key=PMI_process_mapping [proxy:0:0 at sandyhp1] PMI
> > >>>response: cmd=get_result rc=0 msg=success
> > >>>value=(vector,(0,1,2))
> > >>># OSU MPI Bandwidth Test v3.6
> > >>># Size Bandwidth (MB/s)
> > >>>1 2.76
> > >>>2 5.54
> > >>>4 11.09
> > >>>8 22.27
> > >>>16 44.48
> > >>>32 87.19
> > >>>64 171.52
> > >>>128 331.24
> > >>>256 612.25
> > >>>512 1014.16
> > >>>1024 1585.68
> > >>>2048 2590.26
> > >>>4096 3744.42
> > >>>8192 4721.57
> > >>>16384 4940.24
> > >>>32768 4258.99
> > >>>65536 4450.17
> > >>>131072 4461.91
> > >>>262144 4126.22
> > >>>524288 4103.56
> > >>>1048576 5106.87
> > >>>2097152 5126.66
> > >>>4194304 5161.63
> > >>>[proxy:0:0 at sandyhp1] got pmi command (from 6): finalize
> > >>>
> > >>>[proxy:0:0 at sandyhp1] PMI response: cmd=finalize_ack
> > >>>[proxy:0:0 at sandyhp1] got pmi command (from 0): finalize
> > >>>
> > >>>[proxy:0:0 at sandyhp1] PMI response: cmd=finalize_ack
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>**************** Running on the remote server
> > >>>
> > >>>[hoot at sandyhp1 osu_benchmarks]$ mpiexec -verbose -hosts
> > >>>10.0.0.2,10.0.0.2
> > >>>/home/hoot/mvapich2-1.8-r5435/osu_benchmarks/osu_bw
> > >>>host: 10.0.0.2
> > >>>
> > >>>===================================================================
> > >>>===
> > >>>============================
> > >>>mpiexec options:
> > >>>----------------
> > >>> Base path: /usr/local/bin/
> > >>> Launcher: (null)
> > >>> Debug level: 1
> > >>> Enable X: -1
> > >>>
> > >>> Global environment:
> > >>> -------------------
> > >>> HOSTNAME=sandyhp1
> > >>> SELINUX_ROLE_REQUESTED=
> > >>> TERM=xterm
> > >>> SHELL=/bin/bash
> > >>> HISTSIZE=1000
> > >>> SSH_CLIENT=169.154.148.10 42095 22
> > >>> SELINUX_USE_CURRENT_RANGE=
> > >>> QTDIR=/usr/lib64/qt-3.3
> > >>> QTINC=/usr/lib64/qt-3.3/include
> > >>> SSH_TTY=/dev/pts/0
> > >>> USER=hoot
> > >>>
> > >>LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd
> > >>=40;33;0
> > >>1:cd=40;33;01:or=40;31;01:mi=01;05;37;41:su=37;41:sg=30;43:ca=30;41:
> > >>tw=30;42
> >
> >>:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arj=01;31:*.taz=01;3
> 1:
> > >>*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.zip=01;31:*.z=01;
> > >>31:*.Z=0
> > >>1;31:*.dz=01;31:*.gz=01;31:*.lz=01;31:*.xz=01;31:*.bz2=01;31:*.tbz=0
> > >>1;31:*.t
> > >>bz2=01;31:*.bz=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:
> > >>*.rar=01
> > >>;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.jpg
> > >>=01;35:*
> > >>.jpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01
> > >>;35:*.tg
> > >>a=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35
> > >>:*.svg=0
> > >>1;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.
> > >>mpeg=01;
> >
> >>35:*.m2v=01;35:*.mkv=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;3
> 5:
> > >>*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;3
> > >>5:*.rmvb
> > >>=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.
> > >>dl=01;35
> > >>:!
> > >>*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.axv=01
> > >>;35:*.an
> > >>x=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=01;36:*.au=01;36:*.flac=01;36:
> > >>*.mid=01
> >
> >>;36:*.midi=01;36:*.mka=01;36:*.mp3=01;36:*.mpc=01;36:*.ogg=01;36:*.ra=01;3
> 6:
> > >>*.wav=01;36:*.axa=01;36:*.oga=01;36:*.spx=01;36:*.xspf=01;36:
> > >>> MAIL=/var/spool/mail/hoot
> > >>>
> > >>PATH=/usr/lib64/qt-3.3/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/s
> > >>bin:/usr
> > >>/sbin:/sbin:/home/hoot/bin
> > >>> PWD=/home/hoot/mvapich2-1.8-r5435/osu_benchmarks
> > >>> LANG=en_US.UTF-8
> > >>> SELINUX_LEVEL_REQUESTED=
> > >>> SSH_ASKPASS=/usr/libexec/openssh/gnome-ssh-askpass
> > >>> HISTCONTROL=ignoredups
> > >>> SHLVL=1
> > >>> HOME=/home/hoot
> > >>> LOGNAME=hoot
> > >>> QTLIB=/usr/lib64/qt-3.3/lib
> > >>> CVS_RSH=ssh
> > >>> SSH_CONNECTION=169.154.148.10 42095 169.154.148.45 22
> > >>> LESSOPEN=|/usr/bin/lesspipe.sh %s
> > >>> G_BROKEN_FILENAMES=1
> > >>> _=/usr/local/bin/mpiexec
> > >>> OLDPWD=/home/hoot/mvapich2-1.8-r5435
> > >>>
> > >>> Hydra internal environment:
> > >>> ---------------------------
> > >>> GFORTRAN_UNBUFFERED_PRECONNECTED=y
> > >>>
> > >>>
> > >>> Proxy information:
> > >>> *********************
> > >>> [1] proxy: 10.0.0.2 (2 cores)
> > >>> Exec list:
> > >>>/home/hoot/mvapich2-1.8-r5435/osu_benchmarks/osu_bw
> > >>>(2 processes);
> > >>>
> > >>>
> > >>>===================================================================
> > >>>===
> > >>>============================
> > >>>
> > >>>[mpiexec at sandyhp1] Timeout set to -1 (-1 means infinite)
> > >>>[mpiexec at sandyhp1] Got a control port string of sandyhp1:39504
> > >>>
> > >>>Proxy launch args: /usr/local/bin/hydra_pmi_proxy --control-port
> > >>>sandyhp1:39504 --debug --rmk user --launcher ssh --demux poll
> > >>>--pgid
> > >>>0 --retries 10 --proxy-id
> > >>>
> > >>>[mpiexec at sandyhp1] PMI FD: (null); PMI PORT: (null); PMI ID/RANK:
> > >>>-1 Arguments being passed to proxy 0:
> > >>>--version 1.4.1p1 --iface-ip-env-name MPICH_INTERFACE_HOSTNAME
> > >>>--hostname 10.0.0.2 --global-core-map 0,2,0 --filler-process-map
> > >>>0,2,0 --global-process-count 2 --auto-cleanup 1 --pmi-rank -1
> > >>>--pmi-kvsname kvs_7496_0 --pmi-process-mapping (vector,(0,1,2))
> > >>>--ckpoint-num -1 --global-inherited-env 29 'HOSTNAME=sandyhp1'
> > >>>'SELINUX_ROLE_REQUESTED=' 'TERM=xterm' 'SHELL=/bin/bash'
> > >>>'HISTSIZE=1000' 'SSH_CLIENT=169.154.148.10 42095 22'
> > >>>'SELINUX_USE_CURRENT_RANGE=' 'QTDIR=/usr/lib64/qt-3.3'
> > >>>'QTINC=/usr/lib64/qt-3.3/include' 'SSH_TTY=/dev/pts/0' 'USER=hoot'
> > >>'LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:b
> > >>d=40;33;
> > >>01:cd=40;33;01:or=40;31;01:mi=01;05;37;41:su=37;41:sg=30;43:ca=30;41
> > >>:tw=30;4
> > >>2:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arj=01;31:*.t
> > >>az=01;31
> > >>:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.zip=01;31:*.z=01
> > >>;31:*.Z=
> >
> >>01;31:*.dz=01;31:*.gz=01;31:*.lz=01;31:*.xz=01;31:*.bz2=01;31:*.tbz=01;31:
> *.
> > >>tbz2=01;31:*.bz=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31
> > >>:*.rar=0
> >
> >>1;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.jpg=01;3
> 5:
> > >>*.jpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=0
> > >>1;35:*.t
> > >>ga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;3
> > >>5:*.svg=
> > >>01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*
> > >>.mpeg=01
> > >>;35:*.m2v=01;35:*.mkv=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp
> > >>4v=01;35
> > >>:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;
> > >>35:*.rmv
> > >>b=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*
> > >>.dl=01;3
> > >>5!
> > >>:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.axv=0
> > >>1;35:*.a
> > >>nx=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=01;36:*.au=01;36:*.flac=01;36
> > >>:*.mid=0
> > >>1;36:*.midi=01;36:*.mka=01;36:*.mp3=01;36:*.mpc=01;36:*.ogg=01;36:*.
> > >>ra=01;36
> > >>:*.wav=01;36:*.axa=01;36:*.oga=01;36:*.spx=01;36:*.xspf=01;36:'
> > >>>'MAIL=/var/spool/mail/hoot'
> > >>'PATH=/usr/lib64/qt-3.3/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/
> > >>sbin:/us
> > >>r/sbin:/sbin:/home/hoot/bin'
> > >>>'PWD=/home/hoot/mvapich2-1.8-r5435/osu_benchmarks'
> > >>>'LANG=en_US.UTF-8' 'SELINUX_LEVEL_REQUESTED='
> > >>>'SSH_ASKPASS=/usr/libexec/openssh/gnome-ssh-askpass'
> > >>>'HISTCONTROL=ignoredups' 'SHLVL=1' 'HOME=/home/hoot' 'LOGNAME=hoot'
> > >>>'QTLIB=/usr/lib64/qt-3.3/lib' 'CVS_RSH=ssh'
> > >>>'SSH_CONNECTION=169.154.148.10 42095 169.154.148.45 22'
> > >>>'LESSOPEN=|/usr/bin/lesspipe.sh %s' 'G_BROKEN_FILENAMES=1'
> > >>>'_=/usr/local/bin/mpiexec' 'OLDPWD=/home/hoot/mvapich2-1.8-r5435'
> > >>>--global-user-env 0 --global-system-env 1
> > >>>'GFORTRAN_UNBUFFERED_PRECONNECTED=y' --proxy-core-count 2 --exec
> > >>>--exec-appnum 0 --exec-proc-count 2 --exec-local-env 0 --exec-wdir
> > >>>/home/hoot/mvapich2-1.8-r5435/osu_benchmarks --exec-args 1
> > >>>/home/hoot/mvapich2-1.8-r5435/osu_benchmarks/osu_bw
> > >>>
> > >>>[mpiexec at sandyhp1] Launch arguments: /usr/bin/ssh -x 10.0.0.2
> > >>>"/usr/local/bin/hydra_pmi_proxy" --control-port sandyhp1:39504
> > >>>--debug --rmk user --launcher ssh --demux poll --pgid 0 --retries
> > >>>10 --proxy-id 0 [proxy:0:0 at sandyhp2] got pmi command (from 4): init
> > >>>pmi_version=1 pmi_subversion=1
> > >>>[proxy:0:0 at sandyhp2] PMI response: cmd=response_to_init
> > >>>pmi_version=1 pmi_subversion=1 rc=0 [proxy:0:0 at sandyhp2] got pmi
> > >>>command (from 4): get_maxes
> > >>>
> > >>>[proxy:0:0 at sandyhp2] PMI response: cmd=maxes kvsname_max=256
> > >>>keylen_max=64 vallen_max=1024
> > >>>[proxy:0:0 at sandyhp2] got pmi command (from 5): init
> > >>>pmi_version=1 pmi_subversion=1
> > >>>[proxy:0:0 at sandyhp2] PMI response: cmd=response_to_init
> > >>>pmi_version=1 pmi_subversion=1 rc=0 [proxy:0:0 at sandyhp2] got pmi
> > >>>command (from 4): get_appnum
> > >>>
> > >>>[proxy:0:0 at sandyhp2] PMI response: cmd=appnum appnum=0
> > >>>[proxy:0:0 at sandyhp2] got pmi command (from 4): get_my_kvsname
> > >>>
> > >>>[proxy:0:0 at sandyhp2] PMI response: cmd=my_kvsname
> > >>>kvsname=kvs_7496_0 [proxy:0:0 at sandyhp2] got pmi command (from 4):
> > >>>get_my_kvsname
> > >>>
> > >>>[proxy:0:0 at sandyhp2] PMI response: cmd=my_kvsname
> > >>>kvsname=kvs_7496_0 [proxy:0:0 at sandyhp2] got pmi command (from 5):
> > >>>get_maxes
> > >>>
> > >>>[proxy:0:0 at sandyhp2] PMI response: cmd=maxes kvsname_max=256
> > >>>keylen_max=64 vallen_max=1024
> > >>>[proxy:0:0 at sandyhp2] got pmi command (from 5): get_appnum
> > >>>
> > >>>[proxy:0:0 at sandyhp2] PMI response: cmd=appnum appnum=0
> > >>>[proxy:0:0 at sandyhp2] got pmi command (from 5): get_my_kvsname
> > >>>
> > >>>[proxy:0:0 at sandyhp2] PMI response: cmd=my_kvsname
> > >>>kvsname=kvs_7496_0 [proxy:0:0 at sandyhp2] got pmi command (from 5):
> > >>>get_my_kvsname
> > >>>
> > >>>[proxy:0:0 at sandyhp2] PMI response: cmd=my_kvsname
> > >>>kvsname=kvs_7496_0 [proxy:0:0 at sandyhp2] got pmi command (from 5):
> > >>>get
> > >>>kvsname=kvs_7496_0 key=PMI_process_mapping [proxy:0:0 at sandyhp2] PMI
> > >>>response: cmd=get_result rc=0 msg=success
> > >>>value=(vector,(0,1,2))
> > >>>[proxy:0:0 at sandyhp2] got pmi command (from 4): get
> > >>>kvsname=kvs_7496_0 key=PMI_process_mapping [proxy:0:0 at sandyhp2] PMI
> > >>>response: cmd=get_result rc=0 msg=success
> > >>>value=(vector,(0,1,2))
> > >>># OSU MPI Bandwidth Test v3.6
> > >>># Size Bandwidth (MB/s)
> > >>>1 2.76
> > >>>2 5.55
> > >>>4 11.07
> > >>>8 21.98
> > >>>16 44.50
> > >>>32 86.66
> > >>>64 170.38
> > >>>128 327.30
> > >>>256 607.49
> > >>>512 1016.71
> > >>>1024 1555.88
> > >>>2048 2598.09
> > >>>4096 3748.63
> > >>>8192 4724.61
> > >>>16384 4910.18
> > >>>32768 4245.63
> > >>>65536 4427.21
> > >>>131072 4467.74
> > >>>262144 4155.35
> > >>>524288 4066.51
> > >>>1048576 4351.23
> > >>>2097152 5121.67
> > >>>4194304 5669.35
> > >>>[proxy:0:0 at sandyhp2] got pmi command (from 5): finalize
> > >>>
> > >>>[proxy:0:0 at sandyhp2] PMI response: cmd=finalize_ack
> > >>>[proxy:0:0 at sandyhp2] got pmi command (from 4): finalize
> > >>>
> > >>>[proxy:0:0 at sandyhp2] PMI response: cmd=finalize_ack
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>_______________________________________________
> > >>>mvapich-discuss mailing list
> > >>>mvapich-discuss at cse.ohio-state.edu
> > >>>http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss
> > >>>
> > >>--
> > >>Jonathan Perkins
> > >>http://www.cse.ohio-state.edu/~perkinjo
> > >>
> > >>
> >
>
> --
> Jonathan Perkins
> http://www.cse.ohio-state.edu/~perkinjo
>
>
--
Jonathan Perkins
http://www.cse.ohio-state.edu/~perkinjo
More information about the mvapich-discuss
mailing list