[mvapich-discuss] MPI INIT error
Hoot Thompson
hoot at ptpnow.com
Sat Apr 6 10:00:12 EDT 2013
I've been down this path before and I believe I've taken care of my
usual oversights. Here's the background, it's a RHEL6.4 setup using the
distro IB modules (not an OFED download). I'm trying to run the micro
benchmarks and I'm getting (debug output attached) ....
=====================================================================================
= BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
= EXIT CODE: 256
= CLEANING UP REMAINING PROCESSES
= YOU CAN IGNORE THE BELOW CLEANUP MESSAGES
=====================================================================================
[proxy:0:1 at rh64-3-ib] got pmi command (from 4): init
pmi_version=1 pmi_subversion=1
[proxy:0:1 at rh64-3-ib] PMI response: cmd=response_to_init pmi_version=1
pmi_subversion=1 rc=0
[proxy:0:1 at rh64-3-ib] got pmi command (from 4): get_maxes
[proxy:0:1 at rh64-3-ib] PMI response: cmd=maxes kvsname_max=256
keylen_max=64 vallen_max=1024
[proxy:0:1 at rh64-3-ib] got pmi command (from 4): get_appnum
[proxy:0:1 at rh64-3-ib] PMI response: cmd=appnum appnum=0
[proxy:0:1 at rh64-3-ib] got pmi command (from 4): get_my_kvsname
[proxy:0:1 at rh64-3-ib] PMI response: cmd=my_kvsname kvsname=kvs_4129_0
[proxy:0:1 at rh64-3-ib] got pmi command (from 4): get_my_kvsname
[proxy:0:1 at rh64-3-ib] PMI response: cmd=my_kvsname kvsname=kvs_4129_0
[proxy:0:1 at rh64-3-ib] got pmi command (from 4): get
kvsname=kvs_4129_0 key=PMI_process_mapping
[proxy:0:1 at rh64-3-ib] PMI response: cmd=get_result rc=0 msg=success
value=(vector,(0,2,1))
[cli_1]: aborting job:
Fatal error in MPI_Init:
Other MPI error
=====================================================================================
= BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
= EXIT CODE: 256
= CLEANING UP REMAINING PROCESSES
= YOU CAN IGNORE THE BELOW CLEANUP MESSAGES
=====================================================================================
Here's the output of ulimit on both ends (configured in limits.conf)
[jhthomps at rh64-1-ib ~]$ ulimit -l
unlimited
[root at rh64-3-ib jhthomps]# ulimit -l
unlimited
Firewalls are down and I think the /etc/hosts files are right.
Suggestions?
Thanks,
Hoot
-------------- next part --------------
[jhthomps at rh64-1-ib ~]$ /usr/local/other/utilities/mvapich2/bin/mpirun -verbose -n 2 -hosts rh64-1-ib,rh64-3-ib /usr/local/other/utilities/mvapich2/libexec/osu-micro-benchmarks/osu_bw
host: rh64-1-ib
host: rh64-3-ib
==================================================================================================
mpiexec options:
----------------
Base path: /usr/local/other/utilities/mvapich2/bin/
Launcher: (null)
Debug level: 1
Enable X: -1
Global environment:
-------------------
MPICH_CLINKER=/usr/local/other/utilities/intel/composerxe/bin/icc
HOSTNAME=rh64-1-ib
SELINUX_ROLE_REQUESTED=
MPICH_CCC=/usr/local/other/utilities/intel/composerxe/bin/icpc
TERM=xterm
SHELL=/bin/bash
HISTSIZE=1000
TMPDIR=/tmp
LIBRARY_PATH=/usr/local/other/utilities/mvapich2/lib:/usr/local/other/utilities/intel/composer_xe_2013.0.079/compiler/lib/intel64:
SELINUX_USE_CURRENT_RANGE=
FPATH=/usr/local/other/utilities/mvapich2/include:/usr/local/other/utilities/intel/composerxe/mkl/include:
QTDIR=/usr/lib64/qt-3.3
MV2_NPROCS_THRESHOLD=18000
CCC=/usr/local/other/utilities/intel/composerxe/bin/icpc
QTINC=/usr/lib64/qt-3.3/include
SSH_TTY=/dev/pts/0
USER=jhthomps
LD_LIBRARY_PATH=/usr/local/other/utilities/mvapich2/lib:/usr/local/other/utilities/intel/composer_xe_2013.0.079/compiler/lib/intel64:
LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=01;05;37;41:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arj=01;31:*.taz=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.lz=01;31:*.xz=01;31:*.bz2=01;31:*.tbz=01;31:*.tbz2=01;31:*.bz=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.rar=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.jpg=01;35:*.jpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.axv=01;35:*.anx=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=01;36:*.au=01;36:*.flac=01;36:*.mid=01;36:*.midi=01;36:*.mka=01;36:*.mp3=01;36:*.mpc=01;36:*.ogg=01;36:*.ra=01;36:*.wav=01;36:*.axa=01;36:*.oga=01;36:*.spx=01;36:*.xspf=01;36:
CPATH=/usr/local/other/utilities/mvapich2/include:/usr/local/other/utilities/intel/composerxe/ipp/include:/usr/local/other/utilities/intel/composerxe/mkl/include:/usr/local/other/utilities/intel/composerxe/tbb/include/tbb:
MPICH_F77LINKER=/usr/local/other/utilities/intel/composerxe/bin/ifort
INTELDIR=/usr/local/other/utilities/intel/composerxe
MAIL=/var/spool/mail/jhthomps
PATH=/usr/local/other/utilities/mvapich2/bin:/usr/local/other/utilities/intel/composerxe/bin/:/usr/lib64/qt-3.3/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/jhthomps/bin
MPICH_F77=/usr/local/other/utilities/intel/composerxe/bin/ifort
MVAPICH_ROOT=/usr/local/other/utilities/mvapich2
UTILDIR=/usr/local/other/utilities
PWD=/home/jhthomps
MPICH_CC=/usr/local/other/utilities/intel/composerxe/bin/icc
LANG=en_US.UTF-8
F77=/usr/local/other/utilities/intel/composerxe/bin/ifort
SELINUX_LEVEL_REQUESTED=
MPICH_F90=/usr/local/other/utilities/intel/composerxe/bin/ifort
CXX=/usr/local/other/utilities/intel/composerxe/bin/icpc
HISTCONTROL=ignoredups
MPICH_F90LINKER=/usr/local/other/utilities/intel/composerxe/bin/ifort
SHLVL=1
HOME=/home/jhthomps
MPD_BIN=/bin
FC=/usr/local/other/utilities/intel/composerxe/bin/ifort
MPICH_CCLINKER=/usr/local/other/utilities/intel/composerxe/bin/icpc
LOGNAME=jhthomps
QTLIB=/usr/lib64/qt-3.3/lib
CVS_RSH=ssh
LESSOPEN=|/usr/bin/lesspipe.sh %s
MV2_FASTSSH_THRESHOLD=1
CC=/usr/local/other/utilities/intel/composerxe/bin/icc
INCLUDE=/usr/local/other/utilities/mvapich2/include:/usr/local/other/utilities/intel/composer_xe_2013.0.079/compiler/include/intel64:
G_BROKEN_FILENAMES=1
_=/usr/local/other/utilities/mvapich2/bin/mpirun
Hydra internal environment:
---------------------------
GFORTRAN_UNBUFFERED_PRECONNECTED=y
Proxy information:
*********************
[1] proxy: rh64-1-ib (1 cores)
Exec list: /usr/local/other/utilities/mvapich2/libexec/osu-micro-benchmarks/osu_bw (1 processes);
[2] proxy: rh64-3-ib (1 cores)
Exec list: /usr/local/other/utilities/mvapich2/libexec/osu-micro-benchmarks/osu_bw (1 processes);
==================================================================================================
[mpiexec at rh64-1-ib] Timeout set to -1 (-1 means infinite)
[mpiexec at rh64-1-ib] Got a control port string of rh64-1-ib:59143
Proxy launch args: /usr/local/other/utilities/mvapich2/bin/hydra_pmi_proxy --control-port rh64-1-ib:59143 --debug --rmk user --launcher ssh --demux poll --pgid 0 --retries 10 --proxy-id
[mpiexec at rh64-1-ib] PMI FD: (null); PMI PORT: (null); PMI ID/RANK: -1
Arguments being passed to proxy 0:
--version 1.4.1p1 --iface-ip-env-name MPICH_INTERFACE_HOSTNAME --hostname rh64-1-ib --global-core-map 0,1,1 --filler-process-map 0,1,1 --global-process-count 2 --auto-cleanup 1 --pmi-rank -1 --pmi-kvsname kvs_4129_0 --pmi-process-mapping (vector,(0,2,1)) --ckpoint-num -1 --global-inherited-env 52 'MPICH_CLINKER=/usr/local/other/utilities/intel/composerxe/bin/icc' 'HOSTNAME=rh64-1-ib' 'SELINUX_ROLE_REQUESTED=' 'MPICH_CCC=/usr/local/other/utilities/intel/composerxe/bin/icpc' 'TERM=xterm' 'SHELL=/bin/bash' 'HISTSIZE=1000' 'TMPDIR=/tmp' '' 'LIBRARY_PATH=/usr/local/other/utilities/mvapich2/lib:/usr/local/other/utilities/intel/composer_xe_2013.0.079/compiler/lib/intel64:' 'SELINUX_USE_CURRENT_RANGE=' 'FPATH=/usr/local/other/utilities/mvapich2/include:/usr/local/other/utilities/intel/composerxe/mkl/include:' 'QTDIR=/usr/lib64/qt-3.3' 'MV2_NPROCS_THRESHOLD=18000' 'CCC=/usr/local/other/utilities/intel/composerxe/bin/icpc' 'QTINC=/usr/lib64/qt-3.3/include' 'SSH_TTY=/dev/pts/0' 'USER=jhthomps' 'LD_LIBRARY_PATH=/usr/local/other/utilities/mvapich2/lib:/usr/local/other/utilities/intel/composer_xe_2013.0.079/compiler/lib/intel64:' 'LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=01;05;37;41:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arj=01;31:*.taz=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.lz=01;31:*.xz=01;31:*.bz2=01;31:*.tbz=01;31:*.tbz2=01;31:*.bz=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.rar=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.jpg=01;35:*.jpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.axv=01;35:*.anx=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=01;36:*.au=01;36:*.flac=01;36:*.mid=01;36:*.midi=01;36:*.mka=01;36:*.mp3=01;36:*.mpc=01;36:*.ogg=01;36:*.ra=01;36:*.wav=01;36:*.axa=01;36:*.oga=01;36:*.spx=01;36:*.xspf=01;36:' 'CPATH=/usr/local/other/utilities/mvapich2/include:/usr/local/other/utilities/intel/composerxe/ipp/include:/usr/local/other/utilities/intel/composerxe/mkl/include:/usr/local/other/utilities/intel/composerxe/tbb/include/tbb:' 'MPICH_F77LINKER=/usr/local/other/utilities/intel/composerxe/bin/ifort' 'INTELDIR=/usr/local/other/utilities/intel/composerxe' 'MAIL=/var/spool/mail/jhthomps' 'PATH=/usr/local/other/utilities/mvapich2/bin:/usr/local/other/utilities/intel/composerxe/bin/:/usr/lib64/qt-3.3/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/jhthomps/bin' 'MPICH_F77=/usr/local/other/utilities/intel/composerxe/bin/ifort' 'MVAPICH_ROOT=/usr/local/other/utilities/mvapich2' 'UTILDIR=/usr/local/other/utilities' 'PWD=/home/jhthomps' 'MPICH_CC=/usr/local/other/utilities/intel/composerxe/bin/icc' 'LANG=en_US.UTF-8' 'F77=/usr/local/other/utilities/intel/composerxe/bin/ifort' 'SELINUX_LEVEL_REQUESTED=' 'MPICH_F90=/usr/local/other/utilities/intel/composerxe/bin/ifort' 'CXX=/usr/local/other/utilities/intel/composerxe/bin/icpc' 'HISTCONTROL=ignoredups' 'MPICH_F90LINKER=/usr/local/other/utilities/intel/composerxe/bin/ifort' 'SHLVL=1' 'HOME=/home/jhthomps' 'MPD_BIN=/bin' 'FC=/usr/local/other/utilities/intel/composerxe/bin/ifort' 'MPICH_CCLINKER=/usr/local/other/utilities/intel/composerxe/bin/icpc' 'LOGNAME=jhthomps' 'QTLIB=/usr/lib64/qt-3.3/lib' 'CVS_RSH=ssh' '' 'LESSOPEN=|/usr/bin/lesspipe.sh %s' 'MV2_FASTSSH_THRESHOLD=1' 'CC=/usr/local/other/utilities/intel/composerxe/bin/icc' 'INCLUDE=/usr/local/other/utilities/mvapich2/include:/usr/local/other/utilities/intel/composer_xe_2013.0.079/compiler/include/intel64:' 'G_BROKEN_FILENAMES=1' '_=/usr/local/other/utilities/mvapich2/bin/mpirun' --global-user-env 0 --global-system-env 1 'GFORTRAN_UNBUFFERED_PRECONNECTED=y' --proxy-core-count 1 --exec --exec-appnum 0 --exec-proc-count 1 --exec-local-env 0 --exec-wdir /home/jhthomps --exec-args 1 /usr/local/other/utilities/mvapich2/libexec/osu-micro-benchmarks/osu_bw
[mpiexec at rh64-1-ib] PMI FD: (null); PMI PORT: (null); PMI ID/RANK: -1
Arguments being passed to proxy 1:
--version 1.4.1p1 --iface-ip-env-name MPICH_INTERFACE_HOSTNAME --hostname rh64-3-ib --global-core-map 1,1,0 --filler-process-map 1,1,0 --global-process-count 2 --auto-cleanup 1 --pmi-rank -1 --pmi-kvsname kvs_4129_0 --pmi-process-mapping (vector,(0,2,1)) --ckpoint-num -1 --global-inherited-env 52 'MPICH_CLINKER=/usr/local/other/utilities/intel/composerxe/bin/icc' 'HOSTNAME=rh64-1-ib' 'SELINUX_ROLE_REQUESTED=' 'MPICH_CCC=/usr/local/other/utilities/intel/composerxe/bin/icpc' 'TERM=xterm' 'SHELL=/bin/bash' 'HISTSIZE=1000' 'TMPDIR=/tmp' '' 'LIBRARY_PATH=/usr/local/other/utilities/mvapich2/lib:/usr/local/other/utilities/intel/composer_xe_2013.0.079/compiler/lib/intel64:' 'SELINUX_USE_CURRENT_RANGE=' 'FPATH=/usr/local/other/utilities/mvapich2/include:/usr/local/other/utilities/intel/composerxe/mkl/include:' 'QTDIR=/usr/lib64/qt-3.3' 'MV2_NPROCS_THRESHOLD=18000' 'CCC=/usr/local/other/utilities/intel/composerxe/bin/icpc' 'QTINC=/usr/lib64/qt-3.3/include' 'SSH_TTY=/dev/pts/0' 'USER=jhthomps' 'LD_LIBRARY_PATH=/usr/local/other/utilities/mvapich2/lib:/usr/local/other/utilities/intel/composer_xe_2013.0.079/compiler/lib/intel64:' 'LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=01;05;37;41:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arj=01;31:*.taz=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.lz=01;31:*.xz=01;31:*.bz2=01;31:*.tbz=01;31:*.tbz2=01;31:*.bz=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.rar=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.jpg=01;35:*.jpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.axv=01;35:*.anx=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=01;36:*.au=01;36:*.flac=01;36:*.mid=01;36:*.midi=01;36:*.mka=01;36:*.mp3=01;36:*.mpc=01;36:*.ogg=01;36:*.ra=01;36:*.wav=01;36:*.axa=01;36:*.oga=01;36:*.spx=01;36:*.xspf=01;36:' 'CPATH=/usr/local/other/utilities/mvapich2/include:/usr/local/other/utilities/intel/composerxe/ipp/include:/usr/local/other/utilities/intel/composerxe/mkl/include:/usr/local/other/utilities/intel/composerxe/tbb/include/tbb:' 'MPICH_F77LINKER=/usr/local/other/utilities/intel/composerxe/bin/ifort' 'INTELDIR=/usr/local/other/utilities/intel/composerxe' 'MAIL=/var/spool/mail/jhthomps' 'PATH=/usr/local/other/utilities/mvapich2/bin:/usr/local/other/utilities/intel/composerxe/bin/:/usr/lib64/qt-3.3/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/jhthomps/bin' 'MPICH_F77=/usr/local/other/utilities/intel/composerxe/bin/ifort' 'MVAPICH_ROOT=/usr/local/other/utilities/mvapich2' 'UTILDIR=/usr/local/other/utilities' 'PWD=/home/jhthomps' 'MPICH_CC=/usr/local/other/utilities/intel/composerxe/bin/icc' 'LANG=en_US.UTF-8' 'F77=/usr/local/other/utilities/intel/composerxe/bin/ifort' 'SELINUX_LEVEL_REQUESTED=' 'MPICH_F90=/usr/local/other/utilities/intel/composerxe/bin/ifort' 'CXX=/usr/local/other/utilities/intel/composerxe/bin/icpc' 'HISTCONTROL=ignoredups' 'MPICH_F90LINKER=/usr/local/other/utilities/intel/composerxe/bin/ifort' 'SHLVL=1' 'HOME=/home/jhthomps' 'MPD_BIN=/bin' 'FC=/usr/local/other/utilities/intel/composerxe/bin/ifort' 'MPICH_CCLINKER=/usr/local/other/utilities/intel/composerxe/bin/icpc' 'LOGNAME=jhthomps' 'QTLIB=/usr/lib64/qt-3.3/lib' 'CVS_RSH=ssh' '' 'LESSOPEN=|/usr/bin/lesspipe.sh %s' 'MV2_FASTSSH_THRESHOLD=1' 'CC=/usr/local/other/utilities/intel/composerxe/bin/icc' 'INCLUDE=/usr/local/other/utilities/mvapich2/include:/usr/local/other/utilities/intel/composer_xe_2013.0.079/compiler/include/intel64:' 'G_BROKEN_FILENAMES=1' '_=/usr/local/other/utilities/mvapich2/bin/mpirun' --global-user-env 0 --global-system-env 1 'GFORTRAN_UNBUFFERED_PRECONNECTED=y' --proxy-core-count 1 --exec --exec-appnum 0 --exec-proc-count 1 --exec-local-env 0 --exec-wdir /home/jhthomps --exec-args 1 /usr/local/other/utilities/mvapich2/libexec/osu-micro-benchmarks/osu_bw
[mpiexec at rh64-1-ib] Launch arguments: /usr/local/other/utilities/mvapich2/bin/hydra_pmi_proxy --control-port rh64-1-ib:59143 --debug --rmk user --launcher ssh --demux poll --pgid 0 --retries 10 --proxy-id 0
[mpiexec at rh64-1-ib] Launch arguments: /usr/bin/ssh -x rh64-3-ib "/usr/local/other/utilities/mvapich2/bin/hydra_pmi_proxy" --control-port rh64-1-ib:59143 --debug --rmk user --launcher ssh --demux poll --pgid 0 --retries 10 --proxy-id 1
[proxy:0:0 at rh64-1-ib] got pmi command (from 0): init
pmi_version=1 pmi_subversion=1
[proxy:0:0 at rh64-1-ib] PMI response: cmd=response_to_init pmi_version=1 pmi_subversion=1 rc=0
[proxy:0:0 at rh64-1-ib] got pmi command (from 0): get_maxes
[proxy:0:0 at rh64-1-ib] PMI response: cmd=maxes kvsname_max=256 keylen_max=64 vallen_max=1024
[proxy:0:0 at rh64-1-ib] got pmi command (from 0): get_appnum
[proxy:0:0 at rh64-1-ib] PMI response: cmd=appnum appnum=0
[proxy:0:0 at rh64-1-ib] got pmi command (from 0): get_my_kvsname
[proxy:0:0 at rh64-1-ib] PMI response: cmd=my_kvsname kvsname=kvs_4129_0
[proxy:0:0 at rh64-1-ib] got pmi command (from 0): get_my_kvsname
[proxy:0:0 at rh64-1-ib] PMI response: cmd=my_kvsname kvsname=kvs_4129_0
[proxy:0:0 at rh64-1-ib] got pmi command (from 0): get
kvsname=kvs_4129_0 key=PMI_process_mapping
[proxy:0:0 at rh64-1-ib] PMI response: cmd=get_result rc=0 msg=success value=(vector,(0,2,1))
[cli_0]: aborting job:
Fatal error in MPI_Init:
Other MPI error
=====================================================================================
= BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
= EXIT CODE: 256
= CLEANING UP REMAINING PROCESSES
= YOU CAN IGNORE THE BELOW CLEANUP MESSAGES
=====================================================================================
[proxy:0:1 at rh64-3-ib] got pmi command (from 4): init
pmi_version=1 pmi_subversion=1
[proxy:0:1 at rh64-3-ib] PMI response: cmd=response_to_init pmi_version=1 pmi_subversion=1 rc=0
[proxy:0:1 at rh64-3-ib] got pmi command (from 4): get_maxes
[proxy:0:1 at rh64-3-ib] PMI response: cmd=maxes kvsname_max=256 keylen_max=64 vallen_max=1024
[proxy:0:1 at rh64-3-ib] got pmi command (from 4): get_appnum
[proxy:0:1 at rh64-3-ib] PMI response: cmd=appnum appnum=0
[proxy:0:1 at rh64-3-ib] got pmi command (from 4): get_my_kvsname
[proxy:0:1 at rh64-3-ib] PMI response: cmd=my_kvsname kvsname=kvs_4129_0
[proxy:0:1 at rh64-3-ib] got pmi command (from 4): get_my_kvsname
[proxy:0:1 at rh64-3-ib] PMI response: cmd=my_kvsname kvsname=kvs_4129_0
[proxy:0:1 at rh64-3-ib] got pmi command (from 4): get
kvsname=kvs_4129_0 key=PMI_process_mapping
[proxy:0:1 at rh64-3-ib] PMI response: cmd=get_result rc=0 msg=success value=(vector,(0,2,1))
[cli_1]: aborting job:
Fatal error in MPI_Init:
Other MPI error
=====================================================================================
= BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
= EXIT CODE: 256
= CLEANING UP REMAINING PROCESSES
= YOU CAN IGNORE THE BELOW CLEANUP MESSAGES
=====================================================================================
More information about the mvapich-discuss
mailing list