[mvapich-discuss] failure during startup when using mvapich2-1.2

[Dan Kokron]

My first attempt to use mvapich2-1.2 fails with the following errors.
Any ideas why this is failing?

The config.log is attached.

mpirun_rsh -ssh -np 240 -hostfile machinefile ./GEOSgcm.x
[cli_14]: readline failed
Segmentation fault
[cli_10]: readline failed
[cli_11]: readline failed
[cli_12]: readline failed
[cli_8]: readline failed
[cli_15]: readline failed
[cli_9]: readline failed
[cli_13]: readline failed
Fatal error in MPI_Init_thread:
Other MPI error, error stack:
MPIR_Init_thread(310)...........: Initialization failed
MPID_Init(113)..................: channel initialization failed
MPIDI_CH3_Init(182).............: 
MPIDI_CH3I_SMP_Init(652)........: 
MPIDI_CH3I_SMP_pull_header(2599): PMI_KVS_Get returned 0
Fatal error in MPI_Init_thread:
Other MPI error, error stack:
MPIR_Init_thread(310)...........: Initialization failed
MPID_Init(113)..................: channel initialization failed
MPIDI_CH3_Init(182).............: 
MPIDI_CH3I_SMP_Init(652)........: 
MPIDI_CH3I_SMP_pull_header(2599): PMI_KVS_Get returned 0
Fatal error in MPI_Init_thread:
Other MPI error, error stack:
MPIR_Init_thread(310)...........: Initialization failed
MPID_Init(113)..................: channel initialization failed
MPIDI_CH3_Init(182).............: 
MPIDI_CH3I_SMP_Init(652)........: 
MPIDI_CH3I_SMP_pull_header(2599): PMI_KVS_Get returned 0
Fatal error in MPI_Init_thread:
Other MPI error, error stack:
MPIR_Init_thread(310)...........: Initialization failed
MPID_Init(113)..................: channel initialization failed
MPIDI_CH3_Init(182).............: 
MPIDI_CH3I_SMP_Init(652)........: 
MPIDI_CH3I_SMP_pull_header(2599): PMI_KVS_Get returned 0
Fatal error in MPI_Init_thread:
Other MPI error, error stack:
MPIR_Init_thread(310)...........: Initialization failed
MPID_Init(113)..................: channel initialization failed
MPIDI_CH3_Init(182).............: 
MPIDI_CH3I_SMP_Init(652)........: 
MPIDI_CH3I_SMP_pull_header(2599): PMI_KVS_Get returned 0
[cli_0]: readline failed
Fatal error in MPI_Init_thread:
Other MPI error, error stack:
MPIR_Init_thread(310)...........: Initialization failed
MPID_Init(113)..................: channel initialization failed
MPIDI_CH3_Init(182).............: 
MPIDI_CH3I_SMP_Init(652)........: 
MPIDI_CH3I_SMP_pull_header(2599): PMI_KVS_Get returned 0
Fatal error in MPI_Init_thread:
Other MPI error, error stack:
MPIR_Init_thread(310)...........: Initialization failed
MPID_Init(113)..................: channel initialization failed
MPIDI_CH3_Init(182).............: 
MPIDI_CH3I_SMP_Init(652)........: 
MPIDI_CH3I_SMP_pull_header(2599): PMI_KVS_Get returned 0
Fatal error in MPI_Init_thread:
Other MPI error, error stack:
MPIR_Init_thread(310)...........: Initialization failed
MPID_Init(113)..................: channel initialization failed
MPIDI_CH3_Init(182).............: 
MPIDI_CH3I_SMP_Init(652)........: 
MPIDI_CH3I_SMP_pull_header(2599): PMI_KVS_Get returned 0

Child exited abnormally!
cleanupKilling remote processes...[cli_27]: readline failed
Broken pipe
[cli_4]: readline failed
[cli_6]: [cli_3]: readline failed
[cli_7]: readline failed
readline failed
[cli_1]: readline failed
[cli_2]: readline failed
[cli_5]: readline failed
Fatal error in MPI_Init_thread:
Other MPI error, error stack:
MPIR_Init_thread(310)...........: Initialization failed
MPID_Init(113)..................: channel initialization failed
MPIDI_CH3_Init(182).............: 
MPIDI_CH3I_SMP_Init(652)........: 
MPIDI_CH3I_SMP_pull_header(2599): PMI_KVS_Get returned 0
Fatal error in MPI_Init_thread:
Other MPI error, error stack:
MPIR_Init_thread(310)...........: Initialization failed
MPID_Init(113)..................: channel initialization failed
MPIDI_CH3_Init(182).............: 
MPIDI_CH3I_SMP_Init(652)........: 
MPIDI_CH3I_SMP_pull_header(2599): PMI_KVS_Get returned 0
Fatal error in MPI_Init_thread:
Other MPI error, error stack:
MPIR_Init_thread(310)...........: Initialization failed
MPID_Init(113)..................: channel initialization failed
MPIDI_CH3_Init(182).............: 
MPIDI_CH3I_SMP_Init(652)........: 
MPIDI_CH3I_SMP_pull_header(2599): PMI_KVS_Get returned 0
Fatal error in MPI_Init_thread:
Other MPI error, error stack:
MPIR_Init_thread(310)...........: Initialization failed
MPID_Init(113)..................: channel initialization failed
MPIDI_CH3_Init(182).............: 
MPIDI_CH3I_SMP_Init(652)........: 
MPIDI_CH3I_SMP_pull_header(2599): PMI_KVS_Get returned 0
Fatal error in MPI_Init_thread:
Other MPI error, error stack:
MPIR_Init_thread(310)...........: Initialization failed
MPID_Init(113)..................: channel initialization failed
MPIDI_CH3_Init(182).............: 
MPIDI_CH3I_SMP_Init(652)........: 
MPIDI_CH3I_SMP_pull_header(2599): PMI_KVS_Get returned 0
Fatal error in MPI_Init_thread:
Other MPI error, error stack:
MPIR_Init_thread(310)...........: Initialization failed
MPID_Init(113)..................: channel initialization failed
MPIDI_CH3_Init(182).............: 
MPIDI_CH3I_SMP_Init(652)........: 
MPIDI_CH3I_SMP_pull_header(2599): PMI_KVS_Get returned 0
Fatal error in MPI_Init_thread:
Other MPI error, error stack:
MPIR_Init_thread(310)...........: Initialization failed
MPID_Init(113)..................: channel initialization failed
MPIDI_CH3_Init(182).............: 
MPIDI_CH3I_SMP_Init(652)........: 
MPIDI_CH3I_SMP_pull_header(2599): PMI_KVS_Get returned 0
Fatal error in MPI_Init_thread:
Other MPI error, error stack:
MPIR_Init_thread(310)...........: Initialization failed
MPID_Init(113)..................: channel initialization failed
MPIDI_CH3_Init(182).............: 
MPIDI_CH3I_SMP_Init(652)........: 
MPIDI_CH3I_SMP_pull_header(2599): PMI_KVS_Get returned 0
Broken pipe
[cli_24]: readline failed
[cli_30]: [cli_28]: readline failed
readline failed
[cli_25]: readline failed
[cli_29]: readline failed
[cli_31]: readline failed
[cli_26]: readline failed
Fatal error in MPI_Init_thread:
Other MPI error, error stack:
MPIR_Init_thread(310)...........: Initialization failed
MPID_Init(113)..................: channel initialization failed
MPIDI_CH3_Init(182).............: 
MPIDI_CH3I_SMP_Init(652)........: 
MPIDI_CH3I_SMP_pull_header(2599): PMI_KVS_Get returned 0
Fatal error in MPI_Init_thread:
Other MPI error, error stack:
MPIR_Init_thread(310)...........: Initialization failed
MPID_Init(113)..................: channel initialization failed
MPIDI_CH3_Init(182).............: 
MPIDI_CH3I_SMP_Init(652)........: 
MPIDI_CH3I_SMP_pull_header(2599): PMI_KVS_Get returned 0
Fatal error in MPI_Init_thread:
Other MPI error, error stack:
MPIR_Init_thread(310)...........: Initialization failed
MPID_Init(113)..................: channel initialization failed
MPIDI_CH3_Init(182).............: 
MPIDI_CH3I_SMP_Init(652)........: 
MPIDI_CH3I_SMP_pull_header(2599): PMI_KVS_Get returned 0
Fatal error in MPI_Init_thread:
Other MPI error, error stack:
MPIR_Init_thread(310)...........: Initialization failed
MPID_Init(113)..................: channel initialization failed
MPIDI_CH3_Init(182).............: 
MPIDI_CH3I_SMP_Init(652)........: 
MPIDI_CH3I_SMP_pull_header(2599): PMI_KVS_Get returned 0
Fatal error in MPI_Init_thread:
Other MPI error, error stack:
MPIR_Init_thread(310)...........: Initialization failed
MPID_Init(113)..................: channel initialization failed
MPIDI_CH3_Init(182).............: 
MPIDI_CH3I_SMP_Init(652)........: 
MPIDI_CH3I_SMP_pull_header(2599): PMI_KVS_Get returned 0
Fatal error in MPI_Init_thread:
Other MPI error, error stack:
MPIR_Init_thread(310)...........: Initialization failed
MPID_Init(113)..................: channel initialization failed
MPIDI_CH3_Init(182).............: 
MPIDI_CH3I_SMP_Init(652)........: 
MPIDI_CH3I_SMP_pull_header(2599): PMI_KVS_Get returned 0
Fatal error in MPI_Init_thread:
Other MPI error, error stack:
MPIR_Init_thread(310)...........: Initialization failed
MPID_Init(113)..................: channel initialization failed
MPIDI_CH3_Init(182).............: 
MPIDI_CH3I_SMP_Init(652)........: 
MPIDI_CH3I_SMP_pull_header(2599): PMI_KVS_Get returned 0
Fatal error in MPI_Init_thread:
Other MPI error, error stack:
MPIR_Init_thread(310)...........: Initialization failed
MPID_Init(113)..................: channel initialization failed
MPIDI_CH3_Init(182).............: 
MPIDI_CH3I_SMP_Init(652)........: 
MPIDI_CH3I_SMP_pull_header(2599): PMI_KVS_Get returned 0
-- 
Dan Kokron
Global Modeling and Assimilation Office
NASA Goddard Space Flight Center
Greenbelt, MD 20771
Daniel.S.Kokron at nasa.gov
Phone: (301) 614-5192
Fax:   (301) 614-5304
-------------- next part --------------
A non-text attachment was scrubbed...
Name: config.log
Type: text/x-log
Size: 204629 bytes
Desc: not available
Url : http://mail.cse.ohio-state.edu/pipermail/mvapich-discuss/attachments/20081118/34d9e2a5/config-0001.bin