[mvapich-discuss] error in ibv_channel_manager.c

[Ben]

I'm getting this error message from mvapich while running a program at 
large core count. It appears to be during a read being performed by the 
root process. Does anyone have any ideas on what could be causing it?


Got FATAL event 0
at line 990 in file ibv_channel_manager.c

my run command is:

mpirun_rsh -hostfile $PBS_NODEIFLE -np 7200 MV2_USE_UD_HYBRID=0 executable


and a mpiname -a shows:

MVAPICH2 1.8.1 Thu Sep 27 18:55:23 EDT 2012 ch3:mrail

Compilation
CC: icc -fpic -m64   -DNDEBUG -DNVALGRIND -O2
CXX: icpc -fpic -m64  -DNDEBUG -DNVALGRIND -O2
F77: ifort -fpic  -O2
FC: ifort -fpic  -O2

Configuration
CC=icc CXX=icpc F77=ifort FC=ifort CFLAGS=-fpic -m64 CXXFLAGS=-fpic -m64 
FFLAGS=-fpic FCFLAGS=-fpic --enable-f77 --enable-fc --enable-cxx 
--enable-romio --enable-threads=default --with-hwloc 
--disable-multi-aliases --enable-xrc=yes --enable-hybrid 
--prefix=/usr/local/other/SLES11.1/mvapich2/1.8.1/intel-13.1.2.183

-- 
Ben Auer, PhD   SSAI, Scientific Programmer/Analyst
NASA GSFC,  Global Modeling and Assimilation Office
Code 610.1, 8800 Greenbelt Rd, Greenbelt, MD  20771
Phone: 301-286-9176               Fax: 301-614-6246