[mvapich-discuss] time spent in mpi_init and MV2_ON_DEMAND_THRESHOLD

[Qi Gao]

Hi Greg,

Thanks for trying our MVAPICH2 and letting us know the problem. We are glad 
to work with you to solve the problem.

I assume this stack trace represents the place where most time is spent. And 
it seems that it's in SMP channel initialization function, and somehow it 
stucks at the PMI calls to process manager.

> #3  0x00000000004158b2 in MPIDI_CH3I_SMP_init ()
> #4  0x00000000004456f9 in MPIDI_CH3_Init ()

To help us narrow down the problem, would you try to disable SMP channel and 
try to see whether that will help?

To disable SMP channel, you need to patch the default make script 
"make.mvapich2.ofa". Here is the patch:

========
--- make.mvapich2.ofa.nosmp     2007-05-03 11:44:53.000000000 -0400
+++ make.mvapich2.ofa   2007-03-08 22:47:23.000000000 -0500
@@ -129,8 +129,6 @@
     SHARED_LIBS=""
 fi

+export SMP_FLAG=""
+
 export LD_LIBRARY_PATH=$OPEN_IB_LIB:$LD_LIBRARY_PATH
 export LIBS=${LIBS:--L${OPEN_IB_LIB} ${BLCR_LIB} 
${RDMA_CM_LIBS} -libverbs -libumad -lpthread}
 export FFLAGS=${FFLAGS:--L${OPEN_IB_LIB}}
========


Thanks!

--Qi