[mvapich-discuss] MPIDI_CH3I_CM_Init: Error initializing MVAPICH2 malloc library

[Gilles Civario]

/* re-send in plain text rather than HTML. Sorry guys */

Hi guys,

I installed mvapich2-1.7-rc1 on one of our cluster to test some features, and with the code I played with, I encountered a problem: while scaling my runs using powers of 2 processes, the code ran beautifully up to 64 processes, and with 128 gave me, for each process, the following message:

Fatal error in MPI_Init:
Other MPI error, error stack:
MPIR_Init_thread(310)..: Initialization failed
MPID_Init(113).........: channel initialization failed
MPIDI_CH3_Init(161)....:
MPIDI_CH3I_CM_Init(828): Error initializing MVAPICH2 malloc library

After googleing the web and browsing the sources, I suspected an issue of some sort with my max locked memory limit. But I managed to rule it out.
Then I suspected some issue with our mpiexec (which is not the one coming with the library), but here again, I ruled it out.
So I tried a more systematic approach and I searched in the on line documentation all environment variables I could play with which default to 128 and set them one after the other to 256... Still no joy. Then I did the same with the ones default to 64 and set them to 128. And then, MV2_ON_DEMAND_THRESHOLD did the trick!
So my issue is now solved. However, I have a few questions / requests:
   * what is the impact for me to set this specific environment variable, to a value large enough so that I don't experiment those crashes in the future?
    * Could it be possible to either document the error / message, or to make it more explicit so that people experimenting the same issue can sort it out more easily than I did?

Thank you for your great job guys.

Cheers.

Gilles Civario