[mvapich-discuss] (no subject)

[Troy Telford]

Quick question:  In building MVAPICH, I've noticed the CFLAG  
-DLAZY_MEM_UNREGISTER in the make.mvapich.* scripts.

Now when I've run benchmarks such as HPL or HPCC, (where it's possible to  
vary the amount of memory used), I've noticed that if I set the benchmark  
to use, say, 70% of the systems memory, it gets allocated.  If the next  
problem size uses 1% of the memory, the memory doesn't get freed.  This  
causes a problem if the third problem size uses 75% of the memory, as it  
will then allocate another 75% of the system's memory (ie. total  
allocation=145%; hits the swapfile, everything screeches to a halt).

With other MPI implementations I've tried (although this also includes  
non-IB interconnects), I can use this exact same HPL/HPCC configuration,  
and it will have no problem completing the HPL or HPCC run.

Now, if I remove -DLAZY_MEM_UNREGISTER, the MVAPICH handles  
allocation/deallocation like the other MPI implementations.

I realize there are some performance implications with removing  
-DLAZY_MEM_UNREGISTER, but I am willing to live with that if I can  
actually finish my jobs.

But I'm wondering if there are any other side-effects of disabling  
-DLAZY_MEM_UNREGISTER.

Thanks,
Troy Telford