[mvapich-discuss] (no subject)

[Sayantan Sur]

Hello Troy,

* On Mar,1 Troy Telford<ttelford at linuxnetworx.com> wrote :
> Quick question:  In building MVAPICH, I've noticed the CFLAG  
> -DLAZY_MEM_UNREGISTER in the make.mvapich.* scripts.
> 
> Now when I've run benchmarks such as HPL or HPCC, (where it's possible to  
> vary the amount of memory used), I've noticed that if I set the benchmark  
> to use, say, 70% of the systems memory, it gets allocated.  If the next  
> problem size uses 1% of the memory, the memory doesn't get freed.  This  
> causes a problem if the third problem size uses 75% of the memory, as it  
> will then allocate another 75% of the system's memory (ie. total  
> allocation=145%; hits the swapfile, everything screeches to a halt).

I have some questions about how you are running these programs:

1) Are you *simultaneously* running three MPI applications on a cluster,
with memory usage configured to be 70%, 1% and 75%?

OR

2) Are you starting the job which uses 70% of memory and while that is
executing, you start the job with 1% memory and that finishes (but the
70% memory job is still running) ... on top of that you are starting the
job with 75% memory consumption? Even in this mode, the total memory
requirements imposed on the nodes is 145% of their capacity.

The memory consumed by each process should be freed as soon as the
process has ended. However, if the process (ie. HPL) which is running,
still has the memory allocated, then there is not much we can do :-)

> With other MPI implementations I've tried (although this also includes  
> non-IB interconnects), I can use this exact same HPL/HPCC configuration,  
> and it will have no problem completing the HPL or HPCC run.

Can you please tell us if these MPI implementations had the registration
caching mechanism a.k.a. -DLAZY_MEM_UNREGISTER enabled during these
runs? IMHO, if you were able to get this config to run on other MPI
implementations (which required memory registration) to work without
`caching' of registrations, you should be able to do the exact same
thing with MVAPICH, by disabling -DLAZY_MEM_UNREGISTER.

> Now, if I remove -DLAZY_MEM_UNREGISTER, the MVAPICH handles  
> allocation/deallocation like the other MPI implementations.
> 
> I realize there are some performance implications with removing  
> -DLAZY_MEM_UNREGISTER, but I am willing to live with that if I can  
> actually finish my jobs.
> 
> But I'm wondering if there are any other side-effects of disabling  
> -DLAZY_MEM_UNREGISTER.

Right. As long as you are aware of the performance implications of
turning registration cache off, it should be fine. There will be no
other side effects.

Thanks,
Sayantan.

> 
> Thanks,
> Troy Telford
> _______________________________________________
> mvapich-discuss mailing list
> mvapich-discuss at cse.ohio-state.edu
> http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss

-- 
http://www.cse.ohio-state.edu/~surs