[mvapich-discuss] tcmalloc issues

Adam T. Moody moody20 at llnl.gov
Thu Dec 18 15:25:40 EST 2014 

Hi Hari,
To echo Jeff's request, we also have a number of applications that use 
tcmalloc.  It significantly outperforms the default malloc/free on the 
system, and some of our apps spend a lot of time in malloc/free.  
TCMalloc is especially beneficial for threaded apps, which you can see 
in the performance graphs here:

http://gperftools.googlecode.com/svn/trunk/doc/tcmalloc.html

As Jeff suggested, even if one disables the registration cache, it would 
be a nice if an app gets the benefit of pre-registered memory if it 
allocates its communication buffers via MPI_ALLOC_MEM.

Also, if use of tcmalloc is becoming more common, it may be worth 
looking at adding support for tcmalloc in place of ptmalloc within MVAPICH.
-Adam


Hari Subramoni wrote:

>Dear Jeff,
>
>It looks like this is more of a runtime issue than a compile time issue.
>For this, don't need to disable registration cache at configure time. The
>environment variable you are currently using is fine.
>
>Is it possible for us to get a reproducer for the issue so that we can
>debug it locally?
>
>Regards,
>Hari.
>
>On Tue, Dec 16, 2014 at 10:54 AM, Jeff Hammond <jeff.science at gmail.com>
>wrote:
>  
>
>>This is the issue we see. Let me know if there's a good way to debug.
>>
>>Jeff
>>
>>Begin forwarded message:
>>
>>*From:* Justus Calvin <>
>>*Date:* December 16, 2014 at 4:59:59 AM PST
>>*To:* madness-developers at googlegroups.com
>>*Subject:* *Re: [madness-developers: 3845] Fwd: [mvapich-discuss]
>>tcmalloc issues*
>>*Reply-To:* madness-developers at googlegroups.com
>>
>>I found that it was necessary to use MV2_USE_LAZY_MEM_UNREGISTER=0 when
>>running my code on our small cluster. The problem that lead us to disable
>>this feature was a consistent, but random, assertion failure. The failure
>>that I saw was:
>>
>>
>>
>>Assertion failed in file
>>../src/mpid/ch3/channels/common/src/reg_cache/dreg.c at line 904:
>>d->is_valid == 0
>>[cli_5]: aborting job:
>>internal ABORT - process 5
>>
>>
>>Sent from my iPhone
>>
>>On Dec 15, 2014, at 7:53 PM, Hari Subramoni <subramoni.1 at osu.edu> wrote:
>>
>>Dear Jeff,
>>
>>Whether disabling registration cache will have a negative effect on
>>communication performance depends entirely on the communication pattern of
>>the application. If the application uses mostly small to medium sized
>>messages (say < 16KB), then disabling registration cache will mostly have
>>no impact on the application. However, if the application uses messages of
>>larger size, then there might be an impact depending on the frequency of
>>communication. If this is the case, then it might be useful to increase the
>>VBUF (MV2_VBUF_TOTAL_SIZE) size and eager-rendezvous threshold
>>(MV2_IBA_EAGER_THRESHOLD) to a larger value. In this scenario, we recommend
>>that you set both to the same value (possibly slightly greater than the
>>median message size being used by your application). Please refer to the
>>following sections of the userguide for more information about these two
>>parameters.
>>
>>
>>http://mvapich.cse.ohio-state.edu/static/media/mvapich/mvapich2-2.1a-userguide.html#x1-26000011.105
>>
>>http://mvapich.cse.ohio-state.edu/static/media/mvapich/mvapich2-2.1a-userguide.html#x1-17800011.23
>>
>>MVAPICH2 has the capability to disable registration cache support at
>>configure time (using --disable-registration-cache as I mentioned) and
>>also at runtime by setting MV2_USE_LAZY_MEM_UNREGISTER=0. From your
>>original e-mail, I understood that you were having a compile/link time
>>issue with the MADNESS code. That is why I suggested disabling registration
>>cache at configure time. If the issue you were facing is at runtime, then
>>you can just use the runtime parameter mentioned above to disable
>>registration cache at runtime. Please refer to the following sections of
>>the userguide for more information about this parameters.
>>
>>
>>http://mvapich.cse.ohio-state.edu/static/media/mvapich/mvapich2-2.1a-userguide.html#x1-23700011.82
>>
>>I am looking into the RMA / MPI_Alloc_mem question you posed. My initial
>>impression is that it will work even if registration cache is disabled.
>>However, I will double check on that and get back to you soon.
>>
>>Regards,
>>Hari.
>>
>>PS: My e-mails to the madness-developers groups is bouncing. So you may
>>have to let them know about this reply.
>>
>>On Mon, Dec 15, 2014 at 3:48 PM, Jeff Hammond <jeff.science at gmail.com>
>>wrote:
>>    
>>
>>>Hi Hari,
>>>
>>>Thanks for the tip.  We can try this.
>>>
>>>Do you think it might be necessary to increase the eager-rendezvous
>>>cutoff to ameliorate the performance loss associated with not caching
>>>registration?  On a related note, does this disable the registration
>>>cache for allocation via MPI_Alloc_mem as well as malloc?  Because we
>>>might be able to use MPI_Alloc_mem for at least some of our
>>>communication buffers to reduce the registration penalty whenever
>>>possible.  Furthermore, can we increase use of RDMA without the
>>>registration cache if we use RMA instead of two-sided?
>>>
>>>Is there any chance that a future version of MVAPICH could enable
>>>--disable-registration-cache as a runtime option instead?  I build
>>>MVAPICH from source regularly, but this is certainly not the norm.
>>>The users of MADNESS will almost certainly count on their local
>>>sysadmin to install MPI for them.
>>>
>>>Obviously, I know changing build-time options to runtime-options is
>>>not trivial, so I'm not expecting this to happen overnight.
>>>
>>>Thanks,
>>>
>>>Jeff
>>>
>>>On Sun, Dec 14, 2014 at 4:08 PM, Hari Subramoni <subramoni.1 at osu.edu>
>>>wrote:
>>>      
>>>
>>>>Hello Jeff/MADNESS team,
>>>>
>>>>Sorry to hear that you are having issues compiling your application with
>>>>MVAPICH2. We are seeing this issue for the first time. Nobody had posted
>>>>anything along this line to mvapich-discuss earlier.
>>>>
>>>>Could you please try configuring MVAPICH2 without registration cache
>>>>        
>>>>
>>>using
>>>      
>>>
>>>>the "-disable-registration-cache" switch at configure time? This should
>>>>prevent PTMalloc related issues that you are encountering. Please let us
>>>>know if this helps. The following section of the userguide has more
>>>>information about how to configure MVPAICH2
>>>>
>>>>
>>>>        
>>>>
>>>http://mvapich.cse.ohio-state.edu/static/media/mvapich/mvapich2-2.1a-userguide.html#x1-110004.4
>>>      
>>>
>>>>Please let us know if you encounter any other performance or
>>>>        
>>>>
>>>functionality
>>>      
>>>
>>>>issues in using your application with MVAPICH2 and we will be glad to
>>>>        
>>>>
>>>help.
>>>      
>>>
>>>>Regards,
>>>>Hari.
>>>>
>>>>On Sun, Dec 14, 2014 at 5:35 PM, Jeff Hammond <jeff.science at gmail.com>
>>>>wrote:
>>>>        
>>>>
>>>>>Hi,
>>>>>
>>>>>The MADNESS team (copied) has encountered an issue with MVAPICH2 and
>>>>>Google TCMalloc, which is not the first time this has happened (see
>>>>>GraphLab:
>>>>>
>>>>>          
>>>>>
>>>http://forum.graphlab.com/discussion/51/unable-to-compile-graphlab-analytics-with-mpi
>>>).
>>>      
>>>
>>>>>I don't have the details, but the nature of the issue seems pretty
>>>>>obvious from the GraphLab explanation.
>>>>>
>>>>>TCMalloc improves the performance of MADNESS in a nontrivial way,
>>>>>hence it is a very bad user experience for us if MVAPICH2 does not
>>>>>allow us to use it.  We can, of course, disable TCMalloc, but since
>>>>>small and non-small object allocation from C++ on many threads is
>>>>>something MADNESS does constantly, PTMalloc is going to be suboptimal,
>>>>>since it is known to not handle this usage well.
>>>>>
>>>>>What is the workaround for this and is there a long-term solution that
>>>>>will allow MVAPICH2 to interoperate with memory allocators other than
>>>>>ptmalloc?  It is particularly important to have MVAPICH2 interoperate
>>>>>with JEMalloc in the future (see
>>>>>https://github.com/memkind/memkind/blob/master/README for clues as to
>>>>>why).
>>>>>
>>>>>Thanks,
>>>>>
>>>>>Jeff
>>>>>
>>>>>--
>>>>>Jeff Hammond
>>>>>jeff.science at gmail.com
>>>>>http://jeffhammond.github.io/
>>>>>_______________________________________________
>>>>>mvapich-discuss mailing list
>>>>>mvapich-discuss at cse.ohio-state.edu
>>>>>http://mailman.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss
>>>>>          
>>>>>
>>>
>>>--
>>>Jeff Hammond
>>>jeff.science at gmail.com
>>>http://jeffhammond.github.io/
>>>
>>>      
>>>
>
>  
>
>------------------------------------------------------------------------
>
>_______________________________________________
>mvapich-discuss mailing list
>mvapich-discuss at cse.ohio-state.edu
>http://mailman.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss
>  
>

More information about the mvapich-discuss mailing list