[mvapich-discuss] tcmalloc issues

Thu Dec 18 16:47:02 EST 2014

An abstract heap allocator interface would be most useful since, like
I said before, there are contexts wherein jemalloc support may be very
important.

Jeff

On Thu, Dec 18, 2014 at 12:25 PM, Adam T. Moody <moody20 at llnl.gov> wrote:
> Hi Hari,
> To echo Jeff's request, we also have a number of applications that use
> tcmalloc.  It significantly outperforms the default malloc/free on the
> system, and some of our apps spend a lot of time in malloc/free.  TCMalloc
> is especially beneficial for threaded apps, which you can see in the
> performance graphs here:
>
> http://gperftools.googlecode.com/svn/trunk/doc/tcmalloc.html
>
> As Jeff suggested, even if one disables the registration cache, it would be
> a nice if an app gets the benefit of pre-registered memory if it allocates
> its communication buffers via MPI_ALLOC_MEM.
>
> Also, if use of tcmalloc is becoming more common, it may be worth looking at
> adding support for tcmalloc in place of ptmalloc within MVAPICH.
> -Adam
>
>
> Hari Subramoni wrote:
>
>> Dear Jeff,
>>
>> It looks like this is more of a runtime issue than a compile time issue.
>> For this, don't need to disable registration cache at configure time. The
>> environment variable you are currently using is fine.
>>
>> Is it possible for us to get a reproducer for the issue so that we can
>> debug it locally?
>>
>> Regards,
>> Hari.
>>
>> On Tue, Dec 16, 2014 at 10:54 AM, Jeff Hammond <jeff.science at gmail.com>
>> wrote:
>>
>>>
>>> This is the issue we see. Let me know if there's a good way to debug.
>>>
>>> Jeff
>>>
>>> Begin forwarded message:
>>>
>>> *From:* Justus Calvin <>
>>> *Date:* December 16, 2014 at 4:59:59 AM PST
>>> *To:* madness-developers at googlegroups.com
>>> *Subject:* *Re: [madness-developers: 3845] Fwd: [mvapich-discuss]
>>> tcmalloc issues*
>>> *Reply-To:* madness-developers at googlegroups.com
>>>
>>>
>>> I found that it was necessary to use MV2_USE_LAZY_MEM_UNREGISTER=0 when
>>> running my code on our small cluster. The problem that lead us to disable
>>> this feature was a consistent, but random, assertion failure. The failure
>>> that I saw was:
>>>
>>>
>>>
>>> Assertion failed in file
>>> ../src/mpid/ch3/channels/common/src/reg_cache/dreg.c at line 904:
>>> d->is_valid == 0
>>> [cli_5]: aborting job:
>>> internal ABORT - process 5
>>>
>>>
>>> Sent from my iPhone
>>>
>>> On Dec 15, 2014, at 7:53 PM, Hari Subramoni <subramoni.1 at osu.edu> wrote:
>>>
>>> Dear Jeff,
>>>
>>> Whether disabling registration cache will have a negative effect on
>>> communication performance depends entirely on the communication pattern
>>> of
>>> the application. If the application uses mostly small to medium sized
>>> messages (say < 16KB), then disabling registration cache will mostly have
>>> no impact on the application. However, if the application uses messages
>>> of
>>> larger size, then there might be an impact depending on the frequency of
>>> communication. If this is the case, then it might be useful to increase
>>> the
>>> VBUF (MV2_VBUF_TOTAL_SIZE) size and eager-rendezvous threshold
>>> (MV2_IBA_EAGER_THRESHOLD) to a larger value. In this scenario, we
>>> recommend
>>> that you set both to the same value (possibly slightly greater than the
>>> median message size being used by your application). Please refer to the
>>> following sections of the userguide for more information about these two
>>> parameters.
>>>
>>>
>>>
>>> http://mvapich.cse.ohio-state.edu/static/media/mvapich/mvapich2-2.1a-userguide.html#x1-26000011.105
>>>
>>>
>>> http://mvapich.cse.ohio-state.edu/static/media/mvapich/mvapich2-2.1a-userguide.html#x1-17800011.23
>>>
>>> MVAPICH2 has the capability to disable registration cache support at
>>> configure time (using --disable-registration-cache as I mentioned) and
>>> also at runtime by setting MV2_USE_LAZY_MEM_UNREGISTER=0. From your
>>> original e-mail, I understood that you were having a compile/link time
>>> issue with the MADNESS code. That is why I suggested disabling
>>> registration
>>> cache at configure time. If the issue you were facing is at runtime, then
>>> you can just use the runtime parameter mentioned above to disable
>>> registration cache at runtime. Please refer to the following sections of
>>> the userguide for more information about this parameters.
>>>
>>>
>>>
>>> http://mvapich.cse.ohio-state.edu/static/media/mvapich/mvapich2-2.1a-userguide.html#x1-23700011.82
>>>
>>> I am looking into the RMA / MPI_Alloc_mem question you posed. My initial
>>> impression is that it will work even if registration cache is disabled.
>>> However, I will double check on that and get back to you soon.
>>>
>>> Regards,
>>> Hari.
>>>
>>> PS: My e-mails to the madness-developers groups is bouncing. So you may
>>> have to let them know about this reply.
>>>
>>> On Mon, Dec 15, 2014 at 3:48 PM, Jeff Hammond <jeff.science at gmail.com>
>>> wrote:
>>>
>>>>
>>>> Hi Hari,
>>>>
>>>> Thanks for the tip.  We can try this.
>>>>
>>>> Do you think it might be necessary to increase the eager-rendezvous
>>>> cutoff to ameliorate the performance loss associated with not caching
>>>> registration?  On a related note, does this disable the registration
>>>> cache for allocation via MPI_Alloc_mem as well as malloc?  Because we
>>>> might be able to use MPI_Alloc_mem for at least some of our
>>>> communication buffers to reduce the registration penalty whenever
>>>> possible.  Furthermore, can we increase use of RDMA without the
>>>> registration cache if we use RMA instead of two-sided?
>>>>
>>>> Is there any chance that a future version of MVAPICH could enable
>>>> --disable-registration-cache as a runtime option instead?  I build
>>>> MVAPICH from source regularly, but this is certainly not the norm.
>>>> The users of MADNESS will almost certainly count on their local
>>>> sysadmin to install MPI for them.
>>>>
>>>> Obviously, I know changing build-time options to runtime-options is
>>>> not trivial, so I'm not expecting this to happen overnight.
>>>>
>>>> Thanks,
>>>>
>>>> Jeff
>>>>
>>>> On Sun, Dec 14, 2014 at 4:08 PM, Hari Subramoni <subramoni.1 at osu.edu>
>>>> wrote:
>>>>
>>>>>
>>>>> Hello Jeff/MADNESS team,
>>>>>
>>>>> Sorry to hear that you are having issues compiling your application
>>>>> with
>>>>> MVAPICH2. We are seeing this issue for the first time. Nobody had
>>>>> posted
>>>>> anything along this line to mvapich-discuss earlier.
>>>>>
>>>>> Could you please try configuring MVAPICH2 without registration cache
>>>>>
>>>>
>>>> using
>>>>
>>>>>
>>>>> the "-disable-registration-cache" switch at configure time? This should
>>>>> prevent PTMalloc related issues that you are encountering. Please let
>>>>> us
>>>>> know if this helps. The following section of the userguide has more
>>>>> information about how to configure MVPAICH2
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>> http://mvapich.cse.ohio-state.edu/static/media/mvapich/mvapich2-2.1a-userguide.html#x1-110004.4
>>>>
>>>>>
>>>>> Please let us know if you encounter any other performance or
>>>>>
>>>>
>>>> functionality
>>>>
>>>>>
>>>>> issues in using your application with MVAPICH2 and we will be glad to
>>>>>
>>>>
>>>> help.
>>>>
>>>>>
>>>>> Regards,
>>>>> Hari.
>>>>>
>>>>> On Sun, Dec 14, 2014 at 5:35 PM, Jeff Hammond <jeff.science at gmail.com>
>>>>> wrote:
>>>>>
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> The MADNESS team (copied) has encountered an issue with MVAPICH2 and
>>>>>> Google TCMalloc, which is not the first time this has happened (see
>>>>>> GraphLab:
>>>>>>
>>>>>>
>>>>
>>>>
>>>> http://forum.graphlab.com/discussion/51/unable-to-compile-graphlab-analytics-with-mpi
>>>> ).
>>>>
>>>>>>
>>>>>> I don't have the details, but the nature of the issue seems pretty
>>>>>> obvious from the GraphLab explanation.
>>>>>>
>>>>>> TCMalloc improves the performance of MADNESS in a nontrivial way,
>>>>>> hence it is a very bad user experience for us if MVAPICH2 does not
>>>>>> allow us to use it.  We can, of course, disable TCMalloc, but since
>>>>>> small and non-small object allocation from C++ on many threads is
>>>>>> something MADNESS does constantly, PTMalloc is going to be suboptimal,
>>>>>> since it is known to not handle this usage well.
>>>>>>
>>>>>> What is the workaround for this and is there a long-term solution that
>>>>>> will allow MVAPICH2 to interoperate with memory allocators other than
>>>>>> ptmalloc?  It is particularly important to have MVAPICH2 interoperate
>>>>>> with JEMalloc in the future (see
>>>>>> https://github.com/memkind/memkind/blob/master/README for clues as to
>>>>>> why).
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> Jeff
>>>>>>
>>>>>> --
>>>>>> Jeff Hammond
>>>>>> jeff.science at gmail.com
>>>>>> http://jeffhammond.github.io/
>>>>>> _______________________________________________
>>>>>> mvapich-discuss mailing list
>>>>>> mvapich-discuss at cse.ohio-state.edu
>>>>>> http://mailman.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss
>>>>>>
>>>>
>>>>
>>>> --
>>>> Jeff Hammond
>>>> jeff.science at gmail.com
>>>> http://jeffhammond.github.io/
>>>>
>>>>
>>
>>
>>
>> ------------------------------------------------------------------------
>>
>> _______________________________________________
>> mvapich-discuss mailing list
>> mvapich-discuss at cse.ohio-state.edu
>> http://mailman.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss
>>
>
>

-- 
Jeff Hammond
jeff.science at gmail.com
http://jeffhammond.github.io/