[mvapich-discuss] tcmalloc issues

Tue Dec 16 12:29:07 EST 2014

Dear Jeff,

It looks like this is more of a runtime issue than a compile time issue.
For this, don't need to disable registration cache at configure time. The
environment variable you are currently using is fine.

Is it possible for us to get a reproducer for the issue so that we can
debug it locally?

Regards,
Hari.

On Tue, Dec 16, 2014 at 10:54 AM, Jeff Hammond <jeff.science at gmail.com>
wrote:
>
> This is the issue we see. Let me know if there's a good way to debug.
>
> Jeff
>
> Begin forwarded message:
>
> *From:* Justus Calvin <>
> *Date:* December 16, 2014 at 4:59:59 AM PST
> *To:* madness-developers at googlegroups.com
> *Subject:* *Re: [madness-developers: 3845] Fwd: [mvapich-discuss]
> tcmalloc issues*
> *Reply-To:* madness-developers at googlegroups.com
>
> I found that it was necessary to use MV2_USE_LAZY_MEM_UNREGISTER=0 when
> running my code on our small cluster. The problem that lead us to disable
> this feature was a consistent, but random, assertion failure. The failure
> that I saw was:
>
>
>
> Assertion failed in file
> ../src/mpid/ch3/channels/common/src/reg_cache/dreg.c at line 904:
> d->is_valid == 0
> [cli_5]: aborting job:
> internal ABORT - process 5
>
>
> Sent from my iPhone
>
> On Dec 15, 2014, at 7:53 PM, Hari Subramoni <subramoni.1 at osu.edu> wrote:
>
> Dear Jeff,
>
> Whether disabling registration cache will have a negative effect on
> communication performance depends entirely on the communication pattern of
> the application. If the application uses mostly small to medium sized
> messages (say < 16KB), then disabling registration cache will mostly have
> no impact on the application. However, if the application uses messages of
> larger size, then there might be an impact depending on the frequency of
> communication. If this is the case, then it might be useful to increase the
> VBUF (MV2_VBUF_TOTAL_SIZE) size and eager-rendezvous threshold
> (MV2_IBA_EAGER_THRESHOLD) to a larger value. In this scenario, we recommend
> that you set both to the same value (possibly slightly greater than the
> median message size being used by your application). Please refer to the
> following sections of the userguide for more information about these two
> parameters.
>
>
> http://mvapich.cse.ohio-state.edu/static/media/mvapich/mvapich2-2.1a-userguide.html#x1-26000011.105
>
> http://mvapich.cse.ohio-state.edu/static/media/mvapich/mvapich2-2.1a-userguide.html#x1-17800011.23
>
> MVAPICH2 has the capability to disable registration cache support at
> configure time (using --disable-registration-cache as I mentioned) and
> also at runtime by setting MV2_USE_LAZY_MEM_UNREGISTER=0. From your
> original e-mail, I understood that you were having a compile/link time
> issue with the MADNESS code. That is why I suggested disabling registration
> cache at configure time. If the issue you were facing is at runtime, then
> you can just use the runtime parameter mentioned above to disable
> registration cache at runtime. Please refer to the following sections of
> the userguide for more information about this parameters.
>
>
> http://mvapich.cse.ohio-state.edu/static/media/mvapich/mvapich2-2.1a-userguide.html#x1-23700011.82
>
> I am looking into the RMA / MPI_Alloc_mem question you posed. My initial
> impression is that it will work even if registration cache is disabled.
> However, I will double check on that and get back to you soon.
>
> Regards,
> Hari.
>
> PS: My e-mails to the madness-developers groups is bouncing. So you may
> have to let them know about this reply.
>
> On Mon, Dec 15, 2014 at 3:48 PM, Jeff Hammond <jeff.science at gmail.com>
> wrote:
>>
>> Hi Hari,
>>
>> Thanks for the tip.  We can try this.
>>
>> Do you think it might be necessary to increase the eager-rendezvous
>> cutoff to ameliorate the performance loss associated with not caching
>> registration?  On a related note, does this disable the registration
>> cache for allocation via MPI_Alloc_mem as well as malloc?  Because we
>> might be able to use MPI_Alloc_mem for at least some of our
>> communication buffers to reduce the registration penalty whenever
>> possible.  Furthermore, can we increase use of RDMA without the
>> registration cache if we use RMA instead of two-sided?
>>
>> Is there any chance that a future version of MVAPICH could enable
>> --disable-registration-cache as a runtime option instead?  I build
>> MVAPICH from source regularly, but this is certainly not the norm.
>> The users of MADNESS will almost certainly count on their local
>> sysadmin to install MPI for them.
>>
>> Obviously, I know changing build-time options to runtime-options is
>> not trivial, so I'm not expecting this to happen overnight.
>>
>> Thanks,
>>
>> Jeff
>>
>> On Sun, Dec 14, 2014 at 4:08 PM, Hari Subramoni <subramoni.1 at osu.edu>
>> wrote:
>> > Hello Jeff/MADNESS team,
>> >
>> > Sorry to hear that you are having issues compiling your application with
>> > MVAPICH2. We are seeing this issue for the first time. Nobody had posted
>> > anything along this line to mvapich-discuss earlier.
>> >
>> > Could you please try configuring MVAPICH2 without registration cache
>> using
>> > the "-disable-registration-cache" switch at configure time? This should
>> > prevent PTMalloc related issues that you are encountering. Please let us
>> > know if this helps. The following section of the userguide has more
>> > information about how to configure MVPAICH2
>> >
>> >
>> http://mvapich.cse.ohio-state.edu/static/media/mvapich/mvapich2-2.1a-userguide.html#x1-110004.4
>> >
>> > Please let us know if you encounter any other performance or
>> functionality
>> > issues in using your application with MVAPICH2 and we will be glad to
>> help.
>> >
>> > Regards,
>> > Hari.
>> >
>> > On Sun, Dec 14, 2014 at 5:35 PM, Jeff Hammond <jeff.science at gmail.com>
>> > wrote:
>> >>
>> >> Hi,
>> >>
>> >> The MADNESS team (copied) has encountered an issue with MVAPICH2 and
>> >> Google TCMalloc, which is not the first time this has happened (see
>> >> GraphLab:
>> >>
>> http://forum.graphlab.com/discussion/51/unable-to-compile-graphlab-analytics-with-mpi
>> ).
>> >> I don't have the details, but the nature of the issue seems pretty
>> >> obvious from the GraphLab explanation.
>> >>
>> >> TCMalloc improves the performance of MADNESS in a nontrivial way,
>> >> hence it is a very bad user experience for us if MVAPICH2 does not
>> >> allow us to use it.  We can, of course, disable TCMalloc, but since
>> >> small and non-small object allocation from C++ on many threads is
>> >> something MADNESS does constantly, PTMalloc is going to be suboptimal,
>> >> since it is known to not handle this usage well.
>> >>
>> >> What is the workaround for this and is there a long-term solution that
>> >> will allow MVAPICH2 to interoperate with memory allocators other than
>> >> ptmalloc?  It is particularly important to have MVAPICH2 interoperate
>> >> with JEMalloc in the future (see
>> >> https://github.com/memkind/memkind/blob/master/README for clues as to
>> >> why).
>> >>
>> >> Thanks,
>> >>
>> >> Jeff
>> >>
>> >> --
>> >> Jeff Hammond
>> >> jeff.science at gmail.com
>> >> http://jeffhammond.github.io/
>> >> _______________________________________________
>> >> mvapich-discuss mailing list
>> >> mvapich-discuss at cse.ohio-state.edu
>> >> http://mailman.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss
>>
>>
>>
>> --
>> Jeff Hammond
>> jeff.science at gmail.com
>> http://jeffhammond.github.io/
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.cse.ohio-state.edu/pipermail/mvapich-discuss/attachments/20141216/4a2f1d04/attachment.html>