[mvapich-discuss] tcmalloc issues

Mon Dec 15 22:53:07 EST 2014

Dear Jeff,

Whether disabling registration cache will have a negative effect on
communication performance depends entirely on the communication pattern of
the application. If the application uses mostly small to medium sized
messages (say < 16KB), then disabling registration cache will mostly have
no impact on the application. However, if the application uses messages of
larger size, then there might be an impact depending on the frequency of
communication. If this is the case, then it might be useful to increase the
VBUF (MV2_VBUF_TOTAL_SIZE) size and eager-rendezvous threshold
(MV2_IBA_EAGER_THRESHOLD) to a larger value. In this scenario, we recommend
that you set both to the same value (possibly slightly greater than the
median message size being used by your application). Please refer to the
following sections of the userguide for more information about these two
parameters.

http://mvapich.cse.ohio-state.edu/static/media/mvapich/mvapich2-2.1a-userguide.html#x1-26000011.105
http://mvapich.cse.ohio-state.edu/static/media/mvapich/mvapich2-2.1a-userguide.html#x1-17800011.23

MVAPICH2 has the capability to disable registration cache support at
configure time (using --disable-registration-cache as I mentioned) and also
at runtime by setting MV2_USE_LAZY_MEM_UNREGISTER=0. From your original
e-mail, I understood that you were having a compile/link time issue with
the MADNESS code. That is why I suggested disabling registration cache at
configure time. If the issue you were facing is at runtime, then you can
just use the runtime parameter mentioned above to disable registration
cache at runtime. Please refer to the following sections of the userguide
for more information about this parameters.

http://mvapich.cse.ohio-state.edu/static/media/mvapich/mvapich2-2.1a-userguide.html#x1-23700011.82

I am looking into the RMA / MPI_Alloc_mem question you posed. My initial
impression is that it will work even if registration cache is disabled.
However, I will double check on that and get back to you soon.

Regards,
Hari.

PS: My e-mails to the madness-developers groups is bouncing. So you may
have to let them know about this reply.

On Mon, Dec 15, 2014 at 3:48 PM, Jeff Hammond <jeff.science at gmail.com>
wrote:
>
> Hi Hari,
>
> Thanks for the tip.  We can try this.
>
> Do you think it might be necessary to increase the eager-rendezvous
> cutoff to ameliorate the performance loss associated with not caching
> registration?  On a related note, does this disable the registration
> cache for allocation via MPI_Alloc_mem as well as malloc?  Because we
> might be able to use MPI_Alloc_mem for at least some of our
> communication buffers to reduce the registration penalty whenever
> possible.  Furthermore, can we increase use of RDMA without the
> registration cache if we use RMA instead of two-sided?
>
> Is there any chance that a future version of MVAPICH could enable
> --disable-registration-cache as a runtime option instead?  I build
> MVAPICH from source regularly, but this is certainly not the norm.
> The users of MADNESS will almost certainly count on their local
> sysadmin to install MPI for them.
>
> Obviously, I know changing build-time options to runtime-options is
> not trivial, so I'm not expecting this to happen overnight.
>
> Thanks,
>
> Jeff
>
> On Sun, Dec 14, 2014 at 4:08 PM, Hari Subramoni <subramoni.1 at osu.edu>
> wrote:
> > Hello Jeff/MADNESS team,
> >
> > Sorry to hear that you are having issues compiling your application with
> > MVAPICH2. We are seeing this issue for the first time. Nobody had posted
> > anything along this line to mvapich-discuss earlier.
> >
> > Could you please try configuring MVAPICH2 without registration cache
> using
> > the "-disable-registration-cache" switch at configure time? This should
> > prevent PTMalloc related issues that you are encountering. Please let us
> > know if this helps. The following section of the userguide has more
> > information about how to configure MVPAICH2
> >
> >
> http://mvapich.cse.ohio-state.edu/static/media/mvapich/mvapich2-2.1a-userguide.html#x1-110004.4
> >
> > Please let us know if you encounter any other performance or
> functionality
> > issues in using your application with MVAPICH2 and we will be glad to
> help.
> >
> > Regards,
> > Hari.
> >
> > On Sun, Dec 14, 2014 at 5:35 PM, Jeff Hammond <jeff.science at gmail.com>
> > wrote:
> >>
> >> Hi,
> >>
> >> The MADNESS team (copied) has encountered an issue with MVAPICH2 and
> >> Google TCMalloc, which is not the first time this has happened (see
> >> GraphLab:
> >>
> http://forum.graphlab.com/discussion/51/unable-to-compile-graphlab-analytics-with-mpi
> ).
> >> I don't have the details, but the nature of the issue seems pretty
> >> obvious from the GraphLab explanation.
> >>
> >> TCMalloc improves the performance of MADNESS in a nontrivial way,
> >> hence it is a very bad user experience for us if MVAPICH2 does not
> >> allow us to use it.  We can, of course, disable TCMalloc, but since
> >> small and non-small object allocation from C++ on many threads is
> >> something MADNESS does constantly, PTMalloc is going to be suboptimal,
> >> since it is known to not handle this usage well.
> >>
> >> What is the workaround for this and is there a long-term solution that
> >> will allow MVAPICH2 to interoperate with memory allocators other than
> >> ptmalloc?  It is particularly important to have MVAPICH2 interoperate
> >> with JEMalloc in the future (see
> >> https://github.com/memkind/memkind/blob/master/README for clues as to
> >> why).
> >>
> >> Thanks,
> >>
> >> Jeff
> >>
> >> --
> >> Jeff Hammond
> >> jeff.science at gmail.com
> >> http://jeffhammond.github.io/
> >> _______________________________________________
> >> mvapich-discuss mailing list
> >> mvapich-discuss at cse.ohio-state.edu
> >> http://mailman.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss
>
>
>
> --
> Jeff Hammond
> jeff.science at gmail.com
> http://jeffhammond.github.io/
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.cse.ohio-state.edu/pipermail/mvapich-discuss/attachments/20141215/ca3bbbc4/attachment-0001.html>