[mvapich-discuss] tcmalloc issues

[Jeff Hammond]

Hi Hari,

Thanks for the tip.  We can try this.

Do you think it might be necessary to increase the eager-rendezvous
cutoff to ameliorate the performance loss associated with not caching
registration?  On a related note, does this disable the registration
cache for allocation via MPI_Alloc_mem as well as malloc?  Because we
might be able to use MPI_Alloc_mem for at least some of our
communication buffers to reduce the registration penalty whenever
possible.  Furthermore, can we increase use of RDMA without the
registration cache if we use RMA instead of two-sided?

Is there any chance that a future version of MVAPICH could enable
--disable-registration-cache as a runtime option instead?  I build
MVAPICH from source regularly, but this is certainly not the norm.
The users of MADNESS will almost certainly count on their local
sysadmin to install MPI for them.

Obviously, I know changing build-time options to runtime-options is
not trivial, so I'm not expecting this to happen overnight.

Thanks,

Jeff

On Sun, Dec 14, 2014 at 4:08 PM, Hari Subramoni <subramoni.1 at osu.edu> wrote:
> Hello Jeff/MADNESS team,
>
> Sorry to hear that you are having issues compiling your application with
> MVAPICH2. We are seeing this issue for the first time. Nobody had posted
> anything along this line to mvapich-discuss earlier.
>
> Could you please try configuring MVAPICH2 without registration cache using
> the "-disable-registration-cache" switch at configure time? This should
> prevent PTMalloc related issues that you are encountering. Please let us
> know if this helps. The following section of the userguide has more
> information about how to configure MVPAICH2
>
> http://mvapich.cse.ohio-state.edu/static/media/mvapich/mvapich2-2.1a-userguide.html#x1-110004.4
>
> Please let us know if you encounter any other performance or functionality
> issues in using your application with MVAPICH2 and we will be glad to help.
>
> Regards,
> Hari.
>
> On Sun, Dec 14, 2014 at 5:35 PM, Jeff Hammond <jeff.science at gmail.com>
> wrote:
>>
>> Hi,
>>
>> The MADNESS team (copied) has encountered an issue with MVAPICH2 and
>> Google TCMalloc, which is not the first time this has happened (see
>> GraphLab:
>> http://forum.graphlab.com/discussion/51/unable-to-compile-graphlab-analytics-with-mpi).
>> I don't have the details, but the nature of the issue seems pretty
>> obvious from the GraphLab explanation.
>>
>> TCMalloc improves the performance of MADNESS in a nontrivial way,
>> hence it is a very bad user experience for us if MVAPICH2 does not
>> allow us to use it.  We can, of course, disable TCMalloc, but since
>> small and non-small object allocation from C++ on many threads is
>> something MADNESS does constantly, PTMalloc is going to be suboptimal,
>> since it is known to not handle this usage well.
>>
>> What is the workaround for this and is there a long-term solution that
>> will allow MVAPICH2 to interoperate with memory allocators other than
>> ptmalloc?  It is particularly important to have MVAPICH2 interoperate
>> with JEMalloc in the future (see
>> https://github.com/memkind/memkind/blob/master/README for clues as to
>> why).
>>
>> Thanks,
>>
>> Jeff
>>
>> --
>> Jeff Hammond
>> jeff.science at gmail.com
>> http://jeffhammond.github.io/
>> _______________________________________________
>> mvapich-discuss mailing list
>> mvapich-discuss at cse.ohio-state.edu
>> http://mailman.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss



-- 
Jeff Hammond
jeff.science at gmail.com
http://jeffhammond.github.io/