[mvapich-discuss] -DDISABLE_PTMALLOC, MPI_Bcast, and -DMCST_SUPPORT

[David_Kewley at Dell.com]

Topic #1:

What are the likely performance impacts of using -DDISABLE_PTMALLOC
(including memory use)?  Does this differ between MVAPICH and MVAPICH2?
We are considering seeing what effect this has on certain applications
that have seen problems with realloc.

Topic #2:

We are using the OpenIB components of OFED 1.2.5.5, and are building our
own MVAPICH and MVAPICH2, with various versions of MV* and compiler.

We have an application apparently failing during MVAPICH MPI_Bcast of a
many MB of data to dozens to hundreds of MPI ranks.  (Actually I believe
it's Fortran, so I guess MPI_BCAST.)  We have already implemented
VIADEV_USE_SHMEM_BCAST=0 just in case, but we are still having problems.
(I'm not 100% reassured by the user's reports that the problem is still
in MPI_Bcast, but I think it's likely.)

If you have any thoughts on the crashes, please do comment.  Meanwhile
I'm continuing to chase it down locally.

Topic #3:

As I looked through the MVAPICH code to see how MPI_Bcast is implemented
for ch_gen2, I see MCST_SUPPORT repeatedly checked.  It appears this is
not set by default (by make.mvapich.gen2).

If MCST_SUPPORT is disabled, what algorithm is used to implement
MPI_Bcast?  If MCST_SUPPORT is enabled, does MPI_Bcast use IB multicast?
Should it greatly speed up MPI_Bcast if enabled?

It seems like MCST_SUPPORT would be beneficial, but the fact that it is
not enabled by default makes me wonder what the risks are of enabling
it?

Thanks,
David

David Kewley
Dell Infrastructure Consulting Services
Onsite Engineer at the Maui HPC Center
Cell: 602-460-7617
David_Kewley at Dell.com

Dell Services: http://www.dell.com/services/
How am I doing? Email my manager Russell_Kelly at Dell.com with any
feedback.