[mvapich-discuss] CM and listening ports

[Jonathan Perkins]

On Thu, Jul 22, 2010 at 7:37 AM, TJC Ward <tjcw at us.ibm.com> wrote:

> I've made significant progress with scale-up of connection establishment.
>
> In drivers/infiniband/core/ucma.c (linux kernel), there's a setting
> enum {
>         UCMA_MAX_BACKLOG        = 128
> };
> On my BlueGene Linux system, with 1024 compute nodes and
> MV2_ON_DEMAND_THRESHOLD set 'high', I was getting cases where this was being
> overflowed, and ucma_event_handler was (misleadingly) returning with ENOMEM.
>
> I've changed UCMA_MAX_BACKLOG        to 8192; this seems to resolve the
> problem.
>
> Other things I'm setting are
> MV2_RDMA_CM_PORT=1026  # to prevent EADDRINUSE-type collisions
> setting 'backlog' high in the 'listen' syscall
> in src/include mpiimpl.h, setting
> #define MPIR_ALLTOALL_MEDIUM_MSG      8388608   // because the 'long
> message' support in mvapich uses the network inefficiently, on machines with
> mesh networks.
>
> Is anyone on the mailing list in a position to feed back the
> UCMA_MAX_BACKLOG item to the maintainers of Infiniband support in the Linux
> kernel ? My workaround probably isn't the ideal solution ... and may not
> even be sufficient as clusters grow larger ... but might be a starting point
> for a better solution.
>
>
Hi, you may want to consider posting a summary of your findings at the
linux-rdma at vger.kernel.org mailing list.  The maintainers follow this list.
They should be able to provide feedback on your findings as well as take a
look at the misleading ENOMEM message that gets generated.

-- 
Jonathan Perkins
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.cse.ohio-state.edu/pipermail/mvapich-discuss/attachments/20100722/ccbc23b3/attachment.html