[mvapich-discuss] Tuning for bandwidth

[Hari Subramoni]

Hi,

The unit for MV2_VBUF_TOTAL_SIZE & MV2_IBA_EAGER_THRESHOLD is bytes.

Idealy, you would want to set

MV2_IBA_EAGER_THRESHOLD = MV2_VBUF_TOTAL_SIZE - 4 bytes.

Thx,
Hari.

On Wed, 14 Jul 2010, TJC Ward wrote:

> I'm trying to tune mvapich2 1.5 to exploit as much of the bisection
> bandwidth as possible of a cluster.
>
> An initial target is 512 nodes, with a 1MB 'MPI_Alltoall' (i.e. a total
> data size of 512MB on each node).
>
> Timer-tick profiling shows that the core on each node running the mvapich
> application is 100% busy. It looks as if data is being divided up into
> small chunks and handed off to the underlying layers in small units; in
> such a fashion that if mvapich were to hand off the whole transfer
> instruction to the RDMA provider in one go, and then wait for the whole
> thing to complete, the operation would run considerably faster.
>
> I'm looking at the mvapich2 user manual and the environment variables,
> particularly at
> MV2_DEFAULT_MAX_SEND_WQE
> MV2_DEFAULT_MAX_RECV_WQE
> MV2_VBUF_TOTAL_SIZE
> MPIR_ALLTOALL_MEDIUM_MSG
> to try to understand what might be helpful to change. Am I looking at the
> right place, and do you have recommended values for this data transfer
> pattern ?
>
> The user manual doesn't say what units MV2_VBUF_TOTAL_SIZE is measured in;
> "bytes", maybe. it's also not clear about MV2_IBA_EAGER_THRESHOLD and the
> circumstances where you would want it the same as, or different from,
> MV2_VBUF_TOTAL_SIZE .
>
> All help and advice appreciated.
>
> T J (Chris) Ward, IBM Research
> Scalable Data-Centric Computing - Active Storage Fabrics - IBM System
> BlueGene
> IBM United Kingdom Ltd., Hursley Park, Winchester, Hants, SO21 2JN
> 011-44-1962-818679
> IBM Intranet http://hurgsa.ibm.com/~tjcw/
>
> IBM System BlueGene Research
> IBM System BlueGene Marketing
>
> IBM Resources for Global Servants
> IBM Branded Products IBM Branded Swag
>
>
> UNIX in the Cloud - Find A Place Where There's Room To Grow, with the
> original Open Standard. Free Trial Here Today
> New Lamps For Old - Diskless Remote Boot Linux from National Center for
> High-Performance Computing, Taiwan
>