[mvapich-discuss] VIADEV_CLUSTER_SIZE

[Matthew Koop]

>     Explicitly, does your description mean that for ch_gen2 MVAPICH
>     the default value of VIADEV_VBUF_TOTAL_SIZE is automataically
>     adjusted based on job size w/o need for VIADEV_CLUSTER_SIZE
>     setting as currently described in the User Guide?

It does not do exactly the same thing since it is not required at this
point. When that parameter was in use buffers needed to be posted per QP
(essentially per peer) in InfiniBand. Thus, as the number of processes in
a job increased, the amount of memory for communication buffers required
increased.

Since then, InfiniBand released a Shared Send Queue (SRQ) to get around
this inherent issue and MVAPICH incorporated support for it. So no longer
are buffers posted peer peer, but as a shared pool instead. As such, the
buffer size does not make a large difference.  Additionally, we've added a
number of other changes to maintain scalability at large process counts.

I hope this helps. Let us know if you have additional questions,

Matt


>            regards,
>
> Matthew Koop wrote:
> > Mark,
> >
> > It depends on what device you are using. For the OpenFabrics/Gen2 device,
> > that parameter is not available since it is no longer needed. The library
> > automatically limits resources according to job size as required. Other
> > scalability features such as SRQ and on-demand connection setup remove
> > the need for that variable.
> >
> > We'll update that entry in the user guide to make it apparent that
> > variable is only used in the VAPI device.
> >
> > For any of the variables, they can be set either in the mvapich.conf file
> > or specified at runtime. e.g.
> >
> > mpirun_rsh -np 2 m0 m1 VIADEV_USE_SHARED_MEM=0 ./a.out
> >
> > Let us know if you have any other questions or if this didn't help.
> >
> > Thanks,
> >
> > Matt
> >
> > On Wed, 27 Jun 2007, Mark Potts wrote:
> >
> >> Hi,
> >>     I see a description of an attractive MVAPICH parameter in the 0.9.9
> >>     User Guide: VIADEV_CLUSTER_SIZE with possible values of SMALL,
> >>     MEDIUM, LARGE, and AUTO.  There is no matching default for this
> >>     parameter contained in the mvapich.conf file.
> >>
> >>     Is this parameter truly implemented, as an app run time parameter,
> >>     in MVAPICH 0.9.9 and if so how/where does its default value get
> >>     assigned?
> >>
> >>              regards,
> >> --
> >> ***********************************
> >>  >> Mark J. Potts, PhD
> >>  >>
> >>  >> HPC Applications Inc.
> >>  >> phone: 410-992-8360 Bus
> >>  >>        410-313-9318 Home
> >>  >>        443-418-4375 Cell
> >>  >> email: potts at hpcapplications.com
> >>  >>        potts at excray.com
> >> ***********************************
> >> _______________________________________________
> >> mvapich-discuss mailing list
> >> mvapich-discuss at cse.ohio-state.edu
> >> http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss
> >>
>
> --
> ***********************************
>  >> Mark J. Potts, PhD
>  >>
>  >> HPC Applications Inc.
>  >> phone: 410-992-8360 Bus
>  >>        410-313-9318 Home
>  >>        443-418-4375 Cell
>  >> email: potts at hpcapplications.com
>  >>        potts at excray.com
> ***********************************
>