[mvapich-discuss] Number of IB QP's

[Matthew Koop]

> So for a N=4097 node system with P=16 processes per node (i.e. a total
> of 65552 MPI processes - not completely unlike a full TACC Ranger run
> utilizing all cores) the number of QP's per node with or without APM
> will be :
>
> QPpN = (N-1)*P*P = 4k*16*16 = 1M

For RC, yes, this is the total number of QPs per node. Per process
there will be (N-1)P QPs, or 64K QPs.

Instead if the XRC transport, included in ConnectX, is used this number
can potentially be (N-1) QPs per process, so less by a factor of P. The
paper that Lei sent has some additional details.

We also have a new hybrid device for MVAPICH, which allows either UD-RC or
UD-XRC operation. Since UD QPs are not connection-oriented, there can be
significantly less numbers of QPs needed in the system. Then for
high-volume connections a RC or XRC connection can be used. Additional
details:

http://nowlab.cse.ohio-state.edu/publications/conf-papers/2008/koop-ipdps08.pdf

Hope this helps,

Matt

> /lars paul
>
> On 10/14/08 07:45 PM, Lei Chai wrote:
> > Hi Lars,
> >
> > I need to clarify the number of QP's used for automatic path migration
> > (APM). My answer below (4 QP's per port and 2 QP's per port) actually
> > referred to the hot-spot avoidance feature (MV2_USE_HSAM=1) in mvapich2.
> > APM doesn't need additional QP's. Sorry if this has caused confusion.
> >
> > Lei
> >
> >
> > Lei Chai wrote:
> >> Hi Lars,
> >>
> >> My answers are inline:
> >>
> >>> How many QP's will be used for a large application run with MVAPICH /
> >>> MVAPICH2 1.x ? aka' given an application running all-to-all traffic
> >>> of random size on <P> MPI processes  (= #cores) behind each HCA on
> >>> <N> nodes/HCA's (i.e. a total of N*P application processes) will use
> >>> <Q1> QP's on rank 0 and <Q2> QP's on the rest...
> >>
> >> If it's all-to-all and the number of processes>64 (on-demand
> >> connection model is used),  then each process will setup (N-1)*P QP's.
> >> Basically each process will connect to all the other processes except
> >> those that are on the same node, and each connection by default uses
> >> one QP.
> >>
> >>> Will Automatic path migration affect the number of QP's used ?
> >>
> >> Yes. It will setup 4 QP's per port for each connection, if the number
> >> of processes is in the range of 8-64, and 2 QP's per port for each
> >> connection otherwise.
> >>
> >>> Will using the MVAPICH2 multi-rail feature affect number of QP's used ?
> >>
> >> Yes, each rail will need to setup a QP.
> >>
> >>> Other factors / features that will affect the number of QP's ?
> >> You can use env variables to modify the number of QP's for each
> >> connection. And if you are using the XRC mode in mvapich-1.1 then you
> >> will need to setup less number of QP's. The details are discussed in
> >> the following paper:
> >>
> >> http://nowlab.cse.ohio-state.edu/publications/conf-papers/2008/koop-cluster08.pdf
> >>
> >>
> >> Hope this helps.
> >>
> >> Lei
> >>
> >>
> >>> /lars paul
> >>>
> >>>
> >>> _______________________________________________
> >>> mvapich-discuss mailing list
> >>> mvapich-discuss at cse.ohio-state.edu
> >>> http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss
> >>
> >> _______________________________________________
> >> mvapich-discuss mailing list
> >> mvapich-discuss at cse.ohio-state.edu
> >> http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss
> >
>
> _______________________________________________
> mvapich-discuss mailing list
> mvapich-discuss at cse.ohio-state.edu
> http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss
>