[mvapich-discuss] Number of IB QP's

[Lei Chai]

Hi Lars,

My answers are inline:

> How many QP's will be used for a large application run with MVAPICH / 
> MVAPICH2 1.x ? aka' given an application running all-to-all traffic of 
> random size on <P> MPI processes  (= #cores) behind each HCA on <N> 
> nodes/HCA's (i.e. a total of N*P application processes) will use <Q1> 
> QP's on rank 0 and <Q2> QP's on the rest...

If it's all-to-all and the number of processes>64 (on-demand connection 
model is used),  then each process will setup (N-1)*P QP's. Basically 
each process will connect to all the other processes except those that 
are on the same node, and each connection by default uses one QP.

> Will Automatic path migration affect the number of QP's used ?

Yes. It will setup 4 QP's per port for each connection, if the number of 
processes is in the range of 8-64, and 2 QP's per port for each 
connection otherwise.

> Will using the MVAPICH2 multi-rail feature affect number of QP's used ?

Yes, each rail will need to setup a QP.

> Other factors / features that will affect the number of QP's ?
You can use env variables to modify the number of QP's for each 
connection. And if you are using the XRC mode in mvapich-1.1 then you 
will need to setup less number of QP's. The details are discussed in the 
following paper:

http://nowlab.cse.ohio-state.edu/publications/conf-papers/2008/koop-cluster08.pdf

Hope this helps.

Lei


> /lars paul
>
>
> _______________________________________________
> mvapich-discuss mailing list
> mvapich-discuss at cse.ohio-state.edu
> http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss