[mvapich-discuss] Number of IB QP's

[Lei Chai]

Hi Lars,

I need to clarify the number of QP's used for automatic path migration 
(APM). My answer below (4 QP's per port and 2 QP's per port) actually 
referred to the hot-spot avoidance feature (MV2_USE_HSAM=1) in mvapich2. 
APM doesn't need additional QP's. Sorry if this has caused confusion.

Lei


Lei Chai wrote:
> Hi Lars,
>
> My answers are inline:
>
>> How many QP's will be used for a large application run with MVAPICH / 
>> MVAPICH2 1.x ? aka' given an application running all-to-all traffic 
>> of random size on <P> MPI processes  (= #cores) behind each HCA on 
>> <N> nodes/HCA's (i.e. a total of N*P application processes) will use 
>> <Q1> QP's on rank 0 and <Q2> QP's on the rest...
>
> If it's all-to-all and the number of processes>64 (on-demand 
> connection model is used),  then each process will setup (N-1)*P QP's. 
> Basically each process will connect to all the other processes except 
> those that are on the same node, and each connection by default uses 
> one QP.
>
>> Will Automatic path migration affect the number of QP's used ?
>
> Yes. It will setup 4 QP's per port for each connection, if the number 
> of processes is in the range of 8-64, and 2 QP's per port for each 
> connection otherwise.
>
>> Will using the MVAPICH2 multi-rail feature affect number of QP's used ?
>
> Yes, each rail will need to setup a QP.
>
>> Other factors / features that will affect the number of QP's ?
> You can use env variables to modify the number of QP's for each 
> connection. And if you are using the XRC mode in mvapich-1.1 then you 
> will need to setup less number of QP's. The details are discussed in 
> the following paper:
>
> http://nowlab.cse.ohio-state.edu/publications/conf-papers/2008/koop-cluster08.pdf 
>
>
> Hope this helps.
>
> Lei
>
>
>> /lars paul
>>
>>
>> _______________________________________________
>> mvapich-discuss mailing list
>> mvapich-discuss at cse.ohio-state.edu
>> http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss
>
> _______________________________________________
> mvapich-discuss mailing list
> mvapich-discuss at cse.ohio-state.edu
> http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss