[mvapich-discuss] Number of IB QP's

[Lars Paul Huse]

Hi Lei,

Thanks for the reply and the pointer :-)

So for a N=4097 node system with P=16 processes per node (i.e. a total 
of 65552 MPI processes - not completely unlike a full TACC Ranger run 
utilizing all cores) the number of QP's per node with or without APM 
will be :

QPpN = (N-1)*P*P = 4k*16*16 = 1M

/lars paul

On 10/14/08 07:45 PM, Lei Chai wrote:
> Hi Lars,
> 
> I need to clarify the number of QP's used for automatic path migration 
> (APM). My answer below (4 QP's per port and 2 QP's per port) actually 
> referred to the hot-spot avoidance feature (MV2_USE_HSAM=1) in mvapich2. 
> APM doesn't need additional QP's. Sorry if this has caused confusion.
> 
> Lei
> 
> 
> Lei Chai wrote:
>> Hi Lars,
>>
>> My answers are inline:
>>
>>> How many QP's will be used for a large application run with MVAPICH / 
>>> MVAPICH2 1.x ? aka' given an application running all-to-all traffic 
>>> of random size on <P> MPI processes  (= #cores) behind each HCA on 
>>> <N> nodes/HCA's (i.e. a total of N*P application processes) will use 
>>> <Q1> QP's on rank 0 and <Q2> QP's on the rest...
>>
>> If it's all-to-all and the number of processes>64 (on-demand 
>> connection model is used),  then each process will setup (N-1)*P QP's. 
>> Basically each process will connect to all the other processes except 
>> those that are on the same node, and each connection by default uses 
>> one QP.
>>
>>> Will Automatic path migration affect the number of QP's used ?
>>
>> Yes. It will setup 4 QP's per port for each connection, if the number 
>> of processes is in the range of 8-64, and 2 QP's per port for each 
>> connection otherwise.
>>
>>> Will using the MVAPICH2 multi-rail feature affect number of QP's used ?
>>
>> Yes, each rail will need to setup a QP.
>>
>>> Other factors / features that will affect the number of QP's ?
>> You can use env variables to modify the number of QP's for each 
>> connection. And if you are using the XRC mode in mvapich-1.1 then you 
>> will need to setup less number of QP's. The details are discussed in 
>> the following paper:
>>
>> http://nowlab.cse.ohio-state.edu/publications/conf-papers/2008/koop-cluster08.pdf 
>>
>>
>> Hope this helps.
>>
>> Lei
>>
>>
>>> /lars paul
>>>
>>>
>>> _______________________________________________
>>> mvapich-discuss mailing list
>>> mvapich-discuss at cse.ohio-state.edu
>>> http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss
>>
>> _______________________________________________
>> mvapich-discuss mailing list
>> mvapich-discuss at cse.ohio-state.edu
>> http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss
>