[mvapich-discuss] regestration cache limits in mvapich 0.9.7

[Pavel Shamis (Pasha)]

Hello Sayantan,

Thank you for full dreg.* stuff review.

Actually the check #3 can be removed, you have the same check in #6 , is 
not it?


Some other issue.

ib_init() include follow #ifdef (viainit.c):
-----------------------------------------------------------------
#ifndef _PPC64_
         attr.cap.max_inline_data = viadev_max_inline_size;
#else
         attr.cap.max_inline_data = -1;
#endif
-----------------------------------------------------------------

max_inline_data is uint32_t and mthca_create_qp 
(src/userspace/libmthca/src/verbs.c) include:
-----------------------------------------------------------------
if (attr->cap.max_send_wr     > 65536 ||
             attr->cap.max_recv_wr     > 65536 ||
             attr->cap.max_send_sge    > 64    ||
             attr->cap.max_recv_sge    > 64    ||
             attr->cap.max_inline_data > 1024)
                 return NULL;
-----------------------------------------------------------------

And as result the code does not run on PPC64
Is it some reason for this ifdef ?

Regards,
Pavel Shamis (Pasha)



Sayantan Sur wrote:
> Hello Pasha,
> 
> Pavel Shamis (Pasha) wrote:
> 
>> Hi,
>> In mvapich-0.9.7 you have viadev_dreg_cache_limit limit that should 
>> limit the size of registration cache. From review in dreg.* stuff I see
>> that actually this parameter limits only the maximum number of pages 
>> per _single_ buffer registration (see dreg_new_entry()) and not total 
>> number of registered pages in the cache.  Can you please review this 
>> point ? Because I'm not sure that we want limit # of pages per single 
>> buffer registration.
> 
> Thanks for looking at the code in detail. Here's my explanation of the 
> code:
> 
> 1) dreg cache interface is called using dreg_register()
> 2) If no cache entry is found, dreg_new_entry() is called
> 3) dreg_new_entry() checks if the single registration size itself will 
> violate the cache limit. It doesn't allow a single registration to go 
> past the user defined limit.
> 4) If it is found that the single registration is below the user defined 
> limit, then the flow proceeds to dreg_insert()
> 5) In dreg_insert(), we call vma_new() to insert a new virtual memory 
> region
> 6) In vma_new() the number of pages requested is added to a variable 
> called `pinned_pages_count'. If this pinned_pages_count exceeds the user 
> set limit `viadev_dreg_cache_limit', then NULL is returned and this is 
> propagated to the upper layer to report a failed registration. If not, 
> then the `pinned_pages_count' variable is incremented accordingly.
> 7) `pinned_pages_count' is decremented when a memory area is removed 
> using vma_remove()
> 
> I hope this helps.
> 
> Thanks,
> Sayantan.
> 
>>
>> Thanks,
>> Pavel Shamis (Pasha)
>> _______________________________________________
>> mvapich-discuss mailing list
>> mvapich-discuss at cse.ohio-state.edu
>> http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss
> 
> 
>