[mvapich-discuss] regestration cache limits in mvapich 0.9.7

[Pavel Shamis (Pasha)]

Hello,

Thank you for the fix, we will check it.

 >Actually the check #3 can be removed, you have the same check in #6 ,
 >is not it?

What do you think about this point (it is from previous email in this 
thread) ?

Thanks,
Pasha

Abhinav Vishnu wrote:
> Hi Pasha,
> 
>> ib_init() include follow #ifdef (viainit.c):
>> -----------------------------------------------------------------
>> #ifndef _PPC64_
>>          attr.cap.max_inline_data = viadev_max_inline_size;
>> #else
>>          attr.cap.max_inline_data = -1;
>> #endif
>> -----------------------------------------------------------------
>>
>> max_inline_data is uint32_t and mthca_create_qp
>> (src/userspace/libmthca/src/verbs.c) include:
>> -----------------------------------------------------------------
>> if (attr->cap.max_send_wr     > 65536 ||
>>              attr->cap.max_recv_wr     > 65536 ||
>>              attr->cap.max_send_sge    > 64    ||
>>              attr->cap.max_recv_sge    > 64    ||
>>              attr->cap.max_inline_data > 1024)
>>                  return NULL;
>> -----------------------------------------------------------------
>>
>> And as result the code does not run on PPC64
>> Is it some reason for this ifdef ?
>>
> 
> Thanks for pointing out this problem to us. Since the IBM EHCA does not
> support inline data transfer, we defined the max inline size to be -1.
> However as you have pointed out, this solution does not seem to be
> completely correct.
> 
> We have taken care of this issue by allowing the messages over
> IBM EHCA to use only IBV_SEND_SIGNALED flag. This change has been checked
> in the SVN, both for single-rail and multi-rail devices. Please let you
> know if the solution helps solve your problem.
> 
> Thanks and regards,
> 
> -- Abhinav
> 
>> Regards,
>> Pavel Shamis (Pasha)
>>
>>
>>
>> Sayantan Sur wrote:
>>> Hello Pasha,
>>>
>>> Pavel Shamis (Pasha) wrote:
>>>
>>>> Hi,
>>>> In mvapich-0.9.7 you have viadev_dreg_cache_limit limit that should
>>>> limit the size of registration cache. From review in dreg.* stuff I see
>>>> that actually this parameter limits only the maximum number of pages
>>>> per _single_ buffer registration (see dreg_new_entry()) and not total
>>>> number of registered pages in the cache.  Can you please review this
>>>> point ? Because I'm not sure that we want limit # of pages per single
>>>> buffer registration.
>>> Thanks for looking at the code in detail. Here's my explanation of the
>>> code:
>>>
>>> 1) dreg cache interface is called using dreg_register()
>>> 2) If no cache entry is found, dreg_new_entry() is called
>>> 3) dreg_new_entry() checks if the single registration size itself will
>>> violate the cache limit. It doesn't allow a single registration to go
>>> past the user defined limit.
>>> 4) If it is found that the single registration is below the user defined
>>> limit, then the flow proceeds to dreg_insert()
>>> 5) In dreg_insert(), we call vma_new() to insert a new virtual memory
>>> region
>>> 6) In vma_new() the number of pages requested is added to a variable
>>> called `pinned_pages_count'. If this pinned_pages_count exceeds the user
>>> set limit `viadev_dreg_cache_limit', then NULL is returned and this is
>>> propagated to the upper layer to report a failed registration. If not,
>>> then the `pinned_pages_count' variable is incremented accordingly.
>>> 7) `pinned_pages_count' is decremented when a memory area is removed
>>> using vma_remove()
>>>
>>> I hope this helps.
>>>
>>> Thanks,
>>> Sayantan.
>>>
>>>> Thanks,
>>>> Pavel Shamis (Pasha)
>>>> _______________________________________________
>>>> mvapich-discuss mailing list
>>>> mvapich-discuss at cse.ohio-state.edu
>>>> http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss
>>>
>>>
>> _______________________________________________
>> mvapich-discuss mailing list
>> mvapich-discuss at cse.ohio-state.edu
>> http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss
>>
> 
> _______________________________________________
> mvapich-discuss mailing list
> mvapich-discuss at cse.ohio-state.edu
> http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss
>