[mvapich-discuss] regestration cache limits in mvapich 0.9.7

[Abhinav Vishnu]

Hi Pasha,

>
> ib_init() include follow #ifdef (viainit.c):
> -----------------------------------------------------------------
> #ifndef _PPC64_
>          attr.cap.max_inline_data = viadev_max_inline_size;
> #else
>          attr.cap.max_inline_data = -1;
> #endif
> -----------------------------------------------------------------
>
> max_inline_data is uint32_t and mthca_create_qp
> (src/userspace/libmthca/src/verbs.c) include:
> -----------------------------------------------------------------
> if (attr->cap.max_send_wr     > 65536 ||
>              attr->cap.max_recv_wr     > 65536 ||
>              attr->cap.max_send_sge    > 64    ||
>              attr->cap.max_recv_sge    > 64    ||
>              attr->cap.max_inline_data > 1024)
>                  return NULL;
> -----------------------------------------------------------------
>
> And as result the code does not run on PPC64
> Is it some reason for this ifdef ?
>

Thanks for pointing out this problem to us. Since the IBM EHCA does not
support inline data transfer, we defined the max inline size to be -1.
However as you have pointed out, this solution does not seem to be
completely correct.

We have taken care of this issue by allowing the messages over
IBM EHCA to use only IBV_SEND_SIGNALED flag. This change has been checked
in the SVN, both for single-rail and multi-rail devices. Please let you
know if the solution helps solve your problem.

Thanks and regards,

-- Abhinav

> Regards,
> Pavel Shamis (Pasha)
>
>
>
> Sayantan Sur wrote:
> > Hello Pasha,
> >
> > Pavel Shamis (Pasha) wrote:
> >
> >> Hi,
> >> In mvapich-0.9.7 you have viadev_dreg_cache_limit limit that should
> >> limit the size of registration cache. From review in dreg.* stuff I see
> >> that actually this parameter limits only the maximum number of pages
> >> per _single_ buffer registration (see dreg_new_entry()) and not total
> >> number of registered pages in the cache.  Can you please review this
> >> point ? Because I'm not sure that we want limit # of pages per single
> >> buffer registration.
> >
> > Thanks for looking at the code in detail. Here's my explanation of the
> > code:
> >
> > 1) dreg cache interface is called using dreg_register()
> > 2) If no cache entry is found, dreg_new_entry() is called
> > 3) dreg_new_entry() checks if the single registration size itself will
> > violate the cache limit. It doesn't allow a single registration to go
> > past the user defined limit.
> > 4) If it is found that the single registration is below the user defined
> > limit, then the flow proceeds to dreg_insert()
> > 5) In dreg_insert(), we call vma_new() to insert a new virtual memory
> > region
> > 6) In vma_new() the number of pages requested is added to a variable
> > called `pinned_pages_count'. If this pinned_pages_count exceeds the user
> > set limit `viadev_dreg_cache_limit', then NULL is returned and this is
> > propagated to the upper layer to report a failed registration. If not,
> > then the `pinned_pages_count' variable is incremented accordingly.
> > 7) `pinned_pages_count' is decremented when a memory area is removed
> > using vma_remove()
> >
> > I hope this helps.
> >
> > Thanks,
> > Sayantan.
> >
> >>
> >> Thanks,
> >> Pavel Shamis (Pasha)
> >> _______________________________________________
> >> mvapich-discuss mailing list
> >> mvapich-discuss at cse.ohio-state.edu
> >> http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss
> >
> >
> >
>
> _______________________________________________
> mvapich-discuss mailing list
> mvapich-discuss at cse.ohio-state.edu
> http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss
>