[Mvapich-discuss] RDMA memory registration for AllGather

[Aaron Knister]

I think the question on the libfabric list was interpreted as asking if you
can use an ibv_mr returned by ibv_reg_mr with multiple NICs rather than I
think what you're asking which is "can I register this memory buffer to
multiple RDMA NICs". If I understand this correctly, in the latter case you
would have multiple ibv_mr's (one per RDMA NIC) that all point to the same
memory address and in the former case you would be attempting to use a
single ibv_mr on multiple NICs.

On Wed, Aug 21, 2024 at 12:21 PM Niyaz Murshed via Mvapich-discuss <
mvapich-discuss at lists.osu.edu> wrote:

> ibv_reg_mr takes in PD and buffer. Each PD is associated with a HCA. If we
> register the buffer with multiple PDs, it can be access by all HCAs? My
> understanding from another source is that “ibverbs” do not support this.
> ‘Some providers (e. g.
>
> ibv_reg_mr  takes in PD and buffer. Each PD is associated with a HCA.
>
> If we register the buffer with multiple PDs, it can be access by all  HCAs?
>
>
>
> My understanding from another source is that “ibverbs” do not support this.
>
> ‘Some providers (e.g. verbs) map domain to a specific nic and thus can't
> share MR among nics.’
>
>
> https://urldefense.com/v3/__https://lists.openfabrics.org/pipermail/libfabric-users/2024-August/001072.html__;!!KGKeukY!z32yRnM82oJHzEe9xDj3tjI4HszkzcRtRZUmmAW127rSHVoupYuVSW4WqgupV4zeE1ZEIW3CGEEkK6yxg6eDIS0Ibcbg-9xWpw$ 
> <https://urldefense.com/v3/__https://lists.openfabrics.org/pipermail/libfabric-users/2024-August/001072.html__;!!KGKeukY!zUrZMd5JRh1pubI8Fuu1XwdWwMy2uLWiA-K4phQvn8l2tX9sKky66SvWrNI6KNax2SuTyiVci0mUwI9et7neP0SHWlrREto$>
>
>
>
> *From: *Subramoni, Hari <subramoni.1 at osu.edu>
> *Date: *Wednesday, August 21, 2024 at 11:15 AM
> *To: *Niyaz Murshed <Niyaz.Murshed at arm.com>, Announcement about MVAPICH
> (MPI over InfiniBand, RoCE, Omni-Path, Slingshot, iWARP and EFA) Libraries
> developed at NBCL/OSU <mvapich-discuss at lists.osu.edu>
> *Cc: *nd <nd at arm.com>, Subramoni, Hari <subramoni.1 at osu.edu>
> *Subject: *RE: RDMA memory registration for AllGather
>
> Hi, Niyaz.
>
>
>
> One buffer can be registered with multiple HCAs. There is no problem with
> this. You just to call ibv_reg_mr multiple times.
>
>
>
> Thx,
>
> Hari.
>
>
>
> *From:* Mvapich-discuss <mvapich-discuss-bounces at lists.osu.edu> *On
> Behalf Of *Niyaz Murshed via Mvapich-discuss
> *Sent:* Tuesday, August 20, 2024 3:18 PM
> *To:* Niyaz Murshed via Mvapich-discuss <mvapich-discuss at lists.osu.edu>
> *Cc:* nd <nd at arm.com>
> *Subject:* [Mvapich-discuss] RDMA memory registration for AllGather
>
>
>
> Hello all , I am trying to understand how a single memory buffer is
> registered to multiple rdma verbvs context in case there are data coming in
> from different interfaces. As I understand, we can only register RDMA
> memory to a single protection
>
> Hello all ,
>
>
>
> I am trying to understand how a single memory buffer is registered to
> multiple rdma verbvs context in case there are data coming in from
> different interfaces.
>
> As I understand, we can only register RDMA memory to a single protection
> domain which is linked to one interface.
>
>
>
> Assuming we receive data from 3 ranks on interface1 and 2 ranks from
> interface 2, how do these interfaces write to same common buffer ?
>
> How is the same buffer registered to 2 interfaces?
>
>
>
> Regards,
>
> Niyaz
> _______________________________________________
> Mvapich-discuss mailing list
> Mvapich-discuss at lists.osu.edu
> https://lists.osu.edu/mailman/listinfo/mvapich-discuss
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osu.edu/pipermail/mvapich-discuss/attachments/20240821/e39e0aaf/attachment-0002.html>