[Mvapich-discuss] RDMA memory registration for AllGather

[Subramoni, Hari]

To the best of my knowledge, IB verbs does support this. Please look at the MVAPICH code for more details.

Best,
Hari.

From: Niyaz Murshed <Niyaz.Murshed at arm.com>
Sent: Wednesday, August 21, 2024 12:20 PM
To: Subramoni, Hari <subramoni.1 at osu.edu>; Announcement about MVAPICH (MPI over InfiniBand, RoCE, Omni-Path, Slingshot, iWARP and EFA) Libraries developed at NBCL/OSU <mvapich-discuss at lists.osu.edu>
Cc: nd <nd at arm.com>; Subramoni, Hari <subramoni.1 at osu.edu>
Subject: Re: RDMA memory registration for AllGather

ibv_reg_mr takes in PD and buffer. Each PD is associated with a HCA. If we register the buffer with multiple PDs, it can be access by all HCAs? My understanding from another source is that “ibverbs” do not support this. ‘Some providers (e. g. 
ZjQcmQRYFpfptBannerStart
This Message Is From an External Sender
This message came from outside your organization.
    Report Suspicious  <https://us-phishalarm-ewt.proofpoint.com/EWT/v1/KGKeukY!vwQdMSZND6YBZRdwP87EmWXS6Y97NEu4enmmpDPsBZ5klIXK3WFg9faxwumef21rksMl9U6XVJC8OUi9XJCKOuuCA8Lm7ZIeUhkYNUVkDC_kZBEXfU3jNqZzZ1FpINLCb4YFB-t89hrKZw$>   ‌
ZjQcmQRYFpfptBannerEnd
ibv_reg_mr  takes in PD and buffer. Each PD is associated with a HCA.
If we register the buffer with multiple PDs, it can be access by all  HCAs?

My understanding from another source is that “ibverbs” do not support this.
‘Some providers (e.g. verbs) map domain to a specific nic and thus can't share MR among nics.’
https://lists.openfabrics.org/pipermail/libfabric-users/2024-August/001072.html<https://urldefense.com/v3/__https:/lists.openfabrics.org/pipermail/libfabric-users/2024-August/001072.html__;!!KGKeukY!xMFkSH86qg6fBW46tJgDLuhq5P_0tpSXJ3qurJx7iZBbndqKnHCU0DQuOCoJwgo3ht4dC7kXZu8uGBQpjH3sOCM$>

From: Subramoni, Hari <subramoni.1 at osu.edu<mailto:subramoni.1 at osu.edu>>
Date: Wednesday, August 21, 2024 at 11:15 AM
To: Niyaz Murshed <Niyaz.Murshed at arm.com<mailto:Niyaz.Murshed at arm.com>>, Announcement about MVAPICH (MPI over InfiniBand, RoCE, Omni-Path, Slingshot, iWARP and EFA) Libraries developed at NBCL/OSU <mvapich-discuss at lists.osu.edu<mailto:mvapich-discuss at lists.osu.edu>>
Cc: nd <nd at arm.com<mailto:nd at arm.com>>, Subramoni, Hari <subramoni.1 at osu.edu<mailto:subramoni.1 at osu.edu>>
Subject: RE: RDMA memory registration for AllGather
Hi, Niyaz.

One buffer can be registered with multiple HCAs. There is no problem with this. You just to call ibv_reg_mr multiple times.

Thx,
Hari.

From: Mvapich-discuss <mvapich-discuss-bounces at lists.osu.edu<mailto:mvapich-discuss-bounces at lists.osu.edu>> On Behalf Of Niyaz Murshed via Mvapich-discuss
Sent: Tuesday, August 20, 2024 3:18 PM
To: Niyaz Murshed via Mvapich-discuss <mvapich-discuss at lists.osu.edu<mailto:mvapich-discuss at lists.osu.edu>>
Cc: nd <nd at arm.com<mailto:nd at arm.com>>
Subject: [Mvapich-discuss] RDMA memory registration for AllGather

Hello all , I am trying to understand how a single memory buffer is registered to multiple rdma verbvs context in case there are data coming in from different interfaces. As I understand, we can only register RDMA memory to a single protection
Hello all ,

I am trying to understand how a single memory buffer is registered to multiple rdma verbvs context in case there are data coming in from different interfaces.
As I understand, we can only register RDMA memory to a single protection domain which is linked to one interface.

Assuming we receive data from 3 ranks on interface1 and 2 ranks from interface 2, how do these interfaces write to same common buffer ?
How is the same buffer registered to 2 interfaces?

Regards,
Niyaz
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osu.edu/pipermail/mvapich-discuss/attachments/20240822/d4df93f3/attachment-0002.html>