[mvapich-discuss] Re: data placement in-order guarantee ?

[Håkon Bugge]

On Feb 2, 2010, at 16:14 , yogeshwar sonawane wrote:

> Hi all,
> 
> Following question can be of naïve nature. Kindly, help me to clarify
> the points.
> Thanks in Advance.
> 
> [snip].
> 
> It will be helpful if some information is shared as how mvapich2
> handles the situation regarding above point (while using rdma
> operations) ?

Snip from the InfiniBandTM Architecture Specification:

o9-20: An application shall not depend of the contents of an RDMA WRITE buffer at the responder until one of the following has occurred:
19 20 21
•	Arrival and Completion of the last RDMA WRITE request packet when used with Immediate data.
•	Arrival and completion of a subsequent SEND message.
•	Update of a memory element by a subsequent ATOMIC operation.

Now, most MPI implementation does not obey this; IIRC MVAPICH does rely on the last byte (word?) being written after all other payload, Scali MPI relies on 4-byte write atomicy and uses a special encoding/decoding which sustains any reordering of the payload (as long as each 4-byte word is written as an atomic unit). The benefit, besides robustness, is the capability of a worm-hole message consumption (the message can be decoded and copied into the MPI user buffer _before_ it has been fully received). This method is currently a patent application, so I do not recommend you to copy-cat this method.

So your choice is to be a "nice guy" using one of the preferred methods from the IB standard or rely on how different IB _implementations_ work.



Håkon




> 
> I thought that i should mail to both the mailing lists as topic covers
> both domains.
> 
> Thanks and Regards,
> Yogeshwar
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo at vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Håkon Bugge
Haakon.bugge at Sun.com
+47 924 84 514