[mvapich-discuss] MVAPICH2 with MV2_USE_RDMA_CM=1 flag

[Hari Subramoni]

Hi Malek,

MVAPICH2 supports different high performance network protocols like IB,
iWARP and RoCE. Although all of these support the "verbs" interface to
perform network communication, they require the applications and
middlewares (like MVAPICH2) that are written on top of them to use
different methods to establish the network connections between the
processes.

The iWARP protocol requires RDMA_CM to establish connections between
processes. This support is enabled in MVAPICH2 through the
"MV2_USE_RDMA_CM" environment variable. The following section of our
userguide has more information on running an MPI program with RDMA_CM.

https://mvapich.cse.ohio-state.edu/support/user_guide_mvapich2-1.9.html#x1-800006.16

The RDMA over Converged Ethernet (RoCE) protocol requires the special
handling to establish connections. This support is enabled through the
environment variable "MV2_USE_RDMAOE". Please note that we have renamed
this variable to "MV2_USE_RoCE" in our latest MVAPICH2 release (1.9).
The following section of our userguide has more information on running an
MPI program over RoCE.

https://mvapich.cse.ohio-state.edu/support/user_guide_mvapich2-1.9.html#x1-380005.2.7

InfiniBand networks support both RDMA_CM and IPoIB based connection
establishment methods. However, for the same underlying network, both of
these should deliver the same communication performance. The following
webpage has a detailed comparison of the performance MVAPICH2 can deliver
over various networks.

https://mvapich.cse.ohio-state.edu/performance/mvapich2/interNode.shtml

We recommend that you update your MPI installation to our latest release
(1.9) for the best performance, latest features and all the bug fixes.

Please let us know if you have any more questions.

Thanks,
Hari.


On Mon, Aug 5, 2013 at 12:25 PM, Malek Musleh <malek.musleh at gmail.com>wrote:

> Hi,
>
> I am testing the OSU_benchmarks over my infiniband setup, some machines
> which have connectx-gen2 and others gen3. I recently came across this fag
> in the user guide, and thought it should impact the performance results
> (improve it) since it would be using RDMA instead of IPoIB.
>
> At first, it did seem like for a few cases (test runs) I was generally
> seeing an improvement (for the large message sizes), but overall it does
> not seem like it, which seems to match the discussion in this thread:
>
>
> http://mail.cse.ohio-state.edu/pipermail/mvapich-discuss/2008-May/001644.html
>
> If it is only used in the setup part, what general benefit would there to
> be use the flag? Similarly, what benefit would there be to use the
> MV2_USE_RDMAOE=1 flag?
>
> I have only been using 2 test nodes at a time (for the moment), and using
> revision 1.8r527.
>
> Generally speaking, are specific benchmarks (osu_passive_acc_latency vs.
> osu_acc_latency) likely/should benefit, or perhaps intra-node tests vs.
> inter-node?
>
> Thanks,
>
> Malek
>
> _______________________________________________
> mvapich-discuss mailing list
> mvapich-discuss at cse.ohio-state.edu
> http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.cse.ohio-state.edu/pipermail/mvapich-discuss/attachments/20130806/01ae4e5c/attachment-0001.html