[mvapich-discuss] Which mvapich version to installonaGPUcluster without Mellanox OFED ?

Wed Apr 25 04:06:20 EDT 2018

Dear Hari,

Thank you for the information! I will check it!

Thank you for your help,
Yussuf 

From: Subramoni, Hari
Sent: Monday, April 23, 2018 8:50 PM
To: Yussuf Ali; mvapich-discuss at cse.ohio-state.edu
Cc: Subramoni, Hari
Subject: RE: [mvapich-discuss] Which mvapich version to installonaGPUcluster without Mellanox OFED ?

===( By JAEA Mail System )===============================
  URL中の文字「:」を「*」に置換しました。
  Characters of ":" in URL have been replaced with "*". 
=========================================================

Hi, Yussuf.

Good to know that you were able to run MVAPICH2-GDR.

By default, MVAPICH2 uses the Cross Memory Attach (CMA) feature available in the Linux Kernel to perform intra-node host to host transfers with high-performance. It looks like the Linux kernel you have does not support this feature. That is why the warning message is showing up. If you can upgrade to a kernel version which support CMA, you will see better performance for larger messages transferred from host to host inside one node. The following articles has some details on CMA and its associated benefits.

https*//lwn.net/Articles/405284/
https*//dl.acm.org/citation.cfm?id=2616532

Please let us know if you face any other issues (performance or functionality) in running MVAPICH2-GDR.

Thx,
Hari.

From: Yussuf Ali [mailto:Yussuf.ali at jaea.go.jp]
Sent: Monday, April 23, 2018 2:14 AM
To: Subramoni, Hari <subramoni.1 at osu.edu>; mvapich-discuss at cse.ohio-state.edu <mvapich-discuss at mailman.cse.ohio-state.edu>
Cc: Subramoni, Hari <subramoni.1 at osu.edu>
Subject: RE: [mvapich-discuss] Which mvapich version to install onaGPUcluster without Mellanox OFED ?

Dear Hari,

thank you for your answer! With your help I was able to execute the osu_benchmark successfully!

When I executed the  benchmark I got this message:

“CMA is not available. Set MV2_SMP_USE_CMA=0 to disable CMA.”

So I also set MV2_SMP_USE_CMA=0

But I’m not sure what this means, I can’t find an online documentation about this flag.

Thank you for your help,
Yussuf

From: Subramoni, Hari<mailto:subramoni.1 at osu.edu>
Sent: Saturday, April 21, 2018 10:17 PM
To: Yussuf Ali<mailto:Yussuf.ali at jaea.go.jp>; mvapich-discuss at cse.ohio-state.edu<mailto:mvapich-discuss at mailman.cse.ohio-state.edu>
Cc: Subramoni, Hari<mailto:subramoni.1 at osu.edu>
Subject: RE: [mvapich-discuss] Which mvapich version to install onaGPUcluster without Mellanox OFED ?

===( By JAEA Mail System )===============================
  URL中の文字「:」を「*」に置換しました。
  Characters of ":" in URL have been replaced with "*".
=========================================================

Hi, Yussuf.

Sorry about the delay in getting back to you.

Using an older Mellanox OFED (4.2) for instance is fine.

Yes, you can use the without SLURM option.

Yes, it means you cannot use MVAPICH2 with GDRCopy. Since your system does not have GDRCopy or GPUDirect RDMA, you can set MV2_USE_GPUDIRECT=0 MV2_USE_GPUDIRECT_GDRCOPY=0 to disable these features at runtime. Note that you will also have to set LD_PRELOAD=<MV2_INSTALL_PATH>/lib64/libmpi.so on the command before running applications or benchmarks (like osu_bw).

Please let us know if you face any issues.

Thx,
Hari.

From: Yussuf Ali [mailto:Yussuf.ali at jaea.go.jp]
Sent: Friday, April 20, 2018 1:24 AM
To: Subramoni, Hari <subramoni.1 at osu.edu<mailto:subramoni.1 at osu.edu>>; mvapich-discuss at cse.ohio-state.edu<mailto:mvapich-discuss at cse.ohio-state.edu> <mvapich-discuss at mailman.cse.ohio-state.edu<mailto:mvapich-discuss at mailman.cse.ohio-state.edu>>
Subject: RE: [mvapich-discuss] Which mvapich version to install on aGPUcluster without Mellanox OFED ?

Dear Hari,

Thank you for your answer! Sorry for my misleading description! Yes, it is only a single node not a cluster

My goal is to run the osu benchmark suite on this node in order to measure the performance for the device to device case “./osu_bw D D”.

The system is running Ubuntu  14.04 but OFED 4.3 version only supports Ubuntu 16.04 and newer. So I have to use OFED 4.2 ?

The system is not running any PBS/Slurm software so I choose GNU 4.8.5 (w/o SLURM) MLNX-OFED 4.2, is this correct?

I downloaded the GDRcopy library and tried to run the examples but they did execute successfully. The GPUs used in this system are GeforeGTX 1080 Ti
according to NVIDIA Geforce cards are not supported for GDRcopy.

Does this mean I can’t use MVAPICH together with the GDRcopy library?

Thank you for your help,
Yussuf

From: Subramoni, Hari<mailto:subramoni.1 at osu.edu>
Sent: Thursday, April 19, 2018 11:20 PM
To: Yussuf Ali<mailto:Yussuf.ali at jaea.go.jp>; mvapich-discuss at cse.ohio-state.edu<mailto:mvapich-discuss at mailman.cse.ohio-state.edu<mailto:mvapich-discuss at cse.ohio-state.edu%3cmailto:mvapich-discuss at mailman.cse.ohio-state.edu>>
Cc: Subramoni, Hari<mailto:subramoni.1 at osu.edu>
Subject: RE: [mvapich-discuss] Which mvapich version to install on aGPUcluster without Mellanox OFED ?

Hi, Yussuf.

When you said you had a “GPU cluster”, I assumed that you had multiple such nodes interconnected using some network. If all you have is one node, then please ignore the question about the network.

Yes, the error message is related to OFED not being found in normal system paths (/usr/lib, /usr/local/lib etc). If you do not have OFED on the system, I would request that you download and install Mellanox OFED from the following site. This will work even if you do not have any network adapter. Note that installing OFED needs administrative privileges.

http*//www.mellanox.com/page/products_dyn?product_family=26

Once you have installed Mellanox OFED, please download the appropriate version of MVAPICH2-GDR from our download page. Note that the GDR package built for Mellanox OFED 4.2 should work with the latest Mellanox OFED package (4.3).

http*//mvapich.cse.ohio-state.edu/downloads/

For best performance, I would recommend installing GDRCopy as well.

Please let me know if you’ve any further questions.

Regards,
Hari.

From: Yussuf Ali [mailto:Yussuf.ali at jaea.go.jp]
Sent: Wednesday, April 18, 2018 9:57 PM
To: Subramoni, Hari <subramoni.1 at osu.edu<mailto:subramoni.1 at osu.edu<mailto:subramoni.1 at osu.edu%3cmailto:subramoni.1 at osu.edu>>>; mvapich-discuss at cse.ohio-state.edu<mailto:mvapich-discuss at cse.ohio-state.edu<mailto:mvapich-discuss at cse.ohio-state.edu%3cmailto:mvapich-discuss at cse.ohio-state.edu>> <mvapich-discuss at mailman.cse.ohio-state.edu<mailto:mvapich-discuss at mailman.cse.ohio-state.edu<mailto:mvapich-discuss at mailman.cse.ohio-state.edu%3cmailto:mvapich-discuss at mailman.cse.ohio-state.edu>>>
Cc: Subramoni, Hari <subramoni.1 at osu.edu<mailto:subramoni.1 at osu.edu<mailto:subramoni.1 at osu.edu%3cmailto:subramoni.1 at osu.edu>>>
Subject: RE: [mvapich-discuss] Which mvapich version to install on a GPUcluster without Mellanox OFED ?

Hi Hari,

Thank you for your answer!

What exactly you mean by interconnect? I can definitely say it does not have a Mellnox infniband or Intel Omni-Path adapter.
It is a two socket CPU system, each CPU connected(?) to four GPUS. I think the interconnect becomes than the (QPI/UPI) ?
The server seems to be some kind of “PCI-Express GPU server”
According to (nvidia-smi topo -m) there is some kind of PCI interconnect (PIX) between GPU pairs.
If its helps it is a box like this (All in Japanese: http*//www.gdep.co.jp/products/list/v/56a9c55e26b1c  , serial nr: MAS-XE5-SV4U/8X)

I don’t think OFED is present on this system. I downloaded MVAPICH-GDR and tried to compile a simple program, but it shows me the
Error message:

“libibumad.so.3, needed by /opt/mvapich2/gdr/2.3a/mcast/no-openacc/cuda8.0/mofed3.4/pbs/gnu4.8.5/lib64/libmpi.so, not found”

Is this error message related to OFED?

GDRCopy is currently not installed on the system.

Thank you for your help,
Yussuf

From: Subramoni, Hari<mailto:subramoni.1 at osu.edu>
Sent: Wednesday, April 18, 2018 7:22 PM
To: Yussuf Ali<mailto:Yussuf.ali at jaea.go.jp>; mvapich-discuss at cse.ohio-state.edu<mailto:mvapich-discuss at mailman.cse.ohio-state.edu<mailto:mvapich-discuss at cse.ohio-state.edu%3cmailto:mvapich-discuss at mailman.cse.ohio-state.edu>>
Cc: Subramoni, Hari<mailto:subramoni.1 at osu.edu>
Subject: RE: [mvapich-discuss] Which mvapich version to install on a GPUcluster without Mellanox OFED ?

Hi, Yussuf.

You can use the optimized MVAPICH2-GDR for single node application runs even if Mellanox hardware is not present. This should give you the best performance within one node.

Could you please let us know the answer to the following questions

a.       What sort of interconnect the system has?

b.      What version of OFED is available on the system?

c.       Is GDRCopy available on the system?

a.     https*//github.com/NVIDIA/gdrcopy

This will enable us to help you better.

Best Regards,
Hari.

From: mvapich-discuss-bounces at cse.ohio-state.edu<mailto:mvapich-discuss-bounces at cse.ohio-state.edu<mailto:mvapich-discuss-bounces at cse.ohio-state.edu%3cmailto:mvapich-discuss-bounces at cse.ohio-state.edu>> On Behalf Of Yussuf Ali
Sent: Wednesday, April 18, 2018 12:20 AM
To: mvapich-discuss at cse.ohio-state.edu<mailto:mvapich-discuss at cse.ohio-state.edu<mailto:mvapich-discuss at cse.ohio-state.edu%3cmailto:mvapich-discuss at cse.ohio-state.edu>> <mvapich-discuss at mailman.cse.ohio-state.edu<mailto:mvapich-discuss at mailman.cse.ohio-state.edu<mailto:mvapich-discuss at mailman.cse.ohio-state.edu%3cmailto:mvapich-discuss at mailman.cse.ohio-state.edu>>>
Subject: [mvapich-discuss] Which mvapich version to install on a GPU cluster without Mellanox OFED ?

Dear Mvapich usergroup,

we have the following GPU cluster system with 8 GPUs (GeForce 1080Ti) but without any Mallanox hardware.

Out goal is to use MPI so send data between different GPUs directly from CUDA device buffers.
Is this possible with any Mvapich version and this particular GPU hardware cluster?

Output from: nvidia-smi topo -m

          GPU0          GPU1         GPU2          GPU3          GPU4          GPU5          GPU6          GPU7         CPU-Affinity
GPU0          X       PIX    PHB   PHB   SYS    SYS    SYS    SYS    0-7,16-23
GPU1          PIX    X       PHB   PHB   SYS    SYS    SYS    SYS    0-7,16-23
GPU2          PHB   PHB  X       PIX    SYS    SYS    SYS    SYS    0-7,16-23
GPU3          PHB   PHB  PIX    X       SYS    SYS    SYS    SYS    0-7,16-23
GPU4          SYS    SYS   SYS    SYS    X       PIX    PHB   PHB   8-15,24-31
GPU5          SYS    SYS   SYS    SYS    PIX    X       PHB   PHB   8-15,24-31
GPU6          SYS    SYS   SYS    SYS    PHB   PHB   X       PIX    8-15,24-31
GPU7          SYS    SYS   SYS    SYS    PHB   PHB   PIX    X       8-15,24-31

X    = Self
SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
PXB  = Connection traversing multiple PCIe switches (without traversing the PCIe Host Bridge)
PIX  = Connection traversing a single PCIe switch
NV#  = Connection traversing a bonded set of # NVLinks

Thank you for your help,
Yussuf

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.cse.ohio-state.edu/pipermail/mvapich-discuss/attachments/20180425/d8a6784e/attachment-0001.html>