[mvapich-discuss] Which mvapich version to install on a GPUcluster without Mellanox OFED ?

Thu Apr 19 10:19:32 EDT 2018

Hi, Yussuf.

When you said you had a “GPU cluster”, I assumed that you had multiple such nodes interconnected using some network. If all you have is one node, then please ignore the question about the network.

Yes, the error message is related to OFED not being found in normal system paths (/usr/lib, /usr/local/lib etc). If you do not have OFED on the system, I would request that you download and install Mellanox OFED from the following site. This will work even if you do not have any network adapter. Note that installing OFED needs administrative privileges.

http://www.mellanox.com/page/products_dyn?product_family=26

Once you have installed Mellanox OFED, please download the appropriate version of MVAPICH2-GDR from our download page. Note that the GDR package built for Mellanox OFED 4.2 should work with the latest Mellanox OFED package (4.3).

http://mvapich.cse.ohio-state.edu/downloads/

For best performance, I would recommend installing GDRCopy as well.

Please let me know if you’ve any further questions.

Regards,

Hari.

From: Yussuf Ali [mailto:Yussuf.ali at jaea.go.jp]
Sent: Wednesday, April 18, 2018 9:57 PM
To: Subramoni, Hari <subramoni.1 at osu.edu>; mvapich-discuss at cse.ohio-state.edu <mvapich-discuss at mailman.cse.ohio-state.edu>
Cc: Subramoni, Hari <subramoni.1 at osu.edu>
Subject: RE: [mvapich-discuss] Which mvapich version to install on a GPUcluster without Mellanox OFED ?

Hi Hari,

Thank you for your answer!

What exactly you mean by interconnect? I can definitely say it does not have a Mellnox infniband or Intel Omni-Path adapter.

It is a two socket CPU system, each CPU connected(?) to four GPUS. I think the interconnect becomes than the (QPI/UPI) ?

The server seems to be some kind of “PCI-Express GPU server”

According to (nvidia-smi topo -m) there is some kind of PCI interconnect (PIX) between GPU pairs.

If its helps it is a box like this (All in Japanese: http://www.gdep.co.jp/products/list/v/56a9c55e26b1c  , serial nr: MAS-XE5-SV4U/8X)

I don’t think OFED is present on this system. I downloaded MVAPICH-GDR and tried to compile a simple program, but it shows me the

Error message:

“libibumad.so.3, needed by /opt/mvapich2/gdr/2.3a/mcast/no-openacc/cuda8.0/mofed3.4/pbs/gnu4.8.5/lib64/libmpi.so, not found”

Is this error message related to OFED?

GDRCopy is currently not installed on the system.

Thank you for your help,

Yussuf

From: Subramoni, Hari<mailto:subramoni.1 at osu.edu>
Sent: Wednesday, April 18, 2018 7:22 PM
To: Yussuf Ali<mailto:Yussuf.ali at jaea.go.jp>; mvapich-discuss at cse.ohio-state.edu<mailto:mvapich-discuss at mailman.cse.ohio-state.edu>
Cc: Subramoni, Hari<mailto:subramoni.1 at osu.edu>
Subject: RE: [mvapich-discuss] Which mvapich version to install on a GPUcluster without Mellanox OFED ?

Hi, Yussuf.

You can use the optimized MVAPICH2-GDR for single node application runs even if Mellanox hardware is not present. This should give you the best performance within one node.

Could you please let us know the answer to the following questions

a.       What sort of interconnect the system has?

b.      What version of OFED is available on the system?

c.       Is GDRCopy available on the system?

a.     https://github.com/NVIDIA/gdrcopy