[mvapich-discuss] OpenACC not supported with mvapich2-gdr 2.3 rpm - Is there another version that does?

Subramoni, Hari subramoni.1 at osu.edu
Tue Jan 29 10:57:53 EST 2019 

Hi, Raghu.

Thanks for confirming.

I'm CC'ing discuss so that others are also aware.

Thx,
Hari.

From: Raghu Reddy <raghu.reddy at noaa.gov>
Sent: Tuesday, January 29, 2019 10:53 AM
To: Subramoni, Hari <subramoni.1 at osu.edu>
Cc: 'Brian Osmond' <brian.osmond at noaa.gov>; _ENG CSE Mvapich-Core <ENG-cse-mvapich-core at osu.edu>
Subject: RE: [mvapich-discuss] OpenACC not supported with mvapich2-gdr 2.3 rpm - Is there another version that does?

Hi Hari,

Yes, that does indeed seem to be the case!

We are currently using Slurm/18.08.3, and I believe PMI has been moved into a separate package!

Thank you for pointing this out!

Raghu!

From: Subramoni, Hari [mailto:subramoni.1 at osu.edu]
Sent: Tuesday, January 29, 2019 10:32 AM
To: Raghu Reddy <raghu.reddy at noaa.gov<mailto:raghu.reddy at noaa.gov>>
Cc: 'Brian Osmond' <brian.osmond at noaa.gov<mailto:brian.osmond at noaa.gov>>; _ENG CSE Mvapich-Core <ENG-cse-mvapich-core at osu.edu<mailto:ENG-cse-mvapich-core at osu.edu>>; Subramoni, Hari <subramoni.1 at osu.edu<mailto:subramoni.1 at osu.edu>>
Subject: RE: [mvapich-discuss] OpenACC not supported with mvapich2-gdr 2.3 rpm - Is there another version that does?

Hi, Raghu.

It looks like the RPM was not able to find libpmi.so.

Could you please let us know which version of SLURM you installed? For some reason, the default installation procedure of SLURM 18 does not seem to install libpmi.so (but older versions like SLURM 17* does install libpmi.so).

I would like to see if this could be the cause. So, could you please send output of module show slurm and the contents of the "lib" and "lib64" directory where libslurm.so is present?

Thx,
Hari.

From: Raghu Reddy <raghu.reddy at noaa.gov<mailto:raghu.reddy at noaa.gov>>
Sent: Tuesday, January 29, 2019 10:18 AM
To: Subramoni, Hari <subramoni.1 at osu.edu<mailto:subramoni.1 at osu.edu>>; mvapich-discuss at cse.ohio-state.edu<mailto:mvapich-discuss at cse.ohio-state.edu> <mvapich-discuss at mailman.cse.ohio-state.edu<mailto:mvapich-discuss at mailman.cse.ohio-state.edu>>
Cc: 'Brian Osmond' <brian.osmond at noaa.gov<mailto:brian.osmond at noaa.gov>>; Raghu Reddy <raghu.reddy at noaa.gov<mailto:raghu.reddy at noaa.gov>>
Subject: RE: [mvapich-discuss] OpenACC not supported with mvapich2-gdr 2.3 rpm - Is there another version that does?

Hi Hari,

We have installed the PGI version and GNU version on our system and are currently trying to test the PGI version and it is complaining about a missing PMI library, can you please let us know what we may be doing wrong?

Because our images are readonly we use cpio to install the RPM in a different location, but this is the same process we have used before with your intel distributions and those have worked fine.

We install the library in /apps instead of /opt and change the wrappers to point to the new path.

Here are the details:

[Raghu.Reddy at sg001 hello]$ module purge
[Raghu.Reddy at sg001 hello]$ module load slurm cuda/9.2 pgi/18.7 mvapich2-gdr/2.3-1-cuda9.2-pgi17.10-slurm
[Raghu.Reddy at sg001 hello]$
[Raghu.Reddy at sg001 hello]$ mpicc -o foo hello_mpi_c.c
/bin/ld: warning: libpmi.so.0, needed by /apps/mvapich2-gdr/2.3-1/cuda9.2-pgi17.10-mofed4.2-slurm-2.3-1/lib64/libmpi.so, not found (try using -rpath or -rpath-link)
/apps/mvapich2-gdr/2.3-1/cuda9.2-pgi17.10-mofed4.2-slurm-2.3-1/lib64/libmpi.so: undefined reference to `PMI_Get_size'
/apps/mvapich2-gdr/2.3-1/cuda9.2-pgi17.10-mofed4.2-slurm-2.3-1/lib64/libmpi.so: undefined reference to `PMI_Get_rank'
/apps/mvapich2-gdr/2.3-1/cuda9.2-pgi17.10-mofed4.2-slurm-2.3-1/lib64/libmpi.so: undefined reference to `PMI_Unpublish_name'
/apps/mvapich2-gdr/2.3-1/cuda9.2-pgi17.10-mofed4.2-slurm-2.3-1/lib64/libmpi.so: undefined reference to `PMI_KVS_Get_name_length_max'
/apps/mvapich2-gdr/2.3-1/cuda9.2-pgi17.10-mofed4.2-slurm-2.3-1/lib64/libmpi.so: undefined reference to `PMI_KVS_Get_key_length_max'
/apps/mvapich2-gdr/2.3-1/cuda9.2-pgi17.10-mofed4.2-slurm-2.3-1/lib64/libmpi.so: undefined reference to `PMI_Finalize'
/apps/mvapich2-gdr/2.3-1/cuda9.2-pgi17.10-mofed4.2-slurm-2.3-1/lib64/libmpi.so: undefined reference to `PMI_KVS_Get_my_name'
/apps/mvapich2-gdr/2.3-1/cuda9.2-pgi17.10-mofed4.2-slurm-2.3-1/lib64/libmpi.so: undefined reference to `PMI_Spawn_multiple'
/apps/mvapich2-gdr/2.3-1/cuda9.2-pgi17.10-mofed4.2-slurm-2.3-1/lib64/libmpi.so: undefined reference to `PMI_Lookup_name'
/apps/mvapich2-gdr/2.3-1/cuda9.2-pgi17.10-mofed4.2-slurm-2.3-1/lib64/libmpi.so: undefined reference to `PMI_KVS_Get_value_length_max'
/apps/mvapich2-gdr/2.3-1/cuda9.2-pgi17.10-mofed4.2-slurm-2.3-1/lib64/libmpi.so: undefined reference to `PMI_Barrier'
/apps/mvapich2-gdr/2.3-1/cuda9.2-pgi17.10-mofed4.2-slurm-2.3-1/lib64/libmpi.so: undefined reference to `PMI_Init'
/apps/mvapich2-gdr/2.3-1/cuda9.2-pgi17.10-mofed4.2-slurm-2.3-1/lib64/libmpi.so: undefined reference to `PMI_KVS_Put'
/apps/mvapich2-gdr/2.3-1/cuda9.2-pgi17.10-mofed4.2-slurm-2.3-1/lib64/libmpi.so: undefined reference to `PMI_Initialized'
/apps/mvapich2-gdr/2.3-1/cuda9.2-pgi17.10-mofed4.2-slurm-2.3-1/lib64/libmpi.so: undefined reference to `PMI_Abort'
/apps/mvapich2-gdr/2.3-1/cuda9.2-pgi17.10-mofed4.2-slurm-2.3-1/lib64/libmpi.so: undefined reference to `PMI_KVS_Commit'
/apps/mvapich2-gdr/2.3-1/cuda9.2-pgi17.10-mofed4.2-slurm-2.3-1/lib64/libmpi.so: undefined reference to `PMI_Get_universe_size'
/apps/mvapich2-gdr/2.3-1/cuda9.2-pgi17.10-mofed4.2-slurm-2.3-1/lib64/libmpi.so: undefined reference to `PMI_Get_appnum'
/apps/mvapich2-gdr/2.3-1/cuda9.2-pgi17.10-mofed4.2-slurm-2.3-1/lib64/libmpi.so: undefined reference to `PMI_KVS_Get'
/apps/mvapich2-gdr/2.3-1/cuda9.2-pgi17.10-mofed4.2-slurm-2.3-1/lib64/libmpi.so: undefined reference to `PMI_Publish_name'
[Raghu.Reddy at sg001 hello]$


sfe01% ll /apps/mvapich2-gdr/2.3-1/cuda9.2-pgi17.10-mofed4.2-slurm-2.3-1/lib64/
total 123200
lrwxrwxrwx 1 Brian.Osmond nesccmgmt        13 Jan 22 20:52 libfmpich.so -> libmpifort.so*
lrwxrwxrwx 1 Brian.Osmond nesccmgmt        12 Jan 22 20:52 libmpichcxx.so -> libmpicxx.so*
lrwxrwxrwx 1 Brian.Osmond nesccmgmt        13 Jan 22 20:52 libmpichf90.so -> libmpifort.so*
lrwxrwxrwx 1 Brian.Osmond nesccmgmt         9 Jan 22 20:52 libmpich.so -> libmpi.so*
lrwxrwxrwx 1 Brian.Osmond nesccmgmt        19 Jan 22 20:52 libmpicxx.so -> libmpicxx.so.12.1.1*
lrwxrwxrwx 1 Brian.Osmond nesccmgmt        19 Jan 22 20:52 libmpicxx.so.12 -> libmpicxx.so.12.1.1*
-rwxr-xr-x 1 Brian.Osmond nesccmgmt    488552 Jan 22 20:52 libmpicxx.so.12.1.1*
lrwxrwxrwx 1 Brian.Osmond nesccmgmt        20 Jan 22 20:52 libmpifort.so -> libmpifort.so.12.1.1*
lrwxrwxrwx 1 Brian.Osmond nesccmgmt        20 Jan 22 20:52 libmpifort.so.12 -> libmpifort.so.12.1.1*
-rwxr-xr-x 1 Brian.Osmond nesccmgmt   6548352 Jan 22 20:52 libmpifort.so.12.1.1*
lrwxrwxrwx 1 Brian.Osmond nesccmgmt        16 Jan 22 20:52 libmpi.so -> libmpi.so.12.1.1*
lrwxrwxrwx 1 Brian.Osmond nesccmgmt        16 Jan 22 20:52 libmpi.so.12 -> libmpi.so.12.1.1*
-rwxr-xr-x 1 Brian.Osmond nesccmgmt 118675144 Jan 22 20:52 libmpi.so.12.1.1*
lrwxrwxrwx 1 Brian.Osmond nesccmgmt         9 Jan 22 20:52 libmpl.so -> libmpi.so*
lrwxrwxrwx 1 Brian.Osmond nesccmgmt         9 Jan 22 20:52 libopa.so -> libmpi.so*
drwxr-xr-x 2 Brian.Osmond nesccmgmt      2048 Jan 22 20:52 pkgconfig/
sfe01%

Please let us know if you need any additional information that may help in diagnose and the problem.

Thank you very much for your help!

Thanks,
Raghu



From: Subramoni, Hari [mailto:subramoni.1 at osu.edu]
Sent: Monday, January 21, 2019 4:17 PM
To: Raghu Reddy <raghu.reddy at noaa.gov<mailto:raghu.reddy at noaa.gov>>; mvapich-discuss at cse.ohio-state.edu<mailto:mvapich-discuss at cse.ohio-state.edu> <mvapich-discuss at mailman.cse.ohio-state.edu<mailto:mvapich-discuss at mailman.cse.ohio-state.edu>>
Cc: Brian Osmond <brian.osmond at noaa.gov<mailto:brian.osmond at noaa.gov>>; Subramoni, Hari <subramoni.1 at osu.edu<mailto:subramoni.1 at osu.edu>>
Subject: RE: [mvapich-discuss] OpenACC not supported with mvapich2-gdr 2.3 rpm - Is there another version that does?

Hi, Raghu.

We just meant that the OMB shipped with the RPM does not have support for OpenACC related benchmarks. The MVAPICH2-GDR codebase should still support OpenACC.

Thx,
Hari.

From: mvapich-discuss <mvapich-discuss-bounces at cse.ohio-state.edu<mailto:mvapich-discuss-bounces at cse.ohio-state.edu>> On Behalf Of Raghu Reddy
Sent: Monday, January 21, 2019 3:50 PM
To: mvapich-discuss at cse.ohio-state.edu<mailto:mvapich-discuss at cse.ohio-state.edu> <mvapich-discuss at mailman.cse.ohio-state.edu<mailto:mvapich-discuss at mailman.cse.ohio-state.edu>>
Cc: Brian Osmond <brian.osmond at noaa.gov<mailto:brian.osmond at noaa.gov>>
Subject: [mvapich-discuss] OpenACC not supported with mvapich2-gdr 2.3 rpm - Is there another version that does?

Dear MVAPICH-2 GDR team,

We have downloaded the following rpm:

mvapich2-gdr-mcast.cuda9.2.mofed4.2.gnu4.8.5.slurm-2.3-1.el7.x86_64.rpm

When we extract this file, I only the following tree:

               /opt/mvapich2/gdr/2.3/mcast/no-openacc/*

Based on this it looks like OpenACC would not be supported with this RPM.  Is there a version available that support OpenACC?

Thanks,
Raghu

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.cse.ohio-state.edu/pipermail/mvapich-discuss/attachments/20190129/a5762e03/attachment-0001.html>

More information about the mvapich-discuss mailing list