[mvapich-discuss] OpenACC not supported with mvapich2-gdr 2.3 rpm - Is there another version that does?

Tue Jan 29 10:21:07 EST 2019

I forgot to mention that this is the first time we are trying to install the
"slurm" version of the library.

So it is possible we need to make additional changes to the wrapper?  I
assume libpmi is something that is provided by slurm?

Thanks,

Raghu

From: Raghu Reddy [mailto:raghu.reddy at noaa.gov] 
Sent: Tuesday, January 29, 2019 10:18 AM
To: 'Subramoni, Hari' <subramoni.1 at osu.edu>;
'mvapich-discuss at cse.ohio-state.edu'
<mvapich-discuss at mailman.cse.ohio-state.edu>
Cc: 'Brian Osmond' <brian.osmond at noaa.gov>; Raghu Reddy
<raghu.reddy at noaa.gov>
Subject: RE: [mvapich-discuss] OpenACC not supported with mvapich2-gdr 2.3
rpm - Is there another version that does?

Hi Hari,

We have installed the PGI version and GNU version on our system and are
currently trying to test the PGI version and it is complaining about a
missing PMI library, can you please let us know what we may be doing wrong?

Because our images are readonly we use cpio to install the RPM in a
different location, but this is the same process we have used before with
your intel distributions and those have worked fine.

We install the library in /apps instead of /opt and change the wrappers to
point to the new path.

Here are the details:

[Raghu.Reddy at sg001 hello]$ module purge

[Raghu.Reddy at sg001 hello]$ module load slurm cuda/9.2 pgi/18.7
mvapich2-gdr/2.3-1-cuda9.2-pgi17.10-slurm

[Raghu.Reddy at sg001 hello]$

[Raghu.Reddy at sg001 hello]$ mpicc -o foo hello_mpi_c.c

/bin/ld: warning: libpmi.so.0, needed by
/apps/mvapich2-gdr/2.3-1/cuda9.2-pgi17.10-mofed4.2-slurm-2.3-1/lib64/libmpi.
so, not found (try using -rpath or -rpath-link)

/apps/mvapich2-gdr/2.3-1/cuda9.2-pgi17.10-mofed4.2-slurm-2.3-1/lib64/libmpi.
so: undefined reference to `PMI_Get_size'

/apps/mvapich2-gdr/2.3-1/cuda9.2-pgi17.10-mofed4.2-slurm-2.3-1/lib64/libmpi.
so: undefined reference to `PMI_Get_rank'

/apps/mvapich2-gdr/2.3-1/cuda9.2-pgi17.10-mofed4.2-slurm-2.3-1/lib64/libmpi.
so: undefined reference to `PMI_Unpublish_name'

/apps/mvapich2-gdr/2.3-1/cuda9.2-pgi17.10-mofed4.2-slurm-2.3-1/lib64/libmpi.
so: undefined reference to `PMI_KVS_Get_name_length_max'

/apps/mvapich2-gdr/2.3-1/cuda9.2-pgi17.10-mofed4.2-slurm-2.3-1/lib64/libmpi.
so: undefined reference to `PMI_KVS_Get_key_length_max'

/apps/mvapich2-gdr/2.3-1/cuda9.2-pgi17.10-mofed4.2-slurm-2.3-1/lib64/libmpi.
so: undefined reference to `PMI_Finalize'

/apps/mvapich2-gdr/2.3-1/cuda9.2-pgi17.10-mofed4.2-slurm-2.3-1/lib64/libmpi.
so: undefined reference to `PMI_KVS_Get_my_name'

/apps/mvapich2-gdr/2.3-1/cuda9.2-pgi17.10-mofed4.2-slurm-2.3-1/lib64/libmpi.
so: undefined reference to `PMI_Spawn_multiple'

/apps/mvapich2-gdr/2.3-1/cuda9.2-pgi17.10-mofed4.2-slurm-2.3-1/lib64/libmpi.
so: undefined reference to `PMI_Lookup_name'

/apps/mvapich2-gdr/2.3-1/cuda9.2-pgi17.10-mofed4.2-slurm-2.3-1/lib64/libmpi.
so: undefined reference to `PMI_KVS_Get_value_length_max'

/apps/mvapich2-gdr/2.3-1/cuda9.2-pgi17.10-mofed4.2-slurm-2.3-1/lib64/libmpi.
so: undefined reference to `PMI_Barrier'

/apps/mvapich2-gdr/2.3-1/cuda9.2-pgi17.10-mofed4.2-slurm-2.3-1/lib64/libmpi.
so: undefined reference to `PMI_Init'

/apps/mvapich2-gdr/2.3-1/cuda9.2-pgi17.10-mofed4.2-slurm-2.3-1/lib64/libmpi.
so: undefined reference to `PMI_KVS_Put'

/apps/mvapich2-gdr/2.3-1/cuda9.2-pgi17.10-mofed4.2-slurm-2.3-1/lib64/libmpi.
so: undefined reference to `PMI_Initialized'

/apps/mvapich2-gdr/2.3-1/cuda9.2-pgi17.10-mofed4.2-slurm-2.3-1/lib64/libmpi.
so: undefined reference to `PMI_Abort'

/apps/mvapich2-gdr/2.3-1/cuda9.2-pgi17.10-mofed4.2-slurm-2.3-1/lib64/libmpi.
so: undefined reference to `PMI_KVS_Commit'

/apps/mvapich2-gdr/2.3-1/cuda9.2-pgi17.10-mofed4.2-slurm-2.3-1/lib64/libmpi.
so: undefined reference to `PMI_Get_universe_size'

/apps/mvapich2-gdr/2.3-1/cuda9.2-pgi17.10-mofed4.2-slurm-2.3-1/lib64/libmpi.
so: undefined reference to `PMI_Get_appnum'

/apps/mvapich2-gdr/2.3-1/cuda9.2-pgi17.10-mofed4.2-slurm-2.3-1/lib64/libmpi.
so: undefined reference to `PMI_KVS_Get'

/apps/mvapich2-gdr/2.3-1/cuda9.2-pgi17.10-mofed4.2-slurm-2.3-1/lib64/libmpi.
so: undefined reference to `PMI_Publish_name'

[Raghu.Reddy at sg001 hello]$

sfe01% ll
/apps/mvapich2-gdr/2.3-1/cuda9.2-pgi17.10-mofed4.2-slurm-2.3-1/lib64/

total 123200

lrwxrwxrwx 1 Brian.Osmond nesccmgmt        13 Jan 22 20:52 libfmpich.so ->
libmpifort.so*

lrwxrwxrwx 1 Brian.Osmond nesccmgmt        12 Jan 22 20:52 libmpichcxx.so ->
libmpicxx.so*

lrwxrwxrwx 1 Brian.Osmond nesccmgmt        13 Jan 22 20:52 libmpichf90.so ->
libmpifort.so*

lrwxrwxrwx 1 Brian.Osmond nesccmgmt         9 Jan 22 20:52 libmpich.so ->
libmpi.so*

lrwxrwxrwx 1 Brian.Osmond nesccmgmt        19 Jan 22 20:52 libmpicxx.so ->
libmpicxx.so.12.1.1*

lrwxrwxrwx 1 Brian.Osmond nesccmgmt        19 Jan 22 20:52 libmpicxx.so.12
-> libmpicxx.so.12.1.1*

-rwxr-xr-x 1 Brian.Osmond nesccmgmt    488552 Jan 22 20:52
libmpicxx.so.12.1.1*

lrwxrwxrwx 1 Brian.Osmond nesccmgmt        20 Jan 22 20:52 libmpifort.so ->
libmpifort.so.12.1.1*

lrwxrwxrwx 1 Brian.Osmond nesccmgmt        20 Jan 22 20:52 libmpifort.so.12
-> libmpifort.so.12.1.1*

-rwxr-xr-x 1 Brian.Osmond nesccmgmt   6548352 Jan 22 20:52
libmpifort.so.12.1.1*

lrwxrwxrwx 1 Brian.Osmond nesccmgmt        16 Jan 22 20:52 libmpi.so ->
libmpi.so.12.1.1*

lrwxrwxrwx 1 Brian.Osmond nesccmgmt        16 Jan 22 20:52 libmpi.so.12 ->
libmpi.so.12.1.1*

-rwxr-xr-x 1 Brian.Osmond nesccmgmt 118675144 Jan 22 20:52 libmpi.so.12.1.1*

lrwxrwxrwx 1 Brian.Osmond nesccmgmt         9 Jan 22 20:52 libmpl.so ->
libmpi.so*

lrwxrwxrwx 1 Brian.Osmond nesccmgmt         9 Jan 22 20:52 libopa.so ->
libmpi.so*

drwxr-xr-x 2 Brian.Osmond nesccmgmt      2048 Jan 22 20:52 pkgconfig/

sfe01%

Please let us know if you need any additional information that may help in
diagnose and the problem.

Thank you very much for your help!

Thanks,

Raghu

From: Subramoni, Hari [mailto:subramoni.1 at osu.edu] 
Sent: Monday, January 21, 2019 4:17 PM
To: Raghu Reddy <raghu.reddy at noaa.gov <mailto:raghu.reddy at noaa.gov> >;
mvapich-discuss at cse.ohio-state.edu
<mailto:mvapich-discuss at cse.ohio-state.edu>
<mvapich-discuss at mailman.cse.ohio-state.edu
<mailto:mvapich-discuss at mailman.cse.ohio-state.edu> >
Cc: Brian Osmond <brian.osmond at noaa.gov <mailto:brian.osmond at noaa.gov> >;
Subramoni, Hari <subramoni.1 at osu.edu <mailto:subramoni.1 at osu.edu> >
Subject: RE: [mvapich-discuss] OpenACC not supported with mvapich2-gdr 2.3
rpm - Is there another version that does?

Hi, Raghu.

We just meant that the OMB shipped with the RPM does not have support for
OpenACC related benchmarks. The MVAPICH2-GDR codebase should still support
OpenACC.

Thx,

Hari.

From: mvapich-discuss <mvapich-discuss-bounces at cse.ohio-state.edu
<mailto:mvapich-discuss-bounces at cse.ohio-state.edu> > On Behalf Of Raghu
Reddy
Sent: Monday, January 21, 2019 3:50 PM
To: mvapich-discuss at cse.ohio-state.edu
<mailto:mvapich-discuss at cse.ohio-state.edu>
<mvapich-discuss at mailman.cse.ohio-state.edu
<mailto:mvapich-discuss at mailman.cse.ohio-state.edu> >
Cc: Brian Osmond <brian.osmond at noaa.gov <mailto:brian.osmond at noaa.gov> >
Subject: [mvapich-discuss] OpenACC not supported with mvapich2-gdr 2.3 rpm -
Is there another version that does?

Dear MVAPICH-2 GDR team,

We have downloaded the following rpm:

mvapich2-gdr-mcast.cuda9.2.mofed4.2.gnu4.8.5.slurm-2.3-1.el7.x86_64.rpm

When we extract this file, I only the following tree:

               /opt/mvapich2/gdr/2.3/mcast/no-openacc/*

Based on this it looks like OpenACC would not be supported with this RPM.
Is there a version available that support OpenACC?

Thanks,

Raghu

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.cse.ohio-state.edu/pipermail/mvapich-discuss/attachments/20190129/da648bba/attachment-0001.html>