[mvapich-discuss] Problems installing mvapich2/2.3 with Slurm

[Raghu Reddy]

Hi Peter/Hari,

We installed the psm rpms (but did not remove psm2 rpms):

sfe01% rpm -qa | grep psm
infinipath-psm-3.3-26_g604758e_open.2.el7.x86_64
psmisc-22.20-15.el7.x86_64
libpsm2-10.3.58-1.el7.x86_64
infinipath-psm-devel-3.3-26_g604758e_open.2.el7.x86_64
libpsm2-devel-10.3.58-1.el7.x86_64
sfe01%

Our configure line was:

./configure --prefix=$INSTALLDIR --with-device=ch3:psm --enable-romio=yes
--enable-shared -enable-fortran=yes CC=icc CXX=icpc F77=ifort FC=ifort | &
tee configure-$t.out-rr

And yet, the make file seems to use the psm2 library (extracted from the
output of make):

libtool: install: (cd /home/Raghu.Reddy/S2/apps/mvapich2-2.3; /bin/sh
"/home/Raghu.Reddy/S2/apps/mvapich2-2.3/libtool"  --silent --tag CXX
--mode=relink icpc -DNDEBUG -DNVAL\
GRIND -O2 -version-info 13:1:1 -L/lib -Wl,-rpath,/lib -L/lib -Wl,-rpath,/lib
-o lib/libmpicxx.la -rpath /apps/mvapich2/2.3-pbs/intel/18.0.3.222/lib
src/binding/cxx/initcxx.l\
o lib/libmpi.la -libverbs -lpsm2 -lpthread )

The build completes successfully, but hang when trying to run hello world.

Hari, is there a separate device for psm vs psm2?  That is:

	--with-device=ch3:psm		for TruScale
	--with-device=ch3:psm2		for OPA

Maybe as Peter suggested, we need to remove the psm2 libraries?

I can include the full configure output and make output if that will be
helpful.

Thanks,
Raghu




-----Original Message-----
From: Peter Kjellström [mailto:cap at nsc.liu.se] 
Sent: Wednesday, March 13, 2019 11:02 AM
To: Raghu Reddy <raghu.reddy at noaa.gov>
Cc: 'Subramoni, Hari' <subramoni.1 at osu.edu>; 'Carlson, Timothy S'
<Timothy.Carlson at pnnl.gov>; 'mvapich-discuss at cse.ohio-state.edu'
<mvapich-discuss at mailman.cse.ohio-state.edu>; 'Brian Osmond'
<brian.osmond at noaa.gov>; 'Kyle Stern' <kstern at redlineperf.com>
Subject: Re: [mvapich-discuss] Problems installing mvapich2/2.3 with Slurm

On Wed, 13 Mar 2019 10:40:30 -0400
Raghu Reddy <raghu.reddy at noaa.gov> wrote:

> Hi Peter,
> 
> Thank you very much for this information!  We are trying to follow up 
> on this, this information is very helpful!
> 
> This is what we currently have:
> 
> tfe03.% rpm -qa | grep psm
> infinipath-psm-3.3-26_g604758e_open.2.el7.x86_64
> psmisc-22.20-15.el7.x86_64
> libpsm2-devel-10.3.58-1.el7.x86_64
> libpsm2-10.3.58-1.el7.x86_64
> tfe03.%
> 
> We will try to install the "psm" libraries that you have mentioned and 
> try again.

You don't have to have my exact version. But you must have an
infinipath-psm-devel (or else ch3:psm cannot build for psm).

Also it may or may not need you to remove libpsm2 and its devel. If you
don't have OPA they are not needed anyway.

/Peter