[mvapich-discuss] recompile with -fPIC

Fri Sep 7 19:05:22 EDT 2012

Hello Jonathan,

Apologies for the extremely tardy response. Just to close the loop on this A) I am using bash B) I did include all variables in my bashrc however I'm still running into the same error as before. For debugging purposes I placed a :

echo LD_LIBRARY_PATH FROM BASHRC
echo ===========================
echo $LD_LIBRARY_PATH
echo ===========================

at the end of my bashrc file and 

echo "Reading ~/.bashrc"

at the beginning.

The log shows that bashrc runs and LD_LIBRARY_PATH includes the folder /users/p20558/apps/mvapich2-1.8-gcc/lib which has the missing libmpl.so.1 library. This occurs right after the ssh calls hydra_pmi_proxy on the second node & it's also present in the Global environment section above. I've attached the log for viewing. 

I got mpirun_rsh to run and managed to run a few benchmarks on 2 nodes of 12 processors. Note that when I tried to run OpenFOAM it yielded the same error. I've attached the log to this email. 

I'm not exactly sure how to proceed from here, it appears to me that the library should be found but at the same time I've got a lot of junk in my environment which might not allow it to be found in time. I'm proceeding now as if something is wrong with my environment which is not exclusive to mvapich. Should you see anything odd/enlightening in either log please let me know.

Thanks

Scott Haynes
Senior Meteorologist
Wind Development
Iberdrola Renewables
1320 Pearl St., Suite 330, Boulder, CO, 80302
Telephone (303) 245-8156, Mobile (503) 841-8524, Fax (303) 245-8155
scott.haynes at iberdrolaren.com
  In the interests of the environment, please print only if necessary and recycle

-----Original Message-----
From: Jonathan Perkins [mailto:perkinjo at cse.ohio-state.edu] 
Sent: Friday, August 24, 2012 6:26 PM
To: Haynes, Scott
Cc: mvapich-core at cse.ohio-state.edu
Subject: Re: [mvapich-discuss] recompile with -fPIC

Hi Scott, I'm cc'ing an internal developer list.

On Fri, Aug 24, 2012 at 05:23:43PM -0700, Haynes, Scott wrote:
> Unfortunately putting LD_LIBRARY_PATH in my ~/.bashrc didn't work &
> yields the same error.

Are you using bash or some other shell?

> 
> Also, mpirun_rsh doesn't work in the version I'm working with.
> mvapich2 version 1.6 was previously installed on our cluster and
> mpirun_rsh never worked in that version so I'd prefer to keep
> debugging hydra (at least until this horse is dead).

There have been a few bugs and improvements to mpirun_rsh since 1.6.
Not sure what issue you faced there.

Here is the latest changelog for mvapich2.
http://mvapich.cse.ohio-state.edu/download/mvapich2/changes.shtml

Also here is the 1.8 userguide.
http://mvapich.cse.ohio-state.edu/support/user_guide_mvapich2-1.8.html

> 
> Can you think of another way to get LD_LIBRARY_PATH into the
> environment before hydra_pmi_proxy starts running?

No, if the startup rc file for the shell isn't being used I'm not sure
what else would.  You could try adding the path in a /etc/ld.so.conf.d/
file but that isn't very flexible in allowing multiple installs.

> 
> Thanks
> Scott
> 
> Scott Haynes
> Senior Meteorologist
> Wind Development
> Iberdrola Renewables
> 1320 Pearl St., Suite 330, Boulder, CO, 80302
> Telephone (303) 245-8156, Mobile (503) 841-8524, Fax (303) 245-8155
> scott.haynes at iberdrolaren.com
>   In the interests of the environment, please print only if necessary and recycle
>  
> 
> -----Original Message-----
> From: Jonathan Perkins [mailto:perkinjo at cse.ohio-state.edu] 
> Sent: Friday, August 24, 2012 3:55 PM
> To: Haynes, Scott
> Cc: mvapich-discuss at cse.ohio-state.edu
> Subject: Re: [mvapich-discuss] recompile with -fPIC
> 
> On Fri, Aug 24, 2012 at 03:13:38PM -0700, Haynes, Scott wrote:
> > Thanks Jonathan,
> > 
> > That worked .... I feel dumb for asking now. 
> 
> :-), I'm just happy my suggestion worked.
> 
> > 
> > After re-compiling I was able to run several of the OSU benchmarks
> > including osu_alltoall on 2 nodes of 8 processors so I can confirm
> > that mvapich2-1.8 is functional. Also the Pstream lib in OpenFOAM
> > linked to the .so with no issues so I was pleased to see that is
> > working now too. 
> > 
> > When I tried to run a test OpenFOAM case decomposed over 20
> > processors, i.e. 2 nodes of 10 processors I got the following error.
> > 
> > =====(error)=======
> > [mpiexec at compute-0-9.local] Launch arguments: /usr/bin/ssh -x
> > compute-0-10 "/users/p20558/apps/mvapich2-1.8-gcc/bin/hydra_pmi_proxy"
> > --control-port compute-0-9:54875 --debug --rmk user --launcher ssh
> > --demux poll --pgid 0 --retries 10 --proxy-id 1 
> > /users/p20558/apps/mvapich2-1.8-gcc/bin/hydra_pmi_proxy: error while loading shared libraries: libmpl.so.1: cannot open shared object file: No such file or directory
> > SHbuoyantPimpleFoam: symbol lookup error:
> > /users/p20558/apps/mvapich2-1.8-gcc/lib/libmpich.so.3: undefined
> > symbol: MPL_env2str
> > ====================
> > 
> > 
> > I verified that libmpl.so.1 was in my LD_LIBRARY_PATH and I even used the -env option to specify both LD_LIBRARY_PATH & PATH variables ensuring that the path to libmpl.so.1 was included, again it didn't work. Next I tried to run the 20 processor job on one node & I received the error below:
> > 
> > =====(error)=======
> > [mpiexec at compute-0-9.local] Launch arguments:
> > /users/p20558/apps/mvapich2-1.8-gcc/bin/hydra_pmi_proxy --control-port
> > compute-0-9:55519 --debug --rmk user --launcher ssh --demux poll
> > --pgid 0 --retries 10 --proxy-id 0 
> > SHbuoyantPimpleFoam: symbol lookup error:
> > /users/p20558/apps/mvapich2-1.8-gcc/lib/libmpich.so.3: undefined
> > symbol: MPL_env2str
> > ====================
> > 
> > This time it skips the libmpl.so.1 part and goes straight to the
> > "undefined symbol: MPL_env2str"
> > 
> > A " nm ~/apps/mvapich2-1.8-gcc/lib/libmpl.so.1 | grep MPL_env2str"
> > confirms that MPL_env2str is in libmpl.so.1 so it seems both of these
> > errors are related to the failure to find libmpl.so.1
> > 
> > I am wondering if I need to perform some additional environmental
> > setup to make use of shared libraries, they're used all over the place
> > in OpenFOAM.  I have attached to this email my -v log from mpiexec for
> > the error listed above.
> > 
> > Please let me know if you see anything obviously wrong or if I can
> > provide any additional information.
> 
> I've seen this issue before when using hydra.  This should not happen if
> you use mpirun_rsh instead.
> 
> If you want this to work with hydra try setting LD_LIBRARY_PATH in your
> .bashrc.  The problem is because the LD_LIBRARY_PATH doesn't take affect
> until after hydra_pmi_proxy is running.
> 
> > 
> > Thanks & Best Regards
> > 
> > Scott 
> > 
> > 
> > 
> > Scott Haynes
> > Senior Meteorologist
> > Wind Development
> > Iberdrola Renewables
> > 1320 Pearl St., Suite 330, Boulder, CO, 80302
> > Telephone (303) 245-8156, Mobile (503) 841-8524, Fax (303) 245-8155
> > scott.haynes at iberdrolaren.com
> >   In the interests of the environment, please print only if necessary and recycle
> >  
> > 
> > 
> > -----Original Message-----
> > From: Jonathan Perkins [mailto:perkinjo at cse.ohio-state.edu] 
> > Sent: Friday, August 24, 2012 6:48 AM
> > To: Haynes, Scott
> > Cc: mvapich-discuss at cse.ohio-state.edu
> > Subject: Re: [mvapich-discuss] recompile with -fPIC
> > 
> > Hello Scott.
> > 
> > Try building mvapich2 with the following configure...
> >     ./configure --with-slurm=../slurm/usr/local/ --enable-shared
> > 
> > Then make sure OpenFOAM links against the libmpich.so instead of
> > libmpich.a.
> > 
> > On Thu, Aug 23, 2012 at 04:58:10PM -0700, Haynes, Scott wrote:
> > > Hello List,
> > > 
> > >  
> > > 
> > > I have been trying to build OpenFOAM 2.1 with mvapich2-1.8-r5609.  The
> > > error I am receiving is pasted below.  "
> > > 
> > >  
> > > 
> > > ===== (Error)======
> > > 
> > > /users/p20558/apps/binutils/bin/ld:
> > > /users/p20558/apps/mvapich2-1.8-gcc/lib/libmpich.a(setbotf.o):
> > > relocation R_X86_64_32 against `mpifcmb2_' can not be used when making a
> > > shared object; recompile with -fPIC
> > > 
> > > /users/p20558/apps/mvapich2-1.8-gcc/lib/libmpich.a: could not read
> > > symbols: Bad value
> > > 
> > > collect2: ld returned 1 exit status
> > > 
> > > =================
> > > 
> > >  
> > > 
> > > I am using gcc 4.6 to compile both mvapich2 & OpenFOAM with the -fPIC
> > > flag (position independent code). I think the error above is the linker
> > > telling me that mvapich2 is not PIC & I need to recompile libmpich.a  .
> > > I have tried rebuilding mvapich2 using several different configurations
> > > each time libmpich.a is not PIC & I have been unsuccessful changing
> > > that.  My best guess at a configure line is below:
> > > 
> > >  
> > > 
> > > ./configure --with-slurm=../slurm/usr/local/ --enable-shared=gcc
> > > CFLAGS=-fPIC CXXFLAGS=-fPIC CPPFLAGS=-fPIC
> > > 
> > >  
> > > 
> > > I assume it is possible to compile mvapich2 as PIC? Is there something
> > > obvious I could add or delete from the configure statement above?  Any
> > > other debugging information I could provide?
> > > 
> > >  
> > > 
> > > Thanks & Best Regards
> > > 
> > >  
> > > 
> > > Scott 
> > > 
> > >  
> > > 
> > >  
> > > 
> > > Scott Haynes
> > > Senior Meteorologist
> > > Wind Development
> > > Iberdrola Renewables
> > > 
> > > 1320 Pearl St., Suite 330, Boulder, CO, 80302
> > > Telephone (303) 245-8156, Mobile (503) 841-8524, Fax (303) 245-8155
> > > 
> > > scott.haynes at iberdrolaren.com
> > > P  In the interests of the environment, please print only if necessary
> > > and recycle
> > > 
> > >  
> > > 
> > >  
> > > 
> > > 
> > > This message is intended for the exclusive attention of the recipient(s) indicated.  Any information contained herein is strictly confidential and privileged.  If you are not the intended recipient, please notify us by return e-mail and delete this message from your computer system. Any unauthorized use, reproduction, alteration, filing or sending of this message and/or any attached files may lead to legal action being taken against the party(ies) responsible for said unauthorized use. Any opinion expressed herein is solely that of the author(s) and does not necessarily represent the opinion of the Company. The sender does not guarantee the integrity, speed or safety of this message, and does not accept responsibility for any possible damage arising from the interception, incorporation of viruses, or any other damage as a result of manipulation.
> > > 
> > > 
> > 
> > 
> > 
> > > _______________________________________________
> > > mvapich-discuss mailing list
> > > mvapich-discuss at cse.ohio-state.edu
> > > http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss
> > 
> > 
> > -- 
> > Jonathan Perkins
> > http://www.cse.ohio-state.edu/~perkinjo
> 
> 
> 
> -- 
> Jonathan Perkins
> http://www.cse.ohio-state.edu/~perkinjo
> 

-- 
Jonathan Perkins
http://www.cse.ohio-state.edu/~perkinjo
-------------- next part --------------
A non-text attachment was scrubbed...
Name: mpiexec.log
Type: application/octet-stream
Size: 34016 bytes
Desc: mpiexec.log
Url : http://mail.cse.ohio-state.edu/pipermail/mvapich-discuss/attachments/20120907/c90d42c3/mpiexec-0001.obj
-------------- next part --------------
A non-text attachment was scrubbed...
Name: mpirun_rsh.log
Type: application/octet-stream
Size: 9816 bytes
Desc: mpirun_rsh.log
Url : http://mail.cse.ohio-state.edu/pipermail/mvapich-discuss/attachments/20120907/c90d42c3/mpirun_rsh-0001.obj