[mvapich-discuss] mvapich 1.1 and ofed 1.3.1

[Jonathan Perkins]

On Wed, Mar 11, 2009 at 12:06:46AM +0100, Pawel Dziekonski wrote:
> On Mon, 09 Mar 2009 at 12:37:50PM -0400, Jonathan Perkins wrote:
> > Pawel:
> > My reply is inline below.
> > 
> > On Mon, Mar 09, 2009 at 04:02:13PM +0100, Pawel Dziekonski wrote:
> > > 
> > > hello,
> > > 
> > > I've just compiled mvapich 1.1 against ofed 1.3.1 and intel compilers on intel
> > > em64t. IB net is based on mthca0's and Voltaire. script and logs are here:
> > > https://cefeid.wcss.wroc.pl/d/tmp/mvapich-1.1/
> > > 
> > > my changes to make.mvapich.gen2 script:
> > > 
> > > diff mvapich-1.1/make.mvapich.gen2 MVAPICH-1.1/make.mvapich.gen2
> > > 84c84
> > > < export LIBS=${LIBS:--L${IBHOME_LIB} -Wl,-rpath=${IBHOME_LIB} -libverbs -libumad -lpthread}
> > > ---
> > > > export LIBS="${IBHOME_LIB}/libibverbs.a ${IBHOME_LIB}/libibumad.a ${IBHOME_LIB}/libibcommon.a -lpthread"
> > > 
> > > as you see, I wanted to have IB and MPI libs statically linked in.
> > 
> > Instead of providing the paths to the static libraries in this manner
> > can you try just replacing the -Wl,rpath=${IBHOME_LIB} portion with
> > -static?  Let us know if this solves your problem.
> 
> In this way it will link everything static, including libc, and fail. Instead
> I used:
> 
> export LIBS="-L/usr/lib64 -Wl,-Bstatic -libverbs -libumad -libcommon -Wl,-Bdynamic -lpthread"

Thanks for the info.

> 
> and it works, however test job still fails with same error. Maybe this is an OFED design?

The output you provided does seem to point to some interaction with the
way the verbs library works.  The Infinihost cards require libmthca.a.
I believe that you'll want to insert -lmthca before -Wl,-Bdynamic.

> 
> regards, Pawel
> 
> -- 
> Pawel Dziekonski <pawel.dziekonski at wcss.pl>
> Wroclaw Centre for Networking & Supercomputing, HPC Department
> Politechnika Wr., pl. Grunwaldzki 9, bud. D2/101, 50-377 Wroclaw, POLAND
> phone: +48 71 3202043, fax: +48 71 3225797, http://www.wcss.wroc.pl
> _______________________________________________
> mvapich-discuss mailing list
> mvapich-discuss at cse.ohio-state.edu
> http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss

-- 
Jonathan Perkins
http://www.cse.ohio-state.edu/~perkinjo
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 197 bytes
Desc: not available
Url : http://mail.cse.ohio-state.edu/pipermail/mvapich-discuss/attachments/20090310/92ede9e4/attachment-0001.bin