[mvapich-discuss] Re: MVAPICH2 for rnfs capable 2.6.17.10 kernel
Sundeep Narravula
narravul at cse.ohio-state.edu
Thu Nov 30 19:18:12 EST 2006
Helen,
> Is the installation directory you mentioned above for the MPI software?
No. I was refering to the iwarp stack installation in my previous mail.
> I am actually trying to figure out how to compile the iwarp-stack
> userspace software. The following are the directory tree containing
> the userspace packages. How do I compile and install them? Is there a
> script fot that?
The instructions for compiling and installing the iwarp branch is
available on the OFA wiki. For chelsio T3 the link is
https://openib.org/tiki/tiki-index.php?page=Install+OpenIB+for+Chelsio+T3
The steps for installing the iwarp stack userspace software are listed
there. I am not aware of any script for that.
> I assume I can then customize the MVAPICH2 Makefile to pick the
> installation target directory?
Yes. Currently the MVAPICH2 make script (make.mvapich2.iwarp) defaults to
/usr/local. This can be changed to your iwarp stack userspace software
installation directory.
--Sundeep.
>
> hycsw at fc14 /root/iWARP/iwarp-2.6.17-stable/userspace $ ls
> dapl libamso libibat libipathverbs libsdp perftest useraccess-lib
> examples libcxgb3 libibcm libmthca management srptools
> imgen libehca libibverbs librdmacm mstflint tvflash
>
> Thanks,
> Helen
>
> -----Original Message-----
> From: Sundeep Narravula [mailto:narravul at cse.ohio-state.edu]
> Sent: Thu 11/30/2006 2:33 PM
> To: Chen, Helen Y
> Cc: panda at cse.ohio-state.edu; mvapich-discuss at cse.ohio-state.edu; Fischer, Noah
> Subject: RE: [mvapich-discuss] Re: MVAPICH2 for rnfs capable 2.6.17.10 kernel
>
> > > Please note that you will need to download and install the iwarp branch
> > > from the OpenFabrics svn located at
> > > https://openib.org/svn/gen2/branches/iwarp/tags/iwarp-2.6.17-stable
> >
> > How do I compile and install the userspace libraries and utilities after I download
> > the iwarp branch without replacing the stack of the running kernel and modules?
>
> In theory, you should be able to configure all the userspace libraries
> with an installation target directory specified other than the default
> (where your existing installation would be).
>
> $ ./autogen.sh ; ./configure --help
> ....
> By default, `make install' will install all the files in
> `/usr/local/bin', `/usr/local/lib' etc. You can specify
> an installation prefix other than `/usr/local' using `--prefix',
> for instance `--prefix=$HOME'.
> ....
>
>
> However, we have not tried this kind of setup of the iwarp branch.
>
> Hope that helps.
>
> --Sundeep.
>
>
>
> >
> > Thanks,
> > Helen
> > -----Original Message-----
> > From: Sundeep Narravula [mailto:narravul at cse.ohio-state.edu]
> > Sent: Thu 11/30/2006 12:56 PM
> > To: Chen, Helen Y
> > Cc: panda at cse.ohio-state.edu; mvapich-discuss at cse.ohio-state.edu; Fischer, Noah
> > Subject: Re: [mvapich-discuss] Re: MVAPICH2 for rnfs capable 2.6.17.10 kernel
> >
> > Hi Helen,
> >
> > The instructions for building and installing MVAPICH2 using iWARP are
> > available in our user guide at
> > http://nowlab.cse.ohio-state.edu/projects/mpi-iba/download-mvapich2/mvapich2_user_guide.html#x1-110004.4.2
> > http://nowlab.cse.ohio-state.edu/projects/mpi-iba/download-mvapich2/mvapich2_user_guide.html
> >
> > I am listing the basic instructions here for your conveniance.
> >
> > -------------------------------------------
> > Using Default build configuration:
> >
> > Go to the mvapich2-0.9.8 directory. We have included a single build script
> > for Gen2-iWARP (make.mvapich2.iwarp) that takes care of different
> > platforms, compilers and architectures. By default, the compilation script
> > uses gcc. In order to use a specific compiler, set the CC, CXX, F77, and
> > F90 variables to the appropriate commands. MVAPICH2 currently supports the
> > GCC, Intel and PGI compiler suites for Gen2-iWARP.
> >
> > Please note that you will need to download and install the iwarp branch
> > from the OpenFabrics svn located at
> > https://openib.org/svn/gen2/branches/iwarp/tags/iwarp-2.6.17-stable
> >
> > After setting all the required parameters, the script make.mvapich2.iwarp
> > configures, builds and installs the entire package in the directory
> > specified by the variable PREFIX.
> >
> >
> >
> > Running MPI applications with Gen2-iWARP support:
> >
> > In MVAPICH2, Gen2-iWARP support is enabled with the two CFLAGS: (i)
> > -DRDMA_CM and (ii) -DRDMA_CM_RNIC. These can be enabled from our build
> > script (make.mvapich2.iwarp or make.mvapich2.gen2).
> >
> > In addition to these, the system needs further one time setup for enabling
> > RDMA_CM usage. The required steps are as follows:
> >
> > 1. Setup the OpenFabrics iWARP Branch: The OpenFabrics iwarp branch
> > located at https://openib.org/svn/gen2/branches/iwarp/tags/iwarp-2.6.17-stable needs
> > to be installed. Detailed instructions for this installation are
> > available on the OpenFabrics Wiki at
> > https://openib.org/tiki/tiki-index.php?page=Install+OpenIB+for+Chelsio+T3
> >
> > 2. Setup the RDMA_CM device: RDMA_CM device needs to be setup, configured
> > with an IP address and connected to the network.
> >
> > 3. Setup the Local Address File: Create the file (/etc/mv2.conf) with the
> > local IP address to be used by RDMA_CM. Example:
> > $ echo 10.1.1.1 >> /etc/mv2.conf
> >
> > -------------------------------------------
> >
> > Further troubleshooting instructions are present at:
> >
> > http://nowlab.cse.ohio-state.edu/projects/mpi-iba/download-mvapich2/mvapich2_user_guide.html#x1-400007.3
> >
> > Regards,
> > --Sundeep.
> >
> >
> >
> >
> > On Thu, 30 Nov 2006, Chen, Helen Y wrote:
> >
> > > Hi,
> > >
> > > We are running a 2.6.17.10 kernel with rnfs patch and the stable
> > > iwarp-stack from openib. We'd like to installed MVAPICH2 that can use
> > > either IB or iWARP RDMA. What are the exact instructions for building
> > > the package? We are new at this so any help from you will be
> > > appreciated.
> > >
> > > Thanks,
> > > Helen
> > >
> > >
> > >
> > > -----Original Message-----
> > > From: Dhabaleswar Panda [mailto:panda at cse.ohio-state.edu]
> > > Sent: Thu 11/30/2006 12:13 PM
> > > To: Chen, Helen Y
> > > Cc: panda at cse.ohio-state.edu; Chen, Helen Y
> > > Subject: Re: [openib-general] openMPI for 2.6.17.10 kernel
> > >
> > > Hi Helen,
> > >
> > > > Thanks for your information. Actually my student had tried to install =
> > > > MVAPICH2 and failed, which I believe is because we are new at this. So =
> > > > my questions remains how to keep my iwarp-satck intact and install =
> > > > MVAPICH2 on i
> > > > the rnfs capable 2.6.17.10 kernel? =20
> > >
> > > Thanks for your prompt reply.
> > >
> > > Sorry to know that you are experiencing problem here. We are able to
> > > run it sucessfully in our lab. Steve Wise is also able to run it.
> > >
> > > I believe you are trying to use it with Chelsio adapter. I am
> > > wondering whether your student took a detailed look at the MVAPICH2
> > > 0.9.8 user guide (available from mvapich page). It has a detailed set
> > > of instructions on how to install and run MVAPICH2 with the iWARP
> > > stack.
> > >
> > > We will be happy to extend help here so that you can use MVAPICH2 with
> > > the iWARP stack. May I request your student to send out a note to us
> > > using our public `mvapich-discuss at cse.ohio-state.edu' mailing list
> > > with the problems he is encountering and we will be happy to help.
> > >
> > > Best Regards,
> > >
> > > DK
> > >
> > > > Thanks,
> > > > Helen
> > > >
> > > > -----Original Message-----
> > > > From: Dhabaleswar Panda [mailto:panda at cse.ohio-state.edu]
> > > > Sent: Thu 11/30/2006 10:58 AM
> > > > To: Chen, Helen Y
> > > > Cc: panda at cse.ohio-state.edu
> > > > Subject: Re: [openib-general] openMPI for 2.6.17.10 kernel
> > > > =20
> > > > Hi Helen,=20
> > > >
> > > > > I am running a 2.6.17.10 kernel with Tom Tucker's rnfs patch and would =
> > > > =3D
> > > > > like to build Open MPI that uses either IB or iWARP RDMA through this =
> > > > =3D
> > > > > kernel. Can I achieve this by getting the latest stable git and =
> > > > compile =3D
> > > > > just the userspace software? If so, how do I go about getting the =
> > > > latest =3D
> > > > > stable git, and how to compile just the userspace stuff? I am new at =
> > > > =3D
> > > > > this and would really appreciate your help.
> > > >
> > > > This is just a private e-mail I am sending to you. Not cc'ing to OpenIB.
> > > >
> > > > Not sure whether you have noticed that our latest MVAPICH2 0.9.8
> > > > release (done immediately before SC '06) supports RDMA CM and can run
> > > > on both IB and iWARP hardware. It has been tested with Chelsio T3
> > > > adapter with iWARP support and OFED 1.1 stack with 2.6.17 kernel. I
> > > > think this is what you are looking for. This release also has added
> > > > features like Checkpoint-Restart for IB.
> > > >
> > > > If you are interested in running a single MPI stack for both IB and
> > > > iWARP, you may try this release.
> > > >
> > > > We will be happy to get your feedbacks.=20
> > > >
> > > > Best Regards,=20
> > > >
> > > > DK
> > > >
> > > >
> > > > > Thanks,
> > > > > Helen
> > > > >=20
> > > > > ------_=3D_NextPart_001_01C71415.11E95606
> > > > > Content-Type: text/html;
> > > > > charset=3Diso-8859-1
> > > > > Content-Transfer-Encoding: quoted-printable
> > > > >=20
> > > > > <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2//EN">
> > > > > <HTML>
> > > > > <HEAD>
> > > > > <META HTTP-EQUIV=3D3D"Content-Type" CONTENT=3D3D"text/html; =3D
> > > > > charset=3D3Diso-8859-1">
> > > > > <META NAME=3D3D"Generator" CONTENT=3D3D"MS Exchange Server version =3D
> > > > > 6.5.7638.1">
> > > > > <TITLE>Re: openMPI for 2.6.17.10 kernel</TITLE>
> > > > > </HEAD>
> > > > > <BODY>
> > > > > <!-- Converted from text/plain format -->
> > > > >=20
> > > > > <P><FONT SIZE=3D3D2>Hi,<BR>
> > > > > <BR>
> > > > > I am running a 2.6.17.10 kernel with Tom Tucker's rnfs patch and would =
> > > > =3D
> > > > > like to build Open MPI that uses either IB or iWARP RDMA through this =
> > > > =3D
> > > > > kernel. Can I achieve this by getting the latest stable git and =
> > > > compile =3D
> > > > > just the userspace software? If so, how do I go about getting the =
> > > > latest =3D
> > > > > stable git, and how to compile just the userspace stuff? I am =
> > > > new =3D
> > > > > at this and would really appreciate your help.<BR>
> > > > > <BR>
> > > > > Thanks,<BR>
> > > > > Helen<BR>
> > > > > </FONT>
> > > > > </P>
> > > > >=20
> > > > > </BODY>
> > > > > </HTML>
> > > > > ------_=3D_NextPart_001_01C71415.11E95606--
> > > > >=20
> > > > > --=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D0218902636=3D=3D
> > > > > MIME-Version: 1.0
> > > > > Content-Type: text/plain;
> > > > > charset=3Dus-ascii
> > > > > Content-Transfer-Encoding: 7bit
> > > > > Content-Disposition: inline
> > > > >=20
> > > > > _______________________________________________
> > > > > openib-general mailing list
> > > > > openib-general at openib.org
> > > > > http://openib.org/mailman/listinfo/openib-general
> > > > >=20
> > > > > To unsubscribe, please visit =
> > > > http://openib.org/mailman/listinfo/openib-general
> > > > > --=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D0218902636=3D=3D--
> > > > >=20
> > > >
> > > >
> > > >
> > > > ------_=_NextPart_001_01C714B0.E043C99C
> > > > Content-Type: text/html;
> > > > charset=iso-8859-1
> > > > Content-Transfer-Encoding: quoted-printable
> > > >
> > > > <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2//EN">
> > > > <HTML>
> > > > <HEAD>
> > > > <META HTTP-EQUIV=3D"Content-Type" CONTENT=3D"text/html; =
> > > > charset=3Diso-8859-1">
> > > > <META NAME=3D"Generator" CONTENT=3D"MS Exchange Server version =
> > > > 6.5.7638.1">
> > > > <TITLE>RE: [openib-general] openMPI for 2.6.17.10 kernel</TITLE>
> > > > </HEAD>
> > > > <BODY>
> > > > <!-- Converted from text/plain format -->
> > > >
> > > > <P><FONT SIZE=3D2>Panda,<BR>
> > > > <BR>
> > > > Thanks for your information. Actually my student had tried to =
> > > > install MVAPICH2 and failed, which I believe is because we are new at =
> > > > this. So my questions remains how to keep my iwarp-satck intact =
> > > > and install MVAPICH2 on i<BR>
> > > > the rnfs capable 2.6.17.10 kernel? <BR>
> > > > <BR>
> > > > Thanks,<BR>
> > > > Helen<BR>
> > > > <BR>
> > > > -----Original Message-----<BR>
> > > > From: Dhabaleswar Panda [<A =
> > > > HREF=3D"mailto:panda at cse.ohio-state.edu">mailto:panda at cse.ohio-state.edu<=
> > > > /A>]<BR>
> > > > Sent: Thu 11/30/2006 10:58 AM<BR>
> > > > To: Chen, Helen Y<BR>
> > > > Cc: panda at cse.ohio-state.edu<BR>
> > > > Subject: Re: [openib-general] openMPI for 2.6.17.10 kernel<BR>
> > > > <BR>
> > > > Hi Helen,<BR>
> > > > <BR>
> > > > > I am running a 2.6.17.10 kernel with Tom Tucker's rnfs patch and =
> > > > would =3D<BR>
> > > > > like to build Open MPI that uses either IB or iWARP RDMA through =
> > > > this =3D<BR>
> > > > > kernel. Can I achieve this by getting the latest stable git and =
> > > > compile =3D<BR>
> > > > > just the userspace software? If so, how do I go about getting the =
> > > > latest =3D<BR>
> > > > > stable git, and how to compile just the userspace stuff? I am =
> > > > new at =3D<BR>
> > > > > this and would really appreciate your help.<BR>
> > > > <BR>
> > > > This is just a private e-mail I am sending to you. Not cc'ing to =
> > > > OpenIB.<BR>
> > > > <BR>
> > > > Not sure whether you have noticed that our latest MVAPICH2 0.9.8<BR>
> > > > release (done immediately before SC '06) supports RDMA CM and can =
> > > > run<BR>
> > > > on both IB and iWARP hardware. It has been tested with Chelsio T3<BR>
> > > > adapter with iWARP support and OFED 1.1 stack with 2.6.17 kernel. I<BR>
> > > > think this is what you are looking for. This release also has added<BR>
> > > > features like Checkpoint-Restart for IB.<BR>
> > > > <BR>
> > > > If you are interested in running a single MPI stack for both IB and<BR>
> > > > iWARP, you may try this release.<BR>
> > > > <BR>
> > > > We will be happy to get your feedbacks.<BR>
> > > > <BR>
> > > > Best Regards,<BR>
> > > > <BR>
> > > > DK<BR>
> > > > <BR>
> > > > <BR>
> > > > > Thanks,<BR>
> > > > > Helen<BR>
> > > > ><BR>
> > > > > ------_=3D_NextPart_001_01C71415.11E95606<BR>
> > > > > Content-Type: text/html;<BR>
> > > > > charset=3Diso-8859-1<BR>
> > > > > Content-Transfer-Encoding: quoted-printable<BR>
> > > > ><BR>
> > > > > <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML =
> > > > 3.2//EN"><BR>
> > > > > <HTML><BR>
> > > > > <HEAD><BR>
> > > > > <META HTTP-EQUIV=3D3D"Content-Type" =
> > > > CONTENT=3D3D"text/html; =3D<BR>
> > > > > charset=3D3Diso-8859-1"><BR>
> > > > > <META NAME=3D3D"Generator" CONTENT=3D3D"MS =
> > > > Exchange Server version =3D<BR>
> > > > > 6.5.7638.1"><BR>
> > > > > <TITLE>Re: openMPI for 2.6.17.10 kernel</TITLE><BR>
> > > > > </HEAD><BR>
> > > > > <BODY><BR>
> > > > > <!-- Converted from text/plain format --><BR>
> > > > ><BR>
> > > > > <P><FONT SIZE=3D3D2>Hi,<BR><BR>
> > > > > <BR><BR>
> > > > > I am running a 2.6.17.10 kernel with Tom Tucker's rnfs patch and =
> > > > would =3D<BR>
> > > > > like to build Open MPI that uses either IB or iWARP RDMA through =
> > > > this =3D<BR>
> > > > > kernel. Can I achieve this by getting the latest stable git and =
> > > > compile =3D<BR>
> > > > > just the userspace software? If so, how do I go about getting the =
> > > > latest =3D<BR>
> > > > > stable git, and how to compile just the userspace stuff?&nbsp; =
> > > > I am new =3D<BR>
> > > > > at this and would really appreciate your help.<BR><BR>
> > > > > <BR><BR>
> > > > > Thanks,<BR><BR>
> > > > > Helen<BR><BR>
> > > > > </FONT><BR>
> > > > > </P><BR>
> > > > ><BR>
> > > > > </BODY><BR>
> > > > > </HTML><BR>
> > > > > ------_=3D_NextPart_001_01C71415.11E95606--<BR>
> > > > ><BR>
> > > > > --=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D0218902636=3D=3D<BR>
> > > > > MIME-Version: 1.0<BR>
> > > > > Content-Type: text/plain;<BR>
> > > > > charset=3Dus-ascii<BR>
> > > > > Content-Transfer-Encoding: 7bit<BR>
> > > > > Content-Disposition: inline<BR>
> > > > ><BR>
> > > > > _______________________________________________<BR>
> > > > > openib-general mailing list<BR>
> > > > > openib-general at openib.org<BR>
> > > > > <A =
> > > > HREF=3D"http://openib.org/mailman/listinfo/openib-general">http://openib.=
> > > > org/mailman/listinfo/openib-general</A><BR>
> > > > ><BR>
> > > > > To unsubscribe, please visit <A =
> > > > HREF=3D"http://openib.org/mailman/listinfo/openib-general">http://openib.=
> > > > org/mailman/listinfo/openib-general</A><BR>
> > > > > =
> > > > --=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D0218902636=3D=3D--<BR>
> > > > ><BR>
> > > > <BR>
> > > > <BR>
> > > > </FONT>
> > > > </P>
> > > >
> > > > </BODY>
> > > > </HTML>
> > > > ------_=_NextPart_001_01C714B0.E043C99C--
> > > >
> > >
> > >
> > >
> >
> >
> >
>
>
>
More information about the mvapich-discuss
mailing list