[mvapich-discuss] MPI-2 over ibverbs/rdma

[Vivek Satpute]

Hi Jonathan,

On Wed, Nov 25, 2009 at 10:17 PM, Jonathan Perkins <
perkinjo at cse.ohio-state.edu> wrote:

> On Wed, Nov 25, 2009 at 08:59:44PM +0530, Vivek Satpute wrote:
> > Hi Jonathan, On Wed, Nov 25, 2009 at 8:21 PM, Jonathan Perkins wrote:
> > > On Wed, Nov 25, 2009 at 02:04:13PM +0530, Vivek Satpute wrote:
> > > > On Wed, Nov 25, 2009 at 1:38 PM, Dhabaleswar Panda wrote:
> > > > > You can use the OpenFabrics-Gen2/IB interface of MVAPICH2 to run
> > > > > it directly over ibverbs.
> > > >
> > > > If  I configured above then for executing MPI application from
> > > > node1 to node2, node3 and node4 (e.g mpirun_rsh -np 4 n0 n1 n2 n3
> > > > ./cpi), I will have to mention a IB interfaces of each n0, n1, n3,
> > > > n4. So ultimately MPI will use IPoIB module. Am i right ?
> > >
> > > You shouldn't need to specify any additional information.  The
> > > mpirun_rsh command only needs to know which nodes you're trying to
> > > launch mpi program on.  It appears that you're basing your questions
> > > on some additional assumptions.  Are you asking this question based
> > > on some problem that you're run into?
> >
> > Yes, you are right. :-) I think I should explain my problem and
> > assumptions here :-)
> >
> > I am working on OFED-1.4, it has many stuffs related to Infiniband. I
> > have installed OFED on Infiniband hardwares, and IB devices are active
> > and they are working similar to normal ethernet devices. Means I can
> > ping/ssh to remote machine through IB devices.
>
> So, it looks like you have configured IPoIB so normal TCP/IP socket
> programs can work over your shiny new Infiniband network.
>
> Yope

> >
> > Whenever I send ping packets or do ssh to remote machine, all traffic
> > goes through IPoIB module, mean it uses IPoIB code path.
>
> Yep.
>
> >
> > My understanding/assumption is, Infiniband also work without using
> > IPoIB.  So, I want to find out application/utility which work without
> > IPoIB module.  All the traffic generated by such xyz application
> > should go to remote machine without IPoIB so for achieving this I can
> > use LID/GID of IB devices.
>
> These programs are written with Infiniband in mind and do not use
> sockets but instead the ibverbs API.
>
> Exactly.. I am looking for these sample programs.

> Is there any such technology/utility/application exists which work
> > without IPoIB ?  Is there any packet path which goes to remote node
> > without passing through IPoIB ?
>
> Yes, but these aren't usually general use applications.
>
> Ok. OFED-1.4 has few sample programs in libibverbs-1.1.2 package. But
all those programs expect IP address of remote machine. So those are not
I am looking for as they will pass through IPoIB module. Could you please
point me to sample applications/programs which uses purely ibverbs APIs ?

And why those examples in libibverbs-1.1.2 package takes IP address ?
Do you have any idea about that ?

> > My this assumption can be wrong so please correct me if I am wrong ..
> >
> > [  As per my understanding libibverbs are below the IPoIB. So
> > applications (examples of libibvers in OFED-1.4) should work on
> > LID/GID, but those applications also uses IP address means ultimately
> > packets of those application also passes through IPoIB module. ]
>
> It may be good for you to subscribe to the linux-rdma at vger.kernel.orga
> list and learn more about this technology there.  You can also take a
> look at http://www.openfabrics.org/downloads/OFA_Backgrounder.pdf.  It
> doesn't seem that your questions are related to using the MPI API to
> develop parallel programs but just taking advantage of RDMA technology
> in general.  I hope this helps.
>
> --
> Jonathan Perkins
> http://www.cse.ohio-state.edu/~perkinjo<http://www.cse.ohio-state.edu/%7Eperkinjo>
>

Thanks,
Vivek
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.cse.ohio-state.edu/pipermail/mvapich-discuss/attachments/20091126/40e7e06d/attachment.html