[mvapich-discuss] MPI-2 over ibverbs/rdma

[Jonathan Perkins]

On Wed, Nov 25, 2009 at 08:59:44PM +0530, Vivek Satpute wrote:
> Hi Jonathan, On Wed, Nov 25, 2009 at 8:21 PM, Jonathan Perkins wrote:
> > On Wed, Nov 25, 2009 at 02:04:13PM +0530, Vivek Satpute wrote:
> > > On Wed, Nov 25, 2009 at 1:38 PM, Dhabaleswar Panda wrote:
> > > > You can use the OpenFabrics-Gen2/IB interface of MVAPICH2 to run
> > > > it directly over ibverbs.
> > >
> > > If  I configured above then for executing MPI application from
> > > node1 to node2, node3 and node4 (e.g mpirun_rsh -np 4 n0 n1 n2 n3
> > > ./cpi), I will have to mention a IB interfaces of each n0, n1, n3,
> > > n4. So ultimately MPI will use IPoIB module. Am i right ?
> >
> > You shouldn't need to specify any additional information.  The
> > mpirun_rsh command only needs to know which nodes you're trying to
> > launch mpi program on.  It appears that you're basing your questions
> > on some additional assumptions.  Are you asking this question based
> > on some problem that you're run into?
>
> Yes, you are right. :-) I think I should explain my problem and
> assumptions here :-)
> 
> I am working on OFED-1.4, it has many stuffs related to Infiniband. I
> have installed OFED on Infiniband hardwares, and IB devices are active
> and they are working similar to normal ethernet devices. Means I can
> ping/ssh to remote machine through IB devices.

So, it looks like you have configured IPoIB so normal TCP/IP socket
programs can work over your shiny new Infiniband network.

> 
> Whenever I send ping packets or do ssh to remote machine, all traffic
> goes through IPoIB module, mean it uses IPoIB code path.

Yep.

> 
> My understanding/assumption is, Infiniband also work without using
> IPoIB.  So, I want to find out application/utility which work without
> IPoIB module.  All the traffic generated by such xyz application
> should go to remote machine without IPoIB so for achieving this I can
> use LID/GID of IB devices.

These programs are written with Infiniband in mind and do not use
sockets but instead the ibverbs API.

> Is there any such technology/utility/application exists which work
> without IPoIB ?  Is there any packet path which goes to remote node
> without passing through IPoIB ?

Yes, but these aren't usually general use applications.

> My this assumption can be wrong so please correct me if I am wrong ..
> 
> [  As per my understanding libibverbs are below the IPoIB. So
> applications (examples of libibvers in OFED-1.4) should work on
> LID/GID, but those applications also uses IP address means ultimately
> packets of those application also passes through IPoIB module. ]

It may be good for you to subscribe to the linux-rdma at vger.kernel.orga
list and learn more about this technology there.  You can also take a
look at http://www.openfabrics.org/downloads/OFA_Backgrounder.pdf.  It
doesn't seem that your questions are related to using the MPI API to
develop parallel programs but just taking advantage of RDMA technology
in general.  I hope this helps.

-- 
Jonathan Perkins
http://www.cse.ohio-state.edu/~perkinjo
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 197 bytes
Desc: not available
Url : http://mail.cse.ohio-state.edu/pipermail/mvapich-discuss/attachments/20091125/06ee1044/attachment-0001.bin