[mvapich-discuss] using OSC's iwarp-1.1 with MVAPICH2 0.9.8

[Sundeep Narravula]

Hi Brad,

  Thanks for trying out mvapich2. We have tested mvapich2-0.9.8 on OFED
1.1 and OFED 1.2. Your build procedure is OK, however, umad related
funtionality is currently needed in mvpaich2. We have not tried our
package with OSC's software verbs.

Regards,
  --Sundeep.

On Thu, 24 May 2007, Brad Penoff wrote:

> Greetings,
>
> Currently we don't have iWARP hardware but we have been using the OSC
> user-space iWARP-in-software implementation to familiarize ourselves with
> iWARP.  In doing so, we were interested in trying to use some MPI
> implementation that runs using OSC's iwarp-1.1 (with "Support for
> OpenFabrics API", says their webpage); do you have any experience using
> this (or any?) user-space iWARP-in-software implementation together with
> MVAPICH2 0.9.8?
>
> On a whim, I tried doing a "make -C iwarp" within OSC's iwarp-1.1,
> producing iwarp/libiwarp.a and verbs/libverbs.a but when I downloaded
> MVAPICH2 0.9.8 and tried to configure (using the supplied
> make.mvapich2.iwarp), that seems to look for libibverbs.a and libibumad.a
> .  Following the settings that the make.mvapich2.iwarp made, and knowing
> our setup was unique (i.e. all software emulation, so I didn't use
> make.mvapich2.iwarp directly) I tried to do the following:
>
> 1) in my environment, I set CFLAGS
> to -I/home/penoff/iwarp-1.1/verbs/openfab
> -I/home/penoff/iwarp-1.1/iwarp -Di686 -DUSE_HEADER_CACHING -DONE_SIDED
> -DMPID_USE_SEQUENCE_NUMBERS
>
> 2) in my environment, I set my LIBS
> to -L/home/penoff/iwarp-1.1/verbs -lverbs
>
>
> 3) I did a vpath build in a directory I created named "build" with these
> args (again, trying to mock the contents of make.mvapich2.iwarp):
>
> ../configure --prefix=/pcl/scratch/penoff/installs/mvapich
> --with-device=osu_ch3:mrail --with-rdma=gen2 --with-pm=mpd --without-mpe
> --disable-romio ; make
>
> ..where /home/penoff/iwarp-1.1 is the OSC iwarp-1.1 location that I had
> already ran "make -C iwarp" in.
>
> The MVAPICH2 0.9.8 make failed looking for the infiniband/umad.h file; no
> umad.h is provided with the OSC iwarp-1.1.  I also didn't see it in
> OFED-1.1...
>
> So I'm wondering, is it possible to compile and run MVAPICH2 using the
> osu_ch3:mrail device with the RDMA portions using iWARP entirely ran
> entirely in software over Ethernet NICs ?
>
> I'd be really curious to hear your experiences.
>
> Thanks ahead of time,
> brad
>
> p.s. I just saw that MVAPICH2 0.9.8p2 is out now.  I glimpsed at the
> contents of the make.mvapich2.ofa file suggested in the user guide and I'm
> guessing I'd run into the same issue since the LIBS is set in a similar
> way to the make.mvapich2.iwarp file that I had looked at when I wrote this
> mail...
> _______________________________________________
> mvapich-discuss mailing list
> mvapich-discuss at cse.ohio-state.edu
> http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss
>