[mvapich-discuss] using OSC's iwarp-1.1 with MVAPICH2 0.9.8

[Brad Penoff]

Greetings,

Currently we don't have iWARP hardware but we have been using the OSC
user-space iWARP-in-software implementation to familiarize ourselves with 
iWARP.  In doing so, we were interested in trying to use some MPI 
implementation that runs using OSC's iwarp-1.1 (with "Support for 
OpenFabrics API", says their webpage); do you have any experience using 
this (or any?) user-space iWARP-in-software implementation together with 
MVAPICH2 0.9.8?

On a whim, I tried doing a "make -C iwarp" within OSC's iwarp-1.1, 
producing iwarp/libiwarp.a and verbs/libverbs.a but when I downloaded 
MVAPICH2 0.9.8 and tried to configure (using the supplied 
make.mvapich2.iwarp), that seems to look for libibverbs.a and libibumad.a 
.  Following the settings that the make.mvapich2.iwarp made, and knowing 
our setup was unique (i.e. all software emulation, so I didn't use 
make.mvapich2.iwarp directly) I tried to do the following:

1) in my environment, I set CFLAGS 
to -I/home/penoff/iwarp-1.1/verbs/openfab 
-I/home/penoff/iwarp-1.1/iwarp -Di686 -DUSE_HEADER_CACHING -DONE_SIDED 
-DMPID_USE_SEQUENCE_NUMBERS

2) in my environment, I set my LIBS
to -L/home/penoff/iwarp-1.1/verbs -lverbs


3) I did a vpath build in a directory I created named "build" with these 
args (again, trying to mock the contents of make.mvapich2.iwarp):

../configure --prefix=/pcl/scratch/penoff/installs/mvapich 
--with-device=osu_ch3:mrail --with-rdma=gen2 --with-pm=mpd --without-mpe 
--disable-romio ; make

..where /home/penoff/iwarp-1.1 is the OSC iwarp-1.1 location that I had 
already ran "make -C iwarp" in.

The MVAPICH2 0.9.8 make failed looking for the infiniband/umad.h file; no 
umad.h is provided with the OSC iwarp-1.1.  I also didn't see it in 
OFED-1.1...

So I'm wondering, is it possible to compile and run MVAPICH2 using the 
osu_ch3:mrail device with the RDMA portions using iWARP entirely ran 
entirely in software over Ethernet NICs ?

I'd be really curious to hear your experiences.

Thanks ahead of time,
brad

p.s. I just saw that MVAPICH2 0.9.8p2 is out now.  I glimpsed at the 
contents of the make.mvapich2.ofa file suggested in the user guide and I'm 
guessing I'd run into the same issue since the LIBS is set in a similar 
way to the make.mvapich2.iwarp file that I had looked at when I wrote this 
mail...