[mvapich-discuss] Errors trying to build MVAPICH2-1.5 with the PANFS IO device

[David Gunter]

Here at LANL we use the Panasas parallel file system.  The recent MVAPICH2-1.5 refuses to build and I think I have figured out why. It has to do with some changes made that aren't present in the 1.4.1 release (the last version we have working here).

mvapich2-1.5/src/mpi/romio/adio/ad_panfs/ad_panfs.h requires adio.h which requires adioi.h. The latter has been changed to now include a file mpiimpl.h which lives at

mvapich2-1.5/src/include/mpiimpl.h

Things would be fine except that mpiimpl.h includes files in directories that are not included in 

mvapich2-1.5/src/mpi/romio/adio/ad_panfs/Makefile.in. Namely, mpiimpl.h includes

mvapich2-1.5/src/mpid/ch3/include/mpidpre.h

and

mvapich2-1.5/src/mpid/ch3/include/mpid_thread.h

Thus a path to that include directory should be added to the above mentioned Makefile.in.  However, that doesn't fix the problem as mpidpre.h includes other files that are not in the list of include directories.

mpidpre.h includes "mpid_dataloop.h" and "mpidi_ch3_pre.h". The first of these is

mvapich2-1.5/src/mpid/common/datatype/mpid_dataloop.h

but the second is found in multiple places depending on the channel one is building for, mrail in our case, so

mvapich2-1.5/src/mpid/ch3/channels/mrail/include

So that means two more sets of directories added to the Makefile.in in the ad_panfs directory.

Finally, mpidi_ch3_pre.h includes one more file, "mpidi_ch3_rdma_pre.h", which is found in two different places depending on the device (gen2 or udapl). We build for gen2, so it is found in

mvapich2-1.5/src/mpid/ch3/channels/mrail/src/gen2/mpidi_ch3_rdma_pre.h 

and that is one more include-directory to add to Makefile.in.

After doing all that, I finally get past the Romio build portion and the ad_panfs device is built with some warnings, but no errors.  The make process proceeds smoothly and dies again towards the end. I'm not sure if the final errors are related to the above issue.

make[4]: Entering directory `/usr/projects/hpctools/dog/mvapich2/mvapich2-1.5/src/pm/util'
gcc -I/opt/ofed/include  -g -DNDEBUG -O2   -L/opt/ofed/lib64  -o mpiexec mpiexec.o  \
        ../util/libmpiexec.a ../../../lib/libmpich.a -L/opt/ofed/lib64 -libverbs -libumad -lpthread 
../../../lib/libmpich.a(ibv_param.o): In function `rdma_cm_get_hca_type':
/usr/projects/hpctools/dog/mvapich2/mvapich2-1.5/src/mpid/ch3/channels/mrail/src/gen2/ibv_param.c:370: undefined referen
ce to `rdma_get_devices'
/usr/projects/hpctools/dog/mvapich2/mvapich2-1.5/src/mpid/ch3/channels/mrail/src/gen2/ibv_param.c:416: undefined referen
ce to `rdma_free_devices'
../../../lib/libmpich.a(rdma_cm.o): In function `ib_finalize_rdma_cm':
/usr/projects/hpctools/dog/mvapich2/mvapich2-1.5/src/mpid/ch3/channels/mrail/src/gen2/rdma_cm.c:1229: undefined reference to `rdma_disconnect'
/usr/projects/hpctools/dog/mvapich2/mvapich2-1.5/src/mpid/ch3/channels/mrail/src/gen2/rdma_cm.c:1230: undefined reference to `rdma_destroy_qp'
/usr/projects/hpctools/dog/mvapich2/mvapich2-1.5/src/mpid/ch3/channels/mrail/src/gen2/rdma_cm.c:1274: undefined reference to `rdma_destroy_id'
/usr/projects/hpctools/dog/mvapich2/mvapich2-1.5/src/mpid/ch3/channels/mrail/src/gen2/rdma_cm.c:1283: undefined reference to `rdma_destroy_id'
/usr/projects/hpctools/dog/mvapich2/mvapich2-1.5/src/mpid/ch3/channels/mrail/src/gen2/rdma_cm.c:1285: undefined reference to `rdma_destroy_event_channel'
../../../lib/libmpich.a(rdma_cm.o): In function `rdma_cm_connect_to_server':
/usr/projects/hpctools/dog/mvapich2/mvapich2-1.5/src/mpid/ch3/channels/mrail/src/gen2/rdma_cm.c:964: undefined reference to `rdma_create_id'
/usr/projects/hpctools/dog/mvapich2/mvapich2-1.5/src/mpid/ch3/channels/mrail/src/gen2/rdma_cm.c:976: undefined reference to `rdma_resolve_addr'
../../../lib/libmpich.a(rdma_cm.o): In function `rdma_cm_create_qp':
/usr/projects/hpctools/dog/mvapich2/mvapich2-1.5/src/mpid/ch3/channels/mrail/src/gen2/rdma_cm.c:794: undefined reference to `rdma_create_qp'
../../../lib/libmpich.a(rdma_cm.o): In function `rdma_cm_get_contexts':
/usr/projects/hpctools/dog/mvapich2/mvapich2-1.5/src/mpid/ch3/channels/mrail/src/gen2/rdma_cm.c:728: undefined reference to `rdma_create_id'
/usr/projects/hpctools/dog/mvapich2/mvapich2-1.5/src/mpid/ch3/channels/mrail/src/gen2/rdma_cm.c:737: undefined reference to `rdma_resolve_addr'
/usr/projects/hpctools/dog/mvapich2/mvapich2-1.5/src/mpid/ch3/channels/mrail/src/gen2/rdma_cm.c:748: undefined reference to `rdma_destroy_id'
../../../lib/libmpich.a(rdma_cm.o): In function `ib_init_rdma_cm':
/usr/projects/hpctools/dog/mvapich2/mvapich2-1.5/src/mpid/ch3/channels/mrail/src/gen2/rdma_cm.c:599: undefined reference to `rdma_create_event_channel'
/usr/projects/hpctools/dog/mvapich2/mvapich2-1.5/src/mpid/ch3/channels/mrail/src/gen2/rdma_cm.c:653: undefined reference to `rdma_create_id'
../../../lib/libmpich.a(rdma_cm.o): In function `bind_listen_port':
/usr/projects/hpctools/dog/mvapich2/mvapich2-1.5/src/mpid/ch3/channels/mrail/src/gen2/rdma_cm.c:547: undefined reference to `rdma_bind_addr'
/usr/projects/hpctools/dog/mvapich2/mvapich2-1.5/src/mpid/ch3/channels/mrail/src/gen2/rdma_cm.c:557: undefined reference to `rdma_bind_addr'
/usr/projects/hpctools/dog/mvapich2/mvapich2-1.5/src/mpid/ch3/channels/mrail/src/gen2/rdma_cm.c:566: undefined reference to `rdma_listen'
../../../lib/libmpich.a(rdma_cm.o): In function `ib_cma_event_handler':
/usr/projects/hpctools/dog/mvapich2/mvapich2-1.5/src/mpid/ch3/channels/mrail/src/gen2/rdma_cm.c:124: undefined reference to `rdma_resolve_route'
/usr/projects/hpctools/dog/mvapich2/mvapich2-1.5/src/mpid/ch3/channels/mrail/src/gen2/rdma_cm.c:184: undefined reference to `rdma_connect'
/usr/projects/hpctools/dog/mvapich2/mvapich2-1.5/src/mpid/ch3/channels/mrail/src/gen2/rdma_cm.c:281: undefined reference to `rdma_accept'
/usr/projects/hpctools/dog/mvapich2/mvapich2-1.5/src/mpid/ch3/channels/mrail/src/gen2/rdma_cm.c:240: undefined reference to `rdma_reject'
../../../lib/libmpich.a(rdma_cm.o): In function `cm_thread':
/usr/projects/hpctools/dog/mvapich2/mvapich2-1.5/src/mpid/ch3/channels/mrail/src/gen2/rdma_cm.c:420: undefined reference to `rdma_ack_cm_event'
/usr/projects/hpctools/dog/mvapich2/mvapich2-1.5/src/mpid/ch3/channels/mrail/src/gen2/rdma_cm.c:403: undefined reference to `rdma_get_cm_event'
collect2: ld returned 1 exit status
make[3]: *** [mpiexec] Error 1
make[3]: Leaving directory `/usr/projects/hpctools/dog/mvapich2/mvapich2-1.5/src/pm/gforker'
make[2]: *** [all-redirect] Error 1
make[2]: Leaving directory `/usr/projects/hpctools/dog/mvapich2/mvapich2-1.5/src/pm'
make[1]: *** [all-redirect] Error 2
make[1]: Leaving directory `/usr/projects/hpctools/dog/mvapich2/mvapich2-1.5/src'
make: *** [all-redirect] Error 2

-david
--
David Gunter
HPC-3: Infrastructure Team
Los Alamos National Laboratory