[mvapich-discuss] Errors trying to build MVAPICH2-1.5 with the PANFS IO device

Tue Aug 17 09:45:40 EDT 2010

Unfortunately this release has the exact same issue I just encountered. In fact, mvapich2-1.5-2010-08-15/src/mpi/romio/adio/ad_panfs/Makefile.in is identical to the 1.5 release.  

-david

--
David Gunter
HPC-3: Infrastructure Team
Los Alamos National Laboratory

On Aug 16, 2010, at 8:50 PM, Jonathan Perkins wrote:

> Hi David, thanks for the post.  This issue came up earlier in our
> discussion list
> 
> http://mail.cse.ohio-state.edu/pipermail/mvapich-discuss/2010-July/002926.html
> 
> We have committed a fix to our 1.5 branch late last month for this.
> Can you try our latest nightly tarball and see if this resolves the
> problem for you?  Here is a link for your convenience...
> 
> http://mvapich.cse.ohio-state.edu/nightly/mvapich2/branches/1.5/mvapich2-1.5-2010-08-15.tar.gz
> 
> On Mon, Aug 16, 2010 at 7:20 PM, David Gunter <dog at lanl.gov> wrote:
>> Here at LANL we use the Panasas parallel file system.  The recent MVAPICH2-1.5 refuses to build and I think I have figured out why. It has to do with some changes made that aren't present in the 1.4.1 release (the last version we have working here).
>> 
>> mvapich2-1.5/src/mpi/romio/adio/ad_panfs/ad_panfs.h requires adio.h which requires adioi.h. The latter has been changed to now include a file mpiimpl.h which lives at
>> 
>> mvapich2-1.5/src/include/mpiimpl.h
>> 
>> Things would be fine except that mpiimpl.h includes files in directories that are not included in
>> 
>> mvapich2-1.5/src/mpi/romio/adio/ad_panfs/Makefile.in. Namely, mpiimpl.h includes
>> 
>> mvapich2-1.5/src/mpid/ch3/include/mpidpre.h
>> 
>> and
>> 
>> mvapich2-1.5/src/mpid/ch3/include/mpid_thread.h
>> 
>> Thus a path to that include directory should be added to the above mentioned Makefile.in.  However, that doesn't fix the problem as mpidpre.h includes other files that are not in the list of include directories.
>> 
>> mpidpre.h includes "mpid_dataloop.h" and "mpidi_ch3_pre.h". The first of these is
>> 
>> mvapich2-1.5/src/mpid/common/datatype/mpid_dataloop.h
>> 
>> but the second is found in multiple places depending on the channel one is building for, mrail in our case, so
>> 
>> mvapich2-1.5/src/mpid/ch3/channels/mrail/include
>> 
>> So that means two more sets of directories added to the Makefile.in in the ad_panfs directory.
>> 
>> Finally, mpidi_ch3_pre.h includes one more file, "mpidi_ch3_rdma_pre.h", which is found in two different places depending on the device (gen2 or udapl). We build for gen2, so it is found in
>> 
>> mvapich2-1.5/src/mpid/ch3/channels/mrail/src/gen2/mpidi_ch3_rdma_pre.h
>> 
>> and that is one more include-directory to add to Makefile.in.
>> 
>> After doing all that, I finally get past the Romio build portion and the ad_panfs device is built with some warnings, but no errors.  The make process proceeds smoothly and dies again towards the end. I'm not sure if the final errors are related to the above issue.
>> 
>> make[4]: Entering directory `/usr/projects/hpctools/dog/mvapich2/mvapich2-1.5/src/pm/util'
>> gcc -I/opt/ofed/include  -g -DNDEBUG -O2   -L/opt/ofed/lib64  -o mpiexec mpiexec.o  \
>>        ../util/libmpiexec.a ../../../lib/libmpich.a -L/opt/ofed/lib64 -libverbs -libumad -lpthread
>> ../../../lib/libmpich.a(ibv_param.o): In function `rdma_cm_get_hca_type':
>> /usr/projects/hpctools/dog/mvapich2/mvapich2-1.5/src/mpid/ch3/channels/mrail/src/gen2/ibv_param.c:370: undefined referen
>> ce to `rdma_get_devices'
>> /usr/projects/hpctools/dog/mvapich2/mvapich2-1.5/src/mpid/ch3/channels/mrail/src/gen2/ibv_param.c:416: undefined referen
>> ce to `rdma_free_devices'
>> ../../../lib/libmpich.a(rdma_cm.o): In function `ib_finalize_rdma_cm':
>> /usr/projects/hpctools/dog/mvapich2/mvapich2-1.5/src/mpid/ch3/channels/mrail/src/gen2/rdma_cm.c:1229: undefined reference to `rdma_disconnect'
>> /usr/projects/hpctools/dog/mvapich2/mvapich2-1.5/src/mpid/ch3/channels/mrail/src/gen2/rdma_cm.c:1230: undefined reference to `rdma_destroy_qp'
>> /usr/projects/hpctools/dog/mvapich2/mvapich2-1.5/src/mpid/ch3/channels/mrail/src/gen2/rdma_cm.c:1274: undefined reference to `rdma_destroy_id'
>> /usr/projects/hpctools/dog/mvapich2/mvapich2-1.5/src/mpid/ch3/channels/mrail/src/gen2/rdma_cm.c:1283: undefined reference to `rdma_destroy_id'
>> /usr/projects/hpctools/dog/mvapich2/mvapich2-1.5/src/mpid/ch3/channels/mrail/src/gen2/rdma_cm.c:1285: undefined reference to `rdma_destroy_event_channel'
>> ../../../lib/libmpich.a(rdma_cm.o): In function `rdma_cm_connect_to_server':
>> /usr/projects/hpctools/dog/mvapich2/mvapich2-1.5/src/mpid/ch3/channels/mrail/src/gen2/rdma_cm.c:964: undefined reference to `rdma_create_id'
>> /usr/projects/hpctools/dog/mvapich2/mvapich2-1.5/src/mpid/ch3/channels/mrail/src/gen2/rdma_cm.c:976: undefined reference to `rdma_resolve_addr'
>> ../../../lib/libmpich.a(rdma_cm.o): In function `rdma_cm_create_qp':
>> /usr/projects/hpctools/dog/mvapich2/mvapich2-1.5/src/mpid/ch3/channels/mrail/src/gen2/rdma_cm.c:794: undefined reference to `rdma_create_qp'
>> ../../../lib/libmpich.a(rdma_cm.o): In function `rdma_cm_get_contexts':
>> /usr/projects/hpctools/dog/mvapich2/mvapich2-1.5/src/mpid/ch3/channels/mrail/src/gen2/rdma_cm.c:728: undefined reference to `rdma_create_id'
>> /usr/projects/hpctools/dog/mvapich2/mvapich2-1.5/src/mpid/ch3/channels/mrail/src/gen2/rdma_cm.c:737: undefined reference to `rdma_resolve_addr'
>> /usr/projects/hpctools/dog/mvapich2/mvapich2-1.5/src/mpid/ch3/channels/mrail/src/gen2/rdma_cm.c:748: undefined reference to `rdma_destroy_id'
>> ../../../lib/libmpich.a(rdma_cm.o): In function `ib_init_rdma_cm':
>> /usr/projects/hpctools/dog/mvapich2/mvapich2-1.5/src/mpid/ch3/channels/mrail/src/gen2/rdma_cm.c:599: undefined reference to `rdma_create_event_channel'
>> /usr/projects/hpctools/dog/mvapich2/mvapich2-1.5/src/mpid/ch3/channels/mrail/src/gen2/rdma_cm.c:653: undefined reference to `rdma_create_id'
>> ../../../lib/libmpich.a(rdma_cm.o): In function `bind_listen_port':
>> /usr/projects/hpctools/dog/mvapich2/mvapich2-1.5/src/mpid/ch3/channels/mrail/src/gen2/rdma_cm.c:547: undefined reference to `rdma_bind_addr'
>> /usr/projects/hpctools/dog/mvapich2/mvapich2-1.5/src/mpid/ch3/channels/mrail/src/gen2/rdma_cm.c:557: undefined reference to `rdma_bind_addr'
>> /usr/projects/hpctools/dog/mvapich2/mvapich2-1.5/src/mpid/ch3/channels/mrail/src/gen2/rdma_cm.c:566: undefined reference to `rdma_listen'
>> ../../../lib/libmpich.a(rdma_cm.o): In function `ib_cma_event_handler':
>> /usr/projects/hpctools/dog/mvapich2/mvapich2-1.5/src/mpid/ch3/channels/mrail/src/gen2/rdma_cm.c:124: undefined reference to `rdma_resolve_route'
>> /usr/projects/hpctools/dog/mvapich2/mvapich2-1.5/src/mpid/ch3/channels/mrail/src/gen2/rdma_cm.c:184: undefined reference to `rdma_connect'
>> /usr/projects/hpctools/dog/mvapich2/mvapich2-1.5/src/mpid/ch3/channels/mrail/src/gen2/rdma_cm.c:281: undefined reference to `rdma_accept'
>> /usr/projects/hpctools/dog/mvapich2/mvapich2-1.5/src/mpid/ch3/channels/mrail/src/gen2/rdma_cm.c:240: undefined reference to `rdma_reject'
>> ../../../lib/libmpich.a(rdma_cm.o): In function `cm_thread':
>> /usr/projects/hpctools/dog/mvapich2/mvapich2-1.5/src/mpid/ch3/channels/mrail/src/gen2/rdma_cm.c:420: undefined reference to `rdma_ack_cm_event'
>> /usr/projects/hpctools/dog/mvapich2/mvapich2-1.5/src/mpid/ch3/channels/mrail/src/gen2/rdma_cm.c:403: undefined reference to `rdma_get_cm_event'
>> collect2: ld returned 1 exit status
>> make[3]: *** [mpiexec] Error 1
>> make[3]: Leaving directory `/usr/projects/hpctools/dog/mvapich2/mvapich2-1.5/src/pm/gforker'
>> make[2]: *** [all-redirect] Error 1
>> make[2]: Leaving directory `/usr/projects/hpctools/dog/mvapich2/mvapich2-1.5/src/pm'
>> make[1]: *** [all-redirect] Error 2
>> make[1]: Leaving directory `/usr/projects/hpctools/dog/mvapich2/mvapich2-1.5/src'
>> make: *** [all-redirect] Error 2
>> 
>> -david
>> --
>> David Gunter
>> HPC-3: Infrastructure Team
>> Los Alamos National Laboratory
>> 
>> 
>> 
>> 
>> 
>> _______________________________________________
>> mvapich-discuss mailing list
>> mvapich-discuss at cse.ohio-state.edu
>> http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss
>> 
>> 
> 
> -- 
> Jonathan Perkins