[mvapich-discuss] Configure/make fails when using --with-shared-memory=sysv

[Jonathan Perkins]

On Thu, Sep 11, 2014 at 08:56:39AM +0200, Åsmund Ervik wrote:
> Hello all,
> 
> I'm compiling mvapich2 on an old quirky cluster where I'm not root.
> After having problems with shared memory using mvapich2 with plain
> configure/make (I had to disable shared memory for any jobs doing
> allreduce), I decided to try to configure/make using
> --with-shared-memory=sysv . This failed at the "make" step, with the
> following error:
>    CC       src/mpid/ch3/channels/mrail/src/gen2/rdma_iba_1sc.lo
> src/mpid/ch3/channels/mrail/src/gen2/rdma_iba_1sc.c: In function
> 'mv2_allocate_shm_local':
> src/mpid/ch3/channels/mrail/src/gen2/rdma_iba_1sc.c:182: error:
> 'O_CREAT' undeclared (first use in this function)
> src/mpid/ch3/channels/mrail/src/gen2/rdma_iba_1sc.c:182: error: (Each
> undeclared identifier is reported only once
> src/mpid/ch3/channels/mrail/src/gen2/rdma_iba_1sc.c:182: error: for each
> function it appears in.)
> src/mpid/ch3/channels/mrail/src/gen2/rdma_iba_1sc.c:182: error: 'O_RDWR'
> undeclared (first use in this function)
> src/mpid/ch3/channels/mrail/src/gen2/rdma_iba_1sc.c:182: error: 'O_EXCL'
> undeclared (first use in this function)
> src/mpid/ch3/channels/mrail/src/gen2/rdma_iba_1sc.c: In function
> 'mv2_rma_allocate_shm':
> src/mpid/ch3/channels/mrail/src/gen2/rdma_iba_1sc.c:375: error:
> 'O_CREAT' undeclared (first use in this function)
> src/mpid/ch3/channels/mrail/src/gen2/rdma_iba_1sc.c:375: error: 'O_RDWR'
> undeclared (first use in this function)
> make[2]: *** [src/mpid/ch3/channels/mrail/src/gen2/rdma_iba_1sc.lo] Error 1
> 
> The full configure command was
> ./configure --prefix=/home/asmunder/mvapich2-2.0
> --with-ib-libpath=/cvos/shared/apps/ofed/1.3.1/lib64/
> --with-ib-include=/cvos/shared/apps/ofed/1.3.1/include
> --with-shared-memory=sysv --enable-libpci
> 
> 
> I traced this down to lines 22 and 26 in rdma_iba_1sc.c ,
> which is an #ifdef that only includes fcntl.h (containing O_CREAT et al)
> when the flag _SMP_LIMIC_ is defined. Obviously this flag was not
> defined in my case. I removed these two lines, and now configure/make
> works fine.
> 
> 
> Also, in the end, I found that the sysv thing was not necessary, and
> that the
> correct and much simpler solution was to set the environment variable
> MV2_SHMEM_DIR to a tempdir I am allowed to write to. For future googlers,
> the error message I originally had was e.g.
> 
> [node096:mpi_rank_0][mv2_shm_coll_init] shmem open failed for
> file:/dev/shm/slot_shmem-coll-kvs_20417_0-node096-0-57926.tmp

Thanks for the report.  We more heavily depend on posix shared memory
now so we may modify this configure option in the future and update the
macros guarding such code in the future.

-- 
Jonathan Perkins