[mvapich-discuss] mvapich2-x, symmetric memory sizing

[Riebs, Andy]

Hi All,

I've got a SHMEM application that wants to use well over half of the available memory on all of the compute nodes that it can find. Unfortunately, this is a shared cluster, so I don't think we can simply reallocate /dev/shm to be a huge chunk of memory across the board. (Or can that actually be done without affecting other users or applications, so long as no one fills it up with junk?)

>From the UG at < http://mvapich.cse.ohio-state.edu/static/media/mvapich/mvapich2-x-2.3rc2-userguide.pdf>, the usage of OOSHM_SYMMETRIC_HEAP_SIZE and OOSHM_USE_SHARED_MEM isn't clear; I get "shmalloc failed" at every turn.

I first tried setting OOSHM_SYMMETRIC_HEAP_SIZE to the value that I would use with Open MPI/OSHMEM's SHMEM_SYMMETRIC_HEAP_SIZE or OSSS-UCX's SHMEM_SYMMETRIC_SIZE, with no luck. I then tried setting that, or not, in conjunction with OOSHM_USE_SHARED_MEM=0, but still had no luck.

Is it possible to use a large symmetric heap without adjusting the size of /dev/shm, or did I misread the documentation?

I'm using the mvapich2-x-mofed4.2-gnu4.8.5-2.3rc2-1.el7.tgz kit, running CentOS-7 on x86_64 compute nodes.

Andy

--
Andy Riebs
andy.riebs at hpe.com
Hewlett Packard Enterprise
High Performance Computing Software Engineering

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.cse.ohio-state.edu/pipermail/mvapich-discuss/attachments/20191108/249b4165/attachment.html>