[mvapich-discuss] Using environment modules -- libimf.so not found

[Jonathan Perkins]

On Mon, Jan 12, 2015 at 03:01:36PM -0500, Novosielski, Ryan wrote:
> Hi all,
> 
> Since we use different compilers for different software packages and
> have a pretty complicated setup for some of the dependencies,
> generally what we do for submitting jobs is to load the modules on the
> master node and submit jobs so that the environment variables are
> passed through. However, this does not appear to be working right for
> MVAPICH2. LD_LIBRARY_PATH is defined, according to the output of env
> at the beginning of the submission script, but apparently does not get
> through to whatever happens when mpiexec/mpirun start the MPI job. The
> FAQ for MPICH2 recommends making sure the environment is correct for
> interactive and non-interactive sessions. However, I do not want to
> define a single PATH to the Intel compiler libraries. People on our
> systems make use of software compiled with different versions, etc.
> 
> Does anyone have an idea of how I might get around this? 

Hi Ryan.  I think that you're close to getting it working but we might
need a bit more details on how and when the failure is happening
exactly.

With mpirun_rsh I believe your setup will work, however with
mpiexec.hydra (mpiexec and mpirun point to this by default) I don't
think LD_LIBRARY_PATH will be seen early enough before the proxy
attempts to execute on the remote nodes.

If you're willing to rebuild MVAPICH2 you can use RPATH during the build
so that the proxy will be able to find its dependencies.

Example:
    ./configure ... LDFLAGS='-Wl,rpath,/path/to/intel/lib/dir' ...

-- 
Jonathan Perkins