[mvapich-discuss] Shell Environment Variables and mpirun_rsh

[Jonathan Perkins]

On Fri, Apr 19, 2013 at 08:33:51AM -0400, Thompson, Matt (GSFC-610.1)[SCIENCE SYSTEMS AND APPLICATIONS INC] wrote:
> All,
> 
> I just had a question about the use of environment variables with
> MVAPICH2 1.8.1 as stated in the MVAPICH2 User's Guide in regards to
> mpirun_rsh. I understand that the best way is to pass them in on the
> command-line directly before the executable. But the User's Guide
> also says:
> 
> >Alternatively, you may also place environmental variables in your
> >shell environment (e.g. .bashrc). These will be automatically picked
> >up when the application starts executing.
> 
> I've tried this by doing this:
> 
>   setenv MV2_ENABLE_AFFINITY    0
>   setenv MV2_CPU_BINDING_POLICY scatter
>   setenv MV2_CPU_BINDING_LEVEL  socket
>   mpirun_rsh -hostfile $PBS_NODEFILE -np $PROCS ./GEOSgcm.x

The above blurb "shell environment" should read "shell personal
initialization file".  Your example above doesn't work because
mpirun_rsh does not forward your environment to the remote processes.

It looks like you're using some version of CSH.  Based on what the blurb
above intended you would want to place the setenv statements in your
.cshrc or .tcshrc file.  The reason that works is because when the new
shells are launched for your remote processes they will use the rc files
to initialize their environment.

Since you're using mvapich2-1.8.1, I suggest you put your environment
settings for mvapich2 in a ~/.mvapich2.conf.
http://mvapich.cse.ohio-state.edu/support/user_guide_mvapich2-1.9rc1.html#x1-490006.3

MV2_ENABLE_AFFINITY = 0
MV2_CPU_BINDING_POLICY = scatter # no effect because affinity is disabled
MV2_CPU_BINDING_LEVEL = socket # no effect because affinity is disabled

If you move to mvapich2-1.9rc1 you can take advantage of the new -export
option of mpirun_rsh where the environment is forwarded for you.

setenv MV2_ENABLE_AFFINITY    0
setenv MV2_CPU_BINDING_POLICY scatter
setenv MV2_CPU_BINDING_LEVEL  socket
mpirun_rsh -export -hostfile $PBS_NODEFILE -np $PROCS ./GEOSgcm.x

-- 
Jonathan Perkins
http://www.cse.ohio-state.edu/~perkinjo