[mvapich-discuss] Shell Environment Variables and mpirun_rsh

[Thompson, Matt (GSFC-610.1)[SCIENCE SYSTEMS AND APPLICATIONS INC]]

All,

I just had a question about the use of environment variables with 
MVAPICH2 1.8.1 as stated in the MVAPICH2 User's Guide in regards to 
mpirun_rsh. I understand that the best way is to pass them in on the 
command-line directly before the executable. But the User's Guide also says:

> Alternatively, you may also place environmental variables in your
> shell environment (e.g. .bashrc). These will be automatically picked
> up when the application starts executing.

I've tried this by doing this:

   setenv MV2_ENABLE_AFFINITY    0
   setenv MV2_CPU_BINDING_POLICY scatter
   setenv MV2_CPU_BINDING_LEVEL  socket
   mpirun_rsh -hostfile $PBS_NODEFILE -np $PROCS ./GEOSgcm.x

in a tcsh run script. These setenvs are directly before the call to 
mpirun_rsh, and yet, it does not have the effect that doing this:

   mpirun_rsh -hostfile $PBS_NODEFILE -np $PROCS \
      MV2_ENABLE_AFFINITY=0 MV2_CPU_BINDING_POLICY=scatter \
      MV2_CPU_BINDING_LEVEL=socket ./GEOSgcm.x

does. This executable, in this example, is running on some nodes with 
GPUs and their hardware configuration is such that if you don't pass in 
scatter/socket, you *know* you have an issue. In this case data transfer 
times to the GPU skyrocket:

   Binding in environment:  REFRESHKS_DATA       :     350.240
   Binding on command line: REFRESHKS_DATA       :      41.477

since you're taking a slow path to the second GPU.

I can show that the slow time is essentially equivalent to passing in 
MV2_CPU_BINDING_POLICY=bunch MV2_CPU_BINDING_LEVEL=core (aka, the 
default), which is why I suspect the environment isn't being read.

So, I'm confused. The User's Guide says I can "place environmental 
variables in (my) shell environment", but it doesn't seem to work. Is 
there a flag I need to pass to mpirun_rsh to expose the environment? Do 
I have to use bash?

Thanks,
Matt Thompson
-- 
Matt Thompson, PhD     SSAI, Sr Software Test Engr
NASA GSFC, Global Modeling and Assimilation Office
Code 610.1, 8800 Greenbelt Rd, Greenbelt, MD 20771
Phone: 301-614-6712              Fax: 301-614-6246