[mvapich-discuss] SGE+mvapich2 tight integration

[Mark Dixon]

On Wed, 28 Jul 2010, Robert Soliday wrote:

> I have it working now. All I had to do was change mpirun_rsh.h from
> #define RSH_CMD		    "/usr/bin/rsh"
> to
> #define RSH_CMD		    "/act/sge/mvapich2-1.5/rsh"
>
> This new rsh command is a wrapper script for qrsh. I then had to add
> every node into my .ssh/known_hosts file. I already had passwordless ssh
> setup but the query related to a host not in the known_hosts file was
> keeping it from working. Anyway, now I can submit the mvapich2 job to
> SGE and when I go to delete it with qdel all the processes on all the
> nodes get deleted correctly.
>
> --Bob Soliday

We're using mvapich2 with SGE tight integration too.

What we do, is:

1) Alter the #define for RSH_CMD to "rsh", and SSH_CMD to "ssh".

2) Replace calls to execv with execvp, and execl with execlp.

This means that the path is searched for the rsh or ssh binary. You can 
then use the same copy of mvapich2 inside of the SGE queuing environment, 
or outside (e.g. users can run short tests on the login node, sysadmins 
can run tests without needing the batch queues).

Mark
-- 
-----------------------------------------------------------------
Mark Dixon                       Email    : m.c.dixon at leeds.ac.uk
HPC/Grid Systems Support         Tel (int): 35429
Information Systems Services     Tel (ext): +44(0)113 343 5429
University of Leeds, LS2 9JT, UK
-----------------------------------------------------------------