[mvapich-discuss] How to keep gid status

[Dhabaleswar Panda]

Hi,

Thanks for your note. I shared your question with Dr. Bill Barth from
TACC. Folks from TACC have been using MVAPICH with mpirun_rsh in their
production environment on Ranger for quite some time. I am including his
reply below. I hope his suggested approach will work for you. Let us know.

I am cc'ing Dr. Barth on this e-mail also.  If there are any additional
questions, two of you might exchange additional information on this issue.

Thanks,

DK

====================================================================

As you may recall, we have wrapper scripts that we use on Ranger and
Lonestar to hide the details of the mpirun_rsh command line from the
users. We call it 'ibrun'. It interacts with the scheduler (through the
environment) to generate the host list and establish the number of tasks
to start. I don't see why it would be hard to add a call to /usr/bin/sg in
there.

If the user would have invoked

mpirun_rsh -np 5 -hostfile hosts ./foo

he simply runs

ibrun ./foo

on Ranger or Lonestar. 'ibrun' is basically structured as:

#!/bin/bash
....find NP from the envrionment....
....find host list....
$MPICH_HOME/bin/mpirun_rsh -np $NP -hostfile $HOSTFILE "$@"
So it just takes the command line args of ibrun and passes them directly
to mpirun_rsh

There's no reason it couldn't do

#!/bin/bash
....find NP....
....find host list....
GROUP_ID=`id -gn`
$MPICH_HOME/bin/mpirun_rsh -np $NP -hostfile $HOSTFILE /usr/bin/sg
$GROUP_ID
"$@"

It should be this straightforward.

Bill.

======


On Sun, 14 Jun 2009, Satoshi Isono wrote:

> Dear all,
>
> I would like to know how to keep gid status when launching MPI
> processes. We know that, with sg command in mpirun_rsh command line, it
> is successful in this case. Can you please advise me. I show a example
> as below.
>
> Most of users belong multiple group. And accounting system is managed
> based on a group ID (GID). So, all files created from each user must be
> owned with appropriate group owner information.
>
> A problem here is that the state of GID not saved. I would show you a
> example. Could you read it, according to numbers.
>
> 1) User logins into a login node.
>
>    $ id
>    uid=1002(craysp) gid=1002(cray)
> groups=10(wheel),1002(cray),8001(GAUSSIAN)
>
> This is showing default gid is 1002(cray). This "cray" is primary group
> ID.
>
> 2) User changes arbitrary group with newgrp command.
>
>    $ newgrp GAUSSIAN
>    $ id
>    uid=1002(craysp) gid=8001(GAUSSIAN)
> groups=10(wheel),1002(cray),8001(GAUSSIAN)
>
> This case is that a user wants to change another group like "GAUSSIAN".
> Certainly, I make sure it was changed to GAUSSIAN from cray.
>
> 3) User runs a MPI job with mpirun_rsh
>
> This is the simple MPI code which generates a output file.
>
>    $ cat gid.c
>    #include <stdio.h>
>    #include <mpi.h>
>    #include <string.h>
>
>    int main(int argc,char *argv[])
>    {
>        int rank,size,namelen;
>        char name[MPI_MAX_PROCESSOR_NAME],comm[512];
>
>        MPI_Init(&argc,&argv);
>
>        MPI_Comm_rank(MPI_COMM_WORLD,&rank);
>        MPI_Comm_size(MPI_COMM_WORLD,&size);
>        MPI_Get_processor_name(name,&namelen);
>
>        sprintf(comm,"touch testfile_%s_%d",name,rank);
>        system(comm);
>
>        MPI_Finalize();
>        return 0;
>    }
>
> After running this code, I want that a output file was owned by
> "GAUSSIAN" group. But it was different from that I want. Below is a run
> script including mpirun_rsh.
>
>    $ cat run_i.sh
>    #!/bin/bash
>    . /opt/Modules/init/bash
>    module load pgi mvapich2/pgi
>    mpirun_rsh -np 1 com-0644 ./gid-mv2
>
> 4) User confirms that a created file doesn't owned appropriate group ID.
>
>    $ ls -l testfile_com-0644_0
>    -rw-r--r-- 1 craysp cray 0 Jun  8 17:50 testfile_com-0644_0
>
> You can confirm that this file is owned "cray" not "GAUSSIAN". This
> problem is caused on mpirun_rsh command or SSH server configuration, I
> think.
>
> 5) The way to solve it.
>
> I am considering that better way is inserting "sg" command just before
> a.out in mpirun_rsh command line. I would show you a example.
>
>    $ grep mpirun_rsh run_i.sh
>    mpirun_rsh -np 1 com-0644 /usr/bin/sg `id -gn` ./gid-mv2
>
> By specifying sg command just before a.out, It works well.
>
>    $ ls -l testfile_com-0644_0
>    -rw-r--r-- 1 craysp GAUSSIAN 0 Jun  8 18:33 testfile_com-0644_0
>
> 6) Request to you
>
> I thought that the wrapper script of mpirun_rsh would be created at
> first. But it is difficult to specify executable file location on
> command lines. There are various patterns that user describes in
> mpirun_rsh line. For example:
>
>    mpirun_rsh -np 2048 -hostfile hosts.txt ./a.out Inputfile | tee -a
> Outputfile
>    mpirun_rsh -np 256 -hostfile hostlist ./a.out input >> log
>    mpirun_rsh -np 8 -hostfile hostfile MV2_ENABLE_AFFINITY=0
> MV2_NUM_HCAS=4 ./numarun_mv2.sh ./a.out
>    ...
>
> And we can take a look on line 1607.
>
>    1607     /* add the arguments */
>    1608     for (i = aout_index + 1; i < argc; i++) {
>    1609         strcat(command_name, " ");
>    1610         strcat(command_name, argv[i]);
>    1611     }
>
> An example of edit:
>
>    1607     /* add the arguments */
>    1608     strcat(command_name, " /usr/bin/sg $(id -gn)");
>    1609     for (i = aout_index + 1; i < argc; i++) {
>    1610         strcat(command_name, " ");
>    1611         strcat(command_name, argv[i]);
>    1612     }
>
> I have edited showing above and done recompile it, but it doesn't apply.
> If you know other way which is able to solve this problem, can you
> please tell me?
>
> Best regards,
> Satoshi Isono
>
>
> _______________________________________________
> mvapich-discuss mailing list
> mvapich-discuss at cse.ohio-state.edu
> http://mail.cse.ohio-state.edu/mailman/listinfo/mvapich-discuss
>