[mvapich-discuss] How to keep gid status

[Satoshi Isono]

Dear all,

I would like to know how to keep gid status when launching MPI
processes. We know that, with sg command in mpirun_rsh command line, it
is successful in this case. Can you please advise me. I show a example
as below.

Most of users belong multiple group. And accounting system is managed
based on a group ID (GID). So, all files created from each user must be
owned with appropriate group owner information.

A problem here is that the state of GID not saved. I would show you a
example. Could you read it, according to numbers.

1) User logins into a login node.

   $ id
   uid=1002(craysp) gid=1002(cray)
groups=10(wheel),1002(cray),8001(GAUSSIAN)

This is showing default gid is 1002(cray). This "cray" is primary group
ID.

2) User changes arbitrary group with newgrp command.

   $ newgrp GAUSSIAN
   $ id
   uid=1002(craysp) gid=8001(GAUSSIAN)
groups=10(wheel),1002(cray),8001(GAUSSIAN)

This case is that a user wants to change another group like "GAUSSIAN".
Certainly, I make sure it was changed to GAUSSIAN from cray.

3) User runs a MPI job with mpirun_rsh

This is the simple MPI code which generates a output file.

   $ cat gid.c
   #include <stdio.h>
   #include <mpi.h>
   #include <string.h>
   
   int main(int argc,char *argv[])
   {
       int rank,size,namelen;
       char name[MPI_MAX_PROCESSOR_NAME],comm[512];
   
       MPI_Init(&argc,&argv);
   
       MPI_Comm_rank(MPI_COMM_WORLD,&rank);
       MPI_Comm_size(MPI_COMM_WORLD,&size);
       MPI_Get_processor_name(name,&namelen);
   
       sprintf(comm,"touch testfile_%s_%d",name,rank);
       system(comm);
   
       MPI_Finalize();
       return 0;
   }

After running this code, I want that a output file was owned by
"GAUSSIAN" group. But it was different from that I want. Below is a run
script including mpirun_rsh.

   $ cat run_i.sh
   #!/bin/bash
   . /opt/Modules/init/bash
   module load pgi mvapich2/pgi
   mpirun_rsh -np 1 com-0644 ./gid-mv2

4) User confirms that a created file doesn't owned appropriate group ID.

   $ ls -l testfile_com-0644_0
   -rw-r--r-- 1 craysp cray 0 Jun  8 17:50 testfile_com-0644_0

You can confirm that this file is owned "cray" not "GAUSSIAN". This
problem is caused on mpirun_rsh command or SSH server configuration, I
think.

5) The way to solve it.

I am considering that better way is inserting "sg" command just before
a.out in mpirun_rsh command line. I would show you a example.

   $ grep mpirun_rsh run_i.sh
   mpirun_rsh -np 1 com-0644 /usr/bin/sg `id -gn` ./gid-mv2

By specifying sg command just before a.out, It works well.

   $ ls -l testfile_com-0644_0
   -rw-r--r-- 1 craysp GAUSSIAN 0 Jun  8 18:33 testfile_com-0644_0

6) Request to you

I thought that the wrapper script of mpirun_rsh would be created at
first. But it is difficult to specify executable file location on
command lines. There are various patterns that user describes in
mpirun_rsh line. For example:

   mpirun_rsh -np 2048 -hostfile hosts.txt ./a.out Inputfile | tee -a
Outputfile
   mpirun_rsh -np 256 -hostfile hostlist ./a.out input >> log
   mpirun_rsh -np 8 -hostfile hostfile MV2_ENABLE_AFFINITY=0
MV2_NUM_HCAS=4 ./numarun_mv2.sh ./a.out
   ...

And we can take a look on line 1607.

   1607     /* add the arguments */
   1608     for (i = aout_index + 1; i < argc; i++) {
   1609         strcat(command_name, " ");
   1610         strcat(command_name, argv[i]);
   1611     }

An example of edit:

   1607     /* add the arguments */
   1608     strcat(command_name, " /usr/bin/sg $(id -gn)");
   1609     for (i = aout_index + 1; i < argc; i++) {
   1610         strcat(command_name, " ");
   1611         strcat(command_name, argv[i]);
   1612     }

I have edited showing above and done recompile it, but it doesn't apply.
If you know other way which is able to solve this problem, can you
please tell me?

Best regards,
Satoshi Isono