[OOD-users] Modify Job Composer to just use a working directory

[Franz, Eric]

Morgan,

Sounds great. I went ahead and opened a Discourse topic to continue the discussion: https://discourse.osc.edu/t/job-composer-to-support-using-existing-directories-as-job-directories/38

There are some thoughts on design and implementation. Hopefully that will be helpful. I think this would be a useful feature for others, and would work well with our plans for future development for the Job Composer. Here is the GitHub repo: https://github.com/OSC/ood-myjobs if you want to fork the code and make a Pull Request. Or even just participation on that Discourse topic on your own experiments and design thoughts are helpful.

Thanks,
Eric

---
Eric Franz, Senior Web & Interface App Engineer
Ohio Supercomputer Center
An Ohio Technology Consortium (OH-TECH) Member
1224 Kinnear Road
Columbus, OH 43212
email: efranz at osc.edu

On 10/1/18, 2:05 PM, "OOD-users on behalf of mludwig" <ood-users-bounces+efranz=osc.edu at lists.osc.edu on behalf of mludwig at intranet.techsquare.com> wrote:

    Hi Eric,
    
    Thank you for the explanation. I now understand the reason for copying 
    the directory. It looks like our deployment does not need to be as 
    feature rich, instead I would like the workflow for our deployment to 
    treat a provided directory as the staging directory (not copying). Maybe 
    just adding a manifest.yml to that provided directory. Ideally allowing 
    users to just use the GUI rather than navigate over a plain ssh 
    connection. I understand that this is not how the Job Composer is 
    intended but its close and so darn pretty. I am going to pursue 
    modifying the app so suggestions on modifying it are welcome ( I know it 
    wont be officially supported ). I will follow up with a thread on 
    Discourse with any progress. Thank you for all your hard work!
    
    best,
    
    Morgan
    
    
    
    On 09/28/2018 05:36 PM, Franz, Eric wrote:
    > Morgan,
    >
    > The way the Job Composer currently works is there are one or more template directories that contain jobs (and a default template). Users make a copy of this directory, which may contain a job script and maybe contain related files such as input files. You can see what we use at OSC here: https://github.com/OSC/osc-ood-config/tree/master/ondemand.osc.edu/apps/myjobs/templates. The workflow is copy the directory, modify files, submit the job, and the job copies files back to that directory.
    >
    > Most of our adapters do pass the job script to stdin of the submit command (whether that is sbatch or bsub etc.) and many templates have no need for input files or separate directories. The Job Composer doesn't support this workflow yet.
    >
    > As far as where these stage directories are created, the default staged directory root path looks like (for my user efranz):
    >
    > ~efranz/ondemand/data/sys/myjobs/projects/default/
    >
    > This part is configurable today to anything you want:
    >
    > ~efranz/ondemand/data/sys/myjobs
    >
    > The "projects/default/" part is hardcoded (https://github.com/OSC/ood-myjobs/blob/43e2fdfc8f2c5943ea4589a8b89909bf416b6595/app/models/workflow.rb#L63-L71) and thus looks like it can only be changed in OnDemand 1.3 using a monkey patch in a custom initializer. I can share with you how to do that but of course it is a temporary fix that could stop working in future versions.
    >
    > Also, the actual staged directories created follow a simple incrementing number naming convention:
    >
    > efranz at owens-login01:~$ ls -1 ~/ondemand/data/sys/myjobs/projects/default/ | sort -n | head
    > 1
    > 2
    > 3
    > 4
    > 5
    > 6
    > 7
    > 8
    > 9
    > 10
    >
    > So that’s probably not what you want.
    >
    > We have a number of feature issues opened in the Job Composer's GitHub project that might provide some of what you want to do, but it might not cover everything. If you are willing we could also flesh out in more detail the workflow you hope to achieve that is not possible now. A mailing list wouldn't be ideal for that, but we do have a Discourse instance (https://discourse.osc.edu/c/open-ondemand) or GitHub issues itself (https://github.com/OSC/ood-myjobs/issues) that might be easier to have that discussion.
    >
    > Thanks,
    > Eric
    >
    > ---
    > Eric Franz, Senior Web & Interface App Engineer
    > Ohio Supercomputer Center
    > An Ohio Technology Consortium (OH-TECH) Member
    > 1224 Kinnear Road
    > Columbus, OH 43212
    > email: efranz at osc.edu
    >
    > On 9/28/18, 4:08 PM, "OOD-users on behalf of mludwig" <ood-users-bounces at lists.osc.edu on behalf of mludwig at intranet.techsquare.com> wrote:
    >
    >      Hello all,
    >      
    >      I fear I may not have a solid understanding of the need for a staging
    >      directory when making a job in Job Composer. I am preparing a deployment
    >      of OOD for a slurm cluster where users have been doing all work through
    >      ssh. They already have years of batch scripts and data files in their
    >      home directory so I would like to be able to have Open OnDemand work out
    >      of these established directories instead of needing to make a copy to
    >      $HOME/ondemand/data/sys/myjobs/projects/{etc}. Is this possible?
    >      
    >      Thanks,
    >      
    >      Morgan
    >      
    >      
    >      _______________________________________________
    >      OOD-users mailing list
    >      OOD-users at lists.osc.edu
    >      https://lists.osu.edu/mailman/listinfo/ood-users
    >      
    >
    > _______________________________________________
    > OOD-users mailing list
    > OOD-users at lists.osc.edu
    > https://lists.osu.edu/mailman/listinfo/ood-users
    
    _______________________________________________
    OOD-users mailing list
    OOD-users at lists.osc.edu
    https://lists.osu.edu/mailman/listinfo/ood-users