[OOD-users] Modify Job Composer to just use a working directory

[Franz, Eric]

Morgan,

The way the Job Composer currently works is there are one or more template directories that contain jobs (and a default template). Users make a copy of this directory, which may contain a job script and maybe contain related files such as input files. You can see what we use at OSC here: https://github.com/OSC/osc-ood-config/tree/master/ondemand.osc.edu/apps/myjobs/templates. The workflow is copy the directory, modify files, submit the job, and the job copies files back to that directory.

Most of our adapters do pass the job script to stdin of the submit command (whether that is sbatch or bsub etc.) and many templates have no need for input files or separate directories. The Job Composer doesn't support this workflow yet.

As far as where these stage directories are created, the default staged directory root path looks like (for my user efranz):

~efranz/ondemand/data/sys/myjobs/projects/default/

This part is configurable today to anything you want:

~efranz/ondemand/data/sys/myjobs

The "projects/default/" part is hardcoded (https://github.com/OSC/ood-myjobs/blob/43e2fdfc8f2c5943ea4589a8b89909bf416b6595/app/models/workflow.rb#L63-L71) and thus looks like it can only be changed in OnDemand 1.3 using a monkey patch in a custom initializer. I can share with you how to do that but of course it is a temporary fix that could stop working in future versions.

Also, the actual staged directories created follow a simple incrementing number naming convention:

efranz at owens-login01:~$ ls -1 ~/ondemand/data/sys/myjobs/projects/default/ | sort -n | head
1
2
3
4
5
6
7
8
9
10

So that’s probably not what you want.

We have a number of feature issues opened in the Job Composer's GitHub project that might provide some of what you want to do, but it might not cover everything. If you are willing we could also flesh out in more detail the workflow you hope to achieve that is not possible now. A mailing list wouldn't be ideal for that, but we do have a Discourse instance (https://discourse.osc.edu/c/open-ondemand) or GitHub issues itself (https://github.com/OSC/ood-myjobs/issues) that might be easier to have that discussion.

Thanks,
Eric

---
Eric Franz, Senior Web & Interface App Engineer
Ohio Supercomputer Center
An Ohio Technology Consortium (OH-TECH) Member
1224 Kinnear Road
Columbus, OH 43212
email: efranz at osc.edu

On 9/28/18, 4:08 PM, "OOD-users on behalf of mludwig" <ood-users-bounces at lists.osc.edu on behalf of mludwig at intranet.techsquare.com> wrote:

    Hello all,
    
    I fear I may not have a solid understanding of the need for a staging 
    directory when making a job in Job Composer. I am preparing a deployment 
    of OOD for a slurm cluster where users have been doing all work through 
    ssh. They already have years of batch scripts and data files in their 
    home directory so I would like to be able to have Open OnDemand work out 
    of these established directories instead of needing to make a copy to 
    $HOME/ondemand/data/sys/myjobs/projects/{etc}. Is this possible?
    
    Thanks,
    
    Morgan
    
    
    _______________________________________________
    OOD-users mailing list
    OOD-users at lists.osc.edu
    https://lists.osu.edu/mailman/listinfo/ood-users