[OOD-users] Modify Job Composer to just use a working directory

[mludwig]

Hey Eric,

Follow up question regarding your response. Where is the configuration 
that points the default staged directory? I would like to cut it down so 
that just the hardcoded "projects/default" is off $HOME/$USER.

Thanks,

Morgan


On 09/28/2018 05:36 PM, Franz, Eric wrote:
> Morgan,
>
> The way the Job Composer currently works is there are one or more template directories that contain jobs (and a default template). Users make a copy of this directory, which may contain a job script and maybe contain related files such as input files. You can see what we use at OSC here: https://github.com/OSC/osc-ood-config/tree/master/ondemand.osc.edu/apps/myjobs/templates. The workflow is copy the directory, modify files, submit the job, and the job copies files back to that directory.
>
> Most of our adapters do pass the job script to stdin of the submit command (whether that is sbatch or bsub etc.) and many templates have no need for input files or separate directories. The Job Composer doesn't support this workflow yet.
>
> As far as where these stage directories are created, the default staged directory root path looks like (for my user efranz):
>
> ~efranz/ondemand/data/sys/myjobs/projects/default/
>
> This part is configurable today to anything you want:
>
> ~efranz/ondemand/data/sys/myjobs
>
> The "projects/default/" part is hardcoded (https://github.com/OSC/ood-myjobs/blob/43e2fdfc8f2c5943ea4589a8b89909bf416b6595/app/models/workflow.rb#L63-L71) and thus looks like it can only be changed in OnDemand 1.3 using a monkey patch in a custom initializer. I can share with you how to do that but of course it is a temporary fix that could stop working in future versions.
>
> Also, the actual staged directories created follow a simple incrementing number naming convention:
>
> efranz at owens-login01:~$ ls -1 ~/ondemand/data/sys/myjobs/projects/default/ | sort -n | head
> 1
> 2
> 3
> 4
> 5
> 6
> 7
> 8
> 9
> 10
>
> So that’s probably not what you want.
>
> We have a number of feature issues opened in the Job Composer's GitHub project that might provide some of what you want to do, but it might not cover everything. If you are willing we could also flesh out in more detail the workflow you hope to achieve that is not possible now. A mailing list wouldn't be ideal for that, but we do have a Discourse instance (https://discourse.osc.edu/c/open-ondemand) or GitHub issues itself (https://github.com/OSC/ood-myjobs/issues) that might be easier to have that discussion.
>
> Thanks,
> Eric
>
> ---
> Eric Franz, Senior Web & Interface App Engineer
> Ohio Supercomputer Center
> An Ohio Technology Consortium (OH-TECH) Member
> 1224 Kinnear Road
> Columbus, OH 43212
> email: efranz at osc.edu
>
> On 9/28/18, 4:08 PM, "OOD-users on behalf of mludwig" <ood-users-bounces at lists.osc.edu on behalf of mludwig at intranet.techsquare.com> wrote:
>
>      Hello all,
>      
>      I fear I may not have a solid understanding of the need for a staging
>      directory when making a job in Job Composer. I am preparing a deployment
>      of OOD for a slurm cluster where users have been doing all work through
>      ssh. They already have years of batch scripts and data files in their
>      home directory so I would like to be able to have Open OnDemand work out
>      of these established directories instead of needing to make a copy to
>      $HOME/ondemand/data/sys/myjobs/projects/{etc}. Is this possible?
>      
>      Thanks,
>      
>      Morgan
>      
>      
>      _______________________________________________
>      OOD-users mailing list
>      OOD-users at lists.osc.edu
>      https://lists.osu.edu/mailman/listinfo/ood-users
>      
>
> _______________________________________________
> OOD-users mailing list
> OOD-users at lists.osc.edu
> https://lists.osu.edu/mailman/listinfo/ood-users