[OOD-users] Launching apps outside job system

[Franz, Eric]

Dan,

Were you thinking of starting these on the web node, login node, or just any dedicated node to run these processes on demand, without the wait imposed by the queue?

We proposed to tackle this problem in the CSSI proposal that was recently awarded so this is on the roadmap for next year. We discussed possibly building a “fork” adapter that would implement the same interface as the job adapters, but instead just fork + exec (or in your case, ssh + fork). The replacement for qstat would be a mixture of ps and inspecting /proc/<pid>; the “job id” would be the process id. qdel would just be kill. There would be problems introduced for process cleanup. Some type of containerization would be helpful but then that would require escalated privileges, complicating the implementation. Or we might investigate using systemd for supporting the process management.

At OSC we support such lightweight processing through a “quick cluster” to support “VDI” apps where jobs only get 1 core and we oversubscribe jobs on the nodes. That is explained in the relevant portions in this thread: https://listsprd.osu.edu/pipermail/ood-users/2018-March/000122.html. While they use the same batch submission (the torque adapter, in OSC’s case), jobs typically take 10-30 seconds to start instead of minutes. Would this approach work for you in the interim? Otherwise perhaps we could explore another option till a more robust “fork” adapter is available.

Thanks,
Eric

---
Eric Franz, Senior Web & Interface App Engineer
Ohio Supercomputer Center
An Ohio Technology Consortium (OH-TECH) Member
1224 Kinnear Road
Columbus, OH 43212
email: efranz at osc.edu

From: OOD-users <ood-users-bounces+efranz=osc.edu at lists.osc.edu> on behalf of Dan Dietz via OOD-users <ood-users at lists.osc.edu>
Reply-To: Dan Dietz <ddietz at purdue.edu>, User support mailing list for Open OnDemand <ood-users at lists.osc.edu>
Date: Tuesday, November 20, 2018 at 8:45 AM
To: "ood-users at lists.osc.edu" <ood-users at lists.osc.edu>
Subject: [OOD-users] Launching apps outside job system

I wonder if there is a way to launch the apps outside the job system, ie, directly on the cluster front-ends? We support running a few lightweight apps on cluster front-ends (visualizations, things like that). Is there a way to have it just "ssh" instead of "qsub" to start an interactive session?

Thanks,
Dan

--
Dan Dietz
Senior Scientific Applications Analyst
ITaP Research Computing, Purdue University
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osu.edu/pipermail/ood-users/attachments/20181120/cc36a088/attachment.html>