[OOD-users] OnDemand Cluster Config Issues

[Franz, Eric]

Chris,

Are you trying to debug submitting jobs from the Job Composer or launching Interactive Apps (whose jobs the Dashboard app submits)?

Both Job Composer and Dashboard submit jobs using the ood_core ruby library https://github.com/OSC/ood_core. The adapters all use shell commands to submit jobs: torque and pbspro uses qsub, slurm uses sbatch, lsf uses bsub.

The Dashboard app currently has a monkey patch added as an initializer to log execve calls from the app:

https://github.com/OSC/ood-dashboard/blob/7c4fbf45c5cdd415a9d76a3fda5c05dc8daf172a/config/initializers/open3_extensions.rb

These log statements appear in the user’s NGINX log file in /var/log/nginx/USER/error.log where Passenger by default writes all app specific log statements (replace USER with whatever username you are accessing as).

For example, in OSC OnDemand when I submit a job to start a Desktop session this line appears in the logfile:

App 69933 stdout: [2018-06-21 15:01:35 -0400 ]  INFO "execve = [{\"PBS_DEFAULT\"=>\"oak-batch.osc.edu\", \"LD_LIBRARY_PATH\"=>\"/opt/torque/lib64:/opt/rh/v8314/root/usr/lib64:/opt/rh/nodejs010/root/usr/lib64:/opt/rh/rh-ruby22/root/usr/lib64:/opt/rh/rh-passenger40/root/usr/lib64\"}, \"/opt/torque/bin/qsub\", \"-d\", \"/users/PZS0562/efranz/ondemand/data/sys/dashboard/batch_connect/sys/bc_desktop/oakley/output/56f817ae-7eed-4335-9b6e-6b1729febd5f\", \"-N\", \"ondemand/sys/dashboard/sys/bc_desktop/oakley\", \"-S\", \"/bin/bash\", \"-o\", \"/users/PZS0562/efranz/ondemand/data/sys/dashboard/batch_connect/sys/bc_desktop/oakley/output/56f817ae-7eed-4335-9b6e-6b1729febd5f/output.log\", \"-j\", \"oe\", \"-A\", \"PZS0530\", \"-l\", \"walltime=01:00:00\", \"-l\", \"nodes=1:ppn=12\", \"/tmp/qsub.20180621-69933-1db8ihm\"]"


The Job Composer doesn’t have this monkey patch but you can add it by using curl or wget to download https://raw.githubusercontent.com/OSC/ood-dashboard/7c4fbf45c5cdd415a9d76a3fda5c05dc8daf172a/config/initializers/open3_extensions.rb to /etc/ood/config/apps/myjobs/initializers/open3_extensions.rb. Restarting the per user web server will ensure that the Job Composer app starts with this initializer.

For example, I submit a job through the Job Composer and this line appears in my log file:

App 72779 stdout: [2018-06-21 15:12:09 -0400 ]  INFO "execve = [{\"PBS_DEFAULT\"=>\"oak-batch.osc.edu\", \"LD_LIBRARY_PATH\"=>\"/opt/torque/lib64:/opt/rh/v8314/root/usr/lib64:/opt/rh/nodejs010/root/usr/lib64:/opt/rh/rh-ruby22/root/usr/lib64:/opt/rh/rh-passenger40/root/usr/lib64\"}, \"/opt/torque/bin/qsub\", \"-j\", \"oe\", \"-A\", \"PZS0530\", \"/tmp/qsub.20180621-72779-f3gh05\"]"


Thanks,
Eric

---
Eric Franz, Senior Web & Interface App Engineer
Ohio Supercomputer Center
An Ohio Technology Consortium (OH-TECH) Member
1224 Kinnear Road
Columbus, OH 43212
email: efranz at osc.edu

From: OOD-users <ood-users-bounces at lists.osc.edu> on behalf of Christopher Ramirez <c.ramirez01 at gmail.com>
Reply-To: User support mailing list for Open OnDemand <ood-users at lists.osc.edu>
Date: Thursday, June 21, 2018 at 1:18 PM
To: "ood-users at lists.osc.edu" <ood-users at lists.osc.edu>
Subject: [OOD-users] OnDemand Cluster Config Issues

Hello,

We’re currently having an issue getting Ondemand configured. We can’t get the cluster config done correctly to allow for jobs to submit.
We’re trying to understand how the ondemand portal submits the job, or where it logs info, so we know better what needs to be addressed in our config file. Any assistance would be greatly appreciated.

-Chris

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osu.edu/pipermail/ood-users/attachments/20180621/4da175f4/attachment-0001.html>