[OOD-users] OOD-users Setting up Proper Environments

Fri Oct 13 12:29:21 EDT 2017

Thanks for the response Nicklas, 

If I submit without -V and no modules like this:
  #!/bin/bash
  #PBS -l select=1:ncpus=2:mpiprocs=2
  #PBS -N hostname.pbs
  #PBS -j oe

  mpirun hostname
  sleep 3

I get:
  /var/spool/pbs/mom_priv/jobs/32.vbcluster1.nor-tech.com.SC: line 10: mpirun: command not found

If I try adding my module load command I get this:
  #!/bin/bash
  #PBS -l select=1:ncpus=2:mpiprocs=2
  #PBS -N hostname.pbs
  #PBS -j oe

  module load prun/1.1 gnu7/7.1.0 openmpi/1.10.7 ohpc

  mpirun hostname
  sleep 3

I get:
  /var/spool/pbs/mom_priv/jobs/33.vbcluster1.nor-tech.com.SC: line 8: module: command not found
  /var/spool/pbs/mom_priv/jobs/33.vbcluster1.nor-tech.com.SC: line 10: mpirun: command not found

Thanks, Kyle

Kyle Gross | kyleg at nor-tech.com
Senior HPC Technician
901 East Cliff Road, Burnsville, MN 55337
Direct 952-229-2086 | Fax 952-229-2061
Burnsville, Minn.
People Friendly Technology with a Global Reach

-----Original Message-----
From: OOD-users [mailto:ood-users-bounces+kyleg=nor-tech.com at lists.osc.edu] On Behalf Of ood-users-request at lists.osc.edu
Sent: Friday, October 13, 2017 7:26 AM
To: ood-users at lists.osc.edu
Subject: OOD-users Digest, Vol 8, Issue 16

Send OOD-users mailing list submissions to
	ood-users at lists.osc.edu

To subscribe or unsubscribe via the World Wide Web, visit
	https://lists.osu.edu/mailman/listinfo/ood-users
or, via email, send a message with subject or body 'help' to
	ood-users-request at lists.osc.edu

You can reach the person managing the list at
	ood-users-owner at lists.osc.edu

When replying, please edit your Subject line so it is more specific than "Re: Contents of OOD-users digest..."

Today's Topics:

   1. Re: Setting up Proper Environments (Nicklas, Jeremy)
   2. Selecting Resources on Job Composer (Kyle Gross)

----------------------------------------------------------------------

Message: 1
Date: Thu, 12 Oct 2017 17:18:31 +0000
From: "Nicklas, Jeremy" <jnicklas at osc.edu>
To: User support maiing list for Open OnDemand
	<ood-users at lists.osc.edu>
Subject: Re: [OOD-users] Setting up Proper Environments
Message-ID:
	<16070CF2216CA0479D3C7A6D19ECEDB982F193E8 at CIO-KRC-D1MBX05.osuad.osu.edu>

Content-Type: text/plain; charset="windows-1252"

What happens if you remove the `#PBS -V` line? I was under the assumption that if you don't specify `qsub -V` it will automatically source the .profile and set that all up. It works for me at one of the centers that use PBSPro.

Currently the per-user NGINX (PUN) processes are run under a very clean environment (as they were forked off from a `sudo` process). It is recommended the job itself should setup its own environment. Typically the resource managers (other than LSF) will run the batch job within a login shell.

One reason you wouldn't want to carry over environment from the OnDemand node is that it can be dramatically different than the cluster node the job is run on. Some of the environment variables PATH and LD_LIBRARY_PATH can point to directories that don't exist on the clusters.

Jeremy Nicklas
Web and Interface App Engineer
Ohio Supercomputer Center (OSC)<https://osc.edu/> A member of the Ohio Technology Consortium<https://oh-tech.org/>
1224 Kinnear Road, Columbus, Ohio 43212
Office: (614) 292-6739<tel:+16142926739> ? Mobile: (614) 316-6428<tel:+16143166428> ? Fax: (614) 292-7168<tel:+16142927168> jnicklas at osc.edu<mailto:jnicklas at osc.edu>

Learn more about OSC at https://osc.edu<https://osc.edu/> ________________________________
From: OOD-users [ood-users-bounces+jnicklas=osc.edu at lists.osc.edu] on behalf of Dominic Daninger [domd at nor-tech.com]
Sent: Thursday, October 12, 2017 12:59 PM
To: ood-users at lists.osc.edu
Subject: [OOD-users] Setting up Proper Environments

Hi All, We are going through our first setup of ood and have a question regarding environments.  If I were to submit a job outside of the ood browser using a submission script like this to PBS:

#!/bin/bash
#PBS -V
#PBS -l select=1:ncpus=2:mpiprocs=2
#PBS -l walltime=01:00
#PBS -N hostname.pbs
#PBS -j oe

module load prun/1.1 gnu7/7.1.0 openmpi/1.10.7 ohpc mpirun hostname

Resulting in an output file similar to this:
compute-0-1
compute-0-1

If I submit the same job through the ood console using the Job Composer I end up getting this error:
/var/spool/pbs/mom_priv/jobs/29.vbcluster1.nor-tech.com.SC: line 9: module: command not found
/var/spool/pbs/mom_priv/jobs/29.vbcluster1.nor-tech.com.SC: line 11: mpirun: command not found

I have been able to get around this issue by sourcing the lmod init script inside my job script like this:

#!/bin/bash
#PBS -V
#PBS -l select=1:ncpus=2:mpiprocs=2
#PBS -N hostname.pbs
#PBS -j oe

export MODULEPATH=/opt/ohpc/pub/moduledeps/gnu7-openmpi:/opt/ohpc/pub/moduledeps/gnu7:/opt/ohpc/pub/modulefiles
source /opt/ohpc/admin/lmod/lmod/init/bash
module load prun/1.1 gnu7/7.1.0 openmpi/1.10.7 ohpc

mpirun hostname

Is there a way to pass the user?s full environment to the job so we don?t have to do the sourcing above?  Usually in PBS you?d use the ?#PBS ?V? option but that is not working with the ood.

Dominic Daninger | domd at nor-tech.com<mailto:domd at nor-tech.com>
Vice President of Engineering
Direct 952-229-2070 | Cell 612 251 3505 | Fax 952-229-2061 [NT logo.jpg]Burnsville, Minn.
People Friendly Technology with a Global Reach

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osu.edu/pipermail/ood-users/attachments/20171012/2618a0f2/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image003.jpg
Type: image/jpeg
Size: 1917 bytes
Desc: image003.jpg
URL: <http://lists.osu.edu/pipermail/ood-users/attachments/20171012/2618a0f2/attachment-0001.jpg>

------------------------------

Message: 2
Date: Fri, 13 Oct 2017 12:25:19 +0000
From: Kyle Gross <Kyleg at nor-tech.com>
To: "ood-users at lists.osc.edu" <ood-users at lists.osc.edu>
Subject: [OOD-users] Selecting Resources on Job Composer
Message-ID: <A7EB80A20E6BA74CBB72C6B53204F02F0D543FC6 at NT-EXCH.ntc.com>
Content-Type: text/plain; charset="us-ascii"

How would one go about adding job resources to the Job Composer page?  If you have any tips or docs you could point me to it would be greatly appreciated.

Ideally we would like to end up with drop down menus or fill in the blank for things like:
CORES=
NODES=
QUEUE=

Instead of writing them in our script contents like this?
#!/bin/bash
#PBS -V
#PBS -l select=1:ncpus=2:mpiprocs=2
#PBS -N hostname.pbs
#PBS -q workq
#PBS -j oe

mpirun hostname

Thanks, Kyle

Kyle Gross | kyleg at nor-tech.com<mailto:kyleg at nor-tech.com>
Senior HPC Technician
901 East Cliff Road, Burnsville, MN 55337 Direct 952-229-2086 | Fax 952-229-2061 [NT logo.jpg]Burnsville, Minn.
People Friendly Technology with a Global Reach

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osu.edu/pipermail/ood-users/attachments/20171013/606da153/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.jpg
Type: image/jpeg
Size: 2015 bytes
Desc: image001.jpg
URL: <http://lists.osu.edu/pipermail/ood-users/attachments/20171013/606da153/attachment.jpg>

------------------------------

Subject: Digest Footer

_______________________________________________
OOD-users mailing list
OOD-users at lists.osc.edu
https://lists.osu.edu/mailman/listinfo/ood-users

------------------------------

End of OOD-users Digest, Vol 8, Issue 16
****************************************