<html><head><meta http-equiv="content-type" content="text/html; charset=utf-8"></head><body dir="auto"><div dir="ltr"></div><div dir="ltr">I gotta say, I am trying to read it very very carefully but don’t seem to be able to parse out the subtlety. 😐</div><div dir="ltr"><br></div><div dir="ltr">Not sure how to miss read this, emphasis on all theirs:</div><div dir="ltr"><br></div><div dir="ltr"><br></div><div dir="ltr"><dt style="box-sizing: border-box;"><span style="background-color: rgba(255, 255, 255, 0);"><b style="box-sizing: border-box; margin: 0px; padding: 0px; border: 0px; font-style: inherit; font-variant-caps: inherit; font-stretch: inherit; line-height: inherit; vertical-align: baseline;">--export</b>=<<span style="box-sizing: border-box; margin: 0px; padding: 0px; border: 0px; font-style: italic; font-variant-caps: inherit; font-stretch: inherit; line-height: inherit; vertical-align: baseline;">environment variables [ALL] | NONE</span>></span></dt><dd style="box-sizing: border-box;"><span style="background-color: rgba(255, 255, 255, 0);">Identify which environment variables from the submission environment are propagated to the launched application. By default, <span style="box-sizing: border-box; margin: 0px; padding: 0px; border: 0px; font-style: italic; font-variant-caps: inherit; font-stretch: inherit; line-height: inherit; vertical-align: baseline;">all</span> are propagated. Multiple environment variable names should be comma separated. </span></dd></div><div dir="ltr"><br></div><div dir="ltr"><br></div><div dir="ltr"><br>On Dec 7, 2018, at 10:24 PM, Michael Coleman <<a href="mailto:mcolema5@uoregon.edu">mcolema5@uoregon.edu</a>> wrote:<br><br></div><blockquote type="cite"><div dir="ltr"><span>If you read the man page very, very carefully, I think the new behavior actually matches what it always said. The old behavior didn't actually match. But I definitely agree that it's a pretty significant change.</span><br><span></span><br><span>Michael Coleman (<a href="mailto:mcolema5@uoregon.edu">mcolema5@uoregon.edu</a>), Computational Scientist</span><br><span>Research Advanced Computing Services</span><br><span>6235 University of Oregon</span><br><span>Eugene, OR 97403</span><br><span></span><br><span></span><br><span>________________________________________</span><br><span>From: John-Paul Robinson <<a href="mailto:jprorama@gmail.com">jprorama@gmail.com</a>></span><br><span>Sent: Friday, December 7, 2018 17:19</span><br><span>To: Michael Coleman</span><br><span>Cc: User support mailing list for Open OnDemand</span><br><span>Subject: Re: [OOD-users] no environment set for HPC desktop -- job fails</span><br><span></span><br><span>I’m also scratching my head because documentation (even for the 18 release) still says the default of export is to propagate all environment variables. :/</span><br><span></span><br><blockquote type="cite"><span>On Dec 7, 2018, at 6:59 PM, John-Paul Robinson <<a href="mailto:jprorama@gmail.com">jprorama@gmail.com</a>> wrote:</span><br></blockquote><blockquote type="cite"><span></span><br></blockquote><blockquote type="cite"><span>Its a pretty alarming change to have happened in three ticks of the minor release number.</span><br></blockquote><blockquote type="cite"><span></span><br></blockquote><blockquote type="cite"><span>Any insights from others on this?</span><br></blockquote><blockquote type="cite"><span></span><br></blockquote><blockquote type="cite"><blockquote type="cite"><span>On Dec 7, 2018, at 6:37 PM, Michael Coleman <<a href="mailto:mcolema5@uoregon.edu">mcolema5@uoregon.edu</a>> wrote:</span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span></span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span>Hi John-Paul,</span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span></span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span>As you say, I believe the key event was the transition in SLURM versions. They apparently made a change in the behavior of export of environment variables from the submitting environment to the job environment (the stuff --export controls). There was much wailing of our users here when their sbatch scripts broke as a result. Generally, the "fix" was simply for users that were already using the --export flag to add the "ALL" keyword to that list, which seemed to restore the old behavior.</span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span></span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span>Ultimately, OOD is calling 'sbatch' to create jobs, and this change affects the environment those jobs see. At least in our environment, the --export=ALL flag seems to cure OOD issues. There are probably other ways to change things, but this seemed the simplest.</span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span></span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span>Good luck,</span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span>Mike</span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span></span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span></span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span>-----Original Message-----</span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span>From: John-Paul Robinson <<a href="mailto:jprorama@gmail.com">jprorama@gmail.com</a>></span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span>Sent: Friday, December 7, 2018 03:56 PM</span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span>To: Michael Coleman <<a href="mailto:mcolema5@uoregon.edu">mcolema5@uoregon.edu</a>>; User support mailing list for Open OnDemand <<a href="mailto:ood-users@lists.osc.edu">ood-users@lists.osc.edu</a>></span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span>Subject: Re: [OOD-users] no environment set for HPC desktop -- job fails</span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span></span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span>MIke,</span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span></span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span>Thanks for pointing us to this issue.</span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span></span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span>This does appear to be similar to what's happening in our dev</span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span>environment. (Note our still-working prod environment is Bright CM with</span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span>slurm 17.02.2).</span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span></span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span>The odd thing with our dev environment (built on OpenHPC) is that it was</span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span>working in October and only started failing in builds over the past</span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span>month. This appears to coincide with the OpenHPC 1.3.5 to 1.3.6 update</span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span>(going from slurm 17.11.7 to 17.11.10).</span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span></span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span>We've had some success in restoring the original working configuration</span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span>in one of our test stacks by reverting to the OpenHPC 1.3.5 release.</span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span></span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span>What's odd is this implies the problem is not with OOD but in the</span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span>OpenHPC system env. As far as we can determine, our OOD remains</span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span>identical. We are setting up dev in vagrant with ansible provisioning</span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span>the openhpc + ood cluster based on the CRI_XSEDE work extended to add an</span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span>OOD node via vagrant + ansible (not as a warewulf provision).</span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span></span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span><a href="https://github.com/jprorama/CRI_XCBC">https://github.com/jprorama/CRI_XCBC</a></span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span></span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span>I've read through the github issue below but haven't teased out all the</span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span>details.</span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span></span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span>Is there an obvious transition point where this export behavior could be</span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span>impacted by the underlying system versions OOD is running on?</span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span></span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span>We'll contribute insights on the github issue as we find them.</span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span></span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span>Thanks,</span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span></span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span>John-Paul</span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span></span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span></span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><span>On 12/5/18 5:15 PM, Michael Coleman wrote:</span><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><span>Hi John-Paul,</span><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><span></span><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><span>We worked through something similar. You might find some useful hints on this ticket.</span><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><span></span><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><span> <a href="https://github.com/OSC/ood_core/issues/109">https://github.com/OSC/ood_core/issues/109</a></span><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><span></span><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><span>Cheers,</span><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><span>Mike</span><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><span></span><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><span></span><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><span></span><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><span></span><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><span>-----Original Message-----</span><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><span>From: OOD-users <<a href="mailto:ood-users-bounces+mcolema5=uoregon.edu@lists.osc.edu">ood-users-bounces+mcolema5=uoregon.edu@lists.osc.edu</a>> On Behalf Of John-Paul Robinson via OOD-users</span><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><span>Sent: Wednesday, December 5, 2018 02:49 PM</span><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><span>To: <a href="mailto:ood-users@lists.osc.edu">ood-users@lists.osc.edu</a></span><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><span>Subject: [OOD-users] no environment set for HPC desktop -- job fails</span><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><span></span><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><span>In our dev environment (slurm with ohpc) we have started to see this</span><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><span>error when trying to launch interactive desktops:</span><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><span></span><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><span>/tmp/slurmd/job00079/slurm_script: line 3: module: command not found</span><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><span>Setting VNC password...</span><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><span>Error: no HOME environment variable</span><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><span>Starting VNC server...</span><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><span>vncserver: The HOME environment variable is not set.</span><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><span>vncserver: The HOME environment variable is not set.</span><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><span>vncserver: The HOME environment variable is not set. vncserver: The HOME</span><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><span>environment variable is</span><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><span></span><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><span></span><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><span>As we understand it, the PUN nginx worker launches the batch job that</span><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><span>starts the desktop batch job.</span><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><span></span><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><span>The problem seems to be that the environment for the job is empty, hence</span><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><span>no module function or HOME env or anything else. We checked the env of</span><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><span>the users nginx worker under /proc and it is completely empty. Because</span><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><span>our job env is inherited from the caller (the nginx worker in this case)</span><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><span>the attempt to run the module command and vncserver commands naturally fail.</span><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><span></span><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><span>When we launch an interactive terminal, it runs just fine, but I'm</span><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><span>guessing that's because the interactive session actually reads the</span><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><span>normal shell startup and builds its environment, even if it happened to</span><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><span>be missing in the proxy.</span><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><span></span><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><span>Do you have any pointers on what could cause this situation. We</span><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><span>noticed it after we started adding additional interactive apps but don't</span><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><span>have a clear time point. It was working fine originally and still</span><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><span>functions fine in our prod env (without any of the additional</span><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><span>interactive apps).</span><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><span></span><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><span>Thanks,</span><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><span></span><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><span>John-Paul</span><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><span></span><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><span>_______________________________________________</span><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><span>OOD-users mailing list</span><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><span><a href="mailto:OOD-users@lists.osc.edu">OOD-users@lists.osc.edu</a></span><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><span><a href="https://lists.osu.edu/mailman/listinfo/ood-users">https://lists.osu.edu/mailman/listinfo/ood-users</a></span><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span></span><br></blockquote></blockquote></div></blockquote></body></html>