<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<style type="text/css" style="display:none"><!-- p { margin-top: 0px; margin-bottom: 0px; }--></style>
</head>
<body dir="ltr" style="font-size:10pt;color:#000000;background-color:#FFFFFF;font-family:Consolas,monospace;">
<p>The key change is what happens if you use a flag like<br>
</p>
<p><br>
</p>
<p> --export=MYVAR1,MYVAR2<br>
</p>
<p><br>
</p>
<p>The behavior mentioned in the man page (see <a href="https://github.com/OSC/ood_core/issues/109">https://github.com/OSC/ood_core/issues/109</a>) says that no other environment variables will be passed. In other words, this is equivalent to<br>
</p>
<p><br>
</p>
<p> --export=NONE,MYVAR1,MYVAR2<br>
</p>
<p><br>
</p>
<p>The recent fix actually makes sbatch behave this way. Previously, the implementation of this case was incorrectly equivalent to<br>
</p>
<p><br>
</p>
<p> --export=ALL,MYVAR1,MYVAR2<br>
</p>
<p><br>
</p>
<p>And just to make us all completely insane, it looks like they're changing this further in a newer release (that we don't yet use at our site). See the new man page here:<br>
</p>
<p><br>
</p>
<p> <a href="https://slurm.schedmd.com/sbatch.html">https://slurm.schedmd.com/sbatch.html</a><br>
</p>
<p><br>
</p>
<p>Mike<br>
</p>
<p><br>
</p>
<div id="Signature">
<div name="divtagdefaultwrapper" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:; margin:0">
<p class="MsoAutoSig" style=""><font size="2" face="Consolas, monospace" style="">Michael Coleman (<a tabindex="0" href="mailto:mcolema5@uoregon.edu" style="" id="NoLP">mcolema5@uoregon.edu</a>), Computational Scientist</font></p>
<p class="MsoAutoSig" style=""><font size="2" face="Consolas, monospace">Research Advanced Computing Services</font></p>
<div class="" style="color:rgb(33,33,33); background-color:rgb(255,255,255)"><font size="2" face="Consolas, monospace">6235 University of Oregon</font></div>
<div class="" style="color:rgb(33,33,33); background-color:rgb(255,255,255)"><font size="2" face="Consolas, monospace">Eugene, OR 97403</font></div>
<p class="MsoAutoSig" style="font-family:Calibri,Arial,Helvetica,sans-serif"><span style="font-size:10.0pt; font-family:Courier"><br style="">
</span></p>
</div>
</div>
<div dir="auto" style="color: rgb(33, 33, 33);">
<hr tabindex="-1" style="display:inline-block; width:98%">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" color="#000000" style="font-size:11pt"><b>From:</b> John-Paul Robinson <jprorama@gmail.com><br>
<b>Sent:</b> Saturday, December 8, 2018 10:54<br>
<b>To:</b> Michael Coleman<br>
<b>Cc:</b> User support mailing list for Open OnDemand<br>
<b>Subject:</b> Re: [OOD-users] no environment set for HPC desktop -- job fails</font>
<div> </div>
</div>
<div>
<div dir="ltr"></div>
<div dir="ltr">I gotta say, I am trying to read it very very carefully but don’t seem to be able to parse out the subtlety. 😐</div>
<div dir="ltr"><br>
</div>
<div dir="ltr">Not sure how to miss read this, emphasis on all theirs:</div>
<div dir="ltr"><br>
</div>
<div dir="ltr"><br>
</div>
<div dir="ltr"><dt style=""><span style=""><b style="margin:0px; padding:0px; border:0px; font-style:inherit; line-height:inherit; vertical-align:baseline">--export</b>=<<span style="margin:0px; padding:0px; border:0px; font-style:italic; line-height:inherit; vertical-align:baseline">environment
variables [ALL] | NONE</span>></span></dt><dd style=""><span style="">Identify which environment variables from the submission environment are propagated to the launched application. By default, <span style="margin:0px; padding:0px; border:0px; font-style:italic; line-height:inherit; vertical-align:baseline">all</span> are
propagated. Multiple environment variable names should be comma separated. </span></dd></div>
<div dir="ltr"><br>
</div>
<div dir="ltr"><br>
</div>
<div dir="ltr"><br>
On Dec 7, 2018, at 10:24 PM, Michael Coleman <<a href="mailto:mcolema5@uoregon.edu">mcolema5@uoregon.edu</a>> wrote:<br>
<br>
</div>
<blockquote type="cite">
<div dir="ltr"><span>If you read the man page very, very carefully, I think the new behavior actually matches what it always said. The old behavior didn't actually match. But I definitely agree that it's a pretty significant change.</span><br>
<br>
<span>Michael Coleman (<a href="mailto:mcolema5@uoregon.edu">mcolema5@uoregon.edu</a>), Computational Scientist</span><br>
<span>Research Advanced Computing Services</span><br>
<span>6235 University of Oregon</span><br>
<span>Eugene, OR 97403</span><br>
<br>
<br>
<span>________________________________________</span><br>
<span>From: John-Paul Robinson <<a href="mailto:jprorama@gmail.com">jprorama@gmail.com</a>></span><br>
<span>Sent: Friday, December 7, 2018 17:19</span><br>
<span>To: Michael Coleman</span><br>
<span>Cc: User support mailing list for Open OnDemand</span><br>
<span>Subject: Re: [OOD-users] no environment set for HPC desktop -- job fails</span><br>
<br>
<span>I’m also scratching my head because documentation (even for the 18 release) still says the default of export is to propagate all environment variables. :/</span><br>
<br>
<blockquote type="cite"><span>On Dec 7, 2018, at 6:59 PM, John-Paul Robinson <<a href="mailto:jprorama@gmail.com">jprorama@gmail.com</a>> wrote:</span><br>
</blockquote>
<blockquote type="cite"><br>
</blockquote>
<blockquote type="cite"><span>Its a pretty alarming change to have happened in three ticks of the minor release number.</span><br>
</blockquote>
<blockquote type="cite"><br>
</blockquote>
<blockquote type="cite"><span>Any insights from others on this?</span><br>
</blockquote>
<blockquote type="cite"><br>
</blockquote>
<blockquote type="cite">
<blockquote type="cite"><span>On Dec 7, 2018, at 6:37 PM, Michael Coleman <<a href="mailto:mcolema5@uoregon.edu">mcolema5@uoregon.edu</a>> wrote:</span><br>
</blockquote>
</blockquote>
<blockquote type="cite">
<blockquote type="cite"><br>
</blockquote>
</blockquote>
<blockquote type="cite">
<blockquote type="cite"><span>Hi John-Paul,</span><br>
</blockquote>
</blockquote>
<blockquote type="cite">
<blockquote type="cite"><br>
</blockquote>
</blockquote>
<blockquote type="cite">
<blockquote type="cite"><span>As you say, I believe the key event was the transition in SLURM versions. They apparently made a change in the behavior of export of environment variables from the submitting environment to the job environment (the stuff --export
controls). There was much wailing of our users here when their sbatch scripts broke as a result. Generally, the "fix" was simply for users that were already using the --export flag to add the "ALL" keyword to that list, which seemed to restore the old behavior.</span><br>
</blockquote>
</blockquote>
<blockquote type="cite">
<blockquote type="cite"><br>
</blockquote>
</blockquote>
<blockquote type="cite">
<blockquote type="cite"><span>Ultimately, OOD is calling 'sbatch' to create jobs, and this change affects the environment those jobs see. At least in our environment, the --export=ALL flag seems to cure OOD issues. There are probably other ways to change
things, but this seemed the simplest.</span><br>
</blockquote>
</blockquote>
<blockquote type="cite">
<blockquote type="cite"><br>
</blockquote>
</blockquote>
<blockquote type="cite">
<blockquote type="cite"><span>Good luck,</span><br>
</blockquote>
</blockquote>
<blockquote type="cite">
<blockquote type="cite"><span>Mike</span><br>
</blockquote>
</blockquote>
<blockquote type="cite">
<blockquote type="cite"><br>
</blockquote>
</blockquote>
<blockquote type="cite">
<blockquote type="cite"><br>
</blockquote>
</blockquote>
<blockquote type="cite">
<blockquote type="cite"><span>-----Original Message-----</span><br>
</blockquote>
</blockquote>
<blockquote type="cite">
<blockquote type="cite"><span>From: John-Paul Robinson <<a href="mailto:jprorama@gmail.com">jprorama@gmail.com</a>></span><br>
</blockquote>
</blockquote>
<blockquote type="cite">
<blockquote type="cite"><span>Sent: Friday, December 7, 2018 03:56 PM</span><br>
</blockquote>
</blockquote>
<blockquote type="cite">
<blockquote type="cite"><span>To: Michael Coleman <<a href="mailto:mcolema5@uoregon.edu">mcolema5@uoregon.edu</a>>; User support mailing list for Open OnDemand <<a href="mailto:ood-users@lists.osc.edu">ood-users@lists.osc.edu</a>></span><br>
</blockquote>
</blockquote>
<blockquote type="cite">
<blockquote type="cite"><span>Subject: Re: [OOD-users] no environment set for HPC desktop -- job fails</span><br>
</blockquote>
</blockquote>
<blockquote type="cite">
<blockquote type="cite"><br>
</blockquote>
</blockquote>
<blockquote type="cite">
<blockquote type="cite"><span>MIke,</span><br>
</blockquote>
</blockquote>
<blockquote type="cite">
<blockquote type="cite"><br>
</blockquote>
</blockquote>
<blockquote type="cite">
<blockquote type="cite"><span>Thanks for pointing us to this issue.</span><br>
</blockquote>
</blockquote>
<blockquote type="cite">
<blockquote type="cite"><br>
</blockquote>
</blockquote>
<blockquote type="cite">
<blockquote type="cite"><span>This does appear to be similar to what's happening in our dev</span><br>
</blockquote>
</blockquote>
<blockquote type="cite">
<blockquote type="cite"><span>environment. (Note our still-working prod environment is Bright CM with</span><br>
</blockquote>
</blockquote>
<blockquote type="cite">
<blockquote type="cite"><span>slurm 17.02.2).</span><br>
</blockquote>
</blockquote>
<blockquote type="cite">
<blockquote type="cite"><br>
</blockquote>
</blockquote>
<blockquote type="cite">
<blockquote type="cite"><span>The odd thing with our dev environment (built on OpenHPC) is that it was</span><br>
</blockquote>
</blockquote>
<blockquote type="cite">
<blockquote type="cite"><span>working in October and only started failing in builds over the past</span><br>
</blockquote>
</blockquote>
<blockquote type="cite">
<blockquote type="cite"><span>month. This appears to coincide with the OpenHPC 1.3.5 to 1.3.6 update</span><br>
</blockquote>
</blockquote>
<blockquote type="cite">
<blockquote type="cite"><span>(going from slurm 17.11.7 to 17.11.10).</span><br>
</blockquote>
</blockquote>
<blockquote type="cite">
<blockquote type="cite"><br>
</blockquote>
</blockquote>
<blockquote type="cite">
<blockquote type="cite"><span>We've had some success in restoring the original working configuration</span><br>
</blockquote>
</blockquote>
<blockquote type="cite">
<blockquote type="cite"><span>in one of our test stacks by reverting to the OpenHPC 1.3.5 release.</span><br>
</blockquote>
</blockquote>
<blockquote type="cite">
<blockquote type="cite"><br>
</blockquote>
</blockquote>
<blockquote type="cite">
<blockquote type="cite"><span>What's odd is this implies the problem is not with OOD but in the</span><br>
</blockquote>
</blockquote>
<blockquote type="cite">
<blockquote type="cite"><span>OpenHPC system env. As far as we can determine, our OOD remains</span><br>
</blockquote>
</blockquote>
<blockquote type="cite">
<blockquote type="cite"><span>identical. We are setting up dev in vagrant with ansible provisioning</span><br>
</blockquote>
</blockquote>
<blockquote type="cite">
<blockquote type="cite"><span>the openhpc + ood cluster based on the CRI_XSEDE work extended to add an</span><br>
</blockquote>
</blockquote>
<blockquote type="cite">
<blockquote type="cite"><span>OOD node via vagrant + ansible (not as a warewulf provision).</span><br>
</blockquote>
</blockquote>
<blockquote type="cite">
<blockquote type="cite"><br>
</blockquote>
</blockquote>
<blockquote type="cite">
<blockquote type="cite"><span><a href="https://github.com/jprorama/CRI_XCBC">https://github.com/jprorama/CRI_XCBC</a></span><br>
</blockquote>
</blockquote>
<blockquote type="cite">
<blockquote type="cite"><br>
</blockquote>
</blockquote>
<blockquote type="cite">
<blockquote type="cite"><span>I've read through the github issue below but haven't teased out all the</span><br>
</blockquote>
</blockquote>
<blockquote type="cite">
<blockquote type="cite"><span>details.</span><br>
</blockquote>
</blockquote>
<blockquote type="cite">
<blockquote type="cite"><br>
</blockquote>
</blockquote>
<blockquote type="cite">
<blockquote type="cite"><span>Is there an obvious transition point where this export behavior could be</span><br>
</blockquote>
</blockquote>
<blockquote type="cite">
<blockquote type="cite"><span>impacted by the underlying system versions OOD is running on?</span><br>
</blockquote>
</blockquote>
<blockquote type="cite">
<blockquote type="cite"><br>
</blockquote>
</blockquote>
<blockquote type="cite">
<blockquote type="cite"><span>We'll contribute insights on the github issue as we find them.</span><br>
</blockquote>
</blockquote>
<blockquote type="cite">
<blockquote type="cite"><br>
</blockquote>
</blockquote>
<blockquote type="cite">
<blockquote type="cite"><span>Thanks,</span><br>
</blockquote>
</blockquote>
<blockquote type="cite">
<blockquote type="cite"><br>
</blockquote>
</blockquote>
<blockquote type="cite">
<blockquote type="cite"><span>John-Paul</span><br>
</blockquote>
</blockquote>
<blockquote type="cite">
<blockquote type="cite"><br>
</blockquote>
</blockquote>
<blockquote type="cite">
<blockquote type="cite"><br>
</blockquote>
</blockquote>
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite"><span>On 12/5/18 5:15 PM, Michael Coleman wrote:</span><br>
</blockquote>
</blockquote>
</blockquote>
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite"><span>Hi John-Paul,</span><br>
</blockquote>
</blockquote>
</blockquote>
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite"><br>
</blockquote>
</blockquote>
</blockquote>
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite"><span>We worked through something similar. You might find some useful hints on this ticket.</span><br>
</blockquote>
</blockquote>
</blockquote>
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite"><br>
</blockquote>
</blockquote>
</blockquote>
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite"><span> <a href="https://github.com/OSC/ood_core/issues/109">https://github.com/OSC/ood_core/issues/109</a></span><br>
</blockquote>
</blockquote>
</blockquote>
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite"><br>
</blockquote>
</blockquote>
</blockquote>
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite"><span>Cheers,</span><br>
</blockquote>
</blockquote>
</blockquote>
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite"><span>Mike</span><br>
</blockquote>
</blockquote>
</blockquote>
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite"><br>
</blockquote>
</blockquote>
</blockquote>
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite"><br>
</blockquote>
</blockquote>
</blockquote>
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite"><br>
</blockquote>
</blockquote>
</blockquote>
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite"><br>
</blockquote>
</blockquote>
</blockquote>
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite"><span>-----Original Message-----</span><br>
</blockquote>
</blockquote>
</blockquote>
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite"><span>From: OOD-users <<a href="mailto:ood-users-bounces+mcolema5=uoregon.edu@lists.osc.edu">ood-users-bounces+mcolema5=uoregon.edu@lists.osc.edu</a>> On Behalf Of John-Paul Robinson via OOD-users</span><br>
</blockquote>
</blockquote>
</blockquote>
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite"><span>Sent: Wednesday, December 5, 2018 02:49 PM</span><br>
</blockquote>
</blockquote>
</blockquote>
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite"><span>To: <a href="mailto:ood-users@lists.osc.edu">ood-users@lists.osc.edu</a></span><br>
</blockquote>
</blockquote>
</blockquote>
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite"><span>Subject: [OOD-users] no environment set for HPC desktop -- job fails</span><br>
</blockquote>
</blockquote>
</blockquote>
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite"><br>
</blockquote>
</blockquote>
</blockquote>
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite"><span>In our dev environment (slurm with ohpc) we have started to see this</span><br>
</blockquote>
</blockquote>
</blockquote>
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite"><span>error when trying to launch interactive desktops:</span><br>
</blockquote>
</blockquote>
</blockquote>
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite"><br>
</blockquote>
</blockquote>
</blockquote>
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite"><span>/tmp/slurmd/job00079/slurm_script: line 3: module: command not found</span><br>
</blockquote>
</blockquote>
</blockquote>
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite"><span>Setting VNC password...</span><br>
</blockquote>
</blockquote>
</blockquote>
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite"><span>Error: no HOME environment variable</span><br>
</blockquote>
</blockquote>
</blockquote>
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite"><span>Starting VNC server...</span><br>
</blockquote>
</blockquote>
</blockquote>
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite"><span>vncserver: The HOME environment variable is not set.</span><br>
</blockquote>
</blockquote>
</blockquote>
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite"><span>vncserver: The HOME environment variable is not set.</span><br>
</blockquote>
</blockquote>
</blockquote>
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite"><span>vncserver: The HOME environment variable is not set. vncserver: The HOME</span><br>
</blockquote>
</blockquote>
</blockquote>
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite"><span>environment variable is</span><br>
</blockquote>
</blockquote>
</blockquote>
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite"><br>
</blockquote>
</blockquote>
</blockquote>
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite"><br>
</blockquote>
</blockquote>
</blockquote>
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite"><span>As we understand it, the PUN nginx worker launches the batch job that</span><br>
</blockquote>
</blockquote>
</blockquote>
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite"><span>starts the desktop batch job.</span><br>
</blockquote>
</blockquote>
</blockquote>
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite"><br>
</blockquote>
</blockquote>
</blockquote>
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite"><span>The problem seems to be that the environment for the job is empty, hence</span><br>
</blockquote>
</blockquote>
</blockquote>
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite"><span>no module function or HOME env or anything else. We checked the env of</span><br>
</blockquote>
</blockquote>
</blockquote>
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite"><span>the users nginx worker under /proc and it is completely empty. Because</span><br>
</blockquote>
</blockquote>
</blockquote>
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite"><span>our job env is inherited from the caller (the nginx worker in this case)</span><br>
</blockquote>
</blockquote>
</blockquote>
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite"><span>the attempt to run the module command and vncserver commands naturally fail.</span><br>
</blockquote>
</blockquote>
</blockquote>
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite"><br>
</blockquote>
</blockquote>
</blockquote>
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite"><span>When we launch an interactive terminal, it runs just fine, but I'm</span><br>
</blockquote>
</blockquote>
</blockquote>
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite"><span>guessing that's because the interactive session actually reads the</span><br>
</blockquote>
</blockquote>
</blockquote>
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite"><span>normal shell startup and builds its environment, even if it happened to</span><br>
</blockquote>
</blockquote>
</blockquote>
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite"><span>be missing in the proxy.</span><br>
</blockquote>
</blockquote>
</blockquote>
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite"><br>
</blockquote>
</blockquote>
</blockquote>
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite"><span>Do you have any pointers on what could cause this situation. We</span><br>
</blockquote>
</blockquote>
</blockquote>
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite"><span>noticed it after we started adding additional interactive apps but don't</span><br>
</blockquote>
</blockquote>
</blockquote>
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite"><span>have a clear time point. It was working fine originally and still</span><br>
</blockquote>
</blockquote>
</blockquote>
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite"><span>functions fine in our prod env (without any of the additional</span><br>
</blockquote>
</blockquote>
</blockquote>
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite"><span>interactive apps).</span><br>
</blockquote>
</blockquote>
</blockquote>
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite"><br>
</blockquote>
</blockquote>
</blockquote>
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite"><span>Thanks,</span><br>
</blockquote>
</blockquote>
</blockquote>
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite"><br>
</blockquote>
</blockquote>
</blockquote>
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite"><span>John-Paul</span><br>
</blockquote>
</blockquote>
</blockquote>
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite"><br>
</blockquote>
</blockquote>
</blockquote>
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite"><span>_______________________________________________</span><br>
</blockquote>
</blockquote>
</blockquote>
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite"><span>OOD-users mailing list</span><br>
</blockquote>
</blockquote>
</blockquote>
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite"><span><a href="mailto:OOD-users@lists.osc.edu">OOD-users@lists.osc.edu</a></span><br>
</blockquote>
</blockquote>
</blockquote>
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite"><span><a href="https://lists.osu.edu/mailman/listinfo/ood-users">https://lists.osu.edu/mailman/listinfo/ood-users</a></span><br>
</blockquote>
</blockquote>
</blockquote>
<blockquote type="cite">
<blockquote type="cite"><br>
</blockquote>
</blockquote>
</div>
</blockquote>
</div>
</div>
</body>
</html>