<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Helvetica;
panose-1:0 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Menlo;
panose-1:2 11 6 9 3 8 4 2 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;
mso-ligatures:standardcontextual;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
p.p1, li.p1, div.p1
{mso-style-name:p1;
margin:0in;
font-size:16.0pt;
font-family:Menlo;
color:black;}
span.s1
{mso-style-name:s1;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;
mso-ligatures:none;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:1737967613;
mso-list-template-ids:128065772;}
@list l0:level1
{mso-level-number-format:bullet;
mso-level-text:\F0B7 ;
mso-level-tab-stop:.5in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l0:level2
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:1.0in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:"Courier New";
mso-bidi-font-family:"Times New Roman";}
@list l0:level3
{mso-level-number-format:bullet;
mso-level-text:\F0A7 ;
mso-level-tab-stop:1.5in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l0:level4
{mso-level-number-format:bullet;
mso-level-text:\F0A7 ;
mso-level-tab-stop:2.0in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l0:level5
{mso-level-number-format:bullet;
mso-level-text:\F0A7 ;
mso-level-tab-stop:2.5in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l0:level6
{mso-level-number-format:bullet;
mso-level-text:\F0A7 ;
mso-level-tab-stop:3.0in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l0:level7
{mso-level-number-format:bullet;
mso-level-text:\F0A7 ;
mso-level-tab-stop:3.5in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l0:level8
{mso-level-number-format:bullet;
mso-level-text:\F0A7 ;
mso-level-tab-stop:4.0in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l0:level9
{mso-level-number-format:bullet;
mso-level-text:\F0A7 ;
mso-level-tab-stop:4.5in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
ol
{margin-bottom:0in;}
ul
{margin-bottom:0in;}
--></style>
</head>
<body lang="EN-US" link="#0563C1" vlink="#954F72" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal">Unity community,</p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Efficiency changes were recently made to the Unity scheduler that MAY have affected your multi-core/multi-node jobs.</p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">When submitting parallel (multi-core,multi-node,OMP,MPI) jobs please use these guidelines and terms.</p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Know when to use:</p>
<p class="MsoNormal">--cpus-per-task=# (used for Multithreaded jobs this is used for a single node multi-core job)<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">–ntasks=# (used for MPI usually over many nodes)</p>
<p class="MsoNormal">--ntasks-per-node=# (MPI tasks)</p>
<p class="MsoNormal">--nodes=# </p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:58.5pt;text-indent:-.25in;mso-list:l0 level1 lfo2;vertical-align:baseline">
<![if !supportLists]><span style="font-size:10.0pt;font-family:Symbol;mso-ligatures:none"><span style="mso-list:Ignore">·<span style="font:7.0pt "Times New Roman"">
</span></span></span><![endif]><span style="mso-ligatures:none">you want one process that can use 16 cores for multithreading: <span style="border:none windowtext 1.0pt;padding:0in">--ntasks=1 --cpus-per-task=16</span> (this should be one of the most common
requests)<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:58.5pt;text-indent:-.25in;mso-list:l0 level1 lfo2;vertical-align:baseline">
<![if !supportLists]><span style="font-size:10.0pt;font-family:Symbol;mso-ligatures:none"><span style="mso-list:Ignore">·<span style="font:7.0pt "Times New Roman"">
</span></span></span><![endif]><span style="mso-ligatures:none">you use mpi and do not care about where those cores are distributed: <span style="border:none windowtext 1.0pt;padding:0in">--ntasks=16</span><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:58.5pt;text-indent:-.25in;mso-list:l0 level1 lfo2;vertical-align:baseline">
<![if !supportLists]><span style="font-size:10.0pt;font-family:Symbol;mso-ligatures:none"><span style="mso-list:Ignore">·<span style="font:7.0pt "Times New Roman"">
</span></span></span><![endif]><span style="mso-ligatures:none">you want to launch 16 independent processes (no communication): <span style="border:none windowtext 1.0pt;padding:0in">--ntasks=16</span><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:58.5pt;text-indent:-.25in;mso-list:l0 level1 lfo2;vertical-align:baseline">
<![if !supportLists]><span style="font-size:10.0pt;font-family:Symbol;mso-ligatures:none"><span style="mso-list:Ignore">·<span style="font:7.0pt "Times New Roman"">
</span></span></span><![endif]><span style="mso-ligatures:none">you want those cores to spread across distinct nodes: <span style="border:none windowtext 1.0pt;padding:0in">--ntasks=16 and --ntasks-per-node=1</span> or <span style="border:none windowtext 1.0pt;padding:0in">--ntasks=16
and --nodes=16</span><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:58.5pt;text-indent:-.25in;mso-list:l0 level1 lfo2;vertical-align:baseline">
<![if !supportLists]><span style="font-size:10.0pt;font-family:Symbol;mso-ligatures:none"><span style="mso-list:Ignore">·<span style="font:7.0pt "Times New Roman"">
</span></span></span><![endif]><span style="mso-ligatures:none">you want those cores to spread across distinct nodes and no interference from other jobs: <span style="border:none windowtext 1.0pt;padding:0in">--ntasks=16 --nodes=16 --exclusive</span><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:58.5pt;text-indent:-.25in;mso-list:l0 level1 lfo2;vertical-align:baseline">
<![if !supportLists]><span style="font-size:10.0pt;font-family:Symbol;mso-ligatures:none"><span style="mso-list:Ignore">·<span style="font:7.0pt "Times New Roman"">
</span></span></span><![endif]><span style="mso-ligatures:none">you want 16 processes to spread across 8 nodes to have two processes per node: <span style="border:none windowtext 1.0pt;padding:0in">--ntasks=16 --ntasks-per-node=2</span><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:58.5pt;text-indent:-.25in;mso-list:l0 level1 lfo2;vertical-align:baseline">
<![if !supportLists]><span style="font-size:10.0pt;font-family:Symbol;mso-ligatures:none"><span style="mso-list:Ignore">·<span style="font:7.0pt "Times New Roman"">
</span></span></span><![endif]><span style="mso-ligatures:none">you want 16 processes to stay on the same node: <span style="border:none windowtext 1.0pt;padding:0in">--ntasks=16 --ntasks-per-node=16</span><o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:58.5pt;text-indent:-.25in;mso-list:l0 level1 lfo2;vertical-align:baseline">
<![if !supportLists]><span style="font-size:10.0pt;font-family:Symbol;mso-ligatures:none"><span style="mso-list:Ignore">·<span style="font:7.0pt "Times New Roman"">
</span></span></span><![endif]><span style="mso-ligatures:none">you want 4 processes that can use 4 cores each for multithreading: <span style="border:none windowtext 1.0pt;padding:0in">--ntasks=4 --cpus-per-task=4</span><o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">See:</p>
<p class="MsoNormal"><a href="https://stackoverflow.com/questions/51139711/hpc-cluster-select-the-number-of-cpus-and-threads-in-slurm-sbatch">https://stackoverflow.com/questions/51139711/hpc-cluster-select-the-number-of-cpus-and-threads-in-slurm-sbatch</a></p>
<p class="MsoNormal"><a href="https://support.ceci-hpc.be/doc/_contents/SubmittingJobs/SlurmFAQ.html#Q05">https://support.ceci-hpc.be/doc/_contents/SubmittingJobs/SlurmFAQ.html#Q05</a></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">As always please use “seff $JOBNUM” after your job completes and you can see your efficiency on number of nodes, cores and memory. IT is very possible to reserve MORE THAN you need, and your program may never use the extra memory, cores
and nodes. (wasting resources)</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">Example:</p>
<p class="p1"><span class="s1"><span style="font-size:11.0pt">Job ID: 570XXXXXX</span></span><span style="font-size:11.0pt"><o:p></o:p></span></p>
<p class="p1"><span class="s1"><span style="font-size:11.0pt">Cluster: unity</span></span><span style="font-size:11.0pt"><o:p></o:p></span></p>
<p class="p1"><span class="s1"><span style="font-size:11.0pt">User/Group: yourname.n/group</span></span><span style="font-size:11.0pt"><o:p></o:p></span></p>
<p class="p1"><span class="s1"><span style="font-size:11.0pt">State: COMPLETED (exit code 0)</span></span><span style="font-size:11.0pt"><o:p></o:p></span></p>
<p class="p1"><span class="s1"><span style="font-size:11.0pt">Cores: 1</span></span><span style="font-size:11.0pt"><o:p></o:p></span></p>
<p class="p1"><span class="s1"><span style="font-size:11.0pt">CPU Utilized: 00:59:38</span></span><span style="font-size:11.0pt"><o:p></o:p></span></p>
<p class="p1"><span class="s1"><span style="font-size:11.0pt">CPU Efficiency: 99.94% of 00:59:40 core-walltime</span></span><span style="font-size:11.0pt"><o:p></o:p></span></p>
<p class="p1"><span class="s1"><span style="font-size:11.0pt">Job Wall-clock time: 00:59:40</span></span><span style="font-size:11.0pt"><o:p></o:p></span></p>
<p class="p1"><span class="s1"><span style="font-size:11.0pt">Memory Utilized: 25.60 MB</span></span><span style="font-size:11.0pt"><o:p></o:p></span></p>
<p class="p1"><span class="s1"><span style="font-size:11.0pt">Memory Efficiency: 2.50% of 1.00 GB
<o:p></o:p></span></span></p>
<p class="p1"><span class="s1"><span style="font-size:11.0pt"><o:p> </o:p></span></span></p>
<p class="MsoNormal">In this case, a single CPU was very effectively used and memory was over requested. (This is fine in this case)</p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Keith</p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal"><span style="font-size:13.5pt;color:black;mso-ligatures:none">--<o:p></o:p></span></p>
<p class="MsoNormal"><b><span style="font-size:9.0pt;font-family:Helvetica;color:#FF7300;background:black;mso-ligatures:none">Keith A Stewart,</span></b><span style="font-size:9.0pt;font-family:Helvetica;color:#272727;background:white;mso-ligatures:none"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:Helvetica;color:black;mso-ligatures:none">Senior HPC/Scientific Computing Engineer</span><span style="font-size:13.5pt;color:black;mso-ligatures:none"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:Helvetica;color:#FF7300;background:black;mso-ligatures:none">College of Arts and Sciences</span><span style="font-size:9.0pt;font-family:Helvetica;color:#272727;background:white;mso-ligatures:none"> Arts
and Sciences Technology Services</span><span style="font-size:9.0pt;font-family:Helvetica;color:#272727;mso-ligatures:none"><br>
<span style="background:white">614-688-8291</span></span><span style="font-size:12.0pt;color:black;mso-ligatures:none"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:Helvetica;color:black;mso-ligatures:none">Help link: </span><a href="http://go.osu.edu/asctechhelp" title="http://go.osu.edu/asctechhelp"><span style="font-size:9.0pt;font-family:Helvetica;color:#000068;mso-ligatures:none">service/help
request</span></a><span style="font-size:9.0pt;font-family:Helvetica;color:#272727;background:white;mso-ligatures:none"><br>
</span><a href="mailto:stewart.358@osu.edu" title="mailto:stewart.358@osu.edu"><b><span style="font-size:9.0pt;font-family:Helvetica;color:#044A91;background:white;mso-ligatures:none">stewart.358@osu.edu</span></b></a><span style="font-size:9.0pt;font-family:Helvetica;color:#272727;background:white;mso-ligatures:none"> </span><a href="http://osu.edu/" title="http://osu.edu/"><b><span style="font-size:9.0pt;font-family:Helvetica;color:#044A91;background:white;mso-ligatures:none">osu.edu</span></b></a><span style="font-size:12.0pt;color:black;mso-ligatures:none"><o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</body>
</html>