<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
Hi Ben, <br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
Looks like you are using the OFI sockets provider instead of Cray's CXI provider. We are still working on making sure that provider detection works in such a way that MVAPICH will automatically detect the best performing provider on a given system. For now
though, please use your chosen launcher to set the environment variable <code>MPIR_CVAR_OFI_USE_PROVIDER=cxi</code> for your run. For additional safety, you can also set
<code>FI_PROVIDER=cxi</code> to ensure that libfabrics itself will only support the cxi provider. The former is required, the latter is optional but can provide a nominal improvement in startup time.
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
Please let me know if you have any issues.<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
Thanks,</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
Nat<br>
</div>
<div id="appendonsend"></div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> Mvapich-discuss <mvapich-discuss-bounces@lists.osu.edu> on behalf of Ben Kirk via Mvapich-discuss <mvapich-discuss@lists.osu.edu><br>
<b>Sent:</b> Wednesday, May 3, 2023 15:30<br>
<b>To:</b> mvapich-discuss@lists.osu.edu <mvapich-discuss@lists.osu.edu><br>
<b>Subject:</b> [Mvapich-discuss] mvapich2-3.0a and slingshot 11 - performance questions</font>
<div> </div>
</div>
<style>
<!--
#x_pfptBannerwh0l79k
{display:block!important;
visibility:visible!important;
opacity:1!important;
background-color:#CFD3D7!important;
max-width:none!important;
max-height:none!important}
-->
</style>
<div>
<div style="display:none!important; display:none; visibility:hidden; font-size:1px; color:#ffffff; line-height:1px; height:0px; max-height:0px; opacity:0; overflow:hidden">
Hi, I've recently installed mvapich2-3. 0a with the hopes of testing slingshot 11 support. When ./configuring with the recommended --with-device=ch4: ofi --with-libfabric=/opt/cray/libfabric/1. 15. 2. 0 I am able to build the library, but find
</div>
<div style="display:none!important; display:none; visibility:hidden; font-size:1px; color:#ffffff; line-height:1px; height:0px; max-height:0px; opacity:0; overflow:hidden">
ZjQcmQRYFpfptBannerStart</div>
<div dir="ltr" lang="en" id="x_pfptBannerwh0l79k" style="display:block!important; text-align:left!important; margin:16px 0px 16px 0px!important; padding:8px 16px 8px 16px!important; border-radius:4px!important; min-width:200px!important; background-color:#CFD3D7!important; background-color:#CFD3D7; border-top:4px solid #8c8e91!important; border-top:4px solid #8c8e91">
<div id="x_pfptBannerwh0l79k" style="float:left!important; display:block!important; margin:0px 0px 1px 0px!important; max-width:600px!important">
<div id="x_pfptBannerwh0l79k" style="display:block!important; visibility:visible!important; background-color:#CFD3D7!important; color:#000000!important; color:#000000; font-family:'Arial',sans-serif!important; font-family:'Arial',sans-serif; font-weight:bold!important; font-weight:bold; font-size:14px!important; line-height:18px!important; line-height:18px">
This Message Is From an External Sender </div>
<div id="x_pfptBannerwh0l79k" style="display:block!important; visibility:visible!important; background-color:#CFD3D7!important; color:#000000!important; color:#000000; font-weight:normal; font-family:'Arial',sans-serif!important; font-family:'Arial',sans-serif; font-size:12px!important; line-height:18px!important; line-height:18px; margin-top:2px!important">
This message came from outside your organization. </div>
</div>
<div id="x_pfptBannerwh0l79k" style="float:right!important; display:block!important; display:block; margin:0px 0px 0px 16px!important; text-align:right!important; width:fit-content!important">
<a id="x_pfptBannerwh0l79k" href="https://us-phishalarm-ewt.proofpoint.com/EWT/v1/KGKeukY!vYQd06ipq8rthBdxFCKveDmLblDOoEjery30SgspyufX-nPH7jTI-7DpDF4NjNf7L7ect23Y2JU83EUbKH6-W8PaMzb7J_wyiXUHuzc3gsU5VYN690MA5_DYcoTewBr3u7W_Sg$" style="display:inline-block!important; text-decoration:none">
<div class="x_pfptPrimaryButtonwh0l79k" style="display:inline-block!important; display:inline-block; visibility:visible!important; opacity:1!important; color:#000000!important; color:#000000; font-family:'Arial',sans-serif!important; font-family:'Arial',sans-serif; font-size:14px!important; font-weight:normal!important; text-decoration:none!important; border-radius:2px!important; padding:7.5px 16px!important; margin:3px 0 3px 16px!important; white-space:nowrap!important; width:fit-content!important; border:1px solid #666666">
Report Suspicious </div>
</a></div>
<div style="clear:both!important; display:block!important; visibility:hidden!important; line-height:0!important; font-size:0.01px!important; height:0px">
</div>
</div>
<div style="display:none!important; display:none; visibility:hidden; font-size:1px; color:#ffffff; line-height:1px; height:0px; max-height:0px; opacity:0; overflow:hidden">
ZjQcmQRYFpfptBannerEnd</div>
<div dir="ltr"><span id="x_gmail-docs-internal-guid-1509aefb-7fff-9a8e-db86-6e2f97a24df2">
<p dir="ltr" style="line-height:1.38; margin-top:0pt; margin-bottom:0pt"><span style="font-size:11pt; font-family:Arial; color:rgb(0,0,0); background-color:transparent; font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; vertical-align:baseline; white-space:pre-wrap">Hi,
I've recently installed mvapich2-3.0a with the hopes of testing slingshot 11 support.</span></p>
<br>
<p dir="ltr" style="line-height:1.38; margin-top:0pt; margin-bottom:0pt"><span style="font-size:11pt; font-family:Arial; color:rgb(0,0,0); background-color:transparent; font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; vertical-align:baseline; white-space:pre-wrap">When
./configuring with the recommended </span></p>
<br>
<p dir="ltr" style="line-height:1.38; margin-top:0pt; margin-bottom:0pt"><span style="font-size:11pt; font-family:Arial; color:rgb(0,0,0); background-color:transparent; font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; vertical-align:baseline; white-space:pre-wrap">--with-device=ch4:ofi
--with-libfabric=/opt/cray/libfabric/<a href="https://urldefense.com/v3/__http://1.15.2.0__;!!KGKeukY!yxPNJFqy5IE80I2_l0aHGTnWM9bVDokVT9ludhEL6Qkjh4leZnAVpFkvvSvA_9u5drvGqr4pPhttW1oOoLyqX4Q_Fwk$">1.15.2.0</a></span></p>
<br>
<p dir="ltr" style="line-height:1.38; margin-top:0pt; margin-bottom:0pt"><span style="font-size:11pt; font-family:Arial; color:rgb(0,0,0); background-color:transparent; font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; vertical-align:baseline; white-space:pre-wrap">I
am able to build the library, but find even simple pt2pt performance exceptionally slow. Comparing intra (inside 1 node) to inter (across 2 nodes):</span></p>
<br>
<p dir="ltr" style="line-height:1.38; margin-top:0pt; margin-bottom:0pt"><span style="font-size:11pt; font-family:Arial; color:rgb(0,0,0); background-color:transparent; font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; vertical-align:baseline; white-space:pre-wrap">#*********
Intra-Node-CPU (Bare Metal) *****************</span></p>
<p dir="ltr" style="line-height:1.38; margin-top:0pt; margin-bottom:0pt"><span style="font-size:11pt; font-family:Arial; color:rgb(0,0,0); background-color:transparent; font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; vertical-align:baseline; white-space:pre-wrap">#/glade/work/benkirk/codes/dev_stack/install/osu-micro-benchmarks-6.2-mvapich2-3.0a-cray_libfabric-derecho-gcc-12.2.0/libexec/osu-micro-benchmarks/mpi/pt2pt/osu_bw</span></p>
<p dir="ltr" style="line-height:1.38; margin-top:0pt; margin-bottom:0pt"><span style="font-size:11pt; font-family:Arial; color:rgb(0,0,0); background-color:transparent; font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; vertical-align:baseline; white-space:pre-wrap">#
OSU MPI Bandwidth Test v6.2</span></p>
<p dir="ltr" style="line-height:1.38; margin-top:0pt; margin-bottom:0pt"><span style="font-size:11pt; font-family:Arial; color:rgb(0,0,0); background-color:transparent; font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; vertical-align:baseline; white-space:pre-wrap">#
Size Bandwidth (MB/s)</span></p>
<p dir="ltr" style="line-height:1.38; margin-top:0pt; margin-bottom:0pt"><span style="font-size:11pt; font-family:Arial; color:rgb(0,0,0); background-color:transparent; font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; vertical-align:baseline; white-space:pre-wrap">1
2.90</span></p>
<p dir="ltr" style="line-height:1.38; margin-top:0pt; margin-bottom:0pt"><span style="font-size:11pt; font-family:Arial; color:rgb(0,0,0); background-color:transparent; font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; vertical-align:baseline; white-space:pre-wrap">2
6.10</span></p>
<p dir="ltr" style="line-height:1.38; margin-top:0pt; margin-bottom:0pt"><span style="font-size:11pt; font-family:Arial; color:rgb(0,0,0); background-color:transparent; font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; vertical-align:baseline; white-space:pre-wrap">4
11.44</span></p>
<p dir="ltr" style="line-height:1.38; margin-top:0pt; margin-bottom:0pt"><span style="font-size:11pt; font-family:Arial; color:rgb(0,0,0); background-color:transparent; font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; vertical-align:baseline; white-space:pre-wrap">8
24.28</span></p>
<p dir="ltr" style="line-height:1.38; margin-top:0pt; margin-bottom:0pt"><span style="font-size:11pt; font-family:Arial; color:rgb(0,0,0); background-color:transparent; font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; vertical-align:baseline; white-space:pre-wrap">16
46.73</span></p>
<p dir="ltr" style="line-height:1.38; margin-top:0pt; margin-bottom:0pt"><span style="font-size:11pt; font-family:Arial; color:rgb(0,0,0); background-color:transparent; font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; vertical-align:baseline; white-space:pre-wrap">32
93.20</span></p>
<p dir="ltr" style="line-height:1.38; margin-top:0pt; margin-bottom:0pt"><span style="font-size:11pt; font-family:Arial; color:rgb(0,0,0); background-color:transparent; font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; vertical-align:baseline; white-space:pre-wrap">64
194.77</span></p>
<p dir="ltr" style="line-height:1.38; margin-top:0pt; margin-bottom:0pt"><span style="font-size:11pt; font-family:Arial; color:rgb(0,0,0); background-color:transparent; font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; vertical-align:baseline; white-space:pre-wrap">128
346.13</span></p>
<p dir="ltr" style="line-height:1.38; margin-top:0pt; margin-bottom:0pt"><span style="font-size:11pt; font-family:Arial; color:rgb(0,0,0); background-color:transparent; font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; vertical-align:baseline; white-space:pre-wrap">256
759.24</span></p>
<p dir="ltr" style="line-height:1.38; margin-top:0pt; margin-bottom:0pt"><span style="font-size:11pt; font-family:Arial; color:rgb(0,0,0); background-color:transparent; font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; vertical-align:baseline; white-space:pre-wrap">512
1389.39</span></p>
<p dir="ltr" style="line-height:1.38; margin-top:0pt; margin-bottom:0pt"><span style="font-size:11pt; font-family:Arial; color:rgb(0,0,0); background-color:transparent; font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; vertical-align:baseline; white-space:pre-wrap">1024
2655.62</span></p>
<p dir="ltr" style="line-height:1.38; margin-top:0pt; margin-bottom:0pt"><span style="font-size:11pt; font-family:Arial; color:rgb(0,0,0); background-color:transparent; font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; vertical-align:baseline; white-space:pre-wrap">2048
5258.23</span></p>
<p dir="ltr" style="line-height:1.38; margin-top:0pt; margin-bottom:0pt"><span style="font-size:11pt; font-family:Arial; color:rgb(0,0,0); background-color:transparent; font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; vertical-align:baseline; white-space:pre-wrap">4096
9112.13</span></p>
<p dir="ltr" style="line-height:1.38; margin-top:0pt; margin-bottom:0pt"><span style="font-size:11pt; font-family:Arial; color:rgb(0,0,0); background-color:transparent; font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; vertical-align:baseline; white-space:pre-wrap">8192
12012.65</span></p>
<p dir="ltr" style="line-height:1.38; margin-top:0pt; margin-bottom:0pt"><span style="font-size:11pt; font-family:Arial; color:rgb(0,0,0); background-color:transparent; font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; vertical-align:baseline; white-space:pre-wrap">16384
11235.18</span></p>
<p dir="ltr" style="line-height:1.38; margin-top:0pt; margin-bottom:0pt"><span style="font-size:11pt; font-family:Arial; color:rgb(0,0,0); background-color:transparent; font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; vertical-align:baseline; white-space:pre-wrap">32768
10769.48</span></p>
<p dir="ltr" style="line-height:1.38; margin-top:0pt; margin-bottom:0pt"><span style="font-size:11pt; font-family:Arial; color:rgb(0,0,0); background-color:transparent; font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; vertical-align:baseline; white-space:pre-wrap">65536
19743.95</span></p>
<p dir="ltr" style="line-height:1.38; margin-top:0pt; margin-bottom:0pt"><span style="font-size:11pt; font-family:Arial; color:rgb(0,0,0); background-color:transparent; font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; vertical-align:baseline; white-space:pre-wrap">131072
29157.27</span></p>
<p dir="ltr" style="line-height:1.38; margin-top:0pt; margin-bottom:0pt"><span style="font-size:11pt; font-family:Arial; color:rgb(0,0,0); background-color:transparent; font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; vertical-align:baseline; white-space:pre-wrap">262144
27715.51</span></p>
<p dir="ltr" style="line-height:1.38; margin-top:0pt; margin-bottom:0pt"><span style="font-size:11pt; font-family:Arial; color:rgb(0,0,0); background-color:transparent; font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; vertical-align:baseline; white-space:pre-wrap">524288
27497.12</span></p>
<p dir="ltr" style="line-height:1.38; margin-top:0pt; margin-bottom:0pt"><span style="font-size:11pt; font-family:Arial; color:rgb(0,0,0); background-color:transparent; font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; vertical-align:baseline; white-space:pre-wrap">1048576
13662.50</span></p>
<p dir="ltr" style="line-height:1.38; margin-top:0pt; margin-bottom:0pt"><span style="font-size:11pt; font-family:Arial; color:rgb(0,0,0); background-color:transparent; font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; vertical-align:baseline; white-space:pre-wrap">2097152
10807.83</span></p>
<p dir="ltr" style="line-height:1.38; margin-top:0pt; margin-bottom:0pt"><span style="font-size:11pt; font-family:Arial; color:rgb(0,0,0); background-color:transparent; font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; vertical-align:baseline; white-space:pre-wrap">4194304
10830.14</span></p>
<br>
<p dir="ltr" style="line-height:1.38; margin-top:0pt; margin-bottom:0pt"><span style="font-size:11pt; font-family:Arial; color:rgb(0,0,0); background-color:transparent; font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; vertical-align:baseline; white-space:pre-wrap">#*********
Inter-Node-CPU (Bare Metal) *****************</span></p>
<p dir="ltr" style="line-height:1.38; margin-top:0pt; margin-bottom:0pt"><span style="font-size:11pt; font-family:Arial; color:rgb(0,0,0); background-color:transparent; font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; vertical-align:baseline; white-space:pre-wrap">#/glade/work/benkirk/codes/dev_stack/install/osu-micro-benchmarks-6.2-mvapich2-3.0a-cray_libfabric-derecho-gcc-12.2.0/libexec/osu-micro-benchmarks/mpi/pt2pt/osu_bw</span></p>
<p dir="ltr" style="line-height:1.38; margin-top:0pt; margin-bottom:0pt"><span style="font-size:11pt; font-family:Arial; color:rgb(0,0,0); background-color:transparent; font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; vertical-align:baseline; white-space:pre-wrap">#
OSU MPI Bandwidth Test v6.2</span></p>
<p dir="ltr" style="line-height:1.38; margin-top:0pt; margin-bottom:0pt"><span style="font-size:11pt; font-family:Arial; color:rgb(0,0,0); background-color:transparent; font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; vertical-align:baseline; white-space:pre-wrap">#
Size Bandwidth (MB/s)</span></p>
<p dir="ltr" style="line-height:1.38; margin-top:0pt; margin-bottom:0pt"><span style="font-size:11pt; font-family:Arial; color:rgb(0,0,0); background-color:transparent; font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; vertical-align:baseline; white-space:pre-wrap">1
0.30</span></p>
<p dir="ltr" style="line-height:1.38; margin-top:0pt; margin-bottom:0pt"><span style="font-size:11pt; font-family:Arial; color:rgb(0,0,0); background-color:transparent; font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; vertical-align:baseline; white-space:pre-wrap">2
0.60</span></p>
<p dir="ltr" style="line-height:1.38; margin-top:0pt; margin-bottom:0pt"><span style="font-size:11pt; font-family:Arial; color:rgb(0,0,0); background-color:transparent; font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; vertical-align:baseline; white-space:pre-wrap">4
1.20</span></p>
<p dir="ltr" style="line-height:1.38; margin-top:0pt; margin-bottom:0pt"><span style="font-size:11pt; font-family:Arial; color:rgb(0,0,0); background-color:transparent; font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; vertical-align:baseline; white-space:pre-wrap">8
2.40</span></p>
<p dir="ltr" style="line-height:1.38; margin-top:0pt; margin-bottom:0pt"><span style="font-size:11pt; font-family:Arial; color:rgb(0,0,0); background-color:transparent; font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; vertical-align:baseline; white-space:pre-wrap">16
4.78</span></p>
<p dir="ltr" style="line-height:1.38; margin-top:0pt; margin-bottom:0pt"><span style="font-size:11pt; font-family:Arial; color:rgb(0,0,0); background-color:transparent; font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; vertical-align:baseline; white-space:pre-wrap">32
9.52</span></p>
<p dir="ltr" style="line-height:1.38; margin-top:0pt; margin-bottom:0pt"><span style="font-size:11pt; font-family:Arial; color:rgb(0,0,0); background-color:transparent; font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; vertical-align:baseline; white-space:pre-wrap">64
17.50</span></p>
<p dir="ltr" style="line-height:1.38; margin-top:0pt; margin-bottom:0pt"><span style="font-size:11pt; font-family:Arial; color:rgb(0,0,0); background-color:transparent; font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; vertical-align:baseline; white-space:pre-wrap">128
34.09</span></p>
<p dir="ltr" style="line-height:1.38; margin-top:0pt; margin-bottom:0pt"><span style="font-size:11pt; font-family:Arial; color:rgb(0,0,0); background-color:transparent; font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; vertical-align:baseline; white-space:pre-wrap">256
54.02</span></p>
<p dir="ltr" style="line-height:1.38; margin-top:0pt; margin-bottom:0pt"><span style="font-size:11pt; font-family:Arial; color:rgb(0,0,0); background-color:transparent; font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; vertical-align:baseline; white-space:pre-wrap">512
101.24</span></p>
<p dir="ltr" style="line-height:1.38; margin-top:0pt; margin-bottom:0pt"><span style="font-size:11pt; font-family:Arial; color:rgb(0,0,0); background-color:transparent; font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; vertical-align:baseline; white-space:pre-wrap">1024
196.80</span></p>
<p dir="ltr" style="line-height:1.38; margin-top:0pt; margin-bottom:0pt"><span style="font-size:11pt; font-family:Arial; color:rgb(0,0,0); background-color:transparent; font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; vertical-align:baseline; white-space:pre-wrap">2048
195.56</span></p>
<p dir="ltr" style="line-height:1.38; margin-top:0pt; margin-bottom:0pt"><span style="font-size:11pt; font-family:Arial; color:rgb(0,0,0); background-color:transparent; font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; vertical-align:baseline; white-space:pre-wrap">4096
262.10</span></p>
<p dir="ltr" style="line-height:1.38; margin-top:0pt; margin-bottom:0pt"><span style="font-size:11pt; font-family:Arial; color:rgb(0,0,0); background-color:transparent; font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; vertical-align:baseline; white-space:pre-wrap">8192
316.01</span></p>
<p dir="ltr" style="line-height:1.38; margin-top:0pt; margin-bottom:0pt"><span style="font-size:11pt; font-family:Arial; color:rgb(0,0,0); background-color:transparent; font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; vertical-align:baseline; white-space:pre-wrap">16384
291.34</span></p>
<p dir="ltr" style="line-height:1.38; margin-top:0pt; margin-bottom:0pt"><span style="font-size:11pt; font-family:Arial; color:rgb(0,0,0); background-color:transparent; font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; vertical-align:baseline; white-space:pre-wrap">32768
289.64</span></p>
<p dir="ltr" style="line-height:1.38; margin-top:0pt; margin-bottom:0pt"><span style="font-size:11pt; font-family:Arial; color:rgb(0,0,0); background-color:transparent; font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; vertical-align:baseline; white-space:pre-wrap">65536
290.30</span></p>
<p dir="ltr" style="line-height:1.38; margin-top:0pt; margin-bottom:0pt"><span style="font-size:11pt; font-family:Arial; color:rgb(0,0,0); background-color:transparent; font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; vertical-align:baseline; white-space:pre-wrap">131072
260.94</span></p>
<p dir="ltr" style="line-height:1.38; margin-top:0pt; margin-bottom:0pt"><span style="font-size:11pt; font-family:Arial; color:rgb(0,0,0); background-color:transparent; font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; vertical-align:baseline; white-space:pre-wrap">262144
298.70</span></p>
<p dir="ltr" style="line-height:1.38; margin-top:0pt; margin-bottom:0pt"><span style="font-size:11pt; font-family:Arial; color:rgb(0,0,0); background-color:transparent; font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; vertical-align:baseline; white-space:pre-wrap">524288
342.67</span></p>
<p dir="ltr" style="line-height:1.38; margin-top:0pt; margin-bottom:0pt"><span style="font-size:11pt; font-family:Arial; color:rgb(0,0,0); background-color:transparent; font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; vertical-align:baseline; white-space:pre-wrap">1048576
295.31</span></p>
<p dir="ltr" style="line-height:1.38; margin-top:0pt; margin-bottom:0pt"><span style="font-size:11pt; font-family:Arial; color:rgb(0,0,0); background-color:transparent; font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; vertical-align:baseline; white-space:pre-wrap">2097152
351.07</span></p>
<p dir="ltr" style="line-height:1.38; margin-top:0pt; margin-bottom:0pt"><span style="font-size:11pt; font-family:Arial; color:rgb(0,0,0); background-color:transparent; font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; vertical-align:baseline; white-space:pre-wrap">4194304
334.36</span></p>
<br>
<p dir="ltr" style="line-height:1.38; margin-top:0pt; margin-bottom:0pt"><span style="font-size:11pt; font-family:Arial; color:rgb(0,0,0); background-color:transparent; font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; vertical-align:baseline; white-space:pre-wrap">Are
there any debugging variables or other tricks I should be aware of?</span></p>
<br>
<p dir="ltr" style="line-height:1.38; margin-top:0pt; margin-bottom:0pt"><span style="font-size:11pt; font-family:Arial; color:rgb(0,0,0); background-color:transparent; font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; vertical-align:baseline; white-space:pre-wrap">Thanks!!</span></p>
<p dir="ltr" style="line-height:1.38; margin-top:0pt; margin-bottom:0pt"><span style="font-size:11pt; font-family:Arial; color:rgb(0,0,0); background-color:transparent; font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; vertical-align:baseline; white-space:pre-wrap">--</span></p>
<p dir="ltr" style="line-height:1.38; margin-top:0pt; margin-bottom:0pt"><span style="font-size:11pt; font-family:Arial; color:rgb(0,0,0); background-color:transparent; font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; vertical-align:baseline; white-space:pre-wrap">Ben
Kirk</span></p>
<p dir="ltr" style="line-height:1.38; margin-top:0pt; margin-bottom:0pt"><span style="font-size:11pt; font-family:Arial; color:rgb(0,0,0); background-color:transparent; font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; vertical-align:baseline; white-space:pre-wrap">NCAR
Computational & Information Systems Laboratory</span></p>
<p dir="ltr" style="line-height:1.38; margin-top:0pt; margin-bottom:0pt"><span style="font-size:11pt; font-family:Arial; color:rgb(0,0,0); background-color:transparent; font-variant-numeric:normal; font-variant-east-asian:normal; font-variant-alternates:normal; vertical-align:baseline; white-space:pre-wrap">Consulting
Services Group Head</span></p>
</span><br class="x_gmail-Apple-interchange-newline">
</div>
</div>
</body>
</html>