<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Hi Ben,</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
No, this configuration is not supported. Firstly, MVAPICH 3.0rc does not officially support GPU buffers. Since we are an MPICH derivative, some support can be enabled via MPICH's GPU support, but we do not support those configurations. For GPU support use MVAPICH-Plus
3.0. This version includes our complete set of GPU optimizations. In either case, building on a CUDA supported system and running on a system without CUDA drivers will cause errors, this is a dependency of the Yaksa datatype engine from MPICH which we are
dependent on.</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Since we do not maintain Yaksa in any way, I cannot provide you much guidance on that front. If you would like to see Yaksa support building on a CUDA capable system and running on a non-CUDA system, please contact the MPICH developers on Github, they may be
able to provide more assistance there. </div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Thanks,</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Nat</div>
<div id="appendonsend"></div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> Mvapich-discuss <mvapich-discuss-bounces+shineman.5=osu.edu@lists.osu.edu> on behalf of Ben Kirk via Mvapich-discuss <mvapich-discuss@lists.osu.edu><br>
<b>Sent:</b> Monday, February 12, 2024 10:33<br>
<b>To:</b> mvapich-discuss@lists.osu.edu <mvapich-discuss@lists.osu.edu><br>
<b>Subject:</b> [Mvapich-discuss] mvapich/3.0rc: Disabling CUDA at runtime?</font>
<div> </div>
</div>
<style>
<!--
#x_pfptBanners6j6pbq
{display:block!important;
visibility:visible!important;
opacity:1!important;
background-color:#CFD3D7!important;
max-width:none!important;
max-height:none!important}
-->
</style>
<div>
<div style="display:none!important; display:none; visibility:hidden; font-size:1px; color:#ffffff; line-height:1px; height:0px; max-height:0px; opacity:0; overflow:hidden">
Hi, we're experimenting with mvapich/3. 0rc on a Cray-EX SS11 machine with a hybrid configuration: 2488 CPU nodes and 82 4-way GPU nodes. We build mvapich/3. 0rc with CUDA support but would like to be able to disable at runtime when on our
</div>
<div style="display:none!important; display:none; visibility:hidden; font-size:1px; color:#ffffff; line-height:1px; height:0px; max-height:0px; opacity:0; overflow:hidden">
ZjQcmQRYFpfptBannerStart</div>
<div dir="ltr" lang="en" id="x_pfptBanners6j6pbq" style="display:block!important; text-align:left!important; margin:16px 0px 16px 0px!important; padding:8px 16px 8px 16px!important; border-radius:4px!important; min-width:200px!important; background-color:#CFD3D7!important; background-color:#CFD3D7; border-top:4px solid #8c8e91!important; border-top:4px solid #8c8e91">
<div id="x_pfptBanners6j6pbq" style="float:left!important; display:block!important; margin:0px 0px 1px 0px!important; max-width:600px!important">
<div id="x_pfptBanners6j6pbq" style="display:block!important; visibility:visible!important; background-color:#CFD3D7!important; color:#000000!important; color:#000000; font-family:'Arial',sans-serif!important; font-family:'Arial',sans-serif; font-weight:bold!important; font-weight:bold; font-size:14px!important; line-height:18px!important; line-height:18px">
This Message Is From an External Sender </div>
<div id="x_pfptBanners6j6pbq" style="display:block!important; visibility:visible!important; background-color:#CFD3D7!important; color:#000000!important; color:#000000; font-weight:normal; font-family:'Arial',sans-serif!important; font-family:'Arial',sans-serif; font-size:12px!important; line-height:18px!important; line-height:18px; margin-top:2px!important">
This message came from outside your organization. </div>
</div>
<div id="x_pfptBanners6j6pbq" style="float:right!important; display:block!important; display:block; margin:0px 0px 0px 16px!important; text-align:right!important; width:fit-content!important">
<a id="x_pfptBanners6j6pbq" href="https://us-phishalarm-ewt.proofpoint.com/EWT/v1/KGKeukY!vYQd06ipq8rthHHbc0LoO8ItnrIKNh6PhY-2PIRsIuNs3kMNrDb-a8CBM1-jMbUVYxCoXGNiIHU62hgaEpiy4309Yy8GxWc--dPWVM2fQ1LsnU0LKdb8YcSGwjVn3687b-7yOQ$" style="display:inline-block!important; text-decoration:none">
<div class="x_pfptPrimaryButtons6j6pbq" style="display:inline-block!important; display:inline-block; visibility:visible!important; opacity:1!important; color:#000000!important; color:#000000; font-family:'Arial',sans-serif!important; font-family:'Arial',sans-serif; font-size:14px!important; font-weight:normal!important; text-decoration:none!important; border-radius:2px!important; padding:7.5px 16px!important; margin:3px 0 3px 16px!important; white-space:nowrap!important; width:fit-content!important; border:1px solid #666666">
Report Suspicious </div>
</a></div>
<div style="clear:both!important; display:block!important; visibility:hidden!important; line-height:0!important; font-size:0.01px!important; height:0px">
</div>
</div>
<div style="display:none!important; display:none; visibility:hidden; font-size:1px; color:#ffffff; line-height:1px; height:0px; max-height:0px; opacity:0; overflow:hidden">
ZjQcmQRYFpfptBannerEnd</div>
<div dir="ltr">Hi, we're experimenting with mvapich/3.0rc on a Cray-EX SS11 machine with a hybrid configuration: 2488 CPU nodes and 82 4-way GPU nodes.<br>
<br>
We build mvapich/3.0rc with CUDA support but would like to be able to disable at runtime when on our CPU-only nodes, is this a supported configuration, and if so how? I checked for various MV2_*/MVP_* environment variables and didn't find success, yet.<br>
<br>
<div>Thanks!!<br clear="all">
<div>--<br>
</div>
<div dir="ltr" class="x_gmail_signature">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div>
<div><font color="#000000">Ben Kirk</font><br>
<span style="color:rgb(0,0,0); font-family:arial,helvetica,sans-serif">NCAR </span><span style="font-family:arial,sans-serif; font-size:12.8px">Computational & Information Systems Laboratory</span></div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<br>
<br>
$ mpiexec -n 2 ./hello_world_mpi.mvapich <br>
pbs_attach: process 115770 attached to job: 3047325.desched1<br>
CUDA Error (yaksuri_cuda_init_hook:src/backend/cuda/hooks/yaksuri_cuda_init_hooks.c,114): no CUDA-capable device is detected<br>
<br>
===================================================================================<br>
= BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES<br>
= PID 116214 RUNNING AT <a href="https://urldefense.com/v3/__http://dec0001.hsn.de.hpc.ucar.edu__;!!KGKeukY!0yHd4LCcCIJNzFvGn9dQQFkTRnyyNahd8hESfxkUa78oZ6FOOYU3lLQhmhN7p231PuB8oV5oTjaKdOp6aW7dSGY7kv0$">
dec0001.hsn.de.hpc.ucar.edu</a><br>
= EXIT CODE: 11<br>
= CLEANING UP REMAINING PROCESSES<br>
= YOU CAN IGNORE THE BELOW CLEANUP MESSAGES<br>
===================================================================================<br>
YOUR APPLICATION TERMINATED WITH THE EXIT STRING: Segmentation fault (signal 11)<br>
This typically refers to a problem with your application.<br>
Please see the FAQ page for debugging suggestions<br>
<br>
<br>
<br>
<div dir="ltr" class="x_gmail_signature" data-smartmail="gmail_signature">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div>
<div style="color:rgb(136,136,136)"><br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</body>
</html>