<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Hey Zhi-Qiang,</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Currently, we recommend manually updating the prefix, exec_prefix, sysconfdir, includedir, and libdir paths in mpicc and mpicxx. </div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
As long as your loaded CUDA module version matches that of the RPM, the CUDA prefix paths should be correct for your system. If that's not the case, let us know and we can generate you a new RPM with the correct CUDA version and prefix.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Thanks,</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
-Quentin</div>
<div id="appendonsend"></div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> Mvapich-discuss <mvapich-discuss-bounces+anthony.301=osu.edu@lists.osu.edu> on behalf of You, Zhi-Qiang via Mvapich-discuss <mvapich-discuss@lists.osu.edu><br>
<b>Sent:</b> Friday, July 16, 2021 1:31 PM<br>
<b>To:</b> mvapich-discuss@lists.osu.edu <mvapich-discuss@lists.osu.edu><br>
<b>Subject:</b> Re: [Mvapich-discuss] Cannot build Horovod with mvapich2-gdr</font>
<div> </div>
</div>
<style>
<!--
@font-face
{font-family:PMingLiU}
@font-face
{font-family:"Cambria Math"}
@font-face
{font-family:Calibri}
@font-face
{}
p.x_MsoNormal, li.x_MsoNormal, div.x_MsoNormal
{margin:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif}
span.x_EmailStyle18
{font-family:"Calibri",sans-serif;
color:windowtext}
.x_MsoChpDefault
{font-size:10.0pt}
@page WordSection1
{margin:1.0in 1.0in 1.0in 1.0in}
div.x_WordSection1
{}
-->
</style>
<div lang="EN-US" link="#0563C1" vlink="#954F72" style="word-wrap:break-word">
<div class="x_WordSection1">
<p class="x_MsoNormal">Hello, any update? </p>
<p class="x_MsoNormal"> </p>
<p class="x_MsoNormal">-ZQ</p>
<p class="x_MsoNormal"> </p>
<div style="border:none; border-top:solid #B5C4DF 1.0pt; padding:3.0pt 0in 0in 0in">
<p class="x_MsoNormal" style="margin-bottom:12.0pt"><b><span style="font-size:12.0pt; color:black">From:
</span></b><span style="font-size:12.0pt; color:black">Mvapich-discuss <mvapich-discuss-bounces@lists.osu.edu> on behalf of You, Zhi-Qiang via Mvapich-discuss <mvapich-discuss@lists.osu.edu><br>
<b>Date: </b>Saturday, July 10, 2021 at 3:13 AM<br>
<b>To: </b>mvapich-discuss@lists.osu.edu <mvapich-discuss@lists.osu.edu><br>
<b>Subject: </b>[Mvapich-discuss] Cannot build Horovod with mvapich2-gdr</span></p>
</div>
<p class="x_MsoNormal">Hello,</p>
<p class="x_MsoNormal"> </p>
<p class="x_MsoNormal">I followed the user guide[1] to build horovod with mvapich2-gdr at OSC and got this CMake error:</p>
<p class="x_MsoNormal"> </p>
<p class="x_MsoNormal">$ module reset</p>
<p class="x_MsoNormal">$ module load mvapich2-gdr/2.3.5 cmake</p>
<p class="x_MsoNormal">$ HOROVOD_GPU_OPERATIONS=MPI HOROVOD_CUDA_HOME=$CUDA_HOME HOROVOD_WITH_MPI=1 pip install --no-cache-dir --ignore-installed horovod</p>
<p class="x_MsoNormal">[ .. skipped .. ]</p>
<p class="x_MsoNormal">-- The CXX compiler identification is GNU 8.4.0</p>
<p class="x_MsoNormal"> -- Check for working CXX compiler: /apps/gnu/8.4.0/bin/c++</p>
<p class="x_MsoNormal"> -- Check for working CXX compiler: /apps/gnu/8.4.0/bin/c++ - works</p>
<p class="x_MsoNormal"> -- Detecting CXX compiler ABI info</p>
<p class="x_MsoNormal"> -- Detecting CXX compiler ABI info - done</p>
<p class="x_MsoNormal"> -- Detecting CXX compile features</p>
<p class="x_MsoNormal"> -- Detecting CXX compile features - done</p>
<p class="x_MsoNormal"> -- Build architecture flags: -mf16c -mavx -mfma</p>
<p class="x_MsoNormal"> -- Using command /fs/ess/scratch/PZS0710/zyou/tmp/horovod/bin/python3</p>
<p class="x_MsoNormal"> CMake Error in /tmp/pip-install-xt_qt24z/horovod_6386bb2d1fc94c1f9518143ed589a550/build/temp.linux-x86_64-3.6/RelWithDebInfo/CMakeFiles/CMakeTmp/CMakeLists.txt:</p>
<p class="x_MsoNormal"> Imported target "MPI::MPI_CXX" includes non-existent path</p>
<p class="x_MsoNormal"> </p>
<p class="x_MsoNormal"> "/usr/local/cuda-11.0/include"</p>
<p class="x_MsoNormal"> </p>
<p class="x_MsoNormal"> in its INTERFACE_INCLUDE_DIRECTORIES. Possible reasons include:</p>
<p class="x_MsoNormal">[ .. skipped .. ]</p>
<p class="x_MsoNormal"> </p>
<p class="x_MsoNormal">I noticed "/usr/local/cuda-11.0/include” from the flags in mpi wrappers:</p>
<p class="x_MsoNormal"> </p>
<p class="x_MsoNormal">$ grep final.*flags= `which mpicxx`</p>
<p class="x_MsoNormal">final_cxxflags=" -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic "</p>
<p class="x_MsoNormal">final_cppflags=" -I/usr/local/cuda-11.0/include -I/usr/local/cuda-11.0/include"</p>
<p class="x_MsoNormal">final_ldflags=" -L/usr/local/lib -lcuda -L/usr/local/cuda-11.0/lib64/stubs -L/usr/local/cuda-11.0/lib64 -lcudart -lrt -lstdc++ -Wl,-rpath,/usr/local/cuda-11.0/lib64 -Wl,-rpath,XORIGIN/placeholder -L/usr/local/software/slurm/current/lib64/
-fPIC -m64 "</p>
<p class="x_MsoNormal"> final_ldflags="${final_ldflags} -L/usr/local/cuda-11.0/lib64/ -L/usr/local/lib -lcuda -L/usr/local/cuda-11.0/lib64/stubs -L/usr/local/cuda-11.0/lib64 -lcudart -lrt -lstdc++ -Wl,-rpath,/usr/local/cuda-11.0/lib64 -Wl,-rpath,XORIGIN/placeholder
-L/usr/local/software/slurm/current/lib64/ -fPIC -m64 -L/usr/local/software/slurm/current/lib"</p>
<p class="x_MsoNormal"> </p>
<p class="x_MsoNormal">At OSC, we use rpm2cpio method to install mvapich2-gdr. I know using rpm with –prefix can fix part of the flags but it seems no other method to update cuda path automatically. Besides manual modifications to the flags in the wrappers,
is there other way to let me proceed to build horovod?</p>
<p class="x_MsoNormal"> </p>
<p class="x_MsoNormal">-ZQ</p>
<p class="x_MsoNormal"> </p>
</div>
</div>
</body>
</html>