<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Aptos;
panose-1:2 11 0 4 2 2 2 2 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:10.0pt;
font-family:"Aptos",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;
mso-ligatures:none;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style>
</head>
<body lang="EN-US" link="blue" vlink="purple" style="word-wrap:break-word">
<div class="WordSection1">
<div id="mail-editor-reference-message-container">
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt">Hi Gabriel,<br>
<br>
Can you please try running MPI4Spark with MVAPICH2 2.3.7? Please let us know if this version is working for you or not. Thanks.<br>
<br>
Regards,<br>
Kinan</span><span style="color:#1D1D1D;background:white"><br>
<br>
</span><o:p></o:p></p>
<div id="mail-editor-reference-message-container">
<div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal" style="margin-bottom:12.0pt"><b><span style="font-size:12.0pt;color:black">From:
</span></b><span style="font-size:12.0pt;color:black">Mvapich-discuss <mvapich-discuss-bounces+shafi.16=osu.edu@lists.osu.edu> on behalf of GABRIEL SOTODOSOS MORALES via Mvapich-discuss <mvapich-discuss@lists.osu.edu><br>
<b>Date: </b>Thursday, June 13, 2024 at 3:54</span><span style="font-size:12.0pt;font-family:"Arial",sans-serif;color:black"> </span><span style="font-size:12.0pt;color:black">AM<br>
<b>To: </b>Paniraja Guptha, Akshay <panirajaguptha.1@osu.edu><br>
<b>Cc: </b>Announcement about MVAPICH (MPI over InfiniBand, RoCE, Omni-Path, Slingshot, iWARP and EFA) Libraries developed at NBCL/OSU <mvapich-discuss@lists.osu.edu><br>
<b>Subject: </b>Re: [Mvapich-discuss] Problems trying to run SparkPi example with MPI4Spark</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:1.0pt;color:white">Hi Akshay, Thank you so much for your help. I am testing new things with your library. If I can help you in any way, please let me know. I tried to start the spark cluster with the traditional script
(.</span><span style="font-size:1.0pt;font-family:"Arial",sans-serif;color:white"> </span><span style="font-size:1.0pt;color:white">/sbin/start-all.</span><span style="font-size:1.0pt;font-family:"Arial",sans-serif;color:white"> </span><span style="font-size:1.0pt;color:white">sh)
with the same result, </span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:1.0pt;color:white">ZjQcmQRYFpfptBannerStart</span><o:p></o:p></p>
</div>
<div style="border:none;border-top:solid #8C8E91 3.0pt;padding:0in 0in 0in 0in;display:block!important;text-align:left!important;margin:0px!important;padding:16px!important;border-radius:4px!important;min-width:200px!important;background-color:#CFD3D7!important;border-top:#8c8e91!important" id="pfptBannerjdw98db">
<div id="pfptBannerjdw98db">
<div id="pfptBannerjdw98db">
<p class="MsoNormal" style="line-height:13.5pt;background:#CFD3D7"><b><span lang="EN" style="font-size:12.0pt;font-family:"Arial",sans-serif;color:black">This Message Is From an External Sender
</span></b><o:p></o:p></p>
</div>
<div id="pfptBannerjdw98db">
<p class="MsoNormal" style="line-height:13.5pt;background:#CFD3D7"><span lang="EN" style="font-size:12.0pt;font-family:"Arial",sans-serif;color:black">This message came from outside your organization.
</span><o:p></o:p></p>
</div>
</div>
<div id="pfptBannerjdw98db">
<div style="border:solid #666666 1.0pt;padding:0in 0in 0in 0in;display:inline-block!important;display:inline-block;visibility:visible!important;opacity:1!important;color:#000000!important;font-size:14px!important;font-weight:normal!important;text-decoration:none!important;border-radius:2px!important;padding:16px!important;margin:16px!important;white-space:nowrap!important;width:fit-content!important">
<p class="MsoNormal" style="background:#CFD3D7"><span style="color:black"><a href="https://us-phishalarm-ewt.proofpoint.com/EWT/v1/KGKeukY!vYQd06pBw4oBSdX73OJkWxk97QaxYQXWciJPQDXl_d_Uhs_cNMn5Jltjwq8NHayNpiqmthJJRbFYmw1sc2wfSimD7NcOs6WTkkJgjFna39sQvLXNYU5ViK5Y4fC2JuinJl5Mhw$"><span lang="EN" style="font-size:12.0pt;font-family:"Arial",sans-serif;color:black;text-decoration:none">Report Suspicious
</span></a></span><o:p></o:p></p>
</div>
<p class="MsoNormal" style="background:#CFD3D7"><span lang="EN" style="font-size:12.0pt;color:black"> </span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="background:#CFD3D7"><span lang="EN" style="font-size:12.0pt;color:black"> </span><o:p></o:p></p>
</div>
</div>
<div>
<p class="MsoNormal"><span style="font-size:1.0pt;color:white">ZjQcmQRYFpfptBannerEnd</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt">Hi Akshay,</span><o:p></o:p></p>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt"> </span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt">Thank you so much for your help. I am testing new things with your library. If I can help you in any way, please let me know.</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt"> </span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt">I tried to start the spark cluster with the traditional script (./sbin/start-all.sh) with the same result, for any reason no workers seem to be available.</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt"> </span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt">On the other hand, I have downloaded the tarball you have available on the website, would you have a public repository where the source code is?</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt"> </span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt">Thanks again for your help. Best regards.</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt">Gabriel.</span><o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"><span style="font-size:12.0pt"> </span><o:p></o:p></p>
<div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt">El mié, 12 jun 2024 a las 18:42, Paniraja Guptha, Akshay (<</span><a href="mailto:panirajaguptha.1@osu.edu"><span style="font-size:12.0pt">panirajaguptha.1@osu.edu</span></a><span style="font-size:12.0pt">>)
escribió:</span><o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt">
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt">Hi Gabriel,</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Thanks for contacting us. </span>
<o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">We are taking a look at this. We will get back to you once we have an update.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">-Akshay</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> Mvapich-discuss <mvapich-discuss-bounces+panirajaguptha.1=</span><a href="mailto:osu.edu@lists.osu.edu" target="_blank"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">osu.edu@lists.osu.edu</span></a><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">>
<b>On Behalf Of </b>GABRIEL SOTODOSOS MORALES via Mvapich-discuss<br>
<b>Sent:</b> Tuesday, June 11, 2024 6:57 AM<br>
<b>To:</b> </span><a href="mailto:mvapich-discuss@lists.osu.edu" target="_blank"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">mvapich-discuss@lists.osu.edu</span></a><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><br>
<b>Subject:</b> [Mvapich-discuss] Problems trying to run SparkPi example with MPI4Spark</span><o:p></o:p></p>
</div>
<p class="MsoNormal"><span style="font-size:12.0pt"> </span><o:p></o:p></p>
<div>
<p class="MsoNormal"><span style="font-size:1.0pt;color:white">Hi Mvapich-discuss, I´m trying to run the SparkPi example in my cluster using the Standalone Cluster Manager. However, my executor gets stuck when deploying the tasks to the executors with the following
message: "WARN TaskSchedulerImpl:</span><span style="font-size:1.0pt;font-family:"Arial",sans-serif;color:white"> </span><span style="font-size:1.0pt;color:white">
</span><o:p></o:p></p>
</div>
<p class="MsoNormal"><span style="font-size:12.0pt">Hi Mvapich-discuss,</span><o:p></o:p></p>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt"> </span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt">I´m trying to run the SparkPi example in my cluster using the Standalone Cluster Manager. However, my executor gets stuck when deploying the tasks to the executors with the following message:</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt"> </span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><i><span style="font-size:12.0pt">"WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources"</span></i><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt"> </span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt">I have followed the steps in the user guide, I don´t know if I did something wrong or if I missed something. With the same configuration in Spark, I can run the SparkPi example without problems.</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt"> </span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt">I am using MVAPICH-3.0 compiled as follows: </span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt">--prefix=/beegfs/home/javier.garciablas/gsotodos/bin_noref/mvapich/ --enable-threads=multiple --enable-romio --with-device=ch4:ofi:psm2 --with-libfabric=/opt/libfabric</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt"> </span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt">And here are my configuration files:</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt">spark-env.sh:</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt">export SPARK_HOME=$HOME/mpi4spark-0.2-x86-bin</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt">export SPARK_NO_DAEMONIZE=1</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt">export JAVA_LIBRARY_PATH=$JAVA_LIBRARY_PATH:$MV2J_HOME</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt">export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$MV2J_HOME/lib</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt">export SPARK_LIBRARY_PATH=$MV2J_HOME/lib</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt">export JAVA_BINARY=$JAVA_HOME/bin</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt">export WORK_DIR=$SPARK_HOME/exec-wdir</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt"> </span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt">spark-defaults.conf:</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt">spark.executor.extraJavaOptions -Djava.library.path=$HOME/mvapich2-j-2.3.7/lib</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt"> </span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt">app.sh:</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt">./bin/spark-submit --master spark://$1:7077 --class org.apache.spark.examples.SparkPi examples/jars/spark-examples_2.12-3.3.0-SNAPSHOT.jar 1024</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt"> </span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt">sbin/start-mpi4spark.sh:</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt"> </span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt">HOSTFILE=hostfile<br>
procs=`wc -l < ${HOSTFILE}`<br>
javac -cp $MV2J_HOME/lib/mvapich2-j.jar SparkMPI.java<br>
host=`tail -2 ${HOSTFILE} | head -1`<br>
<br>
{<br>
$MPILIB/bin/mpirun_rsh -export-all -np $procs -hostfile ${HOSTFILE} SLURM_JOB_ID=$SLURM_JOB_ID MV2_RNDV_PROTOCOL=RGET MV2_USE_RDMA_FAST_PATH=0 MV2_USE_COALESCE=0 MV2_SUPPORT_DPM=1 MV2_HOMOGENEOUS_CLUSTER=1 MV2_ENABLE_AFFINITY=0 LD_PRELOAD= $MPILIB/lib/libmpi.so
java -cp $MV2J_HOME/lib/mvapich2-j.jar:. -Djava.library.path=$MV2J_HOME/lib SparkMPI $host<br>
} >& exec.log</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt"> </span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt">After launching sbin/start-mpi4spark.sh the master and workers nodes keep alive but the execution gets stuck as said before. Am I missing something? Thanks for the help in advance.</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt"> </span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt">Best regads.</span><o:p></o:p></p>
</div>
<p class="MsoNormal"><span style="font-size:12.0pt">Gabriel.</span><o:p></o:p></p>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</body>
</html>