<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"MS Gothic";
panose-1:2 11 6 9 7 2 5 8 2 4;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:"PingFang TC";
panose-1:2 11 4 0 0 0 0 0 0 0;}
@font-face
{font-family:"\@MS Gothic";
panose-1:2 11 6 9 7 2 5 8 2 4;}
@font-face
{font-family:"\@PingFang TC";}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
span.EmailStyle19
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;
mso-ligatures:none;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style>
</head>
<body lang="EN-US" link="blue" vlink="purple" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal">Hi,<br>
<br>
Can you please hardcode the path in ` spark-defaults.conf`? The ‘$MV2J_HOME’ does not get expanded here, you would have to write the path in full.
<br>
<br>
One example of defining variables in `conf/spark-defaults.conf` is by doing the following,<br>
<br>
spark.mv2j.path /home/alattar.2/test/mvapich2-j <br>
spark.executor.extraJavaOptions -Djava.library.path=${spark.mv2j.path}/lib<br>
<br>
Please let me know if you are still having issues. Thanks.<br>
<br>
Regards,<br>
Kinan AlAttar<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal" style="margin-bottom:12.0pt"><b><span style="font-size:12.0pt;color:black">From:
</span></b><span style="font-size:12.0pt;font-family:"MS Gothic";color:black">余博</span><span style="font-size:12.0pt;font-family:"PingFang TC",sans-serif;color:black">识</span><span style="font-size:12.0pt;color:black">(</span><span style="font-size:12.0pt;font-family:"MS Gothic";color:black">修羽</span><span style="font-size:12.0pt;color:black">)
<yuboshi.ybs@alibaba-inc.com><br>
<b>Date: </b>Wednesday, May 22, 2024 at 5:06 AM<br>
<b>To: </b>mvapich-discuss <mvapich-discuss@lists.osu.edu><br>
<b>Subject: </b>Got “Initial job has not accepted any resources” when run MPI4Spark<o:p></o:p></span></p>
</div>
<div id="mail-editor-reference-message-container">
<div>
<p class="MsoNormal" style="mso-line-height-alt:.75pt"><span style="font-size:1.0pt;color:white">Hello, I am very interested in the MPI4Spark framework and try to run it in a cluster with 4 nodes equipped with RoCE network. I ran mvapich2 and mvapich2-j examples
successfully. Unfortunately</span><span style="font-size:1.0pt;font-family:"MS Gothic";color:white">，</span><span style="font-size:1.0pt;color:white">I got the “Initial job has not accepted any resources;
<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-line-height-alt:.75pt"><span style="font-size:1.0pt;color:white">ZjQcmQRYFpfptBannerStart<o:p></o:p></span></p>
</div>
<table class="MsoNormalTable" border="0" cellspacing="0" cellpadding="0" width="100%" style="width:100.0%;border-radius:4px">
<tbody>
<tr>
<td style="padding:12.0pt 0in 12.0pt 0in">
<table class="MsoNormalTable" border="1" cellspacing="0" cellpadding="0" width="100%" style="width:100.0%;background:#CFD3D7;border:none;border-top:solid #8C8E91 3.0pt">
<tbody>
<tr>
<td valign="top" style="border:none;padding:0in 7.5pt 3.75pt 4.5pt">
<table class="MsoNormalTable" border="0" cellspacing="0" cellpadding="0" align="left">
<tbody>
<tr>
<td style="padding:3.0pt 6.0pt 3.0pt 6.0pt">
<p class="MsoNormal"><b><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:black">This Message Is From an External Sender
<o:p></o:p></span></b></p>
</td>
</tr>
<tr>
<td style="padding:3.0pt 6.0pt 3.0pt 6.0pt">
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Arial",sans-serif;color:black">This message came from outside your organization.
<o:p></o:p></span></p>
</td>
</tr>
</tbody>
</table>
<table class="MsoNormalTable" border="0" cellspacing="0" cellpadding="0" align="right">
<tbody>
<tr>
<td style="padding:3.0pt 0in 3.0pt 0in">
<p class="MsoNormal"> <a href="https://us-phishalarm-ewt.proofpoint.com/EWT/v1/KGKeukY!vUQZ18dLiYzAINqbX65klHaEBlfJ-dRGfxPW2556GIhccTl_sOnZjyxSfFe9L6MjYqxA7gfbIO4qEnTm5NX8n4SNOql2DZPXaKztbYWGvEjRQKDg-OFniG31PtKrasyuVTo$" target="_blank"><strong><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:black;border:solid #666666 1.0pt;padding:6.0pt;font-weight:normal;text-decoration:none"> Report Suspicious </span></strong></a>
<o:p></o:p></p>
</td>
</tr>
</tbody>
</table>
</td>
</tr>
</tbody>
</table>
</td>
</tr>
</tbody>
</table>
<div>
<p class="MsoNormal" style="mso-line-height-alt:.75pt"><span style="font-size:1.0pt;color:white">ZjQcmQRYFpfptBannerEnd<o:p></o:p></span></p>
</div>
<div>
<div>
<p class="MsoNormal">Hello, I am very interested in the MPI4Spark framework and try to run it in a cluster with 4 nodes equipped with RoCE network. I ran mvapich2 and mvapich2-j examples successfully. Unfortunately<span style="font-family:"MS Gothic"">，</span>I
got the “Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources” warnings when I run MPI4Spark in both standalone and yarn mode. I read the worker.log and found that both workers
got the job request but they seemed not to launch the executor. Is this problem caused by some misconfigurations? The detailed configuraions and logs are listed below. I will appreciate if someone could help me to figure it out. :)<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Hostfile<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">mpi4spark000<o:p></o:p></p>
<div>
<p class="MsoNormal">mpi4spark001<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">mpi4spark002<o:p></o:p></p>
</div>
<p class="MsoNormal">mpi4spark003<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">spark-env.sh<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">export SPARK_HOME=/root/mpi4spark/mpi4spark-0.2-x86-bin<o:p></o:p></p>
<div>
<p class="MsoNormal">export SPARK_NO_DAEMONIZE=1<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">export SPARK_USE_MPI=1<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">export JAVA_LIBRARY_PATH=$JAVA_LIBRARY_PATH:$MV2J_HOME<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$MV2J_HOME/lib<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">export SPARK_LIBRARY_PATH=$MV2J_HOME/lib<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">export JAVA_BINARY=$JAVA_HOME/bin<o:p></o:p></p>
</div>
<p class="MsoNormal">export WORK_DIR=$SPARK_HOME/exec-wdir<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">spark-defaults.conf<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">spark.executor.extraJavaOptions -Djava.library.path=$MV2J_HOME/lib<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">app.sh<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">./bin/spark-submit --master spark://$1:7077 --class org.apache.spark.examples.SparkPi examples/jars/spark-examples_2.12-3.3.0-SNAPSHOT.jar<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Master.log<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">starting org.apache.spark.deploy.master.Master, logging to /root/mpi4spark/mpi4spark-0.2-x86-bin/logs/spark-root-org.apache.spark.deploy.master.Master-1-mpi4spark002.out<o:p></o:p></p>
<div>
<p class="MsoNormal">Spark Command: /root/mpi4spark/jdk1.8.0_321//bin/java -cp /root/mpi4spark/mpi4spark-0.2-x86-bin//conf/:/root/mpi4spark/mpi4spark-0.2-x86-bin/jars/*:/root/mpi4spark/hadoop-3.3.4//etc/hadoop/ -Xmx1g org.apache.spark.deploy.master.Master --host
mpi4spark002 --port 7077 --webui-port 8080<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">========================================<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:50,724 INFO master.Master: Started daemon with process name: 2621571@mpi4spark002<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:50,728 INFO util.SignalUtils: Registering signal handler for TERM<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:50,728 INFO util.SignalUtils: Registering signal handler for HUP<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:50,728 INFO util.SignalUtils: Registering signal handler for INT<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:50,932 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:50,968 INFO spark.SecurityManager: Changing view acls to: root<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:50,968 INFO spark.SecurityManager: Changing modify acls to: root<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:50,968 INFO spark.SecurityManager: Changing view acls groups to:<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:50,969 INFO spark.SecurityManager: Changing modify acls groups to:<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:50,969 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root); groups with view permissions: Set(); users with modify permissions: Set(root); groups
with modify permissions: Set()<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:53,395 INFO nio.NioEventLoop: Starting MPI4Spark.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:53,495 INFO util.Utils: Successfully started service 'sparkMaster' on port 7077.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:53,511 INFO master.Master: Starting Spark master at spark://mpi4spark002:7077<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:53,514 INFO master.Master: Running Spark version 3.3.0-SNAPSHOT<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:53,534 INFO util.log: Logging initialized @3209ms to org.sparkproject.jetty.util.log.Slf4jLog<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:53,567 INFO server.Server: jetty-9.4.43.v20210629; built: 2021-06-30T11:07:22.254Z; git: 526006ecfa3af7f1a27ef3a288e2bef7ea9dd7e8; jvm 1.8.0_321-b07<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:53,579 INFO server.Server: Started @3255ms<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:53,605 INFO server.AbstractConnector: Started ServerConnector@2efb59fc{HTTP/1.1, (http/1.1)}{0.0.0.0:8080}<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:53,605 INFO util.Utils: Successfully started service 'MasterUI' on port 8080.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:53,606 INFO ui.MasterWebUI: Bound MasterWebUI to 0.0.0.0, and started at
<a href="https://urldefense.com/v3/__http:/mpi4spark002:-1__;!!KGKeukY!1yam9MqJGXUuv-UdrWeakBeZ8y-emE258qcPhFWor-ytPm-G1OuIFGfhJjah0Jd4dl55B1r8DghhPvPOMekN7tMAdIspKkkgc5eq$">
http://mpi4spark002:-1</a><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:53,620 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@1332e5ce{/app,null,AVAILABLE,@Spark}<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:53,621 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@1f910681{/app/json,null,AVAILABLE,@Spark}<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:53,622 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@2a186645{/,null,AVAILABLE,@Spark}<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:53,623 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@772642ad{/json,null,AVAILABLE,@Spark}<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:53,628 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@751cafb2{/static,null,AVAILABLE,@Spark}<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:53,628 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@1b2adb3b{/app/kill,null,AVAILABLE,@Spark}<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:53,629 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@1d5107f3{/driver/kill,null,AVAILABLE,@Spark}<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:53,630 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@7e39159e{/workers/kill,null,AVAILABLE,@Spark}<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:53,715 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@64ff458a{/metrics/master/json,null,AVAILABLE,@Spark}<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:53,716 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@5b0b91df{/metrics/applications/json,null,AVAILABLE,@Spark}<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:53,725 INFO master.Master: I have been elected leader! New state: ALIVE<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:53,826 INFO master.Master: Registering worker 200.2.149.6:46337 with 64 cores, 502.3 GiB RAM<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:53,831 INFO master.Master: Registering worker 200.2.149.2:40247 with 64 cores, 502.3 GiB RAM<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:53,906 INFO master.Master: Registering app Spark Pi<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:53,908 INFO master.Master: Registered app Spark Pi with ID app-20240522163753-0000<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:53,923 INFO master.Master: Launching executor app-20240522163753-0000/0 on worker worker-20240522163753-200.2.149.2-40247<o:p></o:p></p>
</div>
<p class="MsoNormal">2024-05-22 16:37:53,927 INFO master.Master: Launching executor app-20240522163753-0000/1 on worker worker-20240522163753-200.2.149.6-46337<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Worker0.log<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">starting org.apache.spark.deploy.worker.Worker, logging to /root/mpi4spark/mpi4spark-0.2-x86-bin/logs/spark-root-org.apache.spark.deploy.worker.Worker-1-mpi4spark000.out<o:p></o:p></p>
<div>
<p class="MsoNormal">Spark Command: /root/mpi4spark/jdk1.8.0_321//bin/java -cp /root/mpi4spark/mpi4spark-0.2-x86-bin//conf/:/root/mpi4spark/mpi4spark-0.2-x86-bin/jars/*:/root/mpi4spark/hadoop-3.3.4//etc/hadoop/ -Xmx1g org.apache.spark.deploy.worker.Worker --webui-port
8081 spark://mpi4spark002:7077<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">========================================<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:51,718 INFO worker.Worker: Started daemon with process name: 2807211@mpi4spark000<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:51,721 INFO util.SignalUtils: Registering signal handler for TERM<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:51,722 INFO util.SignalUtils: Registering signal handler for HUP<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:51,722 INFO util.SignalUtils: Registering signal handler for INT<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:51,931 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:51,966 INFO spark.SecurityManager: Changing view acls to: root<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:51,966 INFO spark.SecurityManager: Changing modify acls to: root<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:51,966 INFO spark.SecurityManager: Changing view acls groups to:<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:51,967 INFO spark.SecurityManager: Changing modify acls groups to:<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:51,967 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root); groups with view permissions: Set(); users with modify permissions: Set(root); groups
with modify permissions: Set()<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">[mpi4spark000:mpi_rank_0][MPID_Init] [Performance Suggestion]: Application has requested for multi-thread capability. If allocating memory from different pthreads/OpenMP threads, please consider setting MV2_USE_ALIGNED_ALLOC=1 for improved
performance.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Use MV2_USE_THREAD_WARNING=0 to suppress this error message.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:53,396 INFO nio.NioEventLoop: Starting MPI4Spark.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:53,496 INFO util.Utils: Successfully started service 'sparkWorker' on port 40247.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:53,497 INFO worker.Worker: Worker decommissioning not enabled.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:53,601 INFO worker.Worker: Starting Spark worker 200.2.149.2:40247 with 64 cores, 502.3 GiB RAM<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:53,604 INFO worker.Worker: Running Spark version 3.3.0-SNAPSHOT<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:53,605 INFO worker.Worker: Spark home: /root/mpi4spark/mpi4spark-0.2-x86-bin<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:53,616 INFO resource.ResourceUtils: ==============================================================<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:53,616 INFO resource.ResourceUtils: No custom resources configured for spark.worker.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:53,616 INFO resource.ResourceUtils: ==============================================================<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:53,635 INFO util.log: Logging initialized @2311ms to org.sparkproject.jetty.util.log.Slf4jLog<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:53,667 INFO server.Server: jetty-9.4.43.v20210629; built: 2021-06-30T11:07:22.254Z; git: 526006ecfa3af7f1a27ef3a288e2bef7ea9dd7e8; jvm 1.8.0_321-b07<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:53,678 INFO server.Server: Started @2355ms<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:53,708 INFO server.AbstractConnector: Started ServerConnector@41cf10a2{HTTP/1.1, (http/1.1)}{0.0.0.0:8081}<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:53,708 INFO util.Utils: Successfully started service 'WorkerUI' on port 8081.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:53,710 INFO ui.WorkerWebUI: Bound WorkerWebUI to 0.0.0.0, and started at
<a href="https://urldefense.com/v3/__http:/mpi4spark000:-1__;!!KGKeukY!1yam9MqJGXUuv-UdrWeakBeZ8y-emE258qcPhFWor-ytPm-G1OuIFGfhJjah0Jd4dl55B1r8DghhPvPOMekN7tMAdIspKj830Qvr$">
http://mpi4spark000:-1</a><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:53,722 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@8e107a9{/logPage,null,AVAILABLE,@Spark}<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:53,724 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@4375cbe0{/logPage/json,null,AVAILABLE,@Spark}<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:53,724 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@5e18e1c2{/,null,AVAILABLE,@Spark}<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:53,726 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@6ef7379{/json,null,AVAILABLE,@Spark}<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:53,731 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@548ada47{/static,null,AVAILABLE,@Spark}<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:53,731 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@4a02c874{/log,null,AVAILABLE,@Spark}<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:53,732 INFO worker.Worker: Connecting to master mpi4spark002:7077...<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:53,742 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@73e20207{/metrics/json,null,AVAILABLE,@Spark}<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:53,775 INFO client.TransportClientFactory: Successfully created connection to mpi4spark002/200.2.151.2:7077 after 31 ms (0 ms spent in bootstraps)<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:53,837 INFO worker.Worker: Successfully registered with master spark://mpi4spark002:7077<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:53,944 INFO worker.Worker: Asked to launch executor app-20240522163753-0000/0 for Spark Pi<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:53,959 INFO spark.SecurityManager: Changing view acls to: root<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:53,959 INFO spark.SecurityManager: Changing modify acls to: root<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:53,959 INFO spark.SecurityManager: Changing view acls groups to:<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:53,960 INFO spark.SecurityManager: Changing modify acls groups to:<o:p></o:p></p>
</div>
<p class="MsoNormal">2024-05-22 16:37:53,960 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root); groups with view permissions: Set(); users with modify permissions: Set(root); groups
with modify permissions: Set()<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Worker1.log<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">starting org.apache.spark.deploy.worker.Worker, logging to /root/mpi4spark/mpi4spark-0.2-x86-bin/logs/spark-root-org.apache.spark.deploy.worker.Worker-1-mpi4spark001.out<o:p></o:p></p>
<div>
<p class="MsoNormal">Spark Command: /root/mpi4spark/jdk1.8.0_321//bin/java -cp /root/mpi4spark/mpi4spark-0.2-x86-bin//conf/:/root/mpi4spark/mpi4spark-0.2-x86-bin/jars/*:/root/mpi4spark/hadoop-3.3.4//etc/hadoop/ -Xmx1g org.apache.spark.deploy.worker.Worker --webui-port
8081 spark://mpi4spark002:7077<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">========================================<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:51,720 INFO worker.Worker: Started daemon with process name: 2883486@mpi4spark001<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:51,723 INFO util.SignalUtils: Registering signal handler for TERM<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:51,724 INFO util.SignalUtils: Registering signal handler for HUP<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:51,724 INFO util.SignalUtils: Registering signal handler for INT<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:51,936 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:51,971 INFO spark.SecurityManager: Changing view acls to: root<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:51,971 INFO spark.SecurityManager: Changing modify acls to: root<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:51,972 INFO spark.SecurityManager: Changing view acls groups to:<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:51,972 INFO spark.SecurityManager: Changing modify acls groups to:<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:51,972 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root); groups with view permissions: Set(); users with modify permissions: Set(root); groups
with modify permissions: Set()<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:53,397 INFO nio.NioEventLoop: Starting MPI4Spark.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:53,506 INFO util.Utils: Successfully started service 'sparkWorker' on port 46337.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:53,507 INFO worker.Worker: Worker decommissioning not enabled.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:53,616 INFO worker.Worker: Starting Spark worker 200.2.149.6:46337 with 64 cores, 502.3 GiB RAM<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:53,619 INFO worker.Worker: Running Spark version 3.3.0-SNAPSHOT<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:53,620 INFO worker.Worker: Spark home: /root/mpi4spark/mpi4spark-0.2-x86-bin<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:53,632 INFO resource.ResourceUtils: ==============================================================<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:53,632 INFO resource.ResourceUtils: No custom resources configured for spark.worker.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:53,632 INFO resource.ResourceUtils: ==============================================================<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:53,651 INFO util.log: Logging initialized @2328ms to org.sparkproject.jetty.util.log.Slf4jLog<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:53,685 INFO server.Server: jetty-9.4.43.v20210629; built: 2021-06-30T11:07:22.254Z; git: 526006ecfa3af7f1a27ef3a288e2bef7ea9dd7e8; jvm 1.8.0_321-b07<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:53,696 INFO server.Server: Started @2373ms<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:53,723 INFO server.AbstractConnector: Started ServerConnector@679db676{HTTP/1.1, (http/1.1)}{0.0.0.0:8081}<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:53,723 INFO util.Utils: Successfully started service 'WorkerUI' on port 8081.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:53,724 INFO ui.WorkerWebUI: Bound WorkerWebUI to 0.0.0.0, and started at
<a href="https://urldefense.com/v3/__http:/mpi4spark001:-1__;!!KGKeukY!1yam9MqJGXUuv-UdrWeakBeZ8y-emE258qcPhFWor-ytPm-G1OuIFGfhJjah0Jd4dl55B1r8DghhPvPOMekN7tMAdIspKlZd_zS9$">
http://mpi4spark001:-1</a><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:53,737 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@648bb0dd{/logPage,null,AVAILABLE,@Spark}<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:53,738 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@15e3912{/logPage/json,null,AVAILABLE,@Spark}<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:53,739 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@5f9f4fef{/,null,AVAILABLE,@Spark}<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:53,740 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@50387b20{/json,null,AVAILABLE,@Spark}<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:53,745 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@1117f451{/static,null,AVAILABLE,@Spark}<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:53,746 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@35f11bf7{/log,null,AVAILABLE,@Spark}<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:53,747 INFO worker.Worker: Connecting to master mpi4spark002:7077...<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:53,757 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@113317dd{/metrics/json,null,AVAILABLE,@Spark}<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:53,785 INFO client.TransportClientFactory: Successfully created connection to mpi4spark002/200.2.151.2:7077 after 25 ms (0 ms spent in bootstraps)<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:53,835 INFO worker.Worker: Successfully registered with master spark://mpi4spark002:7077<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:53,946 INFO worker.Worker: Asked to launch executor app-20240522163753-0000/1 for Spark Pi<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:53,961 INFO spark.SecurityManager: Changing view acls to: root<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:53,961 INFO spark.SecurityManager: Changing modify acls to: root<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:53,961 INFO spark.SecurityManager: Changing view acls groups to:<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:53,961 INFO spark.SecurityManager: Changing modify acls groups to:<o:p></o:p></p>
</div>
<p class="MsoNormal">2024-05-22 16:37:53,961 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root); groups with view permissions: Set(); users with modify permissions: Set(root); groups
with modify permissions: Set()<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">app.log<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:53,006 INFO spark.SparkContext: Running Spark version 3.3.0-SNAPSHOT<o:p></o:p></p>
<div>
<p class="MsoNormal">2024-05-22 16:37:53,043 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:53,116 INFO resource.ResourceUtils: ==============================================================<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:53,116 INFO resource.ResourceUtils: No custom resources configured for spark.driver.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:53,117 INFO resource.ResourceUtils: ==============================================================<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:53,117 INFO spark.SparkContext: Submitted application: Spark Pi<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:53,133 INFO resource.ResourceProfile: Default ResourceProfile created, executor resources: Map(cores -> name: cores, amount: 1, script: , vendor: , memory -> name: memory, amount: 1024, script: , vendor: , offHeap -> name:
offHeap, amount: 0, script: , vendor: ), task resources: Map(cpus -> name: cpus, amount: 1.0)<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:53,141 INFO resource.ResourceProfile: Limiting resource is cpu<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:53,141 INFO resource.ResourceProfileManager: Added ResourceProfile id: 0<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:53,173 INFO spark.SecurityManager: Changing view acls to: root<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:53,174 INFO spark.SecurityManager: Changing modify acls to: root<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:53,174 INFO spark.SecurityManager: Changing view acls groups to:<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:53,174 INFO spark.SecurityManager: Changing modify acls groups to:<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:53,174 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root); groups with view permissions: Set(); users with modify permissions: Set(root); groups
with modify permissions: Set()<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:53,395 INFO nio.NioEventLoop: Starting MPI4Spark.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:53,462 INFO util.Utils: Successfully started service 'sparkDriver' on port 45597.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:53,478 INFO spark.SparkEnv: Registering MapOutputTracker<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:53,498 INFO spark.SparkEnv: Registering BlockManagerMaster<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:53,509 INFO storage.BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:53,510 INFO storage.BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:53,512 INFO spark.SparkEnv: Registering BlockManagerMasterHeartbeat<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:53,524 INFO storage.DiskBlockManager: Created local directory at /tmp/blockmgr-4d5b7f26-5a1f-4c8c-9b8c-f60132a6f261<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:53,538 INFO memory.MemoryStore: MemoryStore started with capacity 408.9 MiB<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:53,548 INFO spark.SparkEnv: Registering OutputCommitCoordinator<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:53,571 INFO util.log: Logging initialized @1227ms to org.sparkproject.jetty.util.log.Slf4jLog<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:53,641 INFO server.Server: jetty-9.4.43.v20210629; built: 2021-06-30T11:07:22.254Z; git: 526006ecfa3af7f1a27ef3a288e2bef7ea9dd7e8; jvm 1.8.0_321-b07<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:53,654 INFO server.Server: Started @1311ms<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:53,680 INFO server.AbstractConnector: Started ServerConnector@5225305d{HTTP/1.1, (http/1.1)}{0.0.0.0:4040}<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:53,680 INFO util.Utils: Successfully started service 'SparkUI' on port 4040.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:53,681 INFO ui.SparkUI: Bound SparkUI to 0.0.0.0, and started at
<a href="https://urldefense.com/v3/__http:/mpi4spark003:-1__;!!KGKeukY!1yam9MqJGXUuv-UdrWeakBeZ8y-emE258qcPhFWor-ytPm-G1OuIFGfhJjah0Jd4dl55B1r8DghhPvPOMekN7tMAdIspKvKpuJRR$">
http://mpi4spark003:-1</a><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:53,694 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@3fcdcf{/,null,AVAILABLE,@Spark}<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:53,703 INFO spark.SparkContext: Added JAR file:/root/mpi4spark/mpi4spark-0.2-x86-bin/examples/jars/spark-examples_2.12-3.3.0-SNAPSHOT.jar at spark://mpi4spark003:45597/jars/spark-examples_2.12-3.3.0-SNAPSHOT.jar with timestamp
1716367073001<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:53,818 INFO client.StandaloneAppClient$ClientEndpoint: Connecting to master spark://mpi4spark002:7077...<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:53,853 INFO client.TransportClientFactory: Successfully created connection to mpi4spark002/200.2.151.2:7077 after 24 ms (0 ms spent in bootstraps)<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:53,912 INFO cluster.StandaloneSchedulerBackend: Connected to Spark cluster with app ID app-20240522163753-0000<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:53,915 INFO util.Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 36911.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:53,916 INFO netty.NettyBlockTransferService: Server created on mpi4spark003:36911<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:53,916 INFO storage.BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:53,920 INFO storage.BlockManagerMaster: Registering BlockManager BlockManagerId(driver, mpi4spark003, 36911, None)<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:53,922 INFO storage.BlockManagerMasterEndpoint: Registering block manager mpi4spark003:36911 with 408.9 MiB RAM, BlockManagerId(driver, mpi4spark003, 36911, None)<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:53,923 INFO storage.BlockManagerMaster: Registered BlockManager BlockManagerId(driver, mpi4spark003, 36911, None)<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:53,924 INFO storage.BlockManager: Initialized BlockManager: BlockManagerId(driver, mpi4spark003, 36911, None)<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:53,929 INFO client.StandaloneAppClient$ClientEndpoint: Executor added: app-20240522163753-0000/0 on worker-20240522163753-200.2.149.2-40247 (200.2.149.2:40247) with 64 core(s)<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:53,930 INFO cluster.StandaloneSchedulerBackend: Granted executor ID app-20240522163753-0000/0 on hostPort 200.2.149.2:40247 with 64 core(s), 1024.0 MiB RAM<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:53,930 INFO client.StandaloneAppClient$ClientEndpoint: Executor added: app-20240522163753-0000/1 on worker-20240522163753-200.2.149.6-46337 (200.2.149.6:46337) with 64 core(s)<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:54,020 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@3fcdcf{/,null,STOPPED,@Spark}<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:54,021 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@28d79cba{/jobs,null,AVAILABLE,@Spark}<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:54,021 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@29f0c4f2{/jobs/json,null,AVAILABLE,@Spark}<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:54,022 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@13047d7d{/jobs/job,null,AVAILABLE,@Spark}<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:54,022 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@65bb9029{/jobs/job/json,null,AVAILABLE,@Spark}<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:54,022 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@2b214b94{/stages,null,AVAILABLE,@Spark}<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:54,023 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@49601f82{/stages/json,null,AVAILABLE,@Spark}<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:54,023 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@24fabd0f{/stages/stage,null,AVAILABLE,@Spark}<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:54,024 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@61f3fbb8{/stages/stage/json,null,AVAILABLE,@Spark}<o:p></o:p></p>
</div>
<p class="MsoNormal">2024-05-22 16:37:54,024 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@432034a{/stages/pool,null,AVAILABLE,@Spark}<o:p></o:p></p>
<div>
<p class="MsoNormal">2024-05-22 16:37:54,025 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@60e5272{/stages/pool/json,null,AVAILABLE,@Spark}<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:54,025 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@69c93ca4{/storage,null,AVAILABLE,@Spark}<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:54,025 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@173373b4{/storage/json,null,AVAILABLE,@Spark}<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:54,026 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@60dd3c23{/storage/rdd,null,AVAILABLE,@Spark}<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:54,026 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@5e9456ae{/storage/rdd/json,null,AVAILABLE,@Spark}<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:54,026 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@1f1cae23{/environment,null,AVAILABLE,@Spark}<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:54,027 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@985696{/environment/json,null,AVAILABLE,@Spark}<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:54,027 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@215a34b4{/executors,null,AVAILABLE,@Spark}<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:54,027 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@35d3ab60{/executors/json,null,AVAILABLE,@Spark}<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:54,028 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@71870da7{/executors/threadDump,null,AVAILABLE,@Spark}<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:54,028 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@45792847{/executors/threadDump/json,null,AVAILABLE,@Spark}<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:54,034 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@4e25147a{/static,null,AVAILABLE,@Spark}<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:54,034 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@5f303ecd{/,null,AVAILABLE,@Spark}<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:54,035 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@25a73de1{/api,null,AVAILABLE,@Spark}<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:54,035 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@260f2144{/jobs/job/kill,null,AVAILABLE,@Spark}<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:54,036 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@51827393{/stages/stage/kill,null,AVAILABLE,@Spark}<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:54,038 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@7ec58feb{/metrics/json,null,AVAILABLE,@Spark}<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:54,038 INFO cluster.StandaloneSchedulerBackend: SchedulerBackend is ready for scheduling beginning after reached minRegisteredResourcesRatio: 0.0<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:54,302 INFO spark.SparkContext: Starting job: reduce at SparkPi.scala:38<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:54,311 INFO scheduler.DAGScheduler: Got job 0 (reduce at SparkPi.scala:38) with 2 output partitions<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:54,311 INFO scheduler.DAGScheduler: Final stage: ResultStage 0 (reduce at SparkPi.scala:38)<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:54,311 INFO scheduler.DAGScheduler: Parents of final stage: List()<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:54,312 INFO scheduler.DAGScheduler: Missing parents: List()<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:54,314 INFO scheduler.DAGScheduler: Submitting ResultStage 0 (MapPartitionsRDD[1] at map at SparkPi.scala:34), which has no missing parents<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:54,349 INFO memory.MemoryStore: Block broadcast_0 stored as values in memory (estimated size 4.0 KiB, free 408.9 MiB)<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:54,365 INFO memory.MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 2.3 KiB, free 408.9 MiB)<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:54,367 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in memory on mpi4spark003:36911 (size: 2.3 KiB, free: 408.9 MiB)<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:54,368 INFO spark.SparkContext: Created broadcast 0 from broadcast at DAGScheduler.scala:1429<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:54,376 INFO scheduler.DAGScheduler: Submitting 2 missing tasks from ResultStage 0 (MapPartitionsRDD[1] at map at SparkPi.scala:34) (first 15 tasks are for partitions Vector(0, 1))<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:37:54,377 INFO scheduler.TaskSchedulerImpl: Adding task set 0.0 with 2 tasks resource profile 0<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:38:09,388 WARN scheduler.TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:38:24,387 WARN scheduler.TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:38:39,387 WARN scheduler.TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:38:54,388 WARN scheduler.TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:39:09,387 WARN scheduler.TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:39:24,387 WARN scheduler.TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:39:39,387 WARN scheduler.TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:39:54,387 WARN scheduler.TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:40:09,387 WARN scheduler.TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:40:24,387 WARN scheduler.TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:40:39,387 WARN scheduler.TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:40:54,387 WARN scheduler.TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:41:09,387 WARN scheduler.TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:41:24,388 WARN scheduler.TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2024-05-22 16:41:39,387 WARN scheduler.TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources<o:p></o:p></p>
</div>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
</div>
</div>
</body>
</html>