<!-- BaNnErBlUrFlE-BoDy-start -->
<!-- Preheader Text : BEGIN -->
<div style="display:none !important;display:none;visibility:hidden;mso-hide:all;font-size:1px;color:#ffffff;line-height:1px;height:0px;max-height:0px;opacity:0;overflow:hidden;">
Hello, I am currently trying to install mpi4spark-0. 2-x86-bin and run the "GroupByTest" (6. 2. 1 in the user guide). I am running into the following error: exec. log: [ffmk-n3: mpispawn_0][readline] Unexpected End-Of-File on file descriptor
</div>
<!-- Preheader Text : END -->
<!-- Email Banner : BEGIN -->
<div style="display:none !important;display:none;visibility:hidden;mso-hide:all;font-size:1px;color:#ffffff;line-height:1px;height:0px;max-height:0px;opacity:0;overflow:hidden;">ZjQcmQRYFpfptBannerStart</div>
<!--[if ((ie)|(mso))]>
<table border="0" cellspacing="0" cellpadding="0" width="100%" style="padding: 16px 0px 16px 0px; direction: ltr" lang="en"><tr><td>
<table border="0" cellspacing="0" cellpadding="0" style="padding: 0px 10px 5px 6px; width: 100%; border-radius:4px; border-top:4px solid #8c8e91;background-color:#CFD3D7;"><tr><td valign="top">
<table align="left" border="0" cellspacing="0" cellpadding="0" style="padding: 4px 8px 4px 8px">
<tr><td style="color:#000000; font-family: 'Arial', sans-serif; font-weight:bold; font-size:14px; direction: ltr">
This Message Is From an External Sender
</td></tr>
<tr><td style="color:#000000; font-weight:normal; font-family: 'Arial', sans-serif; font-size:12px; direction: ltr">
This message came from outside your organization.
</td></tr>
</table>
<![if ie]><br clear="all"><![endif]>
<table align="right" border="0" cellspacing="0" cellpadding="0" style="padding: 4px 0px 4px 0px"><tr>
<td style="direction: ltr"> <a target="_blank" href="https://us-phishalarm-ewt.proofpoint.com/EWT/v1/KGKeukY!vOQd8yZtA6cARRdxP2-Ed4o3zhqbsM51VvDYO-EHBae3XuNJ7weArIhwIiM1cppw3wez14CJsr4VZLT9uba6yz3vmV4OzObSI5n15tchxeGJ-ttnsOhAGEQ0pb7usmuFyXFWd6r_jwngCHw3RcPc$" style="mso-padding-alt: 7.5px; padding: 7.5px; border-radius: 2px; border: 1.5px solid #666666; "><strong style="font-weight: normal; color: #000000; text-decoration: none; font-family: 'Arial', sans-serif; font-size:14px; line-height: 40px; "> Report Suspicious </strong></a> </td>
</tr></table>
</td></tr></table>
</td></tr></table>
<![endif]-->
<![if !((ie)|(mso))]>
<div dir="ltr" lang="en" id="pfptBannerb87ka6v" style="all: revert !important; display:block !important; text-align: left !important; margin:16px 0px 16px 0px !important; padding:8px 16px 8px 16px !important; border-radius: 4px !important; min-width: 200px !important; background-color: #CFD3D7 !important; background-color: #CFD3D7; border-top: 4px solid #8c8e91 !important; border-top: 4px solid #8c8e91;">
<div id="pfptBannerb87ka6v" style="all: unset !important; float:left !important; display:block !important; margin: 0px 0px 1px 0px !important; max-width: 600px !important;">
<div id="pfptBannerb87ka6v" style="all: unset !important; display:block !important; visibility: visible !important; background-color: #CFD3D7 !important; color:#000000 !important; color:#000000; font-family: 'Arial', sans-serif !important; font-family: 'Arial', sans-serif; font-weight:bold !important; font-weight:bold; font-size:14px !important; line-height:18px !important; line-height:18px">
This Message Is From an External Sender
</div>
<div id="pfptBannerb87ka6v" style="all: unset !important; display:block !important; visibility: visible !important; background-color: #CFD3D7 !important; color:#000000 !important; color:#000000; font-weight:normal; font-family: 'Arial', sans-serif !important; font-family: 'Arial', sans-serif; font-size:12px !important; line-height:18px !important; line-height:18px; margin-top:2px !important;">
This message came from outside your organization.
</div>
</div>
<div id="pfptBannerb87ka6v" style="all: unset !important; float: right !important; display: block !important; display: block; margin: 0px 0px 0px 16px !important; text-align: right !important; width: fit-content !important;">
<a id="pfptBannerb87ka6v" href="https://us-phishalarm-ewt.proofpoint.com/EWT/v1/KGKeukY!vOQd8yZtA6cARRdxP2-Ed4o3zhqbsM51VvDYO-EHBae3XuNJ7weArIhwIiM1cppw3wez14CJsr4VZLT9uba6yz3vmV4OzObSI5n15tchxeGJ-ttnsOhAGEQ0pb7usmuFyXFWd6r_jwngCHw3RcPc$"
style="all: unset !important; display: inline-block !important; text-decoration: none">
<div class="pfptPrimaryButtonb87ka6v" style="display: inline-block !important; display: inline-block; visibility: visible !important; opacity: 1 !important; color: #000000 !important; color: #000000; font-family: 'Arial', sans-serif !important; font-family: 'Arial', sans-serif; font-size: 14px !important; font-weight: normal !important; text-decoration: none !important; border-radius: 2px !important; padding: 7.5px 16px !important; margin: 3px 0 3px 16px !important; white-space: nowrap !important; width: fit-content !important;
border: 1px solid #666666">
Report Suspicious
</div>
</a>
</div>
<div style="clear: both !important; display: block !important; visibility: hidden !important; line-height: 0 !important; font-size: 0.01px !important; height: 0px"> </div>
</div>
<![endif]>
<div style="display:none !important;display:none;visibility:hidden;mso-hide:all;font-size:1px;color:#ffffff;line-height:1px;height:0px;max-height:0px;opacity:0;overflow:hidden;">ZjQcmQRYFpfptBannerEnd</div>
<!-- Email Banner : END -->
<!-- BaNnErBlUrFlE-BoDy-end -->
<html>
<head><!-- BaNnErBlUrFlE-HeAdEr-start -->
<style>
#pfptBannerb87ka6v { all: revert !important; display: block !important;
visibility: visible !important; opacity: 1 !important;
background-color: #CFD3D7 !important;
max-width: none !important; max-height: none !important }
.pfptPrimaryButtonb87ka6v:hover, .pfptPrimaryButtonb87ka6v:focus {
background-color: #adb0b4 !important; }
.pfptPrimaryButtonb87ka6v:active {
background-color: #8c8e91 !important; }
</style>
<!-- BaNnErBlUrFlE-HeAdEr-end -->
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
Hello,</div>
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<br>
</div>
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof ContentPasted0">
I am currently trying to install mpi4spark-0.2-x86-bin and run the "GroupByTest" (6.2.1 in the
<a href="http://hibd.cse.ohio-state.edu/static/media/hibd/mpi4spark/mpi4spark_user_guide.pdf" title="http://hibd.cse.ohio-state.edu/static/media/hibd/mpi4spark/mpi4spark_user_guide.pdf" data-loopstyle="linkonly" id="LPlnk408790" class="OWAAutoLink">
user guide</a>).</div>
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof ContentPasted0">
<br>
</div>
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof ContentPasted0">
I am running into the following error:</div>
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof ContentPasted0">
<br>
</div>
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof ContentPasted0">
exec.log:</div>
<blockquote style="margin-top:0;margin-bottom:0">
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof ContentPasted0 ContentPasted1">
[ffmk-n3:mpispawn_0][readline] Unexpected End-Of-File on file descriptor 6. MPI process died?
<div class="ContentPasted1">[ffmk-n3:mpispawn_0][mtpmi_processops] Error while reading PMI socket. MPI process died?</div>
<div class="ContentPasted1">[ffmk-n3:mpispawn_0][child_handler] MPI process (rank: 0, pid: 2197) exited with status 130</div>
</div>
</blockquote>
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof ContentPasted0 ContentPasted1">
<br>
</div>
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof ContentPasted0 ContentPasted1">
app.log:</div>
<blockquote style="margin-top:0;margin-bottom:0">
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof ContentPasted0 ContentPasted1 ContentPasted2">
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
<div class="ContentPasted2">23/09/24 13:28:29 INFO SparkContext: Running Spark version 3.3.0-SNAPSHOT</div>
<div class="ContentPasted2">23/09/24 13:28:29 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable</div>
<div class="ContentPasted2">23/09/24 13:28:29 INFO ResourceUtils: ==============================================================</div>
<div class="ContentPasted2">23/09/24 13:28:29 INFO ResourceUtils: No custom resources configured for spark.driver.</div>
<div class="ContentPasted2">23/09/24 13:28:29 INFO ResourceUtils: ==============================================================</div>
<div class="ContentPasted2">23/09/24 13:28:29 INFO SparkContext: Submitted application: GroupBy Test</div>
<div class="ContentPasted2">23/09/24 13:28:29 INFO ResourceProfile: Default ResourceProfile created, executor resources: Map(cores -> name: cores, amount: 1, script: , vendor: , memory -> name: memory, amount: 1024, script: , vendor: , offHeap -> name: offHeap,
amount: 0, script: , vendor: ), task resources: Map(cpus -> name: cpus, amount: 1.0)</div>
<div class="ContentPasted2">23/09/24 13:28:29 INFO ResourceProfile: Limiting resource is cpu</div>
<div class="ContentPasted2">23/09/24 13:28:29 INFO ResourceProfileManager: Added ResourceProfile id: 0</div>
<div class="ContentPasted2">23/09/24 13:28:29 INFO SecurityManager: Changing view acls to: bengt</div>
<div class="ContentPasted2">23/09/24 13:28:29 INFO SecurityManager: Changing modify acls to: bengt</div>
<div class="ContentPasted2">23/09/24 13:28:29 INFO SecurityManager: Changing view acls groups to:
</div>
<div class="ContentPasted2">23/09/24 13:28:29 INFO SecurityManager: Changing modify acls groups to:
</div>
<div class="ContentPasted2">23/09/24 13:28:29 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(bengt); groups with view permissions: Set(); users with modify permissions: Set(bengt); groups
with modify permissions: Set()</div>
<div class="ContentPasted2">[ffmk-n3:mpi_rank_0][rdma_find_network_type] Unable to find the numa process is bound to. Disabling process placement aware hca mapping.</div>
<div class="ContentPasted2">[ffmk-n3:mpi_rank_0][MPID_Init] [Performance Suggestion]: Application has requested for multi-thread capability. If allocating memory from different pthreads/OpenMP threads, please consider setting MV2_USE_ALIGNED_ALLOC=1 for improved
performance.</div>
<div class="ContentPasted2">Use MV2_USE_THREAD_WARNING=0 to suppress this error message.</div>
<div class="ContentPasted2">23/09/24 13:28:29 INFO NioEventLoop: Starting MPI4Spark.</div>
<div class="ContentPasted2">23/09/24 13:28:29 INFO Utils: Successfully started service 'sparkDriver' on port 33211.</div>
<div class="ContentPasted2">23/09/24 13:28:29 INFO SparkEnv: Registering MapOutputTracker</div>
<div class="ContentPasted2">WARNING: An illegal reflective access operation has occurred</div>
<div class="ContentPasted2">WARNING: Illegal reflective access by org.apache.spark.unsafe.Platform (file:/home/bengt/tmp/mpi4spark-0.2-x86-bin/jars/spark-unsafe_2.12-3.3.0-SNAPSHOT.jar) to constructor java.nio.DirectByteBuffer(long,int)</div>
<div class="ContentPasted2">WARNING: Please consider reporting this to the maintainers of org.apache.spark.unsafe.Platform</div>
<div class="ContentPasted2">WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations</div>
<div class="ContentPasted2">WARNING: All illegal access operations will be denied in a future release</div>
<div class="ContentPasted2">23/09/24 13:28:29 INFO SparkEnv: Registering BlockManagerMaster</div>
<div class="ContentPasted2">23/09/24 13:28:29 INFO BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information</div>
<div class="ContentPasted2">23/09/24 13:28:29 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up</div>
<div class="ContentPasted2">23/09/24 13:28:29 INFO SparkEnv: Registering BlockManagerMasterHeartbeat</div>
<div class="ContentPasted2">23/09/24 13:28:29 INFO DiskBlockManager: Created local directory at /tmp/blockmgr-25cd1d9f-cb83-4262-a938-0057ad98f451</div>
<div class="ContentPasted2">23/09/24 13:28:29 INFO MemoryStore: MemoryStore started with capacity 434.4 MiB</div>
<div class="ContentPasted2">23/09/24 13:28:29 INFO SparkEnv: Registering OutputCommitCoordinator</div>
<div class="ContentPasted2">23/09/24 13:28:30 INFO Utils: Successfully started service 'SparkUI' on port 4040.</div>
<div class="ContentPasted2">23/09/24 13:28:30 INFO SparkUI: Bound SparkUI to 0.0.0.0, and started at <a href="https://urldefense.com/v3/__http://*3Cffmk3-url*3E:-1__;JSU!!KGKeukY!1O8XjuysQLdi7c6JzYF2Zw2YcSnphLcw9QP-DKnQ-HnKS6fJMaN6sUZfl_YFva3lfE8wDWDs-5s8vdO-mHM-oR2dUgQ6SIyl4O6Q-OJJ7-idFg$">http://<ffmk3-url>:-1</a></div>
<div class="ContentPasted2">23/09/24 13:28:30 INFO SparkContext: Added JAR file:/home/bengt/tmp/mpi4spark-0.2-x86-bin/examples/jars/spark-examples_2.12-3.3.0-SNAPSHOT.jar at spark://<ffmk3-url>:33211/jars/spark-examples_2.12-3.3.0-SNAPSHOT.jar with timestamp
1695562109112</div>
<div class="ContentPasted2">23/09/24 13:28:30 INFO StandaloneAppClient$ClientEndpoint: Connecting to master spark://ffmk-n3:7077...</div>
<div class="ContentPasted2">23/09/24 13:28:30 WARN ChannelInitializer: Failed to initialize a channel. Closing: [id: 0x2430e197]</div>
<div class="ContentPasted2">java.lang.NullPointerException</div>
<div class="ContentPasted2"> at java.base/java.lang.String.contains(String.java:2036)</div>
<div class="ContentPasted2"> at org.apache.spark.network.TransportContext.initializePipeline(TransportContext.java:221)</div>
<div class="ContentPasted2"> at org.apache.spark.network.TransportContext.initializePipeline(TransportContext.java:194)</div>
<div class="ContentPasted2"> at org.apache.spark.network.client.TransportClientFactory$1.initChannel(TransportClientFactory.java:302)</div>
<div class="ContentPasted2"> at org.apache.spark.network.client.TransportClientFactory$1.initChannel(TransportClientFactory.java:299)</div>
<div class="ContentPasted2"> at io.netty.channel.ChannelInitializer.initChannel(ChannelInitializer.java:129)</div>
<div class="ContentPasted2"> at io.netty.channel.ChannelInitializer.handlerAdded(ChannelInitializer.java:112)</div>
<div class="ContentPasted2"> at io.netty.channel.AbstractChannelHandlerContext.callHandlerAdded(AbstractChannelHandlerContext.java:943)</div>
<div class="ContentPasted2"> at io.netty.channel.DefaultChannelPipeline.callHandlerAdded0(DefaultChannelPipeline.java:609)</div>
<div class="ContentPasted2"> at io.netty.channel.DefaultChannelPipeline.access$100(DefaultChannelPipeline.java:46)</div>
<div class="ContentPasted2"> at io.netty.channel.DefaultChannelPipeline$PendingHandlerAddedTask.execute(DefaultChannelPipeline.java:1463)</div>
<div class="ContentPasted2"> at io.netty.channel.DefaultChannelPipeline.callHandlerAddedForAllHandlers(DefaultChannelPipeline.java:1115)</div>
<div class="ContentPasted2"> at io.netty.channel.DefaultChannelPipeline.invokeHandlerAddedIfNeeded(DefaultChannelPipeline.java:650)</div>
<div class="ContentPasted2"> at io.netty.channel.AbstractChannel$AbstractUnsafe.register0(AbstractChannel.java:528)</div>
<div class="ContentPasted2"> at io.netty.channel.AbstractChannel$AbstractUnsafe.access$200(AbstractChannel.java:443)</div>
<div class="ContentPasted2"> at io.netty.channel.AbstractChannel$AbstractUnsafe$1.run(AbstractChannel.java:500)</div>
<div class="ContentPasted2"> at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)</div>
<div class="ContentPasted2"> at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:469)</div>
<div class="ContentPasted2"> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:559)</div>
<div class="ContentPasted2"> at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)</div>
<div class="ContentPasted2"> at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)</div>
<div class="ContentPasted2"> at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)</div>
<div class="ContentPasted2"> at java.base/java.lang.Thread.run(Thread.java:834)</div>
<div class="ContentPasted2">23/09/24 13:28:30 WARN StandaloneAppClient$ClientEndpoint: Failed to connect to master ffmk-n3:7077</div>
<div class="ContentPasted2">org.apache.spark.SparkException: Exception thrown in awaitResult:
</div>
<div class="ContentPasted2"> at org.apache.spark.util.ThreadUtils$.awaitResult(ThreadUtils.scala:301)</div>
<div class="ContentPasted2"> at org.apache.spark.rpc.RpcTimeout.awaitResult(RpcTimeout.scala:75)</div>
<div class="ContentPasted2"> at org.apache.spark.rpc.RpcEnv.setupEndpointRefByURI(RpcEnv.scala:101)</div>
<div class="ContentPasted2"> at org.apache.spark.rpc.RpcEnv.setupEndpointRef(RpcEnv.scala:109)</div>
<div class="ContentPasted2"> at org.apache.spark.deploy.client.StandaloneAppClient$ClientEndpoint$$anon$1.run(StandaloneAppClient.scala:126)</div>
<div class="ContentPasted2"> at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)</div>
<div class="ContentPasted2"> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)</div>
<div class="ContentPasted2"> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)</div>
<div class="ContentPasted2"> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)</div>
<div class="ContentPasted2"> at java.base/java.lang.Thread.run(Thread.java:834)</div>
<div class="ContentPasted2">Caused by: java.io.IOException: Failed to connect to ffmk-n3/141.76.48.47:7077</div>
<div class="ContentPasted2"> at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:316)</div>
<div class="ContentPasted2"> at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:246)</div>
<div class="ContentPasted2"> at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:258)</div>
<div class="ContentPasted2"> at org.apache.spark.rpc.netty.NettyRpcEnv.createClient(NettyRpcEnv.scala:204)</div>
<div class="ContentPasted2"> at org.apache.spark.rpc.netty.Outbox$$anon$1.call(Outbox.scala:202)</div>
<div class="ContentPasted2"> at org.apache.spark.rpc.netty.Outbox$$anon$1.call(Outbox.scala:198)</div>
<div class="ContentPasted2"> ... 4 more</div>
<div class="ContentPasted2">Caused by: io.netty.channel.StacklessClosedChannelException</div>
<div class="ContentPasted2"> at io.netty.channel.AbstractChannel$AbstractUnsafe.ensureOpen(ChannelPromise)(Unknown Source)</div>
<div class="ContentPasted2">23/09/24 13:28:50 INFO StandaloneAppClient$ClientEndpoint: Connecting to master spark://ffmk-n3:7077...</div>
<div class="ContentPasted2">23/09/24 13:28:50 WARN ChannelInitializer: Failed to initialize a channel. Closing: [id: 0x564d0fc7]</div>
<div class="ContentPasted2">java.lang.NullPointerException</div>
<div class="ContentPasted2"> at java.base/java.lang.String.contains(String.java:2036)</div>
<div class="ContentPasted2"> at org.apache.spark.network.TransportContext.initializePipeline(TransportContext.java:221)</div>
<div class="ContentPasted2"> at org.apache.spark.network.TransportContext.initializePipeline(TransportContext.java:194)</div>
<div class="ContentPasted2"> at org.apache.spark.network.client.TransportClientFactory$1.initChannel(TransportClientFactory.java:302)</div>
<div class="ContentPasted2"> at org.apache.spark.network.client.TransportClientFactory$1.initChannel(TransportClientFactory.java:299)</div>
<div class="ContentPasted2"> at io.netty.channel.ChannelInitializer.initChannel(ChannelInitializer.java:129)</div>
<div class="ContentPasted2"> at io.netty.channel.ChannelInitializer.handlerAdded(ChannelInitializer.java:112)</div>
<div class="ContentPasted2"> at io.netty.channel.AbstractChannelHandlerContext.callHandlerAdded(AbstractChannelHandlerContext.java:943)</div>
<div class="ContentPasted2"> at io.netty.channel.DefaultChannelPipeline.callHandlerAdded0(DefaultChannelPipeline.java:609)</div>
<div class="ContentPasted2"> at io.netty.channel.DefaultChannelPipeline.access$100(DefaultChannelPipeline.java:46)</div>
<div class="ContentPasted2"> at io.netty.channel.DefaultChannelPipeline$PendingHandlerAddedTask.execute(DefaultChannelPipeline.java:1463)</div>
<div class="ContentPasted2"> at io.netty.channel.DefaultChannelPipeline.callHandlerAddedForAllHandlers(DefaultChannelPipeline.java:1115)</div>
<div class="ContentPasted2"> at io.netty.channel.DefaultChannelPipeline.invokeHandlerAddedIfNeeded(DefaultChannelPipeline.java:650)</div>
<div class="ContentPasted2"> at io.netty.channel.AbstractChannel$AbstractUnsafe.register0(AbstractChannel.java:528)</div>
<div class="ContentPasted2"> at io.netty.channel.AbstractChannel$AbstractUnsafe.access$200(AbstractChannel.java:443)</div>
<div class="ContentPasted2"> at io.netty.channel.AbstractChannel$AbstractUnsafe$1.run(AbstractChannel.java:500)</div>
<div class="ContentPasted2"> at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)</div>
<div class="ContentPasted2"> at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:469)</div>
<div class="ContentPasted2"> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:559)</div>
<div class="ContentPasted2"> at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)</div>
<div class="ContentPasted2"> at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)</div>
<div class="ContentPasted2"> at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)</div>
<div class="ContentPasted2"> at java.base/java.lang.Thread.run(Thread.java:834)</div>
<div class="ContentPasted2">23/09/24 13:28:50 WARN StandaloneAppClient$ClientEndpoint: Failed to connect to master ffmk-n3:7077</div>
<div class="ContentPasted2">org.apache.spark.SparkException: Exception thrown in awaitResult:
</div>
<div class="ContentPasted2"> at org.apache.spark.util.ThreadUtils$.awaitResult(ThreadUtils.scala:301)</div>
<div class="ContentPasted2"> at org.apache.spark.rpc.RpcTimeout.awaitResult(RpcTimeout.scala:75)</div>
<div class="ContentPasted2"> at org.apache.spark.rpc.RpcEnv.setupEndpointRefByURI(RpcEnv.scala:101)</div>
<div class="ContentPasted2"> at org.apache.spark.rpc.RpcEnv.setupEndpointRef(RpcEnv.scala:109)</div>
<div class="ContentPasted2"> at org.apache.spark.deploy.client.StandaloneAppClient$ClientEndpoint$$anon$1.run(StandaloneAppClient.scala:126)</div>
<div class="ContentPasted2"> at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)</div>
<div class="ContentPasted2"> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)</div>
<div class="ContentPasted2"> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)</div>
<div class="ContentPasted2"> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)</div>
<div class="ContentPasted2"> at java.base/java.lang.Thread.run(Thread.java:834)</div>
<div class="ContentPasted2">Caused by: java.io.IOException: Failed to connect to ffmk-n3/141.76.48.47:7077</div>
<div class="ContentPasted2"> at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:316)</div>
<div class="ContentPasted2"> at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:246)</div>
<div class="ContentPasted2"> at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:258)</div>
<div class="ContentPasted2"> at org.apache.spark.rpc.netty.NettyRpcEnv.createClient(NettyRpcEnv.scala:204)</div>
<div class="ContentPasted2"> at org.apache.spark.rpc.netty.Outbox$$anon$1.call(Outbox.scala:202)</div>
<div class="ContentPasted2"> at org.apache.spark.rpc.netty.Outbox$$anon$1.call(Outbox.scala:198)</div>
<div class="ContentPasted2"> ... 4 more</div>
<div class="ContentPasted2">Caused by: io.netty.channel.StacklessClosedChannelException</div>
<div class="ContentPasted2"> at io.netty.channel.AbstractChannel$AbstractUnsafe.ensureOpen(ChannelPromise)(Unknown Source)</div>
<div class="ContentPasted2">23/09/24 13:29:10 INFO StandaloneAppClient$ClientEndpoint: Connecting to master spark://ffmk-n3:7077...</div>
<div class="ContentPasted2">23/09/24 13:29:10 WARN ChannelInitializer: Failed to initialize a channel. Closing: [id: 0x2ee6c860]</div>
<div class="ContentPasted2">java.lang.NullPointerException</div>
<div class="ContentPasted2"> at java.base/java.lang.String.contains(String.java:2036)</div>
<div class="ContentPasted2"> at org.apache.spark.network.TransportContext.initializePipeline(TransportContext.java:221)</div>
<div class="ContentPasted2"> at org.apache.spark.network.TransportContext.initializePipeline(TransportContext.java:194)</div>
<div class="ContentPasted2"> at org.apache.spark.network.client.TransportClientFactory$1.initChannel(TransportClientFactory.java:302)</div>
<div class="ContentPasted2"> at org.apache.spark.network.client.TransportClientFactory$1.initChannel(TransportClientFactory.java:299)</div>
<div class="ContentPasted2"> at io.netty.channel.ChannelInitializer.initChannel(ChannelInitializer.java:129)</div>
<div class="ContentPasted2"> at io.netty.channel.ChannelInitializer.handlerAdded(ChannelInitializer.java:112)</div>
<div class="ContentPasted2"> at io.netty.channel.AbstractChannelHandlerContext.callHandlerAdded(AbstractChannelHandlerContext.java:943)</div>
<div class="ContentPasted2"> at io.netty.channel.DefaultChannelPipeline.callHandlerAdded0(DefaultChannelPipeline.java:609)</div>
<div class="ContentPasted2"> at io.netty.channel.DefaultChannelPipeline.access$100(DefaultChannelPipeline.java:46)</div>
<div class="ContentPasted2"> at io.netty.channel.DefaultChannelPipeline$PendingHandlerAddedTask.execute(DefaultChannelPipeline.java:1463)</div>
<div class="ContentPasted2"> at io.netty.channel.DefaultChannelPipeline.callHandlerAddedForAllHandlers(DefaultChannelPipeline.java:1115)</div>
<div class="ContentPasted2"> at io.netty.channel.DefaultChannelPipeline.invokeHandlerAddedIfNeeded(DefaultChannelPipeline.java:650)</div>
<div class="ContentPasted2"> at io.netty.channel.AbstractChannel$AbstractUnsafe.register0(AbstractChannel.java:528)</div>
<div class="ContentPasted2"> at io.netty.channel.AbstractChannel$AbstractUnsafe.access$200(AbstractChannel.java:443)</div>
<div class="ContentPasted2"> at io.netty.channel.AbstractChannel$AbstractUnsafe$1.run(AbstractChannel.java:500)</div>
<div class="ContentPasted2"> at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)</div>
<div class="ContentPasted2"> at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:469)</div>
<div class="ContentPasted2"> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:559)</div>
<div class="ContentPasted2"> at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)</div>
<div class="ContentPasted2"> at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)</div>
<div class="ContentPasted2"> at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)</div>
<div class="ContentPasted2"> at java.base/java.lang.Thread.run(Thread.java:834)</div>
<div class="ContentPasted2">23/09/24 13:29:10 WARN StandaloneAppClient$ClientEndpoint: Failed to connect to master ffmk-n3:7077</div>
<div class="ContentPasted2">org.apache.spark.SparkException: Exception thrown in awaitResult:
</div>
<div class="ContentPasted2"> at org.apache.spark.util.ThreadUtils$.awaitResult(ThreadUtils.scala:301)</div>
<div class="ContentPasted2"> at org.apache.spark.rpc.RpcTimeout.awaitResult(RpcTimeout.scala:75)</div>
<div class="ContentPasted2"> at org.apache.spark.rpc.RpcEnv.setupEndpointRefByURI(RpcEnv.scala:101)</div>
<div class="ContentPasted2"> at org.apache.spark.rpc.RpcEnv.setupEndpointRef(RpcEnv.scala:109)</div>
<div class="ContentPasted2"> at org.apache.spark.deploy.client.StandaloneAppClient$ClientEndpoint$$anon$1.run(StandaloneAppClient.scala:126)</div>
<div class="ContentPasted2"> at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)</div>
<div class="ContentPasted2"> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)</div>
<div class="ContentPasted2"> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)</div>
<div class="ContentPasted2"> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)</div>
<div class="ContentPasted2"> at java.base/java.lang.Thread.run(Thread.java:834)</div>
<div class="ContentPasted2">Caused by: java.io.IOException: Failed to connect to ffmk-n3/141.76.48.47:7077</div>
<div class="ContentPasted2"> at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:316)</div>
<div class="ContentPasted2"> at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:246)</div>
<div class="ContentPasted2"> at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:258)</div>
<div class="ContentPasted2"> at org.apache.spark.rpc.netty.NettyRpcEnv.createClient(NettyRpcEnv.scala:204)</div>
<div class="ContentPasted2"> at org.apache.spark.rpc.netty.Outbox$$anon$1.call(Outbox.scala:202)</div>
<div class="ContentPasted2"> at org.apache.spark.rpc.netty.Outbox$$anon$1.call(Outbox.scala:198)</div>
<div class="ContentPasted2"> ... 4 more</div>
<div class="ContentPasted2">Caused by: io.netty.channel.StacklessClosedChannelException</div>
<div class="ContentPasted2"> at io.netty.channel.AbstractChannel$AbstractUnsafe.ensureOpen(ChannelPromise)(Unknown Source)</div>
<div class="ContentPasted2">23/09/24 13:29:30 ERROR StandaloneSchedulerBackend: Application has been killed. Reason: All masters are unresponsive! Giving up.</div>
<div class="ContentPasted2">23/09/24 13:29:30 WARN StandaloneSchedulerBackend: Application ID is not initialized yet.</div>
<div class="ContentPasted2 ContentPasted23">23/09/24 13:29:30 INFO SparkUI: Stopped Spark web UI at <a href="https://urldefense.com/v3/__http://*3Cffmk3-url*3E:4040__;JSU!!KGKeukY!1O8XjuysQLdi7c6JzYF2Zw2YcSnphLcw9QP-DKnQ-HnKS6fJMaN6sUZfl_YFva3lfE8wDWDs-5s8vdO-mHM-oR2dUgQ6SIyl4O6Q-OIT6RVKyQ$">http://<ffmk3-url>:4040</a></div>
<div class="ContentPasted2">23/09/24 13:29:30 INFO StandaloneSchedulerBackend: Shutting down all executors</div>
<div class="ContentPasted2">23/09/24 13:29:30 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Asking each executor to shut down</div>
<div class="ContentPasted2">23/09/24 13:29:30 WARN StandaloneAppClient$ClientEndpoint: Drop UnregisterApplication(null) because has not yet connected to master</div>
<div class="ContentPasted2">23/09/24 13:29:30 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!</div>
<div class="ContentPasted2">23/09/24 13:29:30 INFO MemoryStore: MemoryStore cleared</div>
<div class="ContentPasted2">23/09/24 13:29:30 INFO BlockManager: BlockManager stopped</div>
<div class="ContentPasted2">23/09/24 13:29:30 INFO BlockManagerMaster: BlockManagerMaster stopped</div>
<div class="ContentPasted2">23/09/24 13:29:30 WARN MetricsSystem: Stopping a MetricsSystem that is not running</div>
<div class="ContentPasted2">23/09/24 13:29:30 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!</div>
<div class="ContentPasted2">23/09/24 13:29:30 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 33659.</div>
<div class="ContentPasted2 ContentPasted25">23/09/24 13:29:30 INFO NettyBlockTransferService: Server created on <ffmk3-url>:33659</div>
<div class="ContentPasted2">23/09/24 13:29:30 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy</div>
<div class="ContentPasted2 ContentPasted26">23/09/24 13:29:30 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, <ffmk3-url>, 33659, None)</div>
<div class="ContentPasted2">23/09/24 13:29:30 ERROR SparkContext: Error initializing SparkContext.</div>
<div class="ContentPasted2">java.lang.NullPointerException</div>
<div class="ContentPasted2"> at org.apache.spark.storage.BlockManagerMaster.registerBlockManager(BlockManagerMaster.scala:79)</div>
<div class="ContentPasted2"> at org.apache.spark.storage.BlockManager.initialize(BlockManager.scala:518)</div>
<div class="ContentPasted2"> at org.apache.spark.SparkContext.<init>(SparkContext.scala:593)</div>
<div class="ContentPasted2"> at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2690)</div>
<div class="ContentPasted2"> at org.apache.spark.sql.SparkSession$Builder.$anonfun$getOrCreate$2(SparkSession.scala:949)</div>
<div class="ContentPasted2"> at scala.Option.getOrElse(Option.scala:189)</div>
<div class="ContentPasted2"> at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:943)</div>
<div class="ContentPasted2"> at org.apache.spark.examples.GroupByTest$.main(GroupByTest.scala:32)</div>
<div class="ContentPasted2"> at org.apache.spark.examples.GroupByTest.main(GroupByTest.scala)</div>
<div class="ContentPasted2"> at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)</div>
<div class="ContentPasted2"> at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)</div>
<div class="ContentPasted2"> at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)</div>
<div class="ContentPasted2"> at java.base/java.lang.reflect.Method.invoke(Method.java:566)</div>
<div class="ContentPasted2"> at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)</div>
<div class="ContentPasted2"> at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:955)</div>
<div class="ContentPasted2"> at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)</div>
<div class="ContentPasted2"> at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)</div>
<div class="ContentPasted2"> at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)</div>
<div class="ContentPasted2"> at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1043)</div>
<div class="ContentPasted2"> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1052)</div>
<div class="ContentPasted2"> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)</div>
<div class="ContentPasted2">23/09/24 13:29:30 INFO SparkContext: SparkContext already stopped.</div>
<div class="ContentPasted2">Exception in thread "main" java.lang.NullPointerException</div>
<div class="ContentPasted2"> at org.apache.spark.storage.BlockManagerMaster.registerBlockManager(BlockManagerMaster.scala:79)</div>
<div class="ContentPasted2"> at org.apache.spark.storage.BlockManager.initialize(BlockManager.scala:518)</div>
<div class="ContentPasted2"> at org.apache.spark.SparkContext.<init>(SparkContext.scala:593)</div>
<div class="ContentPasted2"> at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2690)</div>
<div class="ContentPasted2"> at org.apache.spark.sql.SparkSession$Builder.$anonfun$getOrCreate$2(SparkSession.scala:949)</div>
<div class="ContentPasted2"> at scala.Option.getOrElse(Option.scala:189)</div>
<div class="ContentPasted2"> at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:943)</div>
<div class="ContentPasted2"> at org.apache.spark.examples.GroupByTest$.main(GroupByTest.scala:32)</div>
<div class="ContentPasted2"> at org.apache.spark.examples.GroupByTest.main(GroupByTest.scala)</div>
<div class="ContentPasted2"> at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)</div>
<div class="ContentPasted2"> at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)</div>
<div class="ContentPasted2"> at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)</div>
<div class="ContentPasted2"> at java.base/java.lang.reflect.Method.invoke(Method.java:566)</div>
<div class="ContentPasted2"> at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)</div>
<div class="ContentPasted2"> at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:955)</div>
<div class="ContentPasted2"> at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)</div>
<div class="ContentPasted2"> at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)</div>
<div class="ContentPasted2"> at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)</div>
<div class="ContentPasted2"> at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1043)</div>
<div class="ContentPasted2"> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1052)</div>
<div class="ContentPasted2"> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)</div>
<div class="ContentPasted2">23/09/24 13:29:30 INFO SparkContext: Successfully stopped SparkContext</div>
<div class="ContentPasted2">23/09/24 13:29:30 INFO ShutdownHookManager: Shutdown hook called</div>
<div class="ContentPasted2">23/09/24 13:29:30 INFO ShutdownHookManager: Deleting directory /tmp/spark-ddf1dc85-4e42-4ff1-b6cd-b2de089b8bdb</div>
<div class="ContentPasted2">23/09/24 13:29:30 INFO ShutdownHookManager: Deleting directory /tmp/spark-f5abcae1-4502-4b9d-8ba1-42184db05c21</div>
</div>
</blockquote>
<div class="ContentPasted2 elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div class="ContentPasted2 elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
So I probably have done something wrong during the setup. I'll explain how I set it up.<br>
</div>
<div class="ContentPasted2 elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
I installed mvapich2 as follows:</div>
<blockquote style="margin-top:0;margin-bottom:0">
<div class="ContentPasted2 elementToProof ContentPasted3" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
wget https://mvapich.cse.ohio-state.edu/download/mvapich/mv2/mvapich2-2.3.7.tar.gz
<div class="ContentPasted3">gzip -dc mvapich2-2.3.7.tar.gz | tar -x</div>
<div class="ContentPasted3">export LIBRARY_PATH="/home/bengt/nooby/baseline/install/lib"</div>
<div class="ContentPasted3">export CPPFLAGS="-I /home/bengt/nooby/baseline/install/include"</div>
<div class="ContentPasted3">export FFLAGS="-w -fallow-argument-mismatch -O2"</div>
<div class="ContentPasted3">export LDFLAGS="-L/home/bengt/nooby/baseline/install/lib -L/home/bengt/nooby/baseline/install/lib64 -Wl,-rpath,/home/bengt/nooby/baseline/install/lib -Wl,-rpath,/home/bengt/nooby/baseline/install/lib64"</div>
<div class="ContentPasted3">export LIBS="-libverbs"</div>
<div class="ContentPasted4">cd mvapich2-2.3.7<br class="ContentPasted3">
</div>
</div>
<div class="ContentPasted2 elementToProof ContentPasted3" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
./configure --with-ibverbs=/home/bengt/nooby/baseline/install --prefix=/home/bengt/nooby/baseline/install</div>
<div class="ContentPasted2 elementToProof ContentPasted3" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
make</div>
<div class="ContentPasted2 elementToProof ContentPasted3" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
make install</div>
</blockquote>
<div class="ContentPasted2 elementToProof ContentPasted3" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div class="ContentPasted2 elementToProof ContentPasted3" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Mvapich2 seems to run fine as the command</div>
<blockquote style="margin-top:0;margin-bottom:0">
<div class="ContentPasted2 elementToProof ContentPasted3 ContentPasted5" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
./mpirun_rsh -np 2 ffmk-n3 ffmk-3 ../libexec/osu-micro-benchmarks/mpi/collective/osu_alltoall<br class="ContentPasted5">
</div>
</blockquote>
<div class="ContentPasted2 elementToProof ContentPasted3 ContentPasted5" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
runs fine.</div>
<div class="ContentPasted2 elementToProof ContentPasted3 ContentPasted5" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div class="ContentPasted2 elementToProof ContentPasted3 ContentPasted5" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Mvapich2-j is also required. I installed this following <a href="http://mvapich.cse.ohio-state.edu/userguide/mv2j/" title="http://mvapich.cse.ohio-state.edu/userguide/mv2j/" data-loopstyle="linkonly" id="LPlnk672729" class="OWAAutoLink">
this user guide</a>.</div>
<div class="ContentPasted2 elementToProof ContentPasted3 ContentPasted5" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
As java was missing I installed java 11.<br>
<blockquote style="margin-top:0;margin-bottom:0" class="ContentPasted6">wget <a href="https://urldefense.com/v3/__https://download.java.net/openjdk/jdk11/ri/openjdk-11*28_linux-x64_bin.tar.gz__;Kw!!KGKeukY!1O8XjuysQLdi7c6JzYF2Zw2YcSnphLcw9QP-DKnQ-HnKS6fJMaN6sUZfl_YFva3lfE8wDWDs-5s8vdO-mHM-oR2dUgQ6SIyl4O6Q-OKaYsuZWA$">https://download.java.net/openjdk/jdk11/ri/openjdk-11+28_linux-x64_bin.tar.gz</a>
<div class="ContentPasted6">gzip -dc openjdk-11+28_linux-x64_bin.tar.gz | tar -x</div>
</blockquote>
<div>I tried java 19 first which ran into an error and then I read online only 8 and 11 work.</div>
<div><br>
</div>
<div>Ant was also missing:</div>
<blockquote style="margin-top:0;margin-bottom:0">
<div class="ContentPasted7">wget <a href="https://urldefense.com/v3/__https://dlcdn.apache.org/*ant/binaries/apache-ant-1.10.14-bin.tar.gz__;Lw!!KGKeukY!1O8XjuysQLdi7c6JzYF2Zw2YcSnphLcw9QP-DKnQ-HnKS6fJMaN6sUZfl_YFva3lfE8wDWDs-5s8vdO-mHM-oR2dUgQ6SIyl4O6Q-OJDtBaxUw$">https://dlcdn.apache.org//ant/binaries/apache-ant-1.10.14-bin.tar.gz</a>
<div class="ContentPasted7">gzip -dc apache-ant-1.10.14-bin.tar.gz | tar -x</div>
</div>
</blockquote>
<div class="ContentPasted7"><br>
</div>
<div class="ContentPasted7">And I got mvapich2-j from here:</div>
<blockquote style="margin-top:0;margin-bottom:0">
<div class="ContentPasted7 ContentPasted8">wget http://mvapich.cse.ohio-state.edu/download/mvapich/mv2j/mvapich2-j-2.3.7.tar.gz
<div class="ContentPasted8">gzip -dc mvapich2-j-2.3.7.tar.gz | tar -x</div>
</div>
</blockquote>
<div class="ContentPasted7 ContentPasted8"><br>
</div>
<div class="ContentPasted7 ContentPasted8">Running ant:</div>
<blockquote style="margin-top:0;margin-bottom:0">
<div class="ContentPasted7 ContentPasted8 ContentPasted9">~/tmp/mvapich2-j-2.3.7/src/java$ $ANT_HOME/bin/ant
<div class="ContentPasted9">Buildfile: /home/bengt/tmp/mvapich2-j-2.3.7/src/java/build.xml</div>
<div><br class="ContentPasted9">
</div>
<div class="ContentPasted9">compile:</div>
<div class="ContentPasted9"> [javac] /home/bengt/tmp/mvapich2-j-2.3.7/src/java/build.xml:18: warning: 'includeantruntime' was not set, defaulting to build.sysclasspath=last; set to false for repeatable builds</div>
<div class="ContentPasted9"> [javac] Compiling 24 source files</div>
<div><br class="ContentPasted9">
</div>
<div class="ContentPasted9">jars:</div>
<div class="ContentPasted9"> [jar] Building jar: /home/bengt/tmp/mvapich2-j-2.3.7/lib/mvapich2-j.jar</div>
<div><br class="ContentPasted9">
</div>
<div class="ContentPasted9">clean:</div>
<div><br class="ContentPasted9">
</div>
<div class="ContentPasted9">all:</div>
<div><br class="ContentPasted9">
</div>
<div class="ContentPasted9">BUILD SUCCESSFUL</div>
<div class="ContentPasted9">Total time: 1 second</div>
</div>
</blockquote>
<div class="ContentPasted7 ContentPasted8 ContentPasted9"><br>
</div>
<div class="ContentPasted7 ContentPasted8 ContentPasted9">Running make:</div>
<blockquote style="margin-top:0;margin-bottom:0">
<div class="ContentPasted7 ContentPasted8 ContentPasted9 ContentPasted10">~/tmp/mvapich2-j-2.3.7/src/c$ make
<div class="ContentPasted10">/home/bengt/tmp/jdk-11/bin/javac -cp /home/bengt/tmp/mvapich2-j-2.3.7/lib/mvapich2-j.jar:. -h . /home/bengt/tmp/mvapich2-j-2.3.7/src/java/mpi/*.java</div>
<div class="ContentPasted10">Note: Some input files use or override a deprecated API.</div>
<div class="ContentPasted10">Note: Recompile with -Xlint:deprecation for details.</div>
<div class="ContentPasted10">Note: Some input files use unchecked or unsafe operations.</div>
<div class="ContentPasted10">Note: Recompile with -Xlint:unchecked for details.</div>
<div class="ContentPasted10">/home/bengt/nooby/baseline/install/bin/mpicc -O3 -o libmvapich2-j.so -shared -fPIC -Wl,-soname -Wl,-rpath -I/home/bengt/tmp/jdk-11/include -I/home/bengt/tmp/jdk-11/include/linux -I/home/bengt/nooby/baseline/install/include -L/usr/lib64
*.c</div>
<div class="ContentPasted10">mpi_Group.c: In function ‘Java_mpi_Group_nativeSize’:</div>
<div class="ContentPasted10">mpi_Group.c:150:9: warning: implicit declaration of function ‘println’; did you mean ‘printf’? [-Wimplicit-function-declaration]</div>
<div class="ContentPasted10"> 150 | println("ERROR: nativeSize for Group failed.");</div>
<div class="ContentPasted10"> | ^~~~~~~</div>
<div class="ContentPasted10"> | printf</div>
<div class="ContentPasted10">mpi_Intracomm.c: In function ‘Java_mpi_Intracomm_nativeSplit’:</div>
<div class="ContentPasted10">mpi_Intracomm.c:424:9: warning: implicit declaration of function ‘println’; did you mean ‘printf’? [-Wimplicit-function-declaration]</div>
<div class="ContentPasted10"> 424 | println("ERROR: Could not split communicator.");</div>
<div class="ContentPasted10"> | ^~~~~~~</div>
<div class="ContentPasted10"> | printf</div>
<div class="ContentPasted10">mv libmvapich2-j.so /home/bengt/tmp/mvapich2-j-2.3.7/lib</div>
<div class="ContentPasted10">/sbin/ldconfig -n /home/bengt/tmp/mvapich2-j-2.3.7/lib</div>
</div>
</blockquote>
<div class="ContentPasted7 ContentPasted8 ContentPasted9 ContentPasted10"><br>
</div>
<div class="ContentPasted7 ContentPasted8 ContentPasted9 ContentPasted10">Then I get mpi4spark:</div>
<blockquote style="margin-top:0;margin-bottom:0">
<div class="ContentPasted7 ContentPasted8 ContentPasted9 ContentPasted10 ContentPasted11">
wget http://hibd.cse.ohio-state.edu/download/hibd/mpi4spark-0.2-x86-bin.tar.gz
<div class="ContentPasted11">gzip -dc mpi4spark-0.2-x86-bin.tar.gz | tar -x</div>
</div>
</blockquote>
<div class="ContentPasted7 ContentPasted8 ContentPasted9 ContentPasted10 ContentPasted11">
<br>
</div>
<div class="ContentPasted7 ContentPasted8 ContentPasted9 ContentPasted10 ContentPasted11">
and copy the mvapich2 jar:</div>
<blockquote style="margin-top:0;margin-bottom:0">
<div class="ContentPasted7 ContentPasted8 ContentPasted9 ContentPasted10 ContentPasted11 ContentPasted12">
cp $MV2J_HOME/lib/mvapich2-j.jar $SPARK_HOME/jars<br class="ContentPasted12">
</div>
</blockquote>
<div class="ContentPasted7 ContentPasted8 ContentPasted9 ContentPasted10 ContentPasted11 ContentPasted12">
<br>
</div>
<div class="ContentPasted7 ContentPasted8 ContentPasted9 ContentPasted10 ContentPasted11 ContentPasted12">
The used environment variables are:</div>
<blockquote style="margin-top:0;margin-bottom:0">
<div class="ContentPasted7 ContentPasted8 ContentPasted9 ContentPasted10 ContentPasted11 ContentPasted12 ContentPasted13">
export JAVA_HOME=/home/bengt/tmp/jdk-11
<div class="ContentPasted13">export MPILIB=/home/bengt/nooby/baseline/install</div>
<div class="ContentPasted13">export MV2J_HOME=/home/bengt/tmp/mvapich2-j-2.3.7</div>
<div class="ContentPasted13">export ANT_HOME=/home/bengt/tmp/apache-ant-1.10.14</div>
<div class="ContentPasted13">export HADOOP_HOME=/home/bengt/tmp/hadoop-3.3.4</div>
<div class="ContentPasted13">export SPARK_HOME=/home/bengt/tmp/mpi4spark-0.2-x86-bin</div>
<div class="ContentPasted13">export MV2_ENABLE_AFFINITY=0</div>
</div>
</blockquote>
<div class="ContentPasted7 ContentPasted8 ContentPasted9 ContentPasted10 ContentPasted11 ContentPasted12 ContentPasted13">
<br>
<br>
Then I am following the mpi4spark guide for the GroupByTest Benchmark:</div>
<div class="ContentPasted7 ContentPasted8 ContentPasted9 ContentPasted10 ContentPasted11 ContentPasted12 ContentPasted13">
The hostfile looks like this:</div>
<blockquote style="margin-top:0;margin-bottom:0">
<div class="ContentPasted7 ContentPasted8 ContentPasted9 ContentPasted10 ContentPasted11 ContentPasted12 ContentPasted13 ContentPasted14">
bengt@ffmk-n3:~/tmp/mpi4spark-0.2-x86-bin$ cat hostfile
<div class="ContentPasted14">ffmk-n3</div>
</div>
</blockquote>
<div class="ContentPasted14">As you can see there is only one node because I am trying to work out the installation process manually before writing a script to deploy the installation to multiple nodes.</div>
<div class="ContentPasted14"><br>
</div>
<div class="ContentPasted14">app.sh:</div>
<blockquote style="margin-top:0;margin-bottom:0">
<div class="ContentPasted14 ContentPasted15">bengt@ffmk-n3:~/tmp/mpi4spark-0.2-x86-bin$ cat app.sh<br class="ContentPasted15">
</div>
<div class="ContentPasted14 ContentPasted15">./bin/spark-submit --master spark://$1:7077 --class org.apache.spark.examples.GroupByTest examples/jars/spark-examples_2.12-3.3.0-SNAPSHOT.jar</div>
</blockquote>
<div class="ContentPasted14 ContentPasted15"><br>
</div>
<div class="ContentPasted14 ContentPasted15">spark-env.sh:</div>
<div class="ContentPasted14 ContentPasted15"><span> < default comments at the beginning ></span><br>
</div>
<blockquote style="margin-top:0;margin-bottom:0">export SPARK_USE_MPI=1
<div class="ContentPasted14 ContentPasted15 ContentPasted16">
<div class="ContentPasted16">export JAVA_HOME=/home/bengt/tmp/jdk-11</div>
<div class="ContentPasted16">export MPILIB=/home/bengt/nooby/baseline/install</div>
<div class="ContentPasted16">export MV2J_HOME=/home/bengt/tmp/mvapich2-j-2.3.7</div>
<div class="ContentPasted16">export ANT_HOME=/home/bengt/tmp/apache-ant-1.10.14</div>
<div class="ContentPasted16">export HADOOP_HOME=/home/bengt/tmp/hadoop-3.3.4</div>
<div class="ContentPasted16">export SPARK_HOME=/home/bengt/tmp/mpi4spark-0.2-x86-bin</div>
<div class="ContentPasted16">export SPARK_NO_DAEMONIZE=1</div>
<div class="ContentPasted16">export JAVA_LIBRARY_PATH=$JAVA_LIBRARY_PATH:$MV2J_HOME</div>
<div class="ContentPasted16">export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$MV2J_HOME/lib</div>
<div class="ContentPasted16">export JAVA_BINARY=/home/bengt/tmp/jdk-11/bin/java</div>
<div class="ContentPasted16">export SPARK_LIBRARY_PATH=$MV2J_HOME/lib</div>
<div class="ContentPasted16">export WORK_DIR=$SPARK_HOME/exec-wdir</div>
</div>
</blockquote>
<div class="ContentPasted14 ContentPasted15 ContentPasted16"><br>
</div>
<div class="ContentPasted14 ContentPasted15 ContentPasted16">spark-default.conf:</div>
<blockquote style="margin-top:0;margin-bottom:0">
<div class="ContentPasted14 ContentPasted15 ContentPasted16 ContentPasted17">bengt@ffmk-n3:~/tmp/mpi4spark-0.2-x86-bin$ cat conf/spark-defaults.conf</div>
<div class="ContentPasted14 ContentPasted15 ContentPasted16 ContentPasted17"><span class="ContentPasted18">< default comments at the beginning ></span><br>
</div>
<div class="ContentPasted14 ContentPasted15 ContentPasted16 ContentPasted17">
<div class="ContentPasted17">spark.executor.extraJavaOptions -Djava.library.path=/home/bengt/tmp/mvapich2-j-2.3.7/lib</div>
</div>
</blockquote>
<div class="ContentPasted14 ContentPasted15 ContentPasted16 ContentPasted17"><br>
</div>
<div class="ContentPasted14 ContentPasted15 ContentPasted16 ContentPasted17">After this i run:<br>
<blockquote style="margin-top:0;margin-bottom:0">./sbin/start-mpi4spark.sh<br>
</blockquote>
<div><br>
</div>
<div>This results in the app.log and exec.log above.</div>
<div><br>
</div>
<div>Trying to figure out what is wrong I also tried running the master and worker manually.<br>
<blockquote style="margin-top:0;margin-bottom:0" class="ContentPasted19">bengt@ffmk-n3:~/tmp/mpi4spark-0.2-x86-bin$ ./sbin/start-master.sh
<div class="ContentPasted19">starting org.apache.spark.deploy.master.Master, logging to /home/bengt/tmp/mpi4spark-0.2-x86-bin/logs/spark-bengt-org.apache.spark.deploy.master.Master-1-ffmk-n3.out</div>
<div class="ContentPasted19">Spark Command: /home/bengt/tmp/jdk-11/bin/java -cp /home/bengt/tmp/mpi4spark-0.2-x86-bin/conf/:/home/bengt/tmp/mpi4spark-0.2-x86-bin/jars/* -Xmx1g org.apache.spark.deploy.master.Master --host <ffmk3-url> --port 7077 --webui-port
8080</div>
<div class="ContentPasted19">========================================</div>
<div class="ContentPasted19">Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties</div>
<div class="ContentPasted19">23/09/24 14:09:23 INFO Master: Started daemon with process name: 2847@ffmk-n3</div>
<div class="ContentPasted19">23/09/24 14:09:23 INFO SignalUtils: Registering signal handler for TERM</div>
<div class="ContentPasted19">23/09/24 14:09:23 INFO SignalUtils: Registering signal handler for HUP</div>
<div class="ContentPasted19">23/09/24 14:09:23 INFO SignalUtils: Registering signal handler for INT</div>
<div class="ContentPasted19">23/09/24 14:09:23 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable</div>
<div class="ContentPasted19">23/09/24 14:09:23 INFO SecurityManager: Changing view acls to: bengt</div>
<div class="ContentPasted19">23/09/24 14:09:23 INFO SecurityManager: Changing modify acls to: bengt</div>
<div class="ContentPasted19">23/09/24 14:09:23 INFO SecurityManager: Changing view acls groups to:
</div>
<div class="ContentPasted19">23/09/24 14:09:23 INFO SecurityManager: Changing modify acls groups to:
</div>
<div class="ContentPasted19">23/09/24 14:09:23 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(bengt); groups with view permissions: Set(); users with modify permissions: Set(bengt); groups
with modify permissions: Set()</div>
<div class="ContentPasted19">[ffmk-n3:mpi_rank_0][MPID_Init] [Performance Suggestion]: Application has requested for multi-thread capability. If allocating memory from different pthreads/OpenMP threads, please consider setting MV2_USE_ALIGNED_ALLOC=1 for improved
performance.</div>
<div class="ContentPasted19">Use MV2_USE_THREAD_WARNING=0 to suppress this error message.</div>
<div class="ContentPasted19">23/09/24 14:09:24 INFO NioEventLoop: Starting MPI4Spark.</div>
<div class="ContentPasted19">23/09/24 14:09:24 INFO Utils: Successfully started service 'sparkMaster' on port 7077.</div>
<div class="ContentPasted19">23/09/24 14:09:24 INFO Master: Starting Spark master at spark://<ffmk3-u>:7077</div>
<div class="ContentPasted19">23/09/24 14:09:24 INFO Master: Running Spark version 3.3.0-SNAPSHOT</div>
<div class="ContentPasted19">23/09/24 14:09:24 INFO Utils: Successfully started service 'MasterUI' on port 8080.</div>
<div class="ContentPasted19">23/09/24 14:09:24 INFO MasterWebUI: Bound MasterWebUI to 0.0.0.0, and started at <a href="https://urldefense.com/v3/__http://*3Cffmk3-url*3E:-1__;JSU!!KGKeukY!1O8XjuysQLdi7c6JzYF2Zw2YcSnphLcw9QP-DKnQ-HnKS6fJMaN6sUZfl_YFva3lfE8wDWDs-5s8vdO-mHM-oR2dUgQ6SIyl4O6Q-OJJ7-idFg$">http://<ffmk3-url>:-1</a></div>
<div class="ContentPasted19">23/09/24 14:09:24 INFO Master: I have been elected leader! New state: ALIVE</div>
<br>
</blockquote>
</div>
</div>
</div>
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof ContentPasted0 ContentPasted1 ContentPasted2">
When I try to telnet into the master:</div>
<blockquote style="margin-top:0;margin-bottom:0">
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof ContentPasted0 ContentPasted1 ContentPasted2 ContentPasted20 ContentPasted27">
bengt@ffmk-n3:~/tmp/mpi4spark-0.2-x86-bin$ telnet<ffmk3-url> 7077
<div class="ContentPasted20">Trying 141.76.48.47...</div>
<div class="ContentPasted20 ContentPasted32">Connected to <ffmk3-url>.</div>
<div class="ContentPasted20">Escape character is '^]'.</div>
<div class="ContentPasted20">Connection closed by foreign host.</div>
</div>
</blockquote>
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof ContentPasted0 ContentPasted1 ContentPasted2 ContentPasted20">
<br>
</div>
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof ContentPasted0 ContentPasted1 ContentPasted2 ContentPasted20">
And on the running master process appears:</div>
<blockquote style="margin-top:0;margin-bottom:0">
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof ContentPasted0 ContentPasted1 ContentPasted2 ContentPasted20 ContentPasted21">
23/09/24 14:11:43 WARN NioServerSocketChannel: Failed to create a new channel from an accepted socket.
<div class="ContentPasted21">java.lang.NullPointerException</div>
<div class="ContentPasted21"> at java.base/java.lang.String.contains(String.java:2036)</div>
<div class="ContentPasted21"> at io.netty.channel.socket.nio.NioServerSocketChannel.doReadMessages(NioServerSocketChannel.java:184)</div>
<div class="ContentPasted21"> at io.netty.channel.nio.AbstractNioMessageChannel$NioMessageUnsafe.read(AbstractNioMessageChannel.java:85)</div>
<div class="ContentPasted21"> at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:802)</div>
<div class="ContentPasted21"> at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:737)</div>
<div class="ContentPasted21"> at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:661)</div>
<div class="ContentPasted21"> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:552)</div>
<div class="ContentPasted21"> at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)</div>
<div class="ContentPasted21"> at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)</div>
<div class="ContentPasted21"> at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)</div>
<div class="ContentPasted21"> at java.base/java.lang.Thread.run(Thread.java:834)</div>
<div><br class="ContentPasted21">
</div>
</div>
</blockquote>
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof ContentPasted0 ContentPasted1 ContentPasted2 ContentPasted20 ContentPasted21">
And running a worker:</div>
<blockquote style="margin-top:0;margin-bottom:0">
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof ContentPasted0 ContentPasted1 ContentPasted2 ContentPasted20 ContentPasted21 ContentPasted22 ContentPasted28">
bengt@ffmk-n3:~/tmp/mpi4spark-0.2-x86-bin$ ./sbin/start-worker.sh spark://<ffmk3-url>:7077</div>
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof ContentPasted0 ContentPasted1 ContentPasted2 ContentPasted20 ContentPasted21 ContentPasted22">
<div class="ContentPasted22">starting org.apache.spark.deploy.worker.Worker, logging to /home/bengt/tmp/mpi4spark-0.2-x86-bin/logs/spark-bengt-org.apache.spark.deploy.worker.Worker-1-ffmk-n3.out</div>
<div class="ContentPasted22 ContentPasted33">Spark Command: /home/bengt/tmp/jdk-11/bin/java -cp /home/bengt/tmp/mpi4spark-0.2-x86-bin/conf/:/home/bengt/tmp/mpi4spark-0.2-x86-bin/jars/* -Xmx1g org.apache.spark.deploy.worker.Worker --webui-port 8081 spark://<ffmk3-url>:7077</div>
<div class="ContentPasted22">========================================</div>
<div class="ContentPasted22">Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties</div>
<div class="ContentPasted22">23/09/24 14:12:43 INFO Worker: Started daemon with process name: 2962@ffmk-n3</div>
<div class="ContentPasted22">23/09/24 14:12:43 INFO SignalUtils: Registering signal handler for TERM</div>
<div class="ContentPasted22">23/09/24 14:12:43 INFO SignalUtils: Registering signal handler for HUP</div>
<div class="ContentPasted22">23/09/24 14:12:43 INFO SignalUtils: Registering signal handler for INT</div>
<div class="ContentPasted22">23/09/24 14:12:43 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable</div>
<div class="ContentPasted22">23/09/24 14:12:43 INFO SecurityManager: Changing view acls to: bengt</div>
<div class="ContentPasted22">23/09/24 14:12:43 INFO SecurityManager: Changing modify acls to: bengt</div>
<div class="ContentPasted22">23/09/24 14:12:43 INFO SecurityManager: Changing view acls groups to:
</div>
<div class="ContentPasted22">23/09/24 14:12:43 INFO SecurityManager: Changing modify acls groups to:
</div>
<div class="ContentPasted22">23/09/24 14:12:43 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(bengt); groups with view permissions: Set(); users with modify permissions: Set(bengt); groups
with modify permissions: Set()</div>
<div class="ContentPasted22">[ffmk-n3:mpi_rank_0][MPID_Init] [Performance Suggestion]: Application has requested for multi-thread capability. If allocating memory from different pthreads/OpenMP threads, please consider setting MV2_USE_ALIGNED_ALLOC=1 for improved
performance.</div>
<div class="ContentPasted22">Use MV2_USE_THREAD_WARNING=0 to suppress this error message.</div>
<div class="ContentPasted22">23/09/24 14:12:44 ERROR NioEventLoop: ERROR: MPI_Init_thread did not reutrn MPI.THREAD_MULTIPLE</div>
<div class="ContentPasted22">23/09/24 14:12:44 INFO NioEventLoop: Starting MPI4Spark.</div>
<div class="ContentPasted22">23/09/24 14:12:44 INFO Utils: Successfully started service 'sparkWorker' on port 39987.</div>
<div class="ContentPasted22">23/09/24 14:12:44 INFO Worker: Worker decommissioning not enabled.</div>
<div class="ContentPasted22">23/09/24 14:12:44 INFO Worker: Starting Spark worker 141.76.48.47:39987 with 8 cores, 14.5 GiB RAM</div>
<div class="ContentPasted22">23/09/24 14:12:44 INFO Worker: Running Spark version 3.3.0-SNAPSHOT</div>
<div class="ContentPasted22">23/09/24 14:12:44 INFO Worker: Spark home: /home/bengt/tmp/mpi4spark-0.2-x86-bin</div>
<div class="ContentPasted22">23/09/24 14:12:44 INFO ResourceUtils: ==============================================================</div>
<div class="ContentPasted22">23/09/24 14:12:44 INFO ResourceUtils: No custom resources configured for spark.worker.</div>
<div class="ContentPasted22">23/09/24 14:12:44 INFO ResourceUtils: ==============================================================</div>
<div class="ContentPasted22">23/09/24 14:12:44 INFO Utils: Successfully started service 'WorkerUI' on port 8081.</div>
<div class="ContentPasted22 ContentPasted29">23/09/24 14:12:44 INFO WorkerWebUI: Bound WorkerWebUI to 0.0.0.0, and started at <a href="https://urldefense.com/v3/__http://*3Cffmk3-url*3E:-1__;JSU!!KGKeukY!1O8XjuysQLdi7c6JzYF2Zw2YcSnphLcw9QP-DKnQ-HnKS6fJMaN6sUZfl_YFva3lfE8wDWDs-5s8vdO-mHM-oR2dUgQ6SIyl4O6Q-OJJ7-idFg$">http://<ffmk3-url>:-1</a></div>
<div class="ContentPasted22 ContentPasted30">23/09/24 14:12:44 INFO Worker: Connecting to master <ffmk3-url>:7077...</div>
<div class="ContentPasted22">23/09/24 14:12:45 WARN ChannelInitializer: Failed to initialize a channel. Closing: [id: 0x32d38e92]</div>
<div class="ContentPasted22">java.lang.NullPointerException</div>
<div class="ContentPasted22"> at java.base/java.lang.String.contains(String.java:2036)</div>
<div class="ContentPasted22"> at org.apache.spark.network.TransportContext.initializePipeline(TransportContext.java:221)</div>
<div class="ContentPasted22"> at org.apache.spark.network.TransportContext.initializePipeline(TransportContext.java:194)</div>
<div class="ContentPasted22"> at org.apache.spark.network.client.TransportClientFactory$1.initChannel(TransportClientFactory.java:302)</div>
<div class="ContentPasted22"> at org.apache.spark.network.client.TransportClientFactory$1.initChannel(TransportClientFactory.java:299)</div>
<div class="ContentPasted22"> at io.netty.channel.ChannelInitializer.initChannel(ChannelInitializer.java:129)</div>
<div class="ContentPasted22"> at io.netty.channel.ChannelInitializer.handlerAdded(ChannelInitializer.java:112)</div>
<div class="ContentPasted22"> at io.netty.channel.AbstractChannelHandlerContext.callHandlerAdded(AbstractChannelHandlerContext.java:943)</div>
<div class="ContentPasted22"> at io.netty.channel.DefaultChannelPipeline.callHandlerAdded0(DefaultChannelPipeline.java:609)</div>
<div class="ContentPasted22"> at io.netty.channel.DefaultChannelPipeline.access$100(DefaultChannelPipeline.java:46)</div>
<div class="ContentPasted22"> at io.netty.channel.DefaultChannelPipeline$PendingHandlerAddedTask.execute(DefaultChannelPipeline.java:1463)</div>
<div class="ContentPasted22"> at io.netty.channel.DefaultChannelPipeline.callHandlerAddedForAllHandlers(DefaultChannelPipeline.java:1115)</div>
<div class="ContentPasted22"> at io.netty.channel.DefaultChannelPipeline.invokeHandlerAddedIfNeeded(DefaultChannelPipeline.java:650)</div>
<div class="ContentPasted22"> at io.netty.channel.AbstractChannel$AbstractUnsafe.register0(AbstractChannel.java:528)</div>
<div class="ContentPasted22"> at io.netty.channel.AbstractChannel$AbstractUnsafe.access$200(AbstractChannel.java:443)</div>
<div class="ContentPasted22"> at io.netty.channel.AbstractChannel$AbstractUnsafe$1.run(AbstractChannel.java:500)</div>
<div class="ContentPasted22"> at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)</div>
<div class="ContentPasted22"> at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:469)</div>
<div class="ContentPasted22"> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:559)</div>
<div class="ContentPasted22"> at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)</div>
<div class="ContentPasted22"> at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)</div>
<div class="ContentPasted22"> at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)</div>
<div class="ContentPasted22"> at java.base/java.lang.Thread.run(Thread.java:834)</div>
<div class="ContentPasted22 ContentPasted31">23/09/24 14:12:45 WARN Worker: Failed to connect to master <ffmk3-url>:7077</div>
<div class="ContentPasted22">org.apache.spark.SparkException: Exception thrown in awaitResult:
</div>
<div class="ContentPasted22"> at org.apache.spark.util.ThreadUtils$.awaitResult(ThreadUtils.scala:301)</div>
<div class="ContentPasted22"> at org.apache.spark.rpc.RpcTimeout.awaitResult(RpcTimeout.scala:75)</div>
<div class="ContentPasted22"> at org.apache.spark.rpc.RpcEnv.setupEndpointRefByURI(RpcEnv.scala:101)</div>
<div class="ContentPasted22"> at org.apache.spark.rpc.RpcEnv.setupEndpointRef(RpcEnv.scala:109)</div>
<div class="ContentPasted22"> at org.apache.spark.deploy.worker.Worker$$anon$1.run(Worker.scala:311)</div>
<div class="ContentPasted22"> at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)</div>
<div class="ContentPasted22"> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)</div>
<div class="ContentPasted22"> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)</div>
<div class="ContentPasted22"> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)</div>
<div class="ContentPasted22"> at java.base/java.lang.Thread.run(Thread.java:834)</div>
<div class="ContentPasted22 ContentPasted24">Caused by: java.io.IOException: Failed to connect to <ffmk3-url>:7077</div>
<div class="ContentPasted22"> at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:316)</div>
<div class="ContentPasted22"> at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:246)</div>
<div class="ContentPasted22"> at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:258)</div>
<div class="ContentPasted22"> at org.apache.spark.rpc.netty.NettyRpcEnv.createClient(NettyRpcEnv.scala:204)</div>
<div class="ContentPasted22"> at org.apache.spark.rpc.netty.Outbox$$anon$1.call(Outbox.scala:202)</div>
<div class="ContentPasted22"> at org.apache.spark.rpc.netty.Outbox$$anon$1.call(Outbox.scala:198)</div>
<div class="ContentPasted22"> ... 4 more</div>
<div class="ContentPasted22">Caused by: io.netty.channel.StacklessClosedChannelException</div>
<div class="ContentPasted22"> at io.netty.channel.AbstractChannel$AbstractUnsafe.ensureOpen(ChannelPromise)(Unknown Source)</div>
</div>
</blockquote>
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof ContentPasted0 ContentPasted1 ContentPasted2 ContentPasted20 ContentPasted21 ContentPasted22">
<br>
</div>
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof ContentPasted0 ContentPasted1 ContentPasted2 ContentPasted20 ContentPasted21 ContentPasted22">
<br>
I don't know if the example with the manual master and worker helps at all.<br>
The goal would be to be able to run the GroupByTest.<br>
Let me know if I left out any information that would be useful for you. <br>
</div>
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof ContentPasted0 ContentPasted1 ContentPasted2 ContentPasted20 ContentPasted21 ContentPasted22">
<br>
</div>
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof ContentPasted0 ContentPasted1 ContentPasted2 ContentPasted20 ContentPasted21 ContentPasted22">
Thanks in advance for your time and effort!<br>
<br>
Kind regards,<br>
Bengt<br>
</div>
<div>
<p style="font-size: 11pt; font-family: Calibri, sans-serif, serif, "EmojiFont"; margin: 0px;">
<br>
<br>
<b>Barkhausen Institut</b> <br>
<a href="https://urldefense.com/v3/__http://www.barkhauseninstitut.org__;!!KGKeukY!1O8XjuysQLdi7c6JzYF2Zw2YcSnphLcw9QP-DKnQ-HnKS6fJMaN6sUZfl_YFva3lfE8wDWDs-5s8vdO-mHM-oR2dUgQ6SIyl4O6Q-OKjL9s2DQ$">www.barkhauseninstitut.org</a> <br>
<br>
</p>
<p style="font-size: 11pt; font-family: Calibri, sans-serif, serif, "EmojiFont"; margin: 0px;">
<span style="font-size:9pt;" lang="de-DE">Barkhausen Institut gGmbH | Sitz: Würzburger Straße 46, 01187 Dresden, Germany | Registergericht: Amtsgericht Dresden, HRB 37267 | Geschäftsführer: Prof. Dr. Gerhard Fettweis, Dr. Tim Hentschel | Vorsitzender der Gesellschafterdelegation:
Jan Gerken <br>
<br>
Hinweise zum Datenschutz und zur Verarbeitung Ihrer Daten finden Sie unter: <a href="https://urldefense.com/v3/__https://barkhauseninstitut.org/data-privacy__;!!KGKeukY!1O8XjuysQLdi7c6JzYF2Zw2YcSnphLcw9QP-DKnQ-HnKS6fJMaN6sUZfl_YFva3lfE8wDWDs-5s8vdO-mHM-oR2dUgQ6SIyl4O6Q-OIATgIgWw$">https://barkhauseninstitut.org/data-privacy</a>
<br>
<br>
</span><span style="font-size:9pt;" lang="en-US">This email and any attachments are intended only for the person to whom this email is addressed and may contain confidential and/or privileged information. If you received this email in error, please do not disclose
the contents to anyone, but notify the sender by return email and delete this email (and any attachments) from your system. Information on data protection and processing of your personal information: <a href="https://urldefense.com/v3/__https://barkhauseninstitut.org/data-privacy__;!!KGKeukY!1O8XjuysQLdi7c6JzYF2Zw2YcSnphLcw9QP-DKnQ-HnKS6fJMaN6sUZfl_YFva3lfE8wDWDs-5s8vdO-mHM-oR2dUgQ6SIyl4O6Q-OIATgIgWw$">https://barkhauseninstitut.org/data-privacy</a>
<br>
<br>
</span></p>
</div>
</body>
</html>