[Hadoop-RDMA-discuss] RDMA Hadoop 1.1.0 on Mellanox ConnectX4 problems

[Mark Goddard]

Hi,

I'm trying to use the RDMA Hadoop 1.1.0 integrated package and having some
problems. Here's my environment:

Cluster: 1 master, 2 slaves
Servers: Dell PowerEdge R630
NICs: Mellanox ConnectX4 VPI, dual port
OS: CentOS 7.3
Kernel: 3.10.0-514.21.1.el7.x86_64
OFED: Mellanox OFED 4.0-2.0.0.1-rhel7.3-x86_64

The NICs are configured with port 0 in Infiniband mode and port 1 in
Ethernet mode. The ethernet port 1 is down. I've used a modified OpenStack
sahara service to

I can start an HDFS namenode on the master node successfully. However, as
soon as I start a datanode on one of the slaves, both the namenode and
datanode processes crash. There is nothing exceptional in the namenode or
datanode logs. Here are the final few lines of the namenode logs for
context:

2017-06-27 14:28:37,427 INFO org.apache.hadoop.ipc.RdmaServer: Starting
RDMAReader
2017-06-27 14:28:37,428 INFO org.apache.hadoop.ipc.RdmaServer: IPC
RdmaServer listener on 9000 with ctx id 0: starting
2017-06-27 14:28:37,430 INFO org.apache.hadoop.ipc.RdmaServer: IPC
RdmaServer handler 0 on 9000: starting
2017-06-27 14:28:37,431 INFO org.apache.hadoop.ipc.RdmaServer: IPC
RdmaServer handler 1 on 9000: starting
2017-06-27 14:28:37,431 INFO org.apache.hadoop.ipc.RdmaServer: IPC
RdmaServer handler 2 on 9000: starting
2017-06-27 14:28:37,432 INFO org.apache.hadoop.ipc.RdmaServer: IPC
RdmaServer handler 3 on 9000: starting
2017-06-27 14:28:37,433 INFO org.apache.hadoop.ipc.RdmaServer: IPC
RdmaServer handler 4 on 9000: starting
2017-06-27 14:28:37,433 INFO org.apache.hadoop.ipc.RdmaServer: IPC
RdmaServer handler 5 on 9000: starting
2017-06-27 14:28:37,435 INFO org.apache.hadoop.ipc.RdmaServer: IPC
RdmaServer handler 6 on 9000: starting
2017-06-27 14:28:37,435 INFO org.apache.hadoop.ipc.RdmaServer: IPC
RdmaServer handler 7 on 9000: starting
2017-06-27 14:28:37,436 INFO org.apache.hadoop.ipc.RdmaServer: IPC
RdmaServer handler 8 on 9000: starting
2017-06-27 14:28:37,437 INFO org.apache.hadoop.ipc.RdmaServer: IPC
RdmaServer handler 9 on 9000: starting
2017-06-27 14:28:37,437 INFO
org.apache.hadoop.hdfs.server.namenode.NameNode: NameNode RPC up at:
vanilla-2-7-3-centos7-vanilla-2-7-3-master-0/10.0.0.1:9000
2017-06-27 14:28:37,437 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Starting services
required for active state
2017-06-27 14:28:37,444 INFO
org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor:
Starting CacheReplicationMonitor with interval 30000 milliseconds

Both the namenode and datanode print the following message on termination:

Could not modify qp to RTR
 at line 153 in file ucr_ep.c
, -c) 0

I ran the namenode under strace with the -f argument to trace child
threads. Here are the final few syscalls from the thread that terminates
first:

bind(210, {sa_family=AF_INET, sin_port=htons(9000),
sin_addr=inet_addr("0.0.0.0")}, 16) = 0
listen(210, 5)                          = 0
accept(210, 0, NULL)                    = 211
read(211,
"\0\0\0\0\0\0\0\0\0\360\366\361\204\177\0\0\277\1\0\0\0\0\0\0\0\360\366\361\204\177\0\0"...,
48) = 48
write(211,
"\0\0\0\0\0\0\0\0YZ\275\350\200\177\0\0|\"\0\0\0\0\0\0\355\266\332\350\200\177\0\0"...,
48) = 48
getpeername(211, {sa_family=AF_INET, sin_port=htons(41348),
sin_addr=inet_addr("10.0.0.2")}, [16]) = 0
write(207,
"\32\0\0\0\36\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"...,
120) = -1 EHOSTUNREACH (No route to host)
write(2, "Could not modify qp to RTR\n", 27) = 27
write(2, " at line 153 in file ucr_ep.c\n", 30) = 30
exit_group(-4)                          = ?
+++ exited with 252 +++

Looking further up the trace we can see what that file descriptor
references:

open("/dev/infiniband/uverbs1", O_RDWR) = 207

So it appears to be using port 1, the ethernet mode port which is down,
rather than infiniband mode port 0 which is up.

Could anyone help with this? Is there a way to disable the ethernet port 1,
or to prevent RDMA hadoop from using it?

Thanks in advance,
Mark
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.cse.ohio-state.edu/pipermail/rdma-hadoop-discuss/attachments/20170627/dc62f579/attachment.html>