Hbase启动报错zookeeper.RecoverableZooKeeper: Node /hbase/rs/slave3

Hbase启动报错，如下：

2016-03-26 01:09:54,039 FATAL [regionserver60020] regionserver.HRegionServer: ABORTING region server slave3,60020,1458925764918: Unhandled: org.apache.hadoop.hbase.ClockOutOfSyncException: Server slave3,60020,1458925764918 has been rejected; Reported time is
too far out of sync with master.  Time difference of 28755135ms > max allowed of 30000ms

    at org.apache.hadoop.hbase.master.ServerManager.checkClockSkew(ServerManager.java:345)

    at org.apache.hadoop.hbase.master.ServerManager.regionServerStartup(ServerManager.java:238)

    at org.apache.hadoop.hbase.master.HMaster.regionServerStartup(HMaster.java:1294)

    at org.apache.hadoop.hbase.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:7910)

    at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2027)

    at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:108)

    at org.apache.hadoop.hbase.ipc.FifoRpcScheduler$1.run(FifoRpcScheduler.java:74)

    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)

    at java.util.concurrent.FutureTask.run(FutureTask.java:262)

    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

    at java.lang.Thread.run(Thread.java:745)

org.apache.hadoop.hbase.ClockOutOfSyncException: org.apache.hadoop.hbase.ClockOutOfSyncException: Server slave3,60020,1458925764918 has been rejected; Reported time is too far out of sync with master.  Time difference of 28755135ms > max allowed of 30000ms

    at org.apache.hadoop.hbase.master.ServerManager.checkClockSkew(ServerManager.java:345)

    at org.apache.hadoop.hbase.master.ServerManager.regionServerStartup(ServerManager.java:238)

    at org.apache.hadoop.hbase.master.HMaster.regionServerStartup(HMaster.java:1294)

    at org.apache.hadoop.hbase.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:7910)

    at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2027)

    at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:108)

    at org.apache.hadoop.hbase.ipc.FifoRpcScheduler$1.run(FifoRpcScheduler.java:74)

    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)

    at java.util.concurrent.FutureTask.run(FutureTask.java:262)

    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

    at java.lang.Thread.run(Thread.java:745)

    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)

    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)

    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)

    at java.lang.reflect.Constructor.newInstance(Constructor.java:526)

    at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)

    at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95)

    at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:284)

    at org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:2118)

    at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:884)

    at java.lang.Thread.run(Thread.java:745)

Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.ClockOutOfSyncException): org.apache.hadoop.hbase.ClockOutOfSyncException: Server slave3,60020,1458925764918 has been rejected; Reported time is too far out of sync with
master.  Time difference of 28755135ms > max allowed of 30000ms

    at org.apache.hadoop.hbase.master.ServerManager.checkClockSkew(ServerManager.java:345)

    at org.apache.hadoop.hbase.master.ServerManager.regionServerStartup(ServerManager.java:238)

    at org.apache.hadoop.hbase.master.HMaster.regionServerStartup(HMaster.java:1294)

    at org.apache.hadoop.hbase.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:7910)

    at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2027)

    at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:108)

    at org.apache.hadoop.hbase.ipc.FifoRpcScheduler$1.run(FifoRpcScheduler.java:74)

    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)

    at java.util.concurrent.FutureTask.run(FutureTask.java:262)

    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

    at java.lang.Thread.run(Thread.java:745)

    at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1457)

    at org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1661)

    at org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1719)

    at org.apache.hadoop.hbase.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$BlockingStub.regionServerStartup(RegionServerStatusProtos.java:8277)

    at org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:2116)

    ... 2 more

2016-03-26 01:09:54,049 FATAL [regionserver60020] regionserver.HRegionServer: RegionServer abort: loaded coprocessors are: []

2016-03-26 01:09:54,049 INFO  [regionserver60020] regionserver.HRegionServer: STOPPED: Unhandled: org.apache.hadoop.hbase.ClockOutOfSyncException: Server slave3,60020,1458925764918 has been rejected; Reported time is too far out of sync with master.  Time difference
of 28755135ms > max allowed of 30000ms

    at org.apache.hadoop.hbase.master.ServerManager.checkClockSkew(ServerManager.java:345)

    at org.apache.hadoop.hbase.master.ServerManager.regionServerStartup(ServerManager.java:238)

    at org.apache.hadoop.hbase.master.HMaster.regionServerStartup(HMaster.java:1294)

    at org.apache.hadoop.hbase.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:7910)

    at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2027)

    at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:108)

    at org.apache.hadoop.hbase.ipc.FifoRpcScheduler$1.run(FifoRpcScheduler.java:74)

    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)

    at java.util.concurrent.FutureTask.run(FutureTask.java:262)

    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

    at java.lang.Thread.run(Thread.java:745)

2016-03-26 01:09:54,049 INFO  [regionserver60020] ipc.RpcServer: Stopping server on 60020

2016-03-26 01:09:54,056 INFO  [regionserver60020] regionserver.HRegionServer: Stopping infoServer

2016-03-26 01:09:54,070 INFO  [regionserver60020] mortbay.log: Stopped SelectChannelConnector@0.0.0.0:60030

2016-03-26 01:09:54,076 INFO  [regionserver60020] snapshot.RegionServerSnapshotManager: Stopping RegionServerSnapshotManager abruptly.

2016-03-26 01:09:54,077 INFO  [regionserver60020] regionserver.HRegionServer: aborting server null

2016-03-26 01:09:54,077 DEBUG [regionserver60020] catalog.CatalogTracker: Stopping catalog tracker org.apache.hadoop.hbase.catalog.CatalogTracker@79373bbc

2016-03-26 01:09:54,077 INFO  [regionserver60020] client.HConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x353ae3663e70003

2016-03-26 01:09:54,098 INFO  [regionserver60020] zookeeper.ZooKeeper: Session: 0x353ae3663e70003 closed

2016-03-26 01:09:54,100 INFO  [regionserver60020-EventThread] zookeeper.ClientCnxn: EventThread shut down

2016-03-26 01:09:54,103 INFO  [regionserver60020] regionserver.HRegionServer: stopping server null; all regions closed.

2016-03-26 01:09:54,106 INFO  [regionserver60020] regionserver.Leases: regionserver60020 closing leases

2016-03-26 01:09:54,106 INFO  [regionserver60020] regionserver.Leases: regionserver60020 closed leases

2016-03-26 01:09:54,107 INFO  [regionserver60020] regionserver.CompactSplitThread: Waiting for Split Thread to finish...

2016-03-26 01:09:54,107 INFO  [regionserver60020] regionserver.CompactSplitThread: Waiting for Merge Thread to finish...

2016-03-26 01:09:54,107 INFO  [regionserver60020] regionserver.CompactSplitThread: Waiting for Large Compaction Thread to finish...

2016-03-26 01:09:54,107 INFO  [regionserver60020] regionserver.CompactSplitThread: Waiting for Small Compaction Thread to finish...

2016-03-26 01:09:54,154 WARN  [regionserver60020] zookeeper.RecoverableZooKeeper: Node /hbase/rs/slave3,60020,1458925764918 already deleted, retry=false

2016-03-26 01:09:54,154 WARN  [regionserver60020] regionserver.HRegionServer: Failed deleting my ephemeral node

org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /hbase/rs/slave3,60020,1458925764918

    at org.apache.zookeeper.KeeperException.create(KeeperException.java:111)

    at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)

    at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:873)

    at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.delete(RecoverableZooKeeper.java:179)

    at org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNode(ZKUtil.java:1273)

    at org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNode(ZKUtil.java:1262)

    at org.apache.hadoop.hbase.regionserver.HRegionServer.deleteMyEphemeralNode(HRegionServer.java:1342)

    at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:1054)

    at java.lang.Thread.run(Thread.java:745)

2016-03-26 01:09:54,249 INFO  [regionserver60020-EventThread] zookeeper.ClientCnxn: EventThread shut down

2016-03-26 01:09:54,249 INFO  [regionserver60020] zookeeper.ZooKeeper: Session: 0x453ae36623c0004 closed

2016-03-26 01:09:54,249 INFO  [regionserver60020] regionserver.HRegionServer: stopping server null; zookeeper connection closed.

2016-03-26 01:09:54,249 INFO  [regionserver60020] regionserver.HRegionServer: regionserver60020 exiting

2016-03-26 01:09:54,249 ERROR [main] regionserver.HRegionServerCommandLine: Region server exiting

java.lang.RuntimeException: HRegionServer Aborted

    at org.apache.hadoop.hbase.regionserver.HRegionServerCommandLine.start(HRegionServerCommandLine.java:66)

    at org.apache.hadoop.hbase.regionserver.HRegionServerCommandLine.run(HRegionServerCommandLine.java:85)

    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)

    at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:126)

    at org.apache.hadoop.hbase.regionserver.HRegionServer.main(HRegionServer.java:2503)

2016-03-26 01:09:54,253 INFO  [Thread-9] regionserver.ShutdownHook: Shutdown hook starting; hbase.shutdown.hook=true; fsShutdownHook=org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer@5d7f50ae

2016-03-26 01:09:54,254 INFO  [Thread-9] regionserver.ShutdownHook: Starting fs shutdown hook thread.

2016-03-26 01:09:54,329 INFO  [Thread-9] regionserver.ShutdownHook: Shutdown hook finished.

解决方法：

Hbase节点的服务器时间不一致，导致HRegionServer启动不了，将节点的服务器时间同步，之后启动正常了。