HBase中Regionserver与Master时间不一致导致RegionServer无法启动ClockOutOfSyncException

启动HBase的时候，无法启动RegionServer，查看日志，错误如下

2012-06-13 09:33:40,057 FATAL org.apache.hadoop.hbase.regionserver.HRegionServer: Master rejected startup because clock is out of sync
org.apache.hadoop.hbase.ClockOutOfSyncException: org.apache.hadoop.hbase.ClockOutOfSyncException: Server hadoop26,60020,1339551219741 has been rejected; Reported time is too far out of sync with master.  Time difference of 107126ms > max allowed of 30000ms
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:95)
at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:79)
at org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:1542)
at org.apache.hadoop.hbase.regionserver.HRegionServer.tryReportForDuty(HRegionServer.java:1499)
at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:572)
at java.lang.Thread.run(Thread.java:662)
Caused by: org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.hbase.ClockOutOfSyncException: Server hadoop26,60020,1339551219741 has been rejected; Reported time is too far out of sync with master.  Time difference of 107126ms > max allowed of 30000ms
at org.apache.hadoop.hbase.master.ServerManager.checkClockSkew(ServerManager.java:181)
at org.apache.hadoop.hbase.master.ServerManager.regionServerStartup(ServerManager.java:129)
at org.apache.hadoop.hbase.master.HMaster.regionServerStartup(HMaster.java:615)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1039)

at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:771)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257)
at $Proxy6.regionServerStartup(Unknown Source)
at org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:1538)
... 3 more
2012-06-13 09:33:40,066 FATAL org.apache.hadoop.hbase.regionserver.HRegionServer: ABORTING region server serverName=hadoop26,60020,1339551219741, load=(requests=0, regions=0, usedHeap=23, maxHeap=991): Unhandled exception: org.apache.hadoop.hbase.ClockOutOfSyncException: Server hadoop26,60020,1339551219741 has been rejected; Reported time is too far out of sync with master.  Time difference of 107126ms > max allowed of 30000ms
org.apache.hadoop.hbase.ClockOutOfSyncException: org.apache.hadoop.hbase.ClockOutOfSyncException: Server hadoop26,60020,1339551219741 has been rejected; Reported time is too far out of sync with master.  Time difference of 107126ms > max allowed of 30000ms
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:95)
at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:79)
at org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:1542)
at org.apache.hadoop.hbase.regionserver.HRegionServer.tryReportForDuty(HRegionServer.java:1499)
at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:572)
at java.lang.Thread.run(Thread.java:662)
Caused by: org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.hbase.ClockOutOfSyncException: Server hadoop26,60020,1339551219741 has been rejected; Reported time is too far out of sync with master.  Time difference of 107126ms > max allowed of 30000ms
at org.apache.hadoop.hbase.master.ServerManager.checkClockSkew(ServerManager.java:181)
at org.apache.hadoop.hbase.master.ServerManager.regionServerStartup(ServerManager.java:129)
at org.apache.hadoop.hbase.master.HMaster.regionServerStartup(HMaster.java:615)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1039)

at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:771)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257)
at $Proxy6.regionServerStartup(Unknown Source)
at org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:1538)
... 3 more
2012-06-13 09:33:40,066 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: STOPPED: Unhandled exception: org.apache.hadoop.hbase.ClockOutOfSyncException: Server hadoop26,60020,1339551219741 has been rejected; Reported time is too far out of sync with master.  Time difference of 107126ms > max allowed of 30000ms
2012-06-13 09:33:40,066 INFO org.apache.hadoop.ipc.HBaseServer: Stopping server on 60020
2012-06-13 09:33:40,069 INFO org.apache.hadoop.hbase.regionserver.StoreFile: Allocating LruBlockCache with maximum size 198.3m
2012-06-13 09:33:40,072 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: aborting server at: hadoop26,60020,1339551219741
2012-06-13 09:33:40,072 DEBUG org.apache.hadoop.hbase.catalog.CatalogTracker: Stopping catalog tracker org.apache.hadoop.hbase.catalog.CatalogTracker@2f60877b
2012-06-13 09:33:40,173 INFO org.apache.hadoop.hbase.regionserver.Leases: regionserver60020 closing leases
2012-06-13 09:33:40,173 INFO org.apache.hadoop.hbase.regionserver.Leases: regionserver60020 closed leases
2012-06-13 09:33:40,173 INFO org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: Closed zookeeper sessionid=0x137e373d9660018
2012-06-13 09:33:40,254 INFO org.apache.zookeeper.ZooKeeper: Session: 0x137e373d9660018 closed
2012-06-13 09:33:40,254 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down
2012-06-13 09:33:40,336 INFO org.apache.zookeeper.ZooKeeper: Session: 0x337d82dda1f110c closed
2012-06-13 09:33:40,336 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down
2012-06-13 09:33:40,362 INFO org.apache.hadoop.hbase.regionserver.ShutdownHook: Shutdown hook starting; hbase.shutdown.hook=true; fsShutdownHook=Thread[Thread-14,5,main]
2012-06-13 09:33:40,362 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: STOPPED: Shutdown hook
2012-06-13 09:33:40,362 INFO org.apache.hadoop.hbase.regionserver.ShutdownHook: Starting fs shutdown hook thread.
2012-06-13 09:33:40,363 INFO org.apache.hadoop.hbase.regionserver.ShutdownHook: Shutdown hook finished.

原来是RegionServer与Master的时间不一致造成的。由错误内容可以看出两台机器之间最大的误差时间为30000ms,一旦超过这个值便无法启动。

解决办法：同步RegionServer与Master的时间，最好周期性同步，具体方法请参考Linux时间同步设置