org.apache.hadoop.hbase.DoNotRetryIOException: Compression algorithm 'snappy' previously failed test
2018-01-16 14:42
6004 查看
执行命令
hbase提示下面的异常
如果我执行
执行
按照Hbase 笔记(11) 性能优化中在hbase-site.xml添加
hbase的异常如下:
最终找到了解决办法,在$HBASE_HOME/lib/native中创建软链接
并且注意环境变量中添加
create 'iw:test11',{NAME=>'i',VERSIONS=>1,COMPRESSION=>'SNAPPY'}
hbase提示下面的异常
2018-01-16 14:07:43,308 INFO [RS_OPEN_REGION-dashuju172:16020-0] coordination.ZkOpenRegionCoordination: Opening of region {ENCODED => 2c52b5a2d05ba1abe8503c904c1f54d9, NAME => 'iw:test11,,1516082862364.2c52b5a2d05ba1abe8503c904c1f54d9.', STARTKEY => '', ENDKEY => ''} failed, transitioning from OPENING to FAILED_OPEN in ZK, expecting version 4 2018-01-16 14:07:43,377 INFO [PriorityRpcServer.handler=16,queue=0,port=16020] regionserver.RSRpcServices: Open iw:test11,,1516082862364.2c52b5a2d05ba1abe8503c904c1f54d9. 2018-01-16 14:07:43,398 ERROR [RS_OPEN_REGION-dashuju172:16020-1] handler.OpenRegionHandler: Failed open of region=iw:test11,,1516082862364.2c52b5a2d05ba1abe8503c904c1f54d9., starting to roll back the global memstore size. org.apache.hadoop.hbase.DoNotRetryIOException: Compression algorithm 'snappy' previously failed test. at org.apache.hadoop.hbase.util.CompressionTest.testCompression(CompressionTest.java:91) at org.apache.hadoop.hbase.regionserver.HRegion.checkCompressionCodecs(HRegion.java:6300) at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6251) at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6218) at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6189) at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6145) at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6096) at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:362) at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:129) at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:129) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745)
如果我执行
create 'iw:test12',{NAME=>'i',VERSIONS=>1},日志是这样的
2018-01-16 14:08:18,432 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=455.85 KB, freeSize=370.80 MB, max=371.25 MB, blockCount=10, accesses=190, hits=180, hitRatio=94.74%, , cachingAccesses=190, cachingHits=180, cachingHitsRatio=94.74%, evictions=209, evicted=0, evictedPerRun=0.0 2018-01-16 14:09:55,773 INFO [PriorityRpcServer.handler=8,queue=0,port=16020] regionserver.RSRpcServices: Open iw:test12,,1516082994955.18e0a7886db8f6a37f55a1c29f1f34fd. 2018-01-16 14:09:55,814 INFO [StoreOpener-18e0a7886db8f6a37f55a1c29f1f34fd-1] hfile.CacheConfig: blockCache=LruBlockCache{blockCount=10, currentSize=466792, freeSize=388817048, maxSize=389283840, heapSize=466792, minSize=369819648, minFactor=0.95, multiSize=184909824, multiFactor=0.5, singleSize=92454912, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=true, prefetchOnOpen=false 2018-01-16 14:09:55,815 INFO [StoreOpener-18e0a7886db8f6a37f55a1c29f1f34fd-1] compactions.CompactionConfiguration: size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000 2018-01-16 14:09:55,863 INFO [RS_OPEN_REGION-dashuju172:16020-2] regionserver.HRegion: Onlined 18e0a7886db8f6a37f55a1c29f1f34fd; next sequenceid=2 2018-01-16 14:09:55,868 WARN [sync.0] wal.FSHLog: Too many consecutive RollWriter requests, it's a sign of the total number of live datanodes is lower than the tolerable replicas. 2018-01-16 14:09:55,870 INFO [PostOpenDeployTasks:18e0a7886db8f6a37f55a1c29f1f34fd] regionserver.HRegionServer: Post open deploy tasks for iw:test12,,1516082994955.18e0a7886db8f6a37f55a1c29f1f34fd. 2018-01-16 14:09:55,875 INFO [PostOpenDeployTasks:18e0a7886db8f6a37f55a1c29f1f34fd] hbase.MetaTableAccessor: Updated row iw:test12,,1516082994955.18e0a7886db8f6a37f55a1c29f1f34fd. with server=dashuju172,16020,1516080795128
执行
hadoop checknative,snappy是支持的。
18/01/16 14:23:42 WARN bzip2.Bzip2Factory: Failed to load/initialize native-bzip2 library system-native, will use pure-Java version 18/01/16 14:23:42 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library Native library checking: hadoop: true /home/hadoop/application/hadoop-2.6.4/lib/native/libhadoop.so.1.0.0 zlib: true /lib64/libz.so.1 snappy: true /home/hadoop/application/hadoop-2.6.4/lib/native/libsnappy.so.1 lz4: true revision:99 bzip2: false openssl: true /usr/lib64/libcrypto.so
按照Hbase 笔记(11) 性能优化中在hbase-site.xml添加
<property> <name>hbase.regionserver.codecs</name> <value>snappy</value> </property>
hbase的异常如下:
2018-01-16 14:34:41,241 WARN [main] util.CompressionTest: Can't instantiate codec: snappy org.apache.hadoop.hbase.DoNotRetryIOException: java.lang.UnsatisfiedLinkError: org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy()Z at org.apache.hadoop.hbase.util.CompressionTest.testCompression(CompressionTest.java:102) at org.apache.hadoop.hbase.util.CompressionTest.testCompression(CompressionTest.java:68) at org.apache.hadoop.hbase.regionserver.HRegionServer.checkCodecs(HRegionServer.java:690) at org.apache.hadoop.hbase.regionserver.HRegionServer.<init>(HRegionServer.java:499) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at org.apache.hadoop.hbase.regionserver.HRegionServer.constructRegionServer(HRegionServer.java:2633) at org.apache.hadoop.hbase.regionserver.HRegionServerCommandLine.start(HRegionServerCommandLine.java:64) at org.apache.hadoop.hbase.regionserver.HRegionServerCommandLine.run(HRegionServerCommandLine.java:87) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:126) at org.apache.hadoop.hbase.regionserver.HRegionServer.main(HRegionServer.java:2650) Caused by: java.lang.UnsatisfiedLinkError: org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy()Z at org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy(Native Method) at org.apache.hadoop.io.compress.SnappyCodec.checkNativeCodeLoaded(SnappyCodec.java:63) at org.apache.hadoop.io.compress.SnappyCodec.getCompressorType(SnappyCodec.java:132) at org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:148) at org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:163) at org.apache.hadoop.hbase.io.compress.Compression$Algorithm.getCompressor(Compression.java:303) at org.apache.hadoop.hbase.util.CompressionTest.testCompression(CompressionTest.java:97) ... 13 more 2018-01-16 14:34:41,244 ERROR [main] regionserver.HRegionServerCommandLine: Region server exiting java.lang.RuntimeException: Failed construction of Regionserver: class org.apache.hadoop.hbase.regionserver.HRegionServer at org.apache.hadoop.hbase.regionserver.HRegionServer.constructRegionServer(HRegionServer.java:2635) at org.apache.hadoop.hbase.regionserver.HRegionServerCommandLine.start(HRegionServerCommandLine.java:64) at org.apache.hadoop.hbase.regionserver.HRegionServerCommandLine.run(HRegionServerCommandLine.java:87) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:126) at org.apache.hadoop.hbase.regionserver.HRegionServer.main(HRegionServer.java:2650) Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at org.apache.hadoop.hbase.regionserver.HRegionServer.constructRegionServer(HRegionServer.java:2633) ... 5 more Caused by: java.io.IOException: Compression codec snappy not supported, aborting RS construction at org.apache.hadoop.hbase.regionserver.HRegionServer.checkCodecs(HRegionServer.java:691) at org.apache.hadoop.hbase.regionserver.HRegionServer.<init>(HRegionServer.java:499) ... 10 more
最终找到了解决办法,在$HBASE_HOME/lib/native中创建软链接
ln -s /home/hadoop/application/hadoop/lib/native Linux-amd64-64
并且注意环境变量中添加
export HBASE_LIBRARY_PATH=/home/hadoop/application/hbase/lib/native/Linux-amd64-64
相关文章推荐
- 关于org.apache.hadoop.hbase.DoNotRetryIOException: java.lang.IllegalAccessError
- org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 6 actions: DoNotRetryIOE
- org.apache.hadoop.hbase.DoNotRetryIOException: Class org.apache.phoenix.coprocessor.MetaDataEndpointImpl cannot be loaded Set hbase.table.sanity.checks to false at conf or table descriptor if you want
- 解决hiveserver2报错:java.io.IOException: Job status not available - Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask
- org.apache.hadoop.hbase.NotServingRegionException: Region is not online 错误
- java.lang.ClassNotFoundException: org.apache.hadoop.hbase.mapreduce.TableOutputFormat解决办法
- spark1.4 读取hbase 0.96 报错 java.io.NotSerializableException: org.apache.hadoop.hbase.io.ImmutableBytes
- 删除hbase表时 org.apache.hadoop.hbase.TableNotDisabledException 解决方法
- org.apache.hadoop.hbase.TableNotFoundException: SYSTEM.CATALOG exception with phoenix 4.5.2
- hbase(ERROR: org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet)
- ERROR: org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet
- hbase无法启动Regionserver:Failed construction of Regionserver: class org.apache.hadoop.hbase.regions问题解决
- 使用 hdfs dfs -put 报错 org.apache.hadoop.ipc.RemoteException(java.io.IOException): File /test/a.txt._C
- org.apache.hadoop.hbase.TableNotDisabledException 解决方法
- org.apache.hadoop.hbase.TableNotFoundException: TRAFODION._DTM_.TDDL
- org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed setting up proxy to /10.10.11.184:6
- hadoop错误Ignoring exception during close for org.apache.hadoop.mapred.MapTask$NewOutputCollector@17bda0f2 java.io.IOException Spill failed
- ClassNotFoundException: org.apache.hadoop.hbase.ipc.controller.ServerRpcControllerFactory
- org.apache.hadoop.hbase.NotServingRegionException: Region is not online 错误