Hbase bulkLoad 批量入库遇到的问题及解决方法

1.2、BulkLoad 入库遇到问题及解决方法

1.2.1、首先就是reduce相关的问题:

  在实际的应用中你就会发现,对于稍大一点的数据量,map过程的执行效率还是比较让人满意的,但是到了reduce阶段就会出现比较严重的卡顿,我的困惑就是?我的代码里明明没有reduce过程,为什么还会有reduce过程来影响我入库的效率呢?  于是,我尝试着在job里,设置reduce的数量,把它设置为0,可是重新执行的时候就会发现还是会有那个烦人的reduce过程, 既然设置它为0没有效果,那我把它的数量设置的多一点,提高它的并行度总能加快效率了吧 于是我又修改了reduce的数量,执行的时候发现还是只有一个..... 后来我才知道, 在这种情况下,我们不用自己写reduce过程,但是会使用Hbase给我们提供的reduce,也就是说,无论你怎么设置reduce数量,都是无效的. 这样我也就释然了

1.2.2、效率严重低下的问题!!!

      首先我用100M的数据量做测试,居然需要30s才能入库完毕!用几个G的数据量测试,效率也没有明显的提升! 也就是说平均每秒的插入速度还不到15000条.,这甚至比mysql的入库还要慢很多,这种效率在实际生产中是完全不能接受的 说好的这是入库最快的方式呢?我不仅产生了怀疑.. 说到底,这种问题还是因为reduce数量只有一个这个蛋疼的问题所导致的,也就是说,不管你的集群有多牛,都值相当于单机版,这显然是不合适的...那么该如何解决这个问题呢????就是在建表的时候进行合理的预分区!!!预分区的数目会决定你的reduce过程的数目!简单来说,在一定的范围内,进行合适预分区的话,reduce的数量增加多少,效率就提高多少倍!!!    有关于hbase的预分区,进行合适的预分区,实际上是一个很复杂的问题,也不是本篇文章讨论的重点. 感兴趣的话可以去看看这位大神写的东西,给了我很大的启发大神的博客链接我只简单介绍一下hbase建表时预分区的shell语句和执行的结果:create 'XUE_BULKLOAD','info',{SPLITS => [ '1','2','3', '4','5','6','7','8','9']}这样就成功的将表名为 'XUE_BULKLOAD',列簇名为'info'的表在建表时预分了10个分区预分区结束之后进行测试:发现reduce的数量为预分区的数量+1,而且执行效率大大提高! 插入效率大致在10W/s~20W/s之间,已经勉强能达到实际工作的要求!

1.2.3、数据量超过某个范围就会导致插入数据库失败的问题!

经过各种各样的调试,效率已经可以接受! 然后开始调大数据量测试,发现哪怕几十个G的数据量,在执行完MapReduce过程之后都会报错,去表里面查看数据,一条记录都没有!!报错信息如下Trying toload more than 32 hfiles to one family of one region18/01/1823:20:36 ERROR mapreduce.LoadIncrementalHFiles: Trying to loadmore than 32 hfiles to family info of region with start key  Exception inthread "main" java.io.IOException: Trying to load morethan 32 hfiles to one family of one region         at org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.doBulkLoad         (LoadIncrementalHFiles.java:377)         at hbase_Insert.Hbase_Insert.main(Hbase_Insert.java:241)         at sun.reflect.NativeMethodAccessorImpl.invoke0(NativeMethod)         at sun.reflect.NativeMethodAccessorImpl.invoke(         NativeMethodAccessorImpl.java:57)         at sun.reflect.DelegatingMethodAccessorImpl.invoke(         DelegatingMethodAccessorImpl.java:43)         at java.lang.reflect.Method.invoke(Method.java:606)         at org.apache.hadoop.util.RunJar.run(RunJar.java:221) 
4000
        at org.apache.hadoop.util.RunJar.main(RunJar.java:136)报错的大致意思就是试图将超过32个Hfile文件导入到hbase里面的一个region导致失败 那这个问题该如何解决呢?实际上就是两个重要的参数限制的一个是:hbase.hregion.max.filesize单个ColumnFamily的region大小，若按照ConstantSizeRegionSplitPolicy策略，超过设置的该值则自动split 默认的大小是1G hbase.mapreduce.bulkload.max.hfiles.perRegion.perFamily允许的hfile的最大个数,默认配置是32 也就是说:这两个参数的默认值决定了,每次批量入库的数据量不能超过1*32也就是32个G,超过这个数量就会导致入库失败可以在代码里,或者在hbase安装路径下conf目录下的hbase-site.xml里面针对这两个参数进行设置为了一劳永逸,我选择在hbase-site.xml里面进行设置,设置结果如下:<property><name>hbase.hregion.max.filesize</name><value>10737418240</value></property><property><name>hbase.mapreduce.bulkload.max.hfiles.perRegion.perFamily</name><value>3200</value></property>这样,每次能够批量入库的数据就达到了32个T,符合公司的数据量需要! 配置完毕后重启集群进行测试,不在报这个错误,执行结果如下:18/01/2012:17:31 INFO mapreduce.Job: map 1% reduce 0%18/01/2012:17:35 INFO mapreduce.Job: map 2% reduce 0%18/01/2012:17:42 INFO mapreduce.Job: map 3% reduce 0%18/01/2012:17:45 INFO mapreduce.Job: map 4% reduce 0%18/01/2012:17:51 INFO mapreduce.Job: map 5% reduce 0%18/01/2012:17:55 INFO mapreduce.Job: map 6% reduce 0%18/01/2012:17:59 INFO mapreduce.Job: map 7% reduce 0%18/01/2012:18:03 INFO mapreduce.Job: map 8% reduce 0%18/01/2012:18:06 INFO mapreduce.Job: map 9% reduce 0%18/01/2012:18:11 INFO mapreduce.Job: map 10% reduce 0%18/01/2012:18:16 INFO mapreduce.Job: map 11% reduce 0%18/01/2012:18:20 INFO mapreduce.Job: map 12% reduce 0%18/01/2012:18:27 INFO mapreduce.Job: map 13% reduce 0%18/01/2012:18:32 INFO mapreduce.Job: map 14% reduce 0%18/01/2012:18:37 INFO mapreduce.Job: map 15% reduce 0%18/01/2012:18:42 INFO mapreduce.Job: map 16% reduce 0%18/01/2012:18:47 INFO mapreduce.Job: map 17% reduce 0%18/01/2012:18:53 INFO mapreduce.Job: map 18% reduce 0%18/01/2012:18:58 INFO mapreduce.Job: map 19% reduce 0%18/01/2012:19:03 INFO mapreduce.Job: map 20% reduce 0%18/01/2012:19:08 INFO mapreduce.Job: map 21% reduce 0%18/01/2012:19:14 INFO mapreduce.Job: map 22% reduce 0%18/01/2012:19:18 INFO mapreduce.Job: map 23% reduce 0%18/01/2012:19:23 INFO mapreduce.Job: map 24% reduce 0%18/01/2012:19:29 INFO mapreduce.Job: map 25% reduce 0%18/01/2012:19:33 INFO mapreduce.Job: map 26% reduce 0%18/01/2012:19:38 INFO mapreduce.Job: map 27% reduce 0%18/01/2012:19:43 INFO mapreduce.Job: map 28% reduce 0%18/01/2012:19:48 INFO mapreduce.Job: map 29% reduce 0%18/01/2012:19:53 INFO mapreduce.Job: map 30% reduce 0%18/01/2012:19:58 INFO mapreduce.Job: map 31% reduce 0%18/01/2012:20:04 INFO mapreduce.Job: map 32% reduce 0%18/01/2012:20:08 INFO mapreduce.Job: map 33% reduce 0%18/01/2012:20:13 INFO mapreduce.Job: map 34% reduce 0%18/01/2012:20:17 INFO mapreduce.Job: map 35% reduce 0%18/01/2012:20:21 INFO mapreduce.Job: map 36% reduce 0%18/01/2012:20:25 INFO mapreduce.Job: map 37% reduce 0%18/01/2012:20:29 INFO mapreduce.Job: map 38% reduce 0%18/01/2012:20:33 INFO mapreduce.Job: map 39% reduce 0%18/01/2012:20:37 INFO mapreduce.Job: map 40% reduce 0%18/01/2012:20:41 INFO mapreduce.Job: map 41% reduce 0%18/01/2012:20:45 INFO mapreduce.Job: map 42% reduce 0%18/01/2012:20:50 INFO mapreduce.Job: map 43% reduce 0%18/01/2012:20:54 INFO mapreduce.Job: map 44% reduce 0%18/01/2012:20:58 INFO mapreduce.Job: map 45% reduce 0%18/01/2012:21:02 INFO mapreduce.Job: map 46% reduce 0%18/01/2012:21:06 INFO mapreduce.Job: map 47% reduce 0%18/01/2012:21:10 INFO mapreduce.Job: map 48% reduce 0%18/01/2012:21:14 INFO mapreduce.Job: map 49% reduce 0%18/01/2012:21:18 INFO mapreduce.Job: map 50% reduce 0%18/01/2012:21:22 INFO mapreduce.Job: map 51% reduce 0% ........ 18/01/2012:29:12 INFO mapreduce.Job: map 100% reduce 81%18/01/2012:29:24 INFO mapreduce.Job: map 100% reduce 82%18/01/2012:29:36 INFO mapreduce.Job: map 100% reduce 83%18/01/2012:29:48 INFO mapreduce.Job: map 100% reduce 84%18/01/2012:30:00 INFO mapreduce.Job: map 100% reduce 85%18/01/2012:30:12 INFO mapreduce.Job: map 100% reduce 86%18/01/2012:30:23 INFO mapreduce.Job: map 100% reduce 87%18/01/2012:30:33 INFO mapreduce.Job:  map 100% reduce 88%18/01/2012:30:45 INFO mapreduce.Job: map 100% reduce 89%18/01/2012:30:59 INFO mapreduce.Job: map 100% reduce 90%18/01/2012:31:11 INFO mapreduce.Job: map 100% reduce 91%18/01/2012:31:21 INFO mapreduce.Job: map 100% reduce 92%18/01/2012:31:33 INFO mapreduce.Job: map 100% reduce 93%18/01/2012:31:45 INFO mapreduce.Job: map 100% reduce 94%18/01/2012:31:57 INFO mapreduce.Job: map 100% reduce 95%18/01/2012:32:10 INFO mapreduce.Job: map 100% reduce 96%18/01/2012:32:28 INFO mapreduce.Job: map 100% reduce 97%18/01/2012:32:57 INFO mapreduce.Job: map 100% reduce 98%18/01/2012:33:28 INFO mapreduce.Job: map 100% reduce 99%18/01/2012:34:43 INFO mapreduce.Job: map 100% reduce 100%18/01/2012:38:02 INFO mapreduce.Job: Job job_1516347580021_0001 completedsuccessfully18/01/2012:38:02 INFO mapreduce.Job: Counters: 52         File System Counters                 FILE: Number of bytes read=87576726096                 FILE: Number of bytes written=142193600747                 FILE: Number of readoperations=0                 FILE: Number of large readoperations=0                 FILE: Number of writeoperations=0                 HDFS: Number of bytes read=83582905128                 HDFS: Number of bytes written=166475667426                 HDFS: Number of readoperations=5468                 HDFS: Number of large readoperations=0                 HDFS: Number of writeoperations=39         Job Counters                  Failed map tasks=6                 Launched map tasks=1086                 Launched reduce tasks=10                 Other local map tasks=6                 Data-local map tasks=465                 Rack-local map tasks=615                 Total time spent by all maps inoccupied slots(ms)=82454392                 Total time spent by all reducesin occupied slots(ms)=47463944                 Total time spent by all map tasks(ms)=10306799                 Total time spent by all reduce tasks(ms)=5932993                 Total vcore-seconds taken byall map tasks=10306799                 Total vcore-seconds taken byall reduce tasks=5932993                 Total megabyte-seconds taken byall map tasks=84433297408                 Total megabyte-seconds taken byall reduce tasks=48603078656         Map-Reduce Framework                 Map input records=568152966                 Map output records=568152966                 Map output bytes=228099087448                 Map output materialized bytes=54476960272                 Input split bytes=186120                 Combine input records=0                 Combine output records=0                 Reduce input groups=292435364                 Reduce shuffle bytes=54476960272                 Reduce input records=568152966                 Reduce output records=2339482912                 Spilled Records=1513624168                 Shuffled Maps =10800                 Failed Shuffles=0                 Merged Map outputs=10800                 GC time elapsed(ms)=794607                 CPU time spent(ms)=21363440                 Physical memory(bytes) snapshot=3038556569600                 Virtual memory(bytes) snapshot=9401710268416                 Total committed heap usage(bytes)=3512994889728         Shuffle Errors                 BAD_ID=0                 CONNECTION=0                 IO_ERROR=0                 WRONG_LENGTH=0                 WRONG_MAP=0                 WRONG_REDUCE=0         File Input Format Counters                  Bytes Read=83582349648         File Output Format Counters                  Bytes Written=16647566742618/01/2012:38:02 INFO zookeeper.RecoverableZooKeeper: Processidentifier=hconnection-0x71f30c76 connecting to ZooKeeper ensemble=node003:4180,node002:4180,node001:4180,master:4180,node009:4180,node008:4180,node007:4180,node010:4180,node006:4180,node005:4180,node004:418018/01/2012:38:02 INFO zookeeper.ZooKeeper: Initiating client connection,connectString=node003:4180,node002:4180,node001:4180,master:4180,node009:4180,node008:4180,node007:4180,node010:4180,node006:4180,node005:4180,node004:4180 sessionTimeout=90000 watcher=hconnection-0x71f30c760x0, quorum=node003:4180,node002:4180,node001:4180,master:4180,node009:4180,node008:4180,node007:4180,node010:4180,node006:4180,node005:4180,node004:4180, baseZNode=/hbase18/01/2012:38:02 INFO zookeeper.ClientCnxn: Opening socket connection toserver node004/192.168.1.38:4180. Will not attempt to authenticate using SASL(unknown error)18/01/2012:38:02 INFO zookeeper.ClientCnxn: Socket connection established tonode004/192.168.1.38:4180, initiating session18/01/2012:38:02 INFO zookeeper.ClientCnxn: Session establishment complete on servernode004/192.168.1.38:4180, sessionid = 0x26001af8d8190002, negotiated timeout = 4000018/01/2012:38:02 WARN mapreduce.LoadIncrementalHFiles: Skippingnon-directory hdfs://192.168.1.31:9000/test_demo/result/test/_SUCCESS18/01/2012:38:02 WARN mapreduce.LoadIncrementalHFiles: Trying to bulkload hfile hdfs://192.168.1.31:9000/test_demo/result/test/info/28e47c53edaf4616a3dfc349d0f0e02awith size: 10931823633 bytes can be problematic as it may lead tooversplitting.18/01/2012:38:02 WARN mapreduce.LoadIncrementalHFiles: Trying to bulkload hfile hdfs://192.168.1.31:9000/test_demo/result/test/info/5e6501ccb7554b82a2e93024d61dbe0ewith size: 10931820982 bytes can be problematic as it may lead tooversplitting.18/01/2012:38:02 WARN mapreduce.LoadIncrementalHFiles: Trying to bulkload hfile hdfs://192.168.1.31:9000/test_demo/result/test/info/6245730468534f85a428ef7fb7acd499with size: 10931829083 bytes can be problematic as it may lead tooversplitting.18/01/2012:38:02 WARN mapreduce.LoadIncrementalHFiles: Trying to bulkload hfile hdfs://192.168.1.31:9000/test_demo/result/test/info/a3b7225320b24e838559d5a5772bdd87with size: 10931823391 bytes can be problematic as it may lead tooversplitting.18/01/2012:38:02 WARN mapreduce.LoadIncrementalHFiles: Trying to bulkload hfile hdfs://192.168.1.31:9000/test_demo/result/test/info/a8306bc4ef3941f5bd131d47f0b1c2c3with size: 10931822321 bytes can be problematic as it may lead tooversplitting.18/01/2012:38:02 WARN mapreduce.LoadIncrementalHFiles: Trying to bulkload hfile hdfs://192.168.1.31:9000/test_demo/result/test/info/c163c568b1c24d88ac0ed7599b81ecbawith size: 10931824861 bytes can be problematic as it may lead tooversplitting.18/01/2012:38:02 WARN mapreduce.LoadIncrementalHFiles: Trying to bulkload hfile hdfs://192.168.1.31:9000/test_demo/result/test/info/d069683ce064411793640f2a0ec6ca98with size: 10931822990 bytes can be problematic as it may lead tooversplitting.18/01/2012:38:02 WARN mapreduce.LoadIncrementalHFiles: Trying to bulkload hfile hdfs://192.168.1.31:9000/test_demo/result/test/info/e5df664e18c54da7b84370b72506923bwith size: 10931821709 bytes can be problematic as it may lead tooversplitting.18/01/2012:38:02 WARN mapreduce.LoadIncrementalHFiles: Trying to bulkload hfile hdfs://192.168.1.31:9000/test_demo/result/test/info/fba38b4d0bd34f6782b844b288780e7bwith size: 10931826385 bytes can be problematic as it may lead tooversplitting.18/01/2012:38:02 INFO hfile.CacheConfig: CacheConfig:disabled18/01/2012:38:02 INFO hfile.CacheConfig: CacheConfig:disabled18/01/2012:38:02 INFO hfile.CacheConfig: CacheConfig:disabled18/01/2012:38:02 INFO hfile.CacheConfig: CacheConfig:disabled18/01/2012:38:02 INFO hfile.CacheConfig: CacheConfig:disabled18/01/2012:38:02 INFO hfile.CacheConfig: CacheConfig:disabled18/01/2012:38:02 INFO hfile.CacheConfig: CacheConfig:disabled18/01/2012:38:02 INFO hfile.CacheConfig: CacheConfig:disabled18/01/2012:38:03 INFO mapreduce.LoadIncrementalHFiles: Trying to loadhfile=hdfs://192.168.1.31:9000/test_demo/result/test/info/11e458c2c1f04654ae1783ec4e6576e8 first=459096918168596876155 last=499999988802494582818/01/2012:38:03 INFO mapreduce.LoadIncrementalHFiles: Trying to load hfile=hdfs://192.168.1.31:9000/test_demo/result/test/info/78067c90799149dbb4a423ef556a4272first=559078464243536377945 last=599999988802494582818/01/2012:38:03 INFO mapreduce.LoadIncrementalHFiles: Trying to load hfile=hdfs://192.168.1.31:9000/test_demo/result/test/info/0049f16fd57b482aa2e68ebe21a0cb72first=15907887724999982915 last=1999999921761149633118/01/2012:38:03 INFO mapreduce.LoadIncrementalHFiles: Trying to load hfile=hdfs://192.168.1.31:9000/test_demo/result/test/info/28e47c53edaf4616a3dfc349d0f0e02a first=80100000359202982424 last=859088818898462383266 hfile=hdfs://192.168.1.31:9000/test_demo/result/test/info/6245730468534f85a428ef7fb7acd499 first=401000000531957283573 last=45909691794129495595418/01/2012:38:03 INFO mapreduce.LoadIncrementalHFiles: Trying to load hfile=hdfs://192.168.1.31:9000/test_demo/result/test/info/18068da4a3f5469a804eee9f6921617afirst=959083192452571451003 last=9999999823997720607818/01/2012:38:03 INFO mapreduce.LoadIncrementalHFiles: Trying to load hfile=hdfs://192.168.1.31:9000/test_demo/result/test/info/5e6501ccb7554b82a2e93024d61dbe0e first=30100000359202982424 last=35908116678630513718518/01/2012:38:03 INFO mapreduce.LoadIncrementalHFiles: Trying to load hfile=hdfs://192.168.1.31:9000/test_demo/result/test/info/0cff66c092004d488db32c3bf549a1d1 first=0100000359202982424 last=099999823997720607818/01/2012:38:03 INFO hfile.CacheConfig: CacheConfig:disabled18/01/2012:38:03 INFO hfile.CacheConfig: CacheConfig:disabled18/01/2012:38:03 INFO hfile.CacheConfig: CacheConfig:disabled18/01/2012:38:03 INFO hfile.CacheConfig: CacheConfig:disabled18/01/2012:38:03 INFO hfile.CacheConfig: CacheConfig:disabled18/01/2012:38:03 INFO hfile.CacheConfig: CacheConfig:disabled18/01/2012:38:03 INFO hfile.CacheConfig: CacheConfig:disabled18/01/2012:38:03 INFO hfile.CacheConfig: CacheConfig:disabled18/01/2012:38:03 INFO mapreduce.LoadIncrementalHFiles: Trying to loadhfile=hdfs://192.168.1.31:9000/test_demo/result/test/info/c163c568b1c24d88ac0ed7599b81ecba first=10100000359202982424 last=15907887393454423668 18/01/2012:38:03 INFO hfile.CacheConfig: CacheConfig:disabled18/01/2012:38:03 INFO mapreduce.LoadIncrementalHFiles: Trying to loadhfile=hdfs://192.168.1.31:9000/test_demo/result/test/info/e5df664e18c54da7b84370b72506923bfirst=501000000531957283573 last=55907845833734074458618/01/2012:38:03 INFO hfile.CacheConfig: CacheConfig:disabled18/01/2012:38:03 INFO mapreduce.LoadIncrementalHFiles: Trying to loadhfile=hdfs://192.168.1.31:9000/test_demo/result/test/info/88df957d66e84b758583c47c9e6aec95first=25908421410455709356 last=2999999823997720607818/01/2012:38:03 INFO hfile.CacheConfig: CacheConfig:disabled18/01/2012:38:03 INFO mapreduce.LoadIncrementalHFiles: Trying to loadhfile=hdfs://192.168.1.31:9000/test_demo/result/test/info/a8306bc4ef3941f5bd131d47f0b1c2c3first=60100000359202982424 last=65907914592917333360018/01/2012:38:03 INFO mapreduce.LoadIncrementalHFiles: Trying to loadhfile=hdfs://192.168.1.31:9000/test_demo/result/test/info/d28c5e918b784127a7faa8afee8b364dfirst=359081168652388606128 last=3999999921761149633118/01/2012:38:03 INFO mapreduce.LoadIncrementalHFiles: Trying to loadhfile=hdfs://192.168.1.31:9000/test_demo/result/test/info/a3b7225320b24e838559d5a5772bdd87first=701000000531957283573 last=75908948961515784114418/01/2012:38:03 INFO mapreduce.LoadIncrementalHFiles: Trying to loadhfile=hdfs://192.168.1.31:9000/test_demo/result/test/info/d069683ce064411793640f2a0ec6ca98first=20100000359202982424 last=2590842137719375424718/01/2012:38:03 INFO mapreduce.LoadIncrementalHFiles: Trying to loadhfile=hdfs://192.168.1.31:9000/test_demo/result/test/info/7b148f47400d49d8aefd92b06a530dc5first=659079146670017258500 last=6999999921761149633118/01/2012:38:03 INFO mapreduce.LoadIncrementalHFiles: Trying to loadhfile=hdfs://192.168.1.31:9000/test_demo/result/test/info/e93f91b50972491d8c600a6bd115bab3first=859088819882023983305 last=8999999823997720607818/01/2012:38:03 INFO mapreduce.LoadIncrementalHFiles: Trying to loadhfile=hdfs://192.168.1.31:9000/test_demo/result/test/info/f73df3c94e9f4c659965dc11f66ddd7bfirst=759089490241357286269 last=799999988802494582818/01/2012:38:03 INFO mapreduce.LoadIncrementalHFiles: Trying to loadhfile=hdfs://192.168.1.31:9000/test_demo/result/test/info/fba38b4d0bd34f6782b844b288780e7bfirst=901000000531957283573 last=959083189020333412493程序的执行时间为:125605318/01/2012:38:03 INFO client.ConnectionManager$HConnectionImplementation:Closing zookeeper sessionid=0x38001af93e73000118/01/2012:38:03 INFO zookeeper.ZooKeeper: Session: 0x38001af93e730001 closed18/01/2012:38:03 INFO zookeeper.ClientCnxn: EventThread shut down去hbase数据库里面查询数据:Current count:100983000, row:405415804076494331733                                                                                                                                          Current count:100984000, row:405418669611868961647                                                                                                                                          Current count:100985000, row:40542152486397650152                                                                                                                                            Current count:100986000, row:405424331577144238851                                                                                                                                          Current count:100987000, row:405427092734454272384                                                                                                                                          Current count:100988000, row:405429767351893163972                                                                                                                                          Current count:100989000, row:40543249871570790691                                                                                                                                            Current count:100990000, row:405435366049740236059  可以看到,不完全统计就已经有1亿条以上的数据.... 这样hbase的批量入库,无论从效率上,还是数据量上都能够慢足要求!
本文源自于：http://blog.csdn.net/weixin_40861707/article/details/79105753，的一部分，详情参考改地址