您的位置:首页 > 运维架构 > Apache

hive-05-Execution Error, return code 3 from org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask

2017-11-21 15:38 706 查看
hive命令行里执行了一句话:

select * from person  join zhanghao   on person.zjhm = zhanghao.zjhm limit 100 ;


就是两个表做连接查询,数据量大小一个是3千万,一个是3亿

结果报错

hive> select * from person  join zhanghao   on person.zjhm = zhanghao.zjhm limit 100 ;
WARNING: Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.
Query ID = root_20171121152644_c3525e56-ea74-474f-8d14-c1d18f3b3f81
Total jobs = 1
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/hzjs/apache-hive-2.1.1-bin/lib/log4j-slf4j-impl-2.4.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/hzjs/apache-hive-2.1.1-bin/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/hzjs/spark-2.1.1-bin-hadoop2.7/jars/slf4j-log4j12-1.7.16.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/hzjs/hadoop-2.7.3/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
2017-11-21 15:27:11,779 main ERROR Unable to locate appender "INFO,DRFA" for logger config "root"
2017-11-21 15:27:15     Starting to launch local task to process map join;      maximum memory = 477626368
2017-11-21 15:27:39     Processing rows:        200000  Hashtable size: 199999  Memory usage:   275244704       percentage:     0.576
2017-11-21 15:27:50     Processing rows:        300000  Hashtable size: 299999  Memory usage:   351999776       percentage:     0.737
2017-11-21 15:28:01     Processing rows:        400000  Hashtable size: 399999  Memory usage:   373283344       percentage:     0.782
2017-11-21 15:28:49     Processing rows:        500000  Hashtable size: 499999  Memory usage:   439679208       percentage:     0.921
Execution failed with exit status: 3
Obtaining error information

Task failed!
Task ID:
Stage-4

Logs:

FAILED: Execution Error, return code 3 from org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask


解决方法:

set hive.mapjoin.localtask.max.memory.usage=0.99

hive.mapjoin.localtask.max.memory.usage

•说明:本地任务可以使用内存的百分比 •默认值: 0.90,如果你的localtask mapjoin 表很小可以试试,但彻底解决需要

set hive.auto.convert.join=false;关闭自动mapjoin 但这个参数用的时候一定要注意,

如果你的sql 很长join会常多,关闭mapjoin任务数会成10倍激增,contener满了任务同样会非常之慢,set hive.auto.convert.join=false;一定要用在localtask级别这种超轻量及的job上。
内容来自用户分享和网络整理,不保证内容的准确性,如有侵权内容,可联系管理员处理 点击这里给我发消息
标签: 
相关文章推荐