hive-05-Execution Error, return code 3 from org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask
2017-11-21 15:38
706 查看
hive命令行里执行了一句话:
就是两个表做连接查询,数据量大小一个是3千万,一个是3亿
结果报错
解决方法:
set hive.mapjoin.localtask.max.memory.usage=0.99
hive.mapjoin.localtask.max.memory.usage
•说明:本地任务可以使用内存的百分比 •默认值: 0.90,如果你的localtask mapjoin 表很小可以试试,但彻底解决需要
set hive.auto.convert.join=false;关闭自动mapjoin 但这个参数用的时候一定要注意,
如果你的sql 很长join会常多,关闭mapjoin任务数会成10倍激增,contener满了任务同样会非常之慢,set hive.auto.convert.join=false;一定要用在localtask级别这种超轻量及的job上。
select * from person join zhanghao on person.zjhm = zhanghao.zjhm limit 100 ;
就是两个表做连接查询,数据量大小一个是3千万,一个是3亿
结果报错
hive> select * from person join zhanghao on person.zjhm = zhanghao.zjhm limit 100 ; WARNING: Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases. Query ID = root_20171121152644_c3525e56-ea74-474f-8d14-c1d18f3b3f81 Total jobs = 1 SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/opt/hzjs/apache-hive-2.1.1-bin/lib/log4j-slf4j-impl-2.4.1.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/opt/hzjs/apache-hive-2.1.1-bin/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/opt/hzjs/spark-2.1.1-bin-hadoop2.7/jars/slf4j-log4j12-1.7.16.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/opt/hzjs/hadoop-2.7.3/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory] 2017-11-21 15:27:11,779 main ERROR Unable to locate appender "INFO,DRFA" for logger config "root" 2017-11-21 15:27:15 Starting to launch local task to process map join; maximum memory = 477626368 2017-11-21 15:27:39 Processing rows: 200000 Hashtable size: 199999 Memory usage: 275244704 percentage: 0.576 2017-11-21 15:27:50 Processing rows: 300000 Hashtable size: 299999 Memory usage: 351999776 percentage: 0.737 2017-11-21 15:28:01 Processing rows: 400000 Hashtable size: 399999 Memory usage: 373283344 percentage: 0.782 2017-11-21 15:28:49 Processing rows: 500000 Hashtable size: 499999 Memory usage: 439679208 percentage: 0.921 Execution failed with exit status: 3 Obtaining error information Task failed! Task ID: Stage-4 Logs: FAILED: Execution Error, return code 3 from org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask
解决方法:
set hive.mapjoin.localtask.max.memory.usage=0.99
hive.mapjoin.localtask.max.memory.usage
•说明:本地任务可以使用内存的百分比 •默认值: 0.90,如果你的localtask mapjoin 表很小可以试试,但彻底解决需要
set hive.auto.convert.join=false;关闭自动mapjoin 但这个参数用的时候一定要注意,
如果你的sql 很长join会常多,关闭mapjoin任务数会成10倍激增,contener满了任务同样会非常之慢,set hive.auto.convert.join=false;一定要用在localtask级别这种超轻量及的job上。
相关文章推荐
- Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask
- FAILED: Execution Error, return code 3 from org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask
- FAILED: Execution Error, return code 3 from org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask
- 解决hiveserver2报错:java.io.IOException: Job status not available - Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask
- Hive jdbc执行seelct 语句时报 return code 1 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask
- FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask.
- Hive报错:FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask
- 程序中执行hive sql出错[FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.MoveTask]
- FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(me
- hive创建表失败:FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask.
- Hive连接Mysql---FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask.
- FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. java.lang.Runtim
- FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask
- FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask
- Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask.
- Hive连接Mysql---FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask.
- FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. org/bson/convers
- org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask
- FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.MoveTask
- Hive 报错 FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. MetaExce