【apache-hive-1.2.1】local mr
2016-01-25 00:00
941 查看
在hive中运行sql有很多是比较小的SQL,数据量小,计算量小。这些比较小的SQL 如果也采用分布式的方式来执行,那么就得不偿失,因为SQL真正执行的时间可能只有10s,但是分布式任务生成的其他过程执行可能要1min。这样小任务采用local mr方式执行,就是本地执行,通过把输入数据拉回到客户端来执行
三个参数来决定:
hive.exec.mode.local.auto=true 是否启动本地mr模式
hive.exec.mode.local.auto.input.files.max=4 input files的数量,默认是4个
hive.exec.mode.local.auto.inputbytes.max=134217728 input files的大小,默认是128M
注意:
hive.exec.mode.local.auto是大前提,只有设置为true,才可能会启用local mr模式
hive.exec.mode.local.auto.input.files.max 和 hive.exec.mode.local.auto.inputbytes.max是 与的关系,只有同时满足才会执行local mr
t_1==> 5个文件
t_2==> 2个文件
三个参数来决定:
hive.exec.mode.local.auto=true 是否启动本地mr模式
hive.exec.mode.local.auto.input.files.max=4 input files的数量,默认是4个
hive.exec.mode.local.auto.inputbytes.max=134217728 input files的大小,默认是128M
注意:
hive.exec.mode.local.auto是大前提,只有设置为true,才可能会启用local mr模式
hive.exec.mode.local.auto.input.files.max 和 hive.exec.mode.local.auto.inputbytes.max是 与的关系,只有同时满足才会执行local mr
t_1==> 5个文件
t_2==> 2个文件
hive>set hive.exec.mode.local.auto=false hive> select * from t_2 order by id; Query ID = hadoop_20160125132157_d767beb0-f674-4962-ac3c-8fbdd2949d01 Total jobs = 1 Launching Job 1 out of 1 Number of reduce tasks determined at compile time: 1 In order to change the average load for a reducer (in bytes): set hive.exec.reducers.bytes.per.reducer=<number> In order to limit the maximum number of reducers: set hive.exec.reducers.max=<number> In order to set a constant number of reducers: set mapreduce.job.reduces=<number> Starting Job = job_1453706740954_0006, Tracking URL = http://hftest0001.webex.com:8088/proxy/application_1453706740954_0006/ Kill Command = /home/hadoop/hadoop-2.7.1/bin/hadoop job -kill job_1453706740954_0006 Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 1 2016-01-25 13:22:19,210 Stage-1 map = 0%, reduce = 0% 2016-01-25 13:22:26,497 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 1.47 sec 2016-01-25 13:22:40,207 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 3.68 sec MapReduce Total cumulative CPU time: 3 seconds 680 msec Ended Job = job_1453706740954_0006 MapReduce Jobs Launched: Stage-Stage-1: Map: 1 Reduce: 1 Cumulative CPU: 3.68 sec HDFS Read: 5465 HDFS Write: 32 SUCCESS Total MapReduce CPU Time Spent: 3 seconds 680 msec OK ... ... hive>set hive.exec.mode.local.auto=true hive> select * from t_2 order by id; hive> select * from t_2 order by id; Automatically selecting local only mode for query ==> 启动用本地模式 Query ID = hadoop_20160125132322_9649b904-ad87-47fa-89ad-5e5f67315ac8 Total jobs = 1 Launching Job 1 out of 1 Number of reduce tasks determined at compile time: 1 In order to change the average load for a reducer (in bytes): set hive.exec.reducers.bytes.per.reducer=<number> In order to limit the maximum number of reducers: set hive.exec.reducers.max=<number> In order to set a constant number of reducers: set mapreduce.job.reduces=<number> Job running in-process (local Hadoop) 2016-01-25 13:23:27,192 Stage-1 map = 100%, reduce = 100% Ended Job = job_local1850780899_0002 MapReduce Jobs Launched: Stage-Stage-1: HDFS Read: 1464 HDFS Write: 1618252652 SUCCESS Total MapReduce CPU Time Spent: 0 msec OK ... ... hive>set hive.exec.mode.local.auto=true hive> select * from t_1 order by id; Query ID = hadoop_20160125132411_3ecd7ee9-8ccb-4bcc-8582-6d797c13babd Total jobs = 1 Launching Job 1 out of 1 Number of reduce tasks determined at compile time: 1 In order to change the average load for a reducer (in bytes): set hive.exec.reducers.bytes.per.reducer=<number> In order to limit the maximum number of reducers: set hive.exec.reducers.max=<number> In order to set a constant number of reducers: set mapreduce.job.reduces=<number> Cannot run job locally: Number of Input Files (= 5) is larger than hive.exec.mode.local.auto.input.files.max(= 4) ==>5 > 4 还是启用了分布式 Starting Job = job_1453706740954_0007, Tracking URL = http://hftest0001.webex.com:8088/proxy/application_1453706740954_0007/ Kill Command = /home/hadoop/hadoop-2.7.1/bin/hadoop job -kill job_1453706740954_0007 Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 1 2016-01-25 13:24:38,775 Stage-1 map = 0%, reduce = 0% 2016-01-25 13:24:52,115 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 1.55 sec 2016-01-25 13:24:59,548 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 3.84 sec MapReduce Total cumulative CPU time: 3 seconds 840 msec Ended Job = job_1453706740954_0007 MapReduce Jobs Launched: Stage-Stage-1: Map: 1 Reduce: 1 Cumulative CPU: 3.84 sec HDFS Read: 5814 HDFS Write: 56 SUCCESS Total MapReduce CPU Time Spent: 3 seconds 840 msec OK ... ... hive>set hive.exec.mode.local.auto=true hive> set hive.exec.mode.local.auto.input.files.max=5; ==> 设置输入文件数max=5 hive> select * from t_1 order by id; Automatically selecting local only mode for query ==> 启用了本地模式 Query ID = hadoop_20160125132558_db2f4fca-f6bf-4b91-9569-c779a3b13386 Total jobs = 1 Launching Job 1 out of 1 Number of reduce tasks determined at compile time: 1 In order to change the average load for a reducer (in bytes): set hive.exec.reducers.bytes.per.reducer=<number> In order to limit the maximum number of reducers: set hive.exec.reducers.max=<number> In order to set a constant number of reducers: set mapreduce.job.reduces=<number> Job running in-process (local Hadoop) 2016-01-25 13:26:03,232 Stage-1 map = 100%, reduce = 100% Ended Job = job_local264155444_0003 MapReduce Jobs Launched: Stage-Stage-1: HDFS Read: 1920 HDFS Write: 1887961792 SUCCESS Total MapReduce CPU Time Spent: 0 msec OK
相关文章推荐
- 源码编译lamp apache启动脚本
- LAMP(Linux+Apache+MySQL+PHP)安装
- 基于Windows的Apache+Git服务器搭建
- 分布式文件存储FastDFS(四)配置fastdfs-apache-module
- Ubuntu apache 虚拟主机配置
- Apache服务器-使用.htaccess实现图片防盗链方法
- Apache 配置屏蔽某些请求头
- macosx下apache的默认用户为daemon
- Apache Commons 工具类介绍及简单使用
- Linux+apache2.4+php5.6环境搭建
- APACHE服务器500错误解决方法
- php.ini+apache配置(持续更新)
- Apache+PHP+MySQL三部曲之:Apache2.4安装详解 - 20160123最新更新
- 在自己机器上实现apache的多域名
- 在自己机器上实现apache的多域名
- apache 安装[转]
- Apache Spark探秘:三种分布式部署方式比较
- Eclipse使用Maven插件创建Web项目时出错:Could not resolve archetype org.apache.maven.archetypes
- Apache HBase 2015年发展回顾与未来展望
- "Failure to transfer org.apache.maven.plugins:maven-compiler-plugin"解决方案