Hive安装及问题解决
2013-12-26 17:07
267 查看
刚刚安装好Hadoop迫不及待要玩下hive,所以自己找了些资料,安装了下hive,这里记录详细的安装过程和碰到的问题:
1.在安装hadoop之前首先确定hadoop已经安装成功,具体安装方法请参考hadoop安装 见 http://www.linuxidc.com/Linux/2011-12/48784.htm 与 http://www.linuxidc.com/Linux/2011-12/48783.htm。
2.下载hive版本 我下载的是
下载地址为http://hive.apache.org/releases.htm
解压文件:
tar -zxvf hive-0.7.0-bin.tar.gz
3.配置环境变量:
[root@localhost hive]# vim /etc/profile
export HADOOP_HOME=/root/hadoop-0.20.2
export PATH=$PATH:$HADOOP_HOME/bin
export HIVE_HOME=/usr/hive/hive-0.7.0-bin
PATH=$PATH:$HIVE_HOME/bin
export PATH
export JAVA_HOME=/usr/java/jdk1.7.0_01
4.我存储元数据是采用第三方的mysql数据库,这种情况下需要下载一个数据包mysql-connector-java-5.1.18-bin.jar,放到hive的lib目录下
在hive的conf目录下创建文件hive-site.xml配置文件,代码如下:
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<property>
<name>hive.metastore.local</name>
<value>true</value>
</property>
<property>
<name>javax.jdo.option.ConnectionURL</name>
<value>jdbc:mysql://192.168.0.237:3306/metastore_db?createDatabaseIfNotExist=true</value>
</property>
<property>
<name>javax.jdo.option.ConnectionDriverName</name>
<value>com.mysql.jdbc.Driver</value>
</property>
<property>
<name>javax.jdo.option.ConnectionUserName</name>
<value>t237</value>
</property>
<property>
<name>javax.jdo.option.ConnectionPassword</name>
<value>123456</value>
</property>
<property>
<name>datanucleus.fixedDatastore</name>
<value>false</value>
</property>
</configuration>
5.运行hive
[root@localhost conf]# hive
Hive history file=/tmp/root/hive_job_log_root_201112061510_1145168119.txt
hive> show tables;
OK
log
log_info
log_new
maptile
result
Time taken: 2.889 seconds
hive>
6.这里遇到点问题:刚开始执行hive时,执行show tables;报如下错误:
FAILED: Hive Internal Error: java.lang.RuntimeException(Error while making MR scratch directory - check filesystem config (null))
java.lang.RuntimeException: Error while making MR scratch directory - check filesystem config (null)
后来询问了google,得到如下解决办法:
原有的hadoop配置文件core-site.xml信息如下:
<pre name="code" class="plain"><!-- Put site-specific property overrides in this file. -->
<configuration>
<property>
<name>hadoop.tmp.dir</name>
<value>/hadoop-root/tmp</value>
</property>
<property>
<name>fs.default.name</name>
<value>hdfs://192.168.30.149:9000/</value>
</property>
</configuration>
改成:
<!-- Put site-specific property overrides in this file. -->
<configuration>
<property>
<name>hadoop.tmp.dir</name>
<value>/hadoop-root/tmp</value>
</property>
<property>
<name>fs.default.name</name>
<value>hdfs://hadoop149:9000/</value>
</property>
</configuration>
重启hadoop,运行hive,问题解决。
之后创建表一直很顺利,但是到查询表例如我执行:select size from log limit 10;时,又报错如下:
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/hive/conf/HiveConf
后来问了下百度和google都没有结果,自己觉得是缺少了类文件,我便把hive/lib下所有的hive-*文件拷贝到hadoop的lib下
重启hadoop,问题解决。但是归根结底我觉得是$HADOOP_CLASSPATH变量没有配置清楚,这个之后再解决这里做个标记。
执行select 后运行正常:
hive> select size from log_info limit 12;
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks is set to 0 since there's no reduce operator
Starting Job = job_201112061127_0004, Tracking URL = http://hadoop149:50030/jobdetails.jsp?jobid=job_201112061127_0004
Kill Command = /root/hadoop-0.20.2/bin/hadoop job -Dmapred.job.tracker=hdfs://192.168.30.149:9004 -kill job_201112061127_0004
2011-12-06 15:35:47,923 Stage-1 map = 0%, reduce = 0%
2011-12-06 15:35:50,952 Stage-1 map = 9%, reduce = 0%
2011-12-06 15:35:52,975 Stage-1 map = 27%, reduce = 0%
2011-12-06 15:35:53,984 Stage-1 map = 45%, reduce = 0%
2011-12-06 15:35:56,001 Stage-1 map = 64%, reduce = 0%
2011-12-06 15:35:59,023 Stage-1 map = 82%, reduce = 0%
2011-12-06 15:36:00,038 Stage-1 map = 100%, reduce = 0%
2011-12-06 15:36:03,062 Stage-1 map = 100%, reduce = 100%
Ended Job = job_201112061127_0004
现在又找到的心得问题:在执行删除表操作时,出现如下错误:
NestedThrowables:
com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Specified key was too long; max key length is 767 bytes
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask
这个问题还没有解决:期待牛人解决
最后附上:内存数据库derby的配置
<configuration>
<property>
<name>hive.metastore.local</name>
<value>true</value>
<description>controls whether to connect to remove metastore server or open a new metastore server in Hive Client JVM</description>
</property>
<property>
<name>javax.jdo.option.ConnectionURL</name>
<value>jdbc:derby://hadoop149:1527/metastore_db;create=true</value>
<description>JDBC connect string for a JDBC metastore</description>
</property>
<property>
<name>javax.jdo.option.ConnectionDriverName</name>
<value>org.apache.derby.jdbc.ClientDriver</value>
<description>Driver class name for a JDBC metastore</description>
</property>
</configuration>
1.在安装hadoop之前首先确定hadoop已经安装成功,具体安装方法请参考hadoop安装 见 http://www.linuxidc.com/Linux/2011-12/48784.htm 与 http://www.linuxidc.com/Linux/2011-12/48783.htm。
2.下载hive版本 我下载的是
下载地址为http://hive.apache.org/releases.htm
解压文件:
tar -zxvf hive-0.7.0-bin.tar.gz
3.配置环境变量:
[root@localhost hive]# vim /etc/profile
export HADOOP_HOME=/root/hadoop-0.20.2
export PATH=$PATH:$HADOOP_HOME/bin
export HIVE_HOME=/usr/hive/hive-0.7.0-bin
PATH=$PATH:$HIVE_HOME/bin
export PATH
export JAVA_HOME=/usr/java/jdk1.7.0_01
4.我存储元数据是采用第三方的mysql数据库,这种情况下需要下载一个数据包mysql-connector-java-5.1.18-bin.jar,放到hive的lib目录下
在hive的conf目录下创建文件hive-site.xml配置文件,代码如下:
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<property>
<name>hive.metastore.local</name>
<value>true</value>
</property>
<property>
<name>javax.jdo.option.ConnectionURL</name>
<value>jdbc:mysql://192.168.0.237:3306/metastore_db?createDatabaseIfNotExist=true</value>
</property>
<property>
<name>javax.jdo.option.ConnectionDriverName</name>
<value>com.mysql.jdbc.Driver</value>
</property>
<property>
<name>javax.jdo.option.ConnectionUserName</name>
<value>t237</value>
</property>
<property>
<name>javax.jdo.option.ConnectionPassword</name>
<value>123456</value>
</property>
<property>
<name>datanucleus.fixedDatastore</name>
<value>false</value>
</property>
</configuration>
5.运行hive
[root@localhost conf]# hive
Hive history file=/tmp/root/hive_job_log_root_201112061510_1145168119.txt
hive> show tables;
OK
log
log_info
log_new
maptile
result
Time taken: 2.889 seconds
hive>
6.这里遇到点问题:刚开始执行hive时,执行show tables;报如下错误:
FAILED: Hive Internal Error: java.lang.RuntimeException(Error while making MR scratch directory - check filesystem config (null))
java.lang.RuntimeException: Error while making MR scratch directory - check filesystem config (null)
后来询问了google,得到如下解决办法:
原有的hadoop配置文件core-site.xml信息如下:
<pre name="code" class="plain"><!-- Put site-specific property overrides in this file. -->
<configuration>
<property>
<name>hadoop.tmp.dir</name>
<value>/hadoop-root/tmp</value>
</property>
<property>
<name>fs.default.name</name>
<value>hdfs://192.168.30.149:9000/</value>
</property>
</configuration>
改成:
<!-- Put site-specific property overrides in this file. -->
<configuration>
<property>
<name>hadoop.tmp.dir</name>
<value>/hadoop-root/tmp</value>
</property>
<property>
<name>fs.default.name</name>
<value>hdfs://hadoop149:9000/</value>
</property>
</configuration>
重启hadoop,运行hive,问题解决。
之后创建表一直很顺利,但是到查询表例如我执行:select size from log limit 10;时,又报错如下:
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/hive/conf/HiveConf
后来问了下百度和google都没有结果,自己觉得是缺少了类文件,我便把hive/lib下所有的hive-*文件拷贝到hadoop的lib下
重启hadoop,问题解决。但是归根结底我觉得是$HADOOP_CLASSPATH变量没有配置清楚,这个之后再解决这里做个标记。
执行select 后运行正常:
hive> select size from log_info limit 12;
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks is set to 0 since there's no reduce operator
Starting Job = job_201112061127_0004, Tracking URL = http://hadoop149:50030/jobdetails.jsp?jobid=job_201112061127_0004
Kill Command = /root/hadoop-0.20.2/bin/hadoop job -Dmapred.job.tracker=hdfs://192.168.30.149:9004 -kill job_201112061127_0004
2011-12-06 15:35:47,923 Stage-1 map = 0%, reduce = 0%
2011-12-06 15:35:50,952 Stage-1 map = 9%, reduce = 0%
2011-12-06 15:35:52,975 Stage-1 map = 27%, reduce = 0%
2011-12-06 15:35:53,984 Stage-1 map = 45%, reduce = 0%
2011-12-06 15:35:56,001 Stage-1 map = 64%, reduce = 0%
2011-12-06 15:35:59,023 Stage-1 map = 82%, reduce = 0%
2011-12-06 15:36:00,038 Stage-1 map = 100%, reduce = 0%
2011-12-06 15:36:03,062 Stage-1 map = 100%, reduce = 100%
Ended Job = job_201112061127_0004
现在又找到的心得问题:在执行删除表操作时,出现如下错误:
NestedThrowables:
com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Specified key was too long; max key length is 767 bytes
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask
这个问题还没有解决:期待牛人解决
最后附上:内存数据库derby的配置
<configuration>
<property>
<name>hive.metastore.local</name>
<value>true</value>
<description>controls whether to connect to remove metastore server or open a new metastore server in Hive Client JVM</description>
</property>
<property>
<name>javax.jdo.option.ConnectionURL</name>
<value>jdbc:derby://hadoop149:1527/metastore_db;create=true</value>
<description>JDBC connect string for a JDBC metastore</description>
</property>
<property>
<name>javax.jdo.option.ConnectionDriverName</name>
<value>org.apache.derby.jdbc.ClientDriver</value>
<description>Driver class name for a JDBC metastore</description>
</property>
</configuration>
相关文章推荐
- Android之Broadcast详解
- TS流分析
- html checked
- 自娱自乐之选择排序
- Set statistics time 输出 CPU time> elapsed time
- 8天学习MongoDB——第一天 基础入门
- KINECT+openni_NITE(kinect开发)
- Linux下Mysql的配置(rpm形式安装)
- Asterisk学习进阶-3
- Android中设置控件可见与不可见
- 写出好的 commit message
- java环境变量配置
- PHP截取字符串,获取长度,获取字符位置的函数 中文字符截取
- C标准库stdio源码分析
- Linux/Unix服务端和客户端Socket编程入门实例(含源码下载)
- hadoop1.0.3 中的hive0.9.0的安装
- 第一次调试微信的Android版本出现的问题和解决方法
- Oracle求当前周和.net求当前周
- test
- [查阅]Dalvik opcodes