hive部署安装(apache-hive-1.1.0)
2016-06-27 13:10
633 查看
http://blog.csdn.net/an342647823/article/details/46048403
以下操作在hdpsrc3 节点上操作
一,下载安装包
1,下载hive http://mirrors.hust.edu.cn/apache/ 得到apache-hive-1.1.0.tar.gz ,放到该目录下
/home/hdpsrc/
2,下载mysql http://dev.mysql.com/downloads/mysql/5.5.html#downloads
得到
mysql-client-5.5.39-2.linux2.6.x86_64.rpm
mysql-devel-5.5.39-2.linux2.6.x86_64.rpm
mysql-server-5.5.39-2.linux2.6.x86_64.rpm
mysql-shared-5.5.39-2.linux2.6.x86_64.rpm
mysql-shared-compat-5.5.39-2.linux2.6.x86_64.rpm
拷贝到该目录下 /home/hdpsrc/Desktop/mysql/
二,安装mysql
1,卸载系统自带的mysql相关安装包,仅卸载 mysql 开头的包
rpm -qa|grep mysql
sudo rpm -e --nodeps mysql-libs-5.1.71-1.el6.x86_64
2,安装
cd /home/hdpsrc/Desktop/mysql/
sudo rpm -ivh mysql-*
sudo cp /usr/share/mysql/my-large.cnf /etc/my.cnf
3,启动设置mysql
启动mysql服务
sudo service mysql start
设置为开机自启动
sudo chkconfig mysql on
设置root用户登录密码
sudo /usr/bin/mysqladmin -u root password 'wu123'
登录mysql 以root用户身份登录
mysql -uroot -pwu123
创建hive用户,数据库等
insert into mysql.user(Host,User,Password) values("localhost","hive",password("hive"));
create database hive;
grant all on hive.* to hive@'%' identified by 'hive';
grant all on hive.* to hive@'localhost' identified by 'hive';
flush privileges;
退出mysql
exit
验证hive用户
mysql -uhive -phive
show databases;
看到如下反馈信息,则说明创建成功
mysql> show databases;
+--------------------+
| Database |
+--------------------+
| information_schema |
| hive |
| test |
+--------------------+
3 rows in set (0.00 sec)
退出mysql
exit
三,安装hive
1,解压安装包
cd ~
tar -zxvf apache-hive-1.1.0-bin.tar.gz
2,建立软连接
ln -s apache-hive-1.1.0-bin hive
3,添加环境变量
vi .bash_profile
导入下面的环境变量
export HIVE_HOME=/home/hdpsrc/hive
export PATH=$PATH:$HIVE_HOME/bin
使其有效
source .bash_profile
4,修改hive-site.xml
主要修改以下参数
<property>
<name>javax.jdo.option.ConnectionURL </name>
<value>jdbc:mysql://localhost:3306/hive </value>
</property>
<property>
<name>javax.jdo.option.ConnectionDriverName </name>
<value>com.mysql.jdbc.Driver </value>
</property>
<property>
<name>javax.jdo.option.ConnectionPassword </name>
<value>hive </value>
</property>
<property>
<name>hive.hwi.listen.port </name>
<value>9999 </value>
<description>This is the port the Hive Web Interface will listen on </descript ion>
</property>
<property>
<name>datanucleus.autoCreateSchema </name>
<value>true</value>
</property>
<property>
<name>datanucleus.fixedDatastore </name>
<value>false</value>
</property>
</property>
<property>
<name>javax.jdo.option.ConnectionUserName</name>
<value>hive</value>
<description>Username to use against metastore database</description>
</property>
<property>
<name>hive.exec.local.scratchdir</name>
<value>/home/hdpsrc/hive/iotmp</value>
<description>Local scratch space for Hive jobs</description>
</property>
<property>
<name>hive.downloaded.resources.dir</name>
<value>/home/hdpsrc/hive/iotmp</value>
<description>Temporary local directory for added resources in the remote file system.</description>
</property>
<property>
<name>hive.querylog.location</name>
<value>/home/hdpsrc/hive/iotmp</value>
<description>Location of Hive run time structured log file</description>
</property>
cp hive/conf/hive-default.xml.template hive/conf/hive-site.xml
编辑hive-site.xml
5,拷贝mysql-connector-java-5.1.6-bin.jar 到hive 的lib下面
mv /home/hdpsrc/Desktop/mysql-connector-java-5.1.6-bin.jar /home/hdpsrc/hive/lib/
6,把jline-2.12.jar拷贝到hadoop相应的目录下,替代jline-0.9.94.jar,否则启动会报错
cp /home/hdpsrc/hive/lib/jline-2.12.jar /home/hdpsrc/hadoop-2.6.0/share/hadoop/yarn/lib/
mv /home/hdpsrc/hadoop-2.6.0/share/hadoop/yarn/lib/jline-0.9.94.jar /home/hdpsrc/hadoop-2.6.0/share/hadoop/yarn/lib/jline-0.9.94.jar.bak /
7,穿件hive临时文件夹
mkdir /home/hdpsrc/hive/iotmp
四,启动测试hive
启动hadoop后,执行hive命令
hive
测试输入 show database;
hive> show databases;
OK
default
Time taken: 0.907 seconds, Fetched: 1 row(s)
遇到问题总结希望可以帮助遇到此类问题的人。
建议先建元数据库,设置编码latin1。否则建好元数据相关可能会出问题,如drop table 卡死, create table too long等等
hive对utf-8支持不好。设置完编码latin1,发现table 字段描述无法显示中文。修改元数据库表的字符
(1)修改表字段注解和表注解
alter table COLUMNS_V2 modify column COMMENT varchar(256) character set utf8
alter table TABLE_PARAMS modify column PARAM_VALUE varchar(4000) character set utf8
(2) 修改分区字段注解:
alter table PARTITION_PARAMS modify column PARAM_VALUE varchar(4000) character set utf8 ;
alter table PARTITION_KEYS modify column PKEY_COMMENT varchar(4000) character set utf8;
(3)修改索引注解:
alter table INDEX_PARAMS modify column PARAM_VALUE varchar(4000) character set utf8;
连接元数据设置
dbc:mysql://192.168.209.1:3306/metastore_hive_db?createDatabaseIfNotExist=true&characterEncoding=UTF-8
对于已经建好的表,不起作用。 最好安装的时候就修改编码格式。
元数据mysql远程模式配置
<property>
<name>hive.metastore.uris</name>
<value>thrift://192.168.223.129:9083</value>
<description>运行hive的主机地址及端口(特别重要ip不要弄错)</description>
</property>
启动元数据
bin/hive --service metastore &
到此hive已经安装完成
参考文档:http://www.mamicode.com/info-detail-516526.html
http://blog.csdn.net/blueheart20/article/details/38460541
备注:
1,遇到的问题:http://www.mamicode.com/info-detail-516526.html
以下操作在hdpsrc3 节点上操作
一,下载安装包
1,下载hive http://mirrors.hust.edu.cn/apache/ 得到apache-hive-1.1.0.tar.gz ,放到该目录下
/home/hdpsrc/
2,下载mysql http://dev.mysql.com/downloads/mysql/5.5.html#downloads
得到
mysql-client-5.5.39-2.linux2.6.x86_64.rpm
mysql-devel-5.5.39-2.linux2.6.x86_64.rpm
mysql-server-5.5.39-2.linux2.6.x86_64.rpm
mysql-shared-5.5.39-2.linux2.6.x86_64.rpm
mysql-shared-compat-5.5.39-2.linux2.6.x86_64.rpm
拷贝到该目录下 /home/hdpsrc/Desktop/mysql/
二,安装mysql
1,卸载系统自带的mysql相关安装包,仅卸载 mysql 开头的包
rpm -qa|grep mysql
sudo rpm -e --nodeps mysql-libs-5.1.71-1.el6.x86_64
2,安装
cd /home/hdpsrc/Desktop/mysql/
sudo rpm -ivh mysql-*
sudo cp /usr/share/mysql/my-large.cnf /etc/my.cnf
3,启动设置mysql
启动mysql服务
sudo service mysql start
设置为开机自启动
sudo chkconfig mysql on
设置root用户登录密码
sudo /usr/bin/mysqladmin -u root password 'wu123'
登录mysql 以root用户身份登录
mysql -uroot -pwu123
创建hive用户,数据库等
insert into mysql.user(Host,User,Password) values("localhost","hive",password("hive"));
create database hive;
grant all on hive.* to hive@'%' identified by 'hive';
grant all on hive.* to hive@'localhost' identified by 'hive';
flush privileges;
退出mysql
exit
验证hive用户
mysql -uhive -phive
show databases;
看到如下反馈信息,则说明创建成功
mysql> show databases;
+--------------------+
| Database |
+--------------------+
| information_schema |
| hive |
| test |
+--------------------+
3 rows in set (0.00 sec)
退出mysql
exit
三,安装hive
1,解压安装包
cd ~
tar -zxvf apache-hive-1.1.0-bin.tar.gz
2,建立软连接
ln -s apache-hive-1.1.0-bin hive
3,添加环境变量
vi .bash_profile
导入下面的环境变量
export HIVE_HOME=/home/hdpsrc/hive
export PATH=$PATH:$HIVE_HOME/bin
使其有效
source .bash_profile
4,修改hive-site.xml
主要修改以下参数
<property>
<name>javax.jdo.option.ConnectionURL </name>
<value>jdbc:mysql://localhost:3306/hive </value>
</property>
<property>
<name>javax.jdo.option.ConnectionDriverName </name>
<value>com.mysql.jdbc.Driver </value>
</property>
<property>
<name>javax.jdo.option.ConnectionPassword </name>
<value>hive </value>
</property>
<property>
<name>hive.hwi.listen.port </name>
<value>9999 </value>
<description>This is the port the Hive Web Interface will listen on </descript ion>
</property>
<property>
<name>datanucleus.autoCreateSchema </name>
<value>true</value>
</property>
<property>
<name>datanucleus.fixedDatastore </name>
<value>false</value>
</property>
</property>
<property>
<name>javax.jdo.option.ConnectionUserName</name>
<value>hive</value>
<description>Username to use against metastore database</description>
</property>
<property>
<name>hive.exec.local.scratchdir</name>
<value>/home/hdpsrc/hive/iotmp</value>
<description>Local scratch space for Hive jobs</description>
</property>
<property>
<name>hive.downloaded.resources.dir</name>
<value>/home/hdpsrc/hive/iotmp</value>
<description>Temporary local directory for added resources in the remote file system.</description>
</property>
<property>
<name>hive.querylog.location</name>
<value>/home/hdpsrc/hive/iotmp</value>
<description>Location of Hive run time structured log file</description>
</property>
cp hive/conf/hive-default.xml.template hive/conf/hive-site.xml
编辑hive-site.xml
5,拷贝mysql-connector-java-5.1.6-bin.jar 到hive 的lib下面
mv /home/hdpsrc/Desktop/mysql-connector-java-5.1.6-bin.jar /home/hdpsrc/hive/lib/
6,把jline-2.12.jar拷贝到hadoop相应的目录下,替代jline-0.9.94.jar,否则启动会报错
cp /home/hdpsrc/hive/lib/jline-2.12.jar /home/hdpsrc/hadoop-2.6.0/share/hadoop/yarn/lib/
mv /home/hdpsrc/hadoop-2.6.0/share/hadoop/yarn/lib/jline-0.9.94.jar /home/hdpsrc/hadoop-2.6.0/share/hadoop/yarn/lib/jline-0.9.94.jar.bak /
7,穿件hive临时文件夹
mkdir /home/hdpsrc/hive/iotmp
四,启动测试hive
启动hadoop后,执行hive命令
hive
测试输入 show database;
hive> show databases;
OK
default
Time taken: 0.907 seconds, Fetched: 1 row(s)
遇到问题总结希望可以帮助遇到此类问题的人。
建议先建元数据库,设置编码latin1。否则建好元数据相关可能会出问题,如drop table 卡死, create table too long等等
hive对utf-8支持不好。设置完编码latin1,发现table 字段描述无法显示中文。修改元数据库表的字符
(1)修改表字段注解和表注解
alter table COLUMNS_V2 modify column COMMENT varchar(256) character set utf8
alter table TABLE_PARAMS modify column PARAM_VALUE varchar(4000) character set utf8
(2) 修改分区字段注解:
alter table PARTITION_PARAMS modify column PARAM_VALUE varchar(4000) character set utf8 ;
alter table PARTITION_KEYS modify column PKEY_COMMENT varchar(4000) character set utf8;
(3)修改索引注解:
alter table INDEX_PARAMS modify column PARAM_VALUE varchar(4000) character set utf8;
连接元数据设置
dbc:mysql://192.168.209.1:3306/metastore_hive_db?createDatabaseIfNotExist=true&characterEncoding=UTF-8
对于已经建好的表,不起作用。 最好安装的时候就修改编码格式。
元数据mysql远程模式配置
<property>
<name>hive.metastore.uris</name>
<value>thrift://192.168.223.129:9083</value>
<description>运行hive的主机地址及端口(特别重要ip不要弄错)</description>
</property>
启动元数据
bin/hive --service metastore &
到此hive已经安装完成
参考文档:http://www.mamicode.com/info-detail-516526.html
http://blog.csdn.net/blueheart20/article/details/38460541
备注:
1,遇到的问题:http://www.mamicode.com/info-detail-516526.html
相关文章推荐
- php五大运行模式CGI,FAST-CGI,CLI,ISAPI,APACHE模式浅谈
- org.apache.commons.lang-RandomStringUtils
- Apache Shiro 认证、授权、加密和会话管理
- Apache Thrift - 可伸缩的跨语言服务开发框架
- centos7 apache2.4 测试
- 使用POI和DynaBean导出Excel的完整例子
- Apache自带的ab测试工具的使用
- ApacheBench压get、post请求
- 初识Apache Kafka+JAVA程序实例
- [置顶] Apache-web多项目部署简单流程
- CentOS 6.3下源码安装LAMP(Linux+Apache+Mysql+Php)环境
- apache限制同一ip大并发访问及下载
- ubuntu下安装Apache + PHP + Mysql
- 使用 Apache Tiles 3 构建页面布局
- Apache Log4j 学习笔记
- Apache Hadoop 2.7.1正式版发布(稳定版)
- linux,apache,php,mysql常用的查看版本信息的方法
- windows上PHP中Apache+MySQL+PHP环境的搭建
- org.apache.catalina.loader.StandardClassLoader@1af33d6
- flask+mod-wsgi+apache在WINDOWS下的部署