Hive安装与问题处理_使用mysql

1. download

wget http://mirror.mel.bkb.net.au/pub/apache//hive/stable/hive-0.8.1.tar.gz
tar zxf hive-0.8.1.tar.gz

只需要在一个节点上安装

2. 设置环境变量

vi .bash_profile

export JAVA_HOME=/usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0.x86_64/jre

export HADOOP_HOME=/home/hadoop/hadoop-1.0.0

export HIVE_HOME=/home/hadoop/hive-0.8.1

export HADOOP_CONF_DIR=$HOME/conf

export HIVE_CONF_DIR=$HOME/hive-conf

export CLASSPATH=$HIVE_HOME/lib:$JAVA_HOME/lib:$JAVA_HOME/jre/lib:$HADOOP_HOME

export PATH=$HIVE_HOME/bin:$HADOOP_HOME/bin:$JAVA_HOME/bin:/sbin/:/bin:$PATH

3. 配置hive

cp -r hive-0.8.1/conf $HIVE_CONF_DIR/

cd $HIVE_CONF_DIR/

cp hive-default.xml.template hive-default.xml

cat hive-env.sh

export HADOOP_HEAPSIZE=512

export HIVE_CONF_DIR=/home/hadoop/hive-conf

3. 测试

$ hive

hive> show tables;

OK

Time taken: 4.824 seconds

hive> create table hwz(id int, name string);

OK

Time taken: 0.566 seconds

hive> select * from hwz;

OK

Time taken: 0.361 seconds

$ hadoop dfs -lsr /user/hive

Warning: $HADOOP_HOME is deprecated.

drwxr-xr-x - hadoop supergroup 0 2012-03-22 12:36 /user/hive/warehouse

drwxr-xr-x - hadoop supergroup 0 2012-03-22 12:36 /user/hive/warehouse/hwz

4. 配置Metastore用mysql数据库,这样才可以多用户同时访问

a. create user and database for hive in mysql

create database hive;

GRANT all ON hive.* TO hive@'%' IDENTIFIED BY 'hivepass';

b. change metastore to use mysql

cat hive-site.xml

<configuration>

<property>

<name>hive.metastore.local</name>

<value>true</value>

</property>

<property>

<name>javax.jdo.option.ConnectionURL</name>

<value>jdbc:mysql://slave1:3306/hive?createDatabaseIfNotExist=true</value>

</property>

<property>

<name>javax.jdo.option.ConnectionDriverName</name>

<value>com.mysql.jdbc.Driver</value>

</property>

<property>

<name>javax.jdo.option.ConnectionUserName</name>

<value>hive</value>

</property>

<property>

<name>javax.jdo.option.ConnectionPassword</name>

<value>hivepass</value>

</property>

</configuration>

c. 检查

$ hive

hive> use dw2;

OK

Time taken: 3.43 seconds

hive> create table hwz2(id int, name string ) row format delimited FIELDS TERMINATED BY ',';

OK

Time taken: 2.519 seconds

hive> show tables;

OK

hwz2

Time taken: 0.419 seconds

hive> load data local inpath 'demo.txt' overwrite into table hwz2;

Copying data from file:/home/hadoop/demo.txt

Copying file: file:/home/hadoop/demo.txt

Loading data to table dw2.hwz2

Deleted hdfs://master:9000/user/hive/warehouse/dw2.db/hwz2

OK

Time taken: 0.557 seconds

hive> select * from hwz2;

OK

12 jack

12 jack

12 jack

12 jack

12 jack

12 jack

12 jack

12 jack

$ hadoop dfs -lsr /user/hive

Warning: $HADOOP_HOME is deprecated.

drwxr-xr-x - hadoop supergroup 0 2012-03-22 15:36 /user/hive/warehouse

drwxr-xr-x - hadoop supergroup 0 2012-03-22 15:48 /user/hive/warehouse/dw2.db

drwxr-xr-x - hadoop supergroup 0 2012-03-22 15:48 /user/hive/warehouse/dw2.db/hwz2

-rw-r--r-- 2 hadoop supergroup 1201 2012-03-22 15:48 /user/hive/warehouse/dw2.db/hwz2/demo.txt

drwxr-xr-x - hadoop supergroup 0 2012-03-22 12:36 /user/hive/warehouse/hwz

drwxr-xr-x - hadoop supergroup 0 2012-03-22 15:36 /user/hive/warehouse/hwz2

-rw-r--r-- 2 hadoop supergroup 1201 2012-03-22 15:36 /user/hive/warehouse/hwz2/demo.txt

$ hadoop dfs -cat /user/hive/warehouse/dw2.db/hwz2/demo.txt |head

Warning: $HADOOP_HOME is deprecated.

12,jack

12,jack

12,jack

12,jack

12,jack

12,jack

12,jack

12,jack

12,jack

12,jack

d. 在mysql中验证建立的新表

mysql> use hive;

Database changed

mysql> show tables;

+-----------------+

| Tables_in_hive |

+-----------------+

| BUCKETING_COLS |

| CDS |

| COLUMNS_V2 |

| DATABASE_PARAMS |

| DBS |

| PARTITION_KEYS |

| SDS |

| SD_PARAMS |

| SEQUENCE_TABLE |

| SERDES |

| SERDE_PARAMS |

| SORT_COLS |

| TABLE_PARAMS |

| TBLS |

+-----------------+

14 rows in set (0.00 sec)

9. 常见错误

error 1:

-------------------------------------------------

hive> show tables;

FAILED: Error in metadata: javax.jdo.JDOFatalInternalException: Error creating transactional connection factory

Solution:

Hive不带mysql JDBC驱动,自己安装:

wget http://dev.mysql.com/get/Downloads/Connector-J/mysql-connector-java-5.1.18.tar.gz/from/http://mysql.mirror.kangaroot.net/
tar zxf mysql-connector-java-5.1.18.tar.gz

cd mysql-connector-java-5.1.18

cp mysql-connector*.jar $HIVE_HOME/lib

error 2:

-------------------------------------------------

hive> show tables;

FAILED: Error in metadata: javax.jdo.JDOException: Couldnt obtain a new sequence (unique id) : Cannot execute statement: impossible to write to binary log since BINLOG_FORMAT = STATEMENT and at least one table uses a storage engine limited to row-based logging.
InnoDB is limited to row-logging when transaction isolation level is READ COMMITTED or READ UNCOMMITTED.

Solution:

在mysql中设置 binlog_format='MIXED'

error3：

-------------------------------------------------
[root@Master hive]# hive

14/11/25 11:40:37 INFO Configuration.deprecation: mapred.reduce.tasks is deprecated. Instead, use mapreduce.job.reduces

14/11/25 11:40:37 INFO Configuration.deprecation: mapred.min.split.size is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize

14/11/25 11:40:37 INFO Configuration.deprecation: mapred.reduce.tasks.speculative.execution is deprecated. Instead, use mapreduce.reduce.speculative

14/11/25 11:40:37 INFO Configuration.deprecation: mapred.min.split.size.per.node is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize.per.node

14/11/25 11:40:37 INFO Configuration.deprecation: mapred.input.dir.recursive is deprecated. Instead, use mapreduce.input.fileinputformat.input.dir.recursive

14/11/25 11:40:37 INFO Configuration.deprecation: mapred.min.split.size.per.rack is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize.per.rack

14/11/25 11:40:37 INFO Configuration.deprecation: mapred.max.split.size is deprecated. Instead, use mapreduce.input.fileinputformat.split.maxsize

14/11/25 11:40:37 INFO Configuration.deprecation: mapred.committer.job.setup.cleanup.needed is deprecated. Instead, use mapreduce.job.committer.setup.cleanup.needed

Logging initialized using configuration in file:/home/hive/conf/hive-log4j.properties

SLF4J: Class path contains multiple SLF4J bindings.

SLF4J: Found binding in [jar:file:/home/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: Found binding in [jar:file:/home/hive/lib/hive-jdbc-0.14.0-standalone.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.

SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]

Exception in thread "main" java.lang.RuntimeException: java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative path in absolute URI: ${system:java.io.tmpdir%7D/$%7Bsystem:user.name%7D

at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:444)

at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:672)

at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:616)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:601)

at org.apache.hadoop.util.RunJar.main(RunJar.java:212)

Caused by: java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative path in absolute URI: ${system:java.io.tmpdir%7D/$%7Bsystem:user.name%7D

at org.apache.hadoop.fs.Path.initialize(Path.java:206)

at org.apache.hadoop.fs.Path.<init>(Path.java:172)

at org.apache.hadoop.hive.ql.session.SessionState.createSessionDirs(SessionState.java:487)

at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:430)

... 7 more

Caused by: java.net.URISyntaxException: Relative path in absolute URI: ${system:java.io.tmpdir%7D/$%7Bsystem:user.name%7D

at java.net.URI.checkPath(URI.java:1804)

at java.net.URI.<init>(URI.java:752)

at org.apache.hadoop.fs.Path.initialize(Path.java:203)

... 10 more

解决方法：
修改配置文件：
hive-default.xml
hive-exec-log4j.properties
hive-log4j.properties
hive-site.xml
讲${system:java.io.tmpdir}，替换为/home/hive/tmp/