impala安装

参考：http://blog.yuanxiaolong.cn/blog/2015/12/27/install-impala/

1.首先安装impala依赖包

添加repo使用yum安装

sudo wget -O /etc/yum.repos.d/bigtop.repo http://www.apache.org/dist/bigtop/bigtop-1.1.0/repos/centos6/bigtop.repo
sudo yum install bigtop-utils

2.下载rpm包
http://archive.cloudera.com/cdh5/redhat/6/x86_64/cdh/5.7.1/RPMS/x86_64/
impala-2.2.0+cdh5.4.0+0-1.cdh5.4.0.p0.75.el6.x86_64.rpm
impala-state-store-2.2.0+cdh5.4.0+0-1.cdh5.4.0.p0.75.el6.x86_64.rpm
impala-shell-2.2.0+cdh5.4.0+0-1.cdh5.4.0.p0.75.el6.x86_64.rpm
impala-catalog-2.2.0+cdh5.4.0+0-1.cdh5.4.0.p0.75.el6.x86_64.rpm
impala-server-2.2.0+cdh5.4.0+0-1.cdh5.4.0.p0.75.el6.x86_64.rpm
impala-udf-devel-2.2.0+cdh5.4.0+0-1.cdh5.4.0.p0.75.el6.x86_64.rpm
impala-debuginfo-2.2.0+cdh5.4.0+0-1.cdh5.4.0.p0.75.el6.x86_64.rpm (可不装)

3.安装

安装，建议上面的顺序

在namenode上安装state-store和catalog服务，在datanode上安装impala-server服务，impala基础服务每台机器都要装

在装impala基础服务时必须使用--force --nodeps，因为rmp包会检查依赖环境

sudo rpm -ivh impala-2.2.0+cdh5.4.0+0-1.cdh5.4.0.p0.75.el6.x86_64.rpm --nodeps --force

由于我的hadoop是通过tar包安装的，所以需要强制安装此rpm，其余正常安装

impala默认的安装目录为

/usr/lib/impala

，jar包地址为

/usr/lib/impala/lib/

4.删除默认的不存在软连接，并添加新连接。

sudo rm -rf /usr/lib/impala/lib/avro*.jar
sudo rm -rf /usr/lib/impala/lib/hadoop-*.jar
sudo rm -rf /usr/lib/impala/lib/hive-*.jar
sudo rm -rf /usr/lib/impala/lib/hbase-*.jar
sudo rm -rf /usr/lib/impala/lib/parquet-hadoop-bundle.jar
sudo rm -rf /usr/lib/impala/lib/sentry-*.jar
sudo rm -rf /usr/lib/impala/lib/zookeeper.jar
sudo rm -rf /usr/lib/impala/lib/libhadoop.so
sudo rm -rf /usr/lib/impala/lib/libhadoop.so.1.0.0
sudo rm -rf /usr/lib/impala/lib/libhdfs.so
sudo rm -rf /usr/lib/impala/lib/libhdfs.so.0.0.0

5.建立新连接，建议利用脚本运行，自己设一下各个home即可，软连后的名字不能改。

sudo ln -s $HBASE_HOME/lib/avro-1.7.6-cdh5.4.0.jar /usr/lib/impala/lib/avro.jar
sudo ln -s $HADOOP_HOME/share/hadoop/common/lib/hadoop-annotations-2.6.0-cdh5.4.0.jar /usr/lib/impala/lib/hadoop-annotations.jar
sudo ln -s $HADOOP_HOME/share/hadoop/common/lib/hadoop-auth-2.6.0-cdh5.4.0.jar /usr/lib/impala/lib/hadoop-auth.jar
sudo ln -s $HADOOP_HOME/share/hadoop/tools/lib/hadoop-aws-2.6.0-cdh5.4.0.jar /usr/lib/impala/lib/hadoop-aws.jar
sudo ln -s $HADOOP_HOME/share/hadoop/common/hadoop-common-2.6.0-cdh5.4.0.jar /usr/lib/impala/lib/hadoop-common.jar
sudo ln -s $HADOOP_HOME/share/hadoop/hdfs/hadoop-hdfs-2.6.0-cdh5.4.0.jar /usr/lib/impala/lib/hadoop-hdfs.jar
sudo ln -s $HADOOP_HOME/share/hadoop/mapreduce/hadoop-mapreduce-client-common-2.6.0-cdh5.4.0.jar /usr/lib/impala/lib/hadoop-mapreduce-client-common.jar
sudo ln -s $HADOOP_HOME/share/hadoop/mapreduce/hadoop-mapreduce-client-core-2.6.0-cdh5.4.0.jar /usr/lib/impala/lib/hadoop-mapreduce-client-core.jar
sudo ln -s $HADOOP_HOME/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.6.0-cdh5.4.0.jar /usr/lib/impala/lib/hadoop-mapreduce-client-jobclient.jar
sudo ln -s $HADOOP_HOME/share/hadoop/mapreduce/hadoop-mapreduce-client-shuffle-2.6.0-cdh5.4.0.jar /usr/lib/impala/lib/hadoop-mapreduce-client-shuffle.jar
sudo ln -s $HADOOP_HOME/share/hadoop/yarn/hadoop-yarn-api-2.6.0-cdh5.4.0.jar /usr/lib/impala/lib/hadoop-yarn-api.jar
sudo ln -s $HADOOP_HOME/share/hadoop/yarn/hadoop-yarn-client-2.6.0-cdh5.4.0.jar /usr/lib/impala/lib/hadoop-yarn-client.jar
sudo ln -s $HADOOP_HOME/share/hadoop/yarn/hadoop-yarn-common-2.6.0-cdh5.4.0.jar /usr/lib/impala/lib/hadoop-yarn-common.jar
sudo ln -s $HADOOP_HOME/share/hadoop/yarn/hadoop-yarn-server-applicationhistoryservice-2.6.0-cdh5.4.0.jar /usr/lib/impala/lib/hadoop-yarn-server-applicationhistoryservice.jar
sudo ln -s $HADOOP_HOME/share/hadoop/yarn/hadoop-yarn-server-common-2.6.0-cdh5.4.0.jar /usr/lib/impala/lib/hadoop-yarn-server-common.jar
sudo ln -s $HADOOP_HOME/share/hadoop/yarn/hadoop-yarn-server-nodemanager-2.6.0-cdh5.4.0.jar /usr/lib/impala/lib/hadoop-yarn-server-nodemanager.jar
sudo ln -s $HADOOP_HOME/share/hadoop/yarn/hadoop-yarn-server-resourcemanager-2.6.0-cdh5.4.0.jar /usr/lib/impala/lib/hadoop-yarn-server-resourcemanager.jar
sudo ln -s $HADOOP_HOME/share/hadoop/yarn/hadoop-yarn-server-web-proxy-2.6.0-cdh5.4.0.jar /usr/lib/impala/lib/hadoop-yarn-server-web-proxy.jar

sudo ln -s $HBASE_HOME/lib/hbase-annotations-1.0.0-cdh5.4.0.jar /usr/lib/impala/lib/hbase-annotations.jar
sudo ln -s $HBASE_HOME/lib/hbase-client-1.0.0-cdh5.4.0.jar /usr/lib/impala/lib/hbase-client.jar
sudo ln -s $HBASE_HOME/lib/hbase-common-1.0.0-cdh5.4.0.jar /usr/lib/impala/lib/hbase-common.jar
sudo ln -s $HBASE_HOME/lib/hbase-protocol-1.0.0-cdh5.4.0.jar /usr/lib/impala/lib/hbase-protocol.jar

sudo ln -s $HIVE_HOME/lib/hive-ant-1.1.0-cdh5.4.0.jar /usr/lib/impala/lib/hive-ant.jar
sudo ln -s $HIVE_HOME/lib/hive-beeline-1.1.0-cdh5.4.0.jar /usr/lib/impala/lib/hive-beeline.jar
sudo ln -s $HIVE_HOME/lib/hive-common-1.1.0-cdh5.4.0.jar /usr/lib/impala/lib/hive-common.jar
sudo ln -s $HIVE_HOME/lib/hive-exec-1.1.0-cdh5.4.0.jar /usr/lib/impala/lib/hive-exec.jar
sudo ln -s $HIVE_HOME/lib/hive-hbase-handler-1.1.0-cdh5.4.0.jar /usr/lib/impala/lib/hive-hbase-handler.jar
sudo ln -s $HIVE_HOME/lib/hive-metastore-1.1.0-cdh5.4.0.jar /usr/lib/impala/lib/hive-metastore.jar
sudo ln -s $HIVE_HOME/lib/hive-serde-1.1.0-cdh5.4.0.jar /usr/lib/impala/lib/hive-serde.jar
sudo ln -s $HIVE_HOME/lib/hive-service-1.1.0-cdh5.4.0.jar /usr/lib/impala/lib/hive-service.jar
sudo ln -s $HIVE_HOME/lib/hive-shims-common-1.1.0-cdh5.4.0.jar /usr/lib/impala/lib/hive-shims-common.jar
sudo ln -s $HIVE_HOME/lib/hive-shims-1.1.0-cdh5.4.0.jar /usr/lib/impala/lib/hive-shims.jar
sudo ln -s $HIVE_HOME/lib/hive-shims-scheduler-1.1.0-cdh5.4.0.jar /usr/lib/impala/lib/hive-shims-scheduler.jar

sudo ln -s $HADOOP_HOME/lib/native/libhadoop.so /usr/lib/impala/lib/libhadoop.so
sudo ln -s $HADOOP_HOME/lib/native/libhadoop.so.1.0.0 /usr/lib/impala/lib/libhadoop.so.1.0.0
sudo ln -s $HADOOP_HOME/lib/native/libhdfs.so /usr/lib/impala/lib/libhdfs.so
sudo ln -s $HADOOP_HOME/lib/native/libhdfs.so.0.0.0 /usr/lib/impala/lib/libhdfs.so.0.0.0

sudo ln -s $HIVE_HOME/lib/parquet-hadoop-bundle-1.5.0-cdh5.4.0.jar /usr/lib/impala/lib/parquet-hadoop-bundle.jar

sudo ln -s $SENTRY_HOME/lib/sentry-binding-hive-1.4.0-cdh5.4.0.jar /usr/lib/impala/lib/sentry-binding-hive.jar
sudo ln -s $SENTRY_HOME/lib/sentry-core-common-1.4.0-cdh5.4.0.jar /usr/lib/impala/lib/sentry-core-common.jar
sudo ln -s $SENTRY_HOME/lib/sentry-core-model-db-1.4.0-cdh5.4.0.jar /usr/lib/impala/lib/sentry-core-model-db.jar
sudo ln -s $SENTRY_HOME/lib/sentry-policy-common-1.4.0-cdh5.4.0.jar /usr/lib/impala/lib/sentry-policy-common.jar
sudo ln -s $SENTRY_HOME/lib/sentry-policy-db-1.4.0-cdh5.4.0.jar /usr/lib/impala/lib/sentry-policy-db.jar
sudo ln -s $SENTRY_HOME/lib/sentry-provider-cache-1.4.0-cdh5.4.0.jar /usr/lib/impala/lib/sentry-provider-cache.jar
sudo ln -s $SENTRY_HOME/lib/sentry-provider-common-1.4.0-cdh5.4.0.jar /usr/lib/impala/lib/sentry-provider-common.jar
sudo ln -s $SENTRY_HOME/lib/sentry-provider-db-1.4.0-cdh5.4.0.jar /usr/lib/impala/lib/sentry-provider-db.jar
sudo ln -s $SENTRY_HOME/lib/sentry-provider-file-1.4.0-cdh5.4.0.jar /usr/lib/impala/lib/sentry-provider-file.jar

sudo ln -s $ZOOKEEPER_HOME/zookeeper-3.4.5-cdh5.3.3.jar /usr/lib/impala/lib/zookeeper.jar

6.修改/etc/default/bigtop-utils

export JAVA_HOME=/usr/local/datacenter/jdk1.7.0_79

7.修改 /etc/default/impala 默认文件

除了修改环境，还需要修改启动参数，显示传递给 statestored 和 catalogd 它们各自需要的ip和port ，不然都默认 localhost ，分开部署会有问题的。

IMPALA_CATALOG_SERVICE_HOST=192.168.7.11
IMPALA_STATE_STORE_HOST=192.168.7.11

IMPALA_CATALOG_ARGS=" -log_dir=${IMPALA_LOG_DIR} -catalog_service_host=${IMPALA_CATALOG_SERVICE_HOST} -state_store_host=${IMPALA_STATE_STORE_HOST}"
IMPALA_STATE_STORE_ARGS=" -log_dir=${IMPALA_LOG_DIR} -state_store_host=${IMPALA_STATE_STORE_HOST} -state_store_port=${IMPALA_STATE_STORE_PORT} -catalog_service_host=${IMPALA_CATALOG_SERVICE_HOST} -catalog_service_port=${CATALOG_SERVICE_PORT} "

MYSQL_CONNECTOR_JAR=/usr/local/datacenter/hive/lib/mysql-connector-java-5.1.34.jar

8.根据实际环境修改impala相关脚本文件

/etc/init.d/impala-state-store、/etc/init.d/impala-server、/etc/init.d/impala-catalog，修改其中两处跟用户相关的地方

###编者注：这里默认是impala
SVC_USER="hadoop"
###编者注：这里默认是impala
install -d -m 0755 -o hadoop -g hadoop /var/run/impala 1>/dev/null 2>&1 || :

为保险起见 手工建立日志目录并修改权限 mkdir /var/log/impala && chmod -R 777 /var/log/impala

9.修改hdfs-site.xml

<property>
<name>dfs.client.read.shortcircuit</name>
<value>true</value>
</property>
<property>
<name>dfs.domain.socket.path</name>
<value>/var/run/hadoop-hdfs/dn</value>
</property>
<property>
<name>dfs.datanode.hdfs-blocks-metadata.enabled</name>
<value>true</value>
</property>
<property>
<name>dfs.client.use.legacy.blockreader.local</name>
<value>false</value> ###编者注：这里需要为false ，官网文档错了
</property>
<property>
<name>dfs.datanode.data.dir.perm</name>
<value>750</value>
</property>
<property>
<name>dfs.block.local-path-access.user</name>
<value>hadoop</value>
</property>
<property>
<name>dfs.client.file-block-storage-locations.timeout</name>
<value>3000</value>
</property>

10.最后启动各项服务

sudo service impala-state-store start
sudo service impala-catalog start
datenode启动sudo service impala-server start

会遇到的问题：

1.Unsupported major.minor version 51.0

解决：/etc/default/bigtop-utils 配置JAVA_HOME

2.mysql jdbc 找不到

解决：I fixed this by copying mysql-connector-java-5.1.25-bin.jar to /var/lib/impala - the startup script was telling the classpath to look here for the connector jar for some reason.

包放在 /var/lib/impala 路径下

3.dfs.client.file-block-storage-locations.timeout 要求至少10s

解决：改成10s