java+scala+zookpeeper+flume+kafka+maven+hadoop+hbase+hive+spark安装流程

系统版本使用的是centos7.0

开始安装之前需要先配置下hosts

vi /etc/hosts
自己的IP hadoop000

关闭防火墙 systemctl stop firewalld.service

java使用的是1.8

java下载地址 http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html
haddop使用的版本都是cdh5.7.0版本

cdh5地址http://archive.cloudera.com/cdh5/cdh/5/

下面是我整理出来的下载地址,可以直接使用

scala https://downloads.lightbend.com/scala/2.11.8/scala-2.11.8.tgz
zookpeeper http://archive.cloudera.com/cdh5/cdh/5/zookeeper-3.4.5-cdh5.7.0.tar.gz
flume http://archive.cloudera.com/cdh5/cdh/5/flume-ng-1.6.0-cdh5.7.0.tar.gz
kafka https://archive.apache.org/dist/kafka/0.9.0.0/kafka_2.11-0.9.0.0.tgz
maven https://mirrors.tuna.tsinghua.edu.cn/apache//maven/maven-3/3.3.9/binaries/apache-maven-3.3.9-bin.tar.gz
hadoop http://archive.cloudera.com/cdh5/cdh/5/hadoop-2.6.0-cdh5.7.0.tar.gz
hbase http://archive.cloudera.com/cdh5/cdh/5/hbase-1.2.0-cdh5.7.0.tar.gz
hive http://archive.cloudera.com/cdh5/cdh/5/hive-1.1.0-cdh5.7.0.tar.gz
spark http://mirror.bit.edu.cn/apache/spark/spark-2.2.0/spark-2.2.0.tgz
安装包存放在/root/tools下，解压后的应用存放在/root/app

一、安装java

    cd /root/tools

    解压 tar -zxvf jdk-8u151-linux-x64.tar.gz -C /root/app

    将java配置系统环境变量中: ~/.bash_profile
export JAVA_HOME=/root/app/jdk1.8.0_151
export PATH=$JAVA_HOME/bin:$PATH

    source ~/.bash_profile 让其配置生效

    检测安装是否正确: java  -version

二、安装scala

    解压 tar -zxvf scala-2.11.8.tgz -C /root/app/

    将scala配置系统环境变量中: ~/.bash_profile
export SCALA_HOME=/root/app/scala-2.11.8
export PATH=$SCALA_HOME/bin:$PATH

    source ~/.bash_profile 让其配置生效

    检测安装是否正确: scala  -version

三、安装zookeeper

    解压 tar -zxvf zookeeper-3.4.5-cdh5.7.0.tar.gz -C /root/app/

    将zookeeper配置系统环境变量中: ~/.bash_profile
export ZOOKEEPER_HOME=/root/app/zookeeper-3.4.5-cdh5.7.0
export PATH=$ZOOKEEPER_HOME/bin:$PATH

    source ~/.bash_profile 让其配置生效

    修改zookeeper配置文件
cd /root/app/zookeeper-3.4.5-cdh5.7.0/conf
cp zoo_sample.cfg zoo.cfg
vi zoo.cfg
修改zookeeper的文件存储地址
dataDir=/root/tmp/zookeeper
mkdir -p /root/tmp/zookeeper
启动zookeeper zkServer.sh start
jps 检测是否启动成功。出现QuorumPeerMain证明启动成功

四、安装flume
cd /root/tools
解压 tar -zxvf flume-ng-1.6.0-cdh5.7.0.tar.gz -C /root/app/
将flume配置系统环境变量中: ~/.bash_profile
export FLUME_HOME=/root/app/apache-flume-1.6.0-cdh5.7.0-bin
export PATH=$FLUME_HOME/bin:$PATH

    source ~/.bash_profile 让其配置生效
检测安装是否正确: flume-ng version

五、安装kafka
解压 tar -zxvf kafka_2.11-0.9.0.0.tgz -C /root/app/
将kafka配置系统环境变量中: ~/.bash_profile
export KAFKA_HOME=/root/app/kafka_2.11-0.9.0.0
export PATH=$KAFKA_HOME/bin:$PATH

    source ~/.bash_profile 让其配置生效
cd /root/app/kafka_2.11-0.9.0.0/config
修改配置文件:
vi server.properties
host.name=hadoop000
log.dirs=/root/tmp/kafka-logs
zookeeper.connect=hadoop000:2181
mkdir -p /root/tmp/kafka-logs
运行kafka-server-start.sh启动kafka
命令说明 kafka-server-start.sh [-daemon] server.properties [--override property=value]
启动kafka kafka-server-start.sh $KAFKA_HOME/config/server.properties

六、安装maven
解压 tar -zxvf apache-maven-3.3.9-bin.tar.gz -C /root/app/
将maven配置系统环境变量中: ~/.bash_profile
export MANVEN_HOME=/root/app/apache-maven-3.3.9
export PATH=$MANVEN_HOME/bin:$PATH
source ~/.bash_profile 让其配置生效
检测安装是否正确：mvn -version

七、安装hadoop
配置ssh无密码登录
ssh-keygen -t rsa
cp ~/.ssh/id_rsa.pub ~/.ssh/authorized_keys
解压tar -zxvf hadoop-2.6.0-cdh5.7.0.tar.gz -C /root/app/
将hadoop配置系统环境变量中: ~/.bash_profile
export HADOOP_HOME=/root/app/hadoop-2.6.0-cdh5.7.0
export PATH=$HADOOP_HOME/bin:$PATH
source ~/.bash_profile 让其配置生效
配置hadoop配置文件
进入hadoop目录 cd /root/app/hadoop-2.6.0-cdh5.7.0/etc/hadoop
vi hadoop_env.sh
export JAVA_HOME=/root/app/jdk1.8.0_151
创建hadoop临时文件
mkdir /root/tmp/hadoop
vi core-site.xml
把下面的内容放到<configuration></configuration>中
<property>

                <name>fs.defaultFS</name>

                <value>hdfs://hadoop000:8020</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>/root/tmp/hadoop</value>
</property>
vi hdfs-site.xml
修改hdfs的副本系数
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
格式化hdfs
hdfs namenode -format
进入hadoop根目录
cd /root/app/hadoop-2.6.0-cdh5.7.0
启动hdfs
sbin/start-dfs.sh
检测hdfs是否启动成功
jps 出现NameNode和DataNode证明启动成功
配置yarn
cd etc/hadoop
cp mapred-site.xml.template mapred-site.xml
vi mapred-site.xml
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
vi yarn-site.xml
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
启动yarn
cd ../../
sbin/start-yarn.sh
检测yarn是否启动成功
jps 出现NodeManager和ResourceManager证明启动成功

八、安装hbase
cd /root/tools
解压tar -zxvf hbase-1.2.0-cdh5.7.0.tar.gz -C /root/app/
将hbase配置系统环境变量中: ~/.bash_profile
export HBASE_HOME=/root/app/hbase-1.2.0-cdh5.7.0
export PATH=$HBASE_HOME/bin:$PATH
source ~/.bash_profile 让其配置生效
修改hbase配置文件
vi $HBASE_HOME/conf/hbase-env.sh
export JAVA_HOME=/root/app/jdk1.8.0_151
export HBASE_MANAGES_ZK=false

vi $HBASE_HOME/conf/hbase-site.xml
<property>
<name>hbase.rootdir</name>
<value>hdfs://hadoop001:8020/hbase</value>
</property>
<property>
<name>hbase.cluster.distributed</name>
<value>true</value>
</property>
<property>
<name>hbase.zookeeper.quorum</name>
<value>hadoop001:2181</value>
</property>
启动hbase
start-hbase.sh
验证hbase启动是否成功
jps 出现HMaster和HRegionServer证明启动成功

九、安装hive
解压tar -zxvf hive-1.1.0-cdh5.7.0.tar.gz -C /root/app/
将hive配置系统环境变量中: ~/.bash_profile
export HIVE_HOME=/root/app/hive-1.1.0-cdh5.7.0
export PATH=$HIVE_HOME/bin:$PATH
source ~/.bash_profile 让其配置生效

十、安装spark
解压tar -zxvf hive-1.1.0-cdh5.7.0.tar.gz -C /root/source
cd /root/source/spark-2.2.0
编译 ./dev/make-distribution.sh --name 2.6.0-cdh5.7.0 --tgz  -Pyarn -Phadoop-2.6 -Phive -Phive-thriftserver -Dhadoop.version=2.6.0-cdh5.7.0
编译报错：failed to execute goal on project spark-launcher_2.11: Could not resolve dependencies for project org.apache.spark:spark-launcher_2.11:jar:2.2.0: Could not find artifact org.apache.hadoop:hadoop-client:jar:2.6.0-cdh5.7.0
in central (https://repo1.maven.org/maven2) -> [Help 1]
解决办法: 在pom.xml添加:
<repository>
<id>cloudera</id>
<url>https://repository.cloudera.com/artifactory/cloudera-repos/</url>
</repository>
编译了一天一夜都没有好,无奈,直接下载官方编译好的版本吧!
cd /root/tools
下载编译好的版本:wget http://mirrors.hust.edu.cn/apache/spark/spark-2.2.0/spark-2.2.0-bin-hadoop2.6.tgz 解压tar -zxvf spark-2.2.0-bin-hadoop2.6.tgz -C /root/app
将spark配置系统环境变量中: ~/.bash_profile
export SPARK_HOME=/root/app/spark-2.2.0-bin-hadoop2.6
export PATH=$SPARK_HOME/bin:$PATH
source ~/.bash_profile 让其配置生效