spark学习1--centOS7.2下基于hadoop2.7.3的spark2.0集群环境搭建
2016-11-23 17:54
1006 查看
安装步骤如没特殊指明哪台服务器,则说明都是在3台服务器上做同样的操作。
node1 192.168.174.133
node2 192.168.174.134
spark-2.0.0-bin-hadoop2.7.tgz
然后上传到3台服务器的/soft目录上
spark-env.sh增加scala_home
让环境变量生效
source /etc/profile
1.准备
1.1 装有centOS7的3台服务器
master 192.168.174.132node1 192.168.174.133
node2 192.168.174.134
1.2 搭建hadoop集群环境
参考“hadoop学习1--hadoop2.7.3集群环境搭建”1.3 准备安装包
scala-2.11.8.tgzspark-2.0.0-bin-hadoop2.7.tgz
然后上传到3台服务器的/soft目录上
2.安装scala
2.1 解压
[root@node1 soft]# mkdir -p /soft/scala [root@node1 soft]# tar -zxvf scala-2.11.8.tgz -C /soft/scala
2.2 配置环境变量
[root@node1 soft]# echo -e "export SCALA_HOME=/soft/scala/scala-2.11.8" >> /etc/profile [root@node1 soft]# echo -e "export PATH=$PATH:$SCALA_HOME/bin" >> /etc/profile
3.安装spark
3.1 解压
[root@node1 soft]# mkdir -p /soft/spark [root@node1 soft]# tar -zxvf spark-2.0.0-bin-hadoop2.7.tgz -C /soft/spark/
3.2 配置环境变量
[root@node1 soft]# echo -e "export SPARK_HOME=/soft/spark/spark-2.0.0-bin-hadoop2.7" >> /etc/profile [root@node1 soft]# echo -e "export PATH=$PATH:$SPARK_HOME/bin" >> /etc/profile
3.3 配置集群
slaves增加node1,node2[root@node1 spark-2.0.0-bin-hadoop2.7]# cd /soft/spark/spark-2.0.0-bin-hadoop2.7/conf/ [root@node1 conf]# cp slaves.template slaves [root@node1 conf]# echo -e "node1\nnode2" > slaves
spark-env.sh增加scala_home
[root@master conf]# cp spark-env.sh.template spark-env.sh
[root@master conf]# echo -e "export SCALA_HOME=/soft/scala/scala-2.11.8" >> spark-env.sh
[root@master conf]# echo -e "JAVA_HOME=/soft/java/jdk1.7.0_79" >> spark-env.sh
让环境变量生效
source /etc/profile
4.启动
启动hadoop集群,master上执行[root@master conf]# $HADOOP_HOME/sbin/start-all.sh This script is Deprecated. Instead use start-dfs.sh and start-yarn.sh Starting namenodes on [master] master: starting namenode, logging to /soft/hadoop/hadoop-2.7.3/logs/hadoop-root-namenode-master.out node1: starting datanode, logging to /soft/hadoop/hadoop-2.7.3/logs/hadoop-root-datanode-node1.out node2: starting datanode, logging to /soft/hadoop/hadoop-2.7.3/logs/hadoop-root-datanode-node2.out Starting secondary namenodes [master] master: starting secondarynamenode, logging to /soft/hadoop/hadoop-2.7.3/logs/hadoop-root-secondarynamenode-master.out starting yarn daemons starting resourcemanager, logging to /soft/hadoop/hadoop-2.7.3/logs/yarn-root-resourcemanager-master.out node1: starting nodemanager, logging to /soft/hadoop/hadoop-2.7.3/logs/yarn-root-nodemanager-node1.out node2: starting nodemanager, logging to /soft/hadoop/hadoop-2.7.3/logs/yarn-root-nodemanager-node2.out启动spark集群,master上执行
[root@master conf]# $SPARK_HOME/sbin/start-all.sh org.apache.spark.deploy.master.Master running as process 3075. Stop it first. node2: starting org.apache.spark.deploy.worker.Worker, logging to /soft/spark/spark-2.0.0-bin-hadoop2.7/logs/spark-root-org.apache.spark.deploy.worker.Worker-1-node2.out node1: starting org.apache.spark.deploy.worker.Worker, logging to /soft/spark/spark-2.0.0-bin-hadoop2.7/logs/spark-root-org.apache.spark.deploy.worker.Worker-1-node1.out [root@master conf]#
5.验证
访问master的8080,比如http://192.168.174.132:8080/相关文章推荐
- CentOS7下 Hadoop2.7.3+Spark2.1.0 集群环境搭建(1NN+2DN)
- CentOS7 基于Hadoop2.7 的Spark2.0集群搭建
- CentOS7下 Hadoop2.7.3+Spark2.1.0 集群环境搭建(1NN+2DN)
- 大数据学习环境搭建(CentOS6.9+Hadoop2.7.3+Hive1.2.1+Hbase1.3.1+Spark2.1.1)
- CentOS7 基于Hadoop2.7 的Spark2.0集群搭建
- Centos7 下 spark1.6.1_hadoop2.6 分布式集群环境搭建
- 基于docker的spark-hadoop分布式集群之一: 环境搭建
- hadoop伪分布式集群环境搭建--基于CentOS
- VirtualBox+CentOs虚拟集群搭建配置hadoop2.2.0学习环境
- Ubuntu16.04 下 Spark2.0.2+Hadoop2.7.3+Zookeeper3.4.9+HBase1.1.7集群环境搭建
- Ubuntu16.04 下 Spark2.0.2+Hadoop2.7.3+Zookeeper3.4.9+HBase1.1.7集群环境搭建--5
- Centos7 下 spark1.6.1_hadoop2.6 分布式集群环境搭建
- 基于vmware workstations 10 、centos6.4和hadoop-2.7.1的hadoop完全分布式集群的开发环境搭建
- 集群RedHat6.5+JDK1.8+Hadoop2.7.3+Spark2.1.1+zookeeper3.4.6+kafka2.11+flume1.6环境搭建步骤
- [置顶] CentOS7 基于Hadoop2.7 的Spark2.0集群搭建
- Ubuntu16.04 下 Spark2.0.2+Hadoop2.7.3+Zookeeper3.4.9+HBase1.1.7集群环境搭建--2
- Ubuntu16.04 下 Spark2.0.2+Hadoop2.7.3+Zookeeper3.4.9+HBase1.1.7集群环境搭建--4
- CentOS 6.5 hadoop 2.7.3 集群环境搭建
- Ubuntu16.04 下 Spark2.0.2+Hadoop2.7.3+Zookeeper3.4.9+HBase1.1.7集群环境搭建--3
- 基于CentOS与VmwareStation10搭建Oracle11G RAC 64集群环境:4.安装Oracle RAC FAQ-4.7.Oracle 11G R2 RAC修改public网络IP