spark standalone集群模式搭建

搭建前，必须准备好hadoop

#scala environment

export SCALA_HOME=/opt/scala-2.10.6

export PATH=$PATH:$SCALA_HOME/bin

centos-master

centos-slave1

centos-slave2

#spark config

SPARK_HOME=/opt/spark-1.6.0

PATH=PATH:$SPARK_HOME/bin:$SPARK_HOME/sbin

配置conf/spark-env.sh

export JAVA_HOME=/usr/java/latest

export SPARK_MASTER_IP=192.168.130.140

export SPARK_MASTER_PORT=7077

export SPARK_WORKER_INSTANCES=1

export SPARK_WORKER_CORES=1

export SPARK_WORKER_MEMORY=1g

分发到各个节点

scp -r /opt/spark-1.6.0 centos-slave1:/opt

scp -r /opt/spark-1.6.0 centos-slave2:/opt

spark standalone启动：

/opt/spark-1.6.0/sbin/start-all.sh