Ubuntu安装Spark和Hadoop集群

一、JDK安装

手动解压JDK的压缩包，然后设置环境变量

1.1在/usr/目录下创建java目录

root@ubuntu:~# mkdir/usr/java
root@ubuntu:~#  cd /usr/java

1.2 下载jdk,然后解压
http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html root@ubuntu:~#  tar -zxvf jdk-8u144-linux-x64.tar.gz

1.3 设置环境变量

root@ubuntu:~#  vi /etc/profile

在profile中添加如下内容:

#set java environment

JAVA_HOME=/usr/java/jdk1.8.0_144

JRE_HOME=/usr/java/jdk1.8.0_144/jre

CLASS_PATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar:$JRE_HOME/lib

PATH=$PATH:$JAVA_HOME/bin:$JRE_HOME/bin

export JAVA_HOME JRE_HOME CLASS_PATH PATH

让修改生效:

root@ubuntu:~#  source /etc/profile

1.4 验证JDK有效性

root@ubuntu:~# java -version

java version "1.8.0_144"

Java(TM) SE Runtime Environment (build 1.8.0_144-b01)

Java HotSpot(TM) 64-Bit Server VM (build 25.144-b01, mixed mode)

root@ubuntu:~# 
二、Scala安装

下载scala：http://www.scala-lang.org/download/2.11.6.html

root@ubuntu:~# cd /usr

root@ubuntu:/usr# mkdir scala

root@ubuntu:/usr# cd scala/

root@ubuntu:/usr/scala# cp /home/baihu/pragrom/scala-2.11.6.tgz .

root@ubuntu:/usr/scala# ls

scala-2.11.6.tgz

root@ubuntu:/usr/scala# tar -zxf scala-2.11.6.tgz 

root@ubuntu:/usr/scala# ls

scala-2.11.6  scala-2.11.6.tgz

root@ubuntu:/usr/scala# 

编辑配置文件添加scala的配置

[root@master scala-2.11.6]# vi /etc/profile

export SCALA_HOME=/usr/scala/scala-2.11.6

export PATH=$PATH:$SCALA_HOME/bin

root@ubuntu:/usr/scala# source /etc/profile

root@ubuntu:/usr/scala#  scala -version

Scala code runner version 2.11.6 -- Copyright 2002-2013, LAMP/EPFL

三、安装Spark

下载spark最新版spark-2.2.0-bin-hadoop2.7.tgz
http://spark.apache.org/downloads.html
root@ubuntu:/usr# mkdir spark

root@ubuntu:/usr# cd spark/

root@ubuntu:/usr/spark# mv /home/baihu/pragrom/spark-2.2.0-bin-hadoop2.7.tgz .

root@ubuntu:/usr/spark# ls

spark-2.2.0-bin-hadoop2.7.tgz

root@ubuntu:/usr/spark# tar -zxf spark-2.2.0-bin-hadoop2.7.tgz 

root@ubuntu:/usr/spark# ls

spark-2.2.0-bin-hadoop2.7  spark-2.2.0-bin-hadoop2.7.tgz

root@ubuntu:/usr/spark# cd spark-2.2.0-bin-hadoop2.7/

root@ubuntu:/usr/spark/spark-2.2.0-bin-hadoop2.7# ls

bin   data      jars     licenses  python  README.md  sbin

conf  examples  LICENSE  NOTICE    R       RELEASE    yarn

root@ubuntu:/usr/spark/spark-2.2.0-bin-hadoop2.7# cd bin/

root@ubuntu:/usr/spark/spark-2.2.0-bin-hadoop2.7/bin# ./pyspark

Python 2.7.6 (default, Jun 22 2015, 17:58:13) 

[GCC 4.8.2] on linux2

Type "help", "copyright", "credits" or "license" for more information.

Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties

Setting default log level to "WARN".

To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).

17/08/02 07:34:23 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

17/08/02 07:34:23 WARN Utils: Your hostname, ubuntu resolves to a loopback address: 127.0.1.1; using 192.168.75.130 instead (on interface eth0)

17/08/02 07:34:23 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address

17/08/02 07:34:52 WARN ObjectStore: Version information not found in metastore. hive.metastore.schema.verification is not enabled so recording the schema version 1.2.0

17/08/02 07:34:53 WARN ObjectStore: Failed to get database default, returning NoSuchObjectException

17/08/02 07:34:55 WARN ObjectStore: Failed to get database global_temp, returning NoSuchObjectException

Welcome to

      ____              __

     / __/__  ___ _____/ /__

    _\ \/ _ \/ _ `/ __/  '_/

   /__ / .__/\_,_/_/ /_/\_\   version 2.2.0

      /_/

Using Python version 2.7.6 (default, Jun 22 2015 17:58:13)

SparkSession available as 'spark'.

>>>