spark编译（官方文档翻译版）

原文地址：http://spark.apache.org/docs/latest/building-spark.html#building-a-runnable-distribution

Building Apache Spark

Apache Maven

The Maven-based build is the build of reference for Apache Spark. Building Spark using Maven requires Maven 3.3.9 or newer and Java 8+. Note that support for Java 7 was removed as of Spark 2.2.0.编译spark需要maven3.3.9和java7以上。编译spark2.2.0的话，至少要java8了。

Setting up Maven’s Memory Usage

You’ll need to configure Maven to use more memory than usual by setting

MAVEN_OPTS

:

export MAVEN_OPTS="-Xmx2g -XX:ReservedCodeCacheSize=512m"

(The

ReservedCodeCacheSize

setting is optional but recommended.) If you don’t add these parameters to

MAVEN_OPTS

, you may see errors and warnings like the following:

[INFO] Compiling 203 Scala sources and 9 Java sources to /Users/me/Development/spark/core/target/scala-2.11/classes...
[ERROR] Java heap space -> [Help 1]

You can fix these problems by setting the

MAVEN_OPTS

variable as discussed before.Note:If using

build/mvn

with no

MAVEN_OPTS

set, the script will automatically add the above options to the

MAVEN_OPTS

environment variable.The

test

phase of the Spark build will automatically add these options to

MAVEN_OPTS

, even when not using

build/mvn

.编译之前最好设置一下MAVEN_OPTS，如果不设置，在使用dev/make-distribution.sh来编译时可能会出现内存不足的错误。如果使用build/mvn指令来编译，则会默认自动加上MAVEN_OPTS="-Xmx2g -XX:ReservedCodeCacheSize=512m"这个配置。

build/mvn

Spark now comes packaged with a self-contained Maven installation to ease building and deployment of Spark from source located under the

build/

directory. This script will automatically download and setup all necessary build requirements (Maven,Scala, andZinc) locally within the

build/

direct4000ory itself. It honors any

mvn

binary if present already, however, will pull down its own copy of Scala and Zinc regardless to ensure proper version requirements are met.

build/mvn

execution acts as a pass through to the

mvn

call allowing easy transition from previous build methods. As an example, one can build a version of Spark as follows:

./build/mvn -DskipTests clean package

Other build examples can be found below.在spark源码的build目录下，spark现在自带maven安装，可以很容易的编译和部署spark。build/mvn这个脚本会自动下载和设置必须的编译环境，maven，scala等。当然如果已经存在了mvn也没问题，只是可能会不满足最适当的版本需求。一个最简单例子如下：./build/mvn -DskipTests clean package

Building a Runnable Distribution

To create a Spark distribution like those distributed by theSpark Downloads page, and that is laid out so as to be runnable, use

./dev/make-distribution.sh

in the project root directory. It can be configured with Maven profile settings and so on like the direct Maven build. Example:

./dev/make-distribution.sh --name custom-spark --pip --r --tgz -Psparkr -Phadoop-2.7 -Phive -Phive-thriftserver -Pmesos -Pyarn

This will build Spark distribution along with Python pip and R packages. For more information on usage, run

./dev/make-distribution.sh --help

想要编译一个下载页面那种分布式的spark，解压展开后即可运行。就使用源码根目录下的dev/make-distribution.sh指令。它可以配置maven的profile设置，以及其他信息，就像maven直接编译一样。想要获取更多的这个指令的使用信息，执行，dev/make-distribution.sh --help。

Specifying the Hadoop Version and Enabling YARN

You can specify the exact version of Hadoop to compile against through the

hadoop.version

property. If unset, Spark will build against Hadoop 2.6.X by default.You can enable the

yarn

profile and optionally set the

yarn.version

property if it is different from

hadoop.version

.Examples:

# Apache Hadoop 2.6.X
./build/mvn -Pyarn -DskipTests clean package

# Apache Hadoop 2.7.X and later
./build/mvn -Pyarn -Phadoop-2.7 -Dhadoop.version=2.7.3 -DskipTests clean packag

指定特定的hadoop版本，并开启spark on yarn。如果不指定的话，就默认匹配hadoop2.6.x的版本，如果yarn的hadoop的版本不一样，可以用yarn.version属性来指定（但是我们下载hadoop都是hdfs和yarn是一套的，版本都是一致的）。

Building With Hive and JDBC Support

To enable Hive integration for Spark SQL along with its JDBC server and CLI, add the

-Phive

and

Phive-thriftserver

profiles to your existing build options. By default Spark will build with Hive 1.2.1 bindings.

# With Hive 1.2.1 support./build/mvn -Pyarn -Phive -Phive-thriftserver -DskipTests clean package

想要开启sparksql的hive集成，和spark的jdbc服务和CLI，就把-Phive 和-Phive-thriftserver两个配置，加到已有的编译选项上，默认编译对hive1.2.1的绑定（如果hive版本更高呢？例如hive2.1.1呢？也没说怎么办，也不知道是不是也可以将就用）。

Packaging without Hadoop Dependencies for YARN

The assembly directory produced by

mvn package

will, by default, include all of Spark’s dependencies, including Hadoop and some of its ecosystem projects. On YARN deployments, this causes multiple versions of these to appear on executor classpaths:the version packaged in the Spark assembly and the version on each node, included with

yarn.application.classpath

. The

hadoop-provided

profile builds the assembly without including Hadoop-ecosystem projects, like ZooKeeper and Hadoop itself.mvn包产生的归类目录会包含几乎所有的spark依赖，包括对Hadoop的依赖和spark生态圈中其他项目的依赖。在on yarn模式下，会导致多个这些版本出现在执行类路径下：spark assembly和每一个节点上，这些版本的包都被包含在了yarn.application.classpath中。hadoop提供的编译配置属性，就不包含hadoop生态圈中的项目，例如zookeeper和hadoop（这部分我也有点没绕清楚，惭愧！！！）

Building with Mesos support

./build/mvn -Pmesos -DskipTests clean package

编译对mesos的支持

Building for Scala 2.10

To produce a Spark package compiled with Scala 2.10, use the

-Dscala-2.10

property:

./dev/change-scala-version.sh 2.10./build/mvn -Pyarn -Dscala-2.10 -DskipTests clean package

Note that support for Scala 2.10 is deprecated as of Spark 2.1.0 and may be removed in Spark 2.2.0.编译对scala2.10的支持，用 -Dscala-2.10配置选项，注意的是，从spark2.1.0开始，对2.10就deprecated（丢弃）了，可能2.2.0之后就不再支持了

Building submodules individually

It’s possible to build Spark sub-modules using the

mvn -pl

option.For instance, you can build the Spark Streaming module using:

./build/mvn -pl :spark-streaming_2.11 clean install

where

spark-streaming_2.11

is the

artifactId

as defined in

streaming/pom.xml

file.单独安装spark子模块，使用-pl配置选项，例如安装spark-streaming 模块。

Continuous Compilation

We use the scala-maven-plugin which supports incremental and continuous compilation. E.g.

./build/mvn scala:cc

should run continuous compilation (i.e. wait for changes). However, this has not been tested extensively. A couple of gotchas to note:it only scans the paths

src/main

and

src/test

(seedocs), so it will only work from within certain submodules that have that structure.you’ll typically need to run

mvn install

from the project root for compilation within specific submodules to work; this is because submodules that depend on other submodules do so via the

spark-parent

module).Thus, the full flow for running continuous-compilation of the

core

submodule may look more like:

$ ./build/mvn install$ cd core$ ../build/mvn scala:cc

使用scala-maven-plugin插件能够支持增量的持续编译，然而没有被广泛的测试。有两点要注意，1）它值扫描src/main和src/test目录下，所以它将只对有that structure（这个我也不知道什么）的子模块起作用。2）（没弄明白- -）。

Building with SBT

Maven is the official build tool recommended for packaging Spark, and is thebuild of reference. But SBT is supported for day-to-day development since it can provide much faster iterative compilation. More advanced developers may wish to use SBT.The SBT build is derived from the Maven POM files, and so the same Maven profiles and variables can be set to control the SBT build. For example:

./build/sbt package

To avoid the overhead of launching sbt each time you need to re-compile, you can launch sbt in interactive mode by running

build/sbt

, and then run all build commands at the command prompt.maven是官方推荐的编译工具，但是sbt也被越来越多的人喜爱。sbt也可以用maven的pom文件，来控制sbt的编译。为了避免超过负荷launching sbt，你可以通过执行build/sbt来使用交互模式，再在命令行，运行所有的编译命令。

Speeding up Compilation

Developers who compile Spark frequently may want to speed up compilation; e.g., by using Zinc (for developers who build with Maven) or by avoiding re-compilation of the assembly JAR (for developers who build with SBT). For more information about how to dothis, refer to theUseful Developer Tools page.加速编译，可以通过zinc（使用maven编译时）或者避免重复编译相似的jar（使用sbt编译时）。更多信息，参考Useful Developer Tools page.

Encrypted Filesystems

When building on an encrypted filesystem (if your home directory is encrypted, for example), then the Spark build might fail with a “Filename too long” error. As a workaround, add the following in the configuration args of the

scala-maven-plugin

in the project

pom.xml

:

<arg>-Xmax-classfile-name</arg><arg>128</arg>

and in

project/SparkBuild.scala

add:

scalacOptions in Compile ++= Seq("-Xmax-classfile-name", "128"),

to the

sharedSettings

val. See alsothis PR if you are unsure of where to add these lines.当在加密的文件系统里面编译spark时，可能会报filename too long的错误，可以在pom.xml中给scala-maven-plugin加上如上参数。并且在sparkbuild.scala中机上如上配置。

IntelliJ IDEA or Eclipse

For help in setting up IntelliJ IDEA or Eclipse for Spark development, and troubleshooting, refer to theUseful Developer Tools page.要整合IDEA或者eclipse参考Useful Developer Tools page.

Running Tests

一下就是一些运行测试的，不再翻译了。估计很少人会用把。Tests are run by default via theScalaTest Maven plugin. Note that tests should not be run as root or an admin user.The following is an example of a command to run the tests:

./build/mvn test

Testing with SBT

The following is an example of a command to run the tests:

./build/sbt test

Running Individual Tests

For information about how to run individual tests, refer to theUseful Developer Tools page.

PySpark pip installable

If you are building Spark for use in a Python environment and you wish to pip install it, you will first need to build the Spark JARs as described above. Then you can construct an sdist package suitable for setup.py and pip installable package.

cd python; python setup.py sdist

Note: Due to packaging requirements you can not directly pip install from the Python directory, rather you must first build the sdist package as described above.Alternatively, you can also run make-distribution with the –pip ob10aption.

PySpark Tests with Maven

If you are building PySpark and wish to run the PySpark tests you will need to build Spark with Hive support.

./build/mvn -DskipTests clean package -Phive./python/run-tests

The run-tests script also can be limited to a specific Python version or a specific module

./python/run-tests --python-executables=python --modules=pyspark-sql

Note: You can also run Python tests with an sbt build, provided you build Spark with Hive support.

Running R Tests

To run the SparkR tests you will need to install theknitr,rmarkdown,testthat,e1071 andsurvival packages first:

R -e "install.packages(c('knitr', 'rmarkdown', 'testthat', 'e1071', 'survival'), repos='http://cran.us.r-project.org')"

You can run just the SparkR tests using the command:

./R/run-tests.sh

Running Docker-based Integration Test Suites

In order to run Docker integration tests, you have to install the

docker

engine on your box. The instructions for installation can be found atthe Docker site. Once installed, the

docker

service needs to be started, if not already running. On Linux, this can be done by

sudo service docker start

.

./build/mvn install -DskipTests./build/mvn test -Pdocker-integration-tests -pl :spark-docker-integration-tests_2.11

or

./build/sbt docker-integration-tests/test