Spark：用Scala和Java实现WordCount

为了在IDEA中编写scala，今天安装配置学习了IDEA集成开发环境。IDEA确实很优秀，学会之后，用起来很顺手。关于如何搭建scala和IDEA开发环境，请看文末的参考资料。
用Scala和Java实现WordCount，其中Java实现的JavaWordCount是spark自带的例子（$SPARK_HOME/examples/src/main/java/org/apache/spark/examples/JavaWordCount.java）
1.环境
OS:Red Hat Enterprise Linux Server release 6.4 (Santiago)
Hadoop:Hadoop 2.4.1
JDK:1.7.0_60
Spark:1.1.0
Scala:2.11.2
集成开发环境：IntelliJ IDEA 13.1.3
注意：需要在客户端windows环境下安装IDEA、Scala、JDK，并且为IDEA下载scala插件。
2.Scala实现单词计数

1 package com.hq
2
3 /**
4  * User: hadoop
5  * Date: 2014/10/10 0010
6  * Time: 18:59
7  */
8 import org.apache.spark.SparkConf
9 import org.apache.spark.SparkContext
10 import org.apache.spark.SparkContext._
11
12 /**
13  * 统计字符出现次数
14  */
15 object WordCount {
16   def main(args: Array[String]) {
17     if (args.length < 1) {
18       System.err.println("Usage: <file>")
19       System.exit(1)
20     }
21
22     val conf = new SparkConf()
23     val sc = new SparkContext(conf)
24     val line = sc.textFile(args(0))
25
26     line.flatMap(_.split(" ")).map((_, 1)).reduceByKey(_+_).collect().foreach(println)
27
28     sc.stop()
29   }
30 }

3.Java实现单词计数


 View
Code
4.IDEA打包和运行
4.1 IDEA的工程结构
在IDEA中建立Scala工程，并导入spark api编程jar包（spark-assembly-1.1.0-hadoop2.4.0.jar：$SPARK_HOME/lib/里面）



 
4.2 打成jar包
File ---> Project Structure 



 
配置完成后，在菜单栏中选择Build->Build Artifacts...，然后使用Build等命令打包。打包完成后会在状态栏中显示“Compilation completed successfully...”的信息，去jar包输出路径下查看jar包，如下所示。



ScalaTest1848.jar就是我们编程所产生的jar包，里面包含了三个类HelloWord、WordCount、JavaWordCount。
可以用这个jar包在spark集群里面运行java或者scala的单词计数程序。
 4.3 以Spark集群standalone方式运行单词计数
上传jar包到服务器，并放置在/home/ebupt/test/WordCount.jar路径下。
上传一个text文本文件到HDFS作为单词计数的输入文件：hdfs://eb170:8020/user/ebupt/text
内容如下


 View
Code
用spark-submit命令提交任务运行，具体使用查看：spark-submit --help

1 [ebupt@eb174 bin]$ spark-submit --help
2 Spark assembly has been built with Hive, including Datanucleus jars on classpath
3 Usage: spark-submit [options] <app jar | python file> [app options]
4 Options:
5   --master MASTER_URL         spark://host:port, mesos://host:port, yarn, or local.
6   --deploy-mode DEPLOY_MODE   Whether to launch the driver program locally ("client") or
7                               on one of the worker machines inside the cluster ("cluster")
8                               (Default: client).
9   --class CLASS_NAME          Your application's main class (for Java / Scala apps).
10   --name NAME                 A name of your application.
11   --jars JARS                 Comma-separated list of local jars to include on the driver
12                               and executor classpaths.
13   --py-files PY_FILES         Comma-separated list of .zip, .egg, or .py files to place
14                               on the PYTHONPATH for Python apps.
15   --files FILES               Comma-separated list of files to be placed in the working
16                               directory of each executor.
17
18   --conf PROP=VALUE           Arbitrary Spark configuration property.
19   --properties-file FILE      Path to a file from which to load extra properties. If not
20                               specified, this will look for conf/spark-defaults.conf.
21
22   --driver-memory MEM         Memory for driver (e.g. 1000M, 2G) (Default: 512M).
23   --driver-java-options       Extra Java options to pass to the driver.
24   --driver-library-path       Extra library path entries to pass to the driver.
25   --driver-class-path         Extra class path entries to pass to the driver. Note that
26                               jars added with --jars are automatically included in the
27                               classpath.
28
29   --executor-memory MEM       Memory per executor (e.g. 1000M, 2G) (Default: 1G).
30
31   --help, -h                  Show this help message and exit
32   --verbose, -v               Print additional debug output
33
34  Spark standalone with cluster deploy mode only:
35   --driver-cores NUM          Cores for driver (Default: 1).
36   --supervise                 If given, restarts the driver on failure.
37
38  Spark standalone and Mesos only:
39   --total-executor-cores NUM  Total cores for all executors.
40
41  YARN-only:
42   --executor-cores NUM        Number of cores per executor (Default: 1).
43   --queue QUEUE_NAME          The YARN queue to submit to (Default: "default").
44   --num-executors NUM         Number of executors to launch (Default: 2).
45   --archives ARCHIVES         Comma separated list of archives to be extracted into the
46                               working directory of each executor.

①提交scala实现的单词计数：
[ebupt@eb174 test]$ spark-submit --master spark://eb174:7077 --name WordCountByscala --class com.hq.WordCount --executor-memory 1G --total-executor-cores 2 ~/test/WordCount.jar hdfs://eb170:8020/user/ebupt/text 
②提交java实现的单词计数：
[ebupt@eb174 test]$ spark-submit --master spark://eb174:7077 --name JavaWordCountByHQ --class com.hq.JavaWordCount --executor-memory 1G --total-executor-cores 2 ~/test/WordCount.jar
hdfs://eb170:8020/user/ebupt/text
③2者运行结果类似，所以只写了一个：

按 Ctrl+C 复制代码
按 Ctrl+C 复制代码