Eclipse 编写的spark版WordCount运行在spark上
2014-11-27 11:35
429 查看
1、代码编写
if (args.length != 3 ){
println("usage is org.test.WordCount <master> <input> <output>")
return
}
val sc = new SparkContext(args(0), "WordCount",
System.getenv("SPARK_HOME"), Seq(System.getenv("SPARK_TEST_JAR")))
val textFile = sc.textFile(args(1))
val result = textFile.flatMap(line => line.split("\\s+"))
.map(word => (word, 1)).reduceByKey(_ + _)
result.saveAsTextFile(args(2))
2、导出jar包,这里我命名为WordCount.jar
3、运行
bin/spark-submit --master spark://master:7077 /home/zkpk/Desktop/WordCount.jar hdfs://master:9000/dedup_in hdfs://master:9000/dedup_out
先记录下,第三部还没执行,抽空试试
if (args.length != 3 ){
println("usage is org.test.WordCount <master> <input> <output>")
return
}
val sc = new SparkContext(args(0), "WordCount",
System.getenv("SPARK_HOME"), Seq(System.getenv("SPARK_TEST_JAR")))
val textFile = sc.textFile(args(1))
val result = textFile.flatMap(line => line.split("\\s+"))
.map(word => (word, 1)).reduceByKey(_ + _)
result.saveAsTextFile(args(2))
2、导出jar包,这里我命名为WordCount.jar
3、运行
bin/spark-submit --master spark://master:7077 /home/zkpk/Desktop/WordCount.jar hdfs://master:9000/dedup_in hdfs://master:9000/dedup_out
先记录下,第三部还没执行,抽空试试
相关文章推荐
- idea运行spark的wordcount与eclipse运行spark的wordcount示例,及本地运行sparkpi
- scala-eclipse 编写spark简单程序 WordCount
- Idea中编写并运行Spark集群(WordCount)
- win7下安装hadoop 2.6.0 的eclipse插件并编写运行WordCount程序
- 利用Scala编写Wordcount并在spark框架下运行
- fedora17中hadoop2.2.0在eclipse下运行wordcount
- eclipse运行wordcount
- eclipse运行hadoop wordcount example
- 在linux下eclipse中运行hadoop自带的WordCount例子出现的两个错误
- Eclipse运行Hadoop WordCount例程
- 在windows上用eclipse远程运行hadoop上的wordcount程序出现的问题,求解决
- eclipse打包jar发布到linux下运行出错(java.lang.ClassNotFoundException: cmd.WordCount$MyMapper )
- 【hadoop学习笔记】4.eclipse运行wordcount实例
- linux下在eclipse上运行hadoop自带例子wordcount
- Hadoop2.2.0源码分析(一)——Eclipse运行WordCount.java
- 解决Eclipse中运行WordCount出现 java.lang.ClassNotFoundException: org.apache.hadoop.examples.WordCount$TokenizerMapper问题【转】
- Hadoop不用eclipse去编译运行WordCount
- Eclipse运行wordcount步骤