Spark Transformation —— distinct 算子

distinct

代码测试

测试数据准备

spark-shell --master spark://master:7077(重要的事情说多遍，spark-shell和spark-submit核心的参数是差不多的)

var data = sc.textFile("/qgzang/1.txt")

测试distinct 算子

scala> data.flatMap(line => line.split("\\s+")).collect
res61: Array[String] = Array(hello, world, hello, spark, hello, hive, hi, spark)

scala> data.flatMap(line => line.split("\\s+")).distinct.collect
res62: Array[String] = Array(hive, hello, world, spark, hi)