spark-shell 数据文件 读成 表 的两种方式！！！ 相对路径！！hdfs dfs -ls
2016-07-20 23:22
543 查看
park SQL应用
Spark Shell启动后，就可以用Spark SQL API执行数据分析查询。
在第一个示例中，我们将从文本文件中加载用户数据并从数据集中创建一个DataFrame对象。然后运行DataFrame函数，执行特定的数据选择查询。
文本文件customers.txt中的内容如下：
100, John Smith, Austin, TX, 78727
200, Joe Johnson, Dallas, TX, 75201
300, Bob Jones, Houston, TX, 77028
400, Andy Davis, San Antonio, TX, 78227
500, James Williams, Austin, TX, 78727
下述代码片段展示了可以在Spark Shell终端执行的Spark SQL命令。
// 首先用已有的Spark Context对象创建SQLContext对象
val sqlContext = new org.apache.spark.sql.SQLContext(sc);
// 导入语句，可以隐式地将RDD转化成DataFrame
import sqlContext.implicits._
// 创建一个表示客户的自定义类
case class Customer(customer_id: Int, name: String, city: String, state: String, zip_code: String)
// 用数据集文本文件创建一个Customer对象的DataFrame
val dfCustomers = sc.textFile("data/customers.txt").map(_.split(",")).map(p => Customer(p(0).trim.toInt, p(1), p(2), p(3), p(4))).toDF();
// 将DataFrame注册为一个表
dfCustomers.registerTempTable("customers");
// 显示DataFrame的内容
dfCustomers.show();
// 打印DF模式
dfCustomers.printSchema();
// 选择客户名称列
dfCustomers.select("name").show();
// 选择客户名称和城市列
dfCustomers.select("name", "city").show()
// 根据id选择客户
dfCustomers.filter(dfCustomers("customer_id").equalTo(500)).show();
// 根据邮政编码统计客户数量
dfCustomers.groupBy("zip_code").count().show();
使用新的数据类型类StructType，StringType和StructField指定模式。
//
// 用编程的方式指定模式
//
// 用已有的Spark Context对象创建SQLContext对象
val sqlContext = new org.apache.spark.sql.SQLContext(sc);
// 创建RDD对象
val rddCustomers = sc.textFile("data/customers.txt");
// 用字符串编码模式
val schemaString = "customer_id name city state zip_code";
// 导入Spark SQL数据类型和Row
import org.apache.spark.sql._
import org.apache.spark.sql.types._;
// 用模式字符串生成模式对象
val schema = StructType(schemaString.split(" ").map(fieldName => StructField(fieldName, StringType, true)));
// 将RDD（rddCustomers）记录转化成Row。
val rowRDD = rddCustomers.map(_.split(",")).map(p => Row(p(0).trim,p(1),p(2),p(3),p(4)));
// 将模式应用于RDD对象。
val dfCustomers = sqlContext.createDataFrame(rowRDD, schema);
// 将DataFrame注册为表
dfCustomers.registerTempTable("customers");
// 用sqlContext对象提供的sql方法执行SQL语句。
val custNames = sqlContext.sql("SELECT name FROM customers");
// SQL查询的返回结果为DataFrame对象，支持所有通用的RDD操作。
// 可以按照顺序访问结果行的各个列。
custNames.map(t => "Name: " + t(0)).collect().foreach(println);
// 用sqlContext对象提供的sql方法执行SQL语句。
val customersByCity = sqlContext.sql("SELECT name,zip_code FROM customers ORDER BY zip_code");
// SQL查询的返回结果为DataFrame对象，支持所有通用的RDD操作。
// 可以按照顺序访问结果行的各个列。
customersByCity.map(t => t(0) + "," + t(1)).collect().foreach(println);
除了文本文件之外，也可以从其他数据源中加载数据，如JSON数据文件，Hive表，甚至可以通过JDBC数据源加载关系型数据库表中的数据<pre name="code" class="sql">Last login: Tue Jul 19 23:21:05 2016 from 192.168.3.103 [root@cdh1 ~]# jps 23852 SparkSubmit 19801 NameNode 23932 Jps 20330 NodeManager 22342 Master 22509 Worker 20231 ResourceManager 20082 SecondaryNameNode 19898 DataNode [root@cdh1 ~]# hdfs dfs -ls /user/ Found 2 items drwxr-xr-x - root supergroup 0 2016-07-19 21:16 /user/hive drwxr-xr-x - root supergroup 0 2016-07-19 23:07 /user/test [root@cdh1 ~]# hdfs dfs -ls /user/test [root@cdh1 ~]# cd /user/test [root@cdh1 test]# ll total 0 [root@cdh1 test]# vim customers.txt [root@cdh1 test]# ll total 4 -rw-r--r--. 1 root root 185 Jul 19 23:25 customers.txt [root@cdh1 test]# cat customers.txt 100, John Smith, Austin, TX, 78727 200, Joe Johnson, Dallas, TX, 75201 300, Bob Jones, Houston, TX, 77028 400, Andy Davis, San Antonio, TX, 78227 500, James Williams, Austin, TX, 78727 [root@cdh1 test]# hdfs dfs -ls /user/test [root@cdh1 test]# hdfs dfs -put /user/test/customers.txt /user/test [root@cdh1 test]# hdfs dfs -ls /user/test Found 1 items -rw-r--r-- 1 root supergroup 185 2016-07-19 23:26 /user/test/customers.txt [root@cdh1 test]# hdfs dfs -ls /user/test Found 1 items -rw-r--r-- 1 root supergroup 185 2016-07-19 23:26 /user/test/customers.txt [root@cdh1 test]# hdfs dfs -ls / Found 2 items drwxr-xr-x - root supergroup 0 2016-07-19 21:31 /tmp drwxr-xr-x - root supergroup 0 2016-07-19 23:07 /user [root@cdh1 test]#
Last login: Tue Jul 19 23:23:11 2016 from 192.168.3.103 [root@cdh1 ~]# cd /user/local/spark-1.4.0-bin-hadoop2.6/ [root@cdh1 spark-1.4.0-bin-hadoop2.6]# ll total 684 drwxr-xr-x. 2 yc yc 4096 Jun 3 2015 bin -rw-r--r--. 1 yc yc 561149 Jun 3 2015 CHANGES.txt drwxr-xr-x. 2 yc yc 4096 Jun 18 00:03 conf drwxr-xr-x. 3 yc yc 4096 Jun 3 2015 data drwxr-xr-x. 3 yc yc 4096 Jun 3 2015 ec2 drwxr-xr-x. 3 yc yc 4096 Jun 3 2015 examples drwxr-xr-x. 2 yc yc 4096 Jun 3 2015 lib -rw-r--r--. 1 yc yc 50902 Jun 3 2015 LICENSE drwxr-xr-x. 2 root root 4096 Jul 19 22:35 logs -rw-r--r--. 1 yc yc 22559 Jun 3 2015 NOTICE drwxr-xr-x. 6 yc yc 4096 Jun 3 2015 python drwxr-xr-x. 3 yc yc 4096 Jun 3 2015 R -rw-r--r--. 1 yc yc 3624 Jun 3 2015 README.md -rw-r--r--. 1 yc yc 134 Jun 3 2015 RELEASE drwxr-xr-x. 2 yc yc 4096 Jul 19 22:39 sbin drwxr-xr-x. 2 root root 4096 Jun 18 00:03 work [root@cdh1 spark-1.4.0-bin-hadoop2.6]# cd work/ [root@cdh1 work]# ll total 0 [root@cdh1 work]# cd .. [root@cdh1 spark-1.4.0-bin-hadoop2.6]# cd data/ [root@cdh1 data]# ll total 4 drwxr-xr-x. 5 yc yc 4096 Jun 3 2015 mllib [root@cdh1 data]# cd mllib/ [root@cdh1 mllib]# ll total 828 drwxr-xr-x. 2 yc yc 4096 Jun 3 2015 als -rw-r--r--. 1 yc yc 63973 Jun 3 2015 gmm_data.txt -rw-r--r--. 1 yc yc 72 Jun 3 2015 kmeans_data.txt drwxr-xr-x. 2 yc yc 4096 Jun 3 2015 lr-data -rw-r--r--. 1 yc yc 197105 Jun 3 2015 lr_data.txt -rw-r--r--. 1 yc yc 24 Jun 3 2015 pagerank_data.txt drwxr-xr-x. 2 yc yc 4096 Jun 3 2015 ridge-data -rw-r--r--. 1 yc yc 104736 Jun 3 2015 sample_binary_classification_data.txt -rw-r--r--. 1 yc yc 68 Jun 3 2015 sample_fpgrowth.txt -rw-r--r--. 1 yc yc 1598 Jun 3 2015 sample_isotonic_regression_data.txt -rw-r--r--. 1 yc yc 264 Jun 3 2015 sample_lda_data.txt -rw-r--r--. 1 yc yc 104736 Jun 3 2015 sample_libsvm_data.txt -rwxr-xr-x. 1 yc yc 119069 Jun 3 2015 sample_linear_regression_data.txt -rw-r--r--. 1 yc yc 14351 Jun 3 2015 sample_movielens_data.txt -rw-r--r--. 1 yc yc 6953 Jun 3 2015 sample_multiclass_classification_data.txt -rw-r--r--. 1 yc yc 48 Jun 3 2015 sample_naive_bayes_data.txt -rw-r--r--. 1 yc yc 39474 Jun 3 2015 sample_svm_data.txt -rw-r--r--. 1 yc yc 115476 Jun 3 2015 sample_tree_data.csv [root@cdh1 mllib]# cd .. [root@cdh1 data]# ll total 4 drwxr-xr-x. 5 yc yc 4096 Jun 3 2015 mllib [root@cdh1 data]# pwd /user/local/spark-1.4.0-bin-hadoop2.6/data [root@cdh1 data]# ls mllib [root@cdh1 data]# vim customers.txt [1]+ Stopped vim customers.txt [root@cdh1 data]# vim customers.txt [root@cdh1 data]# ls customers.txt mllib [root@cdh1 data]# ls -l total 8 -rw-r--r--. 1 root root 185 Jul 19 23:44 customers.txt drwxr-xr-x. 5 yc yc 4096 Jun 3 2015 mllib [root@cdh1 data]# hdfs dfs -l /user/root/data/ -l: Unknown command [root@cdh1 data]# hdfs dfs -ls /user/root/data/ ls: `/user/root/data/': No such file or directory [root@cdh1 data]# hdfs dfs -ls /user/ Found 2 items drwxr-xr-x - root supergroup 0 2016-07-19 21:16 /user/hive drwxr-xr-x - root supergroup 0 2016-07-19 23:26 /user/test [root@cdh1 data]# hdfs dfs -mkdir /user/root/data/ mkdir: `/user/root/data/': No such file or directory [root@cdh1 data]# hdfs dfs -mkdir -p /user/root/data/ [root@cdh1 data]# hdfs dfs -ls /user/root/data/ [root@cdh1 data]# hdfs dfs -ls /user/root Found 1 items drwxr-xr-x - root supergroup 0 2016-07-19 23:47 /user/root/data [root@cdh1 data]# pwd /user/local/spark-1.4.0-bin-hadoop2.6/data [root@cdh1 data]# ll total 8 -rw-r--r--. 1 root root 185 Jul 19 23:44 customers.txt drwxr-xr-x. 5 yc yc 4096 Jun 3 2015 mllib [root@cdh1 data]# hdfs dfs -put customers.txt /user/root/data [root@cdh1 data]# hdfs dfs -ls /user/root/data Found 1 items -rw-r--r-- 1 root supergroup 185 2016-07-19 23:48 /user/root/data/customers.txt [root@cdh1 data]# hdfs dfs -text /user/root/data/customers.txt 100, John Smith, Austin, TX, 78727 200, Joe Johnson, Dallas, TX, 75201 300, Bob Jones, Houston, TX, 77028 400, Andy Davis, San Antonio, TX, 78227 500, James Williams, Austin, TX, 78727 [root@cdh1 data]# hdfs dfs -cat /user/root/data/customers.txt 100, John Smith, Austin, TX, 78727 200, Joe Johnson, Dallas, TX, 75201 300, Bob Jones, Houston, TX, 77028 400, Andy Davis, San Antonio, TX, 78227 500, James Williams, Austin, TX, 78727 [root@cdh1 data]# hdfs dfs -mv /user/root/data/customers.txt /user/root/data/customer.tx [root@cdh1 data]# hdfs dfs -ls /user/root/data Found 1 items -rw-r--r-- 1 root supergroup 185 2016-07-19 23:48 /user/root/data/customer.tx [root@cdh1 data]#
[BEGIN] 2016/7/20 22:46:49
dfCustomers.groupBy("zip_code").count().show();filter(dfCustomers("customer_id").equalTo(500)).show();select("name", "city").show()).show();printSchema();show();registerTempTable("customers");val dfCustomers = sc.textFile("/user/test/customers.txt").map(_.split(",")).map(p => Customer(p(0).trim.toInt, p(1), p(2), p(3), p(4))).toDF();data/customers.txt").map(_.split(",")).map(p => Customer(p(0).trim.toInt, p(1), p(2), p(3), p(4))).toDF();/user/test/customers.txt").map(_.split(",")).map(p => Customer(p(0).trim.toInt, p(1), p(2), p(3), p(4))).toDF();data/customers.txt").map(_.split(",")).map(p => Customer(p(0).trim.toInt, p(1), p(2), p(3), p(4))).toDF();case class Customer(customer_id: Int, name: String, city: String, state: String, zip_code: String)import sqlContext.implicits._val sqlContext = new org.apache.spark.sql.SQLContext(sc);textFile = sc.textFile("/user/test/customers.txt").collect();customers.txt").collect();
16/07/19 23:33:01 INFO storage.MemoryStore: ensureFreeSpace(222752) called with curMem=896994, maxMem=278019440
16/07/19 23:33:01 INFO storage.MemoryStore: Block broadcast_19 stored as values in memory (estimated size 217.5 KB, free 264.1 MB)
16/07/19 23:33:01 INFO storage.MemoryStore: ensureFreeSpace(19999) called with curMem=1119746, maxMem=278019440
16/07/19 23:33:01 INFO storage.MemoryStore: Block broadcast_19_piece0 stored as bytes in memory (estimated size 19.5 KB, free 264.1 MB)
16/07/19 23:33:01 INFO storage.BlockManagerInfo: Added broadcast_19_piece0 in memory on localhost:56137 (size: 19.5 KB, free: 265.0 MB)
16/07/19 23:33:01 INFO spark.SparkContext: Created broadcast 19 from textFile at <console>:26
org.apache.hadoop.mapred.InvalidInputException: Input path does not exist: hdfs://cdh1:9000/user/root/customers.txt
at org.apache.hadoop.mapred.FileInputFormat.singleThreadedListStatus(FileInputFormat.java:285)
at org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:228)
at org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:313)
at org.apache.spark.rdd.HadoopRDD.getPartitions(HadoopRDD.scala:207)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:219)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:217)
at scala.Option.getOrElse(Option.scala:120)
at org.apache.spark.rdd.RDD.partitions(RDD.scala:217)
at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:32)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:219)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:217)
at scala.Option.getOrElse(Option.scala:120)
at org.apache.spark.rdd.RDD.partitions(RDD.scala:217)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:1779)
at org.apache.spark.rdd.RDD$$anonfun$collect$1.apply(RDD.scala:885)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:148)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:109)
at org.apache.spark.rdd.RDD.withScope(RDD.scala:286)
at org.apache.spark.rdd.RDD.collect(RDD.scala:884)
at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:26)
at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:31)
at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:33)
at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:35)
at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:37)
at $iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:39)
at $iwC$$iwC$$iwC$$iwC.<init>(<console>:41)
at $iwC$$iwC$$iwC.<init>(<console>:43)
at $iwC$$iwC.<init>(<console>:45)
at $iwC.<init>(<console>:47)
at <init>(<console>:49)
at .<init>(<console>:53)
at .<clinit>(<console>)
at .<init>(<console>:7)
at .<clinit>(<console>)
at $print(<console>)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:1065)
at org.apache.spark.repl.SparkIMain$Request.loadAndRun(SparkIMain.scala:1338)
at org.apache.spark.repl.SparkIMain.loadAndRunReq$1(SparkIMain.scala:840)
at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:871)
at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:819)
at org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:857)
at org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:902)
at org.apache.spark.repl.SparkILoop.command(SparkILoop.scala:814)
at org.apache.spark.repl.SparkILoop.processLine$1(SparkILoop.scala:657)
at org.apache.spark.repl.SparkILoop.innerLoop$1(SparkILoop.scala:665)
at org.apache.spark.repl.SparkILoop.org$apache$spark$repl$SparkILoop$$loop(SparkILoop.scala:670)
at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply$mcZ$sp(SparkILoop.scala:997)
at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:945)
at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:945)
at scala.tools.nsc.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:135)
at org.apache.spark.repl.SparkILoop.org$apache$spark$repl$SparkILoop$$process(SparkILoop.scala:945)
at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:1059)
at org.apache.spark.repl.Main$.main(Main.scala:31)
at org.apache.spark.repl.Main.main(Main.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:664)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:169)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:192)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:111)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
scala> val textFile = sc.textFile("customers.txt").collect();dfCustomers.groupBy("zip_code").count().show();filter(dfCustomers("customer_id").equalTo(500)).show();select("name", "city").show()).show();printSchema();show();registerTempTable("customers");val dfCustomers = sc.textFile("/user/test/customers.txt").map(_.split(",")).map(p => Customer(p(0).trim.toInt, p(1), p(2), p(3), p(4))).toDF();data/customers.txt").map(_.split(",")).map(p => Customer(p(0).trim.toInt, p(1), p(2), p(3), p(4))).toDF();case class Customer(customer_id: Int, name: String, city: String, state: String, zip_code: String)import sqlContext.implicits._val sqlContext = new org.apache.spark.sql.SQLContext(sc);textFile = sc.textFile("/user/test/customers.txt").collect();
16/07/19 23:33:16 INFO storage.BlockManagerInfo: Removed broadcast_19_piece0 on localhost:56137 in memory (size: 19.5 KB, free: 265.0 MB)
16/07/19 23:33:16 INFO storage.MemoryStore: ensureFreeSpace(222752) called with curMem=896994, maxMem=278019440
16/07/19 23:33:16 INFO storage.MemoryStore: Block broadcast_20 stored as values in memory (estimated size 217.5 KB, free 264.1 MB)
16/07/19 23:33:16 INFO storage.MemoryStore: ensureFreeSpace(19999) called with curMem=1119746, maxMem=278019440
16/07/19 23:33:16 INFO storage.MemoryStore: Block broadcast_20_piece0 stored as bytes in memory (estimated size 19.5 KB, free 264.1 MB)
16/07/19 23:33:16 INFO storage.BlockManagerInfo: Added broadcast_20_piece0 in memory on localhost:56137 (size: 19.5 KB, free: 265.0 MB)
16/07/19 23:33:16 INFO spark.SparkContext: Created broadcast 20 from textFile at <console>:26
16/07/19 23:33:16 INFO mapred.FileInputFormat: Total input paths to process : 1
16/07/19 23:33:16 INFO spark.SparkContext: Starting job: collect at <console>:26
16/07/19 23:33:16 INFO scheduler.DAGScheduler: Got job 13 (collect at <console>:26) with 2 output partitions (allowLocal=false)
16/07/19 23:33:16 INFO scheduler.DAGScheduler: Final stage: ResultStage 17(collect at <console>:26)
16/07/19 23:33:16 INFO scheduler.DAGScheduler: Parents of final stage: List()
16/07/19 23:33:16 INFO scheduler.DAGScheduler: Missing parents: List()
16/07/19 23:33:16 INFO scheduler.DAGScheduler: Submitting ResultStage 17 (MapPartitionsRDD[38] at textFile at <console>:26), which has no missing parents
16/07/19 23:33:16 INFO storage.MemoryStore: ensureFreeSpace(3128) called with curMem=1139745, maxMem=278019440
16/07/19 23:33:16 INFO storage.MemoryStore: Block broadcast_21 stored as values in memory (estimated size 3.1 KB, free 264.1 MB)
16/07/19 23:33:16 INFO storage.MemoryStore: ensureFreeSpace(1795) called with curMem=1142873, maxMem=278019440
16/07/19 23:33:16 INFO storage.MemoryStore: Block broadcast_21_piece0 stored as bytes in memory (estimated size 1795.0 B, free 264.0 MB)
16/07/19 23:33:16 INFO storage.BlockManagerInfo: Added broadcast_21_piece0 in memory on localhost:56137 (size: 1795.0 B, free: 265.0 MB)
16/07/19 23:33:16 INFO spark.SparkContext: Created broadcast 21 from broadcast at DAGScheduler.scala:874
16/07/19 23:33:16 INFO scheduler.DAGScheduler: Submitting 2 missing tasks from ResultStage 17 (MapPartitionsRDD[38] at textFile at <console>:26)
16/07/19 23:33:16 INFO scheduler.TaskSchedulerImpl: Adding task set 17.0 with 2 tasks
16/07/19 23:33:16 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 17.0 (TID 414, localhost, ANY, 1413 bytes)
16/07/19 23:33:16 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 17.0 (TID 415, localhost, ANY, 1413 bytes)
16/07/19 23:33:16 INFO executor.Executor: Running task 0.0 in stage 17.0 (TID 414)
16/07/19 23:33:16 INFO rdd.HadoopRDD: Input split: hdfs://cdh1:9000/user/test/customers.txt:0+92
16/07/19 23:33:16 INFO executor.Executor: Running task 1.0 in stage 17.0 (TID 415)
16/07/19 23:33:16 INFO rdd.HadoopRDD: Input split: hdfs://cdh1:9000/user/test/customers.txt:92+93
16/07/19 23:33:16 INFO executor.Executor: Finished task 1.0 in stage 17.0 (TID 415). 1875 bytes result sent to driver
16/07/19 23:33:16 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 17.0 (TID 415) in 28 ms on localhost (1/2)
16/07/19 23:33:16 INFO executor.Executor: Finished task 0.0 in stage 17.0 (TID 414). 1904 bytes result sent to driver
16/07/19 23:33:16 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 17.0 (TID 414) in 30 ms on localhost (2/2)
16/07/19 23:33:16 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 17.0, whose tasks have all completed, from pool
16/07/19 23:33:16 INFO scheduler.DAGScheduler: ResultStage 17 (collect at <console>:26) finished in 0.030 s
16/07/19 23:33:16 INFO scheduler.DAGScheduler: Job 13 finished: collect at <console>:26, took 0.041509 s
textFile: Array[String] = Array(100, John Smith, Austin, TX, 78727, 200, Joe Johnson, Dallas, TX, 75201, 300, Bob Jones, Houston, TX, 77028, 400, Andy Davis, San Antonio, TX, 78227, 500, James Williams, Austin, TX, 78727)
scala> val textFile = sc.textFile("/user/test/customers.txt").collect();customers.txt").collect();dfCustomers.groupBy("zip_code").count().show();filter(dfCustomers("customer_id").equalTo(500)).show();select("name", "city").show()).show();printSchema();show();registerTempTable("customers");val dfCustomers = sc.textFile("/user/test/customers.txt").map(_.split(",")).map(p => Customer(p(0).trim.toInt, p(1), p(2), p(3), p(4))).toDF();data/customers.txt").map(_.split(",")).map(p => Customer(p(0).trim.toInt, p(1), p(2), p(3), p(4))).toDF();case class Customer(customer_id: Int, name: String, city: String, state: String, zip_code: String)import sqlContext.implicits._val sqlContext = new org.apache.spark.sql.SQLContext(sc);textFile = sc.textFile("/user/test/customers.txt").collect();sqlContext = new org.apache.spark.sql.SQLContext(sc);
sqlContext: org.apache.spark.sql.SQLContext = org.apache.spark.sql.SQLContext@1b57779
scala>
(reverse-i-search)`':
scala> val sqlContext = new org.apache.spark.sql.SQLContext(sc);textFile = sc.textFile("/user/test/customers.txt").collect();sqlContext = new org.apache.spark.sql.SQLContext(sc);val sqlContext = new org.apache.spark.sql.SQLContext(sc);textFile = sc.textFile("/user/test/customers.txt").collect();customers.txt").collect();dfCustomers.groupBy("zip_code").count().show();filter(dfCustomers("customer_id").equalTo(500)).show();select("name", "city").show()).show();printSchema();show();registerTempTable("customers");val dfCustomers = sc.textFile("/user/test/customers.txt").map(_.split(",")).map(p => Customer(p(0).trim.toInt, p(1), p(2), p(3), p(4))).toDF();data/customers.txt").map(_.split(",")).map(p => Customer(p(0).trim.toInt, p(1), p(2), p(3), p(4))).toDF();case class Customer(customer_id: Int, name: String, city: String, state: String, zip_code: String)import sqlContext.implicits._val sqlContext = new org.apache.spark.sql.SQLContext(sc);import sqlContext.implicits._
import sqlContext.implicits._
scala> import sqlContext.implicits._val sqlContext = new org.apache.spark.sql.SQLContext(sc);textFile = sc.textFile("/user/test/customers.txt").collect();customers.txt").collect();dfCustomers.groupBy("zip_code").count().show();filter(dfCustomers("customer_id").equalTo(500)).show();select("name", "city").show()).show();printSchema();show();registerTempTable("customers");val dfCustomers = sc.textFile("/user/test/customers.txt").map(_.split(",")).map(p => Customer(p(0).trim.toInt, p(1), p(2), p(3), p(4))).toDF();data/customers.txt").map(_.split(",")).map(p => Customer(p(0).trim.toInt, p(1), p(2), p(3), p(4))).toDF();case class Customer(customer_id: Int, name: String, city: String, state: String, zip_code: String)import sqlContext.implicits._case class Customer(customer_id: Int, name: String, city: String, state: String, zip_code: String)
defined class Customer
scala> case class Customer(customer_id: Int, name: String, city: String, state: String, zip_code: String)import sqlContext.implicits._val sqlContext = new org.apache.spark.sql.SQLContext(sc);textFile = sc.textFile("/user/test/customers.txt").collect();customers.txt").collect();dfCustomers.groupBy("zip_code").count().show();filter(dfCustomers("customer_id").equalTo(500)).show();select("name", "city").show()).show();printSchema();show();registerTempTable("customers");val dfCustomers = sc.textFile("/user/test/customers.txt").map(_.split(",")).map(p => Customer(p(0).trim.toInt, p(1), p(2), p(3), p(4))).toDF();
16/07/19 23:34:33 INFO storage.BlockManagerInfo: Removed broadcast_21_piece0 on localhost:56137 in memory (size: 1795.0 B, free: 265.0 MB)
16/07/19 23:34:33 INFO storage.BlockManagerInfo: Removed broadcast_20_piece0 on localhost:56137 in memory (size: 19.5 KB, free: 265.0 MB)
16/07/19 23:34:33 INFO storage.MemoryStore: ensureFreeSpace(222752) called with curMem=896994, maxMem=278019440
16/07/19 23:34:33 INFO storage.MemoryStore: Block broadcast_22 stored as values in memory (estimated size 217.5 KB, free 264.1 MB)
16/07/19 23:34:33 INFO storage.MemoryStore: ensureFreeSpace(19999) called with curMem=1119746, maxMem=278019440
16/07/19 23:34:33 INFO storage.MemoryStore: Block broadcast_22_piece0 stored as bytes in memory (estimated size 19.5 KB, free 264.1 MB)
16/07/19 23:34:33 INFO storage.BlockManagerInfo: Added broadcast_22_piece0 in memory on localhost:56137 (size: 19.5 KB, free: 265.0 MB)
16/07/19 23:34:33 INFO spark.SparkContext: Created broadcast 22 from textFile at <console>:33
dfCustomers: org.apache.spark.sql.DataFrame = [customer_id: int, name: string, city: string, state: string, zip_code: string]
scala> dfCustomers.registerTempTable("customers");
scala> dfCustomers.show();
16/07/19 23:34:55 INFO mapred.FileInputFormat: Total input paths to process : 1
16/07/19 23:34:55 INFO spark.SparkContext: Starting job: show at <console>:36
16/07/19 23:34:55 INFO scheduler.DAGScheduler: Got job 14 (show at <console>:36) with 1 output partitions (allowLocal=false)
16/07/19 23:34:55 INFO scheduler.DAGScheduler: Final stage: ResultStage 18(show at <console>:36)
16/07/19 23:34:55 INFO scheduler.DAGScheduler: Parents of final stage: List()
16/07/19 23:34:55 INFO scheduler.DAGScheduler: Missing parents: List()
16/07/19 23:34:55 INFO scheduler.DAGScheduler: Submitting ResultStage 18 (MapPartitionsRDD[44] at show at <console>:36), which has no missing parents
16/07/19 23:34:55 INFO storage.MemoryStore: ensureFreeSpace(4088) called with curMem=1139745, maxMem=278019440
16/07/19 23:34:55 INFO storage.MemoryStore: Block broadcast_23 stored as values in memory (estimated size 4.0 KB, free 264.0 MB)
16/07/19 23:34:55 INFO storage.MemoryStore: ensureFreeSpace(2227) called with curMem=1143833, maxMem=278019440
16/07/19 23:34:55 INFO storage.MemoryStore: Block broadcast_23_piece0 stored as bytes in memory (estimated size 2.2 KB, free 264.0 MB)
16/07/19 23:34:55 INFO storage.BlockManagerInfo: Added broadcast_23_piece0 in memory on localhost:56137 (size: 2.2 KB, free: 265.0 MB)
16/07/19 23:34:55 INFO spark.SparkContext: Created broadcast 23 from broadcast at DAGScheduler.scala:874
16/07/19 23:34:55 INFO scheduler.DAGScheduler: Submitting 1 missing tasks from ResultStage 18 (MapPartitionsRDD[44] at show at <console>:36)
16/07/19 23:34:55 INFO scheduler.TaskSchedulerImpl: Adding task set 18.0 with 1 tasks
16/07/19 23:34:55 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 18.0 (TID 416, localhost, ANY, 1413 bytes)
16/07/19 23:34:55 INFO executor.Executor: Running task 0.0 in stage 18.0 (TID 416)
16/07/19 23:34:55 INFO rdd.HadoopRDD: Input split: hdfs://cdh1:9000/user/test/customers.txt:0+92
16/07/19 23:34:55 INFO executor.Executor: Finished task 0.0 in stage 18.0 (TID 416). 2420 bytes result sent to driver
16/07/19 23:34:55 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 18.0 (TID 416) in 28 ms on localhost (1/1)
16/07/19 23:34:55 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 18.0, whose tasks have all completed, from pool
16/07/19 23:34:55 INFO scheduler.DAGScheduler: ResultStage 18 (show at <console>:36) finished in 0.027 s
16/07/19 23:34:55 INFO scheduler.DAGScheduler: Job 14 finished: show at <console>:36, took 0.057418 s
16/07/19 23:34:55 INFO spark.SparkContext: Starting job: show at <console>:36
16/07/19 23:34:55 INFO scheduler.DAGScheduler: Got job 15 (show at <console>:36) with 1 output partitions (allowLocal=false)
16/07/19 23:34:55 INFO scheduler.DAGScheduler: Final stage: ResultStage 19(show at <console>:36)
16/07/19 23:34:55 INFO scheduler.DAGScheduler: Parents of final stage: List()
16/07/19 23:34:55 INFO scheduler.DAGScheduler: Missing parents: List()
16/07/19 23:34:55 INFO scheduler.DAGScheduler: Submitting ResultStage 19 (MapPartitionsRDD[44] at show at <console>:36), which has no missing parents
16/07/19 23:34:55 INFO storage.MemoryStore: ensureFreeSpace(4088) called with curMem=1146060, maxMem=278019440
16/07/19 23:34:55 INFO storage.MemoryStore: Block broadcast_24 stored as values in memory (estimated size 4.0 KB, free 264.0 MB)
16/07/19 23:34:55 INFO storage.MemoryStore: ensureFreeSpace(2227) called with curMem=1150148, maxMem=278019440
16/07/19 23:34:55 INFO storage.MemoryStore: Block broadcast_24_piece0 stored as bytes in memory (estimated size 2.2 KB, free 264.0 MB)
16/07/19 23:34:55 INFO storage.BlockManagerInfo: Added broadcast_24_piece0 in memory on localhost:56137 (size: 2.2 KB, free: 265.0 MB)
16/07/19 23:34:55 INFO spark.SparkContext: Created broadcast 24 from broadcast at DAGScheduler.scala:874
16/07/19 23:34:55 INFO scheduler.DAGScheduler: Submitting 1 missing tasks from ResultStage 19 (MapPartitionsRDD[44] at show at <console>:36)
16/07/19 23:34:55 INFO scheduler.TaskSchedulerImpl: Adding task set 19.0 with 1 tasks
16/07/19 23:34:55 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 19.0 (TID 417, localhost, ANY, 1413 bytes)
16/07/19 23:34:55 INFO executor.Executor: Running task 0.0 in stage 19.0 (TID 417)
16/07/19 23:34:55 INFO rdd.HadoopRDD: Input split: hdfs://cdh1:9000/user/test/customers.txt:92+93
16/07/19 23:34:55 INFO executor.Executor: Finished task 0.0 in stage 19.0 (TID 417). 2311 bytes result sent to driver
16/07/19 23:34:55 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 19.0 (TID 417) in 14 ms on localhost (1/1)
16/07/19 23:34:55 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 19.0, whose tasks have all completed, from pool
16/07/19 23:34:55 INFO scheduler.DAGScheduler: ResultStage 19 (show at <console>:36) finished in 0.013 s
16/07/19 23:34:55 INFO scheduler.DAGScheduler: Job 15 finished: show at <console>:36, took 0.026368 s
+-----------+---------------+------------+-----+--------+
|customer_id| name| city|state|zip_code|
+-----------+---------------+------------+-----+--------+
| 100| John Smith| Austin| TX| 78727|
| 200| Joe Johnson| Dallas| TX| 75201|
| 300| Bob Jones| Houston| TX| 77028|
| 400| Andy Davis| San Antonio| TX| 78227|
| 500| James Williams| Austin| TX| 78727|
+-----------+---------------+------------+-----+--------+
scala> dfCustomers.printSchema();
root
|-- customer_id: integer (nullable = false)
|-- name: string (nullable = true)
|-- city: string (nullable = true)
|-- state: string (nullable = true)
|-- zip_code: string (nullable = true)
scala> dfCustomers.select("name").show();
16/07/19 23:35:11 INFO spark.SparkContext: Starting job: show at <console>:36
16/07/19 23:35:11 INFO scheduler.DAGScheduler: Got job 16 (show at <console>:36) with 1 output partitions (allowLocal=false)
16/07/19 23:35:11 INFO scheduler.DAGScheduler: Final stage: ResultStage 20(show at <console>:36)
16/07/19 23:35:11 INFO scheduler.DAGScheduler: Parents of final stage: List()
16/07/19 23:35:11 INFO scheduler.DAGScheduler: Missing parents: List()
16/07/19 23:35:11 INFO scheduler.DAGScheduler: Submitting ResultStage 20 (MapPartitionsRDD[46] at show at <console>:36), which has no missing parents
16/07/19 23:35:11 INFO storage.MemoryStore: ensureFreeSpace(5472) called with curMem=1152375, maxMem=278019440
16/07/19 23:35:11 INFO storage.MemoryStore: Block broadcast_25 stored as values in memory (estimated size 5.3 KB, free 264.0 MB)
16/07/19 23:35:11 INFO storage.MemoryStore: ensureFreeSpace(2881) called with curMem=1157847, maxMem=278019440
16/07/19 23:35:11 INFO storage.MemoryStore: Block broadcast_25_piece0 stored as bytes in memory (estimated size 2.8 KB, free 264.0 MB)
16/07/19 23:35:11 INFO storage.BlockManagerInfo: Added broadcast_25_piece0 in memory on localhost:56137 (size: 2.8 KB, free: 265.0 MB)
16/07/19 23:35:11 INFO spark.SparkContext: Created broadcast 25 from broadcast at DAGScheduler.scala:874
16/07/19 23:35:11 INFO scheduler.DAGScheduler: Submitting 1 missing tasks from ResultStage 20 (MapPartitionsRDD[46] at show at <console>:36)
16/07/19 23:35:11 INFO scheduler.TaskSchedulerImpl: Adding task set 20.0 with 1 tasks
16/07/19 23:35:11 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 20.0 (TID 418, localhost, ANY, 1413 bytes)
16/07/19 23:35:11 INFO executor.Executor: Running task 0.0 in stage 20.0 (TID 418)
16/07/19 23:35:11 INFO rdd.HadoopRDD: Input split: hdfs://cdh1:9000/user/test/customers.txt:0+92
16/07/19 23:35:11 INFO executor.Executor: Finished task 0.0 in stage 20.0 (TID 418). 2130 bytes result sent to driver
16/07/19 23:35:11 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 20.0 (TID 418) in 45 ms on localhost (1/1)
16/07/19 23:35:11 INFO scheduler.DAGScheduler: ResultStage 20 (show at <console>:36) finished in 0.045 s
16/07/19 23:35:11 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 20.0, whose tasks have all completed, from pool
16/07/19 23:35:11 INFO scheduler.DAGScheduler: Job 16 finished: show at <console>:36, took 0.064127 s
16/07/19 23:35:11 INFO spark.SparkContext: Starting job: show at <console>:36
16/07/19 23:35:11 INFO scheduler.DAGScheduler: Got job 17 (show at <console>:36) with 1 output partitions (allowLocal=false)
16/07/19 23:35:11 INFO scheduler.DAGScheduler: Final stage: ResultStage 21(show at <console>:36)
16/07/19 23:35:11 INFO scheduler.DAGScheduler: Parents of final stage: List()
16/07/19 23:35:11 INFO scheduler.DAGScheduler: Missing parents: List()
16/07/19 23:35:11 INFO scheduler.DAGScheduler: Submitting ResultStage 21 (MapPartitionsRDD[46] at show at <console>:36), which has no missing parents
16/07/19 23:35:11 INFO storage.MemoryStore: ensureFreeSpace(5472) called with curMem=1160728, maxMem=278019440
16/07/19 23:35:11 INFO storage.MemoryStore: Block broadcast_26 stored as values in memory (estimated size 5.3 KB, free 264.0 MB)
16/07/19 23:35:11 INFO storage.MemoryStore: ensureFreeSpace(2881) called with curMem=1166200, maxMem=278019440
16/07/19 23:35:11 INFO storage.MemoryStore: Block broadcast_26_piece0 stored as bytes in memory (estimated size 2.8 KB, free 264.0 MB)
16/07/19 23:35:11 INFO storage.BlockManagerInfo: Added broadcast_26_piece0 in memory on localhost:56137 (size: 2.8 KB, free: 265.0 MB)
16/07/19 23:35:11 INFO spark.SparkContext: Created broadcast 26 from broadcast at DAGScheduler.scala:874
16/07/19 23:35:11 INFO storage.BlockManagerInfo: Removed broadcast_25_piece0 on localhost:56137 in memory (size: 2.8 KB, free: 265.0 MB)
16/07/19 23:35:11 INFO scheduler.DAGScheduler: Submitting 1 missing tasks from ResultStage 21 (MapPartitionsRDD[46] at show at <console>:36)
16/07/19 23:35:11 INFO scheduler.TaskSchedulerImpl: Adding task set 21.0 with 1 tasks
16/07/19 23:35:11 INFO storage.BlockManagerInfo: Removed broadcast_24_piece0 on localhost:56137 in memory (size: 2.2 KB, free: 265.0 MB)
16/07/19 23:35:11 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 21.0 (TID 419, localhost, ANY, 1413 bytes)
16/07/19 23:35:11 INFO executor.Executor: Running task 0.0 in stage 21.0 (TID 419)
16/07/19 23:35:11 INFO storage.BlockManagerInfo: Removed broadcast_23_piece0 on localhost:56137 in memory (size: 2.2 KB, free: 265.0 MB)
16/07/19 23:35:11 INFO rdd.HadoopRDD: Input split: hdfs://cdh1:9000/user/test/customers.txt:92+93
16/07/19 23:35:11 INFO executor.Executor: Finished task 0.0 in stage 21.0 (TID 419). 2091 bytes result sent to driver
16/07/19 23:35:11 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 21.0 (TID 419) in 18 ms on localhost (1/1)
16/07/19 23:35:11 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 21.0, whose tasks have all completed, from pool
16/07/19 23:35:11 INFO scheduler.DAGScheduler: ResultStage 21 (show at <console>:36) finished in 0.019 s
16/07/19 23:35:11 INFO scheduler.DAGScheduler: Job 17 finished: show at <console>:36, took 0.060116 s
+---------------+
| name|
+---------------+
| John Smith|
| Joe Johnson|
| Bob Jones|
| Andy Davis|
| James Williams|
+---------------+
scala> dfCustomers.select("name", "city").show()
16/07/19 23:35:19 INFO spark.SparkContext: Starting job: show at <console>:36
16/07/19 23:35:19 INFO scheduler.DAGScheduler: Got job 18 (show at <console>:36) with 1 output partitions (allowLocal=false)
16/07/19 23:35:19 INFO scheduler.DAGScheduler: Final stage: ResultStage 22(show at <console>:36)
16/07/19 23:35:19 INFO scheduler.DAGScheduler: Parents of final stage: List()
16/07/19 23:35:19 INFO scheduler.DAGScheduler: Missing parents: List()
16/07/19 23:35:19 INFO scheduler.DAGScheduler: Submitting ResultStage 22 (MapPartitionsRDD[48] at show at <console>:36), which has no missing parents
16/07/19 23:35:19 INFO storage.MemoryStore: ensureFreeSpace(5480) called with curMem=1148098, maxMem=278019440
16/07/19 23:35:19 INFO storage.MemoryStore: Block broadcast_27 stored as values in memory (estimated size 5.4 KB, free 264.0 MB)
16/07/19 23:35:19 INFO storage.MemoryStore: ensureFreeSpace(2883) called with curMem=1153578, maxMem=278019440
16/07/19 23:35:19 INFO storage.MemoryStore: Block broadcast_27_piece0 stored as bytes in memory (estimated size 2.8 KB, free 264.0 MB)
16/07/19 23:35:19 INFO storage.BlockManagerInfo: Added broadcast_27_piece0 in memory on localhost:56137 (size: 2.8 KB, free: 265.0 MB)
16/07/19 23:35:19 INFO spark.SparkContext: Created broadcast 27 from broadcast at DAGScheduler.scala:874
16/07/19 23:35:19 INFO scheduler.DAGScheduler: Submitting 1 missing tasks from ResultStage 22 (MapPartitionsRDD[48] at show at <console>:36)
16/07/19 23:35:19 INFO scheduler.TaskSchedulerImpl: Adding task set 22.0 with 1 tasks
16/07/19 23:35:19 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 22.0 (TID 420, localhost, ANY, 1413 bytes)
16/07/19 23:35:19 INFO executor.Executor: Running task 0.0 in stage 22.0 (TID 420)
16/07/19 23:35:19 INFO rdd.HadoopRDD: Input split: hdfs://cdh1:9000/user/test/customers.txt:0+92
16/07/19 23:35:19 INFO executor.Executor: Finished task 0.0 in stage 22.0 (TID 420). 2200 bytes result sent to driver
16/07/19 23:35:19 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 22.0 (TID 420) in 17 ms on localhost (1/1)
16/07/19 23:35:19 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 22.0, whose tasks have all completed, from pool
16/07/19 23:35:19 INFO scheduler.DAGScheduler: ResultStage 22 (show at <console>:36) finished in 0.013 s
16/07/19 23:35:19 INFO scheduler.DAGScheduler: Job 18 finished: show at <console>:36, took 0.030147 s
16/07/19 23:35:19 INFO spark.SparkContext: Starting job: show at <console>:36
16/07/19 23:35:19 INFO scheduler.DAGScheduler: Got job 19 (show at <console>:36) with 1 output partitions (allowLocal=false)
16/07/19 23:35:19 INFO scheduler.DAGScheduler: Final stage: ResultStage 23(show at <console>:36)
16/07/19 23:35:19 INFO scheduler.DAGScheduler: Parents of final stage: List()
16/07/19 23:35:19 INFO scheduler.DAGScheduler: Missing parents: List()
16/07/19 23:35:19 INFO scheduler.DAGScheduler: Submitting ResultStage 23 (MapPartitionsRDD[48] at show at <console>:36), which has no missing parents
16/07/19 23:35:19 INFO storage.MemoryStore: ensureFreeSpace(5480) called with curMem=1156461, maxMem=278019440
16/07/19 23:35:19 INFO storage.MemoryStore: Block broadcast_28 stored as values in memory (estimated size 5.4 KB, free 264.0 MB)
16/07/19 23:35:19 INFO storage.MemoryStore: ensureFreeSpace(2883) called with curMem=1161941, maxMem=278019440
16/07/19 23:35:19 INFO storage.MemoryStore: Block broadcast_28_piece0 stored as bytes in memory (estimated size 2.8 KB, free 264.0 MB)
16/07/19 23:35:19 INFO storage.BlockManagerInfo: Added broadcast_28_piece0 in memory on localhost:56137 (size: 2.8 KB, free: 265.0 MB)
16/07/19 23:35:19 INFO spark.SparkContext: Created broadcast 28 from broadcast at DAGScheduler.scala:874
16/07/19 23:35:19 INFO scheduler.DAGScheduler: Submitting 1 missing tasks from ResultStage 23 (MapPartitionsRDD[48] at show at <console>:36)
16/07/19 23:35:19 INFO scheduler.TaskSchedulerImpl: Adding task set 23.0 with 1 tasks
16/07/19 23:35:19 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 23.0 (TID 421, localhost, ANY, 1413 bytes)
16/07/19 23:35:19 INFO executor.Executor: Running task 0.0 in stage 23.0 (TID 421)
16/07/19 23:35:19 INFO rdd.HadoopRDD: Input split: hdfs://cdh1:9000/user/test/customers.txt:92+93
16/07/19 23:35:19 INFO executor.Executor: Finished task 0.0 in stage 23.0 (TID 421). 2142 bytes result sent to driver
16/07/19 23:35:19 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 23.0 (TID 421) in 14 ms on localhost (1/1)
16/07/19 23:35:19 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 23.0, whose tasks have all completed, from pool
16/07/19 23:35:19 INFO scheduler.DAGScheduler: ResultStage 23 (show at <console>:36) finished in 0.014 s
16/07/19 23:35:19 INFO scheduler.DAGScheduler: Job 19 finished: show at <console>:36, took 0.027027 s
+---------------+------------+
| name| city|
+---------------+------------+
| John Smith| Austin|
| Joe Johnson| Dallas|
| Bob Jones| Houston|
| Andy Davis| San Antonio|
| James Williams| Austin|
+---------------+------------+
scala> dfCustomers.filter(dfCustomers("customer_id").equalTo(500)).show();
16/07/19 23:35:41 INFO spark.SparkContext: Starting job: show at <console>:36
16/07/19 23:35:41 INFO scheduler.DAGScheduler: Got job 20 (show at <console>:36) with 1 output partitions (allowLocal=false)
16/07/19 23:35:41 INFO scheduler.DAGScheduler: Final stage: ResultStage 24(show at <console>:36)
16/07/19 23:35:41 INFO scheduler.DAGScheduler: Parents of final stage: List()
16/07/19 23:35:41 INFO scheduler.DAGScheduler: Missing parents: List()
16/07/19 23:35:41 INFO scheduler.DAGScheduler: Submitting ResultStage 24 (MapPartitionsRDD[50] at show at <console>:36), which has no missing parents
16/07/19 23:35:41 INFO storage.MemoryStore: ensureFreeSpace(5632) called with curMem=1164824, maxMem=278019440
16/07/19 23:35:41 INFO storage.MemoryStore: Block broadcast_29 stored as values in memory (estimated size 5.5 KB, free 264.0 MB)
16/07/19 23:35:41 INFO storage.MemoryStore: ensureFreeSpace(2924) called with curMem=1170456, maxMem=278019440
16/07/19 23:35:41 INFO storage.MemoryStore: Block broadcast_29_piece0 stored as bytes in memory (estimated size 2.9 KB, free 264.0 MB)
16/07/19 23:35:41 INFO storage.BlockManagerInfo: Added broadcast_29_piece0 in memory on localhost:56137 (size: 2.9 KB, free: 265.0 MB)
16/07/19 23:35:41 INFO spark.SparkContext: Created broadcast 29 from broadcast at DAGScheduler.scala:874
16/07/19 23:35:41 INFO scheduler.DAGScheduler: Submitting 1 missing tasks from ResultStage 24 (MapPartitionsRDD[50] at show at <console>:36)
16/07/19 23:35:41 INFO scheduler.TaskSchedulerImpl: Adding task set 24.0 with 1 tasks
16/07/19 23:35:41 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 24.0 (TID 422, localhost, ANY, 1413 bytes)
16/07/19 23:35:41 INFO executor.Executor: Running task 0.0 in stage 24.0 (TID 422)
16/07/19 23:35:41 INFO rdd.HadoopRDD: Input split: hdfs://cdh1:9000/user/test/customers.txt:0+92
16/07/19 23:35:41 INFO executor.Executor: Finished task 0.0 in stage 24.0 (TID 422). 1800 bytes result sent to driver
16/07/19 23:35:41 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 24.0 (TID 422) in 41 ms on localhost (1/1)
16/07/19 23:35:41 INFO scheduler.DAGScheduler: ResultStage 24 (show at <console>:36) finished in 0.040 s
16/07/19 23:35:41 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 24.0, whose tasks have all completed, from pool
16/07/19 23:35:41 INFO scheduler.DAGScheduler: Job 20 finished: show at <console>:36, took 0.057741 s
16/07/19 23:35:41 INFO spark.SparkContext: Starting job: show at <console>:36
16/07/19 23:35:41 INFO scheduler.DAGScheduler: Got job 21 (show at <console>:36) with 1 output partitions (allowLocal=false)
16/07/19 23:35:41 INFO scheduler.DAGScheduler: Final stage: ResultStage 25(show at <console>:36)
16/07/19 23:35:41 INFO scheduler.DAGScheduler: Parents of final stage: List()
16/07/19 23:35:41 INFO scheduler.DAGScheduler: Missing parents: List()
16/07/19 23:35:41 INFO scheduler.DAGScheduler: Submitting ResultStage 25 (MapPartitionsRDD[50] at show at <console>:36), which has no missing parents
16/07/19 23:35:41 INFO storage.MemoryStore: ensureFreeSpace(5632) called with curMem=1173380, maxMem=278019440
16/07/19 23:35:41 INFO storage.MemoryStore: Block broadcast_30 stored as values in memory (estimated size 5.5 KB, free 264.0 MB)
16/07/19 23:35:41 INFO storage.MemoryStore: ensureFreeSpace(2924) called with curMem=1179012, maxMem=278019440
16/07/19 23:35:41 INFO storage.MemoryStore: Block broadcast_30_piece0 stored as bytes in memory (estimated size 2.9 KB, free 264.0 MB)
16/07/19 23:35:41 INFO storage.BlockManagerInfo: Added broadcast_30_piece0 in memory on localhost:56137 (size: 2.9 KB, free: 265.0 MB)
16/07/19 23:35:41 INFO spark.SparkContext: Created broadcast 30 from broadcast at DAGScheduler.scala:874
16/07/19 23:35:41 INFO scheduler.DAGScheduler: Submitting 1 missing tasks from ResultStage 25 (MapPartitionsRDD[50] at show at <console>:36)
16/07/19 23:35:41 INFO scheduler.TaskSchedulerImpl: Adding task set 25.0 with 1 tasks
16/07/19 23:35:41 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 25.0 (TID 423, localhost, ANY, 1413 bytes)
16/07/19 23:35:41 INFO executor.Executor: Running task 0.0 in stage 25.0 (TID 423)
16/07/19 23:35:41 INFO rdd.HadoopRDD: Input split: hdfs://cdh1:9000/user/test/customers.txt:92+93
16/07/19 23:35:41 INFO executor.Executor: Finished task 0.0 in stage 25.0 (TID 423). 2189 bytes result sent to driver
16/07/19 23:35:41 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 25.0 (TID 423) in 15 ms on localhost (1/1)
16/07/19 23:35:41 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 25.0, whose tasks have all completed, from pool
16/07/19 23:35:41 INFO scheduler.DAGScheduler: ResultStage 25 (show at <console>:36) finished in 0.015 s
16/07/19 23:35:41 INFO scheduler.DAGScheduler: Job 21 finished: show at <console>:36, took 0.038441 s
+-----------+---------------+-------+-----+--------+
|customer_id| name| city|state|zip_code|
+-----------+---------------+-------+-----+--------+
| 500| James Williams| Austin| TX| 78727|
+-----------+---------------+-------+-----+--------+
scala> dfCustomers.groupBy("zip_code").count().show();
16/07/19 23:35:52 INFO execution.Exchange: Using SparkSqlSerializer2.
16/07/19 23:35:52 INFO spark.SparkContext: Starting job: show at <console>:36
16/07/19 23:35:52 INFO scheduler.DAGScheduler: Registering RDD 53 (show at <console>:36)
16/07/19 23:35:52 INFO scheduler.DAGScheduler: Got job 22 (show at <console>:36) with 1 output partitions (allowLocal=false)
16/07/19 23:35:52 INFO scheduler.DAGScheduler: Final stage: ResultStage 27(show at <console>:36)
16/07/19 23:35:52 INFO scheduler.DAGScheduler: Parents of final stage: List(ShuffleMapStage 26)
16/07/19 23:35:52 INFO scheduler.DAGScheduler: Missing parents: List(ShuffleMapStage 26)
16/07/19 23:35:52 INFO scheduler.DAGScheduler: Submitting ShuffleMapStage 26 (MapPartitionsRDD[53] at show at <console>:36), which has no missing parents
16/07/19 23:35:52 INFO storage.MemoryStore: ensureFreeSpace(9456) called with curMem=1181936, maxMem=278019440
16/07/19 23:35:52 INFO storage.MemoryStore: Block broadcast_31 stored as values in memory (estimated size 9.2 KB, free 264.0 MB)
16/07/19 23:35:52 INFO storage.MemoryStore: ensureFreeSpace(4565) called with curMem=1191392, maxMem=278019440
16/07/19 23:35:52 INFO storage.MemoryStore: Block broadcast_31_piece0 stored as bytes in memory (estimated size 4.5 KB, free 264.0 MB)
16/07/19 23:35:52 INFO storage.BlockManagerInfo: Removed broadcast_29_piece0 on localhost:56137 in memory (size: 2.9 KB, free: 265.0 MB)
16/07/19 23:35:52 INFO storage.BlockManagerInfo: Added broadcast_31_piece0 in memory on localhost:56137 (size: 4.5 KB, free: 265.0 MB)
16/07/19 23:35:52 INFO storage.BlockManagerInfo: Removed broadcast_28_piece0 on localhost:56137 in memory (size: 2.8 KB, free: 265.0 MB)
16/07/19 23:35:52 INFO spark.SparkContext: Created broadcast 31 from broadcast at DAGScheduler.scala:874
16/07/19 23:35:52 INFO storage.BlockManagerInfo: Removed broadcast_27_piece0 on localhost:56137 in memory (size: 2.8 KB, free: 265.0 MB)
16/07/19 23:35:52 INFO scheduler.DAGScheduler: Submitting 2 missing tasks from ShuffleMapStage 26 (MapPartitionsRDD[53] at show at <console>:36)
16/07/19 23:35:52 INFO scheduler.TaskSchedulerImpl: Adding task set 26.0 with 2 tasks
16/07/19 23:35:52 INFO storage.BlockManagerInfo: Removed broadcast_26_piece0 on localhost:56137 in memory (size: 2.8 KB, free: 265.0 MB)
16/07/19 23:35:52 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 26.0 (TID 424, localhost, ANY, 1402 bytes)
16/07/19 23:35:52 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 26.0 (TID 425, localhost, ANY, 1402 bytes)
16/07/19 23:35:52 INFO executor.Executor: Running task 0.0 in stage 26.0 (TID 424)
16/07/19 23:35:52 INFO storage.BlockManagerInfo: Removed broadcast_30_piece0 on localhost:56137 in memory (size: 2.9 KB, free: 265.0 MB)
16/07/19 23:35:52 INFO rdd.HadoopRDD: Input split: hdfs://cdh1:9000/user/test/customers.txt:0+92
16/07/19 23:35:52 INFO executor.Executor: Running task 1.0 in stage 26.0 (TID 425)
16/07/19 23:35:52 INFO rdd.HadoopRDD: Input split: hdfs://cdh1:9000/user/test/customers.txt:92+93
16/07/19 23:35:52 INFO executor.Executor: Finished task 0.0 in stage 26.0 (TID 424). 2203 bytes result sent to driver
16/07/19 23:35:52 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 26.0 (TID 424) in 175 ms on localhost (1/2)
16/07/19 23:35:52 INFO executor.Executor: Finished task 1.0 in stage 26.0 (TID 425). 2203 bytes result sent to driver
16/07/19 23:35:52 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 26.0 (TID 425) in 181 ms on localhost (2/2)
16/07/19 23:35:52 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 26.0, whose tasks have all completed, from pool
16/07/19 23:35:52 INFO scheduler.DAGScheduler: ShuffleMapStage 26 (show at <console>:36) finished in 0.180 s
16/07/19 23:35:52 INFO scheduler.DAGScheduler: looking for newly runnable stages
16/07/19 23:35:52 INFO scheduler.DAGScheduler: running: Set()
16/07/19 23:35:52 INFO scheduler.DAGScheduler: waiting: Set(ResultStage 27)
16/07/19 23:35:52 INFO scheduler.DAGScheduler: failed: Set()
16/07/19 23:35:52 INFO scheduler.DAGScheduler: Missing parents for ResultStage 27: List()
16/07/19 23:35:52 INFO scheduler.DAGScheduler: Submitting ResultStage 27 (MapPartitionsRDD[57] at show at <console>:36), which is now runnable
16/07/19 23:35:52 INFO storage.MemoryStore: ensureFreeSpace(10280) called with curMem=1153766, maxMem=278019440
16/07/19 23:35:52 INFO storage.MemoryStore: Block broadcast_32 stored as values in memory (estimated size 10.0 KB, free 264.0 MB)
16/07/19 23:35:52 INFO storage.MemoryStore: ensureFreeSpace(4981) called with curMem=1164046, maxMem=278019440
16/07/19 23:35:52 INFO storage.MemoryStore: Block broadcast_32_piece0 stored as bytes in memory (estimated size 4.9 KB, free 264.0 MB)
16/07/19 23:35:52 INFO storage.BlockManagerInfo: Added broadcast_32_piece0 in memory on localhost:56137 (size: 4.9 KB, free: 265.0 MB)
16/07/19 23:35:52 INFO spark.SparkContext: Created broadcast 32 from broadcast at DAGScheduler.scala:874
16/07/19 23:35:52 INFO scheduler.DAGScheduler: Submitting 1 missing tasks from ResultStage 27 (MapPartitionsRDD[57] at show at <console>:36)
16/07/19 23:35:52 INFO scheduler.TaskSchedulerImpl: Adding task set 27.0 with 1 tasks
16/07/19 23:35:52 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 27.0 (TID 426, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:52 INFO executor.Executor: Running task 0.0 in stage 27.0 (TID 426)
16/07/19 23:35:52 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:52 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
16/07/19 23:35:52 INFO executor.Executor: Finished task 0.0 in stage 27.0 (TID 426). 894 bytes result sent to driver
16/07/19 23:35:52 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 27.0 (TID 426) in 7 ms on localhost (1/1)
16/07/19 23:35:52 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 27.0, whose tasks have all completed, from pool
16/07/19 23:35:52 INFO scheduler.DAGScheduler: ResultStage 27 (show at <console>:36) finished in 0.007 s
16/07/19 23:35:52 INFO scheduler.DAGScheduler: Job 22 finished: show at <console>:36, took 0.261327 s
16/07/19 23:35:52 INFO spark.SparkContext: Starting job: show at <console>:36
16/07/19 23:35:52 INFO spark.MapOutputTrackerMaster: Size of output statuses for shuffle 2 is 169 bytes
16/07/19 23:35:52 INFO scheduler.DAGScheduler: Got job 23 (show at <console>:36) with 199 output partitions (allowLocal=false)
16/07/19 23:35:52 INFO scheduler.DAGScheduler: Final stage: ResultStage 29(show at <console>:36)
16/07/19 23:35:52 INFO scheduler.DAGScheduler: Parents of final stage: List(ShuffleMapStage 28)
16/07/19 23:35:52 INFO scheduler.DAGScheduler: Missing parents: List()
16/07/19 23:35:52 INFO scheduler.DAGScheduler: Submitting ResultStage 29 (MapPartitionsRDD[57] at show at <console>:36), which has no missing parents
16/07/19 23:35:52 INFO storage.MemoryStore: ensureFreeSpace(10280) called with curMem=1169027, maxMem=278019440
16/07/19 23:35:52 INFO storage.MemoryStore: Block broadcast_33 stored as values in memory (estimated size 10.0 KB, free 264.0 MB)
16/07/19 23:35:52 INFO storage.MemoryStore: ensureFreeSpace(4981) called with curMem=1179307, maxMem=278019440
16/07/19 23:35:52 INFO storage.MemoryStore: Block broadcast_33_piece0 stored as bytes in memory (estimated size 4.9 KB, free 264.0 MB)
16/07/19 23:35:52 INFO storage.BlockManagerInfo: Added broadcast_33_piece0 in memory on localhost:56137 (size: 4.9 KB, free: 265.0 MB)
16/07/19 23:35:53 INFO spark.SparkContext: Created broadcast 33 from broadcast at DAGScheduler.scala:874
16/07/19 23:35:53 INFO scheduler.DAGScheduler: Submitting 199 missing tasks from ResultStage 29 (MapPartitionsRDD[57] at show at <console>:36)
16/07/19 23:35:53 INFO scheduler.TaskSchedulerImpl: Adding task set 29.0 with 199 tasks
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 29.0 (TID 427, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 29.0 (TID 428, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO executor.Executor: Running task 0.0 in stage 29.0 (TID 427)
16/07/19 23:35:53 INFO executor.Executor: Running task 1.0 in stage 29.0 (TID 428)
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/07/19 23:35:53 INFO executor.Executor: Finished task 1.0 in stage 29.0 (TID 428). 894 bytes result sent to driver
16/07/19 23:35:53 INFO executor.Executor: Finished task 0.0 in stage 29.0 (TID 427). 894 bytes result sent to driver
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 29.0 (TID 429, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO executor.Executor: Running task 2.0 in stage 29.0 (TID 429)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 29.0 (TID 430, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 29.0 (TID 427) in 11 ms on localhost (1/199)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 29.0 (TID 428) in 10 ms on localhost (2/199)
16/07/19 23:35:53 INFO executor.Executor: Running task 3.0 in stage 29.0 (TID 430)
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/07/19 23:35:53 INFO executor.Executor: Finished task 3.0 in stage 29.0 (TID 430). 894 bytes result sent to driver
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 4.0 in stage 29.0 (TID 431, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO executor.Executor: Running task 4.0 in stage 29.0 (TID 431)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 29.0 (TID 430) in 8 ms on localhost (3/199)
16/07/19 23:35:53 INFO executor.Executor: Finished task 2.0 in stage 29.0 (TID 429). 894 bytes result sent to driver
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 5.0 in stage 29.0 (TID 432, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 29.0 (TID 429) in 12 ms on localhost (4/199)
16/07/19 23:35:53 INFO executor.Executor: Running task 5.0 in stage 29.0 (TID 432)
16/07/19 23:35:53 INFO executor.Executor: Finished task 4.0 in stage 29.0 (TID 431). 894 bytes result sent to driver
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 6.0 in stage 29.0 (TID 433, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 4.0 in stage 29.0 (TID 431) in 6 ms on localhost (5/199)
16/07/19 23:35:53 INFO executor.Executor: Running task 6.0 in stage 29.0 (TID 433)
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/07/19 23:35:53 INFO executor.Executor: Finished task 6.0 in stage 29.0 (TID 433). 894 bytes result sent to driver
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 7.0 in stage 29.0 (TID 434, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO executor.Executor: Finished task 5.0 in stage 29.0 (TID 432). 894 bytes result sent to driver
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 8.0 in stage 29.0 (TID 435, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO executor.Executor: Running task 8.0 in stage 29.0 (TID 435)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 5.0 in stage 29.0 (TID 432) in 13 ms on localhost (6/199)
16/07/19 23:35:53 INFO executor.Executor: Running task 7.0 in stage 29.0 (TID 434)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 6.0 in stage 29.0 (TID 433) in 10 ms on localhost (7/199)
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/07/19 23:35:53 INFO executor.Executor: Finished task 7.0 in stage 29.0 (TID 434). 894 bytes result sent to driver
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 9.0 in stage 29.0 (TID 436, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO executor.Executor: Running task 9.0 in stage 29.0 (TID 436)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 7.0 in stage 29.0 (TID 434) in 10 ms on localhost (8/199)
16/07/19 23:35:53 INFO executor.Executor: Finished task 8.0 in stage 29.0 (TID 435). 894 bytes result sent to driver
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 10.0 in stage 29.0 (TID 437, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 8.0 in stage 29.0 (TID 435) in 11 ms on localhost (9/199)
16/07/19 23:35:53 INFO executor.Executor: Running task 10.0 in stage 29.0 (TID 437)
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/07/19 23:35:53 INFO executor.Executor: Finished task 10.0 in stage 29.0 (TID 437). 894 bytes result sent to driver
16/07/19 23:35:53 INFO executor.Executor: Finished task 9.0 in stage 29.0 (TID 436). 894 bytes result sent to driver
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 11.0 in stage 29.0 (TID 438, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 12.0 in stage 29.0 (TID 439, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO executor.Executor: Running task 11.0 in stage 29.0 (TID 438)
16/07/19 23:35:53 INFO executor.Executor: Running task 12.0 in stage 29.0 (TID 439)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 10.0 in stage 29.0 (TID 437) in 10 ms on localhost (10/199)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 9.0 in stage 29.0 (TID 436) in 13 ms on localhost (11/199)
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/07/19 23:35:53 INFO executor.Executor: Finished task 12.0 in stage 29.0 (TID 439). 894 bytes result sent to driver
16/07/19 23:35:53 INFO executor.Executor: Finished task 11.0 in stage 29.0 (TID 438). 894 bytes result sent to driver
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 13.0 in stage 29.0 (TID 440, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 14.0 in stage 29.0 (TID 441, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO executor.Executor: Running task 14.0 in stage 29.0 (TID 441)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 12.0 in stage 29.0 (TID 439) in 9 ms on localhost (12/199)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 11.0 in stage 29.0 (TID 438) in 11 ms on localhost (13/199)
16/07/19 23:35:53 INFO executor.Executor: Running task 13.0 in stage 29.0 (TID 440)
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/07/19 23:35:53 INFO executor.Executor: Finished task 14.0 in stage 29.0 (TID 441). 894 bytes result sent to driver
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 15.0 in stage 29.0 (TID 442, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 14.0 in stage 29.0 (TID 441) in 9 ms on localhost (14/199)
16/07/19 23:35:53 INFO executor.Executor: Running task 15.0 in stage 29.0 (TID 442)
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/07/19 23:35:53 INFO executor.Executor: Finished task 13.0 in stage 29.0 (TID 440). 894 bytes result sent to driver
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 16.0 in stage 29.0 (TID 443, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO executor.Executor: Running task 16.0 in stage 29.0 (TID 443)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 13.0 in stage 29.0 (TID 440) in 15 ms on localhost (15/199)
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/07/19 23:35:53 INFO executor.Executor: Finished task 15.0 in stage 29.0 (TID 442). 894 bytes result sent to driver
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 17.0 in stage 29.0 (TID 444, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO executor.Executor: Running task 17.0 in stage 29.0 (TID 444)
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/07/19 23:35:53 INFO executor.Executor: Finished task 16.0 in stage 29.0 (TID 443). 894 bytes result sent to driver
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 15.0 in stage 29.0 (TID 442) in 11 ms on localhost (16/199)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 18.0 in stage 29.0 (TID 445, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 16.0 in stage 29.0 (TID 443) in 13 ms on localhost (17/199)
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/07/19 23:35:53 INFO executor.Executor: Running task 18.0 in stage 29.0 (TID 445)
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/07/19 23:35:53 INFO executor.Executor: Finished task 17.0 in stage 29.0 (TID 444). 894 bytes result sent to driver
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 19.0 in stage 29.0 (TID 446, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 17.0 in stage 29.0 (TID 444) in 17 ms on localhost (18/199)
16/07/19 23:35:53 INFO executor.Executor: Running task 19.0 in stage 29.0 (TID 446)
16/07/19 23:35:53 INFO executor.Executor: Finished task 18.0 in stage 29.0 (TID 445). 894 bytes result sent to driver
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 20.0 in stage 29.0 (TID 447, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 18.0 in stage 29.0 (TID 445) in 14 ms on localhost (19/199)
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
16/07/19 23:35:53 INFO executor.Executor: Running task 20.0 in stage 29.0 (TID 447)
16/07/19 23:35:53 INFO executor.Executor: Finished task 19.0 in stage 29.0 (TID 446). 894 bytes result sent to driver
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 21.0 in stage 29.0 (TID 448, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 19.0 in stage 29.0 (TID 446) in 9 ms on localhost (20/199)
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
16/07/19 23:35:53 INFO executor.Executor: Finished task 20.0 in stage 29.0 (TID 447). 894 bytes result sent to driver
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 22.0 in stage 29.0 (TID 449, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 20.0 in stage 29.0 (TID 447) in 12 ms on localhost (21/199)
16/07/19 23:35:53 INFO executor.Executor: Running task 21.0 in stage 29.0 (TID 448)
16/07/19 23:35:53 INFO executor.Executor: Running task 22.0 in stage 29.0 (TID 449)
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/07/19 23:35:53 INFO executor.Executor: Finished task 22.0 in stage 29.0 (TID 449). 894 bytes result sent to driver
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 23.0 in stage 29.0 (TID 450, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO executor.Executor: Running task 23.0 in stage 29.0 (TID 450)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 22.0 in stage 29.0 (TID 449) in 7 ms on localhost (22/199)
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/07/19 23:35:53 INFO executor.Executor: Finished task 21.0 in stage 29.0 (TID 448). 894 bytes result sent to driver
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 24.0 in stage 29.0 (TID 451, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO executor.Executor: Running task 24.0 in stage 29.0 (TID 451)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 21.0 in stage 29.0 (TID 448) in 20 ms on localhost (23/199)
16/07/19 23:35:53 INFO executor.Executor: Finished task 23.0 in stage 29.0 (TID 450). 894 bytes result sent to driver
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 25.0 in stage 29.0 (TID 452, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 23.0 in stage 29.0 (TID 450) in 13 ms on localhost (24/199)
16/07/19 23:35:53 INFO executor.Executor: Running task 25.0 in stage 29.0 (TID 452)
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/07/19 23:35:53 INFO executor.Executor: Finished task 25.0 in stage 29.0 (TID 452). 894 bytes result sent to driver
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 26.0 in stage 29.0 (TID 453, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 25.0 in stage 29.0 (TID 452) in 7 ms on localhost (25/199)
16/07/19 23:35:53 INFO executor.Executor: Running task 26.0 in stage 29.0 (TID 453)
16/07/19 23:35:53 INFO executor.Executor: Finished task 24.0 in stage 29.0 (TID 451). 894 bytes result sent to driver
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 27.0 in stage 29.0 (TID 454, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 24.0 in stage 29.0 (TID 451) in 16 ms on localhost (26/199)
16/07/19 23:35:53 INFO executor.Executor: Running task 27.0 in stage 29.0 (TID 454)
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
16/07/19 23:35:53 INFO executor.Executor: Finished task 26.0 in stage 29.0 (TID 453). 894 bytes result sent to driver
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 28.0 in stage 29.0 (TID 455, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO executor.Executor: Finished task 27.0 in stage 29.0 (TID 454). 894 bytes result sent to driver
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 26.0 in stage 29.0 (TID 453) in 10 ms on localhost (27/199)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 29.0 in stage 29.0 (TID 456, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 27.0 in stage 29.0 (TID 454) in 7 ms on localhost (28/199)
16/07/19 23:35:53 INFO executor.Executor: Running task 29.0 in stage 29.0 (TID 456)
16/07/19 23:35:53 INFO executor.Executor: Running task 28.0 in stage 29.0 (TID 455)
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/07/19 23:35:53 INFO executor.Executor: Finished task 29.0 in stage 29.0 (TID 456). 894 bytes result sent to driver
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 30.0 in stage 29.0 (TID 457, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 29.0 in stage 29.0 (TID 456) in 13 ms on localhost (29/199)
16/07/19 23:35:53 INFO executor.Executor: Running task 30.0 in stage 29.0 (TID 457)
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
16/07/19 23:35:53 INFO executor.Executor: Finished task 30.0 in stage 29.0 (TID 457). 894 bytes result sent to driver
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 31.0 in stage 29.0 (TID 458, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 30.0 in stage 29.0 (TID 457) in 8 ms on localhost (30/199)
16/07/19 23:35:53 INFO executor.Executor: Running task 31.0 in stage 29.0 (TID 458)
16/07/19 23:35:53 INFO executor.Executor: Finished task 28.0 in stage 29.0 (TID 455). 894 bytes result sent to driver
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 32.0 in stage 29.0 (TID 459, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 28.0 in stage 29.0 (TID 455) in 24 ms on localhost (31/199)
16/07/19 23:35:53 INFO executor.Executor: Running task 32.0 in stage 29.0 (TID 459)
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/07/19 23:35:53 INFO executor.Executor: Finished task 32.0 in stage 29.0 (TID 459). 894 bytes result sent to driver
16/07/19 23:35:53 INFO executor.Executor: Finished task 31.0 in stage 29.0 (TID 458). 894 bytes result sent to driver
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 33.0 in stage 29.0 (TID 460, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 32.0 in stage 29.0 (TID 459) in 10 ms on localhost (32/199)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 31.0 in stage 29.0 (TID 458) in 14 ms on localhost (33/199)
16/07/19 23:35:53 INFO executor.Executor: Running task 33.0 in stage 29.0 (TID 460)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 34.0 in stage 29.0 (TID 461, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO executor.Executor: Running task 34.0 in stage 29.0 (TID 461)
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/07/19 23:35:53 INFO executor.Executor: Finished task 34.0 in stage 29.0 (TID 461). 894 bytes result sent to driver
16/07/19 23:35:53 INFO executor.Executor: Finished task 33.0 in stage 29.0 (TID 460). 894 bytes result sent to driver
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 35.0 in stage 29.0 (TID 462, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO executor.Executor: Running task 35.0 in stage 29.0 (TID 462)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 36.0 in stage 29.0 (TID 463, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 34.0 in stage 29.0 (TID 461) in 5 ms on localhost (34/199)
16/07/19 23:35:53 INFO executor.Executor: Running task 36.0 in stage 29.0 (TID 463)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 33.0 in stage 29.0 (TID 460) in 9 ms on localhost (35/199)
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO executor.Executor: Finished task 36.0 in stage 29.0 (TID 463). 894 bytes result sent to driver
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 37.0 in stage 29.0 (TID 464, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO executor.Executor: Running task 37.0 in stage 29.0 (TID 464)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 36.0 in stage 29.0 (TID 463) in 8 ms on localhost (36/199)
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/07/19 23:35:53 INFO executor.Executor: Finished task 35.0 in stage 29.0 (TID 462). 894 bytes result sent to driver
16/07/19 23:35:53 INFO executor.Executor: Finished task 37.0 in stage 29.0 (TID 464). 894 bytes result sent to driver
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 38.0 in stage 29.0 (TID 465, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO executor.Executor: Running task 38.0 in stage 29.0 (TID 465)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 39.0 in stage 29.0 (TID 466, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 35.0 in stage 29.0 (TID 462) in 16 ms on localhost (37/199)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 37.0 in stage 29.0 (TID 464) in 9 ms on localhost (38/199)
16/07/19 23:35:53 INFO executor.Executor: Running task 39.0 in stage 29.0 (TID 466)
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/07/19 23:35:53 INFO executor.Executor: Finished task 38.0 in stage 29.0 (TID 465). 894 bytes result sent to driver
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 40.0 in stage 29.0 (TID 467, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 38.0 in stage 29.0 (TID 465) in 7 ms on localhost (39/199)
16/07/19 23:35:53 INFO executor.Executor: Running task 40.0 in stage 29.0 (TID 467)
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
16/07/19 23:35:53 INFO executor.Executor: Finished task 39.0 in stage 29.0 (TID 466). 894 bytes result sent to driver
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 41.0 in stage 29.0 (TID 468, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 39.0 in stage 29.0 (TID 466) in 11 ms on localhost (40/199)
16/07/19 23:35:53 INFO executor.Executor: Running task 41.0 in stage 29.0 (TID 468)
16/07/19 23:35:53 INFO executor.Executor: Finished task 40.0 in stage 29.0 (TID 467). 894 bytes result sent to driver
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 42.0 in stage 29.0 (TID 469, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO executor.Executor: Running task 42.0 in stage 29.0 (TID 469)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 40.0 in stage 29.0 (TID 467) in 8 ms on localhost (41/199)
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/07/19 23:35:53 INFO executor.Executor: Finished task 41.0 in stage 29.0 (TID 468). 894 bytes result sent to driver
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 43.0 in stage 29.0 (TID 470, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 41.0 in stage 29.0 (TID 468) in 8 ms on localhost (42/199)
16/07/19 23:35:53 INFO executor.Executor: Running task 43.0 in stage 29.0 (TID 470)
16/07/19 23:35:53 INFO executor.Executor: Finished task 42.0 in stage 29.0 (TID 469). 894 bytes result sent to driver
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 44.0 in stage 29.0 (TID 471, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO executor.Executor: Running task 44.0 in stage 29.0 (TID 471)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 42.0 in stage 29.0 (TID 469) in 10 ms on localhost (43/199)
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
16/07/19 23:35:53 INFO executor.Executor: Finished task 43.0 in stage 29.0 (TID 470). 894 bytes result sent to driver
16/07/19 23:35:53 INFO executor.Executor: Finished task 44.0 in stage 29.0 (TID 471). 894 bytes result sent to driver
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 45.0 in stage 29.0 (TID 472, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 46.0 in stage 29.0 (TID 473, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 44.0 in stage 29.0 (TID 471) in 6 ms on localhost (44/199)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 43.0 in stage 29.0 (TID 470) in 11 ms on localhost (45/199)
16/07/19 23:35:53 INFO executor.Executor: Running task 45.0 in stage 29.0 (TID 472)
16/07/19 23:35:53 INFO executor.Executor: Running task 46.0 in stage 29.0 (TID 473)
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
16/07/19 23:35:53 INFO executor.Executor: Finished task 46.0 in stage 29.0 (TID 473). 1211 bytes result sent to driver
16/07/19 23:35:53 INFO executor.Executor: Finished task 45.0 in stage 29.0 (TID 472). 894 bytes result sent to driver
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 47.0 in stage 29.0 (TID 474, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO executor.Executor: Running task 47.0 in stage 29.0 (TID 474)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 48.0 in stage 29.0 (TID 475, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 46.0 in stage 29.0 (TID 473) in 9 ms on localhost (46/199)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 45.0 in stage 29.0 (TID 472) in 9 ms on localhost (47/199)
16/07/19 23:35:53 INFO executor.Executor: Running task 48.0 in stage 29.0 (TID 475)
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
16/07/19 23:35:53 INFO executor.Executor: Finished task 48.0 in stage 29.0 (TID 475). 894 bytes result sent to driver
16/07/19 23:35:53 INFO executor.Executor: Finished task 47.0 in stage 29.0 (TID 474). 894 bytes result sent to driver
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 49.0 in stage 29.0 (TID 476, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO executor.Executor: Running task 49.0 in stage 29.0 (TID 476)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 48.0 in stage 29.0 (TID 475) in 10 ms on localhost (48/199)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 50.0 in stage 29.0 (TID 477, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 47.0 in stage 29.0 (TID 474) in 14 ms on localhost (49/199)
16/07/19 23:35:53 INFO executor.Executor: Running task 50.0 in stage 29.0 (TID 477)
16/07/19 23:35:53 INFO storage.BlockManagerInfo: Removed broadcast_32_piece0 on localhost:56137 in memory (size: 4.9 KB, free: 265.0 MB)
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.BlockManagerInfo: Removed broadcast_31_piece0 on localhost:56137 in memory (size: 4.5 KB, free: 265.0 MB)
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/07/19 23:35:53 INFO executor.Executor: Finished task 50.0 in stage 29.0 (TID 477). 894 bytes result sent to driver
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 51.0 in stage 29.0 (TID 478, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 50.0 in stage 29.0 (TID 477) in 27 ms on localhost (50/199)
16/07/19 23:35:53 INFO executor.Executor: Finished task 49.0 in stage 29.0 (TID 476). 894 bytes result sent to driver
16/07/19 23:35:53 INFO executor.Executor: Running task 51.0 in stage 29.0 (TID 478)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 52.0 in stage 29.0 (TID 479, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 49.0 in stage 29.0 (TID 476) in 31 ms on localhost (51/199)
16/07/19 23:35:53 INFO executor.Executor: Running task 52.0 in stage 29.0 (TID 479)
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO executor.Executor: Finished task 51.0 in stage 29.0 (TID 478). 894 bytes result sent to driver
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 53.0 in stage 29.0 (TID 480, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO executor.Executor: Running task 53.0 in stage 29.0 (TID 480)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 51.0 in stage 29.0 (TID 478) in 11 ms on localhost (52/199)
16/07/19 23:35:53 INFO executor.Executor: Finished task 52.0 in stage 29.0 (TID 479). 894 bytes result sent to driver
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 52.0 in stage 29.0 (TID 479) in 13 ms on localhost (53/199)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 54.0 in stage 29.0 (TID 481, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO executor.Executor: Running task 54.0 in stage 29.0 (TID 481)
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/07/19 23:35:53 INFO executor.Executor: Finished task 53.0 in stage 29.0 (TID 480). 894 bytes result sent to driver
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 53.0 in stage 29.0 (TID 480) in 10 ms on localhost (54/199)
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 55.0 in stage 29.0 (TID 482, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO executor.Executor: Running task 55.0 in stage 29.0 (TID 482)
16/07/19 23:35:53 INFO executor.Executor: Finished task 54.0 in stage 29.0 (TID 481). 894 bytes result sent to driver
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 56.0 in stage 29.0 (TID 483, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO executor.Executor: Running task 56.0 in stage 29.0 (TID 483)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 54.0 in stage 29.0 (TID 481) in 8 ms on localhost (55/199)
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/07/19 23:35:53 INFO executor.Executor: Finished task 56.0 in stage 29.0 (TID 483). 894 bytes result sent to driver
16/07/19 23:35:53 INFO executor.Executor: Finished task 55.0 in stage 29.0 (TID 482). 894 bytes result sent to driver
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 57.0 in stage 29.0 (TID 484, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO executor.Executor: Running task 57.0 in stage 29.0 (TID 484)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 58.0 in stage 29.0 (TID 485, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO executor.Executor: Running task 58.0 in stage 29.0 (TID 485)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 55.0 in stage 29.0 (TID 482) in 11 ms on localhost (56/199)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 56.0 in stage 29.0 (TID 483) in 10 ms on localhost (57/199)
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/07/19 23:35:53 INFO executor.Executor: Finished task 58.0 in stage 29.0 (TID 485). 894 bytes result sent to driver
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 59.0 in stage 29.0 (TID 486, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 58.0 in stage 29.0 (TID 485) in 8 ms on localhost (58/199)
16/07/19 23:35:53 INFO executor.Executor: Running task 59.0 in stage 29.0 (TID 486)
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/07/19 23:35:53 INFO executor.Executor: Finished task 57.0 in stage 29.0 (TID 484). 894 bytes result sent to driver
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 60.0 in stage 29.0 (TID 487, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 57.0 in stage 29.0 (TID 484) in 13 ms on localhost (59/199)
16/07/19 23:35:53 INFO executor.Executor: Running task 60.0 in stage 29.0 (TID 487)
16/07/19 23:35:53 INFO executor.Executor: Finished task 59.0 in stage 29.0 (TID 486). 894 bytes result sent to driver
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 61.0 in stage 29.0 (TID 488, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 59.0 in stage 29.0 (TID 486) in 9 ms on localhost (60/199)
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/07/19 23:35:53 INFO executor.Executor: Running task 61.0 in stage 29.0 (TID 488)
16/07/19 23:35:53 INFO executor.Executor: Finished task 60.0 in stage 29.0 (TID 487). 894 bytes result sent to driver
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 62.0 in stage 29.0 (TID 489, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 60.0 in stage 29.0 (TID 487) in 9 ms on localhost (61/199)
16/07/19 23:35:53 INFO executor.Executor: Running task 62.0 in stage 29.0 (TID 489)
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/07/19 23:35:53 INFO executor.Executor: Finished task 61.0 in stage 29.0 (TID 488). 894 bytes result sent to driver
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 63.0 in stage 29.0 (TID 490, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 61.0 in stage 29.0 (TID 488) in 12 ms on localhost (62/199)
16/07/19 23:35:53 INFO executor.Executor: Finished task 62.0 in stage 29.0 (TID 489). 894 bytes result sent to driver
16/07/19 23:35:53 INFO executor.Executor: Running task 63.0 in stage 29.0 (TID 490)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 64.0 in stage 29.0 (TID 491, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 62.0 in stage 29.0 (TID 489) in 9 ms on localhost (63/199)
16/07/19 23:35:53 INFO executor.Executor: Running task 64.0 in stage 29.0 (TID 491)
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
16/07/19 23:35:53 INFO executor.Executor: Finished task 64.0 in stage 29.0 (TID 491). 894 bytes result sent to driver
16/07/19 23:35:53 INFO executor.Executor: Finished task 63.0 in stage 29.0 (TID 490). 894 bytes result sent to driver
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 65.0 in stage 29.0 (TID 492, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 64.0 in stage 29.0 (TID 491) in 6 ms on localhost (64/199)
16/07/19 23:35:53 INFO executor.Executor: Running task 65.0 in stage 29.0 (TID 492)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 66.0 in stage 29.0 (TID 493, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 63.0 in stage 29.0 (TID 490) in 8 ms on localhost (65/199)
16/07/19 23:35:53 INFO executor.Executor: Running task 66.0 in stage 29.0 (TID 493)
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/07/19 23:35:53 INFO executor.Executor: Finished task 65.0 in stage 29.0 (TID 492). 894 bytes result sent to driver
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 67.0 in stage 29.0 (TID 494, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO executor.Executor: Running task 67.0 in stage 29.0 (TID 494)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 65.0 in stage 29.0 (TID 492) in 9 ms on localhost (66/199)
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/07/19 23:35:53 INFO executor.Executor: Finished task 66.0 in stage 29.0 (TID 493). 894 bytes result sent to driver
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 68.0 in stage 29.0 (TID 495, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO executor.Executor: Running task 68.0 in stage 29.0 (TID 495)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 66.0 in stage 29.0 (TID 493) in 11 ms on localhost (67/199)
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/07/19 23:35:53 INFO executor.Executor: Finished task 68.0 in stage 29.0 (TID 495). 894 bytes result sent to driver
16/07/19 23:35:53 INFO executor.Executor: Finished task 67.0 in stage 29.0 (TID 494). 894 bytes result sent to driver
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 69.0 in stage 29.0 (TID 496, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO executor.Executor: Running task 69.0 in stage 29.0 (TID 496)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 70.0 in stage 29.0 (TID 497, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 67.0 in stage 29.0 (TID 494) in 10 ms on localhost (68/199)
16/07/19 23:35:53 INFO executor.Executor: Running task 70.0 in stage 29.0 (TID 497)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 68.0 in stage 29.0 (TID 495) in 6 ms on localhost (69/199)
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/07/19 23:35:53 INFO executor.Executor: Finished task 70.0 in stage 29.0 (TID 497). 894 bytes result sent to driver
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 71.0 in stage 29.0 (TID 498, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO executor.Executor: Running task 71.0 in stage 29.0 (TID 498)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 70.0 in stage 29.0 (TID 497) in 5 ms on localhost (70/199)
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/07/19 23:35:53 INFO executor.Executor: Finished task 69.0 in stage 29.0 (TID 496). 894 bytes result sent to driver
16/07/19 23:35:53 INFO executor.Executor: Finished task 71.0 in stage 29.0 (TID 498). 894 bytes result sent to driver
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 72.0 in stage 29.0 (TID 499, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 73.0 in stage 29.0 (TID 500, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 69.0 in stage 29.0 (TID 496) in 12 ms on localhost (71/199)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 71.0 in stage 29.0 (TID 498) in 7 ms on localhost (72/199)
16/07/19 23:35:53 INFO executor.Executor: Running task 73.0 in stage 29.0 (TID 500)
16/07/19 23:35:53 INFO executor.Executor: Running task 72.0 in stage 29.0 (TID 499)
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/07/19 23:35:53 INFO executor.Executor: Finished task 73.0 in stage 29.0 (TID 500). 894 bytes result sent to driver
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 74.0 in stage 29.0 (TID 501, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO executor.Executor: Running task 74.0 in stage 29.0 (TID 501)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 73.0 in stage 29.0 (TID 500) in 7 ms on localhost (73/199)
16/07/19 23:35:53 INFO executor.Executor: Finished task 72.0 in stage 29.0 (TID 499). 894 bytes result sent to driver
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 75.0 in stage 29.0 (TID 502, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 72.0 in stage 29.0 (TID 499) in 12 ms on localhost (74/199)
16/07/19 23:35:53 INFO executor.Executor: Finished task 74.0 in stage 29.0 (TID 501). 894 bytes result sent to driver
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 76.0 in stage 29.0 (TID 503, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 74.0 in stage 29.0 (TID 501) in 7 ms on localhost (75/199)
16/07/19 23:35:53 INFO executor.Executor: Running task 76.0 in stage 29.0 (TID 503)
16/07/19 23:35:53 INFO executor.Executor: Running task 75.0 in stage 29.0 (TID 502)
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/07/19 23:35:53 INFO executor.Executor: Finished task 76.0 in stage 29.0 (TID 503). 894 bytes result sent to driver
16/07/19 23:35:53 INFO executor.Executor: Finished task 75.0 in stage 29.0 (TID 502). 894 bytes result sent to driver
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 77.0 in stage 29.0 (TID 504, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO executor.Executor: Running task 77.0 in stage 29.0 (TID 504)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 78.0 in stage 29.0 (TID 505, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 75.0 in stage 29.0 (TID 502) in 11 ms on localhost (76/199)
16/07/19 23:35:53 INFO executor.Executor: Running task 78.0 in stage 29.0 (TID 505)
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 76.0 in stage 29.0 (TID 503) in 13 ms on localhost (77/199)
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/07/19 23:35:53 INFO executor.Executor: Finished task 78.0 in stage 29.0 (TID 505). 894 bytes result sent to driver
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 79.0 in stage 29.0 (TID 506, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 78.0 in stage 29.0 (TID 505) in 9 ms on localhost (78/199)
16/07/19 23:35:53 INFO executor.Executor: Finished task 77.0 in stage 29.0 (TID 504). 894 bytes result sent to driver
16/07/19 23:35:53 INFO executor.Executor: Running task 79.0 in stage 29.0 (TID 506)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 77.0 in stage 29.0 (TID 504) in 11 ms on localhost (79/199)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 80.0 in stage 29.0 (TID 507, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO executor.Executor: Running task 80.0 in stage 29.0 (TID 507)
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/07/19 23:35:53 INFO executor.Executor: Finished task 79.0 in stage 29.0 (TID 506). 894 bytes result sent to driver
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 81.0 in stage 29.0 (TID 508, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 79.0 in stage 29.0 (TID 506) in 10 ms on localhost (80/199)
16/07/19 23:35:53 INFO executor.Executor: Running task 81.0 in stage 29.0 (TID 508)
16/07/19 23:35:53 INFO executor.Executor: Finished task 80.0 in stage 29.0 (TID 507). 894 bytes result sent to driver
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/07/19 23:35:53 INFO executor.Executor: Finished task 81.0 in stage 29.0 (TID 508). 894 bytes result sent to driver
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 82.0 in stage 29.0 (TID 509, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 80.0 in stage 29.0 (TID 507) in 11 ms on localhost (81/199)
16/07/19 23:35:53 INFO executor.Executor: Running task 82.0 in stage 29.0 (TID 509)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 83.0 in stage 29.0 (TID 510, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 81.0 in stage 29.0 (TID 508) in 6 ms on localhost (82/199)
16/07/19 23:35:53 INFO executor.Executor: Running task 83.0 in stage 29.0 (TID 510)
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/07/19 23:35:53 INFO executor.Executor: Finished task 82.0 in stage 29.0 (TID 509). 894 bytes result sent to driver
16/07/19 23:35:53 INFO executor.Executor: Finished task 83.0 in stage 29.0 (TID 510). 894 bytes result sent to driver
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 84.0 in stage 29.0 (TID 511, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 82.0 in stage 29.0 (TID 509) in 8 ms on localhost (83/199)
16/07/19 23:35:53 INFO executor.Executor: Running task 84.0 in stage 29.0 (TID 511)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 85.0 in stage 29.0 (TID 512, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 83.0 in stage 29.0 (TID 510) in 8 ms on localhost (84/199)
16/07/19 23:35:53 INFO executor.Executor: Running task 85.0 in stage 29.0 (TID 512)
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/07/19 23:35:53 INFO executor.Executor: Finished task 84.0 in stage 29.0 (TID 511). 894 bytes result sent to driver
16/07/19 23:35:53 INFO executor.Executor: Finished task 85.0 in stage 29.0 (TID 512). 894 bytes result sent to driver
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 86.0 in stage 29.0 (TID 513, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO executor.Executor: Running task 86.0 in stage 29.0 (TID 513)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 87.0 in stage 29.0 (TID 514, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 84.0 in stage 29.0 (TID 511) in 8 ms on localhost (85/199)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 85.0 in stage 29.0 (TID 512) in 7 ms on localhost (86/199)
16/07/19 23:35:53 INFO executor.Executor: Running task 87.0 in stage 29.0 (TID 514)
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/07/19 23:35:53 INFO executor.Executor: Finished task 86.0 in stage 29.0 (TID 513). 894 bytes result sent to driver
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 88.0 in stage 29.0 (TID 515, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO executor.Executor: Finished task 87.0 in stage 29.0 (TID 514). 1211 bytes result sent to driver
16/07/19 23:35:53 INFO executor.Executor: Running task 88.0 in stage 29.0 (TID 515)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 86.0 in stage 29.0 (TID 513) in 7 ms on localhost (87/199)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 89.0 in stage 29.0 (TID 516, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO executor.Executor: Running task 89.0 in stage 29.0 (TID 516)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 87.0 in stage 29.0 (TID 514) in 10 ms on localhost (88/199)
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/07/19 23:35:53 INFO executor.Executor: Finished task 88.0 in stage 29.0 (TID 515). 894 bytes result sent to driver
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 90.0 in stage 29.0 (TID 517, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 88.0 in stage 29.0 (TID 515) in 10 ms on localhost (89/199)
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/07/19 23:35:53 INFO executor.Executor: Running task 90.0 in stage 29.0 (TID 517)
16/07/19 23:35:53 INFO executor.Executor: Finished task 89.0 in stage 29.0 (TID 516). 894 bytes result sent to driver
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 91.0 in stage 29.0 (TID 518, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 89.0 in stage 29.0 (TID 516) in 11 ms on localhost (90/199)
16/07/19 23:35:53 INFO executor.Executor: Running task 91.0 in stage 29.0 (TID 518)
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/07/19 23:35:53 INFO executor.Executor: Finished task 91.0 in stage 29.0 (TID 518). 894 bytes result sent to driver
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 92.0 in stage 29.0 (TID 519, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO executor.Executor: Running task 92.0 in stage 29.0 (TID 519)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 91.0 in stage 29.0 (TID 518) in 7 ms on localhost (91/199)
16/07/19 23:35:53 INFO executor.Executor: Finished task 90.0 in stage 29.0 (TID 517). 894 bytes result sent to driver
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 93.0 in stage 29.0 (TID 520, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO executor.Executor: Running task 93.0 in stage 29.0 (TID 520)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 90.0 in stage 29.0 (TID 517) in 13 ms on localhost (92/199)
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/07/19 23:35:53 INFO executor.Executor: Finished task 92.0 in stage 29.0 (TID 519). 1211 bytes result sent to driver
16/07/19 23:35:53 INFO executor.Executor: Finished task 93.0 in stage 29.0 (TID 520). 894 bytes result sent to driver
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 94.0 in stage 29.0 (TID 521, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO executor.Executor: Running task 94.0 in stage 29.0 (TID 521)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 95.0 in stage 29.0 (TID 522, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 92.0 in stage 29.0 (TID 519) in 8 ms on localhost (93/199)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 93.0 in stage 29.0 (TID 520) in 7 ms on localhost (94/199)
16/07/19 23:35:53 INFO executor.Executor: Running task 95.0 in stage 29.0 (TID 522)
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/07/19 23:35:53 INFO executor.Executor: Finished task 95.0 in stage 29.0 (TID 522). 894 bytes result sent to driver
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 96.0 in stage 29.0 (TID 523, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 95.0 in stage 29.0 (TID 522) in 11 ms on localhost (95/199)
16/07/19 23:35:53 INFO executor.Executor: Running task 96.0 in stage 29.0 (TID 523)
16/07/19 23:35:53 INFO executor.Executor: Finished task 94.0 in stage 29.0 (TID 521). 894 bytes result sent to driver
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 97.0 in stage 29.0 (TID 524, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 94.0 in stage 29.0 (TID 521) in 14 ms on localhost (96/199)
16/07/19 23:35:53 INFO executor.Executor: Running task 97.0 in stage 29.0 (TID 524)
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/07/19 23:35:53 INFO executor.Executor: Finished task 96.0 in stage 29.0 (TID 523). 894 bytes result sent to driver
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 98.0 in stage 29.0 (TID 525, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 96.0 in stage 29.0 (TID 523) in 11 ms on localhost (97/199)
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/07/19 23:35:53 INFO executor.Executor: Running task 98.0 in stage 29.0 (TID 525)
16/07/19 23:35:53 INFO executor.Executor: Finished task 97.0 in stage 29.0 (TID 524). 894 bytes result sent to driver
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 99.0 in stage 29.0 (TID 526, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 97.0 in stage 29.0 (TID 524) in 12 ms on localhost (98/199)
16/07/19 23:35:53 INFO executor.Executor: Running task 99.0 in stage 29.0 (TID 526)
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
16/07/19 23:35:53 INFO executor.Executor: Finished task 98.0 in stage 29.0 (TID 525). 894 bytes result sent to driver
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 100.0 in stage 29.0 (TID 527, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 98.0 in stage 29.0 (TID 525) in 11 ms on localhost (99/199)
16/07/19 23:35:53 INFO executor.Executor: Running task 100.0 in stage 29.0 (TID 527)
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 2 ms
16/07/19 23:35:53 INFO executor.Executor: Finished task 99.0 in stage 29.0 (TID 526). 894 bytes result sent to driver
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 101.0 in stage 29.0 (TID 528, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 99.0 in stage 29.0 (TID 526) in 11 ms on localhost (100/199)
16/07/19 23:35:53 INFO executor.Executor: Running task 101.0 in stage 29.0 (TID 528)
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
16/07/19 23:35:53 INFO executor.Executor: Finished task 100.0 in stage 29.0 (TID 527). 894 bytes result sent to driver
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 102.0 in stage 29.0 (TID 529, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 100.0 in stage 29.0 (TID 527) in 9 ms on localhost (101/199)
16/07/19 23:35:53 INFO executor.Executor: Running task 102.0 in stage 29.0 (TID 529)
16/07/19 23:35:53 INFO executor.Executor: Finished task 101.0 in stage 29.0 (TID 528). 894 bytes result sent to driver
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 103.0 in stage 29.0 (TID 530, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 101.0 in stage 29.0 (TID 528) in 9 ms on localhost (102/199)
16/07/19 23:35:53 INFO executor.Executor: Running task 103.0 in stage 29.0 (TID 530)
16/07/19 23:35:53 INFO executor.Executor: Finished task 102.0 in stage 29.0 (TID 529). 894 bytes result sent to driver
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 104.0 in stage 29.0 (TID 531, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 102.0 in stage 29.0 (TID 529) in 7 ms on localhost (103/199)
16/07/19 23:35:53 INFO executor.Executor: Running task 104.0 in stage 29.0 (TID 531)
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
16/07/19 23:35:53 INFO executor.Executor: Finished task 104.0 in stage 29.0 (TID 531). 894 bytes result sent to driver
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 105.0 in stage 29.0 (TID 532, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 104.0 in stage 29.0 (TID 531) in 7 ms on localhost (104/199)
16/07/19 23:35:53 INFO executor.Executor: Running task 105.0 in stage 29.0 (TID 532)
16/07/19 23:35:53 INFO executor.Executor: Finished task 103.0 in stage 29.0 (TID 530). 894 bytes result sent to driver
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 106.0 in stage 29.0 (TID 533, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 103.0 in stage 29.0 (TID 530) in 13 ms on localhost (105/199)
16/07/19 23:35:53 INFO executor.Executor: Running task 106.0 in stage 29.0 (TID 533)
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/07/19 23:35:53 INFO executor.Executor: Finished task 105.0 in stage 29.0 (TID 532). 894 bytes result sent to driver
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 107.0 in stage 29.0 (TID 534, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 105.0 in stage 29.0 (TID 532) in 9 ms on localhost (106/199)
16/07/19 23:35:53 INFO executor.Executor: Running task 107.0 in stage 29.0 (TID 534)
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/07/19 23:35:53 INFO executor.Executor: Finished task 107.0 in stage 29.0 (TID 534). 894 bytes result sent to driver
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 108.0 in stage 29.0 (TID 535, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 107.0 in stage 29.0 (TID 534) in 7 ms on localhost (107/199)
16/07/19 23:35:53 INFO executor.Executor: Running task 108.0 in stage 29.0 (TID 535)
16/07/19 23:35:53 INFO executor.Executor: Finished task 106.0 in stage 29.0 (TID 533). 894 bytes result sent to driver
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
16/07/19 23:35:53 INFO executor.Executor: Finished task 108.0 in stage 29.0 (TID 535). 894 bytes result sent to driver
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 109.0 in stage 29.0 (TID 536, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 106.0 in stage 29.0 (TID 533) in 19 ms on localhost (108/199)
16/07/19 23:35:53 INFO executor.Executor: Running task 109.0 in stage 29.0 (TID 536)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 110.0 in stage 29.0 (TID 537, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 108.0 in stage 29.0 (TID 535) in 9 ms on localhost (109/199)
16/07/19 23:35:53 INFO executor.Executor: Running task 110.0 in stage 29.0 (TID 537)
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/07/19 23:35:53 INFO executor.Executor: Finished task 110.0 in stage 29.0 (TID 537). 894 bytes result sent to driver
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 111.0 in stage 29.0 (TID 538, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 110.0 in stage 29.0 (TID 537) in 7 ms on localhost (110/199)
16/07/19 23:35:53 INFO executor.Executor: Running task 111.0 in stage 29.0 (TID 538)
16/07/19 23:35:53 INFO executor.Executor: Finished task 109.0 in stage 29.0 (TID 536). 894 bytes result sent to driver
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 112.0 in stage 29.0 (TID 539, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO executor.Executor: Running task 112.0 in stage 29.0 (TID 539)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 109.0 in stage 29.0 (TID 536) in 13 ms on localhost (111/199)
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 2 ms
16/07/19 23:35:53 INFO executor.Executor: Finished task 111.0 in stage 29.0 (TID 538). 894 bytes result sent to driver
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 113.0 in stage 29.0 (TID 540, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO executor.Executor: Running task 113.0 in stage 29.0 (TID 540)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 111.0 in stage 29.0 (TID 538) in 8 ms on localhost (112/199)
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/07/19 23:35:53 INFO executor.Executor: Finished task 112.0 in stage 29.0 (TID 539). 894 bytes result sent to driver
16/07/19 23:35:53 INFO executor.Executor: Finished task 113.0 in stage 29.0 (TID 540). 894 bytes result sent to driver
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 114.0 in stage 29.0 (TID 541, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO executor.Executor: Running task 114.0 in stage 29.0 (TID 541)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 115.0 in stage 29.0 (TID 542, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO executor.Executor: Running task 115.0 in stage 29.0 (TID 542)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 112.0 in stage 29.0 (TID 539) in 12 ms on localhost (113/199)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 113.0 in stage 29.0 (TID 540) in 10 ms on localhost (114/199)
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/07/19 23:35:53 INFO executor.Executor: Finished task 114.0 in stage 29.0 (TID 541). 894 bytes result sent to driver
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 114.0 in stage 29.0 (TID 541) in 6 ms on localhost (115/199)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 116.0 in stage 29.0 (TID 543, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO executor.Executor: Running task 116.0 in stage 29.0 (TID 543)
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/07/19 23:35:53 INFO executor.Executor: Finished task 116.0 in stage 29.0 (TID 543). 894 bytes result sent to driver
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 116.0 in stage 29.0 (TID 543) in 5 ms on localhost (116/199)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 117.0 in stage 29.0 (TID 544, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO executor.Executor: Running task 117.0 in stage 29.0 (TID 544)
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
16/07/19 23:35:53 INFO executor.Executor: Finished task 115.0 in stage 29.0 (TID 542). 894 bytes result sent to driver
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 118.0 in stage 29.0 (TID 545, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 115.0 in stage 29.0 (TID 542) in 18 ms on localhost (117/199)
16/07/19 23:35:53 INFO executor.Executor: Finished task 117.0 in stage 29.0 (TID 544). 894 bytes result sent to driver
16/07/19 23:35:53 INFO executor.Executor: Running task 118.0 in stage 29.0 (TID 545)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 119.0 in stage 29.0 (TID 546, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 117.0 in stage 29.0 (TID 544) in 10 ms on localhost (118/199)
16/07/19 23:35:53 INFO executor.Executor: Running task 119.0 in stage 29.0 (TID 546)
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO executor.Executor: Finished task 119.0 in stage 29.0 (TID 546). 894 bytes result sent to driver
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 6 ms
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 120.0 in stage 29.0 (TID 547, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 119.0 in stage 29.0 (TID 546) in 15 ms on localhost (119/199)
16/07/19 23:35:53 INFO executor.Executor: Running task 120.0 in stage 29.0 (TID 547)
16/07/19 23:35:53 INFO executor.Executor: Finished task 118.0 in stage 29.0 (TID 545). 894 bytes result sent to driver
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/07/19 23:35:53 INFO executor.Executor: Finished task 120.0 in stage 29.0 (TID 547). 894 bytes result sent to driver
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 121.0 in stage 29.0 (TID 548, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 122.0 in stage 29.0 (TID 549, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 118.0 in stage 29.0 (TID 545) in 24 ms on localhost (120/199)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 120.0 in stage 29.0 (TID 547) in 9 ms on localhost (121/199)
16/07/19 23:35:53 INFO executor.Executor: Running task 122.0 in stage 29.0 (TID 549)
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/07/19 23:35:53 INFO executor.Executor: Running task 121.0 in stage 29.0 (TID 548)
16/07/19 23:35:53 INFO executor.Executor: Finished task 122.0 in stage 29.0 (TID 549). 894 bytes result sent to driver
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 123.0 in stage 29.0 (TID 550, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 122.0 in stage 29.0 (TID 549) in 14 ms on localhost (122/199)
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/07/19 23:35:53 INFO executor.Executor: Running task 123.0 in stage 29.0 (TID 550)
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
16/07/19 23:35:53 INFO executor.Executor: Finished task 123.0 in stage 29.0 (TID 550). 894 bytes result sent to driver
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 124.0 in stage 29.0 (TID 551, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO executor.Executor: Running task 124.0 in stage 29.0 (TID 551)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 123.0 in stage 29.0 (TID 550) in 9 ms on localhost (123/199)
16/07/19 23:35:53 INFO executor.Executor: Finished task 121.0 in stage 29.0 (TID 548). 894 bytes result sent to driver
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 125.0 in stage 29.0 (TID 552, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO executor.Executor: Running task 125.0 in stage 29.0 (TID 552)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 121.0 in stage 29.0 (TID 548) in 25 ms on localhost (124/199)
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/07/19 23:35:53 INFO executor.Executor: Finished task 124.0 in stage 29.0 (TID 551). 894 bytes result sent to driver
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 126.0 in stage 29.0 (TID 553, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO executor.Executor: Running task 126.0 in stage 29.0 (TID 553)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 124.0 in stage 29.0 (TID 551) in 8 ms on localhost (125/199)
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/07/19 23:35:53 INFO executor.Executor: Finished task 125.0 in stage 29.0 (TID 552). 894 bytes result sent to driver
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 127.0 in stage 29.0 (TID 554, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 125.0 in stage 29.0 (TID 552) in 9 ms on localhost (126/199)
16/07/19 23:35:53 INFO executor.Executor: Finished task 126.0 in stage 29.0 (TID 553). 894 bytes result sent to driver
16/07/19 23:35:53 INFO executor.Executor: Running task 127.0 in stage 29.0 (TID 554)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 128.0 in stage 29.0 (TID 555, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 126.0 in stage 29.0 (TID 553) in 6 ms on localhost (127/199)
16/07/19 23:35:53 INFO executor.Executor: Running task 128.0 in stage 29.0 (TID 555)
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/07/19 23:35:53 INFO executor.Executor: Finished task 127.0 in stage 29.0 (TID 554). 894 bytes result sent to driver
16/07/19 23:35:53 INFO executor.Executor: Finished task 128.0 in stage 29.0 (TID 555). 894 bytes result sent to driver
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 129.0 in stage 29.0 (TID 556, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 127.0 in stage 29.0 (TID 554) in 8 ms on localhost (128/199)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 130.0 in stage 29.0 (TID 557, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 128.0 in stage 29.0 (TID 555) in 8 ms on localhost (129/199)
16/07/19 23:35:53 INFO executor.Executor: Running task 130.0 in stage 29.0 (TID 557)
16/07/19 23:35:53 INFO executor.Executor: Running task 129.0 in stage 29.0 (TID 556)
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/07/19 23:35:53 INFO executor.Executor: Finished task 130.0 in stage 29.0 (TID 557). 894 bytes result sent to driver
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 131.0 in stage 29.0 (TID 558, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 130.0 in stage 29.0 (TID 557) in 7 ms on localhost (130/199)
16/07/19 23:35:53 INFO executor.Executor: Running task 131.0 in stage 29.0 (TID 558)
16/07/19 23:35:53 INFO executor.Executor: Finished task 129.0 in stage 29.0 (TID 556). 894 bytes result sent to driver
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 132.0 in stage 29.0 (TID 559, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 129.0 in stage 29.0 (TID 556) in 12 ms on localhost (131/199)
16/07/19 23:35:53 INFO executor.Executor: Finished task 131.0 in stage 29.0 (TID 558). 894 bytes result sent to driver
16/07/19 23:35:53 INFO executor.Executor: Running task 132.0 in stage 29.0 (TID 559)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 133.0 in stage 29.0 (TID 560, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 131.0 in stage 29.0 (TID 558) in 7 ms on localhost (132/199)
16/07/19 23:35:53 INFO executor.Executor: Running task 133.0 in stage 29.0 (TID 560)
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/07/19 23:35:53 INFO executor.Executor: Finished task 132.0 in stage 29.0 (TID 559). 894 bytes result sent to driver
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 134.0 in stage 29.0 (TID 561, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 132.0 in stage 29.0 (TID 559) in 10 ms on localhost (133/199)
16/07/19 23:35:53 INFO executor.Executor: Running task 134.0 in stage 29.0 (TID 561)
16/07/19 23:35:53 INFO executor.Executor: Finished task 133.0 in stage 29.0 (TID 560). 894 bytes result sent to driver
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 135.0 in stage 29.0 (TID 562, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 133.0 in stage 29.0 (TID 560) in 10 ms on localhost (134/199)
16/07/19 23:35:53 INFO executor.Executor: Running task 135.0 in stage 29.0 (TID 562)
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/07/19 23:35:53 INFO executor.Executor: Finished task 135.0 in stage 29.0 (TID 562). 894 bytes result sent to driver
16/07/19 23:35:53 INFO executor.Executor: Finished task 134.0 in stage 29.0 (TID 561). 894 bytes result sent to driver
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 136.0 in stage 29.0 (TID 563, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO executor.Executor: Running task 136.0 in stage 29.0 (TID 563)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 135.0 in stage 29.0 (TID 562) in 7 ms on localhost (135/199)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 137.0 in stage 29.0 (TID 564, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO executor.Executor: Running task 137.0 in stage 29.0 (TID 564)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 134.0 in stage 29.0 (TID 561) in 13 ms on localhost (136/199)
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/07/19 23:35:53 INFO executor.Executor: Finished task 136.0 in stage 29.0 (TID 563). 894 bytes result sent to driver
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 138.0 in stage 29.0 (TID 565, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 136.0 in stage 29.0 (TID 563) in 10 ms on localhost (137/199)
16/07/19 23:35:53 INFO executor.Executor: Running task 138.0 in stage 29.0 (TID 565)
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/07/19 23:35:53 INFO executor.Executor: Finished task 137.0 in stage 29.0 (TID 564). 894 bytes result sent to driver
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/07/19 23:35:53 INFO executor.Executor: Finished task 138.0 in stage 29.0 (TID 565). 894 bytes result sent to driver
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 139.0 in stage 29.0 (TID 566, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 140.0 in stage 29.0 (TID 567, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 137.0 in stage 29.0 (TID 564) in 16 ms on localhost (138/199)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 138.0 in stage 29.0 (TID 565) in 11 ms on localhost (139/199)
16/07/19 23:35:53 INFO executor.Executor: Running task 140.0 in stage 29.0 (TID 567)
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO executor.Executor: Running task 139.0 in stage 29.0 (TID 566)
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
16/07/19 23:35:53 INFO executor.Executor: Finished task 140.0 in stage 29.0 (TID 567). 894 bytes result sent to driver
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 141.0 in stage 29.0 (TID 568, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 140.0 in stage 29.0 (TID 567) in 8 ms on localhost (140/199)
16/07/19 23:35:53 INFO executor.Executor: Running task 141.0 in stage 29.0 (TID 568)
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/07/19 23:35:53 INFO executor.Executor: Finished task 141.0 in stage 29.0 (TID 568). 894 bytes result sent to driver
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 142.0 in stage 29.0 (TID 569, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO executor.Executor: Running task 142.0 in stage 29.0 (TID 569)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 141.0 in stage 29.0 (TID 568) in 6 ms on localhost (141/199)
16/07/19 23:35:53 INFO executor.Executor: Finished task 139.0 in stage 29.0 (TID 566). 894 bytes result sent to driver
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 143.0 in stage 29.0 (TID 570, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO executor.Executor: Running task 143.0 in stage 29.0 (TID 570)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 139.0 in stage 29.0 (TID 566) in 16 ms on localhost (142/199)
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/07/19 23:35:53 INFO executor.Executor: Finished task 143.0 in stage 29.0 (TID 570). 894 bytes result sent to driver
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 144.0 in stage 29.0 (TID 571, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 143.0 in stage 29.0 (TID 570) in 6 ms on localhost (143/199)
16/07/19 23:35:53 INFO executor.Executor: Running task 144.0 in stage 29.0 (TID 571)
16/07/19 23:35:53 INFO executor.Executor: Finished task 142.0 in stage 29.0 (TID 569). 894 bytes result sent to driver
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 145.0 in stage 29.0 (TID 572, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 142.0 in stage 29.0 (TID 569) in 9 ms on localhost (144/199)
16/07/19 23:35:53 INFO executor.Executor: Running task 145.0 in stage 29.0 (TID 572)
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
16/07/19 23:35:53 INFO executor.Executor: Finished task 145.0 in stage 29.0 (TID 572). 894 bytes result sent to driver
16/07/19 23:35:53 INFO executor.Executor: Finished task 144.0 in stage 29.0 (TID 571). 894 bytes result sent to driver
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 146.0 in stage 29.0 (TID 573, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO executor.Executor: Running task 146.0 in stage 29.0 (TID 573)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 147.0 in stage 29.0 (TID 574, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 145.0 in stage 29.0 (TID 572) in 7 ms on localhost (145/199)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 144.0 in stage 29.0 (TID 571) in 9 ms on localhost (146/199)
16/07/19 23:35:53 INFO executor.Executor: Running task 147.0 in stage 29.0 (TID 574)
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/07/19 23:35:53 INFO executor.Executor: Finished task 146.0 in stage 29.0 (TID 573). 894 bytes result sent to driver
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 148.0 in stage 29.0 (TID 575, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 146.0 in stage 29.0 (TID 573) in 9 ms on localhost (147/199)
16/07/19 23:35:53 INFO executor.Executor: Running task 148.0 in stage 29.0 (TID 575)
16/07/19 23:35:53 INFO executor.Executor: Finished task 147.0 in stage 29.0 (TID 574). 894 bytes result sent to driver
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 147.0 in stage 29.0 (TID 574) in 10 ms on localhost (148/199)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 149.0 in stage 29.0 (TID 576, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/07/19 23:35:53 INFO executor.Executor: Running task 149.0 in stage 29.0 (TID 576)
16/07/19 23:35:53 INFO executor.Executor: Finished task 148.0 in stage 29.0 (TID 575). 894 bytes result sent to driver
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/07/19 23:35:53 INFO executor.Executor: Finished task 149.0 in stage 29.0 (TID 576). 894 bytes result sent to driver
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 148.0 in stage 29.0 (TID 575) in 10 ms on localhost (149/199)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 150.0 in stage 29.0 (TID 577, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO executor.Executor: Running task 150.0 in stage 29.0 (TID 577)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 149.0 in stage 29.0 (TID 576) in 9 ms on localhost (150/199)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 151.0 in stage 29.0 (TID 578, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO executor.Executor: Running task 151.0 in stage 29.0 (TID 578)
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/07/19 23:35:53 INFO executor.Executor: Finished task 150.0 in stage 29.0 (TID 577). 894 bytes result sent to driver
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 152.0 in stage 29.0 (TID 579, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO executor.Executor: Finished task 151.0 in stage 29.0 (TID 578). 894 bytes result sent to driver
16/07/19 23:35:53 INFO executor.Executor: Running task 152.0 in stage 29.0 (TID 579)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 150.0 in stage 29.0 (TID 577) in 9 ms on localhost (151/199)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 151.0 in stage 29.0 (TID 578) in 7 ms on localhost (152/199)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 153.0 in stage 29.0 (TID 580, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO executor.Executor: Running task 153.0 in stage 29.0 (TID 580)
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/07/19 23:35:53 INFO executor.Executor: Finished task 152.0 in stage 29.0 (TID 579). 894 bytes result sent to driver
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 154.0 in stage 29.0 (TID 581, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO executor.Executor: Running task 154.0 in stage 29.0 (TID 581)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 152.0 in stage 29.0 (TID 579) in 10 ms on localhost (153/199)
16/07/19 23:35:53 INFO executor.Executor: Finished task 153.0 in stage 29.0 (TID 580). 894 bytes result sent to driver
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 155.0 in stage 29.0 (TID 582, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 153.0 in stage 29.0 (TID 580) in 9 ms on localhost (154/199)
16/07/19 23:35:53 INFO executor.Executor: Running task 155.0 in stage 29.0 (TID 582)
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/07/19 23:35:53 INFO executor.Executor: Finished task 154.0 in stage 29.0 (TID 581). 894 bytes result sent to driver
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 156.0 in stage 29.0 (TID 583, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 154.0 in stage 29.0 (TID 581) in 7 ms on localhost (155/199)
16/07/19 23:35:53 INFO executor.Executor: Running task 156.0 in stage 29.0 (TID 583)
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/07/19 23:35:53 INFO executor.Executor: Finished task 155.0 in stage 29.0 (TID 582). 894 bytes result sent to driver
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 157.0 in stage 29.0 (TID 584, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 155.0 in stage 29.0 (TID 582) in 11 ms on localhost (156/199)
16/07/19 23:35:53 INFO executor.Executor: Finished task 156.0 in stage 29.0 (TID 583). 894 bytes result sent to driver
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 158.0 in stage 29.0 (TID 585, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO executor.Executor: Running task 157.0 in stage 29.0 (TID 584)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 156.0 in stage 29.0 (TID 583) in 8 ms on localhost (157/199)
16/07/19 23:35:53 INFO executor.Executor: Running task 158.0 in stage 29.0 (TID 585)
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/07/19 23:35:53 INFO executor.Executor: Finished task 157.0 in stage 29.0 (TID 584). 894 bytes result sent to driver
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 159.0 in stage 29.0 (TID 586, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO executor.Executor: Running task 159.0 in stage 29.0 (TID 586)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 157.0 in stage 29.0 (TID 584) in 11 ms on localhost (158/199)
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO executor.Executor: Finished task 158.0 in stage 29.0 (TID 585). 894 bytes result sent to driver
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 160.0 in stage 29.0 (TID 587, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 158.0 in stage 29.0 (TID 585) in 13 ms on localhost (159/199)
16/07/19 23:35:53 INFO executor.Executor: Running task 160.0 in stage 29.0 (TID 587)
16/07/19 23:35:53 INFO executor.Executor: Finished task 159.0 in stage 29.0 (TID 586). 894 bytes result sent to driver
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 161.0 in stage 29.0 (TID 588, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 159.0 in stage 29.0 (TID 586) in 7 ms on localhost (160/199)
16/07/19 23:35:53 INFO executor.Executor: Running task 161.0 in stage 29.0 (TID 588)
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/07/19 23:35:53 INFO executor.Executor: Finished task 160.0 in stage 29.0 (TID 587). 894 bytes result sent to driver
16/07/19 23:35:53 INFO executor.Executor: Finished task 161.0 in stage 29.0 (TID 588). 894 bytes result sent to driver
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 162.0 in stage 29.0 (TID 589, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO executor.Executor: Running task 162.0 in stage 29.0 (TID 589)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 163.0 in stage 29.0 (TID 590, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 160.0 in stage 29.0 (TID 587) in 9 ms on localhost (161/199)
16/07/19 23:35:53 INFO executor.Executor: Running task 163.0 in stage 29.0 (TID 590)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 161.0 in stage 29.0 (TID 588) in 8 ms on localhost (162/199)
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/07/19 23:35:53 INFO executor.Executor: Finished task 163.0 in stage 29.0 (TID 590). 894 bytes result sent to driver
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 164.0 in stage 29.0 (TID 591, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO executor.Executor: Finished task 162.0 in stage 29.0 (TID 589). 894 bytes result sent to driver
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 163.0 in stage 29.0 (TID 590) in 9 ms on localhost (163/199)
16/07/19 23:35:53 INFO executor.Executor: Running task 164.0 in stage 29.0 (TID 591)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 165.0 in stage 29.0 (TID 592, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 162.0 in stage 29.0 (TID 589) in 11 ms on localhost (164/199)
16/07/19 23:35:53 INFO executor.Executor: Running task 165.0 in stage 29.0 (TID 592)
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
16/07/19 23:35:53 INFO executor.Executor: Finished task 164.0 in stage 29.0 (TID 591). 894 bytes result sent to driver
16/07/19 23:35:53 INFO executor.Executor: Finished task 165.0 in stage 29.0 (TID 592). 894 bytes result sent to driver
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 166.0 in stage 29.0 (TID 593, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 167.0 in stage 29.0 (TID 594, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO executor.Executor: Running task 167.0 in stage 29.0 (TID 594)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 164.0 in stage 29.0 (TID 591) in 9 ms on localhost (165/199)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 165.0 in stage 29.0 (TID 592) in 8 ms on localhost (166/199)
16/07/19 23:35:53 INFO executor.Executor: Running task 166.0 in stage 29.0 (TID 593)
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/07/19 23:35:53 INFO executor.Executor: Finished task 167.0 in stage 29.0 (TID 594). 894 bytes result sent to driver
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 168.0 in stage 29.0 (TID 595, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO executor.Executor: Running task 168.0 in stage 29.0 (TID 595)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 167.0 in stage 29.0 (TID 594) in 5 ms on localhost (167/199)
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/07/19 23:35:53 INFO executor.Executor: Finished task 166.0 in stage 29.0 (TID 593). 894 bytes result sent to driver
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 169.0 in stage 29.0 (TID 596, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 166.0 in stage 29.0 (TID 593) in 10 ms on localhost (168/199)
16/07/19 23:35:53 INFO executor.Executor: Finished task 168.0 in stage 29.0 (TID 595). 894 bytes result sent to driver
16/07/19 23:35:53 INFO executor.Executor: Running task 169.0 in stage 29.0 (TID 596)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 170.0 in stage 29.0 (TID 597, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 168.0 in stage 29.0 (TID 595) in 7 ms on localhost (169/199)
16/07/19 23:35:53 INFO executor.Executor: Running task 170.0 in stage 29.0 (TID 597)
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/07/19 23:35:53 INFO executor.Executor: Finished task 169.0 in stage 29.0 (TID 596). 894 bytes result sent to driver
16/07/19 23:35:53 INFO executor.Executor: Finished task 170.0 in stage 29.0 (TID 597). 894 bytes result sent to driver
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 171.0 in stage 29.0 (TID 598, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 169.0 in stage 29.0 (TID 596) in 10 ms on localhost (170/199)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 172.0 in stage 29.0 (TID 599, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO executor.Executor: Running task 171.0 in stage 29.0 (TID 598)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 170.0 in stage 29.0 (TID 597) in 9 ms on localhost (171/199)
16/07/19 23:35:53 INFO executor.Executor: Running task 172.0 in stage 29.0 (TID 599)
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/07/19 23:35:53 INFO executor.Executor: Finished task 172.0 in stage 29.0 (TID 599). 894 bytes result sent to driver
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 173.0 in stage 29.0 (TID 600, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO executor.Executor: Running task 173.0 in stage 29.0 (TID 600)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 172.0 in stage 29.0 (TID 599) in 6 ms on localhost (172/199)
16/07/19 23:35:53 INFO executor.Executor: Finished task 171.0 in stage 29.0 (TID 598). 894 bytes result sent to driver
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 174.0 in stage 29.0 (TID 601, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 171.0 in stage 29.0 (TID 598) in 11 ms on localhost (173/199)
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/07/19 23:35:53 INFO executor.Executor: Running task 174.0 in stage 29.0 (TID 601)
16/07/19 23:35:53 INFO executor.Executor: Finished task 173.0 in stage 29.0 (TID 600). 894 bytes result sent to driver
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 175.0 in stage 29.0 (TID 602, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 173.0 in stage 29.0 (TID 600) in 7 ms on localhost (174/199)
16/07/19 23:35:53 INFO executor.Executor: Running task 175.0 in stage 29.0 (TID 602)
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
16/07/19 23:35:53 INFO executor.Executor: Finished task 174.0 in stage 29.0 (TID 601). 894 bytes result sent to driver
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 174.0 in stage 29.0 (TID 601) in 9 ms on localhost (175/199)
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 176.0 in stage 29.0 (TID 603, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO executor.Executor: Running task 176.0 in stage 29.0 (TID 603)
16/07/19 23:35:53 INFO executor.Executor: Finished task 175.0 in stage 29.0 (TID 602). 1211 bytes result sent to driver
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 175.0 in stage 29.0 (TID 602) in 8 ms on localhost (176/199)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 177.0 in stage 29.0 (TID 604, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO executor.Executor: Running task 177.0 in stage 29.0 (TID 604)
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/07/19 23:35:53 INFO executor.Executor: Finished task 176.0 in stage 29.0 (TID 603). 894 bytes result sent to driver
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 176.0 in stage 29.0 (TID 603) in 8 ms on localhost (177/199)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 178.0 in stage 29.0 (TID 605, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO executor.Executor: Running task 178.0 in stage 29.0 (TID 605)
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/07/19 23:35:53 INFO executor.Executor: Finished task 177.0 in stage 29.0 (TID 604). 894 bytes result sent to driver
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 179.0 in stage 29.0 (TID 606, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 177.0 in stage 29.0 (TID 604) in 11 ms on localhost (178/199)
16/07/19 23:35:53 INFO executor.Executor: Running task 179.0 in stage 29.0 (TID 606)
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO executor.Executor: Finished task 179.0 in stage 29.0 (TID 606). 894 bytes result sent to driver
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 180.0 in stage 29.0 (TID 607, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO executor.Executor: Running task 180.0 in stage 29.0 (TID 607)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 179.0 in stage 29.0 (TID 606) in 6 ms on localhost (179/199)
16/07/19 23:35:53 INFO executor.Executor: Finished task 178.0 in stage 29.0 (TID 605). 894 bytes result sent to driver
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 181.0 in stage 29.0 (TID 608, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 178.0 in stage 29.0 (TID 605) in 15 ms on localhost (180/199)
16/07/19 23:35:53 INFO executor.Executor: Running task 181.0 in stage 29.0 (TID 608)
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/07/19 23:35:53 INFO executor.Executor: Finished task 180.0 in stage 29.0 (TID 607). 894 bytes result sent to driver
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 180.0 in stage 29.0 (TID 607) in 8 ms on localhost (181/199)
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 182.0 in stage 29.0 (TID 609, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO executor.Executor: Finished task 181.0 in stage 29.0 (TID 608). 894 bytes result sent to driver
16/07/19 23:35:53 INFO executor.Executor: Running task 182.0 in stage 29.0 (TID 609)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 181.0 in stage 29.0 (TID 608) in 9 ms on localhost (182/199)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 183.0 in stage 29.0 (TID 610, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO executor.Executor: Running task 183.0 in stage 29.0 (TID 610)
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/07/19 23:35:53 INFO executor.Executor: Finished task 182.0 in stage 29.0 (TID 609). 894 bytes result sent to driver
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 182.0 in stage 29.0 (TID 609) in 8 ms on localhost (183/199)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 184.0 in stage 29.0 (TID 611, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO executor.Executor: Running task 184.0 in stage 29.0 (TID 611)
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
16/07/19 23:35:53 INFO executor.Executor: Finished task 183.0 in stage 29.0 (TID 610). 894 bytes result sent to driver
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 183.0 in stage 29.0 (TID 610) in 9 ms on localhost (184/199)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 185.0 in stage 29.0 (TID 612, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO executor.Executor: Running task 185.0 in stage 29.0 (TID 612)
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/07/19 23:35:53 INFO executor.Executor: Finished task 184.0 in stage 29.0 (TID 611). 894 bytes result sent to driver
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 186.0 in stage 29.0 (TID 613, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO executor.Executor: Running task 186.0 in stage 29.0 (TID 613)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 184.0 in stage 29.0 (TID 611) in 9 ms on localhost (185/199)
16/07/19 23:35:53 INFO executor.Executor: Finished task 185.0 in stage 29.0 (TID 612). 894 bytes result sent to driver
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/07/19 23:35:53 INFO executor.Executor: Finished task 186.0 in stage 29.0 (TID 613). 894 bytes result sent to driver
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 187.0 in stage 29.0 (TID 614, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 186.0 in stage 29.0 (TID 613) in 7 ms on localhost (186/199)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 185.0 in stage 29.0 (TID 612) in 11 ms on localhost (187/199)
16/07/19 23:35:53 INFO executor.Executor: Running task 187.0 in stage 29.0 (TID 614)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 188.0 in stage 29.0 (TID 615, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO executor.Executor: Running task 188.0 in stage 29.0 (TID 615)
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/07/19 23:35:53 INFO executor.Executor: Finished task 188.0 in stage 29.0 (TID 615). 894 bytes result sent to driver
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 189.0 in stage 29.0 (TID 616, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO executor.Executor: Finished task 187.0 in stage 29.0 (TID 614). 894 bytes result sent to driver
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 190.0 in stage 29.0 (TID 617, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO executor.Executor: Running task 190.0 in stage 29.0 (TID 617)
16/07/19 23:35:53 INFO executor.Executor: Running task 189.0 in stage 29.0 (TID 616)
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 188.0 in stage 29.0 (TID 615) in 9 ms on localhost (188/199)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 187.0 in stage 29.0 (TID 614) in 12 ms on localhost (189/199)
16/07/19 23:35:53 INFO executor.Executor: Finished task 190.0 in stage 29.0 (TID 617). 894 bytes result sent to driver
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 191.0 in stage 29.0 (TID 618, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 190.0 in stage 29.0 (TID 617) in 7 ms on localhost (190/199)
16/07/19 23:35:53 INFO executor.Executor: Running task 191.0 in stage 29.0 (TID 618)
16/07/19 23:35:53 INFO executor.Executor: Finished task 189.0 in stage 29.0 (TID 616). 894 bytes result sent to driver
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 192.0 in stage 29.0 (TID 619, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO executor.Executor: Running task 192.0 in stage 29.0 (TID 619)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 189.0 in stage 29.0 (TID 616) in 12 ms on localhost (191/199)
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/07/19 23:35:53 INFO executor.Executor: Finished task 191.0 in stage 29.0 (TID 618). 894 bytes result sent to driver
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 193.0 in stage 29.0 (TID 620, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO executor.Executor: Running task 193.0 in stage 29.0 (TID 620)
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 191.0 in stage 29.0 (TID 618) in 12 ms on localhost (192/199)
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/07/19 23:35:53 INFO executor.Executor: Finished task 193.0 in stage 29.0 (TID 620). 894 bytes result sent to driver
16/07/19 23:35:53 INFO executor.Executor: Finished task 192.0 in stage 29.0 (TID 619). 894 bytes result sent to driver
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 194.0 in stage 29.0 (TID 621, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 195.0 in stage 29.0 (TID 622, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 192.0 in stage 29.0 (TID 619) in 11 ms on localhost (193/199)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 193.0 in stage 29.0 (TID 620) in 7 ms on localhost (194/199)
16/07/19 23:35:53 INFO executor.Executor: Running task 195.0 in stage 29.0 (TID 622)
16/07/19 23:35:53 INFO executor.Executor: Running task 194.0 in stage 29.0 (TID 621)
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/07/19 23:35:53 INFO executor.Executor: Finished task 195.0 in stage 29.0 (TID 622). 894 bytes result sent to driver
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 196.0 in stage 29.0 (TID 623, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO executor.Executor: Running task 196.0 in stage 29.0 (TID 623)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 195.0 in stage 29.0 (TID 622) in 7 ms on localhost (195/199)
16/07/19 23:35:53 INFO executor.Executor: Finished task 194.0 in stage 29.0 (TID 621). 894 bytes result sent to driver
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 197.0 in stage 29.0 (TID 624, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO executor.Executor: Running task 197.0 in stage 29.0 (TID 624)
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 194.0 in stage 29.0 (TID 621) in 12 ms on localhost (196/199)
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
16/07/19 23:35:53 INFO executor.Executor: Finished task 196.0 in stage 29.0 (TID 623). 894 bytes result sent to driver
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Starting task 198.0 in stage 29.0 (TID 625, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 196.0 in stage 29.0 (TID 623) in 8 ms on localhost (197/199)
16/07/19 23:35:53 INFO executor.Executor: Running task 198.0 in stage 29.0 (TID 625)
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/07/19 23:35:53 INFO executor.Executor: Finished task 197.0 in stage 29.0 (TID 624). 894 bytes result sent to driver
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 197.0 in stage 29.0 (TID 624) in 9 ms on localhost (198/199)
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:35:53 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/07/19 23:35:53 INFO executor.Executor: Finished task 198.0 in stage 29.0 (TID 625). 894 bytes result sent to driver
16/07/19 23:35:53 INFO scheduler.TaskSetManager: Finished task 198.0 in stage 29.0 (TID 625) in 8 ms on localhost (199/199)
16/07/19 23:35:53 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 29.0, whose tasks have all completed, from pool
16/07/19 23:35:53 INFO scheduler.DAGScheduler: ResultStage 29 (show at <console>:36) finished in 0.914 s
16/07/19 23:35:53 INFO scheduler.DAGScheduler: Job 23 finished: show at <console>:36, took 0.962054 s
+--------+-----+
|zip_code|count|
+--------+-----+
| 75201| 1|
| 78227| 1|
| 78727| 2|
| 77028| 1|
+--------+-----+
scala>
scala>
scala>
scala>
scala> val sqlContext = new org.apache.spark.sql.SQLContext(sc);
sqlContext: org.apache.spark.sql.SQLContext = org.apache.spark.sql.SQLContext@880cb6
scala> val rddCustomers = sc.textFile("/user/test/customers.txt");
16/07/19 23:37:13 INFO storage.MemoryStore: ensureFreeSpace(222752) called with curMem=1155006, maxMem=278019440
16/07/19 23:37:13 INFO storage.MemoryStore: Block broadcast_34 stored as values in memory (estimated size 217.5 KB, free 263.8 MB)
16/07/19 23:37:13 INFO storage.MemoryStore: ensureFreeSpace(19999) called with curMem=1377758, maxMem=278019440
16/07/19 23:37:13 INFO storage.MemoryStore: Block broadcast_34_piece0 stored as bytes in memory (estimated size 19.5 KB, free 263.8 MB)
16/07/19 23:37:13 INFO storage.BlockManagerInfo: Added broadcast_34_piece0 in memory on localhost:56137 (size: 19.5 KB, free: 265.0 MB)
16/07/19 23:37:13 INFO spark.SparkContext: Created broadcast 34 from textFile at <console>:31
rddCustomers: org.apache.spark.rdd.RDD[String] = MapPartitionsRDD[59] at textFile at <console>:31
scala> val schemaString = "customer_id name city state zip_code";
schemaString: String = customer_id name city state zip_code
scala> import org.apache.spark.sql._
import org.apache.spark.sql._
scala>
scala> import org.apache.spark.sql.types._;
import org.apache.spark.sql.types._
scala> val schema = StructType(schemaString.split(" ").map(fieldName => StructField(fieldName, StringType, true)));
schema: org.apache.spark.sql.types.StructType = StructType(StructField(customer_id,StringType,true), StructField(name,StringType,true), StructField(city,StringType,true), StructField(state,StringType,true), StructField(zip_code,StringType,true))
scala> val rowRDD = rddCustomers.map(_.split(",")).map(p => Row(p(0).trim,p(1),p(2),p(3),p(4)));
rowRDD: org.apache.spark.rdd.RDD[org.apache.spark.sql.Row] = MapPartitionsRDD[61] at map at <console>:39
scala> val dfCustomers = sqlContext.createDataFrame(rowRDD, schema);
dfCustomers: org.apache.spark.sql.DataFrame = [customer_id: string, name: string, city: string, state: string, zip_code: string]
scala> dfCustomers.registerTempTable("customers");
scala> val custNames = sqlContext.sql("SELECT name FROM customers");
custNames: org.apache.spark.sql.DataFrame = [name: string]
scala> custNames.map(t => "Name: " + t(0)).collect().foreach(println);
16/07/19 23:40:33 INFO mapred.FileInputFormat: Total input paths to process : 1
16/07/19 23:40:33 INFO spark.SparkContext: Starting job: collect at <console>:42
16/07/19 23:40:33 INFO scheduler.DAGScheduler: Got job 24 (collect at <console>:42) with 2 output partitions (allowLocal=false)
16/07/19 23:40:33 INFO scheduler.DAGScheduler: Final stage: ResultStage 30(collect at <console>:42)
16/07/19 23:40:33 INFO scheduler.DAGScheduler: Parents of final stage: List()
16/07/19 23:40:33 INFO scheduler.DAGScheduler: Missing parents: List()
16/07/19 23:40:33 INFO scheduler.DAGScheduler: Submitting ResultStage 30 (MapPartitionsRDD[65] at map at <console>:42), which has no missing parents
16/07/19 23:40:33 INFO storage.MemoryStore: ensureFreeSpace(6360) called with curMem=1397757, maxMem=278019440
16/07/19 23:40:33 INFO storage.MemoryStore: Block broadcast_35 stored as values in memory (estimated size 6.2 KB, free 263.8 MB)
16/07/19 23:40:33 INFO storage.MemoryStore: ensureFreeSpace(3147) called with curMem=1404117, maxMem=278019440
16/07/19 23:40:33 INFO storage.MemoryStore: Block broadcast_35_piece0 stored as bytes in memory (estimated size 3.1 KB, free 263.8 MB)
16/07/19 23:40:33 INFO storage.BlockManagerInfo: Added broadcast_35_piece0 in memory on localhost:56137 (size: 3.1 KB, free: 265.0 MB)
16/07/19 23:40:33 INFO spark.SparkContext: Created broadcast 35 from broadcast at DAGScheduler.scala:874
16/07/19 23:40:33 INFO scheduler.DAGScheduler: Submitting 2 missing tasks from ResultStage 30 (MapPartitionsRDD[65] at map at <console>:42)
16/07/19 23:40:33 INFO scheduler.TaskSchedulerImpl: Adding task set 30.0 with 2 tasks
16/07/19 23:40:33 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 30.0 (TID 626, localhost, ANY, 1413 bytes)
16/07/19 23:40:33 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 30.0 (TID 627, localhost, ANY, 1413 bytes)
16/07/19 23:40:33 INFO executor.Executor: Running task 0.0 in stage 30.0 (TID 626)
16/07/19 23:40:33 INFO executor.Executor: Running task 1.0 in stage 30.0 (TID 627)
16/07/19 23:40:33 INFO rdd.HadoopRDD: Input split: hdfs://cdh1:9000/user/test/customers.txt:0+92
16/07/19 23:40:33 INFO rdd.HadoopRDD: Input split: hdfs://cdh1:9000/user/test/customers.txt:92+93
16/07/19 23:40:33 INFO executor.Executor: Finished task 0.0 in stage 30.0 (TID 626). 1852 bytes result sent to driver
16/07/19 23:40:33 INFO executor.Executor: Finished task 1.0 in stage 30.0 (TID 627). 1836 bytes result sent to driver
16/07/19 23:40:33 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 30.0 (TID 627) in 69 ms on localhost (1/2)
16/07/19 23:40:33 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 30.0 (TID 626) in 70 ms on localhost (2/2)
16/07/19 23:40:33 INFO scheduler.DAGScheduler: ResultStage 30 (collect at <console>:42) finished in 0.069 s
16/07/19 23:40:33 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 30.0, whose tasks have all completed, from pool
16/07/19 23:40:33 INFO scheduler.DAGScheduler: Job 24 finished: collect at <console>:42, took 0.084842 s
Name: John Smith
Name: Joe Johnson
Name: Bob Jones
Name: Andy Davis
Name: James Williams
scala> val customersByCity = sqlContext.sql("SELECT name,zip_code FROM customers ORDER BY zip_code");
customersByCity: org.apache.spark.sql.DataFrame = [name: string, zip_code: string]
scala> customersByCity.map(t => t(0) + "," + t(1)).collect().foreach(println);
16/07/19 23:41:09 INFO storage.BlockManagerInfo: Removed broadcast_35_piece0 on localhost:56137 in memory (size: 3.1 KB, free: 265.0 MB)
16/07/19 23:41:09 INFO execution.Exchange: Using SparkSqlSerializer2.
16/07/19 23:41:09 INFO spark.SparkContext: Starting job: map at <console>:42
16/07/19 23:41:09 INFO scheduler.DAGScheduler: Got job 25 (map at <console>:42) with 2 output partitions (allowLocal=false)
16/07/19 23:41:09 INFO scheduler.DAGScheduler: Final stage: ResultStage 31(map at <console>:42)
16/07/19 23:41:09 INFO scheduler.DAGScheduler: Parents of final stage: List()
16/07/19 23:41:09 INFO scheduler.DAGScheduler: Missing parents: List()
16/07/19 23:41:09 INFO scheduler.DAGScheduler: Submitting ResultStage 31 (MapPartitionsRDD[69] at map at <console>:42), which has no missing parents
16/07/19 23:41:09 INFO storage.MemoryStore: ensureFreeSpace(6744) called with curMem=1397757, maxMem=278019440
16/07/19 23:41:09 INFO storage.MemoryStore: Block broadcast_36 stored as values in memory (estimated size 6.6 KB, free 263.8 MB)
16/07/19 23:41:09 INFO storage.MemoryStore: ensureFreeSpace(3351) called with curMem=1404501, maxMem=278019440
16/07/19 23:41:09 INFO storage.MemoryStore: Block broadcast_36_piece0 stored as bytes in memory (estimated size 3.3 KB, free 263.8 MB)
16/07/19 23:41:09 INFO storage.BlockManagerInfo: Added broadcast_36_piece0 in memory on localhost:56137 (size: 3.3 KB, free: 265.0 MB)
16/07/19 23:41:09 INFO spark.SparkContext: Created broadcast 36 from broadcast at DAGScheduler.scala:874
16/07/19 23:41:09 INFO scheduler.DAGScheduler: Submitting 2 missing tasks from ResultStage 31 (MapPartitionsRDD[69] at map at <console>:42)
16/07/19 23:41:09 INFO scheduler.TaskSchedulerImpl: Adding task set 31.0 with 2 tasks
16/07/19 23:41:09 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 31.0 (TID 628, localhost, ANY, 1413 bytes)
16/07/19 23:41:09 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 31.0 (TID 629, localhost, ANY, 1413 bytes)
16/07/19 23:41:09 INFO executor.Executor: Running task 0.0 in stage 31.0 (TID 628)
16/07/19 23:41:09 INFO executor.Executor: Running task 1.0 in stage 31.0 (TID 629)
16/07/19 23:41:09 INFO rdd.HadoopRDD: Input split: hdfs://cdh1:9000/user/test/customers.txt:92+93
16/07/19 23:41:09 INFO rdd.HadoopRDD: Input split: hdfs://cdh1:9000/user/test/customers.txt:0+92
16/07/19 23:41:09 INFO executor.Executor: Finished task 0.0 in stage 31.0 (TID 628). 2394 bytes result sent to driver
16/07/19 23:41:09 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 31.0 (TID 628) in 117 ms on localhost (1/2)
16/07/19 23:41:09 INFO executor.Executor: Finished task 1.0 in stage 31.0 (TID 629). 2333 bytes result sent to driver
16/07/19 23:41:09 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 31.0 (TID 629) in 120 ms on localhost (2/2)
16/07/19 23:41:09 INFO scheduler.DAGScheduler: ResultStage 31 (map at <console>:42) finished in 0.113 s
16/07/19 23:41:09 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 31.0, whose tasks have all completed, from pool
16/07/19 23:41:09 INFO scheduler.DAGScheduler: Job 25 finished: map at <console>:42, took 0.135991 s
16/07/19 23:41:09 INFO spark.SparkContext: Starting job: collect at <console>:42
16/07/19 23:41:09 INFO scheduler.DAGScheduler: Registering RDD 70 (map at <console>:42)
16/07/19 23:41:09 INFO scheduler.DAGScheduler: Got job 26 (collect at <console>:42) with 5 output partitions (allowLocal=false)
16/07/19 23:41:09 INFO scheduler.DAGScheduler: Final stage: ResultStage 33(collect at <console>:42)
16/07/19 23:41:09 INFO scheduler.DAGScheduler: Parents of final stage: List(ShuffleMapStage 32)
16/07/19 23:41:09 INFO scheduler.DAGScheduler: Missing parents: List(ShuffleMapStage 32)
16/07/19 23:41:09 INFO scheduler.DAGScheduler: Submitting ShuffleMapStage 32 (MapPartitionsRDD[70] at map at <console>:42), which has no missing parents
16/07/19 23:41:09 INFO storage.MemoryStore: ensureFreeSpace(8176) called with curMem=1407852, maxMem=278019440
16/07/19 23:41:09 INFO storage.MemoryStore: Block broadcast_37 stored as values in memory (estimated size 8.0 KB, free 263.8 MB)
16/07/19 23:41:09 INFO storage.MemoryStore: ensureFreeSpace(4107) called with curMem=1416028, maxMem=278019440
16/07/19 23:41:09 INFO storage.MemoryStore: Block broadcast_37_piece0 stored as bytes in memory (estimated size 4.0 KB, free 263.8 MB)
16/07/19 23:41:09 INFO storage.BlockManagerInfo: Added broadcast_37_piece0 in memory on localhost:56137 (size: 4.0 KB, free: 265.0 MB)
16/07/19 23:41:09 INFO spark.SparkContext: Created broadcast 37 from broadcast at DAGScheduler.scala:874
16/07/19 23:41:09 INFO scheduler.DAGScheduler: Submitting 2 missing tasks from ShuffleMapStage 32 (MapPartitionsRDD[70] at map at <console>:42)
16/07/19 23:41:09 INFO scheduler.TaskSchedulerImpl: Adding task set 32.0 with 2 tasks
16/07/19 23:41:09 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 32.0 (TID 630, localhost, ANY, 1402 bytes)
16/07/19 23:41:09 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 32.0 (TID 631, localhost, ANY, 1402 bytes)
16/07/19 23:41:09 INFO executor.Executor: Running task 1.0 in stage 32.0 (TID 631)
16/07/19 23:41:09 INFO executor.Executor: Running task 0.0 in stage 32.0 (TID 630)
16/07/19 23:41:09 INFO rdd.HadoopRDD: Input split: hdfs://cdh1:9000/user/test/customers.txt:0+92
16/07/19 23:41:09 INFO rdd.HadoopRDD: Input split: hdfs://cdh1:9000/user/test/customers.txt:92+93
16/07/19 23:41:09 INFO executor.Executor: Finished task 1.0 in stage 32.0 (TID 631). 2005 bytes result sent to driver
16/07/19 23:41:09 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 32.0 (TID 631) in 74 ms on localhost (1/2)
16/07/19 23:41:09 INFO executor.Executor: Finished task 0.0 in stage 32.0 (TID 630). 2005 bytes result sent to driver
16/07/19 23:41:09 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 32.0 (TID 630) in 84 ms on localhost (2/2)
16/07/19 23:41:09 INFO scheduler.DAGScheduler: ShuffleMapStage 32 (map at <console>:42) finished in 0.080 s
16/07/19 23:41:09 INFO scheduler.DAGScheduler: looking for newly runnable stages
16/07/19 23:41:09 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 32.0, whose tasks have all completed, from pool
16/07/19 23:41:09 INFO scheduler.DAGScheduler: running: Set()
16/07/19 23:41:09 INFO scheduler.DAGScheduler: waiting: Set(ResultStage 33)
16/07/19 23:41:09 INFO scheduler.DAGScheduler: failed: Set()
16/07/19 23:41:09 INFO scheduler.DAGScheduler: Missing parents for ResultStage 33: List()
16/07/19 23:41:09 INFO scheduler.DAGScheduler: Submitting ResultStage 33 (MapPartitionsRDD[75] at map at <console>:42), which is now runnable
16/07/19 23:41:09 INFO storage.MemoryStore: ensureFreeSpace(9776) called with curMem=1420135, maxMem=278019440
16/07/19 23:41:09 INFO storage.MemoryStore: Block broadcast_38 stored as values in memory (estimated size 9.5 KB, free 263.8 MB)
16/07/19 23:41:09 INFO storage.MemoryStore: ensureFreeSpace(4674) called with curMem=1429911, maxMem=278019440
16/07/19 23:41:09 INFO storage.MemoryStore: Block broadcast_38_piece0 stored as bytes in memory (estimated size 4.6 KB, free 263.8 MB)
16/07/19 23:41:09 INFO storage.BlockManagerInfo: Added broadcast_38_piece0 in memory on localhost:56137 (size: 4.6 KB, free: 265.0 MB)
16/07/19 23:41:09 INFO spark.SparkContext: Created broadcast 38 from broadcast at DAGScheduler.scala:874
16/07/19 23:41:09 INFO scheduler.DAGScheduler: Submitting 5 missing tasks from ResultStage 33 (MapPartitionsRDD[75] at map at <console>:42)
16/07/19 23:41:09 INFO scheduler.TaskSchedulerImpl: Adding task set 33.0 with 5 tasks
16/07/19 23:41:09 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 33.0 (TID 632, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:41:09 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 33.0 (TID 633, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:41:09 INFO executor.Executor: Running task 1.0 in stage 33.0 (TID 633)
16/07/19 23:41:09 INFO executor.Executor: Running task 0.0 in stage 33.0 (TID 632)
16/07/19 23:41:09 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:41:09 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:41:09 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
16/07/19 23:41:09 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
16/07/19 23:41:09 INFO executor.Executor: Finished task 0.0 in stage 33.0 (TID 632). 908 bytes result sent to driver
16/07/19 23:41:09 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 33.0 (TID 634, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:41:09 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 33.0 (TID 632) in 9 ms on localhost (1/5)
16/07/19 23:41:09 INFO executor.Executor: Running task 2.0 in stage 33.0 (TID 634)
16/07/19 23:41:09 INFO executor.Executor: Finished task 1.0 in stage 33.0 (TID 633). 906 bytes result sent to driver
16/07/19 23:41:09 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 33.0 (TID 635, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:41:09 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 33.0 (TID 633) in 12 ms on localhost (2/5)
16/07/19 23:41:09 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:41:09 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
16/07/19 23:41:09 INFO executor.Executor: Finished task 2.0 in stage 33.0 (TID 634). 907 bytes result sent to driver
16/07/19 23:41:09 INFO scheduler.TaskSetManager: Starting task 4.0 in stage 33.0 (TID 636, localhost, PROCESS_LOCAL, 1165 bytes)
16/07/19 23:41:09 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 33.0 (TID 634) in 9 ms on localhost (3/5)
16/07/19 23:41:09 INFO executor.Executor: Running task 3.0 in stage 33.0 (TID 635)
16/07/19 23:41:09 INFO executor.Executor: Running task 4.0 in stage 33.0 (TID 636)
16/07/19 23:41:09 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:41:09 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/07/19 23:41:09 INFO executor.Executor: Finished task 3.0 in stage 33.0 (TID 635). 932 bytes result sent to driver
16/07/19 23:41:09 INFO storage.ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
16/07/19 23:41:09 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/07/19 23:41:09 INFO executor.Executor: Finished task 4.0 in stage 33.0 (TID 636). 886 bytes result sent to driver
16/07/19 23:41:09 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 33.0 (TID 635) in 20 ms on localhost (4/5)
16/07/19 23:41:09 INFO scheduler.TaskSetManager: Finished task 4.0 in stage 33.0 (TID 636) in 13 ms on localhost (5/5)
16/07/19 23:41:09 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 33.0, whose tasks have all completed, from pool
16/07/19 23:41:09 INFO scheduler.DAGScheduler: ResultStage 33 (collect at <console>:42) finished in 0.028 s
16/07/19 23:41:09 INFO scheduler.DAGScheduler: Job 26 finished: collect at <console>:42, took 0.149747 s
Joe Johnson, 75201
Bob Jones, 77028
Andy Davis, 78227
John Smith, 78727
James Williams, 78727
scala>
scala>
scala>
scala> val sqlContext = new org.apache.spark.sql.SQLContext(sc);
sqlContext: org.apache.spark.sql.SQLContext = org.apache.spark.sql.SQLContext@8d0323
scala> import sqlContext.implicits._
import sqlContext.implicits._
scala> case class Customer(customer_id: Int, name: String, city: String, state: String, zip_code: String);
16/07/19 23:50:39 INFO storage.BlockManagerInfo: Removed broadcast_38_piece0 on localhost:56137 in memory (size: 4.6 KB, free: 265.0 MB)
16/07/19 23:50:39 INFO storage.BlockManagerInfo: Removed broadcast_37_piece0 on localhost:56137 in memory (size: 4.0 KB, free: 265.0 MB)
16/07/19 23:50:39 INFO storage.BlockManagerInfo: Removed broadcast_36_piece0 on localhost:56137 in memory (size: 3.3 KB, free: 265.0 MB)
defined class Customer
scala> val dfCustomers = sc.textFile("data/customers.txt").map(_.split(",")).map(p => Customer(p(0).trim.toInt, p(1), p(2), p(3), p(4))).toDF();
16/07/19 23:51:17 INFO storage.MemoryStore: ensureFreeSpace(222752) called with curMem=1397757, maxMem=278019440
16/07/19 23:51:17 INFO storage.MemoryStore: Block broadcast_39 stored as values in memory (estimated size 217.5 KB, free 263.6 MB)
16/07/19 23:51:17 INFO storage.MemoryStore: ensureFreeSpace(19999) called with curMem=1620509, maxMem=278019440
16/07/19 23:51:17 INFO storage.MemoryStore: Block broadcast_39_piece0 stored as bytes in memory (estimated size 19.5 KB, free 263.6 MB)
16/07/19 23:51:17 INFO storage.BlockManagerInfo: Added broadcast_39_piece0 in memory on localhost:56137 (size: 19.5 KB, free: 265.0 MB)
16/07/19 23:51:17 INFO spark.SparkContext: Created broadcast 39 from textFile at <console>:44
dfCustomers: org.apache.spark.sql.DataFrame = [customer_id: int, name: string, city: string, state: string, zip_code: string]
scala> dfCustomers.registerTempTable("customers");
scala> dfCustomers.show();
16/07/19 23:51:55 INFO mapred.FileInputFormat: Total input paths to process : 1
16/07/19 23:51:55 INFO spark.SparkContext: Starting job: show at <console>:47
16/07/19 23:51:55 INFO scheduler.DAGScheduler: Got job 27 (show at <console>:47) with 1 output partitions (allowLocal=false)
16/07/19 23:51:55 INFO scheduler.DAGScheduler: Final stage: ResultStage 34(show at <console>:47)
16/07/19 23:51:55 INFO scheduler.DAGScheduler: Parents of final stage: List()
16/07/19 23:51:55 INFO scheduler.DAGScheduler: Missing parents: List()
16/07/19 23:51:55 INFO scheduler.DAGScheduler: Submitting ResultStage 34 (MapPartitionsRDD[81] at show at <console>:47), which has no missing parents
16/07/19 23:51:55 INFO storage.MemoryStore: ensureFreeSpace(4144) called with curMem=1640508, maxMem=278019440
16/07/19 23:51:55 INFO storage.MemoryStore: Block broadcast_40 stored as values in memory (estimated size 4.0 KB, free 263.6 MB)
16/07/19 23:51:55 INFO storage.MemoryStore: ensureFreeSpace(2210) called with curMem=1644652, maxMem=278019440
16/07/19 23:51:55 INFO storage.MemoryStore: Block broadcast_40_piece0 stored as bytes in memory (estimated size 2.2 KB, free 263.6 MB)
16/07/19 23:51:55 INFO storage.BlockManagerInfo: Added broadcast_40_piece0 in memory on localhost:56137 (size: 2.2 KB, free: 265.0 MB)
16/07/19 23:51:55 INFO spark.SparkContext: Created broadcast 40 from broadcast at DAGScheduler.scala:874
16/07/19 23:51:55 INFO scheduler.DAGScheduler: Submitting 1 missing tasks from ResultStage 34 (MapPartitionsRDD[81] at show at <console>:47)
16/07/19 23:51:55 INFO scheduler.TaskSchedulerImpl: Adding task set 34.0 with 1 tasks
16/07/19 23:51:55 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 34.0 (TID 637, localhost, ANY, 1418 bytes)
16/07/19 23:51:55 INFO executor.Executor: Running task 0.0 in stage 34.0 (TID 637)
16/07/19 23:51:55 INFO rdd.HadoopRDD: Input split: hdfs://cdh1:9000/user/root/data/customers.txt:0+92
16/07/19 23:51:55 INFO executor.Executor: Finished task 0.0 in stage 34.0 (TID 637). 2420 bytes result sent to driver
16/07/19 23:51:55 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 34.0 (TID 637) in 23 ms on localhost (1/1)
16/07/19 23:51:55 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 34.0, whose tasks have all completed, from pool
16/07/19 23:51:55 INFO scheduler.DAGScheduler: ResultStage 34 (show at <console>:47) finished in 0.022 s
16/07/19 23:51:55 INFO scheduler.DAGScheduler: Job 27 finished: show at <console>:47, took 0.040762 s
16/07/19 23:51:55 INFO spark.SparkContext: Starting job: show at <console>:47
16/07/19 23:51:55 INFO scheduler.DAGScheduler: Got job 28 (show at <console>:47) with 1 output partitions (allowLocal=false)
16/07/19 23:51:55 INFO scheduler.DAGScheduler: Final stage: ResultStage 35(show at <console>:47)
16/07/19 23:51:55 INFO scheduler.DAGScheduler: Parents of final stage: List()
16/07/19 23:51:55 INFO scheduler.DAGScheduler: Missing parents: List()
16/07/19 23:51:55 INFO scheduler.DAGScheduler: Submitting ResultStage 35 (MapPartitionsRDD[81] at show at <console>:47), which has no missing parents
16/07/19 23:51:55 INFO storage.MemoryStore: ensureFreeSpace(4144) called with curMem=1646862, maxMem=278019440
16/07/19 23:51:55 INFO storage.MemoryStore: Block broadcast_41 stored as values in memory (estimated size 4.0 KB, free 263.6 MB)
16/07/19 23:51:55 INFO storage.MemoryStore: ensureFreeSpace(2210) called with curMem=1651006, maxMem=278019440
16/07/19 23:51:55 INFO storage.MemoryStore: Block broadcast_41_piece0 stored as bytes in memory (estimated size 2.2 KB, free 263.6 MB)
16/07/19 23:51:55 INFO storage.BlockManagerInfo: Added broadcast_41_piece0 in memory on localhost:56137 (size: 2.2 KB, free: 265.0 MB)
16/07/19 23:51:55 INFO spark.SparkContext: Created broadcast 41 from broadcast at DAGScheduler.scala:874
16/07/19 23:51:55 INFO scheduler.DAGScheduler: Submitting 1 missing tasks from ResultStage 35 (MapPartitionsRDD[81] at show at <console>:47)
16/07/19 23:51:55 INFO scheduler.TaskSchedulerImpl: Adding task set 35.0 with 1 tasks
16/07/19 23:51:55 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 35.0 (TID 638, localhost, ANY, 1418 bytes)
16/07/19 23:51:55 INFO executor.Executor: Running task 0.0 in stage 35.0 (TID 638)
16/07/19 23:51:55 INFO rdd.HadoopRDD: Input split: hdfs://cdh1:9000/user/root/data/customers.txt:92+93
16/07/19 23:51:55 INFO executor.Executor: Finished task 0.0 in stage 35.0 (TID 638). 2311 bytes result sent to driver
16/07/19 23:51:55 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 35.0 (TID 638) in 12 ms on localhost (1/1)
16/07/19 23:51:55 INFO scheduler.DAGScheduler: ResultStage 35 (show at <console>:47) finished in 0.010 s
16/07/19 23:51:55 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 35.0, whose tasks have all completed, from pool
16/07/19 23:51:55 INFO scheduler.DAGScheduler: Job 28 finished: show at <console>:47, took 0.024055 s
+-----------+---------------+------------+-----+--------+
|customer_id| name| city|state|zip_code|
+-----------+---------------+------------+-----+--------+
| 100| John Smith| Austin| TX| 78727|
| 200| Joe Johnson| Dallas| TX| 75201|
| 300| Bob Jones| Houston| TX| 77028|
| 400| Andy Davis| San Antonio| TX| 78227|
| 500| James Williams| Austin| TX| 78727|
+-----------+---------------+------------+-----+--------+
scala> dfCustomers.show();registerTempTable("customers");val dfCustomers = sc.textFile("data/customers.txt").map(_.split(",")).map(p => Customer(p(0).trim.toInt, p(1), p(2), p(3), p(4))).toDF(); = sc.textFile("data/customers.txt").map(_.split(",")).map(p => Customer(p(0).trim.toInt, p(1), p(2), p(3), p(4))).toDF(); = sc.textFile("data/customers.txt").map(_.split(",")).map(p => Customer(p(0).trim.toInt, p(1), p(2), p(3), p(4))).toDF(); = sc.textFile("data/customers.txt").map(_.split(",")).map(p => Customer(p(0).trim.toInt, p(1), p(2), p(3), p(4))).toDF(); = sc.textFile("data/customers.txt").map(_.split(",")).map(p => Customer(p(0).trim.toInt, p(1), p(2), p(3), p(4))).toDF(); = sc.textFile("data/customers.txt").map(_.split(",")).map(p => Customer(p(0).trim.toInt, p(1), p(2), p(3), p(4))).toDF();
16/07/19 23:52:33 INFO storage.MemoryStore: ensureFreeSpace(222752) called with curMem=1653216, maxMem=278019440
16/07/19 23:52:33 INFO storage.MemoryStore: Block broadcast_42 stored as values in memory (estimated size 217.5 KB, free 263.4 MB)
16/07/19 23:52:33 INFO storage.MemoryStore: ensureFreeSpace(19999) called with curMem=1875968, maxMem=278019440
16/07/19 23:52:33 INFO storage.MemoryStore: Block broadcast_42_piece0 stored as bytes in memory (estimated size 19.5 KB, free 263.3 MB)
16/07/19 23:52:33 INFO storage.BlockManagerInfo: Added broadcast_42_piece0 in memory on localhost:56137 (size: 19.5 KB, free: 265.0 MB)
16/07/19 23:52:33 INFO spark.SparkContext: Created broadcast 42 from textFile at <console>:44
dfCust: org.apache.spark.sql.DataFrame = [customer_id: int, name: string, city: string, state: string, zip_code: string]
scala> val dfCust = sc.textFile("data/customers.txt").map(_.split(",")).map(p => Customer(p(0).trim.toInt, p(1), p(2), p(3), p(4))).toDF();dfCustomers.show();registerTempTable("customers");val dfCustomers = sc.textFile("data/customers.txt").map(_.split(",")).map(p => Customer(p(0).trim.toInt, p(1), p(2), p(3), p(4))).toDF();dfCustomers.registerTempTable("customers");.registerTempTable("customers");.registerTempTable("customers");.registerTempTable("customers");.registerTempTable("customers");.registerTempTable("customers");s");s");s");s");
scala> dfCust.registerTempTable("custs");val dfCust = sc.textFile("data/customers.txt").map(_.split(",")).map(p => Customer(p(0).trim.toInt, p(1), p(2), p(3), p(4))).toDF();dfCustomers.show();.show();.show();.show();.show();.show();
16/07/19 23:53:23 INFO storage.BlockManagerInfo: Removed broadcast_41_piece0 on localhost:56137 in memory (size: 2.2 KB, free: 265.0 MB)
16/07/19 23:53:23 INFO storage.BlockManagerInfo: Removed broadcast_40_piece0 on localhost:56137 in memory (size: 2.2 KB, free: 265.0 MB)
16/07/19 23:53:23 INFO mapred.FileInputFormat: Total input paths to process : 1
16/07/19 23:53:23 INFO spark.SparkContext: Starting job: show at <console>:47
16/07/19 23:53:23 INFO scheduler.DAGScheduler: Got job 29 (show at <console>:47) with 1 output partitions (allowLocal=false)
16/07/19 23:53:23 INFO scheduler.DAGScheduler: Final stage: ResultStage 36(show at <console>:47)
16/07/19 23:53:23 INFO scheduler.DAGScheduler: Parents of final stage: List()
16/07/19 23:53:23 INFO scheduler.DAGScheduler: Missing parents: List()
16/07/19 23:53:23 INFO scheduler.DAGScheduler: Submitting ResultStage 36 (MapPartitionsRDD[87] at show at <console>:47), which has no missing parents
16/07/19 23:53:23 INFO storage.MemoryStore: ensureFreeSpace(4144) called with curMem=1883259, maxMem=278019440
16/07/19 23:53:23 INFO storage.MemoryStore: Block broadcast_43 stored as values in memory (estimated size 4.0 KB, free 263.3 MB)
16/07/19 23:53:23 INFO storage.MemoryStore: ensureFreeSpace(2211) called with curMem=1887403, maxMem=278019440
16/07/19 23:53:23 INFO storage.MemoryStore: Block broadcast_43_piece0 stored as bytes in memory (estimated size 2.2 KB, free 263.3 MB)
16/07/19 23:53:23 INFO storage.BlockManagerInfo: Added broadcast_43_piece0 in memory on localhost:56137 (size: 2.2 KB, free: 265.0 MB)
16/07/19 23:53:23 INFO spark.SparkContext: Created broadcast 43 from broadcast at DAGScheduler.scala:874
16/07/19 23:53:23 INFO scheduler.DAGScheduler: Submitting 1 missing tasks from ResultStage 36 (MapPartitionsRDD[87] at show at <console>:47)
16/07/19 23:53:23 INFO scheduler.TaskSchedulerImpl: Adding task set 36.0 with 1 tasks
16/07/19 23:53:23 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 36.0 (TID 639, localhost, ANY, 1418 bytes)
16/07/19 23:53:23 INFO executor.Executor: Running task 0.0 in stage 36.0 (TID 639)
16/07/19 23:53:23 INFO rdd.HadoopRDD: Input split: hdfs://cdh1:9000/user/root/data/customers.txt:0+92
16/07/19 23:53:23 INFO executor.Executor: Finished task 0.0 in stage 36.0 (TID 639). 2420 bytes result sent to driver
16/07/19 23:53:23 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 36.0 (TID 639) in 16 ms on localhost (1/1)
16/07/19 23:53:23 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 36.0, whose tasks have all completed, from pool
16/07/19 23:53:23 INFO scheduler.DAGScheduler: ResultStage 36 (show at <console>:47) finished in 0.016 s
16/07/19 23:53:23 INFO scheduler.DAGScheduler: Job 29 finished: show at <console>:47, took 0.032561 s
16/07/19 23:53:23 INFO spark.SparkContext: Starting job: show at <console>:47
16/07/19 23:53:23 INFO scheduler.DAGScheduler: Got job 30 (show at <console>:47) with 1 output partitions (allowLocal=false)
16/07/19 23:53:23 INFO scheduler.DAGScheduler: Final stage: ResultStage 37(show at <console>:47)
16/07/19 23:53:23 INFO scheduler.DAGScheduler: Parents of final stage: List()
16/07/19 23:53:23 INFO scheduler.DAGScheduler: Missing parents: List()
16/07/19 23:53:23 INFO scheduler.DAGScheduler: Submitting ResultStage 37 (MapPartitionsRDD[87] at show at <console>:47), which has no missing parents
16/07/19 23:53:23 INFO storage.MemoryStore: ensureFreeSpace(4144) called with curMem=1889614, maxMem=278019440
16/07/19 23:53:23 INFO storage.MemoryStore: Block broadcast_44 stored as values in memory (estimated size 4.0 KB, free 263.3 MB)
16/07/19 23:53:23 INFO storage.MemoryStore: ensureFreeSpace(2211) called with curMem=1893758, maxMem=278019440
16/07/19 23:53:23 INFO storage.MemoryStore: Block broadcast_44_piece0 stored as bytes in memory (estimated size 2.2 KB, free 263.3 MB)
16/07/19 23:53:23 INFO storage.BlockManagerInfo: Added broadcast_44_piece0 in memory on localhost:56137 (size: 2.2 KB, free: 265.0 MB)
16/07/19 23:53:23 INFO spark.SparkContext: Created broadcast 44 from broadcast at DAGScheduler.scala:874
16/07/19 23:53:23 INFO scheduler.DAGScheduler: Submitting 1 missing tasks from ResultStage 37 (MapPartitionsRDD[87] at show at <console>:47)
16/07/19 23:53:23 INFO scheduler.TaskSchedulerImpl: Adding task set 37.0 with 1 tasks
16/07/19 23:53:23 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 37.0 (TID 640, localhost, ANY, 1418 bytes)
16/07/19 23:53:23 INFO executor.Executor: Running task 0.0 in stage 37.0 (TID 640)
16/07/19 23:53:23 INFO rdd.HadoopRDD: Input split: hdfs://cdh1:9000/user/root/data/customers.txt:92+93
16/07/19 23:53:23 INFO executor.Executor: Finished task 0.0 in stage 37.0 (TID 640). 2311 bytes result sent to driver
16/07/19 23:53:23 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 37.0 (TID 640) in 13 ms on localhost (1/1)
16/07/19 23:53:23 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 37.0, whose tasks have all completed, from pool
16/07/19 23:53:23 INFO scheduler.DAGScheduler: ResultStage 37 (show at <console>:47) finished in 0.015 s
16/07/19 23:53:23 INFO scheduler.DAGScheduler: Job 30 finished: show at <console>:47, took 0.025694 s
+-----------+---------------+------------+-----+--------+
|customer_id| name| city|state|zip_code|
+-----------+---------------+------------+-----+--------+
| 100| John Smith| Austin| TX| 78727|
| 200| Joe Johnson| Dallas| TX| 75201|
| 300| Bob Jones| Houston| TX| 77028|
| 400| Andy Davis| San Antonio| TX| 78227|
| 500| James Williams| Austin| TX| 78727|
+-----------+---------------+------------+-----+--------+
scala> dfCust.show();registerTempTable("custs");val dfCust = sc.textFile("data/customers.txt").map(_.split(",")).map(p => Customer(p(0).trim.toInt, p(1), p(2), p(3), p(4))).toDF();
16/07/19 23:54:43 INFO storage.MemoryStore: ensureFreeSpace(222752) called with curMem=1895969, maxMem=278019440
16/07/19 23:54:43 INFO storage.MemoryStore: Block broadcast_45 stored as values in memory (estimated size 217.5 KB, free 263.1 MB)
16/07/19 23:54:43 INFO storage.MemoryStore: ensureFreeSpace(19999) called with curMem=2118721, maxMem=278019440
16/07/19 23:54:43 INFO storage.MemoryStore: Block broadcast_45_piece0 stored as bytes in memory (estimated size 19.5 KB, free 263.1 MB)
16/07/19 23:54:43 INFO storage.BlockManagerInfo: Added broadcast_45_piece0 in memory on localhost:56137 (size: 19.5 KB, free: 264.9 MB)
16/07/19 23:54:43 INFO spark.SparkContext: Created broadcast 45 from textFile at <console>:44
dfCust: org.apache.spark.sql.DataFrame = [customer_id: int, name: string, city: string, state: string, zip_code: string]
scala> val dfCust = sc.textFile("data/customers.txt").map(_.split(",")).map(p => Customer(p(0).trim.toInt, p(1), p(2), p(3), p(4))).toDF();dfCust.show();registerTempTable("custs");
scala> dfCust.registerTempTable("custs");val dfCust = sc.textFile("data/customers.txt").map(_.split(",")).map(p => Customer(p(0).trim.toInt, p(1), p(2), p(3), p(4))).toDF();dfCust.show();
org.apache.hadoop.mapred.InvalidInputException: Input path does not exist: hdfs://cdh1:9000/user/root/data/customers.txt
at org.apache.hadoop.mapred.FileInputFormat.singleThreadedListStatus(FileInputFormat.java:285)
at org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:228)
at org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:313)
at org.apache.spark.rdd.HadoopRDD.getPartitions(HadoopRDD.scala:207)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:219)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:217)
at scala.Option.getOrElse(Option.scala:120)
at org.apache.spark.rdd.RDD.partitions(RDD.scala:217)
at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:32)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:219)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:217)
at scala.Option.getOrElse(Option.scala:120)
at org.apache.spark.rdd.RDD.partitions(RDD.scala:217)
at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:32)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:219)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:217)
at scala.Option.getOrElse(Option.scala:120)
at org.apache.spark.rdd.RDD.partitions(RDD.scala:217)
at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:32)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:219)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:217)
at scala.Option.getOrElse(Option.scala:120)
at org.apache.spark.rdd.RDD.partitions(RDD.scala:217)
at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:32)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:219)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:217)
at scala.Option.getOrElse(Option.scala:120)
at org.apache.spark.rdd.RDD.partitions(RDD.scala:217)
at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:32)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:219)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:217)
at scala.Option.getOrElse(Option.scala:120)
at org.apache.spark.rdd.RDD.partitions(RDD.scala:217)
at org.apache.spark.sql.execution.SparkPlan.executeTake(SparkPlan.scala:121)
at org.apache.spark.sql.execution.Limit.executeCollect(basicOperators.scala:125)
at org.apache.spark.sql.DataFrame.collect(DataFrame.scala:1255)
at org.apache.spark.sql.DataFrame.head(DataFrame.scala:1189)
at org.apache.spark.sql.DataFrame.take(DataFrame.scala:1248)
at org.apache.spark.sql.DataFrame.showString(DataFrame.scala:176)
at org.apache.spark.sql.DataFrame.show(DataFrame.scala:331)
at org.apache.spark.sql.DataFrame.show(DataFrame.scala:338)
at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:47)
at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:52)
at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:54)
at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:56)
at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:58)
at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:60)
at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:62)
at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:64)
at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:66)
at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:68)
at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:70)
at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:72)
at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:74)
at $iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:76)
at $iwC$$iwC$$iwC$$iwC.<init>(<console>:78)
at $iwC$$iwC$$iwC.<init>(<console>:80)
at $iwC$$iwC.<init>(<console>:82)
at $iwC.<init>(<console>:84)
at <init>(<console>:86)
at .<init>(<console>:90)
at .<clinit>(<console>)
at .<init>(<console>:7)
at .<clinit>(<console>)
at $print(<console>)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:1065)
at org.apache.spark.repl.SparkIMain$Request.loadAndRun(SparkIMain.scala:1338)
at org.apache.spark.repl.SparkIMain.loadAndRunReq$1(SparkIMain.scala:840)
at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:871)
at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:819)
at org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:857)
at org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:902)
at org.apache.spark.repl.SparkILoop.command(SparkILoop.scala:814)
at org.apache.spark.repl.SparkILoop.processLine$1(SparkILoop.scala:657)
at org.apache.spark.repl.SparkILoop.innerLoop$1(SparkILoop.scala:665)
at org.apache.spark.repl.SparkILoop.org$apache$spark$repl$SparkILoop$$loop(SparkILoop.scala:670)
at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply$mcZ$sp(SparkILoop.scala:997)
at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:945)
at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:945)
at scala.tools.nsc.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:135)
at org.apache.spark.repl.SparkILoop.org$apache$spark$repl$SparkILoop$$process(SparkILoop.scala:945)
at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:1059)
at org.apache.spark.repl.Main$.main(Main.scala:31)
at org.apache.spark.repl.Main.main(Main.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:664)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:169)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:192)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:111)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
scala> Stopping spark context.
16/07/19 23:55:19 INFO storage.BlockManagerInfo: Removed broadcast_44_piece0 on localhost:56137 in memory (size: 2.2 KB, free: 264.9 MB)
16/07/19 23:55:19 INFO storage.BlockManagerInfo: Removed broadcast_43_piece0 on localhost:56137 in memory (size: 2.2 KB, free: 264.9 MB)
16/07/19 23:55:19 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/metrics/json,null}
16/07/19 23:55:19 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/stage/kill,null}
16/07/19 23:55:19 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/api,null}
16/07/19 23:55:19 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/,null}
16/07/19 23:55:19 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/static,null}
16/07/19 23:55:19 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/executors/threadDump/json,null}
16/07/19 23:55:19 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/executors/threadDump,null}
16/07/19 23:55:19 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/executors/json,null}
16/07/19 23:55:19 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/executors,null}
16/07/19 23:55:19 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/environment/json,null}
16/07/19 23:55:19 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/environment,null}
16/07/19 23:55:19 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/storage/rdd/json,null}
16/07/19 23:55:19 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/storage/rdd,null}
16/07/19 23:55:19 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/storage/json,null}
16/07/19 23:55:19 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/storage,null}
16/07/19 23:55:19 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/pool/json,null}
16/07/19 23:55:19 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/pool,null}
16/07/19 23:55:19 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/stage/json,null}
16/07/19 23:55:19 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/stage,null}
16/07/19 23:55:19 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/json,null}
16/07/19 23:55:19 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages,null}
16/07/19 23:55:19 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/jobs/job/json,null}
16/07/19 23:55:19 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/jobs/job,null}
16/07/19 23:55:19 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/jobs/json,null}
16/07/19 23:55:19 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/jobs,null}
16/07/19 23:55:19 INFO ui.SparkUI: Stopped Spark web UI at http://192.168.3.110:4040 16/07/19 23:55:19 INFO scheduler.DAGScheduler: Stopping DAGScheduler
16/07/19 23:55:19 INFO spark.MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
16/07/19 23:55:19 INFO util.Utils: path = /tmp/spark-ae3f749b-b6f3-4a4a-97c0-d474b85a89c5/blockmgr-461632dc-c7ea-4a60-83ab-baed43170006, already present as root for deletion.
16/07/19 23:55:19 INFO storage.MemoryStore: MemoryStore cleared
16/07/19 23:55:19 INFO storage.BlockManager: BlockManager stopped
16/07/19 23:55:19 INFO storage.BlockManagerMaster: BlockManagerMaster stopped
16/07/19 23:55:19 INFO spark.SparkContext: Successfully stopped SparkContext
16/07/19 23:55:19 INFO scheduler.OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
16/07/19 23:55:19 INFO remote.RemoteActorRefProvider$RemotingTerminator: Shutting down remote daemon.
16/07/19 23:55:19 INFO remote.RemoteActorRefProvider$RemotingTerminator: Remote daemon shut down; proceeding with flushing remote transports.
16/07/19 23:55:19 INFO util.Utils: Shutdown hook called
16/07/19 23:55:19 INFO util.Utils: Deleting directory /tmp/spark-ae3f749b-b6f3-4a4a-97c0-d474b85a89c5
16/07/19 23:55:19 INFO util.Utils: Deleting directory /tmp/spark-204cf89b-ce08-42fc-be47-ba2ba033d2c8
16/07/19 23:55:19 INFO remote.RemoteActorRefProvider$RemotingTerminator: Remoting shut down.
16/07/19 23:55:19 INFO util.Utils: Deleting directory /tmp/spark-08615e52-6632-46e7-be70-cd38504dfd2e
[root@cdh1 ~]# hdfs dfs -ls /user/test
Found 1 items
-rw-r--r-- 1 root supergroup 185 2016-07-19 23:26 /user/test/customers.txt
[root@cdh1 ~]# hdfs dfs -ls /user/root/data
Found 1 items
-rw-r--r-- 1 root supergroup 185 2016-07-19 23:48 /user/root/data/customer.tx
[root@cdh1 ~]# hdfs dfs -ls /user/test
hdfs dfs -ls
Found 1 items
-rw-r--r-- 1 root supergroup 185 2016-07-19 23:48 /user/root/data/customer.tx
[root@cdh1 ~]# hdfs dfs -m/customer.tx /user/root/data/customers
[root@cdh1 ~]# hdfs dfs -mv /user/root/data/customer.tx /user/root/data/customers.txt
ls /user/root/data
Found 1 items
-rw-r--r-- 1 root supergroup 185 2016-07-19 23:48 /user/root/data/customers.txt
[root@cdh1 ~]# jps
19801 NameNode
20330 NodeManager
25692 Jps
22342 Master
22509 Worker
20231 ResourceManager
20082 SecondaryNameNode
19898 DataNode
[root@cdh1 ~]# spark-shell
16/07/19 23:57:55 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
16/07/19 23:57:55 INFO spark.SecurityManager: Changing view acls to: root
16/07/19 23:57:55 INFO spark.SecurityManager: Changing modify acls to: root
16/07/19 23:57:55 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root); users with modify permissions: Set(root)
16/07/19 23:57:55 INFO spark.HttpServer: Starting HTTP Server
16/07/19 23:57:55 INFO server.Server: jetty-8.y.z-SNAPSHOT
16/07/19 23:57:55 INFO server.AbstractConnector: Started SocketConnector@0.0.0.0:32775
16/07/19 23:57:55 INFO util.Utils: Successfully started service 'HTTP class server' on port 32775.
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 1.4.0
/_/
Using Scala version 2.10.4 (Java HotSpot(TM) Server VM, Java 1.7.0_67)
Type in expressions to have them evaluated.
Type :help for more information.
16/07/19 23:58:02 INFO spark.SparkContext: Running Spark version 1.4.0
16/07/19 23:58:03 INFO spark.SecurityManager: Changing view acls to: root
16/07/19 23:58:03 INFO spark.SecurityManager: Changing modify acls to: root
16/07/19 23:58:03 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root); users with modify permissions: Set(root)
16/07/19 23:58:04 INFO slf4j.Slf4jLogger: Slf4jLogger started
16/07/19 23:58:04 INFO Remoting: Starting remoting
16/07/19 23:58:04 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriver@192.168.3.110:53811]
16/07/19 23:58:04 INFO util.Utils: Successfully started service 'sparkDriver' on port 53811.
16/07/19 23:58:04 INFO spark.SparkEnv: Registering MapOutputTracker
16/07/19 23:58:04 INFO spark.SparkEnv: Registering BlockManagerMaster
16/07/19 23:58:05 INFO storage.DiskBlockManager: Created local directory at /tmp/spark-e50d3364-3769-4e1f-931c-e4546f928fb1/blockmgr-3264bf2d-b01d-4e98-a050-95dd146208ee
16/07/19 23:58:05 INFO storage.MemoryStore: MemoryStore started with capacity 265.1 MB
16/07/19 23:58:05 INFO spark.HttpFileServer: HTTP File server directory is /tmp/spark-e50d3364-3769-4e1f-931c-e4546f928fb1/httpd-db40a758-4b48-40be-a566-ed73bff6da0b
16/07/19 23:58:05 INFO spark.HttpServer: Starting HTTP Server
16/07/19 23:58:05 INFO server.Server: jetty-8.y.z-SNAPSHOT
16/07/19 23:58:05 INFO server.AbstractConnector: Started SocketConnector@0.0.0.0:33028
16/07/19 23:58:05 INFO util.Utils: Successfully started service 'HTTP file server' on port 33028.
16/07/19 23:58:05 INFO spark.SparkEnv: Registering OutputCommitCoordinator
16/07/19 23:58:05 INFO server.Server: jetty-8.y.z-SNAPSHOT
16/07/19 23:58:05 INFO server.AbstractConnector: Started SelectChannelConnector@0.0.0.0:4040
16/07/19 23:58:05 INFO util.Utils: Successfully started service 'SparkUI' on port 4040.
16/07/19 23:58:05 INFO ui.SparkUI: Started SparkUI at http://192.168.3.110:4040 16/07/19 23:58:05 INFO executor.Executor: Starting executor ID driver on host localhost
16/07/19 23:58:05 INFO executor.Executor: Using REPL class URI: http://192.168.3.110:32775 16/07/19 23:58:06 INFO util.Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 60239.
16/07/19 23:58:06 INFO netty.NettyBlockTransferService: Server created on 60239
16/07/19 23:58:06 INFO storage.BlockManagerMaster: Trying to register BlockManager
16/07/19 23:58:06 INFO storage.BlockManagerMasterEndpoint: Registering block manager localhost:60239 with 265.1 MB RAM, BlockManagerId(driver, localhost, 60239)
16/07/19 23:58:06 INFO storage.BlockManagerMaster: Registered BlockManager
16/07/19 23:58:06 INFO repl.SparkILoop: Created spark context..
Spark context available as sc.
16/07/19 23:58:07 INFO hive.HiveContext: Initializing execution hive, version 0.13.1
16/07/19 23:58:08 INFO metastore.HiveMetaStore: 0: Opening raw store with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore
16/07/19 23:58:08 INFO metastore.ObjectStore: ObjectStore, initialize called
16/07/19 23:58:09 INFO DataNucleus.Persistence: Property datanucleus.cache.level2 unknown - will be ignored
16/07/19 23:58:09 INFO DataNucleus.Persistence: Property hive.metastore.integral.jdo.pushdown unknown - will be ignored
16/07/19 23:58:09 WARN DataNucleus.Connection: BoneCP specified but not present in CLASSPATH (or one of dependencies)
16/07/19 23:58:10 WARN DataNucleus.Connection: BoneCP specified but not present in CLASSPATH (or one of dependencies)
16/07/19 23:58:13 INFO metastore.ObjectStore: Setting MetaStore object pin classes with hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order"
16/07/19 23:58:13 INFO metastore.MetaStoreDirectSql: MySQL check failed, assuming we are not on mysql: Lexical error at line 1, column 5. Encountered: "@" (64), after : "".
16/07/19 23:58:15 INFO DataNucleus.Datastore: The class "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as "embedded-only" so does not have its own datastore table.
16/07/19 23:58:15 INFO DataNucleus.Datastore: The class "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only" so does not have its own datastore table.
16/07/19 23:58:17 INFO DataNucleus.Datastore: The class "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as "embedded-only" so does not have its own datastore table.
16/07/19 23:58:17 INFO DataNucleus.Datastore: The class "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only" so does not have its own datastore table.
16/07/19 23:58:18 INFO metastore.ObjectStore: Initialized ObjectStore
16/07/19 23:58:18 WARN metastore.ObjectStore: Version information not found in metastore. hive.metastore.schema.verification is not enabled so recording the schema version 0.13.1aa
16/07/19 23:58:19 INFO metastore.HiveMetaStore: Added admin role in metastore
16/07/19 23:58:19 INFO metastore.HiveMetaStore: Added public role in metastore
16/07/19 23:58:19 INFO metastore.HiveMetaStore: No user is added in admin role, since config is empty
16/07/19 23:58:19 INFO session.SessionState: No Tez session required at this point. hive.execution.engine=mr.
16/07/19 23:58:19 INFO repl.SparkILoop: Created sql context (with Hive support)..
SQL context available as sqlContext.
scala> dfCust.show();registerTempTable("custs");val dfCust = sc.textFile("data/customers.txt").map(_.split(",")).map(p => Customer(p(0).trim.toInt, p(1), p(2), p(3), p(4))).toDF();
<console>:21: error: not found: value Customer
val dfCust = sc.textFile("data/customers.txt").map(_.split(",")).map(p => Customer(p(0).trim.toInt, p(1), p(2), p(3), p(4))).toDF();
^
scala> val dfCust = sc.textFile("data/customers.txt").map(_.split(",")).map(p => Customer(p(0).trim.toInt, p(1), p(2), p(3), p(4))).toDF();dfCust.show();registerTempTable("custs");val dfCust = sc.textFile("data/customers.txt").map(_.split(",")).map(p => Customer(p(0).trim.toInt, p(1), p(2), p(3), p(4))).toDF();dfCust.show();registerTempTable("custs");val dfCust = sc.textFile("data/customers.txt").map(_.split(",")).map(p => Customer(p(0).trim.toInt, p(1), p(2), p(3), p(4))).toDF();dfCustomers.show();registerTempTable("customers");val dfCustomers = sc.textFile("data/customers.txt").map(_.split(",")).map(p => Customer(p(0).trim.toInt, p(1), p(2), p(3), p(4))).toDF();case class Customer(customer_id: Int, name: String, city: String, state: String, zip_code: String);
defined class Customer
scala> case class Customer(customer_id: Int, name: String, city: String, state: String, zip_code: String);val dfCust = sc.textFile("data/customers.txt").map(_.split(",")).map(p => Customer(p(0).trim.toInt, p(1), p(2), p(3), p(4))).toDF();
16/07/19 23:58:59 WARN util.SizeEstimator: Failed to check whether UseCompressedOops is set; assuming yes
16/07/19 23:58:59 INFO storage.MemoryStore: ensureFreeSpace(85352) called with curMem=0, maxMem=278019440
16/07/19 23:58:59 INFO storage.MemoryStore: Block broadcast_0 stored as values in memory (estimated size 83.4 KB, free 265.1 MB)
16/07/19 23:58:59 INFO storage.MemoryStore: ensureFreeSpace(19999) called with curMem=85352, maxMem=278019440
16/07/19 23:58:59 INFO storage.MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 19.5 KB, free 265.0 MB)
16/07/19 23:58:59 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in memory on localhost:60239 (size: 19.5 KB, free: 265.1 MB)
16/07/19 23:58:59 INFO spark.SparkContext: Created broadcast 0 from textFile at <console>:23
16/07/19 23:59:00 INFO hive.HiveContext: Initializing HiveMetastoreConnection version 0.13.1 using Spark classes.
16/07/19 23:59:01 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
16/07/19 23:59:01 INFO metastore.HiveMetaStore: 0: Opening raw store with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore
16/07/19 23:59:01 INFO metastore.ObjectStore: ObjectStore, initialize called
16/07/19 23:59:02 INFO DataNucleus.Persistence: Property datanucleus.cache.level2 unknown - will be ignored
16/07/19 23:59:02 INFO DataNucleus.Persistence: Property hive.metastore.integral.jdo.pushdown unknown - will be ignored
16/07/19 23:59:02 WARN DataNucleus.Connection: BoneCP specified but not present in CLASSPATH (or one of dependencies)
16/07/19 23:59:02 WARN DataNucleus.Connection: BoneCP specified but not present in CLASSPATH (or one of dependencies)
16/07/19 23:59:05 INFO metastore.ObjectStore: Setting MetaStore object pin classes with hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order"
16/07/19 23:59:05 INFO metastore.MetaStoreDirectSql: MySQL check failed, assuming we are not on mysql: Lexical error at line 1, column 5. Encountered: "@" (64), after : "".
16/07/19 23:59:06 INFO DataNucleus.Datastore: The class "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as "embedded-only" so does not have its own datastore table.
16/07/19 23:59:06 INFO DataNucleus.Datastore: The class "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only" so does not have its own datastore table.
16/07/19 23:59:08 INFO DataNucleus.Datastore: The class "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as "embedded-only" so does not have its own datastore table.
16/07/19 23:59:08 INFO DataNucleus.Datastore: The class "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only" so does not have its own datastore table.
16/07/19 23:59:09 INFO metastore.ObjectStore: Initialized ObjectStore
16/07/19 23:59:09 WARN metastore.ObjectStore: Version information not found in metastore. hive.metastore.schema.verification is not enabled so recording the schema version 0.13.1aa
16/07/19 23:59:10 INFO metastore.HiveMetaStore: Added admin role in metastore
16/07/19 23:59:10 INFO metastore.HiveMetaStore: Added public role in metastore
16/07/19 23:59:10 INFO metastore.HiveMetaStore: No user is added in admin role, since config is empty
16/07/19 23:59:10 INFO session.SessionState: No Tez session required at this point. hive.execution.engine=mr.
dfCust: org.apache.spark.sql.DataFrame = [customer_id: int, name: string, city: string, state: string, zip_code: string]
scala> val dfCust = sc.textFile("data/customers.txt").map(_.split(",")).map(p => Customer(p(0).trim.toInt, p(1), p(2), p(3), p(4))).toDF();case class Customer(customer_id: Int, name: String, city: String, state: String, zip_code: String);val dfCust = sc.textFile("data/customers.txt").map(_.split(",")).map(p => Customer(p(0).trim.toInt, p(1), p(2), p(3), p(4))).toDF();dfCust.show();registerTempTable("custs");val dfCust = sc.textFile("data/customers.txt").map(_.split(",")).map(p => Customer(p(0).trim.toInt, p(1), p(2), p(3), p(4))).toDF();dfCust.registerTempTable("custs");
scala> dfCust.registerTempTable("custs");val dfCust = sc.textFile("data/customers.txt").map(_.split(",")).map(p => Customer(p(0).trim.toInt, p(1), p(2), p(3), p(4))).toDF();case class Customer(customer_id: Int, name: String, city: String, state: String, zip_code: String);val dfCust = sc.textFile("data/customers.txt").map(_.split(",")).map(p => Customer(p(0).trim.toInt, p(1), p(2), p(3), p(4))).toDF();dfCust.show();
16/07/19 23:59:28 INFO mapred.FileInputFormat: Total input paths to process : 1
16/07/19 23:59:28 INFO spark.SparkContext: Starting job: show at <console>:26
16/07/19 23:59:28 INFO scheduler.DAGScheduler: Got job 0 (show at <console>:26) with 1 output partitions (allowLocal=false)
16/07/19 23:59:28 INFO scheduler.DAGScheduler: Final stage: ResultStage 0(show at <console>:26)
16/07/19 23:59:28 INFO scheduler.DAGScheduler: Parents of final stage: List()
16/07/19 23:59:28 INFO scheduler.DAGScheduler: Missing parents: List()
16/07/19 23:59:28 INFO scheduler.DAGScheduler: Submitting ResultStage 0 (MapPartitionsRDD[5] at show at <console>:26), which has no missing parents
16/07/19 23:59:28 INFO storage.MemoryStore: ensureFreeSpace(4040) called with curMem=105351, maxMem=278019440
16/07/19 23:59:28 INFO storage.MemoryStore: Block broadcast_1 stored as values in memory (estimated size 3.9 KB, free 265.0 MB)
16/07/19 23:59:28 INFO storage.MemoryStore: ensureFreeSpace(2214) called with curMem=109391, maxMem=278019440
16/07/19 23:59:28 INFO storage.MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 2.2 KB, free 265.0 MB)
16/07/19 23:59:28 INFO storage.BlockManagerInfo: Added broadcast_1_piece0 in memory on localhost:60239 (size: 2.2 KB, free: 265.1 MB)
16/07/19 23:59:28 INFO spark.SparkContext: Created broadcast 1 from broadcast at DAGScheduler.scala:874
16/07/19 23:59:28 INFO scheduler.DAGScheduler: Submitting 1 missing tasks from ResultStage 0 (MapPartitionsRDD[5] at show at <console>:26)
16/07/19 23:59:28 INFO scheduler.TaskSchedulerImpl: Adding task set 0.0 with 1 tasks
16/07/19 23:59:28 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, localhost, ANY, 1418 bytes)
16/07/19 23:59:28 INFO executor.Executor: Running task 0.0 in stage 0.0 (TID 0)
16/07/19 23:59:28 INFO rdd.HadoopRDD: Input split: hdfs://cdh1:9000/user/root/data/customers.txt:0+92
16/07/19 23:59:28 INFO Configuration.deprecation: mapred.tip.id is deprecated. Instead, use mapreduce.task.id
16/07/19 23:59:28 INFO Configuration.deprecation: mapred.task.id is deprecated. Instead, use mapreduce.task.attempt.id
16/07/19 23:59:28 INFO Configuration.deprecation: mapred.task.is.map is deprecated. Instead, use mapreduce.task.ismap
16/07/19 23:59:28 INFO Configuration.deprecation: mapred.task.partition is deprecated. Instead, use mapreduce.task.partition
16/07/19 23:59:28 INFO Configuration.deprecation: mapred.job.id is deprecated. Instead, use mapreduce.job.id
16/07/19 23:59:28 INFO executor.Executor: Finished task 0.0 in stage 0.0 (TID 0). 2420 bytes result sent to driver
16/07/19 23:59:28 INFO scheduler.DAGScheduler: ResultStage 0 (show at <console>:26) finished in 0.299 s
16/07/19 23:59:29 INFO scheduler.DAGScheduler: Job 0 finished: show at <console>:26, took 0.478343 s
16/07/19 23:59:29 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) in 282 ms on localhost (1/1)
16/07/19 23:59:29 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool
16/07/19 23:59:29 INFO spark.SparkContext: Starting job: show at <console>:26
16/07/19 23:59:29 INFO scheduler.DAGScheduler: Got job 1 (show at <console>:26) with 1 output partitions (allowLocal=false)
16/07/19 23:59:29 INFO scheduler.DAGScheduler: Final stage: ResultStage 1(show at <console>:26)
16/07/19 23:59:29 INFO scheduler.DAGScheduler: Parents of final stage: List()
16/07/19 23:59:29 INFO scheduler.DAGScheduler: Missing parents: List()
16/07/19 23:59:29 INFO scheduler.DAGScheduler: Submitting ResultStage 1 (MapPartitionsRDD[5] at show at <console>:26), which has no missing parents
16/07/19 23:59:29 INFO storage.MemoryStore: ensureFreeSpace(4040) called with curMem=111605, maxMem=278019440
16/07/19 23:59:29 INFO storage.MemoryStore: Block broadcast_2 stored as values in memory (estimated size 3.9 KB, free 265.0 MB)
16/07/19 23:59:29 INFO storage.MemoryStore: ensureFreeSpace(2214) called with curMem=115645, maxMem=278019440
16/07/19 23:59:29 INFO storage.MemoryStore: Block broadcast_2_piece0 stored as bytes in memory (estimated size 2.2 KB, free 265.0 MB)
16/07/19 23:59:29 INFO storage.BlockManagerInfo: Added broadcast_2_piece0 in memory on localhost:60239 (size: 2.2 KB, free: 265.1 MB)
16/07/19 23:59:29 INFO spark.SparkContext: Created broadcast 2 from broadcast at DAGScheduler.scala:874
16/07/19 23:59:29 INFO scheduler.DAGScheduler: Submitting 1 missing tasks from ResultStage 1 (MapPartitionsRDD[5] at show at <console>:26)
16/07/19 23:59:29 INFO scheduler.TaskSchedulerImpl: Adding task set 1.0 with 1 tasks
16/07/19 23:59:29 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 1.0 (TID 1, localhost, ANY, 1418 bytes)
16/07/19 23:59:29 INFO executor.Executor: Running task 0.0 in stage 1.0 (TID 1)
16/07/19 23:59:29 INFO rdd.HadoopRDD: Input split: hdfs://cdh1:9000/user/root/data/customers.txt:92+93
16/07/19 23:59:29 INFO executor.Executor: Finished task 0.0 in stage 1.0 (TID 1). 2311 bytes result sent to driver
16/07/19 23:59:29 INFO scheduler.DAGScheduler: ResultStage 1 (show at <console>:26) finished in 0.059 s
16/07/19 23:59:29 INFO scheduler.DAGScheduler: Job 1 finished: show at <console>:26, took 0.088599 s
16/07/19 23:59:29 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 1.0 (TID 1) in 60 ms on localhost (1/1)
16/07/19 23:59:29 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 1.0, whose tasks have all completed, from pool
+-----------+---------------+------------+-----+--------+
|customer_id| name| city|state|zip_code|
+-----------+---------------+------------+-----+--------+
| 100| John Smith| Austin| TX| 78727|
| 200| Joe Johnson| Dallas| TX| 75201|
| 300| Bob Jones| Houston| TX| 77028|
| 400| Andy Davis| San Antonio| TX| 78227|
| 500| James Williams| Austin| TX| 78727|
+-----------+---------------+------------+-----+--------+
scala> dfCust.show();Stopping spark context.
16/07/20 00:03:35 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/metrics/json,null}
16/07/20 00:03:35 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/stage/kill,null}
16/07/20 00:03:35 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/api,null}
16/07/20 00:03:35 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/,null}
16/07/20 00:03:35 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/static,null}
16/07/20 00:03:35 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/executors/threadDump/json,null}
16/07/20 00:03:35 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/executors/threadDump,null}
16/07/20 00:03:35 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/executors/json,null}
16/07/20 00:03:35 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/executors,null}
16/07/20 00:03:35 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/environment/json,null}
16/07/20 00:03:35 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/environment,null}
16/07/20 00:03:35 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/storage/rdd/json,null}
16/07/20 00:03:35 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/storage/rdd,null}
16/07/20 00:03:35 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/storage/json,null}
16/07/20 00:03:35 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/storage,null}
16/07/20 00:03:35 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/pool/json,null}
16/07/20 00:03:35 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/pool,null}
16/07/20 00:03:35 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/stage/json,null}
16/07/20 00:03:35 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/stage,null}
16/07/20 00:03:35 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/json,null}
16/07/20 00:03:35 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages,null}
16/07/20 00:03:35 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/jobs/job/json,null}
16/07/20 00:03:35 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/jobs/job,null}
16/07/20 00:03:35 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/jobs/json,null}
16/07/20 00:03:35 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/jobs,null}
16/07/20 00:03:35 INFO ui.SparkUI: Stopped Spark web UI at http://192.168.3.110:4040 16/07/20 00:03:35 INFO scheduler.DAGScheduler: Stopping DAGScheduler
16/07/20 00:03:35 INFO spark.MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
16/07/20 00:03:35 INFO util.Utils: path = /tmp/spark-e50d3364-3769-4e1f-931c-e4546f928fb1/blockmgr-3264bf2d-b01d-4e98-a050-95dd146208ee, already present as root for deletion.
16/07/20 00:03:35 INFO storage.MemoryStore: MemoryStore cleared
16/07/20 00:03:35 INFO storage.BlockManager: BlockManager stopped
16/07/20 00:03:35 INFO storage.BlockManagerMaster: BlockManagerMaster stopped
16/07/20 00:03:35 INFO scheduler.OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
16/07/20 00:03:35 INFO spark.SparkContext: Successfully stopped SparkContext
16/07/20 00:03:35 INFO remote.RemoteActorRefProvider$RemotingTerminator: Shutting down remote daemon.
16/07/20 00:03:35 INFO remote.RemoteActorRefProvider$RemotingTerminator: Remote daemon shut down; proceeding with flushing remote transports.
16/07/20 00:03:35 INFO remote.RemoteActorRefProvider$RemotingTerminator: Remoting shut down.
16/07/20 00:03:35 INFO util.Utils: Shutdown hook called
16/07/20 00:03:35 INFO util.Utils: Deleting directory /tmp/spark-e50d3364-3769-4e1f-931c-e4546f928fb1
16/07/20 00:03:35 INFO util.Utils: Deleting directory /tmp/spark-0af3ccea-1dec-4bac-8ee4-c36a8f2ea48f
16/07/20 00:03:35 INFO util.Utils: Deleting directory /tmp/spark-2a3e1bd9-e6ce-4cac-97a4-88d4a2f567eb
[root@cdh1 ~]#
[END] 2016/7/20 23:17:51
内容来自用户分享和网络整理，不保证内容的准确性，如有侵权内容，可联系管理员处理 
相关文章推荐
- Spark RDD API详解(一) Map和Reduce
- 使用spark和spark mllib进行股票预测
- Windows下Scala环境搭建
- Spark随谈——开发指南（译）
- Spark，一种快速数据分析替代方案
- Windows7下安装Scala 2.9.2教程
- eclipse 开发 spark Streaming wordCount
- Understanding Spark Caching
- Scala代码实现列出Hadoop 文件夹下面的所有文件
- ClassNotFoundException:scala.PreDef$
- Windows 下Spark 快速搭建Spark源码阅读环境
- Spark中将对象序列化存储到hdfs
- sbt创建web项目
- 使用java代码提交Spark的hive sql任务，run as java application
- XML 文件解析--含Unicode字符的XML文件
- Scala 学习随笔
- Scala 小程序记录（学习期间的代码片段）
- Spark机器学习（一） -- Machine Learning Library (MLlib)
- Spark机器学习（二） 局部向量 Local-- Data Types - MLlib