基于Spark分析寻医问药网的问答数据

下面一系列将会基于两个节点的yarn，spark来分析寻医问药网上面的问答数据：

第一次分析（updated in 2016.9.19 by RichardMore）：

scala> val in = sqlContext.jsonFile("hdfs:///user/miaofu/healthcare/2016-05-01/content.txt")
warning: there were 1 deprecation warning(s); re-run with -deprecation for details
16/09/19 15:05:17 INFO BlockManagerInfo: Removed broadcast_2_piece0 on master:41990 in memory (size: 1867.0 B, free: 511.5 MB)
16/09/19 15:05:17 INFO BlockManagerInfo: Removed broadcast_2_piece0 on 192.168.202.104:56322 in memory (size: 1867.0 B, free: 511.5 MB)
16/09/19 15:05:17 INFO ContextCleaner: Cleaned accumulator 2
16/09/19 15:05:17 INFO JSONRelation: Listing hdfs://master:8020/user/miaofu/healthcare/2016-05-01/content.txt on driver
16/09/19 15:05:18 INFO MemoryStore: Block broadcast_3 stored as values in memory (estimated size 230.1 KB, free 484.0 KB)
16/09/19 15:05:18 INFO MemoryStore: Block broadcast_3_piece0 stored as bytes in memory (estimated size 19.6 KB, free 503.7 KB)
16/09/19 15:05:18 INFO BlockManagerInfo: Added broadcast_3_piece0 in memory on 192.168.202.104:56322 (size: 19.6 KB, free: 511.5 MB)
16/09/19 15:05:18 INFO SparkContext: Created broadcast 3 from jsonFile at <console>:25
16/09/19 15:05:18 INFO FileInputFormat: Total input paths to process : 1
16/09/19 15:05:18 INFO SparkContext: Starting job: jsonFile at <console>:25
16/09/19 15:05:18 INFO DAGScheduler: Got job 2 (jsonFile at <console>:25) with 2 output partitions
16/09/19 15:05:18 INFO DAGScheduler: Final stage: ResultStage 2 (jsonFile at <console>:25)
16/09/19 15:05:18 INFO DAGScheduler: Parents of final stage: List()
16/09/19 15:05:18 INFO DAGScheduler: Missing parents: List()
16/09/19 15:05:18 INFO DAGScheduler: Submitting ResultStage 2 (MapPartitionsRDD[5] at jsonFile at <console>:25), which has no missing parents
16/09/19 15:05:18 INFO MemoryStore: Block broadcast_4 stored as values in memory (estimated size 4.3 KB, free 508.0 KB)
16/09/19 15:05:18 INFO MemoryStore: Block broadcast_4_piece0 stored as bytes in memory (estimated size 2.5 KB, free 510.5 KB)
16/09/19 15:05:18 INFO BlockManagerInfo: Added broadcast_4_piece0 in memory on 192.168.202.104:56322 (size: 2.5 KB, free: 511.5 MB)
16/09/19 15:05:18 INFO SparkContext: Created broadcast 4 from broadcast at DAGScheduler.scala:1006
16/09/19 15:05:18 INFO DAGScheduler: Submitting 2 missing tasks from ResultStage 2 (MapPartitionsRDD[5] at jsonFile at <console>:25)
16/09/19 15:05:18 INFO YarnScheduler: Adding task set 2.0 with 2 tasks
16/09/19 15:05:18 INFO TaskSetManager: Starting task 0.0 in stage 2.0 (TID 3, master, partition 0,NODE_LOCAL, 2166 bytes)
16/09/19 15:05:18 INFO TaskSetManager: Starting task 1.0 in stage 2.0 (TID 4, slave02, partition 1,NODE_LOCAL, 2166 bytes)
16/09/19 15:05:18 INFO BlockManagerInfo: Added broadcast_4_piece0 in memory on master:41990 (size: 2.5 KB, free: 511.5 MB)
16/09/19 15:05:18 INFO BlockManagerInfo: Added broadcast_4_piece0 in memory on slave02:56196 (size: 2.5 KB, free: 511.5 MB)
16/09/19 15:05:18 INFO BlockManagerInfo: Added broadcast_3_piece0 in memory on slave02:56196 (size: 19.6 KB, free: 511.5 MB)
16/09/19 15:05:18 INFO BlockManagerInfo: Added broadcast_3_piece0 in memory on master:41990 (size: 19.6 KB, free: 511.5 MB)
16/09/19 15:05:20 INFO TaskSetManager: Finished task 1.0 in stage 2.0 (TID 4) in 2304 ms on slave02 (1/2)
16/09/19 15:05:22 INFO TaskSetManager: Finished task 0.0 in stage 2.0 (TID 3) in 3876 ms on master (2/2)
16/09/19 15:05:22 INFO DAGScheduler: ResultStage 2 (jsonFile at <console>:25) finished in 3.877 s
16/09/19 15:05:22 INFO YarnScheduler: Removed TaskSet 2.0, whose tasks have all completed, from pool
16/09/19 15:05:22 INFO DAGScheduler: Job 2 finished: jsonFile at <console>:25, took 4.548309 s
in: org.apache.spark.sql.DataFrame = [answer: array<struct<answer_text:string,answer_time:string,mainpage:string,major:string,name:string,position:string>>, question: struct<question:string,time:string>, userinfo: struct<age:string,gender:string,name:string>]

scala> in
res2: org.apache.spark.sql.DataFrame = [answer: array<struct<answer_text:string,answer_time:string,mainpage:string,major:string,name:string,position:string>>, question: struct<question:string,time:string>, userinfo: struct<age:string,gender:string,name:string>]

scala> in.count()
16/09/19 15:09:23 INFO MemoryStore: Block broadcast_5 stored as values in memory (estimated size 86.5 KB, free 596.9 KB)
16/09/19 15:09:23 INFO MemoryStore: Block broadcast_5_piece0 stored as bytes in memory (estimated size 19.5 KB, free 616.4 KB)
16/09/19 15:09:23 INFO BlockManagerInfo: Added broadcast_5_piece0 in memory on 192.168.202.104:56322 (size: 19.5 KB, free: 511.4 MB)
16/09/19 15:09:23 INFO SparkContext: Created broadcast 5 from count at <console>:28
16/09/19 15:09:23 INFO MemoryStore: Block broadcast_6 stored as values in memory (estimated size 230.1 KB, free 846.5 KB)
16/09/19 15:09:23 INFO MemoryStore: Block broadcast_6_piece0 stored as bytes in memory (estimated size 19.6 KB, free 866.2 KB)
16/09/19 15:09:23 INFO BlockManagerInfo: Added broadcast_6_piece0 in memory on 192.168.202.104:56322 (size: 19.6 KB, free: 511.4 MB)
16/09/19 15:09:23 INFO SparkContext: Created broadcast 6 from count at <console>:28
16/09/19 15:09:24 INFO FileInputFormat: Total input paths to process : 1
16/09/19 15:09:24 INFO SparkContext: Starting job: count at <console>:28
16/09/19 15:09:24 INFO DAGScheduler: Registering RDD 11 (count at <console>:28)
16/09/19 15:09:24 INFO DAGScheduler: Got job 3 (count at <console>:28) with 1 output partitions
16/09/19 15:09:24 INFO DAGScheduler: Final stage: ResultStage 4 (count at <console>:28)
16/09/19 15:09:24 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 3)
16/09/19 15:09:24 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 3)
16/09/19 15:09:24 INFO DAGScheduler: Submitting ShuffleMapStage 3 (MapPartitionsRDD[11] at count at <console>:28), which has no missing parents
16/09/19 15:09:24 INFO MemoryStore: Block broadcast_7 stored as values in memory (estimated size 10.4 KB, free 876.6 KB)
16/09/19 15:09:24 INFO MemoryStore: Block broadcast_7_piece0 stored as bytes in memory (estimated size 5.4 KB, free 881.9 KB)
16/09/19 15:09:24 INFO BlockManagerInfo: Added broadcast_7_piece0 in memory on 192.168.202.104:56322 (size: 5.4 KB, free: 511.4 MB)
16/09/19 15:09:24 INFO SparkContext: Created broadcast 7 from broadcast at DAGScheduler.scala:1006
16/09/19 15:09:24 INFO DAGScheduler: Submitting 2 missing tasks from ShuffleMapStage 3 (MapPartitionsRDD[11] at count at <console>:28)
16/09/19 15:09:24 INFO YarnScheduler: Adding task set 3.0 with 2 tasks
16/09/19 15:09:24 INFO TaskSetManager: Starting task 0.0 in stage 3.0 (TID 5, master, partition 0,NODE_LOCAL, 2155 bytes)
16/09/19 15:09:24 INFO TaskSetManager: Starting task 1.0 in stage 3.0 (TID 6, slave02, partition 1,NODE_LOCAL, 2155 bytes)
16/09/19 15:09:24 INFO BlockManagerInfo: Added broadcast_7_piece0 in memory on slave02:56196 (size: 5.4 KB, free: 511.5 MB)
16/09/19 15:09:24 INFO BlockManagerInfo: Added broadcast_7_piece0 in memory on master:41990 (size: 5.4 KB, free: 511.5 MB)
16/09/19 15:09:24 INFO BlockManagerInfo: Added broadcast_6_piece0 in memory on slave02:56196 (size: 19.6 KB, free: 511.4 MB)
16/09/19 15:09:24 INFO BlockManagerInfo: Added broadcast_6_piece0 in memory on master:41990 (size: 19.6 KB, free: 511.4 MB)
16/09/19 15:09:25 INFO TaskSetManager: Finished task 1.0 in stage 3.0 (TID 6) in 1719 ms on slave02 (1/2)
16/09/19 15:09:26 INFO TaskSetManager: Finished task 0.0 in stage 3.0 (TID 5) in 2596 ms on master (2/2)
16/09/19 15:09:26 INFO DAGScheduler: ShuffleMapStage 3 (count at <console>:28) finished in 2.596 s
16/09/19 15:09:26 INFO YarnScheduler: Removed TaskSet 3.0, whose tasks have all completed, from pool
16/09/19 15:09:26 INFO DAGScheduler: looking for newly runnable stages
16/09/19 15:09:26 INFO DAGScheduler: running: Set()
16/09/19 15:09:26 INFO DAGScheduler: waiting: Set(ResultStage 4)
16/09/19 15:09:26 INFO DAGScheduler: failed: Set()
16/09/19 15:09:26 INFO DAGScheduler: Submitting ResultStage 4 (MapPartitionsRDD[14] at count at <console>:28), which has no missing parents
16/09/19 15:09:26 INFO MemoryStore: Block broadcast_8 stored as values in memory (estimated size 11.5 KB, free 893.5 KB)
16/09/19 15:09:26 INFO MemoryStore: Block broadcast_8_piece0 stored as bytes in memory (estimated size 5.8 KB, free 899.3 KB)
16/09/19 15:09:26 INFO BlockManagerInfo: Added broadcast_8_piece0 in memory on 192.168.202.104:56322 (size: 5.8 KB, free: 511.4 MB)
16/09/19 15:09:26 INFO SparkContext: Created broadcast 8 from broadcast at DAGScheduler.scala:1006
16/09/19 15:09:26 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 4 (MapPartitionsRDD[14] at count at <console>:28)
16/09/19 15:09:26 INFO YarnScheduler: Adding task set 4.0 with 1 tasks
16/09/19 15:09:26 INFO TaskSetManager: Starting task 0.0 in stage 4.0 (TID 7, slave02, partition 0,NODE_LOCAL, 1999 bytes)
16/09/19 15:09:26 INFO BlockManagerInfo: Added broadcast_8_piece0 in memory on slave02:56196 (size: 5.8 KB, free: 511.4 MB)
16/09/19 15:09:26 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 0 to slave02:34081
16/09/19 15:09:27 INFO MapOutputTrackerMaster: Size of output statuses for shuffle 0 is 162 bytes
16/09/19 15:09:27 INFO TaskSetManager: Finished task 0.0 in stage 4.0 (TID 7) in 502 ms on slave02 (1/1)
16/09/19 15:09:27 INFO YarnScheduler: Removed TaskSet 4.0, whose tasks have all completed, from pool
16/09/19 15:09:27 INFO DAGScheduler: ResultStage 4 (count at <console>:28) finished in 0.502 s
16/09/19 15:09:27 INFO DAGScheduler: Job 3 finished: count at <console>:28, took 3.156273 s
res3: Long = 28696

scala> in.printSchema()
root
|-- answer: array (nullable = true)
|    |-- element: struct (containsNull = true)
|    |    |-- answer_text: string (nullable = true)
|    |    |-- answer_time: string (nullable = true)
|    |    |-- mainpage: string (nullable = true)
|    |    |-- major: string (nullable = true)
|    |    |-- name: string (nullable = true)
|    |    |-- position: string (nullable = true)
|-- question: struct (nullable = true)
|    |-- question: string (nullable = true)
|    |-- time: string (nullable = true)
|-- userinfo: struct (nullable = true)
|    |-- age: string (nullable = true)
|    |-- gender: string (nullable = true)
|    |-- name: string (nullable = true)

scala> in.16/09/19 15:10:42 INFO ContextCleaner: Cleaned shuffle 0
16/09/19 15:10:42 INFO ContextCleaner: Cleaned accumulator 12
16/09/19 15:10:42 INFO ContextCleaner: Cleaned accumulator 11
16/09/19 15:10:42 INFO ContextCleaner: Cleaned accumulator 10
16/09/19 15:10:42 INFO ContextCleaner: Cleaned accumulator 9
16/09/19 15:10:42 INFO ContextCleaner: Cleaned accumulator 8
16/09/19 15:10:42 INFO ContextCleaner: Cleaned accumulator 7
16/09/19 15:10:42 INFO ContextCleaner: Cleaned accumulator 6
16/09/19 15:10:42 INFO ContextCleaner: Cleaned accumulator 5
16/09/19 15:10:42 INFO BlockManagerInfo: Removed broadcast_6_piece0 on 192.168.202.104:56322 in memory (size: 19.6 KB, free: 511.4 MB)
16/09/19 15:10:42 INFO BlockManagerInfo: Removed broadcast_6_piece0 on master:41990 in memory (size: 19.6 KB, free: 511.5 MB)
16/09/19 15:10:42 INFO BlockManagerInfo: Removed broadcast_6_piece0 on slave02:56196 in memory (size: 19.6 KB, free: 511.4 MB)
16/09/19 15:10:42 INFO BlockManagerInfo: Removed broadcast_5_piece0 on 192.168.202.104:56322 in memory (size: 19.5 KB, free: 511.4 MB)
16/09/19 15:10:42 INFO BlockManagerInfo: Removed broadcast_4_piece0 on 192.168.202.104:56322 in memory (size: 2.5 KB, free: 511.4 MB)
16/09/19 15:10:42 INFO BlockManagerInfo: Removed broadcast_4_piece0 on slave02:56196 in memory (size: 2.5 KB, free: 511.4 MB)
16/09/19 15:10:42 INFO BlockManagerInfo: Removed broadcast_4_piece0 on master:41990 in memory (size: 2.5 KB, free: 511.5 MB)
16/09/19 15:10:42 INFO ContextCleaner: Cleaned accumulator 3
16/09/19 15:10:42 INFO BlockManagerInfo: Removed broadcast_3_piece0 on 192.168.202.104:56322 in memory (size: 19.6 KB, free: 511.5 MB)
16/09/19 15:10:42 INFO BlockManagerInfo: Removed broadcast_3_piece0 on master:41990 in memory (size: 19.6 KB, free: 511.5 MB)
16/09/19 15:10:42 INFO BlockManagerInfo: Removed broadcast_3_piece0 on slave02:56196 in memory (size: 19.6 KB, free: 511.5 MB)
16/09/19 15:10:42 INFO BlockManagerInfo: Removed broadcast_8_piece0 on 192.168.202.104:56322 in memory (size: 5.8 KB, free: 511.5 MB)
16/09/19 15:10:42 INFO BlockManagerInfo: Removed broadcast_8_piece0 on slave02:56196 in memory (size: 5.8 KB, free: 511.5 MB)
16/09/19 15:10:42 INFO ContextCleaner: Cleaned accumulator 14
16/09/19 15:10:42 INFO BlockManagerInfo: Removed broadcast_7_piece0 on 192.168.202.104:56322 in memory (size: 5.4 KB, free: 511.5 MB)
16/09/19 15:10:42 INFO BlockManagerInfo: Removed broadcast_7_piece0 on master:41990 in memory (size: 5.4 KB, free: 511.5 MB)
16/09/19 15:10:42 INFO BlockManagerInfo: Removed broadcast_7_piece0 on slave02:56196 in memory (size: 5.4 KB, free: 511.5 MB)
16/09/19 15:10:42 INFO ContextCleaner: Cleaned accumulator 13

agg                    alias                  apply                  as                     asInstanceOf           cache                  coalesce
col                    collect                collectAsList          columns                count                  createJDBCTable        cube
describe               distinct               drop                   dropDuplicates         dtypes                 except                 explain
explode                filter                 first                  flatMap                foreach                foreachPartition       groupBy
head                   inputFiles             insertInto             insertIntoJDBC         intersect              isInstanceOf           isLocal
javaRDD                join                   limit                  map                    mapPartitions          na                     orderBy
persist                printSchema            queryExecution         randomSplit            rdd                    registerTempTable      repartition
rollup                 sample                 save                   saveAsParquetFile      saveAsTable            schema                 select
selectExpr             show                   sort                   sortWithinPartitions   sqlContext             stat                   take
takeAsList             toDF                   toJSON                 toJavaRDD              toSchemaRDD            toString               transform
unionAll               unpersist              where                  withColumn             withColumnRenamed      write

scala> in.
agg                    alias                  apply                  as                     asInstanceOf           cache                  coalesce
col                    collect                collectAsList          columns                count                  createJDBCTable        cube
describe               distinct               drop                   dropDuplicates         dtypes                 except                 explain
explode                filter                 first                  flatMap                foreach                foreachPartition       groupBy
head                   inputFiles             insertInto             insertIntoJDBC         intersect              isInstanceOf           isLocal
javaRDD                join                   limit                  map                    mapPartitions          na                     orderBy
persist                printSchema            queryExecution         randomSplit            rdd                    registerTempTable      repartition
rollup                 sample                 save                   saveAsParquetFile      saveAsTable            schema                 select
selectExpr             show                   sort                   sortWithinPartitions   sqlContext             stat                   take
takeAsList             toDF                   toJSON                 toJavaRDD              toSchemaRDD            toString               transform
unionAll               unpersist              where                  withColumn             withColumnRenamed      write

scala> in.cache()
16/09/19 15:10:53 INFO MemoryStore: Block broadcast_9 stored as values in memory (estimated size 229.7 KB, free 483.7 KB)
16/09/19 15:10:53 INFO MemoryStore: Block broadcast_9_piece0 stored as bytes in memory (estimated size 19.5 KB, free 503.1 KB)
16/09/19 15:10:53 INFO BlockManagerInfo: Added broadcast_9_piece0 in memory on 192.168.202.104:56322 (size: 19.5 KB, free: 511.5 MB)
16/09/19 15:10:53 INFO SparkContext: Created broadcast 9 from cache at <console>:28
16/09/19 15:10:53 INFO MemoryStore: Block broadcast_10 stored as values in memory (estimated size 230.1 KB, free 733.3 KB)
16/09/19 15:10:53 INFO MemoryStore: Block broadcast_10_piece0 stored as bytes in memory (estimated size 19.6 KB, free 752.9 KB)
16/09/19 15:10:53 INFO BlockManagerInfo: Added broadcast_10_piece0 in memory on 192.168.202.104:56322 (size: 19.6 KB, free: 511.4 MB)
16/09/19 15:10:53 INFO SparkContext: Created broadcast 10 from cache at <console>:28
res5: in.type = [answer: array<struct<answer_text:string,answer_time:string,mainpage:string,major:string,name:string,position:string>>, question: struct<question:string,time:string>, userinfo: struct<age:string,gender:string,name:string>]

scala> in.isLocal
res6: Boolean = false

scala> in.describe()
res7: org.apache.spark.sql.DataFrame = [summary: string]

scala> in.describe
<console>:28: error: missing arguments for method describe in class DataFrame;
follow this method with `_' if you want to treat it as a partially applied function
in.describe
^

scala> in.head
16/09/19 15:11:47 INFO FileInputFormat: Total input paths to process : 1
16/09/19 15:11:47 INFO SparkContext: Starting job: head at <console>:28
16/09/19 15:11:47 INFO DAGScheduler: Got job 4 (head at <console>:28) with 1 output partitions
16/09/19 15:11:47 INFO DAGScheduler: Final stage: ResultStage 5 (head at <console>:28)
16/09/19 15:11:47 INFO DAGScheduler: Parents of final stage: List()
16/09/19 15:11:47 INFO DAGScheduler: Missing parents: List()
16/09/19 15:11:47 INFO DAGScheduler: Submitting ResultStage 5 (MapPartitionsRDD[22] at head at <console>:28), which has no missing parents
16/09/19 15:11:47 INFO MemoryStore: Block broadcast_11 stored as values in memory (estimated size 9.6 KB, free 762.5 KB)
16/09/19 15:11:47 INFO MemoryStore: Block broadcast_11_piece0 stored as bytes in memory (estimated size 4.8 KB, free 767.4 KB)
16/09/19 15:11:47 INFO BlockManagerInfo: Added broadcast_11_piece0 in memory on 192.168.202.104:56322 (size: 4.8 KB, free: 511.4 MB)
16/09/19 15:11:47 INFO SparkContext: Created broadcast 11 from broadcast at DAGScheduler.scala:1006
16/09/19 15:11:47 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 5 (MapPartitionsRDD[22] at head at <console>:28)
16/09/19 15:11:47 INFO YarnScheduler: Adding task set 5.0 with 1 tasks
16/09/19 15:11:47 INFO TaskSetManager: Starting task 0.0 in stage 5.0 (TID 8, slave02, partition 0,NODE_LOCAL, 2166 bytes)
16/09/19 15:11:47 INFO BlockManagerInfo: Added broadcast_11_piece0 in memory on slave02:56196 (size: 4.8 KB, free: 511.5 MB)
16/09/19 15:11:47 INFO BlockManagerInfo: Added broadcast_10_piece0 in memory on slave02:56196 (size: 19.6 KB, free: 511.5 MB)
16/09/19 15:11:48 INFO BlockManagerInfo: Added rdd_19_0 in memory on slave02:56196 (size: 24.1 MB, free: 487.3 MB)
16/09/19 15:11:48 INFO TaskSetManager: Finished task 0.0 in stage 5.0 (TID 8) in 1474 ms on slave02 (1/1)
16/09/19 15:11:48 INFO DAGScheduler: ResultStage 5 (head at <console>:28) finished in 1.474 s
16/09/19 15:11:48 INFO YarnScheduler: Removed TaskSet 5.0, whose tasks have all completed, from pool
16/09/19 15:11:48 INFO DAGScheduler: Job 4 finished: head at <console>:28, took 1.525440 s
res9: org.apache.spark.sql.Row = [WrappedArray([指导意见：你好，药物要是干吃的话会使人体吸收加慢，在短时间内无法达到药效浓度，从而影响治疗效果。建议正确的服药方式是用开水完全溶解药物，待温度变温搅拌服用！,2016-05-02 00:16:54,http://club.xywy.com/doc_card/92372268,专长：内科,许敏,职称：护士]),[解郁安神颗粒可以干服吗？没有水，长期紧张恐惧，头脑不清醒，总觉得自己会不会死，胸口不适头不适,2016-05-01 23:57:16],[22,男,会员96129648]]

scala> df.filter(df("age")>"21").16/09/19 15:19:04 INFO BlockManagerInfo: Removed broadcast_1_piece0 on 192.168.202.104:56322 in memory (size: 1806.0 B, free: 511.4 MB)
16/09/19 15:19:04 INFO BlockManagerInfo: Removed broadcast_1_piece0 on master:41990 in memory (size: 1806.0 B, free: 511.5 MB)
16/09/19 15:19:04 INFO BlockManagerInfo: Removed broadcast_1_piece0 on slave02:56196 in memory (size: 1806.0 B, free: 487.3 MB)
16/09/19 15:19:04 INFO BlockManagerInfo: Removed broadcast_11_piece0 on 192.168.202.104:56322 in memory (size: 4.8 KB, free: 511.4 MB)
16/09/19 15:19:04 INFO BlockManagerInfo: Removed broadcast_11_piece0 on slave02:56196 in memory (size: 4.8 KB, free: 487.3 MB)
16/09/19 15:19:04 INFO ContextCleaner: Cleaned accumulator 16
16/09/19 15:19:04 INFO BlockManagerInfo: Removed broadcast_9_piece0 on 192.168.202.104:56322 in memory (size: 19.5 KB, free: 511.5 MB)
16/09/19 15:19:04 INFO ContextCleaner: Cleaned accumulator 1

|
|
You typed two blank lines.  Starting a new command.

scala> in.printSchema
def printSchema(): Unit

scala> in.printSchema()
root
|-- answer: array (nullable = true)
|    |-- element: struct (containsNull = true)
|    |    |-- answer_text: string (nullable = true)
|    |    |-- answer_time: string (nullable = true)
|    |    |-- mainpage: string (nullable = true)
|    |    |-- major: string (nullable = true)
|    |    |-- name: string (nullable = true)
|    |    |-- position: string (nullable = true)
|-- question: struct (nullable = true)
|    |-- question: string (nullable = true)
|    |-- time: string (nullable = true)
|-- userinfo: struct (nullable = true)
|    |-- age: string (nullable = true)
|    |-- gender: string (nullable = true)
|    |-- name: string (nullable = true)

scala> in("question")
res11: org.apache.spark.sql.Column = question

scala> in("question")("time")
res12: org.apache.spark.sql.Column = question[time]

scala> in
res13: org.apache.spark.sql.DataFrame = [answer: array<struct<answer_text:string,answer_time:string,mainpage:string,major:string,name:string,position:string>>, question: struct<question:string,time:string>, userinfo: struct<age:string,gender:string,name:string>]

scala> val question = in("question")
question: org.apache.spark.sql.Column = question

scala> question.
!==            %              &&             *              +              -              /              ===            >              >=             alias          and
apply          as             asInstanceOf   asc            between        bitwiseAND     bitwiseOR      bitwiseXOR     cast           contains       desc           divide
endsWith       eqNullSafe     equalTo        explain        geq            getField       getItem        gt             in             isInstanceOf   isNaN          isNotNull
isNull         isin           leq            like           lt             minus          mod            multiply       notEqual       or             otherwise      over
plus           rlike          startsWith     substr         toString       unary_!        unary_-        when           ||

scala> question.
!==            %              &&             *              +              -              /              ===            >              >=             alias          and
apply          as             asInstanceOf   asc            between        bitwiseAND     bitwiseOR      bitwiseXOR     cast           contains       desc           divide
endsWith       eqNullSafe     equalTo        explain        geq            getField       getItem        gt             in             isInstanceOf   isNaN          isNotNull
isNull         isin           leq            like           lt             minus          mod            multiply       notEqual       or             otherwise      over
plus           rlike          startsWith     substr         toString       unary_!        unary_-        when           ||

scala> in.show()
16/09/19 15:33:26 INFO SparkContext: Starting job: show at <console>:28
16/09/19 15:33:26 INFO DAGScheduler: Got job 5 (show at <console>:28) with 1 output partitions
16/09/19 15:33:26 INFO DAGScheduler: Final stage: ResultStage 6 (show at <console>:28)
16/09/19 15:33:26 INFO DAGScheduler: Parents of final stage: List()
16/09/19 15:33:26 INFO DAGScheduler: Missing parents: List()
16/09/19 15:33:26 INFO DAGScheduler: Submitting ResultStage 6 (MapPartitionsRDD[25] at show at <console>:28), which has no missing parents
16/09/19 15:33:26 INFO MemoryStore: Block broadcast_12 stored as values in memory (estimated size 9.6 KB, free 508.6 KB)
16/09/19 15:33:26 INFO MemoryStore: Block broadcast_12_piece0 stored as bytes in memory (estimated size 4.8 KB, free 513.4 KB)
16/09/19 15:33:26 INFO BlockManagerInfo: Added broadcast_12_piece0 in memory on 192.168.202.104:56322 (size: 4.8 KB, free: 511.5 MB)
16/09/19 15:33:26 INFO SparkContext: Created broadcast 12 from broadcast at DAGScheduler.scala:1006
16/09/19 15:33:26 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 6 (MapPartitionsRDD[25] at show at <console>:28)
16/09/19 15:33:26 INFO YarnScheduler: Adding task set 6.0 with 1 tasks
16/09/19 15:33:26 INFO TaskSetManager: Starting task 0.0 in stage 6.0 (TID 9, slave02, partition 0,PROCESS_LOCAL, 2166 bytes)
16/09/19 15:33:26 INFO BlockManagerInfo: Added broadcast_12_piece0 in memory on slave02:56196 (size: 4.8 KB, free: 487.3 MB)
16/09/19 15:33:26 INFO TaskSetManager: Finished task 0.0 in stage 6.0 (TID 9) in 67 ms on slave02 (1/1)
16/09/19 15:33:26 INFO YarnScheduler: Removed TaskSet 6.0, whose tasks have all completed, from pool
16/09/19 15:33:26 INFO DAGScheduler: ResultStage 6 (show at <console>:28) finished in 0.067 s
16/09/19 15:33:26 INFO DAGScheduler: Job 5 finished: show at <console>:28, took 0.084498 s
+--------------------+--------------------+------------------+
|              answer|            question|          userinfo|
+--------------------+--------------------+------------------+
|[[指导意见：你好，药物要是干吃的...|[解郁安神颗粒可以干服吗？没有水，...| [22,男,会员96129648]|
|[[问题分析：阴毛处长红色痘痘你这...|[阴毛边上经常长红色痘痘,2016...|[24岁,女,会员96129643]|
|[[问题分析：按你说的这个问题，这...|[跖骨骨折髓腔可以打钢钉吗,201...|[24岁,男,会员96129642]|
|[[您好朋友据你的情况有可能是腰椎...|[睡觉腰疼，右边特别痛，翻身也疼，...|[28岁,女,会员96129631]|
|[[问题分析：你好根据你所描述的 ...|[和女朋友亲吻口交之后,2016-...|[18岁,男,会员96129641]|
|[[抑郁症患者建议做相关心理咨询，...|[医生你好，我爸是抑郁症患者！今年...|[55岁,男,会员96129640]|
|[[周期多少天呢？按28天算的话，...|[上个月23来28过去，我的排卵期...|[28岁,女,会员96129567]|
|[[问题分析：一般的考虑是痔疮引起...|[医生你好，我是每次上厕所，就是大...|[18岁,女,会员96129636]|
|[[指导意见：你好，根据你的描述这...|[每晚都腿麻，睡觉之前就麻，生气也...|[22岁,女,会员96129635]|
|[[指导意见：像你这个情况大面积的...|[刘医生您好，我父亲被云南省第一人...|[55岁,男,会员96118403]|
|[[问题分析：您好，根据您的病情描...|[最近下颌一按就疼痛，今天晚饭后几...|[21岁,女,会员96129633]|
|[[指导意见：患者男性，21岁，根...|[昨天晚上吃了西红柿和火锅，然后今...|[21岁,男,会员96129619]|
|[[您好，您的这种情况使用皮炎平是...|[你好，我的左脚小拇指脱皮还痒，有...|[32岁,男,会员96129632]|
|[[你好，根据你的描述，建议到该院...|[放射性直肠炎在301应该挂哪个科...|[44岁,女,会员96129522]|
|[[你好，你有包皮过长情况吗？包皮...|[有包皮垢，发出味道，龟头敏感的太...|[16岁,男,会员96129630]|
|[[患者怀孕四十天，未见孕囊，一般...|[你好，怀孕40天，子宫未见孕囊，...|[23岁,女,会员96129628]|
|[[您好，最好的减肥方法是生活方式...|[想请问一下去中医院抓关于减肥的中...|[21岁,女,会员96129507]|
|[[一般这样的情况需要勤换衣服和洗...|[我只要一坐到凳子上屁股就发热，而...|[15岁,男,会员96129625]|
|[[首次来月经会和往常不一样，暂时...|[我产女五个月，哺乳期，但奶水很少...|[34岁,女,会员96129624]|
|[[你好 你这个情况你自己担心的太...|[医生您好，去年春季入夏也是4，5...|[26岁,男,会员96128857]|
+--------------------+--------------------+------------------+
only showing top 20 rows

scala> in
res15: org.apache.spark.sql.DataFrame = [answer: array<struct<answer_text:string,answer_time:string,mainpage:string,major:string,name:string,position:string>>, question: struct<question:string,time:string>, userinfo: struct<age:string,gender:string,name:string>]

scala>

目前的瓶颈应该是如何操作DataFrame，记录一下API的链接：
http://spark.apache.org/docs/1.3.0/api/java/org/apache/spark/sql/DataFrame.html
留着下次分析吧

附上启动的信息，感觉挺有用

miaofu@master:~/spark-1.6.2-bin-hadoop2.6$ spark-shell --master yarn-client
16/09/19 14:48:11 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
16/09/19 14:48:11 INFO SecurityManager: Changing view acls to: miaofu
16/09/19 14:48:11 INFO SecurityManager: Changing modify acls to: miaofu
16/09/19 14:48:11 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(miaofu); users with modify permissions: Set(miaofu)
16/09/19 14:48:12 INFO HttpServer: Starting HTTP Server
16/09/19 14:48:12 INFO Utils: Successfully started service 'HTTP class server' on port 36169.
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 1.6.2
/_/

Using Scala version 2.10.5 (OpenJDK 64-Bit Server VM, Java 1.7.0_95)
Type in expressions to have them evaluated.
Type :help for more information.
16/09/19 14:48:17 INFO SparkContext: Running Spark version 1.6.2
16/09/19 14:48:17 INFO SecurityManager: Changing view acls to: miaofu
16/09/19 14:48:17 INFO SecurityManager: Changing modify acls to: miaofu
16/09/19 14:48:17 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(miaofu); users with modify permissions: Set(miaofu)
16/09/19 14:48:17 INFO Utils: Successfully started service 'sparkDriver' on port 57910.
16/09/19 14:48:18 INFO Slf4jLogger: Slf4jLogger started
16/09/19 14:48:18 INFO Remoting: Starting remoting
16/09/19 14:48:18 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriverActorSystem@192.168.202.104:36129]
16/09/19 14:48:18 INFO Utils: Successfully started service 'sparkDriverActorSystem' on port 36129.
16/09/19 14:48:18 INFO SparkEnv: Registering MapOutputTracker
16/09/19 14:48:18 INFO SparkEnv: Registering BlockManagerMaster
16/09/19 14:48:18 INFO DiskBlockManager: Created local directory at /tmp/blockmgr-fc698a2d-8a1d-4357-902f-dc538549f645
16/09/19 14:48:18 INFO MemoryStore: MemoryStore started with capacity 511.5 MB
16/09/19 14:48:18 INFO SparkEnv: Registering OutputCommitCoordinator
16/09/19 14:48:18 INFO Utils: Successfully started service 'SparkUI' on port 4040.
16/09/19 14:48:18 INFO SparkUI: Started SparkUI at http://192.168.202.104:4040 16/09/19 14:48:18 WARN YarnClientSchedulerBackend: NOTE: SPARK_WORKER_MEMORY is deprecated. Use SPARK_EXECUTOR_MEMORY or --executor-memory through spark-submit instead.
16/09/19 14:48:19 INFO RMProxy: Connecting to ResourceManager at master/192.168.202.104:8032
16/09/19 14:48:31 INFO Client: Requesting a new application from cluster with 2 NodeManagers
16/09/19 14:48:31 INFO Client: Verifying our application has not requested more than the maximum memory capability of the cluster (8192 MB per container)
16/09/19 14:48:31 INFO Client: Will allocate AM container, with 896 MB memory including 384 MB overhead
16/09/19 14:48:31 INFO Client: Setting up container launch context for our AM
16/09/19 14:48:31 INFO Client: Setting up the launch environment for our AM container
16/09/19 14:48:31 INFO Client: Preparing resources for our AM container
16/09/19 14:48:32 INFO Client: Uploading resource file:/home/miaofu/spark-1.6.2-bin-hadoop2.6/lib/spark-assembly-1.6.2-hadoop2.6.0.jar -> hdfs://master:8020/user/miaofu/.sparkStaging/application_1474181125241_0005/spark-assembly-1.6.2-hadoop2.6.0.jar
16/09/19 14:48:48 INFO Client: Uploading resource file:/tmp/spark-16fdbdc6-16df-4159-958c-dcbb3783ed8a/__spark_conf__4841865900258042205.zip -> hdfs://master:8020/user/miaofu/.sparkStaging/application_1474181125241_0005/__spark_conf__4841865900258042205.zip
16/09/19 14:48:48 INFO SecurityManager: Changing view acls to: miaofu
16/09/19 14:48:48 INFO SecurityManager: Changing modify acls to: miaofu
16/09/19 14:48:48 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(miaofu); users with modify permissions: Set(miaofu)
16/09/19 14:48:48 INFO Client: Submitting application 5 to ResourceManager
16/09/19 14:48:56 INFO YarnClientImpl: Submitted application application_1474181125241_0005
16/09/19 14:48:57 INFO Client: Application report for application_1474181125241_0005 (state: ACCEPTED)
16/09/19 14:48:57 INFO Client:
client token: N/A
diagnostics: N/A
ApplicationMaster host: N/A
ApplicationMaster RPC port: -1
queue: default
start time: 1474267729613
final status: UNDEFINED
tracking URL: http://master:8088/proxy/application_1474181125241_0005/ user: miaofu
16/09/19 14:48:58 INFO Client: Application report for application_1474181125241_0005 (state: ACCEPTED)
16/09/19 14:48:59 INFO Client: Application report for application_1474181125241_0005 (state: ACCEPTED)
16/09/19 14:49:00 INFO Client: Application report for application_1474181125241_0005 (state: ACCEPTED)
16/09/19 14:49:01 INFO Client: Application report for application_1474181125241_0005 (state: ACCEPTED)
16/09/19 14:49:01 INFO YarnSchedulerBackend$YarnSchedulerEndpoint: ApplicationMaster registered as NettyRpcEndpointRef(null)
16/09/19 14:49:01 INFO YarnClientSchedulerBackend: Add WebUI Filter. org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter, Map(PROXY_HOSTS -> master, PROXY_URI_BASES -> http://master:8088/proxy/application_1474181125241_0005), /proxy/application_1474181125241_0005
16/09/19 14:49:01 INFO JettyUtils: Adding filter: org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter
16/09/19 14:49:02 INFO Client: Application report for application_1474181125241_0005 (state: RUNNING)
16/09/19 14:49:02 INFO Client:
client token: N/A
diagnostics: N/A
ApplicationMaster host: 192.168.202.105
ApplicationMaster RPC port: 0
queue: default
start time: 1474267729613
final status: UNDEFINED
tracking URL: http://master:8088/proxy/application_1474181125241_0005/ user: miaofu
16/09/19 14:49:02 INFO YarnClientSchedulerBackend: Application application_1474181125241_0005 has started running.
16/09/19 14:49:02 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 56322.
16/09/19 14:49:02 INFO NettyBlockTransferService: Server created on 56322
16/09/19 14:49:02 INFO BlockManagerMaster: Trying to register BlockManager
16/09/19 14:49:02 INFO BlockManagerMasterEndpoint: Registering block manager 192.168.202.104:56322 with 511.5 MB RAM, BlockManagerId(driver, 192.168.202.104, 56322)
16/09/19 14:49:02 INFO BlockManagerMaster: Registered BlockManager
16/09/19 14:49:02 INFO YarnClientSchedulerBackend: SchedulerBackend is ready for scheduling beginning after waiting maxRegisteredResourcesWaitingTime: 30000(ms)
16/09/19 14:49:02 INFO SparkILoop: Created spark context..
Spark context available as sc.
16/09/19 14:49:03 INFO HiveContext: Initializing execution hive, version 1.2.1
16/09/19 14:49:03 INFO ClientWrapper: Inspected Hadoop version: 2.6.0
16/09/19 14:49:03 INFO ClientWrapper: Loaded org.apache.hadoop.hive.shims.Hadoop23Shims for Hadoop version 2.6.0
16/09/19 14:49:03 INFO HiveMetaStore: 0: Opening raw store with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore
16/09/19 14:49:03 INFO ObjectStore: ObjectStore, initialize called
16/09/19 14:49:04 INFO Persistence: Property datanucleus.cache.level2 unknown - will be ignored
16/09/19 14:49:04 INFO Persistence: Property hive.metastore.integral.jdo.pushdown unknown - will be ignored
16/09/19 14:49:04 WARN Connection: BoneCP specified but not present in CLASSPATH (or one of dependencies)
16/09/19 14:49:04 WARN Connection: BoneCP specified but not present in CLASSPATH (or one of dependencies)
16/09/19 14:49:06 INFO YarnClientSchedulerBackend: Registered executor NettyRpcEndpointRef(null) (slave02:34081) with ID 2
16/09/19 14:49:06 INFO BlockManagerMasterEndpoint: Registering block manager slave02:56196 with 511.5 MB RAM, BlockManagerId(2, slave02, 56196)
16/09/19 14:49:09 INFO ObjectStore: Setting MetaStore object pin classes with hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order"
16/09/19 14:49:10 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as "embedded-only" so does not have its own datastore table.
16/09/19 14:49:10 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only" so does not have its own datastore table.
16/09/19 14:49:14 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as "embedded-only" so does not have its own datastore table.
16/09/19 14:49:14 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only" so does not have its own datastore table.
16/09/19 14:49:15 INFO MetaStoreDirectSql: Using direct SQL, underlying DB is DERBY
16/09/19 14:49:15 INFO ObjectStore: Initialized ObjectStore
16/09/19 14:49:15 WARN ObjectStore: Version information not found in metastore. hive.metastore.schema.verification is not enabled so recording the schema version 1.2.0
16/09/19 14:49:15 WARN ObjectStore: Failed to get database default, returning NoSuchObjectException
16/09/19 14:49:16 INFO HiveMetaStore: Added admin role in metastore
16/09/19 14:49:16 INFO HiveMetaStore: Added public role in metastore
16/09/19 14:49:16 INFO HiveMetaStore: No user is added in admin role, since config is empty
16/09/19 14:49:16 INFO HiveMetaStore: 0: get_all_databases
16/09/19 14:49:16 INFO audit: ugi=miaofu ip=unknown-ip-addr cmd=get_all_databases
16/09/19 14:49:16 INFO HiveMetaStore: 0: get_functions: db=default pat=*
16/09/19 14:49:16 INFO audit: ugi=miaofu ip=unknown-ip-addr cmd=get_functions: db=default pat=*
16/09/19 14:49:16 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MResourceUri" is tagged as "embedded-only" so does not have its own datastore table.
16/09/19 14:49:17 INFO SessionState: Created local directory: /tmp/9b870fb9-b54a-4e33-87cd-68bbaef7f497_resources
16/09/19 14:49:17 INFO SessionState: Created HDFS directory: /tmp/hive/miaofu/9b870fb9-b54a-4e33-87cd-68bbaef7f497
16/09/19 14:49:17 INFO SessionState: Created local directory: /tmp/miaofu/9b870fb9-b54a-4e33-87cd-68bbaef7f497
16/09/19 14:49:17 INFO SessionState: Created HDFS directory: /tmp/hive/miaofu/9b870fb9-b54a-4e33-87cd-68bbaef7f497/_tmp_space.db
16/09/19 14:49:17 INFO HiveContext: default warehouse location is /user/hive/warehouse
16/09/19 14:49:17 INFO HiveContext: Initializing HiveMetastoreConnection version 1.2.1 using Spark classes.
16/09/19 14:49:17 INFO ClientWrapper: Inspected Hadoop version: 2.6.0
16/09/19 14:49:17 INFO ClientWrapper: Loaded org.apache.hadoop.hive.shims.Hadoop23Shims for Hadoop version 2.6.0
16/09/19 14:49:18 INFO HiveMetaStore: 0: Opening raw store with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore
16/09/19 14:49:18 INFO ObjectStore: ObjectStore, initialize called
16/09/19 14:49:18 INFO Persistence: Property datanucleus.cache.level2 unknown - will be ignored
16/09/19 14:49:18 INFO Persistence: Property hive.metastore.integral.jdo.pushdown unknown - will be ignored
16/09/19 14:49:18 WARN Connection: BoneCP specified but not present in CLASSPATH (or one of dependencies)
16/09/19 14:49:18 WARN Connection: BoneCP specified but not present in CLASSPATH (or one of dependencies)
16/09/19 14:49:21 INFO ObjectStore: Setting MetaStore object pin classes with hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order"
16/09/19 14:49:22 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as "embedded-only" so does not have its own datastore table.
16/09/19 14:49:22 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only" so does not have its own datastore table.
16/09/19 14:49:23 INFO YarnClientSchedulerBackend: Registered executor NettyRpcEndpointRef(null) (master:53103) with ID 1
16/09/19 14:49:23 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as "embedded-only" so does not have its own datastore table.
16/09/19 14:49:23 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only" so does not have its own datastore table.
16/09/19 14:49:23 INFO BlockManagerMasterEndpoint: Registering block manager master:41990 with 511.5 MB RAM, BlockManagerId(1, master, 41990)
16/09/19 14:49:23 INFO Query: Reading in results for query "org.datanucleus.store.rdbms.query.SQLQuery@0" since the connection used is closing
16/09/19 14:49:23 INFO MetaStoreDirectSql: Using direct SQL, underlying DB is DERBY
16/09/19 14:49:23 INFO ObjectStore: Initialized ObjectStore
16/09/19 14:49:23 INFO HiveMetaStore: Added admin role in metastore
16/09/19 14:49:23 INFO HiveMetaStore: Added public role in metastore
16/09/19 14:49:23 INFO HiveMetaStore: No user is added in admin role, since config is empty
16/09/19 14:49:23 INFO HiveMetaStore: 0: get_all_databases
16/09/19 14:49:23 INFO audit: ugi=miaofu ip=unknown-ip-addr cmd=get_all_databases
16/09/19 14:49:23 INFO HiveMetaStore: 0: get_functions: db=default pat=*
16/09/19 14:49:23 INFO audit: ugi=miaofu ip=unknown-ip-addr cmd=get_functions: db=default pat=*
16/09/19 14:49:23 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MResourceUri" is tagged as "embedded-only" so does not have its own datastore table.
16/09/19 14:49:24 INFO SessionState: Created local directory: /tmp/1ed7cbd6-24a3-4f69-a504-85447022c5bb_resources
16/09/19 14:49:24 INFO SessionState: Created HDFS directory: /tmp/hive/miaofu/1ed7cbd6-24a3-4f69-a504-85447022c5bb
16/09/19 14:49:24 INFO SessionState: Created local directory: /tmp/miaofu/1ed7cbd6-24a3-4f69-a504-85447022c5bb
16/09/19 14:49:24 INFO SessionState: Created HDFS directory: /tmp/hive/miaofu/1ed7cbd6-24a3-4f69-a504-85447022c5bb/_tmp_space.db
16/09/19 14:49:24 INFO SparkILoop: Created sql context (with Hive support)..
SQL context available as sqlContext.


以及关闭的信息
scala> exit
warning: there were 1 deprecation warning(s); re-run with -deprecation for details
16/09/19 15:54:16 INFO SparkContext: Invoking stop() from shutdown hook
16/09/19 15:54:16 INFO SparkUI: Stopped Spark web UI at http://192.168.202.104:4040 16/09/19 15:54:16 INFO YarnClientSchedulerBackend: Shutting down all executors
16/09/19 15:54:16 INFO YarnClientSchedulerBackend: Interrupting monitor thread
16/09/19 15:54:16 INFO YarnClientSchedulerBackend: Asking each executor to shut down
16/09/19 15:54:16 INFO YarnClientSchedulerBackend: Stopped
16/09/19 15:54:16 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
16/09/19 15:54:17 INFO MemoryStore: MemoryStore cleared
16/09/19 15:54:17 INFO BlockManager: BlockManager stopped
16/09/19 15:54:17 INFO BlockManagerMaster: BlockManagerMaster stopped
16/09/19 15:54:17 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
16/09/19 15:54:17 INFO SparkContext: Successfully stopped SparkContext
16/09/19 15:54:17 INFO ShutdownHookManager: Shutdown hook called
16/09/19 15:54:17 INFO ShutdownHookManager: Deleting directory /tmp/spark-16fdbdc6-16df-4159-958c-dcbb3783ed8a
16/09/19 15:54:17 INFO ShutdownHookManager: Deleting directory /tmp/spark-af730c7f-169d-4ed4-9462-860f924ee006
16/09/19 15:54:17 INFO RemoteActorRefProvider$RemotingTerminator: Shutting down remote daemon.
16/09/19 15:54:17 INFO RemoteActorRefProvider$RemotingTerminator: Remote daemon shut down; proceeding with flushing remote transports.
16/09/19 15:54:17 INFO ShutdownHookManager: Deleting directory /tmp/spark-8db9dd25-5833-4b04-b0bf-f318d2336037