Scala学习2之spark学习6之scala版本不同的问题

http://plugins.jetbrains.com/plugin/?idea&id=1347

Spark之scala版本不同的问题：

用sbt编译有问题，但可以执行：

hadoop@Master:~/cloud/testByXubo/sh_spark_xubo/WordCountByTimeNoSort$ sbtRun.sh
[info] Set current project to helloworld project (in build file:/home/hadoop/cloud/testByXubo/sh_spark_xubo/WordCountByTimeNoSort/)
[info] Compiling 1 Scala source to /home/hadoop/cloud/testByXubo/sh_spark_xubo/WordCountByTimeNoSort/target/scala-2.10/classes...
[success] Total time: 7 s, completed Jan 29, 2016 10:24:54 PM
[info] Set current project to helloworld project (in build file:/home/hadoop/cloud/testByXubo/sh_spark_xubo/WordCountByTimeNoSort/)
[info] Packaging /home/hadoop/cloud/testByXubo/sh_spark_xubo/WordCountByTimeNoSort/target/scala-2.10/helloworld-project_2.10-1.0.jar ...
[info] Done packaging.
[success] Total time: 2 s, completed Jan 29, 2016 10:25:01 PM
[info] Set current project to helloworld project (in build file:/home/hadoop/cloud/testByXubo/sh_spark_xubo/WordCountByTimeNoSort/)
[info] Running WordCountByTimeNoSort
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
16/01/29 22:25:08 WARN SparkConf: null jar passed to SparkContext constructor
16/01/29 22:25:08 INFO SparkContext: Running Spark version 1.5.2
16/01/29 22:25:08 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
16/01/29 22:25:08 INFO SecurityManager: Changing view acls to: hadoop
16/01/29 22:25:08 INFO SecurityManager: Changing modify acls to: hadoop
16/01/29 22:25:08 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(hadoop); users with modify permissions: Set(hadoop)
16/01/29 22:25:09 INFO Slf4jLogger: Slf4jLogger started
16/01/29 22:25:09 INFO Remoting: Starting remoting
16/01/29 22:25:09 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriver@219.219.220.149:45087]
16/01/29 22:25:09 INFO Utils: Successfully started service 'sparkDriver' on port 45087.
16/01/29 22:25:09 INFO SparkEnv: Registering MapOutputTracker
16/01/29 22:25:09 INFO SparkEnv: Registering BlockManagerMaster
16/01/29 22:25:09 INFO DiskBlockManager: Created local directory at /tmp/blockmgr-4b725a45-2516-4de9-a231-2c4ebdf7221d
16/01/29 22:25:09 INFO MemoryStore: MemoryStore started with capacity 530.3 MB
16/01/29 22:25:09 INFO HttpFileServer: HTTP File server directory is /tmp/spark-b6c1a87a-7252-4429-8cc4-03516398246d/httpd-b9bda1d1-33a1-45bc-9352-0ace86e08ab4
16/01/29 22:25:09 INFO HttpServer: Starting HTTP Server
16/01/29 22:25:10 INFO Utils: Successfully started service 'HTTP file server' on port 34626.
16/01/29 22:25:10 INFO SparkEnv: Registering OutputCommitCoordinator
16/01/29 22:25:10 INFO Utils: Successfully started service 'SparkUI' on port 4040.
16/01/29 22:25:10 INFO SparkUI: Started SparkUI at http://219.219.220.149:4040 16/01/29 22:25:10 WARN MetricsSystem: Using default name DAGScheduler for source because spark.app.id is not set.
16/01/29 22:25:10 INFO Executor: Starting executor ID driver on host localhost
16/01/29 22:25:10 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 55909.
16/01/29 22:25:10 INFO NettyBlockTransferService: Server created on 55909
16/01/29 22:25:10 INFO BlockManagerMaster: Trying to register BlockManager
16/01/29 22:25:10 INFO BlockManagerMasterEndpoint: Registering block manager localhost:55909 with 530.3 MB RAM, BlockManagerId(driver, localhost, 55909)
16/01/29 22:25:10 INFO BlockManagerMaster: Registered BlockManager
16/01/29 22:25:11 INFO MemoryStore: ensureFreeSpace(106480) called with curMem=0, maxMem=556038881
16/01/29 22:25:11 INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 104.0 KB, free 530.2 MB)
16/01/29 22:25:11 INFO MemoryStore: ensureFreeSpace(10065) called with curMem=106480, maxMem=556038881
16/01/29 22:25:11 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 9.8 KB, free 530.2 MB)
16/01/29 22:25:11 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on localhost:55909 (size: 9.8 KB, free: 530.3 MB)
16/01/29 22:25:11 INFO SparkContext: Created broadcast 0 from textFile at WordCountByTimeNoSort.scala:26
16/01/29 22:25:12 INFO FileInputFormat: Total input paths to process : 4
16/01/29 22:25:12 INFO deprecation: mapred.tip.id is deprecated. Instead, use mapreduce.task.id
16/01/29 22:25:12 INFO deprecation: mapred.task.id is deprecated. Instead, use mapreduce.task.attempt.id
16/01/29 22:25:12 INFO deprecation: mapred.task.is.map is deprecated. Instead, use mapreduce.task.ismap
16/01/29 22:25:12 INFO deprecation: mapred.task.partition is deprecated. Instead, use mapreduce.task.partition
16/01/29 22:25:12 INFO deprecation: mapred.job.id is deprecated. Instead, use mapreduce.job.id
16/01/29 22:25:12 INFO SparkContext: Starting job: saveAsTextFile at WordCountByTimeNoSort.scala:30
16/01/29 22:25:12 INFO DAGScheduler: Registering RDD 3 (map at WordCountByTimeNoSort.scala:28)
16/01/29 22:25:12 INFO DAGScheduler: Got job 0 (saveAsTextFile at WordCountByTimeNoSort.scala:30) with 4 output partitions
16/01/29 22:25:12 INFO DAGScheduler: Final stage: ResultStage 1(saveAsTextFile at WordCountByTimeNoSort.scala:30)
16/01/29 22:25:12 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 0)
16/01/29 22:25:12 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 0)
16/01/29 22:25:12 INFO DAGScheduler: Submitting ShuffleMapStage 0 (MapPartitionsRDD[3] at map at WordCountByTimeNoSort.scala:28), which has no missing parents
16/01/29 22:25:12 INFO MemoryStore: ensureFreeSpace(4064) called with curMem=116545, maxMem=556038881
16/01/29 22:25:12 INFO MemoryStore: Block broadcast_1 stored as values in memory (estimated size 4.0 KB, free 530.2 MB)
16/01/29 22:25:12 INFO MemoryStore: ensureFreeSpace(2318) called with curMem=120609, maxMem=556038881
16/01/29 22:25:12 INFO MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 2.3 KB, free 530.2 MB)
16/01/29 22:25:12 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on localhost:55909 (size: 2.3 KB, free: 530.3 MB)
16/01/29 22:25:12 INFO SparkContext: Created broadcast 1 from broadcast at DAGScheduler.scala:861
16/01/29 22:25:12 INFO DAGScheduler: Submitting 4 missing tasks from ShuffleMapStage 0 (MapPartitionsRDD[3] at map at WordCountByTimeNoSort.scala:28)
16/01/29 22:25:12 INFO TaskSchedulerImpl: Adding task set 0.0 with 4 tasks
16/01/29 22:25:12 INFO TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, localhost, ANY, 2139 bytes)
16/01/29 22:25:12 INFO Executor: Running task 0.0 in stage 0.0 (TID 0)
16/01/29 22:25:12 INFO HadoopRDD: Input split: hdfs://219.219.220.149:9000/input/900.txt:0+478
16/01/29 22:25:12 INFO Executor: Finished task 0.0 in stage 0.0 (TID 0). 2256 bytes result sent to driver
16/01/29 22:25:12 INFO TaskSetManager: Starting task 1.0 in stage 0.0 (TID 1, localhost, ANY, 2158 bytes)
16/01/29 22:25:12 INFO Executor: Running task 1.0 in stage 0.0 (TID 1)
16/01/29 22:25:13 INFO HadoopRDD: Input split: hdfs://219.219.220.149:9000/input/in1/English_shashibiya.txt:0+4560
16/01/29 22:25:13 INFO TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) in 213 ms on localhost (1/4)
16/01/29 22:25:13 INFO Executor: Finished task 1.0 in stage 0.0 (TID 1). 2256 bytes result sent to driver
16/01/29 22:25:13 INFO TaskSetManager: Starting task 2.0 in stage 0.0 (TID 2, localhost, ANY, 2141 bytes)
16/01/29 22:25:13 INFO Executor: Running task 2.0 in stage 0.0 (TID 2)
16/01/29 22:25:13 INFO TaskSetManager: Finished task 1.0 in stage 0.0 (TID 1) in 253 ms on localhost (2/4)
16/01/29 22:25:13 INFO HadoopRDD: Input split: hdfs://219.219.220.149:9000/input/test1.txt:0+13
16/01/29 22:25:13 INFO Executor: Finished task 2.0 in stage 0.0 (TID 2). 2256 bytes result sent to driver
16/01/29 22:25:13 INFO TaskSetManager: Starting task 3.0 in stage 0.0 (TID 3, localhost, ANY, 2141 bytes)
16/01/29 22:25:13 INFO Executor: Running task 3.0 in stage 0.0 (TID 3)
16/01/29 22:25:13 INFO HadoopRDD: Input split: hdfs://219.219.220.149:9000/input/test2.txt:0+12
16/01/29 22:25:13 INFO TaskSetManager: Finished task 2.0 in stage 0.0 (TID 2) in 41 ms on localhost (3/4)
16/01/29 22:25:13 INFO Executor: Finished task 3.0 in stage 0.0 (TID 3). 2256 bytes result sent to driver
16/01/29 22:25:13 INFO TaskSetManager: Finished task 3.0 in stage 0.0 (TID 3) in 23 ms on localhost (4/4)
16/01/29 22:25:13 INFO DAGScheduler: ShuffleMapStage 0 (map at WordCountByTimeNoSort.scala:28) finished in 0.518 s
16/01/29 22:25:13 INFO DAGScheduler: looking for newly runnable stages
16/01/29 22:25:13 INFO TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool
16/01/29 22:25:13 INFO DAGScheduler: running: Set()
16/01/29 22:25:13 INFO DAGScheduler: waiting: Set(ResultStage 1)
16/01/29 22:25:13 INFO DAGScheduler: failed: Set()
16/01/29 22:25:13 INFO DAGScheduler: Missing parents for ResultStage 1: List()
16/01/29 22:25:13 INFO DAGScheduler: Submitting ResultStage 1 (MapPartitionsRDD[5] at saveAsTextFile at WordCountByTimeNoSort.scala:30), which is now runnable
16/01/29 22:25:13 INFO MemoryStore: ensureFreeSpace(95472) called with curMem=122927, maxMem=556038881
16/01/29 22:25:13 INFO MemoryStore: Block broadcast_2 stored as values in memory (estimated size 93.2 KB, free 530.1 MB)
16/01/29 22:25:13 INFO MemoryStore: ensureFreeSpace(31415) called with curMem=218399, maxMem=556038881
16/01/29 22:25:13 INFO MemoryStore: Block broadcast_2_piece0 stored as bytes in memory (estimated size 30.7 KB, free 530.0 MB)
16/01/29 22:25:13 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on localhost:55909 (size: 30.7 KB, free: 530.2 MB)
16/01/29 22:25:13 INFO SparkContext: Created broadcast 2 from broadcast at DAGScheduler.scala:861
16/01/29 22:25:13 INFO DAGScheduler: Submitting 4 missing tasks from ResultStage 1 (MapPartitionsRDD[5] at saveAsTextFile at WordCountByTimeNoSort.scala:30)
16/01/29 22:25:13 INFO TaskSchedulerImpl: Adding task set 1.0 with 4 tasks
16/01/29 22:25:13 INFO TaskSetManager: Starting task 0.0 in stage 1.0 (TID 4, localhost, PROCESS_LOCAL, 1901 bytes)
16/01/29 22:25:13 INFO Executor: Running task 0.0 in stage 1.0 (TID 4)
16/01/29 22:25:13 INFO deprecation: mapred.output.dir is deprecated. Instead, use mapreduce.output.fileoutputformat.outputdir
16/01/29 22:25:13 INFO deprecation: mapred.output.key.class is deprecated. Instead, use mapreduce.job.output.key.class
16/01/29 22:25:13 INFO deprecation: mapred.output.value.class is deprecated. Instead, use mapreduce.job.output.value.class
16/01/29 22:25:13 INFO deprecation: mapred.working.dir is deprecated. Instead, use mapreduce.job.working.dir
16/01/29 22:25:13 INFO ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 4 blocks
16/01/29 22:25:13 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 5 ms
16/01/29 22:25:13 INFO FileOutputCommitter: Saved output of task 'attempt_201601292225_0001_m_000000_4' to hdfs://219.219.220.149:9000/output/20160129222507987/_temporary/0/task_201601292225_0001_m_000000
16/01/29 22:25:13 INFO SparkHadoopMapRedUtil: attempt_201601292225_0001_m_000000_4: Committed
16/01/29 22:25:13 INFO Executor: Finished task 0.0 in stage 1.0 (TID 4). 1165 bytes result sent to driver
16/01/29 22:25:13 INFO TaskSetManager: Starting task 1.0 in stage 1.0 (TID 5, localhost, PROCESS_LOCAL, 1901 bytes)
16/01/29 22:25:13 INFO TaskSetManager: Finished task 0.0 in stage 1.0 (TID 4) in 283 ms on localhost (1/4)
16/01/29 22:25:13 INFO Executor: Running task 1.0 in stage 1.0 (TID 5)
16/01/29 22:25:13 INFO ShuffleBlockFetcherIterator: Getting 3 non-empty blocks out of 4 blocks
16/01/29 22:25:13 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/01/29 22:25:13 INFO FileOutputCommitter: Saved output of task 'attempt_201601292225_0001_m_000001_5' to hdfs://219.219.220.149:9000/output/20160129222507987/_temporary/0/task_201601292225_0001_m_000001
16/01/29 22:25:13 INFO SparkHadoopMapRedUtil: attempt_201601292225_0001_m_000001_5: Committed
16/01/29 22:25:13 INFO Executor: Finished task 1.0 in stage 1.0 (TID 5). 1165 bytes result sent to driver
16/01/29 22:25:13 INFO TaskSetManager: Starting task 2.0 in stage 1.0 (TID 6, localhost, PROCESS_LOCAL, 1901 bytes)
16/01/29 22:25:13 INFO TaskSetManager: Finished task 1.0 in stage 1.0 (TID 5) in 121 ms on localhost (2/4)
16/01/29 22:25:13 INFO Executor: Running task 2.0 in stage 1.0 (TID 6)
16/01/29 22:25:13 INFO ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks
16/01/29 22:25:13 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/01/29 22:25:13 INFO FileOutputCommitter: Saved output of task 'attempt_201601292225_0001_m_000002_6' to hdfs://219.219.220.149:9000/output/20160129222507987/_temporary/0/task_201601292225_0001_m_000002
16/01/29 22:25:13 INFO SparkHadoopMapRedUtil: attempt_201601292225_0001_m_000002_6: Committed
16/01/29 22:25:13 INFO Executor: Finished task 2.0 in stage 1.0 (TID 6). 1165 bytes result sent to driver
16/01/29 22:25:13 INFO TaskSetManager: Starting task 3.0 in stage 1.0 (TID 7, localhost, PROCESS_LOCAL, 1901 bytes)
16/01/29 22:25:13 INFO TaskSetManager: Finished task 2.0 in stage 1.0 (TID 6) in 109 ms on localhost (3/4)
16/01/29 22:25:13 INFO Executor: Running task 3.0 in stage 1.0 (TID 7)
16/01/29 22:25:13 INFO ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 4 blocks
16/01/29 22:25:13 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/01/29 22:25:14 INFO FileOutputCommitter: Saved output of task 'attempt_201601292225_0001_m_000003_7' to hdfs://219.219.220.149:9000/output/20160129222507987/_temporary/0/task_201601292225_0001_m_000003
16/01/29 22:25:14 INFO SparkHadoopMapRedUtil: attempt_201601292225_0001_m_000003_7: Committed
16/01/29 22:25:14 INFO Executor: Finished task 3.0 in stage 1.0 (TID 7). 1165 bytes result sent to driver
16/01/29 22:25:14 INFO TaskSetManager: Finished task 3.0 in stage 1.0 (TID 7) in 110 ms on localhost (4/4)
16/01/29 22:25:14 INFO DAGScheduler: ResultStage 1 (saveAsTextFile at WordCountByTimeNoSort.scala:30) finished in 0.617 s
16/01/29 22:25:14 INFO TaskSchedulerImpl: Removed TaskSet 1.0, whose tasks have all completed, from pool
16/01/29 22:25:14 INFO DAGScheduler: Job 0 finished: saveAsTextFile at WordCountByTimeNoSort.scala:30, took 1.341520 s
success
16/01/29 22:25:14 ERROR Utils: uncaught error in thread SparkListenerBus, stopping SparkContext
java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:996)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1303)
at java.util.concurrent.Semaphore.acquire(Semaphore.java:317)
at org.apache.spark.util.AsynchronousListenerBus$$anon$1$$anonfun$run$1.apply$mcV$sp(AsynchronousListenerBus.scala:65)
at org.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1136)
at org.apache.spark.util.AsynchronousListenerBus$$anon$1.run(AsynchronousListenerBus.scala:63)
16/01/29 22:25:14 ERROR ContextCleaner: Error in cleaning thread
java.lang.InterruptedException
at java.lang.Object.wait(Native Method)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135)
at org.apache.spark.ContextCleaner$$anonfun$org$apache$spark$ContextCleaner$$keepCleaning$1.apply$mcV$sp(ContextCleaner.scala:157)
at org.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1136)
at org.apache.spark.ContextCleaner.org$apache$spark$ContextCleaner$$keepCleaning(ContextCleaner.scala:154)
at org.apache.spark.ContextCleaner$$anon$3.run(ContextCleaner.scala:67)
16/01/29 22:25:14 INFO SparkUI: Stopped Spark web UI at http://219.219.220.149:4040 16/01/29 22:25:14 INFO BlockManagerInfo: Removed broadcast_0_piece0 on localhost:55909 in memory (size: 9.8 KB, free: 530.2 MB)
16/01/29 22:25:14 INFO DAGScheduler: Stopping DAGScheduler
[success] Total time: 9 s, completed Jan 29, 2016 10:25:15 PM
16/01/29 22:25:15 INFO DiskBlockManager: Shutdown hook called
16/01/29 22:25:15 INFO ShutdownHookManager: Shutdown hook called
16/01/29 22:25:15 INFO ShutdownHookManager: Deleting directory /tmp/spark-b6c1a87a-7252-4429-8cc4-03516398246d/userFiles-48b939eb-9bd7-4fe5-9916-87d786bf2d81
16/01/29 22:25:15 INFO ShutdownHookManager: Deleting directory /tmp/spark-b6c1a87a-7252-4429-8cc4-03516398246d
hadoop@Master:~/cloud/testByXubo/sh_spark_xubo/WordCountByTimeNoSort$ ./submitJob.sh
success

从eclipse中导不出jar包，因为：

JAR creation failed. See details for additional information.
Class files on classpath not found or not accessible for: 'SparkScala/src/org/apache/spark/examples/ByXubo/WordCountByTimeNoSort.scala'

JAR creation failed. See details for additional information.
Class files on classpath not found or not accessible for: 'SparkScala/src/TestDate.scala'

新建scala的project中也报错：

Description	Resource	Path	Location	Type
More than one scala library found in the build path (G:/149/spark-assembly-1.5.2-hadoop2.6.0.jar, D:/1win7/eclipse/plugins/org.scala-lang.scala-library_2.11.7.v20150622-112736-1fbce4612c.jar).At least one has an incompatible version. Please update the project build path so it contains only one compatible scala library.	SparkScala		Unknown	Scala Classpath Problem

其中主要是java的date等class的问题：

package test2
import java.util._;
import java.text.SimpleDateFormat

object TestDate {
def main(args: Array[String]) {
//
//  val sdf = new SimpleDateFormat("yyyy-MM-dd H:mm:ss")
//
val iString=new SimpleDateFormat("yyyyMMddHHmmssSSS").format(new Date() );
val s1="hdfs://219.219.220.149:9000/output/"+iString;
println(s1)
val s0= "hdfs://219.219.220.149:9000/input/*"
println(s0)
}
}

//class can not export jar

未解决：