windows环境下IDEA运行spark程序出现的异常问题
2017-05-19 14:40
651 查看
spark程序如下
运行时一直不出结果，控制台打印信息如下
"C:\Program Files\Java\jdk1.8.0_131\bin\java" "-javaagent:C:\NoWinProgram\IntelliJ IDEA Community Edition 2017.1\lib\idea_rt.jar=7435:C:\NoWinProgram\IntelliJ IDEA Community Edition 2017.1\bin" -Dfile.encoding=UTF-8 -classpath "C:\Program Files\Java\jdk1.8.0_131\jre\lib\charsets.jar;C:\Program
Files\Java\jdk1.8.0_131\jre\lib\deploy.jar;C:\Program Files\Java\jdk1.8.0_131\jre\lib\ext\access-bridge-64.jar;C:\Program Files\Java\jdk1.8.0_131\jre\lib\ext\cldrdata.jar;C:\Program Files\Java\jdk1.8.0_131\jre\lib\ext\dnsns.jar;C:\Program Files\Java\jdk1.8.0_131\jre\lib\ext\jaccess.jar;C:\Program
Files\Java\jdk1.8.0_131\jre\lib\ext\jfxrt.jar;C:\Program Files\Java\jdk1.8.0_131\jre\lib\ext\localedata.jar;C:\Program Files\Java\jdk1.8.0_131\jre\lib\ext\nashorn.jar;C:\Program Files\Java\jdk1.8.0_131\jre\lib\ext\sunec.jar;C:\Program Files\Java\jdk1.8.0_131\jre\lib\ext\sunjce_provider.jar;C:\Program
Files\Java\jdk1.8.0_131\jre\lib\ext\sunmscapi.jar;C:\Program Files\Java\jdk1.8.0_131\jre\lib\ext\sunpkcs11.jar;C:\Program Files\Java\jdk1.8.0_131\jre\lib\ext\zipfs.jar;C:\Program Files\Java\jdk1.8.0_131\jre\lib\javaws.jar;C:\Program Files\Java\jdk1.8.0_131\jre\lib\jce.jar;C:\Program
Files\Java\jdk1.8.0_131\jre\lib\jfr.jar;C:\Program Files\Java\jdk1.8.0_131\jre\lib\jfxswt.jar;C:\Program Files\Java\jdk1.8.0_131\jre\lib\jsse.jar;C:\Program Files\Java\jdk1.8.0_131\jre\lib\management-agent.jar;C:\Program Files\Java\jdk1.8.0_131\jre\lib\plugin.jar;C:\Program
Files\Java\jdk1.8.0_131\jre\lib\resources.jar;C:\Program Files\Java\jdk1.8.0_131\jre\lib\rt.jar;D:\scala-idea-workspace\cars\out\production\cars;C:\Program Files (x86)\scala\lib\scala-actors-migration.jar;C:\Program Files (x86)\scala\lib\scala-actors.jar;C:\Program
Files (x86)\scala\lib\scala-library.jar;C:\Program Files (x86)\scala\lib\scala-reflect.jar;C:\Program Files (x86)\scala\lib\scala-swing.jar;C:\Program Files (x86)\scala\src\scala-actors-src.jar;C:\Program Files (x86)\scala\src\scala-library-src.jar;C:\Program
Files (x86)\scala\src\scala-reflect-src.jar;C:\Program Files (x86)\scala\src\scala-swing-src.jar;F:\DownLoad\mysql-connector-java-5.1.42\mysql-connector-java-5.1.42\mysql-connector-java-5.1.42-bin.jar;F:\DownLoad\spark-assembly-1.6.1-hadoop2.6.0.jar" HelloWorld
Hello World
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
17/05/19 14:32:48 INFO SparkContext: Running Spark version 1.6.1
17/05/19 14:32:48 INFO SecurityManager: Changing view acls to: Philon Yun,hadoop
17/05/19 14:32:48 INFO SecurityManager: Changing modify acls to: Philon Yun,hadoop
17/05/19 14:32:48 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(Philon Yun, hadoop); users with modify permissions: Set(Philon Yun, hadoop)
17/05/19 14:32:48 INFO Utils: Successfully started service 'sparkDriver' on port 7472.
17/05/19 14:32:49 INFO Slf4jLogger: Slf4jLogger started
17/05/19 14:32:49 INFO Remoting: Starting remoting
17/05/19 14:32:49 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriverActorSystem@169.254.4.104:7485]
17/05/19 14:32:49 INFO Utils: Successfully started service 'sparkDriverActorSystem' on port 7485.
17/05/19 14:32:49 INFO SparkEnv: Registering MapOutputTracker
17/05/19 14:32:49 INFO SparkEnv: Registering BlockManagerMaster
17/05/19 14:32:49 INFO DiskBlockManager: Created local directory at C:\Users\Philon Yun\AppData\Local\Temp\blockmgr-60dcf5d0-c40b-443b-892d-7f7a5d2e4183
17/05/19 14:32:49 INFO MemoryStore: MemoryStore started with capacity 2.4 GB
17/05/19 14:32:49 INFO SparkEnv: Registering OutputCommitCoordinator
17/05/19 14:32:49 INFO Utils: Successfully started service 'SparkUI' on port 4040.
17/05/19 14:32:49 INFO SparkUI: Started SparkUI at http://169.254.4.104:4040
17/05/19 14:32:49 INFO AppClient$ClientEndpoint: Connecting to master spark://192.168.11.221:7077...
17/05/19 14:32:49 INFO SparkDeploySchedulerBackend: Connected to Spark cluster with app ID app-20170519143251-0005
17/05/19 14:32:49 INFO AppClient$ClientEndpoint: Executor added: app-20170519143251-0005/0 on worker-20170519135622-192.168.11.222-39133 (192.168.11.222:39133) with 1 cores
17/05/19 14:32:49 INFO SparkDeploySchedulerBackend: Granted executor ID app-20170519143251-0005/0 on hostPort 192.168.11.222:39133 with 1 cores, 1024.0 MB RAM
17/05/19 14:32:49 INFO AppClient$ClientEndpoint: Executor added: app-20170519143251-0005/1 on worker-20170519135622-192.168.11.223-37116 (192.168.11.223:37116) with 1 cores
17/05/19 14:32:49 INFO SparkDeploySchedulerBackend: Granted executor ID app-20170519143251-0005/1 on hostPort 192.168.11.223:37116 with 1 cores, 1024.0 MB RAM
17/05/19 14:32:49 INFO AppClient$ClientEndpoint: Executor updated: app-20170519143251-0005/0 is now RUNNING
17/05/19 14:32:49 INFO AppClient$ClientEndpoint: Executor updated: app-20170519143251-0005/1 is now RUNNING
17/05/19 14:32:49 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 7523.
17/05/19 14:32:49 INFO NettyBlockTransferService: Server created on 7523
17/05/19 14:32:49 INFO BlockManagerMaster: Trying to register BlockManager
17/05/19 14:32:49 INFO BlockManagerMasterEndpoint: Registering block manager 169.254.4.104:7523 with 2.4 GB RAM, BlockManagerId(driver, 169.254.4.104, 7523)
17/05/19 14:32:49 INFO BlockManagerMaster: Registered BlockManager
17/05/19 14:32:49 INFO SparkDeploySchedulerBackend: SchedulerBackend is ready for scheduling beginning after reached minRegisteredResourcesRatio: 0.0
17/05/19 14:32:49 INFO HttpFileServer: HTTP File server directory is C:\Users\Philon Yun\AppData\Local\Temp\spark-78170f53-6988-49c6-b17c-42e6544272d0\httpd-d1f185b3-75b9-4b12-becf-fc079368e8ac
17/05/19 14:32:49 INFO HttpServer: Starting HTTP Server
17/05/19 14:32:49 INFO Utils: Successf
c314
ully started service 'HTTP file server' on port 7524.
17/05/19 14:32:50 INFO SparkContext: Added JAR D:\scala-idea-workspace\cars\out\artifacts\cars_jar\cars.jar at http://169.254.4.104:7524/jars/cars.jar with timestamp 1495175570735
17/05/19 14:32:51 INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 127.4 KB, free 127.4 KB)
17/05/19 14:32:51 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 13.9 KB, free 141.3 KB)
17/05/19 14:32:51 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on 169.254.4.104:7523 (size: 13.9 KB, free: 2.4 GB)
17/05/19 14:32:51 INFO SparkContext: Created broadcast 0 from textFile at HelloWorld.scala:18
17/05/19 14:32:52 INFO FileInputFormat: Total input paths to process : 1
17/05/19 14:32:52 INFO SparkContext: Starting job: json at HelloWorld.scala:25
17/05/19 14:32:52 INFO DAGScheduler: Got job 0 (json at HelloWorld.scala:25) with 2 output partitions
17/05/19 14:32:52 INFO DAGScheduler: Final stage: ResultStage 0 (json at HelloWorld.scala:25)
17/05/19 14:32:52 INFO DAGScheduler: Parents of final stage: List()
17/05/19 14:32:52 INFO DAGScheduler: Missing parents: List()
17/05/19 14:32:53 INFO DAGScheduler: Submitting ResultStage 0 (MapPartitionsRDD[9] at json at HelloWorld.scala:25), which has no missing parents
17/05/19 14:32:53 INFO MemoryStore: Block broadcast_1 stored as values in memory (estimated size 7.6 KB, free 148.9 KB)
17/05/19 14:32:53 INFO MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 3.9 KB, free 152.8 KB)
17/05/19 14:32:53 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on 169.254.4.104:7523 (size: 3.9 KB, free: 2.4 GB)
17/05/19 14:32:53 INFO SparkContext: Created broadcast 1 from broadcast at DAGScheduler.scala:1006
17/05/19 14:32:53 INFO DAGScheduler: Submitting 2 missing tasks from ResultStage 0 (MapPartitionsRDD[9] at json at HelloWorld.scala:25)
17/05/19 14:32:53 INFO TaskSchedulerImpl: Adding task set 0.0 with 2 tasks
17/05/19 14:32:55 INFO AppClient$ClientEndpoint: Executor updated: app-20170519143251-0005/1 is now EXITED (Command exited with code 1)
17/05/19 14:32:55 INFO SparkDeploySchedulerBackend: Executor app-20170519143251-0005/1 removed: Command exited with code 1
17/05/19 14:32:55 INFO SparkDeploySchedulerBackend: Asked to remove non-existent executor 1
17/05/19 14:32:55 INFO AppClient$ClientEndpoint: Executor added: app-20170519143251-0005/2 on worker-20170519135622-192.168.11.223-37116 (192.168.11.223:37116) with 1 cores
17/05/19 14:32:55 INFO SparkDeploySchedulerBackend: Granted executor ID app-20170519143251-0005/2 on hostPort 192.168.11.223:37116 with 1 cores, 1024.0 MB RAM
17/05/19 14:32:55 INFO AppClient$ClientEndpoint: Executor updated: app-20170519143251-0005/2 is now RUNNING
17/05/19 14:32:55 INFO AppClient$ClientEndpoint: Executor updated: app-20170519143251-0005/0 is now EXITED (Command exited with code 1)
17/05/19 14:32:55 INFO SparkDeploySchedulerBackend: Executor app-20170519143251-0005/0 removed: Command exited with code 1
17/05/19 14:32:55 INFO SparkDeploySchedulerBackend: Asked to remove non-existent executor 0
17/05/19 14:32:55 INFO AppClient$ClientEndpoint: Executor added: app-20170519143251-0005/3 on worker-20170519135622-192.168.11.222-39133 (192.168.11.222:39133) with 1 cores
17/05/19 14:32:55 INFO SparkDeploySchedulerBackend: Granted executor ID app-20170519143251-0005/3 on hostPort 192.168.11.222:39133 with 1 cores, 1024.0 MB RAM
17/05/19 14:32:55 INFO AppClient$ClientEndpoint: Executor updated: app-20170519143251-0005/3 is now RUNNING
17/05/19 14:32:58 INFO AppClient$ClientEndpoint: Executor updated: app-20170519143251-0005/3 is now EXITED (Command exited with code 1)
17/05/19 14:32:58 INFO SparkDeploySchedulerBackend: Executor app-20170519143251-0005/3 removed: Command exited with code 1
17/05/19 14:32:58 INFO SparkDeploySchedulerBackend: Asked to remove non-existent executor 3
17/05/19 14:32:58 INFO AppClient$ClientEndpoint: Executor added: app-20170519143251-0005/4 on worker-20170519135622-192.168.11.222-39133 (192.168.11.222:39133) with 1 cores
17/05/19 14:32:58 INFO SparkDeploySchedulerBackend: Granted executor ID app-20170519143251-0005/4 on hostPort 192.168.11.222:39133 with 1 cores, 1024.0 MB RAM
17/05/19 14:32:58 INFO AppClient$ClientEndpoint: Executor updated: app-20170519143251-0005/4 is now RUNNING
17/05/19 14:32:58 INFO AppClient$ClientEndpoint: Executor updated: app-20170519143251-0005/2 is now EXITED (Command exited with code 1)
17/05/19 14:32:58 INFO SparkDeploySchedulerBackend: Executor app-20170519143251-0005/2 removed: Command exited with code 1
17/05/19 14:32:58 INFO SparkDeploySchedulerBackend: Asked to remove non-existent executor 2
17/05/19 14:32:58 INFO AppClient$ClientEndpoint: Executor added: app-20170519143251-0005/5 on worker-20170519135622-192.168.11.223-37116 (192.168.11.223:37116) with 1 cores
17/05/19 14:32:58 INFO SparkDeploySchedulerBackend: Granted executor ID app-20170519143251-0005/5 on hostPort 192.168.11.223:37116 with 1 cores, 1024.0 MB RAM
17/05/19 14:32:58 INFO AppClient$ClientEndpoint: Executor updated: app-20170519143251-0005/5 is now RUNNING
17/05/19 14:33:03 INFO AppClient$ClientEndpoint: Executor updated: app-20170519143251-0005/5 is now EXITED (Command exited with code 1)
17/05/19 14:33:03 INFO SparkDeploySchedulerBackend: Executor app-20170519143251-0005/5 removed: Command exited with code 1
17/05/19 14:33:03 INFO SparkDeploySchedulerBackend: Asked to remove non-existent executor 5
17/05/19 14:33:03 INFO AppClient$ClientEndpoint: Executor added: app-20170519143251-0005/6 on worker-20170519135622-192.168.11.223-37116 (192.168.11.223:37116) with 1 cores
17/05/19 14:33:03 INFO SparkDeploySchedulerBackend: Granted executor ID app-20170519143251-0005/6 on hostPort 192.168.11.223:37116 with 1 cores, 1024.0 MB RAM
17/05/19 14:33:03 INFO AppClient$ClientEndpoint: Executor updated: app-20170519143251-0005/6 is now RUNNING
17/05/19 14:33:06 INFO AppClient$ClientEndpoint: Executor updated: app-20170519143251-0005/4 is now EXITED (Command exited with code 1)
17/05/19 14:33:06 INFO SparkDeploySchedulerBackend: Executor app-20170519143251-0005/4 removed: Command exited with code 1
17/05/19 14:33:06 INFO SparkDeploySchedulerBackend: Asked to remove non-existent executor 4
17/05/19 14:33:06 INFO AppClient$ClientEndpoint: Executor added: app-20170519143251-0005/7 on worker-20170519135622-192.168.11.222-39133 (192.168.11.222:39133) with 1 cores
17/05/19 14:33:06 INFO SparkDeploySchedulerBackend: Granted executor ID app-20170519143251-0005/7 on hostPort 192.168.11.222:39133 with 1 cores, 1024.0 MB RAM
17/05/19 14:33:06 INFO AppClient$ClientEndpoint: Executor updated: app-20170519143251-0005/7 is now RUNNING
17/05/19 14:33:08 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
17/05/19 14:33:09 INFO AppClient$ClientEndpoint: Executor updated: app-20170519143251-0005/7 is now EXITED (Command exited with code 1)
17/05/19 14:33:09 INFO SparkDeploySchedulerBackend: Executor app-20170519143251-0005/7 removed: Command exited with code 1
17/05/19 14:33:09 INFO SparkDeploySchedulerBackend: Asked to remove non-existent executor 7
17/05/19 14:33:09 INFO AppClient$ClientEndpoint: Executor added: app-20170519143251-0005/8 on worker-20170519135622-192.168.11.222-39133 (192.168.11.222:39133) with 1 cores
17/05/19 14:33:09 INFO SparkDeploySchedulerBackend: Granted executor ID app-20170519143251-0005/8 on hostPort 192.168.11.222:39133 with 1 cores, 1024.0 MB RAM
17/05/19 14:33:09 INFO AppClient$ClientEndpoint: Executor updated: app-20170519143251-0005/8 is now RUNNING
17/05/19 14:33:12 INFO AppClient$ClientEndpoint: Executor updated: app-20170519143251-0005/6 is now EXITED (Command exited with code 1)
17/05/19 14:33:12 INFO SparkDeploySchedulerBackend: Executor app-20170519143251-0005/6 removed: Command exited with code 1
17/05/19 14:33:12 INFO SparkDeploySchedulerBackend: Asked to remove non-existent executor 6
17/05/19 14:33:12 INFO AppClient$ClientEndpoint: Executor added: app-20170519143251-0005/9 on worker-20170519135622-192.168.11.223-37116 (192.168.11.223:37116) with 1 cores
17/05/19 14:33:12 INFO SparkDeploySchedulerBackend: Granted executor ID app-20170519143251-0005/9 on hostPort 192.168.11.223:37116 with 1 cores, 1024.0 MB RAM
17/05/19 14:33:12 INFO AppClient$ClientEndpoint: Executor updated: app-20170519143251-0005/9 is now RUNNING
17/05/19 14:33:17 INFO AppClient$ClientEndpoint: Executor updated: app-20170519143251-0005/8 is now EXITED (Command exited with code 1)
17/05/19 14:33:17 INFO SparkDeploySchedulerBackend: Executor app-20170519143251-0005/8 removed: Command exited with code 1
17/05/19 14:33:17 INFO SparkDeploySchedulerBackend: Asked to remove non-existent executor 8
17/05/19 14:33:17 INFO AppClient$ClientEndpoint: Executor added: app-20170519143251-0005/10 on worker-20170519135622-192.168.11.222-39133 (192.168.11.222:39133) with 1 cores
17/05/19 14:33:17 INFO SparkDeploySchedulerBackend: Granted executor ID app-20170519143251-0005/10 on hostPort 192.168.11.222:39133 with 1 cores, 1024.0 MB RAM
17/05/19 14:33:17 INFO AppClient$ClientEndpoint: Executor updated: app-20170519143251-0005/10 is now RUNNING
17/05/19 14:33:20 INFO AppClient$ClientEndpoint: Executor updated: app-20170519143251-0005/9 is now EXITED (Command exited with code 1)
17/05/19 14:33:20 INFO SparkDeploySchedulerBackend: Executor app-20170519143251-0005/9 removed: Command exited with code 1
17/05/19 14:33:20 INFO SparkDeploySchedulerBackend: Asked to remove non-existent executor 9
17/05/19 14:33:20 INFO AppClient$ClientEndpoint: Executor added: app-20170519143251-0005/11 on worker-20170519135622-192.168.11.223-37116 (192.168.11.223:37116) with 1 cores
17/05/19 14:33:20 INFO SparkDeploySchedulerBackend: Granted executor ID app-20170519143251-0005/11 on hostPort 192.168.11.223:37116 with 1 cores, 1024.0 MB RAM
17/05/19 14:33:20 INFO AppClient$ClientEndpoint: Executor updated: app-20170519143251-0005/11 is now RUNNING
这一行特别醒目
WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
初始工作没有接受任何资源；检查你的集群用户界面以确保工人注册并有足够的资源
我在网上查了很多，说内存不足的，节点间通信问题的，其实我的集群设置是没问题的，而且有足够的内存用
这是我的提交任务后的集群信息：
最终我找到了解决办法就是在windows环境变量中添加一个环境变量 SPARK_LOCAL_IP
在cmd中运行ipconfig命令查看自己的ip地址，然后赋值给这个环境变量，然后重启IDEA重新提交运行任务就可以解决这个问题。
/**
* Created by Philon Yun on 2017/5/15.
*/
import org.apache.spark.SparkConf
import org.apache.spark.SparkContext
import org.apache.spark.SparkContext._
import org.apache.spark.sql.SQLContext
object HelloWorld {
def main(args: Array[String]): Unit = {
println("Hello World")
val conf = new SparkConf().setMaster("spark://192.168.11.221:7077").setAppName("MyFitst")
val sc = new SparkContext(conf)
sc.addJar("D:\\scala-idea-workspace\\cars\\out\\artifacts\\cars_jar\\cars.jar")
val sqlContext = new SQLContext(sc)
val rdd = sc.textFile("hdfs://master:9000/person.txt").map(_.split(","))
val personRDD = rdd.map(x => Person(x(0).toLong,x(1),x(2).toInt))
import sqlContext.implicits._
val personDF = personRDD.toDF
personDF.registerTempTable("t_person")
val df = sqlContext.sql("select * from t_person order by age desc ")
df.write.json("hdfs://master:9000/out2")
sc.stop()
}
}
case class Person(id:Long,name:String,age:Int)运行时一直不出结果，控制台打印信息如下
"C:\Program Files\Java\jdk1.8.0_131\bin\java" "-javaagent:C:\NoWinProgram\IntelliJ IDEA Community Edition 2017.1\lib\idea_rt.jar=7435:C:\NoWinProgram\IntelliJ IDEA Community Edition 2017.1\bin" -Dfile.encoding=UTF-8 -classpath "C:\Program Files\Java\jdk1.8.0_131\jre\lib\charsets.jar;C:\Program
Files\Java\jdk1.8.0_131\jre\lib\deploy.jar;C:\Program Files\Java\jdk1.8.0_131\jre\lib\ext\access-bridge-64.jar;C:\Program Files\Java\jdk1.8.0_131\jre\lib\ext\cldrdata.jar;C:\Program Files\Java\jdk1.8.0_131\jre\lib\ext\dnsns.jar;C:\Program Files\Java\jdk1.8.0_131\jre\lib\ext\jaccess.jar;C:\Program
Files\Java\jdk1.8.0_131\jre\lib\ext\jfxrt.jar;C:\Program Files\Java\jdk1.8.0_131\jre\lib\ext\localedata.jar;C:\Program Files\Java\jdk1.8.0_131\jre\lib\ext\nashorn.jar;C:\Program Files\Java\jdk1.8.0_131\jre\lib\ext\sunec.jar;C:\Program Files\Java\jdk1.8.0_131\jre\lib\ext\sunjce_provider.jar;C:\Program
Files\Java\jdk1.8.0_131\jre\lib\ext\sunmscapi.jar;C:\Program Files\Java\jdk1.8.0_131\jre\lib\ext\sunpkcs11.jar;C:\Program Files\Java\jdk1.8.0_131\jre\lib\ext\zipfs.jar;C:\Program Files\Java\jdk1.8.0_131\jre\lib\javaws.jar;C:\Program Files\Java\jdk1.8.0_131\jre\lib\jce.jar;C:\Program
Files\Java\jdk1.8.0_131\jre\lib\jfr.jar;C:\Program Files\Java\jdk1.8.0_131\jre\lib\jfxswt.jar;C:\Program Files\Java\jdk1.8.0_131\jre\lib\jsse.jar;C:\Program Files\Java\jdk1.8.0_131\jre\lib\management-agent.jar;C:\Program Files\Java\jdk1.8.0_131\jre\lib\plugin.jar;C:\Program
Files\Java\jdk1.8.0_131\jre\lib\resources.jar;C:\Program Files\Java\jdk1.8.0_131\jre\lib\rt.jar;D:\scala-idea-workspace\cars\out\production\cars;C:\Program Files (x86)\scala\lib\scala-actors-migration.jar;C:\Program Files (x86)\scala\lib\scala-actors.jar;C:\Program
Files (x86)\scala\lib\scala-library.jar;C:\Program Files (x86)\scala\lib\scala-reflect.jar;C:\Program Files (x86)\scala\lib\scala-swing.jar;C:\Program Files (x86)\scala\src\scala-actors-src.jar;C:\Program Files (x86)\scala\src\scala-library-src.jar;C:\Program
Files (x86)\scala\src\scala-reflect-src.jar;C:\Program Files (x86)\scala\src\scala-swing-src.jar;F:\DownLoad\mysql-connector-java-5.1.42\mysql-connector-java-5.1.42\mysql-connector-java-5.1.42-bin.jar;F:\DownLoad\spark-assembly-1.6.1-hadoop2.6.0.jar" HelloWorld
Hello World
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
17/05/19 14:32:48 INFO SparkContext: Running Spark version 1.6.1
17/05/19 14:32:48 INFO SecurityManager: Changing view acls to: Philon Yun,hadoop
17/05/19 14:32:48 INFO SecurityManager: Changing modify acls to: Philon Yun,hadoop
17/05/19 14:32:48 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(Philon Yun, hadoop); users with modify permissions: Set(Philon Yun, hadoop)
17/05/19 14:32:48 INFO Utils: Successfully started service 'sparkDriver' on port 7472.
17/05/19 14:32:49 INFO Slf4jLogger: Slf4jLogger started
17/05/19 14:32:49 INFO Remoting: Starting remoting
17/05/19 14:32:49 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriverActorSystem@169.254.4.104:7485]
17/05/19 14:32:49 INFO Utils: Successfully started service 'sparkDriverActorSystem' on port 7485.
17/05/19 14:32:49 INFO SparkEnv: Registering MapOutputTracker
17/05/19 14:32:49 INFO SparkEnv: Registering BlockManagerMaster
17/05/19 14:32:49 INFO DiskBlockManager: Created local directory at C:\Users\Philon Yun\AppData\Local\Temp\blockmgr-60dcf5d0-c40b-443b-892d-7f7a5d2e4183
17/05/19 14:32:49 INFO MemoryStore: MemoryStore started with capacity 2.4 GB
17/05/19 14:32:49 INFO SparkEnv: Registering OutputCommitCoordinator
17/05/19 14:32:49 INFO Utils: Successfully started service 'SparkUI' on port 4040.
17/05/19 14:32:49 INFO SparkUI: Started SparkUI at http://169.254.4.104:4040
17/05/19 14:32:49 INFO AppClient$ClientEndpoint: Connecting to master spark://192.168.11.221:7077...
17/05/19 14:32:49 INFO SparkDeploySchedulerBackend: Connected to Spark cluster with app ID app-20170519143251-0005
17/05/19 14:32:49 INFO AppClient$ClientEndpoint: Executor added: app-20170519143251-0005/0 on worker-20170519135622-192.168.11.222-39133 (192.168.11.222:39133) with 1 cores
17/05/19 14:32:49 INFO SparkDeploySchedulerBackend: Granted executor ID app-20170519143251-0005/0 on hostPort 192.168.11.222:39133 with 1 cores, 1024.0 MB RAM
17/05/19 14:32:49 INFO AppClient$ClientEndpoint: Executor added: app-20170519143251-0005/1 on worker-20170519135622-192.168.11.223-37116 (192.168.11.223:37116) with 1 cores
17/05/19 14:32:49 INFO SparkDeploySchedulerBackend: Granted executor ID app-20170519143251-0005/1 on hostPort 192.168.11.223:37116 with 1 cores, 1024.0 MB RAM
17/05/19 14:32:49 INFO AppClient$ClientEndpoint: Executor updated: app-20170519143251-0005/0 is now RUNNING
17/05/19 14:32:49 INFO AppClient$ClientEndpoint: Executor updated: app-20170519143251-0005/1 is now RUNNING
17/05/19 14:32:49 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 7523.
17/05/19 14:32:49 INFO NettyBlockTransferService: Server created on 7523
17/05/19 14:32:49 INFO BlockManagerMaster: Trying to register BlockManager
17/05/19 14:32:49 INFO BlockManagerMasterEndpoint: Registering block manager 169.254.4.104:7523 with 2.4 GB RAM, BlockManagerId(driver, 169.254.4.104, 7523)
17/05/19 14:32:49 INFO BlockManagerMaster: Registered BlockManager
17/05/19 14:32:49 INFO SparkDeploySchedulerBackend: SchedulerBackend is ready for scheduling beginning after reached minRegisteredResourcesRatio: 0.0
17/05/19 14:32:49 INFO HttpFileServer: HTTP File server directory is C:\Users\Philon Yun\AppData\Local\Temp\spark-78170f53-6988-49c6-b17c-42e6544272d0\httpd-d1f185b3-75b9-4b12-becf-fc079368e8ac
17/05/19 14:32:49 INFO HttpServer: Starting HTTP Server
17/05/19 14:32:49 INFO Utils: Successf
c314
ully started service 'HTTP file server' on port 7524.
17/05/19 14:32:50 INFO SparkContext: Added JAR D:\scala-idea-workspace\cars\out\artifacts\cars_jar\cars.jar at http://169.254.4.104:7524/jars/cars.jar with timestamp 1495175570735
17/05/19 14:32:51 INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 127.4 KB, free 127.4 KB)
17/05/19 14:32:51 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 13.9 KB, free 141.3 KB)
17/05/19 14:32:51 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on 169.254.4.104:7523 (size: 13.9 KB, free: 2.4 GB)
17/05/19 14:32:51 INFO SparkContext: Created broadcast 0 from textFile at HelloWorld.scala:18
17/05/19 14:32:52 INFO FileInputFormat: Total input paths to process : 1
17/05/19 14:32:52 INFO SparkContext: Starting job: json at HelloWorld.scala:25
17/05/19 14:32:52 INFO DAGScheduler: Got job 0 (json at HelloWorld.scala:25) with 2 output partitions
17/05/19 14:32:52 INFO DAGScheduler: Final stage: ResultStage 0 (json at HelloWorld.scala:25)
17/05/19 14:32:52 INFO DAGScheduler: Parents of final stage: List()
17/05/19 14:32:52 INFO DAGScheduler: Missing parents: List()
17/05/19 14:32:53 INFO DAGScheduler: Submitting ResultStage 0 (MapPartitionsRDD[9] at json at HelloWorld.scala:25), which has no missing parents
17/05/19 14:32:53 INFO MemoryStore: Block broadcast_1 stored as values in memory (estimated size 7.6 KB, free 148.9 KB)
17/05/19 14:32:53 INFO MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 3.9 KB, free 152.8 KB)
17/05/19 14:32:53 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on 169.254.4.104:7523 (size: 3.9 KB, free: 2.4 GB)
17/05/19 14:32:53 INFO SparkContext: Created broadcast 1 from broadcast at DAGScheduler.scala:1006
17/05/19 14:32:53 INFO DAGScheduler: Submitting 2 missing tasks from ResultStage 0 (MapPartitionsRDD[9] at json at HelloWorld.scala:25)
17/05/19 14:32:53 INFO TaskSchedulerImpl: Adding task set 0.0 with 2 tasks
17/05/19 14:32:55 INFO AppClient$ClientEndpoint: Executor updated: app-20170519143251-0005/1 is now EXITED (Command exited with code 1)
17/05/19 14:32:55 INFO SparkDeploySchedulerBackend: Executor app-20170519143251-0005/1 removed: Command exited with code 1
17/05/19 14:32:55 INFO SparkDeploySchedulerBackend: Asked to remove non-existent executor 1
17/05/19 14:32:55 INFO AppClient$ClientEndpoint: Executor added: app-20170519143251-0005/2 on worker-20170519135622-192.168.11.223-37116 (192.168.11.223:37116) with 1 cores
17/05/19 14:32:55 INFO SparkDeploySchedulerBackend: Granted executor ID app-20170519143251-0005/2 on hostPort 192.168.11.223:37116 with 1 cores, 1024.0 MB RAM
17/05/19 14:32:55 INFO AppClient$ClientEndpoint: Executor updated: app-20170519143251-0005/2 is now RUNNING
17/05/19 14:32:55 INFO AppClient$ClientEndpoint: Executor updated: app-20170519143251-0005/0 is now EXITED (Command exited with code 1)
17/05/19 14:32:55 INFO SparkDeploySchedulerBackend: Executor app-20170519143251-0005/0 removed: Command exited with code 1
17/05/19 14:32:55 INFO SparkDeploySchedulerBackend: Asked to remove non-existent executor 0
17/05/19 14:32:55 INFO AppClient$ClientEndpoint: Executor added: app-20170519143251-0005/3 on worker-20170519135622-192.168.11.222-39133 (192.168.11.222:39133) with 1 cores
17/05/19 14:32:55 INFO SparkDeploySchedulerBackend: Granted executor ID app-20170519143251-0005/3 on hostPort 192.168.11.222:39133 with 1 cores, 1024.0 MB RAM
17/05/19 14:32:55 INFO AppClient$ClientEndpoint: Executor updated: app-20170519143251-0005/3 is now RUNNING
17/05/19 14:32:58 INFO AppClient$ClientEndpoint: Executor updated: app-20170519143251-0005/3 is now EXITED (Command exited with code 1)
17/05/19 14:32:58 INFO SparkDeploySchedulerBackend: Executor app-20170519143251-0005/3 removed: Command exited with code 1
17/05/19 14:32:58 INFO SparkDeploySchedulerBackend: Asked to remove non-existent executor 3
17/05/19 14:32:58 INFO AppClient$ClientEndpoint: Executor added: app-20170519143251-0005/4 on worker-20170519135622-192.168.11.222-39133 (192.168.11.222:39133) with 1 cores
17/05/19 14:32:58 INFO SparkDeploySchedulerBackend: Granted executor ID app-20170519143251-0005/4 on hostPort 192.168.11.222:39133 with 1 cores, 1024.0 MB RAM
17/05/19 14:32:58 INFO AppClient$ClientEndpoint: Executor updated: app-20170519143251-0005/4 is now RUNNING
17/05/19 14:32:58 INFO AppClient$ClientEndpoint: Executor updated: app-20170519143251-0005/2 is now EXITED (Command exited with code 1)
17/05/19 14:32:58 INFO SparkDeploySchedulerBackend: Executor app-20170519143251-0005/2 removed: Command exited with code 1
17/05/19 14:32:58 INFO SparkDeploySchedulerBackend: Asked to remove non-existent executor 2
17/05/19 14:32:58 INFO AppClient$ClientEndpoint: Executor added: app-20170519143251-0005/5 on worker-20170519135622-192.168.11.223-37116 (192.168.11.223:37116) with 1 cores
17/05/19 14:32:58 INFO SparkDeploySchedulerBackend: Granted executor ID app-20170519143251-0005/5 on hostPort 192.168.11.223:37116 with 1 cores, 1024.0 MB RAM
17/05/19 14:32:58 INFO AppClient$ClientEndpoint: Executor updated: app-20170519143251-0005/5 is now RUNNING
17/05/19 14:33:03 INFO AppClient$ClientEndpoint: Executor updated: app-20170519143251-0005/5 is now EXITED (Command exited with code 1)
17/05/19 14:33:03 INFO SparkDeploySchedulerBackend: Executor app-20170519143251-0005/5 removed: Command exited with code 1
17/05/19 14:33:03 INFO SparkDeploySchedulerBackend: Asked to remove non-existent executor 5
17/05/19 14:33:03 INFO AppClient$ClientEndpoint: Executor added: app-20170519143251-0005/6 on worker-20170519135622-192.168.11.223-37116 (192.168.11.223:37116) with 1 cores
17/05/19 14:33:03 INFO SparkDeploySchedulerBackend: Granted executor ID app-20170519143251-0005/6 on hostPort 192.168.11.223:37116 with 1 cores, 1024.0 MB RAM
17/05/19 14:33:03 INFO AppClient$ClientEndpoint: Executor updated: app-20170519143251-0005/6 is now RUNNING
17/05/19 14:33:06 INFO AppClient$ClientEndpoint: Executor updated: app-20170519143251-0005/4 is now EXITED (Command exited with code 1)
17/05/19 14:33:06 INFO SparkDeploySchedulerBackend: Executor app-20170519143251-0005/4 removed: Command exited with code 1
17/05/19 14:33:06 INFO SparkDeploySchedulerBackend: Asked to remove non-existent executor 4
17/05/19 14:33:06 INFO AppClient$ClientEndpoint: Executor added: app-20170519143251-0005/7 on worker-20170519135622-192.168.11.222-39133 (192.168.11.222:39133) with 1 cores
17/05/19 14:33:06 INFO SparkDeploySchedulerBackend: Granted executor ID app-20170519143251-0005/7 on hostPort 192.168.11.222:39133 with 1 cores, 1024.0 MB RAM
17/05/19 14:33:06 INFO AppClient$ClientEndpoint: Executor updated: app-20170519143251-0005/7 is now RUNNING
17/05/19 14:33:08 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
17/05/19 14:33:09 INFO AppClient$ClientEndpoint: Executor updated: app-20170519143251-0005/7 is now EXITED (Command exited with code 1)
17/05/19 14:33:09 INFO SparkDeploySchedulerBackend: Executor app-20170519143251-0005/7 removed: Command exited with code 1
17/05/19 14:33:09 INFO SparkDeploySchedulerBackend: Asked to remove non-existent executor 7
17/05/19 14:33:09 INFO AppClient$ClientEndpoint: Executor added: app-20170519143251-0005/8 on worker-20170519135622-192.168.11.222-39133 (192.168.11.222:39133) with 1 cores
17/05/19 14:33:09 INFO SparkDeploySchedulerBackend: Granted executor ID app-20170519143251-0005/8 on hostPort 192.168.11.222:39133 with 1 cores, 1024.0 MB RAM
17/05/19 14:33:09 INFO AppClient$ClientEndpoint: Executor updated: app-20170519143251-0005/8 is now RUNNING
17/05/19 14:33:12 INFO AppClient$ClientEndpoint: Executor updated: app-20170519143251-0005/6 is now EXITED (Command exited with code 1)
17/05/19 14:33:12 INFO SparkDeploySchedulerBackend: Executor app-20170519143251-0005/6 removed: Command exited with code 1
17/05/19 14:33:12 INFO SparkDeploySchedulerBackend: Asked to remove non-existent executor 6
17/05/19 14:33:12 INFO AppClient$ClientEndpoint: Executor added: app-20170519143251-0005/9 on worker-20170519135622-192.168.11.223-37116 (192.168.11.223:37116) with 1 cores
17/05/19 14:33:12 INFO SparkDeploySchedulerBackend: Granted executor ID app-20170519143251-0005/9 on hostPort 192.168.11.223:37116 with 1 cores, 1024.0 MB RAM
17/05/19 14:33:12 INFO AppClient$ClientEndpoint: Executor updated: app-20170519143251-0005/9 is now RUNNING
17/05/19 14:33:17 INFO AppClient$ClientEndpoint: Executor updated: app-20170519143251-0005/8 is now EXITED (Command exited with code 1)
17/05/19 14:33:17 INFO SparkDeploySchedulerBackend: Executor app-20170519143251-0005/8 removed: Command exited with code 1
17/05/19 14:33:17 INFO SparkDeploySchedulerBackend: Asked to remove non-existent executor 8
17/05/19 14:33:17 INFO AppClient$ClientEndpoint: Executor added: app-20170519143251-0005/10 on worker-20170519135622-192.168.11.222-39133 (192.168.11.222:39133) with 1 cores
17/05/19 14:33:17 INFO SparkDeploySchedulerBackend: Granted executor ID app-20170519143251-0005/10 on hostPort 192.168.11.222:39133 with 1 cores, 1024.0 MB RAM
17/05/19 14:33:17 INFO AppClient$ClientEndpoint: Executor updated: app-20170519143251-0005/10 is now RUNNING
17/05/19 14:33:20 INFO AppClient$ClientEndpoint: Executor updated: app-20170519143251-0005/9 is now EXITED (Command exited with code 1)
17/05/19 14:33:20 INFO SparkDeploySchedulerBackend: Executor app-20170519143251-0005/9 removed: Command exited with code 1
17/05/19 14:33:20 INFO SparkDeploySchedulerBackend: Asked to remove non-existent executor 9
17/05/19 14:33:20 INFO AppClient$ClientEndpoint: Executor added: app-20170519143251-0005/11 on worker-20170519135622-192.168.11.223-37116 (192.168.11.223:37116) with 1 cores
17/05/19 14:33:20 INFO SparkDeploySchedulerBackend: Granted executor ID app-20170519143251-0005/11 on hostPort 192.168.11.223:37116 with 1 cores, 1024.0 MB RAM
17/05/19 14:33:20 INFO AppClient$ClientEndpoint: Executor updated: app-20170519143251-0005/11 is now RUNNING
这一行特别醒目
WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
初始工作没有接受任何资源；检查你的集群用户界面以确保工人注册并有足够的资源
我在网上查了很多，说内存不足的，节点间通信问题的，其实我的集群设置是没问题的，而且有足够的内存用
这是我的提交任务后的集群信息：
最终我找到了解决办法就是在windows环境变量中添加一个环境变量 SPARK_LOCAL_IP
在cmd中运行ipconfig命令查看自己的ip地址，然后赋值给这个环境变量，然后重启IDEA重新提交运行任务就可以解决这个问题。
内容来自用户分享和网络整理，不保证内容的准确性，如有侵权内容，可联系管理员处理 
相关文章推荐
- windows在IDEA环境运行spark程序（spark安装在另一台mac机器）
- windows下使用idea maven配置spark运行环境、运行WordCount例子以及碰到的问题
- MSCRM4.0 - 在64位MSCRM4.0 环境中运行32位程序出现的问题及解决方案
- 解决在windows的eclipse上面运行WordCount程序出现的一系列问题详解
- 在windows上用eclipse远程运行hadoop上的wordcount程序出现的问题，求解决
- C程序访问hadoop出现的各种编译错误和运行时各种类DefFound异常的解决方法（makefile的书写和环境变量的配置）
- QT 编译通过但是运行出现程序异常结束或者crashed的问题 在QT creator中最佳解决办法
- 对异常处理中的一个问题的思考（出现异常，程序仍能继续运行）
- windows 环境下部署spark运行环境 （包含遇到的问题和解决方法）
- Windows|Eclipse 运行HDFS程序遇到问题之 AccessControlException【windows中运行hdfs程序后报的异常：访问权限控制异常；方案二，修改MR程序即可】
- 【经验随笔】Java程序远程调试定位特定运行环境上出现的问题
- windows环境下本地运行spark程序
- windows搭建spark运行环境(windows scala,hadoop,spark安装,idea使用配置等)
- windows环境下VS2008编译程序在有些电脑上运行不了的问题
- windows环境开发yii上传程序至linux服务器时程序运行结果不一致的问题
- 在伪分布式环境下通过宿主机上的eclipse运行WordCount小程序出现的问题小结
- 使用idea编辑sparkSql的程序运行出现java.lang.IllegalArgumentException: java.net.URISyntaxException
- 【转】Android 中处理崩溃异常并重启程序出现页面重叠的问题
- nutch2.x在eclipse+windows环境下运行遇到的一些问题的解决方案
- 解决Unity 3D 在运行iOS程序时出现mscorlib.dll找不到的问题