您的位置:首页 > 运维架构 > Apache

spark一个版本问题引发的血案(java.lang.NoSuchMethodError: org.apache.spark.ui.SparkUI.addStaticHandler)

2016-07-09 19:55 671 查看
先看异常:

Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/jerry/.m2/repository/org/slf4j/slf4j-log4j12/1.7.10/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/local/data0/program/spark-1.6.1-bin-hadoop2.6.4/lib/spark-assembly-1.6.1-hadoop2.6.4.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
16/07/09 19:17:42 INFO SparkContext: Running Spark version 1.3.0
16/07/09 19:17:43 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
16/07/09 19:17:43 INFO SecurityManager: Changing view acls to: jerry
16/07/09 19:17:43 INFO SecurityManager: Changing modify acls to: jerry
16/07/09 19:17:43 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(jerry); users with modify permissions: Set(jerry)
16/07/09 19:17:44 INFO Slf4jLogger: Slf4jLogger started
16/07/09 19:17:44 INFO Remoting: Starting remoting
16/07/09 19:17:44 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriver@localhost:4090]
16/07/09 19:17:44 INFO Utils: Successfully started service 'sparkDriver' on port 4090.
16/07/09 19:17:44 INFO SparkEnv: Registering MapOutputTracker
16/07/09 19:17:44 INFO SparkEnv: Registering BlockManagerMaster
16/07/09 19:17:44 INFO DiskBlockManager: Created local directory at /tmp/spark-a250cf38-ad2c-4f20-8cb2-22f33bd7c391/blockmgr-6a6dbba7-a1de-41fc-99d0-bf0856c81fcf
16/07/09 19:17:44 INFO MemoryStore: MemoryStore started with capacity 945.5 MB
16/07/09 19:17:44 INFO HttpFileServer: HTTP File server directory is /tmp/spark-ce6f474b-70a6-4edf-be1f-ab6e904f29c8/httpd-1ef8f1f4-4a16-4f59-b99a-6eeceb9ece31
16/07/09 19:17:44 INFO HttpServer: Starting HTTP Server
16/07/09 19:17:44 INFO Server: jetty-8.y.z-SNAPSHOT
16/07/09 19:17:44 INFO AbstractConnector: Started SocketConnector@0.0.0.0:53269
16/07/09 19:17:44 INFO Utils: Successfully started service 'HTTP file server' on port 53269.
16/07/09 19:17:44 INFO SparkEnv: Registering OutputCommitCoordinator
16/07/09 19:17:44 INFO Server: jetty-8.y.z-SNAPSHOT
16/07/09 19:17:44 INFO AbstractConnector: Started SelectChannelConnector@0.0.0.0:4040
16/07/09 19:17:44 INFO Utils: Successfully started service 'SparkUI' on port 4040.
16/07/09 19:17:44 INFO SparkUI: Started SparkUI at http://debian0:4040 16/07/09 19:17:44 INFO Executor: Starting executor ID <driver> on host localhost
16/07/09 19:17:44 INFO AkkaUtils: Connecting to HeartbeatReceiver: akka.tcp://sparkDriver@localhost:4090/user/HeartbeatReceiver
16/07/09 19:17:45 INFO NettyBlockTransferService: Server created on 46819
16/07/09 19:17:45 INFO BlockManagerMaster: Trying to register BlockManager
16/07/09 19:17:45 INFO BlockManagerMasterActor: Registering block manager localhost:46819 with 945.5 MB RAM, BlockManagerId(<driver>, localhost, 46819)
16/07/09 19:17:45 INFO BlockManagerMaster: Registered BlockManager
Exception in thread "main" java.lang.NoSuchMethodError: org.apache.spark.ui.SparkUI.addStaticHandler(Ljava/lang/String;Ljava/lang/String;)V
at org.apache.spark.sql.execution.ui.SQLTab.<init>(SQLTab.scala:34)
at org.apache.spark.sql.SQLContext$$anonfun$createListenerAndUI$1.apply(SQLContext.scala:1369)
at org.apache.spark.sql.SQLContext$$anonfun$createListenerAndUI$1.apply(SQLContext.scala:1369)
at scala.Option.foreach(Option.scala:236)
at org.apache.spark.sql.SQLContext$.createListenerAndUI(SQLContext.scala:1369)
at org.apache.spark.sql.SQLContext.<init>(SQLContext.scala:77)
at com.spark.test.sql.SparkSqlJDBC2Mysql$.main(SparkSqlJDBC2Mysql.scala:21)
at com.spark.test.sql.SparkSqlJDBC2Mysql.main(SparkSqlJDBC2Mysql.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:144)


一开始想到的是scala版本的问题,所以就将scala版本由 2.10.6 修改成了 2.10.4,修改完以后,问题依旧。
然后就开始想,这是 spark 方法的问题,所以是spark版本的问题,由于自己导入了spark的lib库是1.6.1的,但是项目的pom文件中还是1.3.0版本的,所以修改成1.6.1以后,问题迎刃而解。

总结:注意版本版本版本的统一,重要的事情说三遍,一般scala遇到 java.lang.NoSuchMethodError
异常都是版本的问题,切记,切记,切记,重要的事情再说三遍。
内容来自用户分享和网络整理,不保证内容的准确性,如有侵权内容,可联系管理员处理 点击这里给我发消息
标签: