IDEA 开发环境中 调试Spark SQL及遇到问题解决办法

1.问题

java.lang.OutOfMemoryError: PermGen space

java.lang.OutOfMemoryError: Java heap space

7/04/17 17:00:05 WARN NettyRpcEndpointRef: Error sending message [message = Heartbeat(driver,[Lscala.Tuple2;@631e6c90,BlockManagerId(driver, localhost, 53273))] in 1 attempts
org.apache.spark.rpc.RpcTimeoutException: Futures timed out after [10 seconds]. This timeout is controlled by spark.executor.heartbeatInterval
at org.apache.spark.rpc.RpcTimeout.org$apache$spark$rpc$RpcTimeout$$createRpcTimeoutException(RpcTimeout.scala:48)
at org.apache.spark.rpc.RpcTimeout$$anonfun$addMessageIfTimeout$1.applyOrElse(RpcTimeout.scala:63)
at org.apache.spark.rpc.RpcTimeout$$anonfun$addMessageIfTimeout$1.applyOrElse(RpcTimeout.scala:59)
at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:33)
at org.apache.spark.rpc.RpcTimeout.awaitResult(RpcTimeout.scala:76)
at org.apache.spark.rpc.RpcEndpointRef.askWithRetry(RpcEndpointRef.scala:101)
at org.apache.spark.executor.Executor.org$apache$spark$executor$Executor$$reportHeartBeat(Executor.scala:449)
at org.apache.spark.executor.Executor$$anon$1$$anonfun$run$1.apply$mcV$sp(Executor.scala:470)
at org.apache.spark.executor.Executor$$anon$1$$anonfun$run$1.apply(Executor.scala:470)
at org.apache.spark.executor.Executor$$anon$1$$anonfun$run$1.apply(Executor.scala:470)
at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1765)
at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:470)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.util.concurrent.TimeoutException: Futures timed out after [10 seconds]
at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219)
at scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223)
at scala.concurrent.Await$$anonfun$result$1.apply(package.scala:107)
at scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53)
at scala.concurrent.Await$.result(package.scala:107)
at org.apache.spark.rpc.RpcTimeout.awaitResult(RpcTimeout.scala:75)
... 14 more

17/04/17 17:46:36 ERROR TaskSetManager: Task 1 in stage 3.0 failed 1 times; aborting job
Exception in thread "qtp502891368-59" java.lang.OutOfMemoryError: Java heap space
17/04/17 17:57:36 ERROR Utils: uncaught error in thread Spark Context Cleaner, stopping SparkContext
java.lang.OutOfMemoryError: Java heap space
17/04/17 17:57:36 WARN TaskSetManager: Lost task 0.0 in stage 3.0 (TID 413, localhost): ExecutorLostFailure (executor driver exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 182499 ms
17/04/17 17:57:36 INFO TaskSchedulerImpl: Removed TaskSet 3.0, whose tasks have all completed, from pool
Exception in thread "qtp502891368-62" java.lang.OutOfMemoryError: Java heap space
17/04/17 17:57:36 WARN SingleThreadEventExecutor: Unexpected exception from an event executor:
java.lang.OutOfMemoryError: Java heap space
17/04/17 17:57:36 ERROR Executor: Exception in task 0.0 in stage 3.0 (TID 413)
java.lang.OutOfMemoryError: Java heap space
17/04/17 17:57:36 WARN NettyRpcEnv: Ignored message: true

猜测原因：

Spark对内存的消耗主要分为三部分（即取决于你的应用程序的需求）：

数据集中对象的大小

访问这些对象的内存消耗

垃圾回收GC的消耗

由网络或者gc引起,worker或executor没有接收到executor或task的心跳反馈，导致 Executor&Task Lost，这时要提高 spark.network.timeout 的值，根据情况改成300(5min)或更高。

解决办法：

这个问题，需要设置IEDA的JVM参数: -Xms256m -Xmx512m -XX:PermSize=256m -XX:MaxPermSize=256M



若在Linux上命令方式的话：



参考： Hadoop与Spark常用配置参数总结 http://www.tuicool.com/articles/naaAzq2