Spark远程调试配置,在IDEA中的配置
2017-07-09 14:36
429 查看
关于Spark集群部署参考:http://blog.csdn.net/tototuzuoquan/article/details/74481570
一、Spark远程调试配置:
二、编写程序进行调试
接着:
一、Spark远程调试配置:
#调试Master,在master节点的spark-env.sh中添加SPARK_MASTER_OPTS变量 export SPARK_MASTER_OPTS="-Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=y,address=10000" #启动Master sbin/start-master.sh #调试Worker,在worker节点的spark-env.sh中添加SPARK_WORKER_OPTS变量 export SPARK_WORKER_OPTS="-Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=y,address=10001" #启动Worker sbin/start-slave.sh 1 spark://hadoop1:7077 #调试spark-submit + app bin/spark-submit --class cn.itcast.spark.WordCount --master spark://hadoop1:7077 --driver-java-options "-Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=y,address=10002" /root/wc.jar hdfs://mycluster/wordcount/input/2.txt hdfs://mycluster/out2 #调试spark-submit + app + executor bin/spark-submit --class cn.itcast.spark.WordCount --master spark://hadoop1:7077 --conf "spark.executor.extraJavaOptions=-Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=y,address=10003" --driver-java-options "-Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=y,address=10002" /root/wc.jar hdfs://mycluster/wordcount/input/2.txt hdfs://mycluster/out2
二、编写程序进行调试
接着:
相关文章推荐
- Spark - 使用IDEA远程调试位于Yarn-Client模式下的Spark集群相关配置
- IDEA 2016 配置远程 TOMCAT 部署 调试
- (二)win7下用Intelij IDEA 远程调试spark standalone 集群
- IDEA【spark&mapreduce混合配置】mapreduce spark 本地调试
- idea远程调试 spark
- idea远程调试配置
- Spark1.6 Idea下远程调试
- Win下用idea远程在hadoop上调试spark程序及读取hbase
- 4000 win7下用Intelij IDEA 远程调试spark standalone 集群
- Spark基础随笔:Spark1.6 Idea下远程调试的2种方法
- idea调试远程spark服务器(2017.04图文,非转载)
- Spark基础随笔:Spark1.6 Idea下远程调试的2种方法
- idea配置远程调试tomcat
- Spark基础随笔:Spark1.6 Idea下远程调试的2种方法
- 从零搭建基于sbt和IDEA的远程spark调试工程
- IDEA远程tomcat调试配置
- IDEA远程调试配置
- Intellij IDEA 配置Tomcat远程调试
- (二)win7下用Intelij IDEA 远程调试spark standalone 集群
- Tomcat 和 IDEA 配置远程调试