使用Intellij构建spark源码阅读环境
2016-06-20 15:52
495 查看
步骤1:
下载spark-源码,这里以spark1.3为例。
http://spark.apache.org/downloads.html
2.下载编译后的spark源码,这里下载根据Hadoop2.3版本编译的。
3.使用Intellij新建scala工程(使用scala编写spark application,也可以新建java工程)
5.配置项目scala版本
6.导入spark依赖版-spark-assembly-1.3.0-hadoop2.4.0.jar
导入路径为:
FIle->ProjectStructure->project Settings->Libraries->+java->选择spark-assembly-1.3.0-hadoop2.4.0.jar。
7.导入spark-源码
点加号,选择spark1.3编译前源码,点确认。
8.新建spark-scala-hello-world项目,代码如下:
可以插入断点,可以把断点插入到Executor、Driver等中。
项目执行结果如下:
从执行结果中可以看到spark各个组件的启动过程。
插入断点,可以进行debug。
下载spark-源码,这里以spark1.3为例。
http://spark.apache.org/downloads.html
2.下载编译后的spark源码,这里下载根据Hadoop2.3版本编译的。
3.使用Intellij新建scala工程(使用scala编写spark application,也可以新建java工程)
5.配置项目scala版本
6.导入spark依赖版-spark-assembly-1.3.0-hadoop2.4.0.jar
导入路径为:
FIle->ProjectStructure->project Settings->Libraries->+java->选择spark-assembly-1.3.0-hadoop2.4.0.jar。
7.导入spark-源码
点加号,选择spark1.3编译前源码,点确认。
8.新建spark-scala-hello-world项目,代码如下:
/** * Created With Intellij * User: lisongfeng * Date: 16/6/3 * Time: 下午5:34 */ import org.apache.spark.SparkConf import org.apache.spark.SparkContext import org.apache.spark.SparkContext._ /** * @author Administrator */ object LineCount { def main(args: Array[String]) { val conf = new SparkConf() .setAppName("LineCount") .setMaster("local[3]") val sc = new SparkContext(conf); val lines = sc.textFile("/usr/local/README.md", 3) lines.filter( _.contains("Home")).foreach(println) val lines1 = sc.textFile("/usr/local/README1.md", 3) val rdd2 = lines.union(lines1) rdd2.foreach(println _ ) lines.map(_ * 2) } }配置下Intellij右上角主类,就能跑起来了。
可以插入断点,可以把断点插入到Executor、Driver等中。
项目执行结果如下:
/Library/Java/JavaVirtualMachines/jdk1.8.0_45.jdk/Contents/Home/bin/java -Didea.launcher.port=7532 "-Didea.launcher.bin.path=/Applications/IntelliJ IDEA 14.app/Contents/bin" -Dfile.encoding=UTF-8 -classpath "/Library/Java/JavaVirtualMachines/jdk1.8.0_45.jdk/Contents/Home/lib/ant-javafx.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_45.jdk/Contents/Home/lib/dt.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_45.jdk/Contents/Home/lib/javafx-mx.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_45.jdk/Contents/Home/lib/jconsole.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_45.jdk/Contents/Home/lib/packager.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_45.jdk/Contents/Home/lib/sa-jdi.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_45.jdk/Contents/Home/lib/tools.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_45.jdk/Contents/Home/jre/lib/charsets.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_45.jdk/Contents/Home/jre/lib/deploy.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_45.jdk/Contents/Home/jre/lib/javaws.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_45.jdk/Contents/Home/jre/lib/jce.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_45.jdk/Contents/Home/jre/lib/jfr.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_45.jdk/Contents/Home/jre/lib/jfxswt.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_45.jdk/Contents/Home/jre/lib/jsse.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_45.jdk/Contents/Home/jre/lib/management-agent.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_45.jdk/Contents/Home/jre/lib/plugin.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_45.jdk/Contents/Home/jre/lib/resources.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_45.jdk/Contents/Home/jre/lib/rt.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_45.jdk/Contents/Home/jre/lib/ext/cldrdata.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_45.jdk/Contents/Home/jre/lib/ext/dnsns.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_45.jdk/Contents/Home/jre/lib/ext/jfxrt.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_45.jdk/Contents/Home/jre/lib/ext/localedata.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_45.jdk/Contents/Home/jre/lib/ext/nashorn.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_45.jdk/Contents/Home/jre/lib/ext/sunec.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_45.jdk/Contents/Home/jre/lib/ext/sunjce_provider.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_45.jdk/Contents/Home/jre/lib/ext/sunpkcs11.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_45.jdk/Contents/Home/jre/lib/ext/zipfs.jar:/Users/lisongfeng/gitRepository/scala13/out/production/scala13:/Users/lisongfeng/.ivy2/cache/org.scala-lang/scala-library/jars/scala-library-2.10.5.jar:/Users/lisongfeng/.ivy2/cache/org.scala-lang/scala-reflect/jars/scala-reflect-2.10.5.jar:/Volumes/LISONGFENG/code/spark-1.3.0-bin-hadoop2.4/spark-1.3.0-bin-hadoop2.4/lib/spark-assembly-1.3.0-hadoop2.4.0.jar:/Applications/IntelliJ IDEA 14.app/Contents/lib/idea_rt.jar" com.intellij.rt.execution.application.AppMain LineCount Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties 16/06/19 23:48:02 INFO SparkContext: Running Spark version 1.3.0 16/06/19 23:48:08 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 16/06/19 23:48:09 INFO SecurityManager: Changing view acls to: lisongfeng 16/06/19 23:48:09 INFO SecurityManager: Changing modify acls to: lisongfeng 16/06/19 23:48:09 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(lisongfeng); users with modify permissions: Set(lisongfeng) 16/06/19 23:48:10 INFO Slf4jLogger: Slf4jLogger started 16/06/19 23:48:10 INFO Remoting: Starting remoting 16/06/19 23:48:11 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriver@192.168.1.107:49763] 16/06/19 23:48:11 INFO Utils: Successfully started service 'sparkDriver' on port 49763. 16/06/19 23:48:11 INFO SparkEnv: Registering MapOutputTracker 16/06/19 23:48:11 INFO SparkEnv: Registering BlockManagerMaster 16/06/19 23:48:11 INFO DiskBlockManager: Created local directory at /var/folders/00/tg296f_x5yl42sbgfmxndm400000gn/T/spark-a07c726d-752d-46a0-b2d1-c2f7d6ef89aa/blockmgr-72709721-9dcf-431f-a4ae-a085016b9b92 16/06/19 23:48:11 INFO MemoryStore: MemoryStore started with capacity 983.1 MB 16/06/19 23:48:11 INFO HttpFileServer: HTTP File server directory is /var/folders/00/tg296f_x5yl42sbgfmxndm400000gn/T/spark-ceca0f7b-3167-4e54-98be-8da9d74ebbc6/httpd-b2bab4f3-903e-4007-87b9-84a724c53b73 16/06/19 23:48:11 INFO HttpServer: Starting HTTP Server 16/06/19 23:48:11 INFO Server: jetty-8.y.z-SNAPSHOT 16/06/19 23:48:11 INFO AbstractConnector: Started SocketConnector@0.0.0.0:49764 16/06/19 23:48:11 INFO Utils: Successfully started service 'HTTP file server' on port 49764. 16/06/19 23:48:11 INFO SparkEnv: Registering OutputCommitCoordinator 16/06/19 23:48:11 INFO Server: jetty-8.y.z-SNAPSHOT 16/06/19 23:48:11 INFO AbstractConnector: Started SelectChannelConnector@0.0.0.0:4040 16/06/19 23:48:11 INFO Utils: Successfully started service 'SparkUI' on port 4040. 16/06/19 23:48:11 INFO SparkUI: Started SparkUI at http://192.168.1.107:4040 16/06/19 23:48:12 INFO Executor: Starting executor ID <driver> on host localhost 16/06/19 23:48:12 INFO AkkaUtils: Connecting to HeartbeatReceiver: akka.tcp://sparkDriver@192.168.1.107:49763/user/HeartbeatReceiver 16/06/19 23:48:12 INFO NettyBlockTransferService: Server created on 49765 16/06/19 23:48:12 INFO BlockManagerMaster: Trying to register BlockManager 16/06/19 23:48:12 INFO BlockManagerMasterActor: Registering block manager localhost:49765 with 983.1 MB RAM, BlockManagerId(<driver>, localhost, 49765) 16/06/19 23:4 4000 8:12 INFO BlockManagerMaster: Registered BlockManager 16/06/19 23:48:13 INFO MemoryStore: ensureFreeSpace(159118) called with curMem=0, maxMem=1030823608 16/06/19 23:48:13 INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 155.4 KB, free 982.9 MB) 16/06/19 23:48:13 INFO MemoryStore: ensureFreeSpace(22692) called with curMem=159118, maxMem=1030823608 16/06/19 23:48:13 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 22.2 KB, free 982.9 MB) 16/06/19 23:48:13 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on localhost:49765 (size: 22.2 KB, free: 983.0 MB) 16/06/19 23:48:13 INFO BlockManagerMaster: Updated info of block broadcast_0_piece0 16/06/19 23:48:13 INFO SparkContext: Created broadcast 0 from textFile at MAIN.scala:24 16/06/19 23:48:13 INFO FileInputFormat: Total input paths to process : 1 16/06/19 23:48:13 INFO SparkContext: Starting job: foreach at MAIN.scala:25 16/06/19 23:48:13 INFO DAGScheduler: Got job 0 (foreach at MAIN.scala:25) with 3 output partitions (allowLocal=false) 16/06/19 23:48:13 INFO DAGScheduler: Final stage: Stage 0(foreach at MAIN.scala:25) 16/06/19 23:48:13 INFO DAGScheduler: Parents of final stage: List() 16/06/19 23:48:13 INFO DAGScheduler: Missing parents: List() 16/06/19 23:48:13 INFO DAGScheduler: Submitting Stage 0 (MapPartitionsRDD[2] at filter at MAIN.scala:25), which has no missing parents 16/06/19 23:48:13 INFO MemoryStore: ensureFreeSpace(2872) called with curMem=181810, maxMem=1030823608 16/06/19 23:48:13 INFO MemoryStore: Block broadcast_1 stored as values in memory (estimated size 2.8 KB, free 982.9 MB) 16/06/19 23:48:13 INFO MemoryStore: ensureFreeSpace(2045) called with curMem=184682, maxMem=1030823608 16/06/19 23:48:13 INFO MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 2045.0 B, free 982.9 MB) 16/06/19 23:48:13 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on localhost:49765 (size: 2045.0 B, free: 983.0 MB) 16/06/19 23:48:13 INFO BlockManagerMaster: Updated info of block broadcast_1_piece0 16/06/19 23:48:13 INFO SparkContext: Created broadcast 1 from broadcast at DAGScheduler.scala:839 16/06/19 23:48:13 INFO DAGScheduler: Submitting 3 missing tasks from Stage 0 (MapPartitionsRDD[2] at filter at MAIN.scala:25) 16/06/19 23:48:13 INFO TaskSchedulerImpl: Adding task set 0.0 with 3 tasks 16/06/19 23:48:13 INFO TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, localhost, PROCESS_LOCAL, 1289 bytes) 16/06/19 23:48:14 INFO TaskSetManager: Starting task 1.0 in stage 0.0 (TID 1, localhost, PROCESS_LOCAL, 1289 bytes) 16/06/19 23:48:14 INFO TaskSetManager: Starting task 2.0 in stage 0.0 (TID 2, localhost, PROCESS_LOCAL, 1289 bytes) 16/06/19 23:48:14 INFO Executor: Running task 0.0 in stage 0.0 (TID 0) 16/06/19 23:48:14 INFO Executor: Running task 1.0 in stage 0.0 (TID 1) 16/06/19 23:48:14 INFO Executor: Running task 2.0 in stage 0.0 (TID 2) 16/06/19 23:48:14 INFO HadoopRDD: Input split: file:/usr/local/README.md:1620+812 16/06/19 23:48:14 INFO HadoopRDD: Input split: file:/usr/local/README.md:810+810 16/06/19 23:48:14 INFO HadoopRDD: Input split: file:/usr/local/README.md:0+810 16/06/19 23:48:14 INFO deprecation: mapred.tip.id is deprecated. Instead, use mapreduce.task.id 16/06/19 23:48:14 INFO deprecation: mapred.task.id is deprecated. Instead, use mapreduce.task.attempt.id 16/06/19 23:48:14 INFO deprecation: mapred.task.is.map is deprecated. Instead, use mapreduce.task.ismap 16/06/19 23:48:14 INFO deprecation: mapred.task.partition is deprecated. Instead, use mapreduce.task.partition 16/06/19 23:48:14 INFO deprecation: mapred.job.id is deprecated. Instead, use mapreduce.job.id # Homebrew Code is under the [BSD 2 Clause (NetBSD) license](https://github.com/Homebrew/homebrew/tree/master/LICENSE.txt). Homebrew's current maintainers are [Misty De Meo](https://github.com/mistydemeo), [Andrew Janke](https://github.com/apjanke), [Xu Cheng](https://github.com/xu-cheng), [Mike McQuaid](https://github.com/mikemcquaid), [Baptiste Fontaine](https://github.com/bfontaine), [Brett Koonce](https://github.com/asparagui), [Martin Afanasjew](https://github.com/UniqMartin), [Dominyk Tiller](https://github.com/DomT4), [Tim Smith](https://github.com/tdsmith) and [Alex Dunn](https://github.com/dunn). Former maintainers with significant contributions include [Jack Nagel](https://github.com/jacknagel), [Adam Vandenberg](https://github.com/adamv) and Homebrew's creator: [Max Howell](https://github.com/mxcl). `brew help`, `man brew` or check [our documentation](https://github.com/Homebrew/homebrew/tree/master/share/doc/homebrew#readme). Second, read the [Troubleshooting Checklist](https://github.com/Homebrew/homebrew/blob/master/share/doc/homebrew/Troubleshooting.md#troubleshooting). Our CI infrastructure was paid for by [our Kickstarter supporters](https://github.com/Homebrew/homebrew/blob/master/SUPPORTERS.md). 16/06/19 23:48:14 INFO Executor: Finished task 1.0 in stage 0.0 (TID 1). 1792 bytes result sent to driver 16/06/19 23:48:14 INFO Executor: Finished task 2.0 in stage 0.0 (TID 2). 1792 bytes result sent to driver 16/06/19 23:48:14 INFO Executor: Finished task 0.0 in stage 0.0 (TID 0). 1792 bytes result sent to driver 16/06/19 23:48:14 INFO TaskSetManager: Finished task 1.0 in stage 0.0 (TID 1) in 184 ms on localhost (1/3) 16/06/19 23:48:14 INFO TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) in 218 ms on localhost (2/3) 16/06/19 23:48:14 INFO TaskSetManager: Finished task 2.0 in stage 0.0 (TID 2) in 188 ms on localhost (3/3) 16/06/19 23:48:14 INFO TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool 16/06/19 23:48:14 INFO DAGScheduler: Stage 0 (foreach at MAIN.scala:25) finished in 0.252 s 16/06/19 23:48:14 INFO DAGScheduler: Job 0 finished: foreach at MAIN.scala:25, took 0.481620 s 16/06/19 23:48:14 INFO MemoryStore: ensureFreeSpace(73391) called with curMem=186727, maxMem=1030823608 16/06/19 23:48:14 INFO MemoryStore: Block broadcast_2 stored as values in memory (estimated size 71.7 KB, free 982.8 MB) 16/06/19 23:48:14 INFO BlockManager: Removing broadcast 1 16/06/19 23:48:14 INFO BlockManager: Removing block broadcast_1 16/06/19 23:48:14 INFO MemoryStore: Block broadcast_1 of size 2872 dropped from memory (free 1030566362) 16/06/19 23:48:14 INFO BlockManager: Removing block broadcast_1_piece0 16/06/19 23:48:14 INFO MemoryStore: Block broadcast_1_piece0 of size 2045 dropped from memory (free 1030568407) 16/06/19 23:48:14 INFO BlockManagerInfo: Removed broadcast_1_piece0 on localhost:49765 in memory (size: 2045.0 B, free: 983.0 MB) 16/06/19 23:48:14 INFO BlockManagerMaster: Updated info of block broadcast_1_piece0 16/06/19 23:48:14 INFO ContextCleaner: Cleaned broadcast 1 16/06/19 23:48:14 INFO MemoryStore: ensureFreeSpace(31262) called with curMem=255201, maxMem=1030823608 16/06/19 23:48:14 INFO MemoryStore: Block broadcast_2_piece0 stored as bytes in memory (estimated size 30.5 KB, free 982.8 MB) 16/06/19 23:48:14 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on localhost:49765 (size: 30.5 KB, free: 983.0 MB) 16/06/19 23:48:14 INFO BlockManagerMaster: Updated info of block broadcast_2_piece0 16/06/19 23:48:14 INFO SparkContext: Created broadcast 2 from textFile at MAIN.scala:26 16/06/19 23:48:14 INFO FileInputFormat: Total input paths to process : 1 16/06/19 23:48:14 INFO SparkContext: Starting job: foreach at MAIN.scala:30 16/06/19 23:48:14 INFO DAGScheduler: Got job 1 (foreach at MAIN.scala:30) with 6 output partitions (allowLocal=false) 16/06/19 23:48:14 INFO DAGScheduler: Final stage: Stage 1(foreach at MAIN.scala:30) 16/06/19 23:48:14 INFO DAGScheduler: Parents of final stage: List() 16/06/19 23:48:14 INFO DAGScheduler: Missing parents: List() 16/06/19 23:48:14 INFO DAGScheduler: Submitting Stage 1 (UnionRDD[5] at union at MAIN.scala:29), which has no missing parents 16/06/19 23:48:14 INFO MemoryStore: ensureFreeSpace(3536) called with curMem=286463, maxMem=1030823608 16/06/19 23:48:14 INFO MemoryStore: Block broadcast_3 stored as values in memory (estimated size 3.5 KB, free 982.8 MB) 16/06/19 23:48:14 INFO MemoryStore: ensureFreeSpace(2589) called with curMem=289999, maxMem=1030823608 16/06/19 23:48:14 INFO MemoryStore: Block broadcast_3_piece0 stored as bytes in memory (estimated size 2.5 KB, free 982.8 MB) 16/06/19 23:48:14 INFO BlockManagerInfo: Added broadcast_3_piece0 in memory on localhost:49765 (size: 2.5 KB, free: 983.0 MB) 16/06/19 23:48:14 INFO BlockManagerMaster: Updated info of block broadcast_3_piece0 16/06/19 23:48:14 INFO SparkContext: Created broadcast 3 from broadcast at DAGScheduler.scala:839 16/06/19 23:48:14 INFO DAGScheduler: Submitting 6 missing tasks from Stage 1 (UnionRDD[5] at union at MAIN.scala:29) 16/06/19 23:48:14 INFO TaskSchedulerImpl: Adding task set 1.0 with 6 tasks 16/06/19 23:48:14 INFO TaskSetManager: Starting task 0.0 in stage 1.0 (TID 3, localhost, PROCESS_LOCAL, 1398 bytes) 16/06/19 23:48:14 INFO TaskSetManager: Starting task 1.0 in stage 1.0 (TID 4, localhost, PROCESS_LOCAL, 1398 bytes) 16/06/19 23:48:14 INFO TaskSetManager: Starting task 2.0 in stage 1.0 (TID 5, localhost, PROCESS_LOCAL, 1398 bytes) 16/06/19 23:48:14 INFO Executor: Running task 1.0 in stage 1.0 (TID 4) 16/06/19 23:48:14 INFO Executor: Running task 2.0 in stage 1.0 (TID 5) 16/06/19 23:48:14 INFO Executor: Running task 0.0 in stage 1.0 (TID 3) 16/06/19 23:48:14 INFO HadoopRDD: Input split: file:/usr/local/README.md:1620+812 16/06/19 23:48:14 INFO HadoopRDD: Input split: file:/usr/local/README.md:810+810 16/06/19 23:48:14 INFO HadoopRDD: Input split: file:/usr/local/README.md:0+810 16/06/19 23:48:14 INFO Executor: Finished task 1.0 in stage 1.0 (TID 4). 1792 bytes result sent to driver 16/06/19 23:48:14 INFO Executor: Finished task 2.0 in stage 1.0 (TID 5). 1792 bytes result sent to driver This is our PGP key which is valid until June 17, 2016. * Key ID: `0xE33A3D3CCE59E297` * Fingerprint: `C657 8F76 2E23 441E C879 EC5C E33A 3D3C CE59 E297` * Full key: https://keybase.io/homebrew/key.asc ## License Code is under the [BSD 2 Clause (NetBSD) license](https://github.com/Homebrew/homebrew/tree/master/LICENSE.txt). Documentation is under the [Creative Commons Attribution license](https://creativecommons.org/licenses/by/4.0/). ## Sponsors Our CI infrastructure was paid for by [our Kickstarter supporters](https://github.com/Homebrew/homebrew/blob/master/SUPPORTERS.md). Our CI infrastructure is hosted by [The Positive Internet Company](http://www.positive-internet.com). ## Who Are You? # Homebrew Homebrew's current maintainers are [Misty De Meo](https://github.com/mistydemeo), [Andrew Janke](https://github.com/apjanke), [Xu Cheng](https://github.com/xu-cheng), [Mike McQuaid](https://github.com/mikemcquaid), [Baptiste Fontaine](https://github.com/bfontaine), [Brett Koonce](https://github.com/asparagui), [Martin Afanasjew](https://github.com/UniqMartin), [Dominyk Tiller](https://github.com/DomT4), [Tim Smith](https://github.com/tdsmith) and [Alex Dunn](https://github.com/dunn). Our bottles (binary packages) are hosted by Bintray. Former maintainers with significant contributions include [Jack Nagel](https://github.com/jacknagel), [Adam Vandenberg](https://github.com/adamv) and Homebrew's creator: [Max Howell](https://github.com/mxcl). Features, usage and installation instructions are [summarised on the homepage](http://brew.sh). [![Downloads by Bintray](https://bintray.com/docs/images/downloads_by_bintray_96.png)](https://bintray.com/homebrew) ## What Packages Are Available? 1. Type `brew search` for a list. 2. Or visit [braumeister.org](http://braumeister.org) to browse packages online. 3. Or use `brew search --desc` to browse packages from the command line. ## More Documentation `brew help`, `man brew` or check [our documentation](https://github.com/Homebrew/homebrew/tree/master/share/doc/homebrew#readme). ## Troubleshooting First, please run `brew update` and `brew doctor`. Second, read the [Troubleshooting Checklist](https://github.com/Homebrew/homebrew/blob/master/share/doc/homebrew/Troubleshooting.md#troubleshooting). **If you don't read these it will take us far longer to help you with your problem.** ## Security Please report security issues to security@brew.sh. 16/06/19 23:48:14 INFO Executor: Finished task 0.0 in stage 1.0 (TID 3). 1792 bytes result sent to driver 16/06/19 23:48:14 INFO TaskSetManager: Starting task 3.0 in stage 1.0 (TID 6, localhost, PROCESS_LOCAL, 1399 bytes) 16/06/19 23:48:14 INFO Executor: Running task 3.0 in stage 1.0 (TID 6) 16/06/19 23:48:14 INFO TaskSetManager: Finished task 1.0 in stage 1.0 (TID 4) in 11 ms on localhost (1/6) 16/06/19 23:48:14 INFO TaskSetManager: Starting task 4.0 in stage 1.0 (TID 7, localhost, PROCESS_LOCAL, 1399 bytes) 16/06/19 23:48:14 INFO TaskSetManager: Finished task 2.0 in stage 1.0 (TID 5) in 12 ms on localhost (2/6) 16/06/19 23:48:14 INFO Executor: Running task 4.0 in stage 1.0 (TID 7) 16/06/19 23:48:14 INFO TaskSetManager: Starting task 5.0 in stage 1.0 (TID 8, localhost, PROCESS_LOCAL, 1399 bytes) 16/06/19 23:48:14 INFO Executor: Running task 5.0 in stage 1.0 (TID 8) 16/06/19 23:48:14 INFO TaskSetManager: Finished task 0.0 in stage 1.0 (TID 3) in 16 ms on localhost (3/6) 16/06/19 23:48:14 INFO HadoopRDD: Input split: file:/usr/local/README1.md:0+810 16/06/19 23:48:14 INFO HadoopRDD: Input split: file:/usr/local/README1.md:810+810 16/06/19 23:48:14 INFO HadoopRDD: Input split: file:/usr/local/README1.md:1620+812 16/06/19 23:48:14 INFO Executor: Finished task 4.0 in stage 1.0 (TID 7). 1792 bytes result sent to driver 16/06/19 23:48:14 INFO Executor: Finished task 3.0 in stage 1.0 (TID 6). 1792 bytes result sent to driver 16/06/19 23:48:14 INFO TaskSetManager: Finished task 4.0 in stage 1.0 (TID 7) in 11 ms on localhost (4/6) 16/06/19 23:48:14 INFO Executor: Finished task 5.0 in stage 1.0 (TID 8). 1792 bytes result sent to driver 16/06/19 23:48:14 INFO TaskSetManager: Finished task 3.0 in stage 1.0 (TID 6) in 14 ms on localhost (5/6) 16/06/19 23:48:14 INFO TaskSetManager: Finished task 5.0 in stage 1.0 (TID 8) in 12 ms on localhost (6/6) This is our PGP key which is valid until June 17, 2016. * Key ID: `0xE33A3D3CCE59E297` * Fingerprint: `C657 8F76 2E23 441E C879 EC5C E33A 3D3C CE59 E297` * Full key: https://keybase.io/homebrew/key.asc # Homebrew ## Who Are You? Homebrew's current maintainers are [Misty De Meo](https://github.com/mistydemeo), [Andrew Janke](https://github.com/apjanke), [Xu Cheng](https://github.com/xu-cheng), [Mike McQuaid](https://github.com/mikemcquaid), [Baptiste Fontaine](https://github.com/bfontaine), [Brett Koonce](https://github.com/asparagui), [Martin Afanasjew](https://github.com/UniqMartin), [Dominyk Tiller](https://github.com/DomT4), [Tim Smith](https://github.com/tdsmith) and [Alex Dunn](https://github.com/dunn). Former maintainers with significant contributions include [Jack Nagel](https://github.com/jacknagel), [Adam Vandenberg](https://github.com/adamv) and Homebrew's creator: [Max Howell](https://github.com/mxcl). Features, usage and installation instructions are [summarised on the homepage](http://brew.sh). ## What Packages Are Available? 1. Type `brew search` for a list. 2. Or visit [braumeister.org](http://braumeister.org) to browse packages online. 3. Or use `brew search --desc` to browse packages from the command line. ## More Documentation `brew help`, `man brew` or check [our documentation](https://github.com/Homebrew/homebrew/tree/master/share/doc/homebrew#readme). ## Troubleshooting First, please run `brew update` and `brew doctor`. Second, read the [Troubleshooting Checklist](https://github.com/Homebrew/homebrew/blob/master/share/doc/homebrew/Troubleshooting.md#troubleshooting). **If you don't read these it will take us far longer to help you with your problem.** ## Security Please report security issues to security@brew.sh. ## License Code is under the [BSD 2 Clause (NetBSD) license](https://github.com/Homebrew/homebrew/tree/master/LICENSE.txt). Documentation is under the [Creative Commons Attribution license](https://creativecommons.org/licenses/by/4.0/). ## Sponsors Our CI infrastructure was paid for by [our Kickstarter supporters](https://github.com/Homebrew/homebrew/blob/master/SUPPORTERS.md). Our CI infrastructure is hosted by [The Positive In a5b7 ternet Company](http://www.positive-internet.com). Our bottles (binary packages) are hosted by Bintray. 16/06/19 23:48:14 INFO TaskSchedulerImpl: Removed TaskSet 1.0, whose tasks have all completed, from pool [![Downloads by Bintray](https://bintray.com/docs/images/downloads_by_bintray_96.png)](https://bintray.com/homebrew) 16/06/19 23:48:14 INFO DAGScheduler: Stage 1 (foreach at MAIN.scala:30) finished in 0.029 s 16/06/19 23:48:14 INFO DAGScheduler: Job 1 finished: foreach at MAIN.scala:30, took 0.054069 s Process finished with exit code 0
从执行结果中可以看到spark各个组件的启动过程。
插入断点,可以进行debug。
相关文章推荐
- 超大数相加
- sina面试题--用图形用户界面模拟聊天室
- java.lang.RuntimeException: Method setUp in android.test.ApplicationTestCase not mocked. See http://g.co/androidstudio/not-mocked for details.
- ASP.NET Web API(二):安全验证之使用HTTP基本认证
- 朝鮮歷史筆寫本原稿 (韓長庚編)
- Win7系统无法清除DNS缓存的解决方法
- Java Annotation
- JQ-has和hasClass
- 后台分页实现方式总结
- CentOS 6.5 安装与配置LAMP FTP
- NodeJS 跨语言子进程持续通讯
- 试用mysql的infobright引擎
- A*搜索 概念
- 让Nginx处理Django的静态文件
- 试用mysql的infobright引擎
- java.lang.IllegalStateException: No activity
- 导读ICML2016 - Learning Convolutional Neural Networks for Graphs
- 羽毛球单打和双打的有效边界区域
- 严重:The web application [web01] appears to have started a thread named ...
- 华为OJ平台——计算字符串的相似度