Spark学习笔记(23)Transformation、Action等源码图解
2017-08-31 10:55
381 查看
本期内容:
1. Spark Streaming的Transformation、Action源码图解
2. Spark Streaming的Input、Output源码图解
StreamingContext成员:socketStream
/**
* Create a input stream from TCP source hostname:port. Data is received using
* a TCP socket and the receive bytes it interepreted as object using the given
* converter.
* @param hostname Hostname to connect to for receiving data
* @param port Port to connect to for receiving data
* @param converter Function to convert the byte stream to objects
* @param storageLevel Storage level to use for storing the received objects
* @tparam T Type of the objects received (after converting bytes to objects)
*/
def socketStream[T: ClassTag](
hostname: String,
port: Int,
converter: (InputStream) => Iterator[T],
storageLevel: StorageLevel
): ReceiverInputDStream[T] = {
new SocketInputDStream[T](this, hostname, port, converter, storageLevel)
}
SocketInputDStream:
class SocketInputDStream[T: ClassTag](
ssc_ : StreamingContext,
host: String,
port: Int,
bytesToObjects: InputStream => Iterator[T],
storageLevel: StorageLevel
) extends ReceiverInputDStream[T](ssc_) {
def getReceiver(): Receiver[T] = {
new SocketReceiver(host, port, bytesToObjects, storageLevel)
}
}
ReceiverInputDStream:
abstract class ReceiverInputDStream[T: ClassTag](ssc_ : StreamingContext)
extends InputDStream[T](ssc_) {
...
InputDStream:
ssc.graph.addInputStream(this)
把ImputDStream放入到了graph中。
DStream.foreachRDD产生ForEachDStream对象,该对象通过register对象也放入了graph中。
1. Spark Streaming的Transformation、Action源码图解
2. Spark Streaming的Input、Output源码图解
StreamingContext成员:socketStream
/**
* Create a input stream from TCP source hostname:port. Data is received using
* a TCP socket and the receive bytes it interepreted as object using the given
* converter.
* @param hostname Hostname to connect to for receiving data
* @param port Port to connect to for receiving data
* @param converter Function to convert the byte stream to objects
* @param storageLevel Storage level to use for storing the received objects
* @tparam T Type of the objects received (after converting bytes to objects)
*/
def socketStream[T: ClassTag](
hostname: String,
port: Int,
converter: (InputStream) => Iterator[T],
storageLevel: StorageLevel
): ReceiverInputDStream[T] = {
new SocketInputDStream[T](this, hostname, port, converter, storageLevel)
}
SocketInputDStream:
class SocketInputDStream[T: ClassTag](
ssc_ : StreamingContext,
host: String,
port: Int,
bytesToObjects: InputStream => Iterator[T],
storageLevel: StorageLevel
) extends ReceiverInputDStream[T](ssc_) {
def getReceiver(): Receiver[T] = {
new SocketReceiver(host, port, bytesToObjects, storageLevel)
}
}
ReceiverInputDStream:
abstract class ReceiverInputDStream[T: ClassTag](ssc_ : StreamingContext)
extends InputDStream[T](ssc_) {
...
InputDStream:
ssc.graph.addInputStream(this)
把ImputDStream放入到了graph中。
DStream.foreachRDD产生ForEachDStream对象,该对象通过register对象也放入了graph中。
相关文章推荐
- 第24课:Spark Streaming的Transformation、Action、Input和Output源码图解
- Spark学习笔记(24)StreamingContext及JobScheduler源码图解
- Spark学习笔记 --- Transformation and Action
- Spark学习笔记--Transformation 和 action
- Spark Streaming的Transformation、Action、Input和Output源码图解(第24课)
- Spark的transformation 和 action的操作学习笔记
- cocos2d-x 2.X demo学习笔记 3 ----ActionTest源码 主要是滚动菜单的操作
- 第67讲:Scala并发编程匿名Actor、消息传递、偏函数实战解析及其在Spark源码中的应用解析学习笔记
- spark源码之RDD(2)transformation和action
- 第74讲:从Spark源码的角度思考Scala中的模式匹配学习笔记
- 第46讲:ClassTag 、Manifest、ClassManifest、TypeTag代码实战及其在Spark中的应用源码解析学习笔记
- Scala类型约束代码实战及其在Spark中的应用源码解析之Scala学习笔记-39
- spark源码之RDD(2)transformation和action
- 第44讲:Scala中View Bounds代码实战及其在Spark中的应用源码解析学习笔记
- 2016书单总结--spring-boot-in-action学习笔记--源码简析
- Scala中隐式参数与隐式转换的联合使用实战详解及其在Spark中的应用源码解析之Scala学习笔记-51
- Scala中上下文界定内幕中的隐式参数与隐式参数的实战详解及其在Spark中的应用源码解析之Scala学习笔记-52
- Scala中Variance代码实战及其在Spark中的应用源码解析之Scala学习笔记-40
- 第43讲:Scala中类型变量Bounds代码实战及其在Spark中的应用源码解析学习笔记
- Scala中类型变量Bounds代码实战及其在Spark中的应用源码解析之Scala学习笔记-34