Spark学习八：spark streaming与flume和kafka集成

Spark学习八：spark streaming与flume和kafka集成

标签（空格分隔）： Spark

Spark学习八spark streaming与flume和kafka集成
一Kafka

二flume和kafka的集成

三kafka和spark streaming的集成方式一kafka推送

四kafka和spark streaming的集成方式一spark streaam主动获取

五spark stream的高级应用updateStateByKey实现累加功能

六spark stream的高级应用窗口函数

一，Kafka

1，概述











2,kafka的安装

解压安装文件

修改server.properties文件：

############################# Log Basics #############################

# A comma seperated list of directories under which to store log files
log.dirs=/opt/app/kafka_2.10-0.8.2.1/ka-logs

############################# Zookeeper #############################

# Zookeeper connection string (see zookeeper docs for details).
# This is a comma separated host:port pairs, each corresponding to a zk
# server. e.g. "127.0.0.1:3000,127.0.0.1:3001,127.0.0.1:3002".
# You can also append an optional chroot string to the urls to specify the
# root directory for all kafka znodes.
zookeeper.connect=study.com.cn:2181

启动zookeeper:

bin/zkServer.sh start

启动kafka:

bin/kafka-server-start.sh config/server.properties

bin/kafka-server-start.sh -daemon config/server.properties   //运行在后台

创建topic

bin/kafka-topics.sh --create --zookeeper study.com.cn:2181 --replication-factor 1 --partitions 1 --topic topicTest

bin/kafka-topics.sh --list --zookeeper study.com.cn:2181

推送信息(生产者)：

bin/kafka-console-producer.sh --broker-list study.com.cn:9092 --topic topicTest
eeeeeeeeeeeeeeeeeeeeeeeeeeee

接受信息（消费者）：

bin/kafka-console-consumer.sh --zookeeper study.com.cn:2181 --topic topicTest --from-beginning

二，flume和kafka的集成

创建topicFlume:

bin/kafka-topics.sh --create --zookeeper study.com.cn:2181 --replication-factor 1 --partitions 1 --topic topicFlume

bin/kafka-topics.sh --list --zookeeper study.com.cn:2181

修改flume-kafka.xml的配置：

agent002.sources = sources002
agent002.channels = channels002
agent002.sinks = sinks002

## define sources
agent002.sources.sources002.type = exec
agent002.sources.sources002.command = tail -F /opt/app/apache-flume-1.5.0-bin/monitor/log.input

## define channels
agent002.channels.channels002.type = memory
agent002.channels.channels002.capacity = 10000
agent002.channels.channels002.transactionCapacity = 10000
agent002.channels.channels002.byteCapacityBufferPercentage = 20
agent002.channels.channels002.byteCapacity = 800000

##define sinks
agent002.sinks.sinks002.type =org.apache.flume.sink.kafka.KafkaSink
agent002.sinks.sinks002.brokerList=study.com.cn:9092
agent002.sinks.sinks002.topic=topicFlume

##relationship
agent002.sources.sources002.channels = channels002
agent002.sinks.sinks002.channel = channels002

启动flume:

bin/flume-ng agent --conf conf --name agent002 --conf-file conf/flume-kafka001.properties -Dflume.root.logger=INFO,console

启动kafka消费者：

bin/kafka-console-consumer.sh --zookeeper study.com.cn:2181 --topic topicFlume --from-beginning

测试：

echo "ddddddddddddddddddddd" >>log.input

三，kafka和spark streaming的集成方式一（kafka推送）

创建topicFlume:

bin/kafka-topics.sh --create --zookeeper study.com.cn:2181 --replication-factor 1 --partitions 1 --topic topicSpark

bin/kafka-topics.sh --list --zookeeper study.com.cn:2181

准备需要的jar包：



启动spark本地应用：

bin/spark-shell \
--jars /opt/app/spark-1.3.0-bin-2.5.0/externaljars/kafka_2.10-0.8.2.1.jar,\
/opt/app/spark-1.3.0-bin-2.5.0/externaljars/kafka-clients-0.8.2.1.jar,\
/opt/app/spark-1.3.0-bin-2.5.0/externaljars/metrics-core-2.2.0.jar,\
/opt/app/spark-1.3.0-bin-2.5.0/externaljars/spark-streaming-kafka_2.10-1.3.0.jar,\
/opt/app/spark-1.3.0-bin-2.5.0/externaljars/zkclient-0.3.jar \
--master local[2]

spark源码:

import org.apache.spark._
import org.apache.spark.streaming._
import org.apache.spark.streaming.StreamingContext._
import org.apache.spark.streaming.kafka._

val ssc = new StreamingContext(sc, Seconds(30))

val topicMap = Map("topicSpark" -> 1)
val kafkaStream = KafkaUtils.createStream(ssc, "study.com.cn:2181", "topicGroup", topicMap).map(_._2)

val wordCountDStream = kafkaStream.flatMap(_.split(" ")).map((_, 1)).reduceByKey(_ + _)

wordCountDStream.print()

ssc.start()
ssc.awaitTermination()

测试:

bin/kafka-console-producer.sh --broker-list study.com.cn:9092 --topic topicSpark
eeeeee eeeeeeeeeeeeeeeeeeeee hadoop

四，kafka和spark streaming的集成方式一（spark streaam主动获取）

创建topicFlume:

bin/kafka-topics.sh --create --zookeeper study.com.cn:2181 --replication-factor 1 --partitions 1 --topic topicSpark

bin/kafka-topics.sh --list --zookeeper study.com.cn:2181

准备需要的jar包：



启动spark本地应用：

bin/spark-shell \
--jars /opt/app/spark-1.3.0-bin-2.5.0/externaljars/kafka_2.10-0.8.2.1.jar,\
/opt/app/spark-1.3.0-bin-2.5.0/externaljars/kafka-clients-0.8.2.1.jar,\
/opt/app/spark-1.3.0-bin-2.5.0/externaljars/metrics-core-2.2.0.jar,\
/opt/app/spark-1.3.0-bin-2.5.0/externaljars/spark-streaming-kafka_2.10-1.3.0.jar,\
/opt/app/spark-1.3.0-bin-2.5.0/externaljars/zkclient-0.3.jar \
--master local[2]

spark源码:

import kafka.serializer.StringDecoder
import org.apache.spark._
import org.apache.spark.streaming._
import org.apache.spark.streaming.StreamingContext._
import org.apache.spark.streaming.kafka._

val ssc = new StreamingContext(sc, Seconds(30))

// kafkaParams: Map[String, String]
val kafkaParams = Map("metadata.broker.list" -> "study.com.cn:9092")
// topics: Set[String]
val topics = Set("topicSpark")
val kafkaStream = KafkaUtils.createDirectStream[String, String, StringDecoder, StringDecoder](ssc, kafkaParams, topics).map(_._2)

val wordCountDStream = kafkaStream.flatMap(_.split(" ")).map((_, 1)).reduceByKey(_ + _)

wordCountDStream.print()

ssc.start()
ssc.awaitTermination()

测试:

bin/kafka-console-producer.sh --broker-list study.com.cn:9092 --topic topicSpark
eeeeee eeeeeeeeeeeeeeeeeeeee hadoop

五，spark stream的高级应用updateStateByKey实现累加功能

创建topicFlume:

bin/kafka-topics.sh --create --zookeeper study.com.cn:2181 --replication-factor 1 --partitions 1 --topic topicSpark

bin/kafka-topics.sh --list --zookeeper study.com.cn:2181

准备需要的jar包：



启动spark本地应用：

bin/spark-shell \
--jars /opt/app/spark-1.3.0-bin-2.5.0/externaljars/kafka_2.10-0.8.2.1.jar,\
/opt/app/spark-1.3.0-bin-2.5.0/externaljars/kafka-clients-0.8.2.1.jar,\
/opt/app/spark-1.3.0-bin-2.5.0/externaljars/metrics-core-2.2.0.jar,\
/opt/app/spark-1.3.0-bin-2.5.0/externaljars/spark-streaming-kafka_2.10-1.3.0.jar,\
/opt/app/spark-1.3.0-bin-2.5.0/externaljars/zkclient-0.3.jar \
--master local[2]

spark源码:

import kafka.serializer.StringDecoder
import org.apache.spark._
import org.apache.spark.streaming._
import org.apache.spark.streaming.StreamingContext._
import org.apache.spark.streaming.kafka._

val ssc = new StreamingContext(sc, Seconds(30))

ssc.checkpoint(".")
val kafkaParams = Map("metadata.broker.list" -> "study.com.cn:9092")
// topics: Set[String]
val topics = Set("topicSpark")

val kafkaStream = KafkaUtils.createDirectStream[String, String, StringDecoder, StringDecoder](ssc, kafkaParams, topics).map(_._2)

val wordDStream = kafkaStream.flatMap(_.split(" ")).map((_, 1))

// definition ,Option[Int]
// SessionInfo: sessionId, ip, count
val updateFunc = (values: Seq[Int], state: Option[Int]) => {
val currentCount = values.sum
val previousCount = state.getOrElse(0)
Some(currentCount + previousCount)
}

// wordDStream[(K, V)]
val wordCountDStream = wordDStream.updateStateByKey(updateFunc)

wordCountDStream.print()

ssc.start()
ssc.awaitTermination()

测试:

bin/kafka-console-producer.sh --broker-list study.com.cn:9092 --topic topicSpark
eeeeee eeeeeeeeeeeeeeeeeeeee hadoop

六，spark stream的高级应用窗口函数

创建topicFlume:

bin/kafka-topics.sh --create --zookeeper study.com.cn:2181 --replication-factor 1 --partitions 1 --topic topicSpark

bin/kafka-topics.sh --list --zookeeper study.com.cn:2181

准备需要的jar包：



启动spark本地应用：

bin/spark-shell \
--jars /opt/app/spark-1.3.0-bin-2.5.0/externaljars/kafka_2.10-0.8.2.1.jar,\
/opt/app/spark-1.3.0-bin-2.5.0/externaljars/kafka-clients-0.8.2.1.jar,\
/opt/app/spark-1.3.0-bin-2.5.0/externaljars/metrics-core-2.2.0.jar,\
/opt/app/spark-1.3.0-bin-2.5.0/externaljars/spark-streaming-kafka_2.10-1.3.0.jar,\
/opt/app/spark-1.3.0-bin-2.5.0/externaljars/zkclient-0.3.jar \
--master local[2]

spark源码:

import kafka.serializer.StringDecoder
import org.apache.spark._
import org.apache.spark.streaming._
import org.apache.spark.streaming.StreamingContext._
import org.apache.spark.streaming.kafka._

val ssc = new StreamingContext(sc, Seconds(5))

// kafkaParams: Map[String, String]
val kafkaParams = Map("metadata.broker.list" -> "bigdata-senior01.ibeifeng.com:9092")
// topics: Set[String]
val topics = Set("sparkTopic")
val kafkaStream = KafkaUtils.createDirectStream[String, String, StringDecoder, StringDecoder](ssc, kafkaParams, topics).map(_._2)

// val wordCountDStream = kafkaStream.flatMap(_.split(" ")).map((_, 1)).reduceByKey(_ + _)
// 每隔十五秒，统计最近五分钟之内的情况
val wordDStream = kafkaStream.flatMap(_.split(" ")).map((_, 1))
//
val reduceFunc = (v1: Int, v1: Int) =>{
v1 + v2
}
val wordCountDStream = wordDStream.reduceByKeyAndWindow(reduceFunc, Seconds(5 * 12 * 5), Seconds(3 * 5))

wordCountDStream.print()

ssc.start()
ssc.awaitTermination()