flume与kafka整合

flume与kafka整合

前提：

flume安装和测试通过，可参考：http://www.cnblogs.com/rwxwsblog/p/5800300.html

kafka安装和测试通过，可参考：http://www.cnblogs.com/rwxwsblog/p/5800224.html

在上诉条件满足的情况下才能进行flume和kafka的整合。

flume与kafka整合(注：flume和kafka两台机器要能连通)

修改/usr/local/flume/conf/flume-conf.properties

agent.sinks.s1.type = org.apache.flume.sink.kafka.KafkaSink
agent.sinks.s1.topic = mytopic
agent.sinks.s1.brokerList = localhost:9092
agent.sinks.s1.requiredAcks = 1
agent.sinks.s1.batchSize = 20
agent.sinks.s1.channel = c1

启动kafka

启动zookeeper

/usr/local/kafka/bin/zookeeper-server-start.sh
-daemon config/zookeeper.properties

启动kafka

/usr/local/kafka/bin/kafka-server-start.sh
config/server.properties

创建主题

test

/usr/local/kafka/bin/kafka-topics.sh
--create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test

启动flume

/usr/local/flume/bin/flume-ng
agent --conf conf -f conf/flume-conf.properties -n agent&

验证

新开一个客户端

telnet localhost 8888

[root@localhost ~]# telnet localhost 8888
Trying ::1...
telnet: connect to address ::1: Connection refused
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
test
OK
chat
OK
this is a flume message to kafka
OK
from flume message.
OK
it's flume and kafka message。
OK

kafka查看消息接收情况

[root@localhost kafka]# /usr/local/kafka/bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic test --from-beginning
test
chat
this is a flume message to kafka
from flume message.
it's flume and kafka message。

如果出现上述效果则说明flume和kafka整合成功。