模拟nginx+flume数据采集+Kafka集群

执行shell脚本模拟nginx生成日志：nginx.sh

for((i=0;i<=500000;i++));
do echo "i am lilei"+$i >> 1.log
done

• 1
• 2
• 3

执行flume程序进行数据采集: 
flume 任务文件 exec.conf

a1.sources = r1
a1.channels = c1
a1.sinks = k1

a1.sources.r1.type = exec
a1.sources.r1.command = tail -F /root/1.log
a1.sources.r1.channels = c1

a1.channels.c1.type = memory
a1.channels.c1.capacity = 10000
a1.channels.c1.transactionCapacity=100

a1.sinks.k1.type = org.apache.flume.sink.kafka.KafkaSink
a1.sinks.k1.topic=all1
a1.sinks.k1.brokerList=mini1:9092
a1.sinks.k1.requiredAcks=1
a1.sinks.k1.batchSize=20
a1.sinks.k1.channel=c1

• 1
• 2
• 3
• 4
• 5
• 6
• 7
• 8
• 9
• 10
• 11
• 12
• 13
• 14
• 15
• 16
• 17
• 18

启动flume 程序：

bin/flume-ng agent -c conf -f conf/exec.conf -n a1  -Dflume.root.logger=INFO,console

• 1

kafka 集群收集数据： 
启动kafka

bin/kafka-server-start.sh config/server.properties

• 1

创建topic

sh bin/kafka-console-consumer.sh --zookeeper mini1:2181 --from-beginning --topic all1

• 1

storm 程序进行单词统计 
https://github.com/JiyangM/stom/tree/master/src/main/java/cn/itcast/storm/kafkastormredis

顶

0

踩

0