Flume学习笔记-windows简单应用

下载安装

http://flume.apache.org/download.html

HelloWorld

flume-conf.properties.template的默认配置为，source为seq，即产生一个0,1,2这样的数字序列，主要为了测试，sink为logger，即打印序列到日志中，日志配置在conf/log4j.properties中，修改为：flume.log.dir=../logs

agent.sources = seqGenSrc
agent.channels = memoryChannel
agent.sinks = loggerSink

agent.sources.seqGenSrc.type = seq
agent.sources.seqGenSrc.channels = memoryChannel
agent.sinks.loggerSink.type = logger
agent.sinks.loggerSink.channel = memoryChannel
agent.channels.memoryChannel.type = memory
agent.channels.memoryChannel.capacity = 100

复制flume-conf.properties.template为helloworld-conf.properties，启动：

bin>flume-ng agent --conf ../conf --conf-file ../conf/helloworld-conf.properties  --name agent

其中--conf指定conf目录位置，即为上层目录，--conf-file指定具体配置文件位置
--name指定agent的名称，一个配置文件里可能配置多个agent

然后在logs目录下可以看到日志文件中的日志：

输出到HDFS,HBase中

agent1.sources = seqSource
agent1.channels = hdfsChannel hbaseChannel
agent1.sinks = hdfsSink hbaseSink

agent1.sources.seqSource.type = seq
agent1.sources.seqSource.channels = hdfsChannel hbaseChannel

agent1.channels.hdfsChannel.type = memory
agent1.channels.hdfsChannel.capacity = 10000
agent1.channels.hbaseChannel.type = memory
agent1.channels.hbaseChannel.capacity = 10000

agent1.sinks.hdfsSink.channel = hdfsChannel
agent1.sinks.hdfsSink.type = hdfs
agent1.sinks.hdfsSink.hdfs.fileType=DataStream
agent1.sinks.hdfsSink.hdfs.writeFormat=Text
agent1.sinks.hdfsSink.hdfs.path = hdfs://127.0.0.1:9000/flume/%Y%m%d
agent1.sinks.hdfsSink.hdfs.filePrefix=log_%H%M
agent1.sinks.hdfsSink.hdfs.fileSuffix=.txt
agent1.sinks.hdfsSink.hdfs.rollInterval=30
agent1.sinks.hdfsSink.hdfs.rollCount=0
agent1.sinks.hdfsSink.hdfs.rollSize=0
agent1.sinks.hdfsSink.hdfs.useLocalTimeStamp=true

agent1.sinks.hbaseSink.channel = hbaseChannel
agent1.sinks.hbaseSink.type = hbase
agent1.sinks.hbaseSink.table = flume
agent1.sinks.hbaseSink.columnFamily = seqence

其中hdfs.rollInterval为30，即30S产生一个文件，为了测试，下面两个配置为0，否则不会30S生成一个文件

hdfs.rollSize表示文件大小超过该值后，会生成新的文件，配置为0表示不基于文件大小生成文件

hdfs.rollCount默认为10，即10个事件生成一个文件，配置为0表示不基于事件数生成文件

HBase要先创建好表
HDFS的目录/flume/%Y%m%d会自动生成，使用useLocalTimeStamp时间填充

启动方式类似，这里的agent改为了agent1

D:\apache-flume-1.7.0-bin\bin>flume-ng agent --conf ../conf --conf-file ../conf/helloworld-conf.properties --name agent1