Flume spooldir主动source示例
2017-07-05 23:33
423 查看
vim test02
8020rpc端口
rollInterval 时间间隔
rollSize 文件大小
rollCount 写的次数,配0则不考虑
三个条件最先满足则关闭流。
mkdir /opt/flume
在/opt/flume中添加文件即可。
# example.conf: A single-node Flume configuration # Name the components on this agent a1.sources = r1 a1.sinks = k1 a1.channels = c1 # Describe/configure the source a1.sources.r1.type =spooldir a1.sources.r1.spoolDir=/opt/flume # Describe the sink a1.sinks.k1.type=hdfs a1.sinks.k1.hdfs.path=hdfs://node1:8020/flume/%Y-%m-%d/%H%M a1.sinks.k1.hdfs.rollCount=0 a1.sinks.k1.hdfs.rollInterval=60 a1.sinks.k1.hdfs.rollSize=10240 a1.sinks.k1.hdfs.idleTimeout=3 a1.sinks.k1.hdfs.fileType=DataStream a1.sinks.k1.hdfs.useLocalTimeStamp=true a1.sinks.k1.hdfs.round=true a1.sinks.k1.hdfs.roundValue=5 a1.sinks.k1.hdfs.roundUnit=minute # Use a channel which buffers events in memory a1.channels.c1.type = memory a1.channels.c1.capacity = 1000 a1.channels.c1.transactionCapacity = 100 # Bind the source and sink to the channel a1.sources.r1.channels = c1 a1.sinks.k1.channel = c1
8020rpc端口
rollInterval 时间间隔
rollSize 文件大小
rollCount 写的次数,配0则不考虑
三个条件最先满足则关闭流。
mkdir /opt/flume
$ bin/flume-ng agent --conf apache-flume-1.6.0-bin/conf/ --conf-file test02 --name a1 -Dflume.root.logger=INFO,console
在/opt/flume中添加文件即可。
相关文章推荐
- Flume(4)实用环境搭建:source(spooldir)+channel(file)+sink(hdfs)方式
- flume配置-生产环境下 Taildir Source to kafka Sink
- Flume 1.7.0 新特性 - Taildir Source
- flume中的TaildirSource(2)
- flume spooldir bug修复
- flume监控spoolDir日志到HDFS整个流程小Demo
- flume监控spoolDir日志到HDFS(从日志产生到hdfs上一整套)
- flume的TaildirSource介绍及升级改造
- Flume Spooldir 源的一些问题
- flume使用(四):taildirSource多文件监控实时采集
- 【Java】【Fulme】Flume-NG源码阅读之SpoolDirectorySource
- Flume TailDirSource问题
- Flume netcat被动source示例
- Flume-NG源码阅读之SpoolDirectorySource(原创)
- Flume中的TaildirSource
- Flume-ng spoolDir目录监控踩过的坑
- flume 使用 spool source的时候字符集出错
- 【Java】【Fulme】Flume-NG源代码阅读之SpoolDirectorySource
- [ETL] Flume 理论与demo(Taildir Source & Hdfs Sink)
- Flume中的TaildirSource