【Flume】flume多个输出分支的配置,多sink,多channel
2015-02-11 09:44
316 查看
这里就简单贴个配置吧,有问题欢迎留言探讨
sink的配置
##############################define [sink] begin##############################
#define the sink k1
a1.sinks.k1.type=hdfs
a1.sinks.k1.channel=c1
a1.sinks.k1.hdfs.useLocalTimeStamp=true
a1.sinks.k1.hdfs.path=hdfs://192.168.11.177:9000/flume/events/%Y/%m/%d
a1.sinks.k1.hdfs.filePrefix=cmcc-%H
a1.sinks.k1.hdfs.fileType=DataStream
a1.sinks.k1.hdfs.minBlockReplicas=1
a1.sinks.k1.hdfs.rollInterval=3600
a1.sinks.k1.hdfs.rollSize=0
a1.sinks.k1.hdfs.rollCount=0
a1.sinks.k1.hdfs.idleTimeout=0
#define the sink k2
a1.sinks.k2.channel=c2
a1.sinks.k2.type=com.cmcc.chiwei.kafka.CmccKafkaSink
a1.sinks.k2.metadata.broker.list=192.168.11.174:9092,192.168.11.175:9092,192.168.11.196:9092
a1.sinks.k2.partition.key=0
a1.sinks.k2.partitioner.class=com.cmcc.chiwei.kafka.CmccPartition
a1.sinks.k2.serializer.class=kafka.serializer.StringEncoder
a1.sinks.k2.request.required.acks=0
a1.sinks.k2.cmcc.encoding=UTF-8
a1.sinks.k2.cmcc.topic.name=cmcc
a1.sinks.k2.producer.type=async
a1.sinks.k2.batchSize=100
##############################define [sink] end##############################这里的是将同样的数据无差异sink到多个输出端,所以通道是复制模式
a1.sources.r1.selector.type=replicating一个是kafka,为了提高性能,用内存通道,一个到hdfs,离线分析,用文件通道
##############################define [channel] begin##############################
#define the channel c1
a1.channels.c1.type=file
a1.channels.c1.checkpointDir=/home/flume/flumeCheckpoint
a1.channels.c1.dataDirs=/home/flume/flumeData,/home/flume/flumeDataExt
a1.channels.c1.capacity=2000000
a1.channels.c1.transactionCapacity=100
#define the channel c2
a1.channels.c2.type=memory
a1.channels.c2.capacity=2000000
a1.channels.c2.transactionCapacity=100
##############################define [channel] end##############################以上就是无差异数据输出多个sink的配置,如果你想不同数据到不同的sink,需要用到通道的多路复用模式,请参考【通道的多路复用】
sink的配置
##############################define [sink] begin##############################
#define the sink k1
a1.sinks.k1.type=hdfs
a1.sinks.k1.channel=c1
a1.sinks.k1.hdfs.useLocalTimeStamp=true
a1.sinks.k1.hdfs.path=hdfs://192.168.11.177:9000/flume/events/%Y/%m/%d
a1.sinks.k1.hdfs.filePrefix=cmcc-%H
a1.sinks.k1.hdfs.fileType=DataStream
a1.sinks.k1.hdfs.minBlockReplicas=1
a1.sinks.k1.hdfs.rollInterval=3600
a1.sinks.k1.hdfs.rollSize=0
a1.sinks.k1.hdfs.rollCount=0
a1.sinks.k1.hdfs.idleTimeout=0
#define the sink k2
a1.sinks.k2.channel=c2
a1.sinks.k2.type=com.cmcc.chiwei.kafka.CmccKafkaSink
a1.sinks.k2.metadata.broker.list=192.168.11.174:9092,192.168.11.175:9092,192.168.11.196:9092
a1.sinks.k2.partition.key=0
a1.sinks.k2.partitioner.class=com.cmcc.chiwei.kafka.CmccPartition
a1.sinks.k2.serializer.class=kafka.serializer.StringEncoder
a1.sinks.k2.request.required.acks=0
a1.sinks.k2.cmcc.encoding=UTF-8
a1.sinks.k2.cmcc.topic.name=cmcc
a1.sinks.k2.producer.type=async
a1.sinks.k2.batchSize=100
##############################define [sink] end##############################这里的是将同样的数据无差异sink到多个输出端,所以通道是复制模式
a1.sources.r1.selector.type=replicating一个是kafka,为了提高性能,用内存通道,一个到hdfs,离线分析,用文件通道
##############################define [channel] begin##############################
#define the channel c1
a1.channels.c1.type=file
a1.channels.c1.checkpointDir=/home/flume/flumeCheckpoint
a1.channels.c1.dataDirs=/home/flume/flumeData,/home/flume/flumeDataExt
a1.channels.c1.capacity=2000000
a1.channels.c1.transactionCapacity=100
#define the channel c2
a1.channels.c2.type=memory
a1.channels.c2.capacity=2000000
a1.channels.c2.transactionCapacity=100
##############################define [channel] end##############################以上就是无差异数据输出多个sink的配置,如果你想不同数据到不同的sink,需要用到通道的多路复用模式,请参考【通道的多路复用】
相关文章推荐
- Flume-ng:multi sink one channel两种配置方式的对比
- Flume开发中常用组件source,channel,sink等配置
- Flume单Channel多Sink配置
- 【Flume】【源码分析】深入flume-ng的三大组件——source,channel,sink
- flume坑之channel.transactionCapacity和HdfsSink.batchSize
- Flume-ng生产环境实践(三)实现文件sink,按照固定格式目录输出
- Flume内置channel,source,sink三组件介绍
- Flume-ng生产环境实践(三)实现文件sink,按照固定格式目录输出
- Flume监听文件目录sink至hdfs配置
- Flume监听文件目录sink至hdfs配置
- MTK MT65XX平台开发分支下配置多个项目的规范说明
- 【Flume】flume中sink到hdfs,文件系统频繁产生文件,文件滚动配置不起作用?
- MT6573台开发分支下配置多个项目的规范说明_mi
- Flume中的HDFS Sink配置参数说明
- flume的source, channel, sink 列表
- Flume(4)实用环境搭建:source(spooldir)+channel(file)+sink(hdfs)方式
- Flume监听文件目录sink至hdfs配置
- Flume NG 学习笔记(十) Transaction、Sink、Source和Channel开发
- 【Flume】【源码分析】flume中sink到hdfs,文件系统频繁产生文件,文件滚动配置不起作用?
- apache配置一个域名读取多个路径代码(包括主干和分支)