关于Flume-ng那些事(四)
2014-03-23 00:00
148 查看
最后一章了,flume-ng的手册总是慢半拍,一些实例可以参考下手册总的图,会很有收获。
http://flume.apache.org/FlumeUserGuide.html 还是1.4版本,有些参数已经发生变化,注意更改。 贴个上hadoop hdfs的例子,Flume-ng write events to HDFS example: update flume-ng 1.3 example: agent config
#List sources, sinks and channels in the agent weblog-agent.sources = tail weblog-agent.sources.tail.interceptors = ts host weblog-agent.sources.tail.interceptors.ts.type = org.apache.flume.interceptor.TimestampInterceptor$Builder weblog-agent.sources.tail.interceptors.host.type = org.apache.flume.interceptor.HostInterceptor$Builder weblog-agent.sources.tail.interceptors.host.useIP = false weblog-agent.sources.tail.interceptors.host.preserveExisting = true weblog-agent.sinks = avro-forward-sink01 weblog-agent.channels = jdbc-channel01 #define the flow #webblog-agent sources config weblog-agent.sources.tail.channels = jdbc-channel01 weblog-agent.sources.tail.type = exec weblog-agent.sources.tail.restart = true #weblog-agent.sources.tail. = true weblog-agent.sources.tail.command = tail -f /opt/nginx/logs/access.log #weblog-agent.sources.tail.selector.type = replicating #avro sink properties weblog-agent.sinks.avro-forward-sink01.channel = jdbc-channel01 weblog-agent.sinks.avro-forward-sink01.type = hdfs weblog-agent.sinks.avro-forward-sink01.hdfs.path = hdfs://ttlsa-hadoop-master.:9000/user/flume/webtest/%{host}/%Y-%m-%d/ weblog-agent.sinks.avro-forward-sink01.hdfs.filePrefix = nginx weblog-agent.sinks.avro-forward-sink01.hdfs.kerberosPrincipal = flume@HADOOP.TTLSA.COM weblog-agent.sinks.avro-forward-sink01.hdfs.kerberosKeytab = /var/run/flume-ng/flume.keytab weblog-agent.sinks.avro-forward-sink01.hdfs.rollInterval = 120 weblog-agent.sinks.avro-forward-sink01.hdfs.rollSize = 0 weblog-agent.sinks.avro-forward-sink01.hdfs.rollCount = 0 weblog-agent.sinks.avro-forward-sink01.hdfs.fileType = DataStream #channels config weblog-agent.channels.jdbc-channel01.type = memory
http://flume.apache.org/FlumeUserGuide.html 还是1.4版本,有些参数已经发生变化,注意更改。 贴个上hadoop hdfs的例子,Flume-ng write events to HDFS example: update flume-ng 1.3 example: agent config
#List sources, sinks and channels in the agent weblog-agent.sources = tail weblog-agent.sources.tail.interceptors = ts host weblog-agent.sources.tail.interceptors.ts.type = org.apache.flume.interceptor.TimestampInterceptor$Builder weblog-agent.sources.tail.interceptors.host.type = org.apache.flume.interceptor.HostInterceptor$Builder weblog-agent.sources.tail.interceptors.host.useIP = false weblog-agent.sources.tail.interceptors.host.preserveExisting = true weblog-agent.sinks = avro-forward-sink01 weblog-agent.channels = jdbc-channel01 #define the flow #webblog-agent sources config weblog-agent.sources.tail.channels = jdbc-channel01 weblog-agent.sources.tail.type = exec weblog-agent.sources.tail.restart = true #weblog-agent.sources.tail. = true weblog-agent.sources.tail.command = tail -f /opt/nginx/logs/access.log #weblog-agent.sources.tail.selector.type = replicating #avro sink properties weblog-agent.sinks.avro-forward-sink01.channel = jdbc-channel01 weblog-agent.sinks.avro-forward-sink01.type = hdfs weblog-agent.sinks.avro-forward-sink01.hdfs.path = hdfs://ttlsa-hadoop-master.:9000/user/flume/webtest/%{host}/%Y-%m-%d/ weblog-agent.sinks.avro-forward-sink01.hdfs.filePrefix = nginx weblog-agent.sinks.avro-forward-sink01.hdfs.kerberosPrincipal = flume@HADOOP.TTLSA.COM weblog-agent.sinks.avro-forward-sink01.hdfs.kerberosKeytab = /var/run/flume-ng/flume.keytab weblog-agent.sinks.avro-forward-sink01.hdfs.rollInterval = 120 weblog-agent.sinks.avro-forward-sink01.hdfs.rollSize = 0 weblog-agent.sinks.avro-forward-sink01.hdfs.rollCount = 0 weblog-agent.sinks.avro-forward-sink01.hdfs.fileType = DataStream #channels config weblog-agent.channels.jdbc-channel01.type = memory
相关文章推荐
- 关于Flume-ng那些事(一)
- 关于Flume-ng那些事(二)
- 关于Flume-ng那些事(三)
- 【Angular】关于angular引用第三方组件库无法改变其组件样式 :host ::ng-deep
- 关于生产者-消费者-订阅者模式的那些事
- 关于Java程序员面试的那些事
- Qt中关于菜单的那些事
- Flume-ng-extends的应用
- 【Flume】【源码分析】深入flume-ng的三大组件——source,channel,sink
- 大数据架构:flume-ng+Kafka+Storm+HDFS 实时系统组合
- 关于cvScalar的那些事
- Android 关于GridView那些事
- iphone,ipad,关于icon图标的那些事(必须1024啦~~)
- 关于aircrack-ng的一些学习笔记及问题
- Flume-NG + HDFS + HIVE 日志收集分析
- 关于前端的那些事(转)
- Flume-og和Flume-ng的变化详解
- 【Hadoop】Flume NG Getting Started(Flume NG 新手入门指南)翻译
- 转载自爱维论坛关于网卡驱动那些事,经验之谈