Flume 收集Nginx日志到Hdfs Tail-to-hdfs sink
2014-03-26 11:04
591 查看
nginx,access.log日志约8000条/s,每100w条数据约253M,需要2min
[/code]
[code]agent1.sources = source1 agent1.sinks = sink1 agent1.channels = channel1 # Describe/configure spooldir source1 #agent1.sources.source1.type = spooldir #agent1.sources.source1.spoolDir = /var/log/apache/flumeSpool1 #agent1.sources.source1.fileHeader = true # Describe/configure tail -F source1 agent1.sources.source1.type = exec agent1.sources.source1.command = tail -F /tmp/log.log agent1.sources.source1.channels = channel1 # Describe/configure nc source1 #agent1.sources.source1.type = netcat #agent1.sources.source1.bind = localhost #agent1.sources.source1.port = 44444 #configure host for source agent1.sources.source1.interceptors = i1 agent1.sources.source1.interceptors.i1.type = host agent1.sources.source1.interceptors.i1.hostHeader = hostname # Describe sink1 #agent1.sinks.sink1.type = logger agent1.sinks.sink1.type = hdfs #a1.sinks.k1.channel = c1 #agent1.sinks.sink1.hdfs.path =hdfs://xxx:9000/tmp/tail/%y-%m-%d/%H%M%S agent1.sinks.sink1.hdfs.path =hdfs://xxx:9000/tmp/tail/%y-%m-%d/%H agent1.sinks.sink1.hdfs.filePrefix = %{hostname}/events- agent1.sinks.sink1.hdfs.maxOpenFiles = 5000 agent1.sinks.sink1.hdfs.batchSize= 500 agent1.sinks.sink1.hdfs.fileType = DataStream agent1.sinks.sink1.hdfs.writeFormat =Text agent1.sinks.sink1.hdfs.rollSize = 0 agent1.sinks.sink1.hdfs.rollCount = 1000000 agent1.sinks.sink1.hdfs.rollInterval = 600 #agent1.sinks.sink1.hdfs.round = true #agent1.sinks.sink1.hdfs.roundValue = 10 #agent1.sinks.sink1.hdfs.roundUnit = minute agent1.sinks.sink1.hdfs.useLocalTimeStamp = true # Use a channel which buffers events in memory agent1.channels.channel1.type = memory agent1.channels.channel1.keep-alive = 120 agent1.channels.channel1.capacity = 500000 agent1.channels.channel1.transactionCapacity = 600 # Bind the source and sink to the channel agent1.sources.source1.channels = channel1 agent1.sinks.sink1.channel = channel1
[/code]
相关文章推荐
- Nginx配置文件详细说明
- 使用python删除nginx缓存文件示例(python文件操作)
- nginx启动,重启,关闭命令
- 使用python删除nginx缓存文件示例(python文件操作)
- nginx 启动错误
- CentOS 安装配置 Apache, nginx, SVN, Trac 日志
- “惊群”,看看nginx是怎么解决它的
- nginx配置
- 源码配置nginx
- 【转】实战Nginx与PHP(FastCGI)的安装、配置与优化
- OPTIMIZING NGINX FOR HIGH TRAFFIC LOADS
- nginx缓存cache的几种方式
- 【转】如何正确配置Nginx+PHP
- Nginx 安装配置magento 1.8.0.0
- 通过apache,和nginx模块去除html中的空格和tab
- ubuntu13.04编译安装lnmp
- nginx交叉编译--好文章要转
- 不能忽略的Nginx做web服务器的favicon.ico图像找不到问题
- nginx日志定时切割脚本
- 分步LVS: 详解利用Keepalived+Nginx解决站点高可用性