flume入门 log4j 输出日志到flume
2017-07-11 09:22
274 查看
将log4j产生的日志直接输出到flume控制台
该配置文件中 配置了一个代理a1
在代理agent中配置了一个source(源)一个sink(接收器)和一个channel(通道),分别为:r1,k1,c1
r1的类型定义为avro(序列化),对应该类型参数为bind和port 分别为0.0.0.0和4141
k1的类型定义为logger,直接输出到日志文件中
cl的类型定义为内存方式,设置其参数capactiy和transactionCapacity分别为1000和1000
指定r1和k1的channel为c1
将上述配置中a1.sinks.k1.type=logger注释,把a1.sinks其他的给打开,就是将日志输入到文件
flume-ng :flume 命令
agent:运行一个Flume Agent
-c:在指定目录下使用配置 use configs in directory
-f:指定配置文件,这个配置文件必须在全局选项的–conf(-c)参数定义的目录下
–name:Agent的名称(必填)
-Dflume.root.logger=INFO,console 该参数将会把flume的日志输出到console
1.编写客户端
package com.study.test; import org.apache.commons.logging.Log; import org.apache.commons.logging.LogFactory; import java.util.Date; public class WriteLog { protected static final Log logger = LogFactory.getLog(WriteLog.class); public static void main(String[] args) throws InterruptedException { while (true) { // 每隔两秒log输出一下当前系统时间戳 logger.info(new Date().getTime()); //System.out.println(new Date().getTime()); Thread.sleep(2000); try { throw new Exception("exception msg"); } catch (Exception e) { logger.error("error:" + e.getMessage()); } } } }
2.pom.xml
<?xml version="1.0" encoding="UTF-8"?> <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"> <modelVersion>4.0.0</modelVersion> <groupId>com.study.test</groupId> <artifactId>log4j2flume</artifactId> <version>1.0-SNAPSHOT</version> <dependencies> <dependency> <groupId>org.apache.flume.flume-ng-clients</groupId> <artifactId>flume-ng-log4jappender</artifactId> <version>1.7.0</version> </dependency> <dependency> <groupId>org.slf4j</groupId> <artifactId>slf4j-simple</artifactId> <version>1.6.1</version> </dependency> </dependencies> </project>
3. log4j.properties
log4j.rootLogger=INFO,flume log4j.appender.console=org.apache.log4j.ConsoleAppender log4j.appender.console.target=System.out log4j.appender.console.layout=org.apache.log4j.PatternLayout log4j.appender.console.layout.ConversionPattern="%d{yyyy-MM-dd HH:mm:ss} %p [%c:%L] - %m%n log4j.appender.flume = org.apache.flume.clients.log4jappender.Log4jAppender log4j.appender.flume.Hostname = 10.45.17.66 log4j.appender.flume.Port = 4444 log4j.appender.flume.UnsafeMode = true log4j.appender.flume.layout=org.apache.log4j.PatternLayout log4j.appender.flume.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss} %p [%c:%L] - %m%n
4.配置flume 建立testlog2flume.conf
a1.sources = r1 a1.sinks = k1 a1.channels = c1 # Describe/configure the source a1.sources.r1.type = avro a1.sources.r1.bind =0.0.0.0 a1.sources.r1.port = 4444 # Describe the sink a1.sinks.k1.type=logger #a1.sinks.k1.type = file_roll #a1.sinks.k1.sink.directory = /data/soft/flume/tmp #a1.sinks.k1.sink.rollInterval=86400 #a1.sinks.k1.sink.batchSize=100 #a1.sinks.k1.sink.serializer=text #a1.sinks.k1.sink.serializer.appendNewline = false # Use a channel which buffers events in memory a1.channels.c1.type = memory a1.channels.c1.capacity = 1000 a1.channels.c1.transactionCapacity = 1000 # Bind the source and sink to the channel a1.sources.r1.channels = c1 a1.sinks.k1.channel = c1
该配置文件中 配置了一个代理a1
在代理agent中配置了一个source(源)一个sink(接收器)和一个channel(通道),分别为:r1,k1,c1
r1的类型定义为avro(序列化),对应该类型参数为bind和port 分别为0.0.0.0和4141
k1的类型定义为logger,直接输出到日志文件中
cl的类型定义为内存方式,设置其参数capactiy和transactionCapacity分别为1000和1000
指定r1和k1的channel为c1
将上述配置中a1.sinks.k1.type=logger注释,把a1.sinks其他的给打开,就是将日志输入到文件
5.到flume目录下启动flume
bin/flume-ng agent -c conf -f conf/testlog2flume.conf --name a1 -Dflume.root.logger=INFO,console
flume-ng :flume 命令
agent:运行一个Flume Agent
-c:在指定目录下使用配置 use configs in directory
-f:指定配置文件,这个配置文件必须在全局选项的–conf(-c)参数定义的目录下
–name:Agent的名称(必填)
-Dflume.root.logger=INFO,console 该参数将会把flume的日志输出到console
相关文章推荐
- Flume-ng在windows环境搭建并测试+log4j日志通过Flume输出到HDFS
- flume学习(一):log4j直接输出日志到flume
- 详解log4j2(下) - Async/MongoDB/Flume Appender 按日志级别区分文件输出
- flume学习(一):log4j直接输出日志到flume
- flume学习(一):log4j直接输出日志到flume
- log4j输出日志到flume
- 详解log4j2(下) - Async/MongoDB/Flume Appender 按日志级别区分文件输出
- flume学习(一):log4j直接输出日志到flume
- flume学习(一):log4jAppender直接输出日志到flume的avro-source
- 详解log4j2(下) - Async/MongoDB/Flume Appender 按日志级别区分文件输出
- 关键字驱动框架入门教程-9-用log4j输出日志
- flume学习(一):log4j直接输出日志到flume
- flume使用(三):实时log4j日志通过flume输出到MySql数据库
- 详解log4j2(下) - Async/MongoDB/Flume Appender 按日志级别区分文件输出
- log4j结合apache flume输出日志到apache kafka
- log4j输出日志到flume-ng(个人记录)
- log4j直接输出日志到flume
- log4j输出日志到flume
- 以包为单位控制Log4j的日志级别和输出(转)
- log4j输出到指定日志文件