Flume 安装与配置

Flume 安装与配置

0 前提：需要安装Java

0.1 从官网下载 jdk

http://www.oracle.com/technetwork/java/javase/downloads/index.html

我下载的是 jdk-7u79-linux-x64.tar.gz

0.2 解压

tar -zxvf jdk-7u79-linux-x64.tar.gz -C /usr/

0.3 设置环境变量

在/etc/profile最后添加

# set java environment
export JAVA_HOME=/usr/jdk1.7.0_79
export JRE_HOME=/usr/jdk1.7.0_79/jre
export CLASSPATH=.:$CLASSPATH:$JAVA_HOME/lib:$JRE_HOME/lib
export PATH=$PATH:$JAVA_HOME/bin:$JRE_HOME/bin

生效 source /etc/profile

1 Flume 下载与解压

从官网下载flume

http://flume.apache.org/download.html

我下载的是 apache-flume-1.6.0-bin.tar.gz

tar -zxvf /mnt/hgfs/share/apache-flume-1.6.0-bin.tar.gz -C /usr/
mv /usr/apache-flume-1.6.0-bin /usr/flume-1.6.0

2 设置环境变量

在/etc/profile最后添加

# set flume environment
export FLUME_HOME=/usr/flume-1.6.0
export PATH=$PATH:$FLUME_HOME/bin

生效 source /etc/profile

3 查看安装成功与否

flume-ng version

4 编写Demo并运行

4.1 配置log位置

vi conf/log4j.properties
修改 flume.log.dir=/usr/flume-1.6.0/logs

4.2 配置属性文件properties（关键文件）

#配置agent的sources channels sinks（这里agent取名为test）
test.sources = src_test
test.channels = channel_test
test.sinks = sink_test

#配置sources所对应的channels
test.sources.src_test.channels = channel_test

#配置sources的类型及这个类型的相关属性
test.sources.src_test.type = spooldir
test.sources.src_test.spoolDir = /usr/flume-1.6.0/data/test
test.sources.src_test.selector.type = replicating

#配置channels的类型及这个类型的相关属性
test.channels.channel_test.type = file
test.channels.channel_test.dataDirs = /usr/flume-1.6.0/fileChannel/data/test
test.channels.channel_test.checkpointDir = /usr/flume-1.6.0/fileChannel/checkPoint/test
test.channels.channel_test.capacity = 2000000
test.channels.channel_test.transactionCapacity = 2000
test.channels.channel_test.write-timeout = 10
test.channels.channel_test.useDualCheckpoints = truetest.channels.channel_test.backupCheckpointDir = /usr/flume-1.6.0/fileChannel/backupCheckPoint/test

#配置sinks的类型及这个类型的相关属性
test.sinks.sink_test.type = file_roll
test.sinks.sink_test.sink.directory = /usr/flume-1.6.0/result/test

#配置sinks所对应的channels
test.sinks.sink_test.channel = channel_test

4.3 建立所需的源文件和目标文件夹（sink输出时不会自动建立文件夹）

mkdir -p /usr/flume-1.6.0/data/test
vi /usr/flume-1.6.0/data/test/test
输入 Hello Flume!
mkdir /usr/flume-1.6.0/result/test

4.4 运行Demo

4.4.1 启动Flume

nohup bin/flume-ng agent --conf conf --conf-file conf/test.properties --name test -Dflume.monitoring.type=http -Dflume.monitoring.port=30000  > /dev/null 2>&1 </dev/null &

4.4.2 查看输出结果

cat /usr/flume-1.6.0/result/test/*

4.4.3 查看日志

cat /usr/flume-1.6.0/logs/flume.log

4.4.4 停止Flume

kill -15 `ps aux | grep flume | grep -v bash | grep -v grep | awk '{print $2}'`

4000