您的位置：首页 > 运维架构 > Apache

apache flume collector 安装

2016-05-11 10:43 441 查看


2、Flume Collector安装（通过extends AbstractSink implements Configurable，直接写入数据库）


2.1 安装环境

系统:

CentOS release 6.6

软件:

flume-collector.tar.gz


2.2 安装步骤


2.2.1 部署Flume Collector

具体脚本（jyapp用户）：
cd /home/jyapp
tar -zxvf flume-collector.tar.gz

cd flume-collector

#修改flume环境变量
vi conf/flume-env.sh
具体配置解析：（注意：红色需修改）
------------------------------------------------------
export JAVA_OPTS="-Xms1000m -Xmx1000m -Dflume.root.logger=INFO,LOGFILE -Dflume.monitoring.type=http -Dflume.monitoring.port=34545 -Dcom.sun.management.jmxremote -Djava.rmi.server.hostname=本机IP -Dcom.sun.management.jmxremote.port=9999
-Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false"
-------------------------------------------------------------------------------------

#修改Collector脚本
vi conf/collector.conf
具体配置解析：（注意：红色需修改）
------------------------------------------------------
collector.sources = r1
collector.channels = c1 c2
collector.sinks = k1 k2 k3 k4 k5 k6

collector.sources.r1.type = avro
collector.sources.r1.bind = 收集端IP #本机IP
collector.sources.r1.port = 收集端绑定端口 #本机接口消息端口

collector.channels.c1.type = memory
collector.channels.c1.capacity = 10000
collector.channels.c1.transactionCapacity = 10000
collector.channels.c1.byteCapacityBufferPercentage = 30
collector.channels.c1.byteCapacity = 1048576000

collector.channels.c2.type = memory
collector.channels.c2.capacity = 10000
collector.channels.c2.transactionCapacity = 10000
collector.channels.c2.byteCapacityBufferPercentage = 30
collector.channels.c2.byteCapacity = 1048576000

collector.sinks.k1.type = com.jy.modules.platform.flume.sink.OracleJdbcSink
collector.sinks.k1.url = jdbc:oracle:thin:@IP:PORT:数据库名
collector.sinks.k1.user =用户名
collector.sinks.k1.password =密码

collector.sinks.k2.type = com.jy.modules.platform.flume.sink.OracleJdbcSink
collector.sinks.k2.url = jdbc:oracle:thin:@IP:PORT:数据库名
collector.sinks.k2.user =用户名
collector.sinks.k2.password =密码

collector.sinks.k3.type = com.jy.modules.platform.flume.sink.OracleJdbcSink
collector.sinks.k3.url = jdbc:oracle:thin:@IP:PORT:数据库名
collector.sinks.k3.user =用户名
collector.sinks.k3.password =密码

collector.sinks.k4.type = com.jy.modules.platform.flume.sink.OracleJdbcSink
collector.sinks.k4.url = jdbc:oracle:thin:@IP:PORT:数据库名
collector.sinks.k4.user =用户名
collector.sinks.k4.password =密码

collector.sinks.k5.type = com.jy.modules.platform.flume.sink.OracleJdbcSink
collector.sinks.k5.url = jdbc:oracle:thin:@IP:PORT:数据库名
collector.sinks.k5.user =用户名
collector.sinks.k5.password =密码

collector.sinks.k6.type = com.jy.modules.platform.flume.sink.OracleJdbcSink
collector.sinks.k6.url = jdbc:oracle:thin:@IP:PORT:数据库名
collector.sinks.k6.user =用户名
collector.sinks.k6.password =密码

collector.sources.r1.channels = c1 c2
collector.sinks.k1.channel = c1
collector.sinks.k2.channel = c1
collector.sinks.k3.channel = c1
collector.sinks.k4.channel = c2
collector.sinks.k5.channel = c2
collector.sinks.k6.channel = c2

collector.sources.r1.selector.type = multiplexing
collector.sources.r1.selector.header = flume.client.system.flag
collector.sources.r1.selector.mapping.S000 = c1
collector.sources.r1.selector.mapping.S001 = c2
collector.sources.r1.selector.mapping.S002 = c1
collector.sources.r1.selector.mapping.S003 = c2
collector.sources.r1.selector.mapping.S004 = c1
collector.sources.r1.selector.mapping.S005 = c2
collector.sources.r1.selector.mapping.S006 = c1
collector.sources.r1.selector.mapping.S007 = c2
collector.sources.r1.selector.mapping.S008 = c1
collector.sources.r1.selector.mapping.S009 = c2
collector.sources.r1.selector.default = c1

collector.sinkgroups = g1 g2
collector.sinkgroups.g1.sinks = k1 k2 k3
collector.sinkgroups.g1.processor.type = load_balance
collector.sinkgroups.g1.processor.backoff = true
collector.sinkgroups.g1.processor.selector = round_robin

collector.sinkgroups.g2.sinks = k4 k5 k6
collector.sinkgroups.g2.processor.type = load_balance
collector.sinkgroups.g2.processor.backoff = true
collector.sinkgroups.g2.processor.selector = round_robin
-------------------------------------------------------------------------------------

后台启动collector
cd /home/jyapp/flume-collector
bin/flume-ng agent --conf conf --conf-file conf/collector.conf --name collector &

停止agent，使用kill命令

内容来自用户分享和网络整理，不保证内容的准确性，如有侵权内容，可联系管理员处理 点击这里给我发消息
标签： 
相关文章推荐
新的分享
章节导航