flume部署
2015-10-31 17:05
661 查看
storm
一.storm:实时版的Hadoop
1.部署zookeeper
(1).useradd shaka
passwd shaka
(2). su shaka
mkdir zookeeper
把zookeeper包放到该目录,并解压
tar -xvf zookeeper.jar包
ifconfig获得本机IP
修改zookeeper配置文件zoo.cfg
a.dataDir=/文件路径/data
b.dataLogDir=/文件路径/logs
c.server=本机IP或者127.0.0.1
(3).启动zookeeper
bin/zkZookeeper.sh start
查看状态(停止stop 重启restart)
bin/zkZookeeper.sh status
启动zookeeper客户端
bin/zkCli.sh -server 127.0.0.1:2181
zookeeper客户端命令:
ls ls2 create get set delete quit help
2.storm 依赖包
(1).shaka下:
mkdir dependence
cd denpence
把俩个依赖文件放到该目录下,解压
cd zeromq-4.0.4 ->./configure
cd jzmq ->./configure
(2)root权限下,
yum install python-devel ->./configure(make make install在root下操作)
3.storm
(1).解压storm包
vim ~/.bash_profile
export STORM_HOME=/文件路径
export PATH=$PATH:$STORM_HOME/bin
重载环境变量 : . ~/.bash_profile
(2).修改storm配置文件storm.yaml
storm.zookeeper.servers:
自己IP或者127.0.0.1
nimbus.host:"ip或127.0.0.1"
storm.local.dir:"原后加/topology"
ui.port:"8080"
(3).启动storm
storm nuimbus &
storm supervisor &
storm ui &
可通过http://127.0.0.1:8080查看
4.应用 starter.jar
修改文件中内容:String host="127.0.0.1";
String db="test";
String user="root";
String passwd="root";
在该目录下ant,且在starter下:storm jar dist/togology-0.0.1.jar starter.WordCountTopology
5.给mysql换权限:grant all privileges on *.* to `myuser`@`127.0.0.1` identified by 'mypasswd';
flume的处理单元为event。flume的数据流图:source-》channel-》sinks
flume的部署:
(1).拷贝flume的jar包到shaka下
(2).配置flume的配置文件:
vim flume-source.properties: (添加如下配置)
source.sinks.sink1.type = org.apache.flume.sink.kafka.KafkaSink
source.sinks.sink1.topic = topic-003
source.sinks.sink1.brokerList = localhost:9092
source.sinks.sink1.requiredAcks = 1
source.sinks.sink1.batchSize = 20
(3).启动:sbin/start-flume.sh
7.kafka简介:
consumer/prodecer生产者与发送者
kafka部署:
(1).解压 打开配置文件修改:
conf/server.properties:
broker.id=0;
host.name=127.0.0.1;
zookeeper.connect=127.0.0.1:2181;
(2).在启动zookeeper的条件下:
启动kafka:bin/kafka-server-start.sh config/server.properties &
查看kafka的list:bin/kafka-topics.sh --list --zookeeper 127.0.0.1:2181
创建topic:bin/kafka-topics.sh --create --zookeeper 127.0.0.1:2181 --replication 1 --topic topic-003
删除topic:bin/kafka-run-class.sh kafka.admin.DeleteTopicCommand --topic topic-003 --zookeeper 127.0.0.1:2181
(3).启动customer:
bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic topic-003
启动producer:
bin/kafka-console-producer.sh --broker-list localhost:9092 --topic topic-003
一.storm:实时版的Hadoop
1.部署zookeeper
(1).useradd shaka
passwd shaka
(2). su shaka
mkdir zookeeper
把zookeeper包放到该目录,并解压
tar -xvf zookeeper.jar包
ifconfig获得本机IP
修改zookeeper配置文件zoo.cfg
a.dataDir=/文件路径/data
b.dataLogDir=/文件路径/logs
c.server=本机IP或者127.0.0.1
(3).启动zookeeper
bin/zkZookeeper.sh start
查看状态(停止stop 重启restart)
bin/zkZookeeper.sh status
启动zookeeper客户端
bin/zkCli.sh -server 127.0.0.1:2181
zookeeper客户端命令:
ls ls2 create get set delete quit help
2.storm 依赖包
(1).shaka下:
mkdir dependence
cd denpence
把俩个依赖文件放到该目录下,解压
cd zeromq-4.0.4 ->./configure
cd jzmq ->./configure
(2)root权限下,
yum install python-devel ->./configure(make make install在root下操作)
3.storm
(1).解压storm包
vim ~/.bash_profile
export STORM_HOME=/文件路径
export PATH=$PATH:$STORM_HOME/bin
重载环境变量 : . ~/.bash_profile
(2).修改storm配置文件storm.yaml
storm.zookeeper.servers:
自己IP或者127.0.0.1
nimbus.host:"ip或127.0.0.1"
storm.local.dir:"原后加/topology"
ui.port:"8080"
(3).启动storm
storm nuimbus &
storm supervisor &
storm ui &
可通过http://127.0.0.1:8080查看
4.应用 starter.jar
修改文件中内容:String host="127.0.0.1";
String db="test";
String user="root";
String passwd="root";
在该目录下ant,且在starter下:storm jar dist/togology-0.0.1.jar starter.WordCountTopology
5.给mysql换权限:grant all privileges on *.* to `myuser`@`127.0.0.1` identified by 'mypasswd';
6.flume简介:
flume是Cloudera提供的一个分布式,可靠,和高可用的海量日志的采集,聚合与传输的日志收集系统,支持在日志系统中定制各类数据的发送方,用于收集数据。同时,flume提供对数据进行简单处理,并写到各种数据接收方的能力。
flume的特性:1可靠 2可伸缩 3高性能 4可延伸 5开源社区的支持flume的处理单元为event。flume的数据流图:source-》channel-》sinks
flume的部署:
(1).拷贝flume的jar包到shaka下
(2).配置flume的配置文件:
vim flume-source.properties: (添加如下配置)
source.sinks.sink1.type = org.apache.flume.sink.kafka.KafkaSink
source.sinks.sink1.topic = topic-003
source.sinks.sink1.brokerList = localhost:9092
source.sinks.sink1.requiredAcks = 1
source.sinks.sink1.batchSize = 20
(3).启动:sbin/start-flume.sh
7.kafka简介:
consumer/prodecer生产者与发送者
kafka部署:
(1).解压 打开配置文件修改:
conf/server.properties:
broker.id=0;
host.name=127.0.0.1;
zookeeper.connect=127.0.0.1:2181;
(2).在启动zookeeper的条件下:
启动kafka:bin/kafka-server-start.sh config/server.properties &
查看kafka的list:bin/kafka-topics.sh --list --zookeeper 127.0.0.1:2181
创建topic:bin/kafka-topics.sh --create --zookeeper 127.0.0.1:2181 --replication 1 --topic topic-003
删除topic:bin/kafka-run-class.sh kafka.admin.DeleteTopicCommand --topic topic-003 --zookeeper 127.0.0.1:2181
(3).启动customer:
bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic topic-003
启动producer:
bin/kafka-console-producer.sh --broker-list localhost:9092 --topic topic-003
相关文章推荐
- Kafka 之 中级
- Release Notes - Apache Storm - Version 0.9.2-incub
- C/C++实现对STORM运行信息查看及控制的方法
- Shell脚本实现自动安装zookeeper
- Flume环境部署和配置详解及案例大全
- 基于Zookeeper的使用详解
- 基于zk的配置管理
- 基于Storm的Nginx log实时监控系统
- Play! Akka Flume实现的完整数据收集
- SolrCloud4.9+zookeeper在CentOS上的搭建与安装
- flume自定义Interceptor
- 基于外部ZooKeeper的GlusterFS作为分布式文件系统的完全分布式HBase集群安装指南
- redis集群搭建
- #Note# Analyzing Twitter Data with Apache Hadoo...
- 使用 RMI + ZooKeeper 实现远程调用框架
- 轻量级分布式 RPC 框架
- 整合Kafka到Spark Streaming——代码示例和挑战
- HBase 系统架构