Storm集群,zookeeper集群搭建
2017-11-07 20:18
337 查看
环境:
CentOS 6.5
storm 1.0.2
jdk 1.8
zookeeper 3.4.6
准备工作:
1、storm 3台机器 例:192.168.168.110,192.168.168.111,192.168.168.112
2、zookeeper 3台 192.168.168.1,192.168.168.2,192.168.168.3
1、安装JDK
2、配置zookeeper环境
3、配置storm集群
其它机器把配置好的storm复制过去,同样环境变量,host不能少
启动storm
脚本如下:
启动 nimbus ui 等
启动 supervisor
然后jps 检查
CentOS 6.5
storm 1.0.2
jdk 1.8
zookeeper 3.4.6
准备工作:
1、storm 3台机器 例:192.168.168.110,192.168.168.111,192.168.168.112
2、zookeeper 3台 192.168.168.1,192.168.168.2,192.168.168.3
1、安装JDK
安装jdk1.8 ,配置环境变量,这里不多说了,不会的自行百度
2、配置zookeeper环境
上传zookeeper安装包到一台机器(192.168.1.1) /usr/local 下 解压安装包 tar zxvf zookeeper-3.4.6.tar.gz 建立软连接 ln -sv zookeeper-3.4.6 zookeeper 配置环境变量 export ZOOKEEPER_HOME=/usr/local/zookeeper export PATH=$ZOOKEEPER_HOME/bin:$JAVA_HOME/bin:$PATH 进入/usr/local/zookeeper/conf文件夹下修改一下配置文件名字 mv zoo_simple.cfg zoo.cfg vi zoo.cfg
首先 修改 dataDir,顾名思义就是【数据目录】了,修改成我们自定义的即可。 server.X=A:B:C X-代表服务器编号 A-代表ip B和C-代表端口,这个端口用来系统之间通信 每台服务器上新建dataDir目录,在data文件夹新建文件myid vi myid server.0=192.168.127.129:2888:3888【192.168.127.129服务器上面的myid填写0】 server.1=192.168.127.130:2888:3888【192.168.127.130服务器上面的myid填写1】 server.2=192.168.127.131:2888:3888【192.168.127.131服务器上面的myid填写2】 然后 zkServer.sh start
3、配置storm集群
安装storm: 下载storm:http://www.apache.org/dyn/closer.lua/storm/apache-storm-1.0.0/apache-storm-1.0.0.tar.gz 解压tar -zxvf apache-storm-1.0.0.tar.gz cp到/usr/local下,并建立软连接 运行授权 修改storm.yaml文件 运行服务
# # # ##### These may optionally be filled in: # ## List of custom serializations # topology.kryo.register: # - org.mycompany.MyType # - org.mycompany.MyType2: org.mycompany.MyType2Serializer # ## List of custom kryo decorators # topology.kryo.decorators: # - org.mycompany.MyDecorator # ## Locations of the drpc servers # drpc.servers: # - "server1" # - "server2" drpc.servers: - "storm018164" ## Metrics Consumers # topology.metrics.consumer.register: # - class: "org.apache.storm.metric.LoggingMetricsConsumer" # parallelism.hint: 1 # - class: "org.mycompany.MyMetricsConsumer" # parallelism.hint: 1 # argument: # - endpoint: "metrics-collector.mycompany.org" storm.local.dir: "/usr/local/storm" storm.zookeeper.servers: - "zookeeper018172" - "zookeeper018173" - "zookeeper018174" storm.zookeeper.port: 2181 nimbus.seeds: ["storm018164"] storm.zookeeper.root: "/storm-1.0.2" storm.cluster.mode: "distributed" storm.log.dir: "/var/log/storm/logs" logviewer.port: 60099 logviewer.appender.name: "A1" ui.port: 9990 topology.debug: false topology.executor.receive.buffer.size: 16384 topology.executor.send.buffer.size: 16384 topology.worker.receiver.thread.count: 4 storm.messaging.netty.server_worker_threads: 4 storm.messaging.netty.client_worker_threads: 4 storm.messaging.transport: "org.apache.storm.messaging.netty.Context" storm.messaging.netty.buffer_size: 52428800 storm.messaging.netty.max_retries: 10 storm.messaging.netty.min_wait_ms: 5000 storm.messaging.netty.max_wait_ms: 10000 nimbus.childopts: "-Xmx2048m -Xms2048m -Xmn500m -XX:PermSize=256M -XX:MaxPermSize=256M -XX:MaxDirectMemorySize=128M -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:+CMSClassUnloadingEnabled -XX:+UseCMSCompactAtFullCollection -XX:CMSFullGCsBeforeCompaction=0 -XX:+CMSParallelRemarkEnabled -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=70 -XX:SoftRefLRUPolicyMSPerMB=0 -XX:+CMSConcurrentMTEnabled -Djava.net.preferIPv4Stack=true" supervisor.childopts: "-Xmx2048m -Xms2048m -Xmn500m -XX:PermSize=256M -XX:MaxPermSize=256M -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:+CMSClassUnloadingEnabled -XX:+UseCMSCompactAtFullCollection -XX:CMSFullGCsBeforeCompaction=0 -XX:+CMSParallelRemarkEnabled -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=70 -XX:SoftRefLRUPolicyMSPerMB=0 -XX:+CMSConcurrentMTEnabled -Djava.net.preferIPv4Stack=true" ui.childopts: "-Xmx2048m -Xms2048m -Xmn500m -XX:PermSize=256M -XX:MaxPermSize=256M -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:+CMSClassUnloadingEnabled -XX:+UseCMSCompactAtFullCollection -XX:CMSFullGCsBeforeCompaction=0 -XX:+CMSParallelRemarkEnabled -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=70 -XX:SoftRefLRUPolicyMSPerMB=0 -XX:+CMSConcurrentMTEnabled -Djava.net.preferIPv4Stack=true" worker.childopts: "-Xmx2048m -Xms2048m -Xmn500m -XX:PermSize=256M -XX:MaxPermSize=256M -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:+CMSClassUnloadingEnabled -XX:+UseCMSCompactAtFullCollection -XX:CMSFullGCsBeforeCompaction=0 -XX:+CMSParallelRemarkEnabled -XX:+UseCMSInitiatingOccupancyOnly -XX 4000 :CMSInitiatingOccupancyFraction=70 -XX:SoftRefLRUPolicyMSPerMB=0 -XX:+CMSConcurrentMTEnabled -Djava.net.preferIPv4Stack=true" supervisor.slots.ports: - 6700 - 6701 - 6702 - 6703 - 6704 - 6705
其它机器把配置好的storm复制过去,同样环境变量,host不能少
启动storm
脚本如下:
启动 nimbus ui 等
nohup $STORM_HOME/bin/storm nimbus > /dev/null 2>&1 & nohup $STORM_HOME/bin/storm ui > /dev/null 2>&1 & nohup $STORM_HOME/bin/storm logviewer > /dev/null 2>&1 & nohup $STORM_HOME/bin/storm drpc > /dev/null 2>&1 &
启动 supervisor
source /etc/profile nohup $STORM_HOME/bin/storm supervisor > /dev/null 2>&1 & nohup $STORM_HOME/bin/storm logviewer > /dev/null 2>&1 &
然后jps 检查
相关文章推荐
- STORM_0002_在做好的zookeeper集群上搭建storm的开发环境
- STORM_0004_windows下zookeeper的伪集群的搭建
- zookeeper+hadoop+hbase+kafka+storm集群搭建
- STORM_0001_用vmware拷贝出三个相同的ubuntu搭建小的zookeeper集群
- Storm和Zookeeper集群搭建及在java项目中的使用
- Zookeeper集群搭建(配置详解)
- storm集群搭建
- zookeeper入门及其集群搭建
- storm伪集群环境搭建
- zookeeper集群搭建centOS6.5
- 搭建JEESZ分布式架构6--ZooKeeper 集群的安装
- zookeeper集群搭建
- CentOS6.5以上搭建zookeeper集群
- zookeeper手把手搭建伪分布集群环境
- 搭建Hbase集群(2) zookeeper和hbase的安装
- [置顶] zookeeper 集群搭建教程之应用案例
- Zookeeper集群搭建
- zookeeper集群环境搭建
- 【Hadoop】ZooKeeper集群搭建中的Connection refused而导致的启动失败