storm on yarn 部署
2016-12-08 16:10
330 查看
1. 环境介绍
1.1 节点与服务映射关系
ip | host | 服务 |
---|---|---|
192.168.40.132 | master | Namenode、NodeManager、DataNode、zookeeper |
192.168.40.133 | slave1 | ResurceManager、NodeManager、DataNode、zookeeper |
192.168.40.134 | slave2 | NodeManager、DataNode、zookeeper |
1.2 jdk版本
[root@slave2 ~]# java -version java version "1.7.0_79" Java(TM) SE Runtime Environment (build 1.7.0_79-b15) Java HotSpot(TM) 64-Bit Server VM (build 24.79-b02, mixed mode)1
2
3
4
1
2
3
4
1.3 hadoop版本
[root@slave2 ~]# hadoop version Hadoop 2.6.0 Subversion https://git-wip-us.apache.org/repos/asf/hadoop.git -r e3496499ecb8d220fba99dc5ed4c99c8f9e33bb1 Compiled by jenkins on 2014-11-13T21:10Z Compiled with protoc 2.5.0 From source with checksum 18e43357c8f927c0695f1e9522859d6a This command was run using /application/hadoop-2.6.0/share/hadoop/common/hadoop-common-2.6.0.jar1
2
3
4
5
6
7
1
2
3
4
5
6
7
1.4 各节点进程 jps
2. 安装maven
2.1 下载、解压maven
wget http://mirrors.tuna.tsinghua.edu.cn/apache/maven/maven-3/3.3.9/binaries/apache-maven-3.3.9-bin.tar.gz tar -xzvf apache-maven-3.3.9-bin.tar.gz ln -s apache-maven-3.3.9/ maven1
2
3
4
5
1
2
3
4
5
2.2 配置maven环境变量
export MAVEN_HOME=/application/maven export PATH=.:$MAVEN_HOME/bin:$JAVA_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$PATH source /etc/profile1
2
3
4
1
2
3
4
2.3 验证,查看maven版本号
mvn -v Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5; 2015-11-11T00:41:47+08:00) Maven home: /application/maven Java version: 1.7.0_79, vendor: Oracle Corporation Java home: /application/jdk1.7.0_79/jre Default locale: en_US, platform encoding: UTF-8 OS name: "linux", version: "2.6.32-504.el6.x86_64", arch: "amd64", family: "unix"1
2
3
4
5
6
7
1
2
3
4
5
6
7
3. 安装storm-yarn
3.1 下载storm-yarn
wget https://github.com/yahoo/storm-yarn/archive/master.zip[/code]1 1
重命名下载的压缩包为storm-yarn.zipmv master.zip storm-yarn.zip1
1
解压unzip storm-yarn.zip1
1根据Hadoop的版本号修改storm-yarn-master/pom.xml
3.2 修改pom.xml<properties> <storm.version>0.9.6</storm.version> <hadoop.version>2.6.0</hadoop.version> </properties>1
2
3
4
1
2
3
4
因为我的环境中hadoop的版本是2.6.0,没有安装storm(也不用安装),所以pom.xml没有修改。
3.3 编译打包storm-yarn-mastermvn package -DskipTests1
1
打包成功[INFO] --- maven-jar-plugin:2.4:jar (default-jar) @ storm-yarn --- [INFO] ---------------------------------------------------------------------- [INFO] BUILD SUCCESS [INFO] ---------------------------------------------------------------------- [INFO] Total time: 08:41 min [INFO] Finished at: 2016-08-26T10:16:27+08:00 [INFO] Final Memory: 14M/34M [INFO] ----------------------------------------------------------------------1
2
3
4
5
6
7
8
1
2
3
4
5
6
7
8解压storm-yarn-master/lib/storm.zip
3.4 解压lib/storm.zip并移动解压后的目录unzip storm-yarn-master/lib/storm.zip [root@master lib]# ls storm-0.9.0-wip21 storm-0.9.0-wip21.zip storm-starter-0.0.1-SNAPSHOT.jar storm.zip1
2
3
4
1
2
3
4
将解压后生成的storm-0.9.0-wip21移到和storm-yarn-master同级的目录[root@master lib]mv storm-0.9.0-wip21 ../.. [root@master lib]# cd ../.. [root@master application]# ls maven storm-yarn.zip apache-maven-3.3.9 tools hadoop zookeeper hadoop-2.6.0 java storm-0.9.0-wip21 zookeeper-3.3.6 jdk1.7.0_79 storm-yarn-master1
2
3
4
1
2
3
4
3.5 配置环境变量export PATH=.:/application/storm-0.9.0-wip21/bin:/application/storm-yarn-master/bin:$MAVEN_HOME/bin::$JAVA_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$PATH1
1
使用source /etc/profile使之生效修改zookeeper地址
3.6 编辑 storm-0.9.0-wip21/conf/storm.yamlstorm.zookeeper.servers: - "master" - "slave1" - "slave2"1
2
3
4
1
2
3
4压缩storm-0.9.0-wip21
3.7 压缩storm.zip并上传到hdfs上zip -r storm.zip storm-0.9.0-wip211
1
将压缩后的文件storm.zip上传到hdfs上的目录/lib/storm/0.9.0-wip21/hadoop fs -mkdir -p /lib/storm/0.9.0-wip21 hdfs dfs -put storm.zip /lib/storm/0.9.0-wip21/ hdfs dfs -ls /lib/storm/0.9.0-wip21/ Found 1 items -rw-r--r-- 3 root supergroup 17141097 2016-08-26 10:27 /lib/storm/0.9.0-wip21/storm.zip1
2
3
4
5
1
2
3
4
5
4. 运行storm-yarn
4.1 启动storm-yarn[root@master application]# storm-yarn launch storm-0.9.0-wip21/conf/storm.yaml SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/application/hadoop-2.6.0/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/application/storm-0.9.0-wip21/lib/logback-classic-1.0.6.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] 16/08/26 10:28:28 INFO client.RMProxy: Connecting to ResourceManager at slave1/192.168.40.133:8032 16/08/26 10:28:29 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 16/08/26 10:28:30 INFO yarn.StormOnYarn: Copy App Master jar from local filesystem and add to local environment 16/08/26 10:28:32 INFO yarn.StormOnYarn: Set the environment for the application master 16/08/26 10:28:32 INFO yarn.StormOnYarn: YARN CLASSPATH COMMAND = [[yarn, classpath]] 16/08/26 10:28:32 INFO yarn.StormOnYarn: Using JAVA_HOME = [/application/java] 16/08/26 10:28:32 INFO yarn.StormOnYarn: Setting up app master command:[/application/java/bin/java, -Dstorm.home=./storm/storm-0.9.0-wip21/, -Dlogfile.name=<LOG_DIR>/master.log, com.yahoo.storm.yarn.MasterServer, 1><LOG_DIR>/stdout, 2><LOG_DIR>/stderr] 16/08/26 10:28:32 INFO impl.YarnClientImpl: Submitted application application_1472175545791_0002 application_1472175545791_00021
2
3
4
5
6
7
8
9
10
11
12
13
14
15
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
通过http://slave2:8088查看yarn的appplication运行情况得到storm的属性信息,并保存到/root/.storm/storm.yaml中
4.2 获取nimbus地址[root@master application]# mkdir -p /root/.storm/ [root@master application]# storm-yarn getStormConfig -appId application_1472175545791_0002 -output ~/.storm/storm.yaml SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/application/hadoop-2.6.0/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/application/storm-0.9.0-wip21/lib/logback-classic-1.0.6.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] 16/08/26 10:30:13 INFO client.RMProxy: Connecting to ResourceManager at slave1/192.168.40.133:8032 16/08/26 10:30:13 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 16/08/26 10:30:14 INFO yarn.StormOnYarn: application report for application_1472175545791_0002 :slave2:9000 16/08/26 10:30:14 INFO yarn.StormOnYarn: Attaching to slave2:9000 to talk to app master application_1472175545791_0002 {10:35:18}16/08/26 10:30:14 INFO yarn.StormMasterCommand: storm.yaml downloaded into /root/.storm/storm.yaml1
2
3
4
5
6
7
8
9
10
11
12
1
2
3
4
5
6
7
8
9
10
11
12
根据/root/.storm/storm.yaml获取nimbus地址[root@master .storm]# cat ~/.storm/storm.yaml | grep nimbus.host nimbus.host: 192.168.40.1341
2
1
2通过访问http://192.168.40.134:7070访问storm UI
4.3 通过nimbus访问storm UI
4.4 提交任务wordcount[root@master storm-yarn-master]# storm jar lib/storm-starter-0.0.1-SNAPSHOT.jar storm.starter.WordCountTopology WordCountTopology -c nimbus.host=192.168.40.134 561 [main] INFO backtype.storm.StormSubmitter - Jar not uploaded to master yet. Submitting jar... 573 [main] INFO backtype.storm.StormSubmitter - Uploading topology jar lib/storm-starter-0.0.1-SNAPSHOT.jar to assigned location: storm-local/nimbus/inbox/stormjar-12a737f5-82e9-43a6-b422-c1740cc36635.jar 596 [main] INFO backtype.storm.StormSubmitter - Successfully uploaded topology jar to assigned location: storm-local/nimbus/inbox/stormjar-12a737f5-82e9-43a6-b422-c1740cc36635.jar 596 [main] INFO backtype.storm.StormSubmitter - Submitting topology WordCountTopology in distributed mode with conf {"nimbus.host":"192.168.40.134","topology.workers":3,"topology.debug":true} 948 [main] INFO backtype.storm.StormSubmitter - Finished submitting topology: WordCountTopology1
2
3
4
5
6
1
2
3
4
5
6
查看Storm UI
查看topology wordcount的运行情况
查看各节点jps
5. 关闭
5.1 关闭storm 提交的任务[root@master ~]# storm kill WordCountTopology 2277 [main] INFO backtype.storm.thrift - Connecting to Nimbus at 192.168.40.134:6627 2387 [main] INFO backtype.storm.command.kill-topology - Killed topology: WordCountTopology1
2
3
1
2
3
5.2 关闭storm on yarn[root@master ~]# yarn application -kill application_1472175545791_0002 Killing application application_1472175545791_0002
相关文章推荐
- storm on yarn 部署
- Storm之——Storm on Yarn 部署
- storm on yarn(CDH5) 部署笔记
- Storm On Yarn 安装部署
- Storm on YARN 安装部署与感悟
- Spark2.0.1 on yarn with hue 集群搭建部署(五)hue安装支持hadoop
- Storm On YARN带来的好处
- storm on Yarn
- Spark On YARN 集群安装部署
- spark集群安装部署(spark on yarn)
- Hbase on Yarn-编译部署slider
- Storm On YARN
- Spark1.0.0 on YARN 模式部署
- Spark On YARN 集群安装部署
- 部署zeppelin时遇到的spark on yarn的submit方式问题
- Spark On YARN部署模式下的内存分配情况
- Spark1.0.0 on YARN 模式部署
- Spark2.0.1 on yarn with hue 集群安装部署(六)hue+hive+sparksql
- Spark On YARN 集群安装部署
- Spark2.0.1 on yarn with hue 集群搭建部署(二)spark on yarn搭建