hadoop安装部署(伪分布及集群)
2017-08-08 14:07
344 查看
hadoop安装部署(伪分布及集群)
@(HADOOP)[hadoop]hadoop安装部署伪分布及集群
第一部分伪分布式
一环境准备
二安装hdfs
三安装YARN
第二部分集群安装
一规划
一硬件资源
二基本资料
二环境配置
一统一用户名密码并为jediael赋予执行所有命令的权限
二创建目录mntjediael
三修改用户名及etchosts文件
第一部分:伪分布式
一、环境准备
1、安装linux、jdk2、下载hadoop2.6.0,并解压
3、配置免密码ssh
(1)检查是否可以免密码:
$ ssh localhost
(2)若否:
$ ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa $ cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys
4、在/etc/profile中添加以下内容
#hadoop setting export PATH=$PATH:/mnt/jediael/hadoop-2.6.0/bin:/mnt/jediael/hadoop-2.6.0/sbin export HADOOP_HOME=/mnt/jediael/hadoop-2.6.0 export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib"
二、安装hdfs
1、配置etc/hadoop/core-site.xml:<configuration> <property> <name>fs.defaultFS</name> <value>hdfs://localhost:9000</value> </property> </configuration>
2、配置etc/hadoop/hdfs-site.xml:
<configuration> <property> <name>dfs.replication</name> <value>1</value> </property> </configuration>
3、格式化namenode
$ bin/hdfs namenode -format
4、启动hdfs
$ sbin/start-dfs.sh
5、打开页面验证hdfs安装成功
http://localhost:50070/
6、运行自带示例
(1)创建目录
$ bin/hdfs dfs -mkdir /user $ bin/hdfs dfs -mkdir /user/jediael
(2)复制文件
bin/hdfs dfs -put etc/hadoop input
(3)运行示例
$ bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0.jar grep input output 'dfs[a-z.]+’
(4)检查输出结果
$ bin/hdfs dfs -cat output/* 6 dfs.audit.logger 4 dfs.class 3 dfs.server.namenode. 2 dfs.period 2 dfs.audit.log.maxfilesize 2 dfs.audit.log.maxbackupindex 1 dfsmetrics.log 1 dfsadmin 1 dfs.servers 1 dfs.replication 1 dfs.file
(5)关闭hdfs
$ sbin/stop-dfs.sh
三、安装YARN
1、配置etc/hadoop/mapred-site.xml<configuration> <property> <name>mapreduce.framework.name</name> <value>yarn</value> </property> </configuration>
2、配置etc/hadoop/yarn-site.xml
<configuration> <property> <name>yarn.nodemanager.aux-services</name> <value>mapreduce_shuffle</value> </property> </configuration>
3、启动yarn
$ sbin/start-yarn.sh
4、打开页面检查yarn
http://localhost:8088/
5、运行一个map-reduce job
$ bin/hadoop fs -mkdir /input $ bin/hadoop fs -copyFromLocal /etc/profile /input $ cd /mnt/jediael/hadoop-2.6.0/share/hadoop/mapreduce $ /mnt/jediael/hadoop-2.6.0/bin/hadoop jar hadoop-mapreduce-examples-2.6.0.jar wordcount /input /output
查看结果:
$/mnt/jediael/hadoop-2.6.0/bin/hadoop fs -cat /output/*
第二部分:集群安装
一、规划
(一)硬件资源
10.171.29.191 master10.171.94.155 slave1
10.251.0.197 slave3
(二)基本资料
用户: jediael目录:/mnt/jediael/
二、环境配置
(一)统一用户名密码,并为jediael赋予执行所有命令的权限
#passwd # useradd jediael # passwd jediael # vi /etc/sudoers
增加以下一行:
jediael ALL=(ALL) ALL
(二)创建目录/mnt/jediael
$sudo chown jediael:jediael /opt $ cd /opt $ sudo mkdir jediael
注意:/opt必须是jediael的,否则会在format namenode时出错。
(三)修改用户名及/etc/hosts文件
1、修改/etc/sysconfig/networkNETWORKING=yes HOSTNAME=*******
2、修改/etc/hosts
10.171.29.191 master
10.171.94.155 slave1
10.251.0.197 slave3
注 意hosts文件不能有127.0.0.1 *配置,否则会导致出现异常。org.apache.hadoop.ipc.Client: Retrying connect to server: master/10.171.29.191:9000. Already trie
3、hostname命令
hostname ****
(四)配置免密码登录
以上命令在master上使用jediael用户执行:
$ ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa $ cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys然后,将authorized_keys复制到slave1,slave2
scp ~/.ssh/authorized_keys slave1:~/.ssh/
scp ~/.ssh/authorized_keys slave2:~/.ssh/
注意
(1)若提示.ssh目录不存在,则表示此机器从未运行过ssh,因此运行一次即可创建.ssh目录。
(2).ssh/的权限为600,authorized_keys的权限为700,权限大了小了都不行。
(五)在3台机器上分别安装java,并设置相关环境变量
参考http://blog.csdn.net/jediael_lu/article/details/38925871
(六)下载hadoop-2.6.0.tar.gz,并将其解压到/mnt/jediael
wget http://mirror.bit.edu.cn/apache/hadoop/common/hadoop-2.6.0/hadoop-2.6.0.tar.gz
tar -zxvf hadoop-2.6.0.tar.gz
三、修改配置文件
【3台机器上均要执行,一般先在一台机器上配置完成,再用scp复制到其它机器】
(一)hadoop_env.sh
export JAVA_HOME=/usr/java/jdk1.7.0_51
(二)修改core-site.xml
<property> <name>hadoop.tmp.dir</name> <value>/mnt/tmp</value> <description>Abase for other temporary directories.</description> </property> <property> <name>fs.defaultFS</name> <value>hdfs://master:9000</value> </property> <property> <name>io.file.buffer.size</name> <value>4096</value> </property>
(三)修改hdfs-site.xml
<property> <name>dfs.replication</name> <value>2</value> </property>
(四)修改mapred-site.xml
<property> <name>mapreduce.framework.name</name> <value>yarn</value> <final>true</final> </property> <property> <name>mapreduce.jobtracker.http.address</name> <value>master:50030</value> </property> <property> <name>mapreduce.jobhistory.address</name> <value>master:10020</value> </property> <property> <name>mapreduce.jobhistory.webapp.address</name> <value>master:19888</value> </property> <property> <name>mapred.job.tracker</name> <value>http://master:9001</value> </property>
(五)修改yarn.xml
<property> <name>yarn.resourcemanager.hostname</name> <value>master</value> </property> <property> <name>yarn.nodemanager.aux-services</name> <value>mapreduce_shuffle</value> </property> <property> <name>yarn.resourcemanager.address</name> <value>master:8032</value> </property> <property> <name>yarn.resourcemanager.scheduler.address</name> <value>master:8030</value> </property> <property> <name>yarn.resourcemanager.resource-tracker.address</name> <value>master:8031</value> </property> <property> <name>yarn.resourcemanager.admin.address</name> <value>master:8033</value> </property> <property> <name>yarn.resourcemanager.webapp.address</name> <value>master:8088</value> </property>
(六)修改slaves
slaves:
slave1 slave3
四、启动并验证
1、格式 化namenode
[jediael@master hadoop-1.2.1]$ bin/hadoop namenode -format
2、启动hadoop【此步骤只需要在master上执行】
[jediael@master hadoop-1.2.1]$ bin/start-all.sh
3、验证1:向hdfs中写入内容
[jediael@master hadoop-2.6.0]$ bin/hadoop fs -ls / [jediael@master hadoop-2.6.0]$ bin/hadoop fs -mkdir /test [jediael@master hadoop-2.6.0]$ bin/hadoop fs -ls / Found 1 items drwxr-xr-x - jediael supergroup 0 2015-04-19 23:41 /test
4、验证:登录页面
NameNode http://ip:50070
5、查看各个主机的java进程
(1)master:
$ jps 3694 NameNode 3882 SecondaryNameNode 7216 Jps 4024 ResourceManager
(2)slave1:
$ jps 1913 NodeManager 2673 Jps 1801 DataNode
(3)slave3:
$ jps 1942 NodeManager 2252 Jps 1840 DataNode
五、运行一个完整的mapreduce程序:运行自带的wordcount程序
$ bin/hadoop fs -mkdir /input $ bin/hadoop fs -ls / Found 2 items drwxr-xr-x - jediael supergroup 0 2015-04-20 18:04 /input drwxr-xr-x - jediael supergroup 0 2015-04-19 23:41 /test $ bin/hadoop fs -copyFromLocal etc/hadoop/mapred-site.xml.template /input $ pwd /mnt/jediael/hadoop-2.6.0/share/hadoop/mapreduce $ /mnt/jediael/hadoop-2.6.0/bin/hadoop jar hadoop-mapreduce-examples-2.6.0.jar wordcount /input /output 15/04/20 18:15:47 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 15/04/20 18:15:48 INFO Configuration.deprecation: session.id is deprecated. Instead, use dfs.metrics.session-id 15/04/20 18:15:48 INFO jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId= 15/04/20 18:15:49 INFO input.FileInputFormat: Total input paths to process : 1 15/04/20 18:15:49 INFO mapreduce.JobSubmitter: number of splits:1 15/04/20 18:15:49 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_local657082309_0001 15/04/20 18:15:50 INFO mapreduce.Job: The url to track the job: http://localhost:8080/ 15/04/20 18:15:50 INFO mapreduce.Job: Running job: job_local657082309_0001 15/04/20 18:15:50 INFO mapred.LocalJobRunner: OutputCommitter set in config null 15/04/20 18:15:50 INFO mapred.LocalJobRunner: OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter 15/04/20 18:15:50 INFO mapred.LocalJobRunner: Waiting for map tasks 15/04/20 18:15:50 INFO mapred.LocalJobRunner: Starting task: attempt_local657082309_0001_m_000000_0 15/04/20 18:15:50 INFO mapred.Task: Using ResourceCalculatorProcessTree : [ ] 15/04/20 18:15:50 INFO mapred.MapTask: Processing split: hdfs://master:9000/input/mapred-site.xml.template:0+2268 15/04/20 18:15:51 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584) 15/04/20 18:15:51 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100 15/04/20 18:15:51 INFO mapred.MapTask: soft limit at 83886080 15/04/20 18:15:51 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600 15/04/20 18:15:51 INFO mapred.MapTask: kvstart = 26214396; length = 6553600 15/04/20 18:15:51 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer 15/04/20 18:15:51 INFO mapred.LocalJobRunner: 15/04/20 18:15:51 INFO mapred.MapTask: Starting flush of map output 15/04/20 18:15:51 INFO mapred.MapTask: Spilling map output 15/04/20 18:15:51 INFO mapred.MapTask: bufstart = 0; bufend = 1698; bufvoid = 104857600 15/04/20 18:15:51 INFO mapred.MapTask: kvstart = 26214396(104857584); kvend = 26213916(104855664); length = 481/6553600 15/04/20 18:15:51 INFO mapred.MapTask: Finished spill 0 15/04/20 18:15:51 INFO mapred.Task: Task:attempt_local657082309_0001_m_000000_0 is done. And is in the process of committing 15/04/20 18:15:51 INFO mapred.LocalJobRunner: map 15/04/20 18:15:51 INFO mapred.Task: Task 'attempt_local657082309_0001_m_000000_0' done. 15/04/20 18:15:51 INFO mapred.LocalJobRunner: Finishing task: attempt_local657082309_0001_m_000000_0 15/04/20 18:15:51 INFO mapred.LocalJobRunner: map task executor complete. 15/04/20 18:15:51 INFO mapred.LocalJobRunner: Waiting for reduce tasks 15/04/20 18:15:51 INFO mapred.LocalJobRunner: Starting task: attempt_local657082309_0001_r_000000_0 15/04/20 18:15:51 INFO mapred.Task: Using ResourceCalculatorProcessTree : [ ] 15/04/20 18:15:51 INFO mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@39be5e01 15/04/20 18:15:51 INFO reduce.MergeManagerImpl: MergerManager: memoryLimit=363285696, maxSingleShuffleLimit=90821424, mergeThreshold=239768576, ioSortFactor=10, memToMemMergeOutputsThreshold=10 15/04/20 18:15:51 INFO reduce.EventFetcher: attempt_local657082309_0001_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events 15/04/20 18:15:51 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local657082309_0001_m_000000_0 decomp: 1566 len: 1570 to MEMORY 15/04/20 18:15:51 INFO reduce.InMemoryMapOutput: Read 1566 bytes from map-output for attempt_local657082309_0001_m_000000_0 15/04/20 18:15:51 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 1566, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->1566 15/04/20 18:15:51 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning 15/04/20 18:15:51 INFO mapred.LocalJobRunner: 1 / 1 copied. 15/04/20 18:15:51 INFO reduce.MergeManagerImpl: finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs 15/04/20 18:15:51 INFO mapred.Merger: Merging 1 sorted segments 15/04/20 18:15:51 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 1560 bytes 15/04/20 18:15:51 INFO reduce.MergeManagerImpl: Merged 1 segments, 1566 bytes to disk to satisfy reduce memory limit 15/04/20 18:15:51 INFO reduce.MergeManagerImpl: Merging 1 files, 1570 bytes from disk 15/04/20 18:15:51 INFO reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce 15/04/20 18:15:51 INFO mapred.Merger: Merging 1 sorted segments 15/04/20 18:15:51 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 1560 bytes 15/04/20 18:15:51 INFO mapred.LocalJobRunner: 1 / 1 copied. 15/04/20 18:15:51 INFO Configuration.deprecation: mapred.skip.on is deprecated. Instead, use mapreduce.job.skiprecords 15/04/20 18:15:51 INFO mapreduce.Job: Job job_local657082309_0001 running in uber mode : false 15/04/20 18:15:51 INFO mapreduce.Job: map 100% reduce 0% 15/04/20 18:15:51 INFO mapred.Task: Task:attempt_local657082309_0001_r_000000_0 is done. And is in the process of committing 15/04/20 18:15:51 INFO mapred.LocalJobRunner: 1 / 1 copied. 15/04/20 18:15:51 INFO mapred.Task: Task attempt_local657082309_0001_r_000000_0 is allowed to commit now 15/04/20 18:15:51 INFO output.FileOutputCommitter: Saved output of task 'attempt_local657082309_0001_r_000000_0' to hdfs://master:9000/output/_temporary/0/task_local657082309_0001_r_000000 15/04/20 18:15:51 INFO mapred.LocalJobRunner: reduce > reduce 15/04/20 18:15:51 INFO mapred.Task: Task 'attempt_local657082309_0001_r_000000_0' done. 15/04/20 18:15:51 INFO mapred.LocalJobRunner: Finishing task: attempt_local657082309_0001_r_000000_0 15/04/20 18:15:51 INFO mapred.LocalJobRunner: reduce task executor complete. 15/04/20 18:15:52 INFO mapreduce.Job: map 100% reduce 100% 15/04/20 18:15:52 INFO mapreduce.Job: Job job_local657082309_0001 completed successfully 15/04/20 18:15:52 INFO mapreduce.Job: Counters: 38 File System Counters FILE: Number of bytes read=544164 FILE: Number of bytes written=1040966 FILE: Number of read operations=0 FILE: Number of large read operations=0 FILE: Number of write operations=0 HDFS: Number of bytes read=4536 HDFS: Number of bytes written=1196 HDFS: Number of read operations=15 HDFS: Number of large read operations=0 HDFS: Number of write operations=4 Map-Reduce Framework Map input records=43 Map output records=121 Map output bytes=1698 Map output materialized bytes=1570 Input split bytes=114 Combine input records=121 Combine output records=92 Reduce input groups=92 Reduce shuffle bytes=1570 Reduce input records=92 Reduce output records=92 Spilled Records=184 Shuffled Maps =1 Failed Shuffles=0 Merged Map outputs=1 GC time elapsed (ms)=123 CPU time spent (ms)=0 Physical memory (bytes) snapshot=0 Virtual memory (bytes) snapshot=0 Total committed heap usage (bytes)=269361152 Shuffle Errors BAD_ID=0 CONNECTION=0 IO_ERROR=0 WRONG_LENGTH=0 WRONG_MAP=0 WRONG_REDUCE=0 File Input Format Counters Bytes Read=2268 File Output Format Counters $ /mnt/jediael/hadoop-2.6.0/bin/hadoop fs -cat /output/*
相关文章推荐
- HBase入门笔记(三)-- 完全分布模式Hadoop集群安装配置
- Hadoop2.2集群安装配置-Spark集群安装部署
- 安装ambari部署hadoop集群
- hadoop学习记(1)--集群安装部署
- Hadoop集群安装部署
- hadoop 集群安装(二) HA部署
- Cloudera Hadoop集群安装部署,cdh3u6,Ubuntu12.04
- [Hadoop培训笔记]03-MapReduce集群的安装与部署
- Spark2.0.1 on yarn with hue 集群搭建部署(五)hue安装支持hadoop
- 完全分布模式hadoop集群安装配置之二 添加新节点组成分布式集群
- Hadoop2.6.0集群安装部署
- Hadoop建设工具Ambari的安装部署及完整使用(五)——Ambari使用之集群卸载
- hadoop-2.4.1 HA 分布式集群安装部署
- 我的Hadoop安装——使用Cloudera部署,管理Hadoop集群(离线安装CDH5.7.0)
- Hadoop教程(五)Hadoop分布式集群部署安装
- Hadoop集群(三节点)安装与部署
- hadoop集群部署安装
- ubuntu64位hadoop2.2.0全分布安装部署
- hadoop集群配置之hive1.2.0安装部署(远程mysql)
- hadoop集群环境搭建之zookeeper集群的安装部署