您的位置：首页 > 运维架构

hadoop2.7.4在ubuntu16桌面版上的集群安装

2017-12-28 10:04 267 查看
Hadoop在ubuntu16桌面版上的集群安装

参考文档1：http://blog.csdn.net/mark_lq/article/details/53384358

参考文档2：http://blog.csdn.net/quiet_girl/article/details/74352190

说明：由于我的ubuntu系统中已经有tensorflow、python2.7、syntaxnet等，并不是全新的系统，安装hadoop集群也是在此之上。安装中也是各种坑，最大的一个坑是ssh免密码登录，其次是安装前要新建一个hadoop账号。我是按照参考文档1进行安装，启动出问题后又想到了参考文档2。在此不说人家的文档写的不好什么的，因为每个人的系统环境不同，你安装启动不了就说人家的文档没写好。言归正传，开始：

本来是写在最后的结束语，但想想还是写在开关：

三个系统建议安装好一个后，把下面的步骤全走完再克隆，要干的事情是修改/etc/hostname，/etc/hosts，以及在master上执行以下命令：

sudoscp./authorized_keys
hadoop@slaver1:~/.ssh

sudoscp./authorized_keys
hadoop@slaver2:~/.ssh

最后是在master上格式化和启动。



1.系统环境

Ubuntu16.04

vmware12.5.2build-4638234

hadoop2.7.4

java1.8.0_131

master：192.168.93.140

slaver1：192.168.93.141

slaver2：192.168.93.142

2.部署步骤

2.1BasicRequirements

1.、添加hadoop用户，并添加到sudoers

sudo[code]adduserhadoop

说明：sudoers文件保存时要用wq!

sudo[code]vim/etc/sudoers

添加如下：

#Userprivilegespecification[code]

root[code]
ALL
=(
ALL
:
ALL
)
ALL

hadoop[code]
ALL
=(
ALL
:
ALL
)
ALL[/code]
2、切换到hadoop用户：

说明：在桌面版下切换用户按右上角的设置图标进行切换，不要用suhadoop

suhadoop

3、修改/etc/hostname主机名为master

sudo[code]vim/etc/hostname

4、修改/etc/hosts

127.0.0.1[code]localhost

127.0.1.1[code]localhost
.localdomain
localhost


#ThefollowinglinesaredesirableforIPv6capablehosts[code]

::
1
localhostip6-localhostip6-loopback

ff02:[code]:
1
ip6-allnodes

ff02:[code]:
2
ip6-allrouters


#hadoopnodes[code]

192.168.93.140[code]master

192.168.93.141[code]slave1

192.168.93.142[code]slave2

说明：jdk我用的是openjdk,默认安装在/usr/lib/jvm/java-1.8.0-openjdk-amd64

5、安装配置java环境

下载jdk1.8解压到/usr/local目录下（为了保证所有用户都能使用），修改/etc/profile，并生效：

#setjdkclasspath[code]

exportJAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-amd64

exportJRE_HOME=$JAVA_HOME/jre


exportPATH=$JAVA_HOME/bin:$JAVA_HOME/jre/bin:$PATH

exportCLASSPATH=$CLASSPATH:.:$JAVA_HOME/lib:$JAVA_HOME/jre/lib

source[code]/etc/profile


验证jdk是否安装配置成功

hadoop@master:~$jjava-version

openjdkversion"1.8.0_131"

OpenJDKRuntimeEnvironment(build1.8.0_131-8u131-b11-2ubuntu1.16.04.3-b11)

OpenJDK64-BitServerVM(build25.131-b11,mixedmode)
1[/code]

6、安装openssh-server

sudoapt-getinstallssh

ssh-keygen-trsa-P''-f~/.ssh/id_rsa

cat~/.ssh/id_rsa.pub>>~/.ssh/authorized_keys


7、对于slave1和slave2可采用虚拟机clone的方法实现复制，复制主机后注意修改/etc/hostname为slave1和slave2

8、配置master节点可通过SSH无密码访问slave1和slave2节点

将生成的authorized_keys文件复制到slave1和slave2的.ssh目录下

sudoscp./authorized_keyshadoop@slaver1:~/.ssh

sudoscp./authorized_keyshadoop@slaver2:~/.ssh

测试master节点无密码访问slave1和slave2节点：

sshslaver1

sshslaver2


输出：

hadoop@master:/usr/lib/jvm/java-1.8.0-openjdk-amd64$sshslaver1

WelcometoUbuntu16.04.3LTS(GNU/Linux4.10.0-35-genericx86_64)


*Documentation:'target='_blank'>https://help.ubuntu.com[/code]
*Management:'target='_blank'>https://landscape.canonical.com[/code]
*Support:'target='_blank'>https://ubuntu.com/advantage[/code]

50packagescanbeupdated.

0updatesaresecurityupdates.


Lastlogin:TueSep1901:20:082017from192.168.93.140

hadoop@slaver1:~$

2.2Hadoop2.7ClusterSetup
说明：先要在hadoop用户下建文件夹software,cd到文件夹下去下载
hadoop-2.7.4.tar.gz
，命令如下：
wget1、hadoop'target='_blank'>http://mirrors.tuna.tsinghua.edu.cn/apache/hadoop/common/hadoop-2.7.4/hadoop-2.7.4.tar.gz
1、hadoop用户目录下解压下载的
hadoop-2.7.4.tar.gz


·
hadoop@master:~/software$ll

·
total260460

·
drwxrwxr-x4hadoophadoop4096Sep1903:43./

·
drwxr-xr-x19hadoophadoop4096Sep1902:37../

·
drwxrwxr-x3hadoophadoop4096Sep1903:43hadoop-2.7.0/

·
drwxr-xr-x11hadoophadoop4096Sep1902:35hadoop-2.7.4/

·[code]-rw-rw-r--1hadoophadoop266688029Aug601:15hadoop-2.7.4.tar.gz

2、配置hadoop的环境变量

sudo[code]vim/etc/profile


配置如下：

#sethadoopclasspath[code]

export[code]HADOOP_HOME=/home/hadoop/software/hadoop-
2.7
.
4

export[code]HADOOP_MAPRED_HOME=
$HADOOP_HOME

export[code]HADOOP_COMMON_HOME=
$HADOOP_HOME

export[code]HADOOP_HDFS_HOME=
$HADOOP_HOME

export[code]YARN_HOME=
$HADOOP_HOME

export[code]JAVA_LIBRARY_PATH=
$HADOOP_HOME
/lib/native

export[code]HADOOP_CONF_DIR=
$HADOOP_HOME
/etc/hadoop

export[code]YARN_CONF_DIR=
$HADOOP_HOME
/etc/hadoop

export[code]HADOOP_PREFIX=
$HADOOP_HOME

export[code]CLASSPATH=
$CLASSPATH
:.:
$HADOOP_HOME
/bin

生效文件:source/etc/profile

3、配置hadoop的配置文件，主要配置core-site.xml、hdfs-site.xml、mapred-site.xml、yarn-site.xml文件。

core-site.xml

<configuration>[code]

<property>

<name>
fs.defaultFS
</name>

<!--master:/etc/hosts配置的域名master-->

<value>
hdfs://master:9000/
</value>

</property>

</configuration>




hdfs-site.xml

<configuration>

<property>

<name>
dfs.namenode.name.dir
</name>

<value>
/home/hadoop/software/hadoop-2.7.0/dfs/namenode
</value>

</property>

<property>

<name>
dfs.datanode.data.dir
</name>

<value>
/home/hadoop/software/hadoop-2.7.0/dfs/datanode
</value>

</property>

<property>

<name>
dfs.replication
</name>

<value>
1
</value>

</property>

<property>

<name>
dfs.namenode.secondary.http-address
</name>

<value>
master:9001
</value>

</property>

</configuration>


说明：mapred-site.xml文件在etc/hadoop下找不到，解决办法是

sudocpmapred-site.xml.templatemapred-site.xml


mapred-site.xml

<configuration>[code]

<property>

<name>
mapreduce.framework.name
</name>

<value>
yarn
</value>

</property>

<property>

<name>
mapreduce.jobhistory.address
</name>

<value>
master:10020
</value>

</property>

<property>

<name>
mapreduce.jobhistory.webapp.address
</name>

<value>
master:19888
</value>

</property>

</configuration>




yarn-site.xml

<configuration>[code]

<property>

<name>
yarn.nodemanager.aux-services
</name>

<value>
mapreduce_shuffle
</value>

</property>

<property>

<name>
yarn.nodemanager.aux-services.mapreduce.shuffle.class
</name>

<value>
org.apache.hadoop.mapred.ShuffleHandler
</value>

</property>

<property>

<name>
yarn.resourcemanager.address
</name>

<value>
master:8032
</value>

</property>

<property>

<name>
yarn.resourcemanager.scheduler.address
</name>

<value>
master:8030
</value>

</property>

<property>

<name>
yarn.resourcemanager.resource-tracker.address
</name>

<value>
master:8031
</value>

</property>

<property>

<name>
yarn.resourcemanager.admin.address
</name>

<value>
master:8033
</value>

</property>

<property>

<name>
yarn.resourcemanager.webapp.address
</name>

<value>
master:8088
</value>

</property>

</configuration>


4、修改env环境变量文件，为hadoop-env.sh、mapred-env.sh、yarn-env.sh文件添加JAVA_HOME：

#Thejavaimplementationtouse.[code]

export
JAVA_HOME
=
/usr/lib/jvm/java-1.8.0-openjdk-amd64[/code]

5、在etc/hadoop下配置masters和slaves文件：

但只找到slaves文件，那就复制一份

cpslavesmasters

文件masters中的内容为master

文件slaves中的内容为slaver1,slaver2

6、向slave1和slave2节点复制hadoop2.7.4整个目录至相同的位置

scp-rhadoop-2.7.4/hadoop@slaver1:~/software
scp-rhadoop-2.7.4/hadoop@slaver2:~/software
2.3StartHadoopclusterfrommaster
1、初始格式化文件系统bin/hdfsnamenode-format

·[code]hadoop@master:~/software/hadoop-2.7.4/bin$./hdfsnamenode-format

输出17/09/1903:43:18INFOcommon.Storage:Storagedirectory/home/hadoop/software/hadoop-2.7.0/dfs/namenodehasbeensuccessfullyformatted.

17/09/1903:43:18INFOnamenode.FSImageFormatProtobuf:Savingimagefile/home/hadoop/software/hadoop-2.7.0/dfs/namenode/current/fsimage.ckpt_0000000000000000000usingnocompression

17/09/1903:43:18INFOnamenode.FSImageFormatProtobuf:Imagefile/home/hadoop/software/hadoop-2.7.0/dfs/namenode/current/fsimage.ckpt_0000000000000000000ofsize322bytessavedin0seconds.

17/09/1903:43:18INFOnamenode.NNStorageRetentionManager:Goingtoretain1imageswithtxid>=0

17/09/1903:43:18INFOutil.ExitUtil:Exitingwithstatus0

17/09/1903:43:18INFOnamenode.NameNode:SHUTDOWN_MSG:

/************************************************************

SHUTDOWN_MSG:ShuttingdownNameNodeatmaster/192.168.93.140

************************************************************/

启动Hadoop集群start-all.sh

hadoop@master:~/software/hadoop-2.7.4/sbin$./start-all.sh


ThisscriptisDeprecated.Insteadusestart-dfs.shandstart-yarn.sh


Startingnamenodeson[master]


master:startingnamenode,loggingto/home/hadoop/software/hadoop-2.7.4/logs/hadoop-hadoop-namenode-master.out


slaver2:startingdatanode,loggingto/home/hadoop/software/hadoop-2.7.4/logs/hadoop-hadoop-datanode-slaver2.out


slaver1:startingdatanode,loggingto/home/hadoop/software/hadoop-2.7.4/logs/hadoop-hadoop-datanode-slaver1.out


Startingsecondarynamenodes[master]


master:startingsecondarynamenode,loggingto/home/hadoop/software/hadoop-2.7.4/logs/hadoop-hadoop-secondarynamenode-master.out


startingyarndaemons


startingresourcemanager,loggingto/home/hadoop/software/hadoop-2.7.4/logs/yarn-hadoop-resourcemanager-master.out


slaver1:startingnodemanager,loggingto/home/hadoop/software/hadoop-2.7.4/logs/yarn-hadoop-nodemanager-slaver1.out


slaver2:startingnodemanager,loggingto/home/hadoop/software/hadoop-2.7.4/logs/yarn-hadoop-nodemanager-slaver2.out


hadoop@master:~/software/hadoop-2.7.4/sbin$


jps输出运行的java进程：

浏览器查看HDFS：http://192.168.93.140:50070

Overview'master:9000'(active)

Started:TueSep1903:43:41PDT2017

Version:2.7.4,rcd915e1e8d9d0131462a0b7301586c175728a282

Compiled:2017-08-01T00:29Zbykshvachkfrombranch-2.7.4

ClusterID:CID-232a6925-5563-411b-9eb9-828aa4623ed0

BlockPoolID:BP-2106543616-192.168.93.140-1505817798572

Summary

Securityisoff.

Safemodeisoff.

1filesanddirectories,0blocks=1totalfilesystemobject(s).

HeapMemoryused44.74MBof221MBHeapMemory.MaxHeapMemoryis889MB.

NonHeapMemoryused45.64MBof46.5MBCommitedNonHeapMemory.MaxNonHeapMemoryis-1B.

浏览器查看mapreduce：http://192.168.93.140:8088

注意：在
hdfsnamenode-format
或
start-all.sh
运行HDFS或Mapreduce无法正常启动时（master节点或slave节点），可将master节点和slave节点目录下的dfs、logs、tmp等目录删除，重新
hdfsnamenode-format
，再运行
start-all.sh


2.4StopHadoopclusterfrommaster
hadoop@master:~/software/hadoop-2.7.4/sbin$jps


9318ResourceManager


8840NameNode


9098SecondaryNameNode


15628Jps


hadoop@master:~/software/hadoop-2.7.4/sbin$./stop-all.sh


ThisscriptisDeprecated.Insteadusestop-dfs.shandstop-yarn.sh


Stoppingnamenodeson[master]


master:stoppingnamenode


slaver1:stoppingdatanode


slaver2:stoppingdatanode


Stoppingsecondarynamenodes[master]


master:stoppingsecondarynamenode


stoppingyarndaemons


stoppingresourcemanager


slaver1:stoppingnodemanager


slaver2:stoppingnodemanager


noproxyservertostop


hadoop@master:~/software/hadoop-2.7.4/sbin$




hadoop
与hbase兼容列表

                                            

                                        

                        
                        内容来自用户分享和网络整理，不保证内容的准确性，如有侵权内容，可联系管理员处理 点击这里给我发消息 
                    

                    

                    

                        

                         标签： 
                                                

                        

                    


                

            


            

                
相关文章推荐

                

                
            

        

        

            

            
            

                

                    
新的分享

                    

                        
                    

                

            


            

                

                    
章节导航