您的位置：首页 > 理论基础 > 计算机网络

ambari，hadoop本地源部署http://blog.chinaunix.net/xmlrpc.php?r=blog/article&uid=26230811&id=4023821

2014-11-19 11:27 751 查看

http://blog.chinaunix.net/xmlrpc.php?r=blog/article&uid=26230811&id=4023821

环境配置

Hostname	IP	OS
Ambari	172.16.235.164	Centos X64 base install
Maste.hadoop	172.16.235.165	Centos X64 base install
Datanode1.hadoop	172.16.235.166	Centos X64 base install

操作系统建议以software server 方式安装，base 方式安装后，需要安装Development tools，本文以base举例。

一：制作ambari，hadoop本地源

首先需要有一台能够联网的linux机器

1：制作ambari（1.4.1.25）本地源

[root@controller ~]# wget http://public-repo-1.hortonworks.com/ambari/centos6/1.x/updates/1.4.1.25/ambari.repo
[root@controller ~]# cp ambari.repo /etc/yum.repos.d/

将网络源中的包同步到本地：

reposync -r HDP-UTILS-1.1.0.16

reposync -r Updates-ambari-1.4.1.25

reposync -r ambari-1.x

会得到相应的三个目录，每个目录中有相应的rpm包

准备一台http服务器，本文以172.16.235.164作为http服务器发布安装源

mkdir /var/www/html/ambari

将 HDP-UTILS-1.1.0.16 Updates-ambari-1.4.1.25 ambari-1.x复制到 /var/www/html/ambari下

cd /var/www/html/ambari

createrepo ./ #没有createrepo命令自己去安装

这样ambari安装源就做好了，下载好jdk-6u31-linux-x64.bin备用

2：准备hortonworks hadoop （HDP）2.0.6本地源

准备HDP网络源

vim /etc/yum.repos.d/HDP.repo，添加以下内容

[HDP-2.0.6]

name=HDP

baseurl=http://public-repo-1.hortonworks.com/HDP/centos6/2.x/updates/2.0.6.0

enabled=1

gpgcheck=0

同步网络源到本地：

reposync -r HDP-2.0.6 #将近1G

讲 HDP-2.0.6 复制到http server的/var/www/html/hdp下

cd /var/www/html/hdp

createrepo ./

这样hadoop的本地源就做好了

3：制作系统本地源（安装ambari和hadoop时，可能会依赖些系统包，这时需要有系统的本地源来安装）

http server 挂载系统光盘

mount /dev/cdrom /var/www/html/centos/

二：安装ambari

系统准备：关闭iptables、selinux,启动ntpd服务

[root@ambari ]vim /etc/hosts

172.16.235.164 ambari

172.16.235.165 master.hadoop

172.16.235.166 datanode1.hadoop

配置与所有主机的无密码登入

[root@ambari ~]# ssh-keygen #一路回车（所有主机都要执行）

[root@ambari ~]# cd .ssh/

[root@ambari .ssh]# ls

id_rsa id_rsa.pub

[root@ambari .ssh]# scp id_rsa.pub master.hadoop:/root/.ssh/1

The authenticity of host 'master.hadoop (172.16.235.165)' can't be established.

RSA key fingerprint is fc:33:53:5f:f2:20:86:92:ac:96:09:e2:dc:e7:33:ea.

Are you sure you want to continue connecting (yes/no)? yes

Warning: Permanently added 'master.hadoop,172.16.235.165' (RSA) to the list of known hosts.

root@master.hadoop's password:

id_rsa.pub 100% 393 0.4KB/s 00:00

[root@ambari .ssh]# scp id_rsa.pub datanode1.hadoop:/root/.ssh/1

The authenticity of host 'datanode1.hadoop (172.16.235.166)' can't be established.

RSA key fingerprint is fc:33:53:5f:f2:20:86:92:ac:96:09:e2:dc:e7:33:ea.

Are you sure you want to continue connecting (yes/no)? yes

Warning: Permanently added 'datanode1.hadoop,172.16.235.166' (RSA) to the list of known hosts.

root@datanode1.hadoop's password:

id_rsa.pub 100% 393 0.4KB/s 00:00

登入master.hadoop

[root@master ~]# cd .ssh/

[root@master .ssh]# cat 1 >authorized_keys

登入datanode1.hadoop

[root@datanode1 ~]# cd .ssh/

[root@datanode1 .ssh]# cat 1 >authorized_keys

回到ambari 测试无密码登入

[root@ambari .ssh]# ssh master.hadoop

Last login: Wed Dec 4 11:11:15 2013 from 172.16.38.48

[root@master ~]# exit

logout

Connection to master.hadoop closed.

[root@ambari .ssh]# ssh datanode1.hadoop

Last login: Wed Dec 4 10:38:06 2013 from 172.16.38.48

[root@datanode1 ~]# exit

logout

Connection to datanode1.hadoop closed.

[root@ambari .ssh]#

安装与配置ambari服务：

[root@ambari ]vim /etc/yum.repos.d/ambari.repo 添加以下内容

[ambari-1.x]

name=Ambari 1.x

baseurl=http://172.16.235.164/ambari/

gpgcheck=0

enabled=1

[HDP-UTILS-1.1.0.16]

name=Hortonworks Data Platform Utils Version - HDP-UTILS-1.1.0.16

baseurl=http://172.16.235.164/ambari/

gpgcheck=0

enabled=1

[Updates-ambari-1.4.1.25]

name=ambari-1.4.1.25 - Updates

baseurl=http://172.16.235.164/ambari/

gpgcheck=

enabled=1

[root@ambari ]vim /etc/yum.repos.d/centos.repo 添加以下内容

[centos]

baseurl=http://172.16.235.164/centos/

enabled=1

gpgcheck=0

执行安装ambari：

[root@ambari yum.repos.d]# yum -y install ambari-server

Loaded plugins: fastestmirror, security

Repository 'Updates-ambari-1.4.1.25': Error parsing config: Error parsing "gpgcheck = ''": invalid boolean value

Repository 'centos' is missing name in configuration, using id

Loading mirror speeds from cached hostfile

Setting up Install Process

Resolving Dependencies

--> Running transaction check

---> Package ambari-server.noarch 0:1.4.1.25-1 will be installed

--> Processing Dependency: postgresql-server >= 8.1 for package: ambari-server-1.4.1.25-1.noarch

--> Running transaction check

---> Package postgresql-server.x86_64 0:8.4.13-1.el6_3 will be installed

--> Processing Dependency: postgresql-libs(x86-64) = 8.4.13-1.el6_3 for package: postgresql-server-8.4.13-1.el6_3.x86_64

--> Processing Dependency: postgresql(x86-64) = 8.4.13-1.el6_3 for package: postgresql-server-8.4.13-1.el6_3.x86_64

--> Processing Dependency: libpq.so.5()(64bit) for package: postgresql-server-8.4.13-1.el6_3.x86_64

--> Running transaction check

---> Package postgresql.x86_64 0:8.4.13-1.el6_3 will be installed

---> Package postgresql-libs.x86_64 0:8.4.13-1.el6_3 will be installed

--> Finished Dependency Resolution

Dependencies Resolved

====================================================================================================================================

Package Arch Version Repository Size

====================================================================================================================================

Installing:

ambari-server noarch 1.4.1.25-1 HDP-UTILS-1.1.0.16 37 M

Installing for dependencies:

postgresql x86_64 8.4.13-1.el6_3 centos 2.8 M

postgresql-libs x86_64 8.4.13-1.el6_3 centos 200 k

postgresql-server x86_64 8.4.13-1.el6_3 centos 3.4 M

Transaction Summary

====================================================================================================================================

Install 4 Package(s)

Total download size: 43 M

Installed size: 88 M

Downloading Packages:

(1/4): ambari-server-1.4.1.25-1.noarch.rpm | 37 MB 00:00

(2/4): postgresql-server-8.4.13-1.el6_3.x86_64.rpm | 3.4 MB 00:00

(3/4): postgresql-libs-8.4.13-1.el6_3.x86_64.rpm | 200 kB 00:00

(4/4): postgresql-8.4.13-1.el6_3.x86_64.rpm | 2.8 MB 00:00

------------------------------------------------------------------------------------------------------------------------------------

Total 43 MB/s | 43 MB 00:00

Running rpm_check_debug

Running Transaction Test

Transaction Test Succeeded

Running Transaction

Installing : postgresql-libs-8.4.13-1.el6_3.x86_64 1/4

Installing : postgresql-8.4.13-1.el6_3.x86_64 2/4

Installing : postgresql-server-8.4.13-1.el6_3.x86_64 3/4

Installing : ambari-server-1.4.1.25-1.noarch 4/4

Verifying : postgresql-server-8.4.13-1.el6_3.x86_64 1/4

Verifying : postgresql-libs-8.4.13-1.el6_3.x86_64 2/4

Verifying : ambari-server-1.4.1.25-1.noarch 3/4

Verifying : postgresql-8.4.13-1.el6_3.x86_64 4/4

Installed:

ambari-server.noarch 0:1.4.1.25-1

Dependency Installed:

postgresql.x86_64 0:8.4.13-1.el6_3 postgresql-libs.x86_64 0:8.4.13-1.el6_3 postgresql-server.x86_64 0:8.4.13-1.el6_3

Complete!

开始配置ambari 服务：

首先上传jdk文件

[root@ambari ]cd /var/lib/ambari-server/resources 将jdk-6u31-linux-x64.bin上传到该目录

[root@ambari resources]# chmod +x jdk-6u31-linux-x64.bin

[root@ambari yum.repos.d]# ambari-server setup

Using python /usr/bin/python2.6

Initializing...

Setup ambari-server

Checking SELinux...

SELinux status is 'enabled'

SELinux mode is 'permissive'

WARNING: SELinux is set to 'permissive' mode and temporarily disabled.

OK to continue [y/n] (y)?

Customize user account for ambari-server daemon [y/n] (n)?

Adjusting ambari-server permissions and ownership...

Checking iptables...

WARNING: iptables is running. Confirm the necessary Ambari ports are accessible. Refer to the Ambari documentation for more details on ports.

OK to continue [y/n] (y)?

Checking JDK...

JDK already exists, using /var/lib/ambari-server/resources/jdk-6u31-linux-x64.bin

Installing JDK to /usr/jdk64

Successfully installed JDK to /usr/jdk64/jdk1.6.0_31

Downloading JCE Policy archive from http://public-repo-1.hortonworks.com/ARTIFACTS/jce_policy-6.zip to /var/lib/ambari-server/resources/jce_policy-6.zip

To download the JCE Policy files you must accept the license terms found at http://www.oracle.com/technetwork/java/javase/terms/license/index.htmlNot accepting will result in errors when configuring Kerberos security.

Do you accept the Oracle Binary Code License Agreement [y/n] (y)?

Successfully downloaded JCE Policy archive to /var/lib/ambari-server/resources/jce_policy-6.zip

Completing setup...

Configuring database...

Enter advanced database configuration [y/n] (n)?

Default properties detected. Using built-in database.

Checking PostgreSQL...

Running initdb: This may take upto a minute.

About to start PostgreSQL

Configuring local database...

Connecting to the database. Attempt 1...

Configuring PostgreSQL...

Restarting PostgreSQL

Ambari Server 'setup' completed successfully.

[root@ambari ~]# ambari-server start

Using python /usr/bin/python2.6

Starting ambari-server

Ambari Server running with 'root' privileges.

Server PID at: /var/run/ambari-server/ambari-server.pid

Server out at: /var/log/ambari-server/ambari-server.out

Server log at: /var/log/ambari-server/ambari-server.log

Ambari Server 'start' completed successfully.

到此ambari安装好

三：本地安装hadoop

给ambari配置本地hadoo源地址：

[root@ambari ~]# cd /var/lib/ambari-server/resources/stacks/HDPLocal/2.0.6/repos/

[root@ambari repos]# vim repoinfo.xml 修改红色部分为本地源地址

<reposinfo>

<os type="centos6">

<repo>

<baseurl>http://172.16.235.164/hdp/</baseurl>

<repoid>HDP-2.0.6</repoid>

<reponame>HDP</reponame>

</repo>

</os>

这样在ambari执行hadoop安装时，会将本地源地址配置到所有主机上，在为所有主机安装ambari-agent时，会将ambari server下的/etc/yum.repo/ambari.repo文件复制到所有主机

主机准备（master.hadoop、datanode1.hadoop）配置相同，只举例master.hadoop

关闭iptables、selinux、启动ntpd服务（略）

[root@master ~]# vim /etc/hosts 添加以下

172.16.235.164 ambari

172.16.235.165 master.hadoop

172.16.235.166 datanode1.hadoop

[root@master ~]# vim /etc/yum.repos.d/centos.repo 添加以下内容

[root

[centos]

baseurl=http://172.16.235.164/centos/

enabled=1

gpgcheck=0

@master ~]# yum -y groupinstall "Development tools"

登入ambari管理页面

http://172.16.235.164:8080/ 账户：admin 密码：admin