Hadoop 离线安装 CDH5.1 第一章:离线环境部署

配置主机映射关系
[hadoop@slave03 ~]$ cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
10.15.5.200 master.hadoop
10.15.5.201 slave01.hadoop
10.15.5.202 slave02.hadoop
10.15.5.203 slave03.hadoop

如上有4台主机，每台hosts都已经编辑过。所有的主机都已经编辑过/etc/ssh/sshd_config

RSAAuthentication yes
PubkeyAuthentication yes
AuthorizedKeysFile     .ssh/authorized_keys

所有的authorized_keys都已经过hadoop用户ssh-keygen -t rsa生成过空密码，并写入成功如下

[hadoop@slave03 ~]$ cat .ssh/authorized_keys
ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAsZbjTNyyW4ZPZ2cCbUk/m3F5LGTsLKzv/TzuG0LJZ9E/UN2dV4j7dhHbN1olbsvFp4oyGGKhFxn036PljJRfEWhdY0dDagWklH3oijRYr+UzBTMf8sIR9FxKD540rhC0NCEP+KaNZXAhndB7BTZTIkytMlfCboOcbRjt1XU+3Yhi7Dlp6Z6cu2oSgYo4iYpX1anHUMRuMLl5/mQ1INeoRMerBTzGfIYAIQF5Du2bE9HRSClyUal2O6QkKhZGHRa0+3jjFY/vER0R+ruLJZLVfQQkJffQJEz1qk4e/V2GEg5ScjAPyojKg+R/mpWNAwivbsJ8pe4YWnUxBNQFphB/7w== hadoop@master.hadoop
ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAwk3DPdzRD41+saaLwnqLkMnLVf5m5br2hb6lCNDF+3vEiezIKpoxp2lzqOp7kUOMrubcJRVcdXJWgrzbp
bmUzkqF1YyaXqZ1OaIR+by20glTnYemkTE3ljknVgrQXXyVUxYDw8TGx3sJ/sw5EgrUEM3xB98Yz9j9JUL7lkGzahjYO02JcaKU83YfqUjCt9QIhnLiIo
7BPs/NVK6UP6Srn0fCvsVXS2XqkHzjPhIjiQAiyDINMz3ZCHxnFsOQm9I//R61YYrHLfNPQKOnBMRV4/Q2FPGXa5wgx022i3VaUiXOSGlTV2ugoC22Q1n
422q9P3kWevhL4nDGizn5rEOCEQ== hadoop@slave01.hadoop
ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA3FI6rK31HE5MGpqQS/60w8lHruIHSomjdWhmiN+FXKoTyZzqHwxyRS7j1Y8mplIpoUzAOlfPWbgm2AQto
jFbLJfOtN4vvBkniw6OEcvjojyOQdyHsEr907L9K9GRq0VtFkjo/YxvYQf823I+14sOdZ8fCZrue0NmcJ0RMifLHrmMGeJ+LLN5fyWYygvDQIenF/DWNN
KNgp15v/eVVaRokJ00sSooeikJlv7DL7EWK8UBc0M6Fum7Y2L4WXVmjavAoG5jxN2tJXQvwgbzFaJ1OWFzZzwsOlI+khVvb13ARGnSd0Lp1PDQ0V64Jyh
Vj70TllSgvrhz2vkMFbe82TJM/w== hadoop@slave02.hadoop
ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA20RPILfJJpN7QsiiGfGaKmcibuwDexslHob9CmzJUFJFxm8sMXtvOMuuipknCl4Aya5+U3AjO5xPh5ZcD
rCMSgUrgerwi9ofF/2Od9efOCd4JMKV0V/nsAHdUtCxCeBGOyaPdZ9rEnEqKtu3dF8fW2zOl3UGJ3GrzgTeAvG1Rlet/+jL4RJ/ob+CzVwX3pZQ5YGrwU
sPbQ70Sn4aZSM56VTjx9QaqWlXxQDiHgyI+FN2OCsfqB9kiDHSu3DZ2Jjyil+7kOeCUuoCOXZqjG3VUGURYVtguEIlRuHEOQV9hWc7zFVADt5AKpi0c4q
i3Bo1P3gYVJ3/wYBqK8YrrHT8Jw== hadoop@slave03.hadoop

给每台主机安装Java, 如果已有Java 请先卸载，从oracle网站得到链接如下:

http://download.oracle.com/otn-pub/java/jdk/7u67-b01/jdk-7u67-linux-x64.tar.gz
下载该文件到本地FTP，因为直接wget会被oracle拒绝

[hadoop@slave03 usr]$ cd java
[hadoop@slave03 java]$ pwd
/usr/java
ftp> bin
200 Type set to I.
ftp> get jdk-7u67-linux-x64.tar.gz
local: jdk-7u67-linux-x64.tar.gz remote: jdk-7u67-linux-x64.tar.gz
227 Entering Passive Mode (10,15,15,63,19,48).
150 Opening data connection for jdk-7u67-linux-x64.tar.gz.

解压数据包到/usr/java目录下

[hadoop@slave03 java]$ sudo tar -zxvf jdk-7u67-linux-x64.tar.gz

修改系统变量，指定java路径，通过source命令 应用该配置,验证Java版本

[hadoop@slave03 jdk1.7.0_67]$ sudo vim /etc/profile
# JAVA environment
export JAVA_HOME=/usr/java/jdk1.7.0_67
export CLASSPATH=.:$CLASSPATH:$JAVA_HOME/lib:$JAVA_HOME/jre/lib
export PATH=$PATH:$JAVA_HOME/bin:$JAVA_HOME/jre/bin
[hadoop@slave03 jdk1.7.0_67]$ source /etc/profile
[hadoop@slave03 jdk1.7.0_67]$ java -version
java version "1.7.0_67"
Java(TM) SE Runtime Environment (build 1.7.0_67-b01)
Java HotSpot(TM) 64-Bit Server VM (build 24.65-b04, mixed mode)

把安装文件分发到其他主机,参照以上步骤，安装java并配置变量并验证。

[hadoop@slave03 java]$ scp jdk-7u67-linux-x64.tar.gz hadoop@master.hadoop:~/
[hadoop@slave03 java]$ scp jdk-7u67-linux-x64.tar.gz hadoop@slave01.hadoop:~/
[hadoop@slave03 java]$ scp jdk-7u67-linux-x64.tar.gz hadoop@master.hadoop:~/

配置NTP服务：集群的时间必须保持一致，否则同步会有问题。所以需要配置NTP服务保证集群正常运行，将master外联到ntp服务器，其他主机和master同步，或者在企业网中搭建专用的NTP服务器都可以,可以使用210.72.145.44 (国家授时中心服务器IP地址)

[hadoop@master ~]$ sudo ntpdate -u 65.55.56.206
21 Aug 23:30:04 ntpdate[2355]: step time server 65.55.56.206 offset 455.994813 sec

看到硬件时间和系统时间不一样，将时间写入硬件(执行同步后，不会自动写).

[hadoop@master ~]$ sudo hwclock -r
Thu 21 Aug 2014 11:25:07 PM CST  -0.736104 seconds
[hadoop@master ~]$ sudo date
Thu Aug 21 23:33:01 CST 2014
[hadoop@master ~]$ sudo hwclock -w
[hadoop@master ~]$ sudo hwclock -r
Thu 21 Aug 2014 11:34:36 PM CST  -0.053046 seconds

配置NTP Server参数,默认的不用动 修改和添加下面两行即可，最终验证如下

[hadoop@master ~]$ sudo vim /etc/ntp.conf
restrict default  nomodify notra
server 65.55.56.206 prefer
[hadoop@master ~]$ sudo service ntpd start
Starting ntpd:                                             [  OK  ]
[hadoop@master ~]$ sudo chkconfig ntpd on
[hadoop@master ~]$ sudo ntpstat
synchronised to NTP server (202.112.31.197) at stratum 3
time correct to within 1053 ms
polling server every 64 s

如下，将所有的datanode 配置NTP Client ，并同步 然后写入硬件时钟

[hadoop@slave01 ~]$ sudo vim /etc/ntp.conf
server 10.15.5.200
[hadoop@slave03 ~]$ sudo ntpdate -u 10.15.5.200
22 Aug 00:00:03 ntpdate[2404]: step time server 10.15.5.200 offset 4964.547679 sec
[hadoop@slave03 ~]$ sudo hwclock -r
Thu 21 Aug 2014 10:37:35 PM CST  -0.447982 seconds
[hadoop@slave03 ~]$ sudo hwclock -w
[hadoop@slave03 ~]$ sudo hwclock -r
Fri 22 Aug 2014 12:00:30 AM CST  -0.248008 seconds

安装mysql，这里参考了一个日本人的博客，使用日本那边大学的源： http://changineer.info/server/mysql/mysql_installation_linux.html

[hadoop@master ~]$ sudo yum install http://ftp.jaist.ac.jp/pub/mysql/Downloads/MySQL-5.6/MySQL-server-5.6.20-1.el6.x86_64.rpm[/code]出现了错误 
file /usr/share/mysql/czech/errmsg.sys from install of MySQL-server-5.6.20-1.el6.x86_64
conflicts with file from package mysql-libs-5.1.73-3.el6_5.x86_64
#解决办法也是参照日本人的博客来做的移除掉mysql-libs然后重新yum安装#
[hadoop@master ~]$ sudo yum remove mysql-libs
[hadoop@master ~]$ sudo yum install
mysql/Downloads/MySQL-5.6/MySQL-server-5.6.20-1.el6.x86_64.rpm
Installed:
MySQL-server.x86_64 0:5.6.20-1.el6
Complete!
启动mysql ,设置为开机启动（需要安装client来连接）

[hadoop@master ~]$ sudo service mysql start
Starting MySQL.. SUCCESS!
[hadoop@master ~]$ sudo chkconfig mysql on
MySQL使用随机密码，通过cat主目录下文件可以得到,进入mysql界面,修改root密码为hadoop
[root@master hadoop]# cat ~/.mysql_secret
# The random password set for the root user at Fri Aug 22 13:55:10 2014 (local time): TPQKIgffMJGcFfFG
[root@master hadoop]# mysql -u root -p
#复制到这里#
Enter password:
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql>
mysql> use mysql;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed
mysql> update user set password=password("hadoop") where user='root';
Query OK, 4 rows affected (0.00 sec)
Rows matched: 4  Changed: 4  Warnings: 0
建立数据库，分配库权限。
mysql> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| mysql              |
| performance_schema |
| test               |
+--------------------+
4 rows in set (0.00 sec)
mysql> create databse amon DEFAULT CHARSET utf8 COLLATE utf8_general_ci;
Query OK, 1 row affected (0.00 sec)
mysql> create database amon DEFAULT CHARSET utf8 COLLATE utf8_general_ci;
Query OK, 1 row affected (0.00 sec)
mysql> create database hue DEFAULT CHARSET utf8 COLLATE utf8_general_ci;
Query OK, 1 row affected (0.00 sec)
mysql> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| amon               |
| hive               |
| hue                |
| mysql              |
| performance_schema |
| test               |
+--------------------+
7 rows in set (0.00 sec)
mysql> grant all on *.* to 'root'@'%' identified by 'hadoop' with grant option;
Query OK, 0 rows affected (0.00 sec)


下载离线的cloudera manager安装包, 放置于/opt目录，并解压，改名

[hadoop@master opt]$ sudo tar -xzvf cloudera-manager-el5-cm5.1.0_x86_64.tar.gz
[hadoop@master opt]$ sudo mv cm-5.1.0 cloudera-manager-5.1.0