Cloudera 5.7集群部署（二）--CM、CDH安装

1 Cloudera安装配置

1.1 安装前注意事项

前期在fengkongj上操作，再将配置好的安装包拷给fengkongb,fengkongc,fengkongd,fengkonge, fengkongf, fengkongg, fengkongh, fengkongi,
基本思路是：先在后台配置好cm(cloudera management)、parcels，然后在页面上操作cm，安装zookeeper、yarn、spark、hbase等。
参考官方文档：http://www.cloudera.com/documentation/enterprise/latest/topics/cm_ig_install_path_c.html

1.2 安装配置cm

1） 在http://archive.cloudera.com/cm5/cm/5/下载：cloudera-manager-el6-cm5.7.0_x86_64.tar.gz

2） 创建/opt/cloudera-manager目录,上传并将tar包解压到该目录下

3） 创建用户

useradd --system --home=/opt/cloudera-manager/cm-5.7.0/run/cloudera-scm-server --no-create-home --comment "Cloudera SCM User" cloudera-scm

4） 修改拥有者用户

chown -R cloudera-scm:cloudera-scm /opt/cloudera-manager

1.3 安装配置parcels

1） cloudera-cdh5 parcels下载地址: http://archive.cloudera.com/cdh5/parcels/5.7.0/



2） 按照官网的文件格式下载源文件, 上传到linux.

目录为/opt/cloudera/parcel-repo,文件夹拥有者要改为cloudera-scm：

chown -R cloudera-scm:cloudera-scm /opt/cloudera

3） 并在/var/www/html上配置链接：

ln -s /opt/cloudera/parcel-repo /var/www/html/CDH5.7.0

打开浏览器，输入192.168.89.10/CDH5.7.3，查看效果：



4） 修改manifest.json,删除剩下以下parcelName的字符簇:

…

“parcelName”: “CDH-5.4.0-1.cdh5.4.0.p0.27-el7.parcel”,

….

5） 修改sha1为sha:

mv CDH-5.5.2-1.cdh5.5.2.p0.4-el7.parcel.sha1 CDH-5.5.2-1.cdh5.5.2.p0.4-el7.parcel.sha

1.4 配置cdh库和用户

注：可选

create database amon DEFAULT CHARACTER SETutf8;

grant all on amon.* TO ’ amon’@’%’ IDENTIFIEDBY ‘amon_123’;

create database rman DEFAULT CHARACTER SETutf8;

grant all on rman.* TO ‘rman’@’%’ IDENTIFIEDBY ‘rman_123’;

create database metastore DEFAULT CHARACTERSET utf8;

grant all on metastore.* TO ‘hive’@’%’IDENTIFIED BY ‘hive_123’;

create database sentry DEFAULT CHARACTER SETutf8;

grant all on sentry.* TO ‘sentry’@’%’IDENTIFIED BY ‘sentry_123’;

create database nav DEFAULT CHARACTER SETutf8;

grant all on nav.* TO ‘nav’@’%’ IDENTIFIED BY’nav_123’;

create database navms DEFAULT CHARACTER SETutf8;

grant all on navms.* TO ‘navms’@’%’ IDENTIFIEDBY ‘navms_123’;

create database hue DEFAULT CHARACTER SETutf8;

grant all on hue.* TO ‘hue’@’%’ IDENTIFIED BY’hue_123’;

create database hive DEFAULT CHARACTER SETutf8;

grant all on hive.* TO ‘hive’@’%’ IDENTIFIEDBY ‘hive_123’;

create database oozie DEFAULT CHARACTER SETutf8;

grant all on oozie.* TO ‘oozie’@’%’ IDENTIFIEDBY ‘oozie_123’;

create database cdhreport DEFAULT CHARACTERSET utf8;

grant all on cdhreport.* TO ‘report’@’%’IDENTIFIED BY ‘report_123’;

create database cdhmonitor DEFAULT CHARACTERSET utf8;

grant all on cdhmonitor.* TO ‘monitor’@’%’IDENTIFIED BY ‘monitor_123’;

mysql> flush privileges;

1.5 mysql客户端驱动

1） 下载http://dev.mysql.com/downloads/connector/j/5.1.html

2） 上传mysql-connector-java-5.1.38.tar.gz

3） 解压后将mysql-connector-java-5.1.38-bin.jar放入/opt/cloudera-manager/cm-5.7.0/share/cmf/lib

4） 也放一份在下面目录

cp mysql-connector-java-5.1.38-bin.jar /usr/share/java/mysql-connector-java.jar

1.6 创建scm数据库

vi /opt/cloudera-manager/cm-5.7.0/share/cmf/schema/scm_prepare_database.sh

JAVA_HOME=/opt/jdk1.8.0_60

/opt/cloudera-manager/cm-5.7.0/share/cmf/schema/scm_prepare_database.sh mysql -h fengkonga -P 3306 -uroot -p123456 --scm-host fengkonga scm scm scm_123

注：输出一下内容，说明成功：

…

Executing: /opt/jdk1.8.0_60/bin/java -cp/usr/share/java/mysql-connector-java.jar:/usr/share/java/oracle-connector-java.jar:/opt/cloudera-manager/cm-5.7.3/share/cmf/schema/.

./lib/* com.cloudera.enterprise.dbutil.DbCommandExecutor /opt/cloudera-manager/cm-5.7.0/etc/cloudera-scm-server/db.propertiescom.cloudera.cmf.db.[main] DbCommandExecutor INFOSuccessfullyconnected to database.

All done,your SCM database is configured correctly!

1.7 Cloudera agent安装

feng
4000
kongb,fengkongc,fengkongd, fengkonge, fengkongf, fengkongg, fengkongh, fengkongi:

vi /opt/cloudera-manager/cm-5.7.0/etc/cloudera-scm-agent/config.ini

…

server_host=192.168.89.10

…

fengkongj:

sudo scp -r cloudera fengkongb:/opt
sudo scp -rcloudera-manager fengkongb:/opt
sudo scp -r cloudera fengkongc:/opt
sudo scp -rcloudera-manager fengkongc:/opt
sudo scp -r cloudera fengkongd:/opt
sudo scp -rcloudera-manager fengkongd:/opt
sudo scp -r cloudera fengkonge:/opt
sudo scp -rcloudera-manager fengkonge:/opt
sudo scp -r cloudera fengkongf:/opt
sudo scp -rcloudera-manager fengkongf:/opt
sudo scp -r cloudera fengkongg:/opt
sudo scp -rcloudera-manager fengkongg:/opt
sudo scp -r cloudera fengkongh:/opt
sudo scp -rcloudera-manager fengkongh:/opt
sudo scp -r cloudera fengkongi:/opt
sudo scp -r cloudera-managerfengkongi:/opt

fengkongb,fengkongc,fengkongd, fengkonge, fengkongf, fengkongg, fengkongh, fengkongi:

useradd --system--home=/opt/cloudera-manager/cm-5.7.0/run/cloudera-scm-server --no-create-home --comment "Cloudera SCM User" cloudera-scm
chown -R cloudera-scm:cloudera-scm /opt/cloudera-manager
chown -R cloudera-scm:cloudera-scm /opt/cloudera

1.8 Cloudera server启动

fengkongj：

/opt/cloudera-manager/cm-5.7.0/etc/init.d/cloudera-scm-server start
tail -f /opt/cloudera-manager/cm-5.7.0/log/cloudera-scm-server/cloudera-scm-server.log

注：server端在初始化，看日志知道出现以下字样表明初始化完成：

。。。

2016-03-23 11:17:50,333 INFOWebServerImpl:org.mortbay.log: jetty-6.1.26.cloudera.4

2016-03-23 11:17:50,334 INFOWebServerImpl:org.mortbay.log: Started SelectChannelConnector@0.0.0.0:7180

2016-03-23 11:17:50,334 INFOWebServerImpl:com.cloudera.server.cmf.WebServerImpl: Started Jetty server.

1.9 Cloudera agent启动

/opt/cloudera-manager/cm-5.7.0/etc/init.d/cloudera-scm-agent start
tail -f /opt/cloudera-manager/cm-5.7.0/log/cloudera-scm-agent/cloudera-scm-agent.log

注：如果启动失败，检查/opt/cloudera-manager/cm-5.7.0/run目录下是否有cloudera-scm-agent文件夹，没有则新建：

mkdir cloudera-scm-agent
chown cloudera-scm:cloudera-scm cloudera-scm-agent/

注：

When you start the Cloudera Manager Server and Agents,Cloudera Manager assumes you are not already running HDFS and MapReduce.

参考

http://www.cloudera.com/documentation/enterprise/5-5-x/topics/cm_ig_install_path_c.html?scroll=cmig_topic_6_7

1.10 Cdh倚赖包安装

yum -y install bind-utils psmisc libxsltzlib sqlite cyrus-sasl-plain cyrus-sasl-gssapi fuse portmap fuse-libsredhat-lsb

1.11 访问Cloudera管理台WEB

浏览器输入：

192.168.89.10:7180

默认：

用户：admin

密码：admin

出现登录界面则表示cm安装成功。