您的位置：首页 > 运维架构

zabbix+flume+hadoop安装及其监控

2016-05-18 17:22 351 查看

1 Hadoop 2.7.1安装部署

1.2 创建用户

[root@iZ94bxra9qaZ ~]# useradd yq

1.3 用户赋权限

[root@iZ94bxra9qaZ ~]# chown yq /home/yq/
[root@iZ94bxra9qaZ ~]# chmod -R 777 /home/yq/

1.4 配置为可以无密码登录本机

如果没有ssh 则需要安装机器可以联网使用 sudo apt-getinstall ssh 命令即可以安装。

查看yq用户下是否存在.ssh文件夹
[root@iZ94bxra9qaZ ~]# ls -a /home/yq/
. .. .bash_logout .bash_profile .bashrc

如果没有，手动创建一个
[yq@iZ94bxra9qaZ ~]$ ssh-keygen -t rsa -f~/.ssh/id_rsa
或者
[yq@iZ94bxra9qaZ ~]$ ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa
Generating public/private dsa key pair.
Your identification has been saved in/home/yq/.ssh/id_dsa.
Your public key has been saved in/home/yq/.ssh/id_dsa.pub.
The key fingerprint is:
71:c1:60:84:36:65:7b:8c:c5:89:20:49:11:3d:73:a5yq@iZ94bxra9qaZ
The key's randomart image is:
+--[ DSA 1024]----+
| .==.=**+. |
| ..Boo*+. |
| . =E + |
| +   |
| S |
|   |
|   |
|   |
|   |
+-----------------+
[yq@iZ94bxra9qaZ ~]$
ssh-keygen 代表生成密钥：-t 表示指定生产的密钥类型；dsa是dsa密钥认证的意思，即密钥类型；-P用于提供蜜语；-f指定生成的密钥文件。

把id_dsa.pub (公钥)追加到授权的key里面去

命令为：
[yq@iZ94bxra9qaZ ~]$ cat ~/.ssh/id_rsa.pub >>~/.ssh/authorized_keys
或者
[yq@iZ94bxra9qaZ ~]$ cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys

另外 /etc/ssh/sshd_config 中下列三行去除注释

RSAAuthentication yes
PubkeyAuthentication yes
AuthorizedKeysFile .ssh/authorized_keys

然后执行service sshd restart
[root@localhost ssh]# service sshd restart

1.5 配置hosts

配置/etc/hosts
master为ye.hadoop
slave为ye.hadoop1
我这里两台机器
127.0.0.1 localhost localhost.localdomainlocalhost4 localhost4.localdomain4
::1 localhostlocalhost.localdomain localhost6 localhost6.localdomain6
192.168.60.128 ye.hadoop
192.168.60.129 ye.hadoop1

配置让master可以访问slave
把ye.hadoop的公钥拿到ye.hadoop1机器上
[yq@localhost .ssh]$ scp yq@ye.hadoop:~/.ssh/id_rsa.pub./master_rsa.pub
id_rsa.pub 100% 406 0.4KB/s 00:00
[yq@localhost .ssh]$ ls
authorized_keys id_rsa id_rsa.pub known_hosts master_rsa.pub
[yq@localhost .ssh]$ cat master_rsa.pub >>authorized_keys

1.6 测试是否能免密码登录

[yq@localhost .ssh]$ ssh ye.hadoop
Last login: Sat Jan 30 09:39:34 2016 from ye.hadoop1
[yq@ye ~]$

1.7 安装hadoop

[yq@ye ~]$ tar -xzvf hadoop-2.7.1.tar.gz

root 用户下：
[root@localhost yq]# rpm -ivh jdk-7u21-linux-x64.rpm
[root@localhost usr]# cd /etc/
[root@localhost usr]vi /etc/profile

export JAVA_HOME=/usr/java/jdk1.7.0_21
export JRE_HOME=$JAVA_HOME/jre
export CLASSPATH=$JAVA_HOME/lib
export PATH=$PATH:/usr/java/jdk1.7.0_21/bin
export HADOOP_HOME=/home/yq/hadoop-2.7.1
exportPATH=$PATH:$JAVA_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin
export HADOOP_PREFIX=/home/yq/hadoop-2.7.1
export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop
export HADOOP_HDFS_HOME=$HADOOP_HOME
export HADOOP_MAPRED_HOME=$HADOOP_HOME
export HADOOP_YARN_HOME=$HADOOP_HOME
export HADOOP_COMMON_HOME=$HADOOP_HOME
exportHADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
exportCLASSPATH=$CLASSPATH:$HADOOP_HOME/libexec/share/hadoop/common/hadoop-common-2.7.1.jar
exportCLASSPATH=$CLASSPATH:$HADOOP_HOME/libexec/share/hadoop/common/hadoop-nfs-2.7.1.jar

[root@localhost etc]# source /etc/profile

配置core-site.xml、hdfs-site.xml、yarn-site.xml、mapred-site.xml 文件，参考官方网址配置。

检测查看hadoop 环境变量
[yq@ye hadoop-2.7.1]$ bin/hadoop classpath
/home/yq/hadoop-2.7.1/libexec/etc/hadoop:/home/yq/hadoop-2.7.1/share/hadoop/common/lib/*:/home/yq/hadoop-2.7.1/share/hadoop/common/*:/home/yq/hadoop-2.7.1/share/hadoop/hdfs:/home/yq/hadoop-2.7.1/share/hadoop/hdfs/lib/*:/home/yq/hadoop-2.7.1/share/hadoop/hdfs/*:/home/yq/hadoop-2.7.1/share/hadoop/yarn/lib/*:/home/yq/hadoop-2.7.1/share/hadoop/yarn/*:/home/yq/hadoop-2.7.1/share/hadoop/mapreduce/lib/*:/home/yq/hadoop-2.7.1/share/hadoop/mapreduce/*
[yq@ye hadoop-2.7.1]$

向节点 ye.hadoop1复制配置好的hadoop

scp -r hadoop-2.7.1 yq@ye.hadoop1:/home/yq/hadoop-2.7.1

在ye.hadoop上格式化namenode
[yq@ye hadoop-2.7.1]$ bin/hdfs namenode -format

启动hadoop
[yq@ye hadoop-2.7.1]$ sbin/start-all.sh

访问地址 http://192.168.60.128:50070
停止hadoop
[yq@ye hadoop-2.7.1]$ sbin/stop-all.sh

1.8 遇到问题

非root用户可能会遇到权限问题

当你去认证登录时候还是要输入密码认证

当我去查看安全日志时有这样一行错误
cat /var/log/secure
Jan 30 08:01:34 localhost sshd[33404]:Authentication refused: bad ownership or modes for directory /home/yq
Jan 30 08:04:35 localhost sshd[33427]:Authentication refused: bad ownership or modes for directory /home/yq
Jan 30 08:04:35 localhost sshd[33427]:Authentication refused: bad ownership or modes for directory /home/yq

解决此问题方法
在用户yq下给/home/yq 赋700的权限
chmod 700 /home/yq
当你再去ssh -vvv localhost时可能会报~/.ssh/下某一个文件 644权限太高的错误解决此办法
chmod 600 ~/.ssh/*

2 Flume 1.6安装部署

安装了hadoop之后，安装flume非常简单。

2.1 下载安装包

去官网下载apache-flume-1.6.0-bin.tar.gz

解压

[yq@ye ~]$ tar zxf apache-flume-1.6.0-bin.tar.gz

2.2 修改flume-env.sh和flume-http.conf文件

[yq@ye ~]$ cd apache-flume-1.6.0-bin

[yq@ye apache-flume-1.6.0-bin]$ ls

bin CHANGELOG conf DEVNOTES docs lib LICENSE NOTICE README RELEASE-NOTES tools

[yq@ye apache-flume-1.6.0-bin]$ cd conf/

[yq@ye conf]$ pwd

/home/yq/apache-flume-1.6.0-bin/conf

[yq@ye conf]$ mv flume-env.sh.template flume-env.sh

[yq@ye conf]$ Vi flume-env.sh

修改标红处

# Licensed to the Apache Software Foundation (ASF) under one

# or more contributor license agreements. See the NOTICE file

# distributed with this work for additional information

# regarding copyright ownership. The ASF licenses this file

# to you under the Apache License, Version 2.0 (the

# "License"); you may not use this file except incompliance

# with the License. You mayobtain a copy of the License at

#

# http://www.apache.org/licenses/LICENSE-2.0
#

# Unless required by applicable law or agreed to in writing,software

# distributed under the License is distributed on an "ASIS" BASIS,

# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express orimplied.

# See the License for the specific language governing permissionsand

# limitations under the License.

# If this file is placed at FLUME_CONF_DIR/flume-env.sh, it will besourced

# during Flume startup.

# Enviroment variables can be set here.

export JAVA_HOME=/usr/java/jdk1.7.0_21

# Give Flume more memory and pre-allocate, enable remote monitoringvia JMX

export JAVA_OPTS="-Xms100m -Xmx2000m-Dcom.sun.management.jmxremote"

# Note that the Flume conf directory is always included in theclasspath.

#FLUME_CLASSPATH=""

[yq@ye conf]$ mv flume-conf.properties.template flume-http.conf

[yq@ye conf]$ vi flume-http.conf

# KIND, either express or implied. See the License for the

# specific language governing permissions and limitations

# under the License.

# The configuration file needs to define the sources,

# the channels and the sinks.

# Sources, channels and sinks are defined per agent,

# in this case called 'agent'

a1.sources = r1

a1.channels = c1

a1.sinks = s1

a1.sources.r1.type = http

a1.sources.r1.bind = 10.10.2.236

a1.sources.r1.port = 5140

a1.sources.r1.handler = org.apache.flume.source.http.JSONHandler

a1.sources.r1.channels = c1

a1.sources.r1.interceptors=i2

a1.sources.r1.interceptors.i2.type=timestamp

a1.channels.c1.type=memory

a1.channels.c1.capacity=10000

a1.channels.c1.transactionCapacity=1000

a1.channels.c1.keep-alive=30

a1.sinks.s1.type=hdfs

a1.sinks.s1.channel=c1

a1.sinks.s1.hdfs.path=hdfs://ye.hadoop:9000/flume/events/%y-%m-%d

a1.sinks.s1.hdfs.fileType=DataStream

a1.sinks.s1.hdfs.writeFormat=Text

a1.sinks.s1.hdfs.timeZone=Asia/Shanghai

a1.sinks.s1.hdfs.rollInterval=0

a1.sinks.s1.hdfs.rollSize=65011712

a1.sinks.s1.hdfs.rollCount=0

a1.sinks.s1.hdfs.idleTimeout=60

2.3 启动flume

[yq@ye apache-flume-1.6.0-bin]$ pwd

/home/yq/apache-flume-1.6.0-bin

[yq@ye apache-flume-1.6.0-bin]$ bin/flume-ng agent --conf conf --conf-fileconf/flume-http.conf --name a1-Dflume.monitoring.type=http-Dflume.monitoring.port=5240
-Dflume.root.logger=INFO,console

使用配置文件为conf/flume-http.conf

-Dflume.monitoring.type=http-Dflume.monitoring.port=5240 为计数器端口，提供给之后zabbix监控使用。

3 Zabbix 3.0安装部署

Zabbix 3.0最低需要PHP5.4版本

3.1 安装yum

3.1.1 卸载原有yum

[root@localhost ~]# rpm -aq | grep yum |xargs rpm -e --nodeps

3.1.2 所需包下载地址：

https://lug.ustc.edu.cn/wiki/_export/code/mirrors/help/centos?codeblock=2%20-O%20/etc/yum.repos.d/CentOS-Base.repo

http://centos.ustc.edu.cn/centos/6/os/x86_64/Packages/python-iniparse-0.3.1-2.1.el6.noarch.rpm

http://centos.ustc.edu.cn/centos/6/os/x86_64/Packages/yum-plugin-fastestmirror-1.1.30-30.el6.noarch.rpm

http://centos.ustc.edu.cn/centos/6/os/x86_64/Packages/yum-metadata-parser-1.1.2-16.el6.x86_64.rpm

http://thunder.mirrors.ustc.edu.cn/centos/6/os/x86_64/Packages/yum-3.2.29-69.el6.centos.noarch.rpm

3.1.3 安装rpm包

[root@localhost home]# rpm -ivhpython-iniparse-0.3.1-2.1.el6.noarch.rpm

Preparing...               ########################################### [100%]

         packagepython-iniparse-0.3.1-2.1.el6.noarch is already installed

[root@localhost home]# rpm -ivh yum-*

Preparing...               ########################################### [100%]

  1:yum-metadata-parser   ##################################          ( 80%)

###########################################[ 33%]

  2:yum-plugin-fastestmirro########################################### [67%]

  3:yum                   ########################################### [100%]

[root@localhost home]#

[root@localhost home]# rpm -Uvhhttp://mirror.webtatic.com/yum/el6/latest.rpm

Retrievinghttp://mirror.webtatic.com/yum/el6/latest.rpm

warning: /var/tmp/rpm-tmp.snnf1M: Header V4DSA/SHA1 Signature, key ID cf4c4ff9: NOKEY

Preparing...                ###########################################[100%]

  1:webtatic-release      ########################################### [100%]

[root@localhost home]# rpm -ivhhttp://mirrors.ustc.edu.cn/epel/6/x86_64/epel-release-6-8.noarch.rpm

Retrievinghttp://mirrors.ustc.edu.cn/epel/6/x86_64/epel-release-6-8.noarch.rpm

warning: /var/tmp/rpm-tmp.Xyd3s5: Header V3RSA/SHA256 Signature, key ID 0608b895: NOKEY

Preparing...               ########################################### [100%]

         packageepel-release-6-8.noarch is already installed

[root@localhost home]# rpm -ivhius-release-1.0-14.ius.el6.noarch.rpm

warning:ius-release-1.0-14.ius.el6.noarch.rpm: Header V4 DSA/SHA1 Signature, key ID9cd4953f: NOKEY

Preparing...               ########################################### [100%]

         packageius-release-1.0-14.ius.el6.noarch is already installed

3.1.4 修改CentOS-Base.repo

[root@localhost home]# cp CentOS-Base.repo/etc/yum.repos.d/CentOS-Base.repo

cp: overwrite`/etc/yum.repos.d/CentOS-Base.repo'? yes

[root@localhost home]#

[root@localhost home]#

[root@localhost home]# sed -i's/\$releasever/6/g' /etc/yum.repos.d/CentOS-Base.repo

[root@localhost home]#

[root@localhost home]#

[root@localhost home]#

[root@localhost home]#

[root@localhost home]# yum makecache

3.1.5 遇到问题

Error:Cannot retrieve metalink for repository: epel. Please verify its path and tryagain

解决办法

[root@localhost home]# su-c "sed -i 's|^#baseurl|baseurl| ; s|^mirrorlist|#mirrorlist|'/etc/yum.repos.d/*"

机器不能访问https造成

是因为DNS配置有问题

修改/etc/resolv.conf文件

3.2 安装lamp环境

[root@ye ~]#yuminstall list |grep php

[root@ye ~]# yum -y install php54w-mysql.x86_64 php54w-gd.x86_64 php54w-mbstring.x86_64 php54w-odbc.x86_64php54w-xml.x86_64 php54w-pear.noarch

[root@ye ~]# yum -y install php54w-bcmath.x86_64

3.3 安装zabbix所需组件

[root@ye ~]# yum install -y curl curl-devel net-snmpnet-snmp-devel perl-DBI

3.4 创建用户

[root@localhost Desktop]# useradd zabbix

3.5 文件夹赋权

[root@ye mysql]# chown zabbix /home/zabbix
[root@ye mysql]#
[root@ye mysql]# chmod -R 777 /home/zabbix

3.6 解压zabbix

zabbix 下载地址：

http://iweb.dl.sourceforge.net/project/zabbix/ZABBIX%20Latest%20Development/3.0.0alpha2/zabbix-3.0.0alpha2.tar.gz

[root@ye zabbix]# ls

zabbix-3.0.0alpha2 (1).tar.gz

[root@ye zabbix]# tar xfzabbix-3.0.0alpha2\ $1$.tar.gz

[root@localhost~]#mysql

mysql>create database zabbix;

mysql>grant all on zabbix.* tozabbix@localhost identified by '123456';

mysql>use zabbix;

mysql>source /home/zabbix/zabbix-3.0.0alpha2/database/mysql/schema.sql

mysql>source/home/zabbix/zabbix-3.0.0alpha2/database/mysql/images.sql

mysql> source/home/zabbix/zabbix-3.0.0alpha2/database/mysql/data.sql

mysql>exit

3.7 编译安装

[root@ye zabbix-3.0.0alpha2]# pwd
/home/zabbix/zabbix-3.0.0alpha2
[root@ye zabbix-3.0.0alpha2]# ./configure--enable-server --enable-agent --with-mysql --with-net-snmp --with-libcurl

参数说明
—prefix=/usr/local/zabbix为指定安装目录为/usr/local/zabbix
—enable-server为安装zabbix服务端程序
—enable-agent为安装agent程序
—with-mysql为使用mysql数据库
—enable-ipv6为启用ipv6支持
—with-net-snmp为启用snmp支持
—with-libcurl为启用curl
—with-libxml2编译xml模块，主要用于监控vm虚拟机

[root@ye zabbix-3.0.0alpha2]# make install

3.8 客户端安装--enable-agent

3.8.1 安装yum

参见3.1节

3.8.2 安装安装zabbix所需组件

[root@ye ~]#
安装客户端
./configure --prefix=/usr/local/zabbix--enable-agent --with-mysql --with-net-snmp --with-libcurl

make install

3.8.3 配置配置文件

修改/usr/local/zabbix/etc/zabbix_agentd.conf 以下几项配置

服务器IP，如果不同网段，需要配置网关

Server=10.10.2.236,10.10.2.1

主动监控，服务器IP

ServerActive=10.10.2.236

Agent主机IP或者主机名

Hostname=10.10.1.233

3.8.4 运行客户端

[root@localhost etc]#/usr/local/zabbix/sbin/zabbix_agentd -c/usr/local/zabbix/etc/zabbix_agentd.conf

3.8.5 遇到问题

1.      sudo: sorry,you must have a tty to run sudo

解决办法：

修改/etc/sudoers

#Defaults    requiretty

#Defaults   !visiblepw

Defaults   visiblepw

2.       执行sudo的时候需要输入密码

解决办法：

修改/etc/sudoers，添加一行。
zabbix ALL=(ALL) NOPASSWD: ALL

3.9 zabbix安装问题

3.9.1 安装完成后汉化

修改include/locales.inc.php文件，如下图

然后用户基本资料中修改即可。

图片中中文乱码

修改defines.inc.php文件
然后下载微软雅黑字体（复制Windows下的字体也可以），改名为msyh传入/home/www/zabbix/fonts目录下即可。

3.9.2 图表X轴时间问题

vim /etc/sysconfig/clock

ZONE="Asia/Shanghai"

cp/usr/share/zoneinfo/Asia/Shanghai /etc/localtime

然后用date 修改系统时间为当前时间即可

3.10 附录

3.10.1 配置flume监控

1.新建一个shell脚本

[root@ye ~]# vi /opt/monitor_flume.sh

curl http://localhost:5240/metrics 2>/dev/null|sed -e's/$[,]$\s*/\1\n/g' -e 's/[{}]/\n/g'-e 's/[",]//g'|grep $1|awk -F: '{print $2}'

2.在代理配置文件中加入

[root@ye ~]# vi /usr/local/zabbix/etc/zabbix_flume_jdk.conf

UserParameter=flume.monitor[*],/bin/bash /opt/monitor_flume.sh $1

3.运行代理

[root@ye ~]#/usr/local/zabbix/sbin/zabbix_agentd -c/usr/local/zabbix/etc/zabbix_flume_jdk.conf

3.      管理平台配置

a.创建主机

b. 创建监控项

c.创建图形

注意创建项目时候的键值如下图：

                                      键值为配置文件中UserParameter的键值

UserParameter键值详解可以参照：https://www.ttlsa.com/zabbix/zabbix-user-parameters/

3.10.2 配置java监控

1. 脚本为：

sudo -u yq /usr/java/jdk1.7.0_21/bin/jmap-histo $(pgrep java|head -1)|grep Total|awk '{print $3}'

2. 配置文件配置：

UserParameter=jvmused,sudo -u hadoop /usr/java/jdk1.7.0_21/bin/jmap -histo $(pgrepjava|head -1)|grep Total|awk '{print$3}'

其他配置参照3.10.1

遇到sudo命令问题解决参照3.8.6

内容来自用户分享和网络整理，不保证内容的准确性，如有侵权内容，可联系管理员处理

标签： hadoop flume zabbix 安装

相关文章推荐

新的分享

章节导航