mikoomi -Zabbix监控Apache Hadoop插件/模板 （翻译）

总揽

这个Hadoop插件可以用来监控Hadoop集群的NameNode和JobTracker。Hadoop is the leading and defacto distributed big data processing system "out there"。然而被像雅虎（据说拥有非常庞大的Hadoop集群），Facebook，Groupon等公司所使用的似乎只有Ganglia和openTSDB两种监控解决方案。当您阅读文档，你会发现这两个监测解决方案是非常紧密的结合Hadoop并且对Hadoop的版本、库等信息十分敏感。
这个Hadoop插件的主旨是在已经运行的Hadoop集群或者Zabbix服务中不需要安装任何软件并且能直接使用。这太好了是真的吗？你为什么不继续把下面的内容读完......

安装和配置

这个Hadoop插件用于从Hadoop的NameNode和JobTracker的Web UI接口截取信息。没有必要添加或者修改任何Hadoop的配置参数或重启你的Hadoop集群。下载这个插件之后你仅需要花费不超过5分钟的时间就能运行这个插件。
这个插件会调用一个叫做curl的命令工具，所以需要先安装这个命令工具。在Zabbix这边，你可以登录root用户（默认密码为zabbix）运行yast -i curl命令。注意虽然curl的包是非常小的，但是yast将花费几分钟的时间重新更新包仓库。接下来下载Hadoop插件它包括2个shell脚本和2个模板xml文件，下载路径是：http://mikoomi.googlecode.com/svn/plugins/。在Zabbix服务器上创建目录/etc/zabbix/externalscripts，并将shell脚本复制到这个目录里面。
完成上述操作后打开浏览器，下载NameNode和JobTracker的模板文件，下载路径是:http://mikoomi.googlecode.com/svn/plugins/。打开一个新的浏览窗口或者标签，登录Zabbix的前端（默认的用户名是admin，密码是zabbix）。
操作如下：
Configuration >> Templates
点击窗口右上角的“Import Template”按钮
在“Import file”对话框内, 找到并选中刚才下载的模板文件。
上传模板
现在你可以开始监控你的Hadoop集群了。使用说明如下：

监控你的Hadoop集群 按照下面的步骤进行：
监控NameNode
登录Zabbix的前端然后点击导航栏上的Configuration >> Hosts
点击右上角的“Create Host”按钮
按照提示填写监控选项 - Name：你选择的名字（在Zabbix中每一个监控实体被称为一个主机 - 但是它可能是一个主机、一个服务、一个程序乃至一个集群）。
完成后单击“templates”选项卡里面的“Add”按钮。
你将看到一个模板列表 - 选择“Template_Hadoop_NameNode”
在“Macros”选项卡里面添加如下宏-
{$HADOOP_NAMENODE_HOST}
{$HADOOP_NAMENODE_METRICS_PORT}
{$ZABBIX_NAME}
{$HADOOP_NAMENODE_HOST}的值应该是NameNode节点服务器的主机名或者完全主机名（可以在网络上ping通）。{$HADOOP_NAMENODE_METRICS_PORT}的值是NameNode的Web UI管理界面的端口。最后{$ZABBIX_NAME}是前面在Zabbix前端定义的NameNode的实体名称。
同样的，安装监控JobTracker的步骤如下 -
监控JobTracker
登录Zabbix的前端然后点击导航栏上的Configuration >> Hosts
点击右上角的“Create Host”按钮
按照提示填写监控选项 - Name：你选择的名字（在Zabbix中每一个监控实体被称为一个主机 - 但是它可能是一个主机、一个服务、一个程序乃至一个集群）。
完成后单击“templates”选项卡里面的“Add”按钮。
你将看到一个模板列表 - 选择“Template_Hadoop_JobTracker”
在“Macros”选项卡里面添加如下宏-
{$HADOOP_JOBTRACKER_HOST}
{$HADOOP_JOBTRACKER_METRICS_PORT}
{$ZABBIX_NAME}
{$HADOOP_NAMENODE_HOST}的值应该是NameNode节点服务器的主机名或者完全主机名（可以在网络上ping通）。{$HADOOP_NAMENODE_METRICS_PORT}的值是NameNode的Web UI管理界面的端口。最后{$ZABBIX_NAME}是前面在Zabbix前端定义的NameNode的实体名称。

NameNode监控指标Configured Cluster Storage
Configured Max. Heap Size (GB)
Hadoop Version
NameNode Process Heap Size (GB)
NameNode Start Time
Number of Dead Nodes
Number of Decommissioned Nodes
Number of Files and Directories in HDFS
Number of HDFS Blocks Used
Number of Live Nodes
Number of Under-Replicated Blocks
Ping Check
Storage Unit
Total % of Storage Available
Total % of Storage Used
Total Storage Available
Total Storage Used by DFS
Total Storage Used by non-DFS
Least (min) Node-level non-DFS Storage Used
Least (min) Node-level Storage Configured
Least (min) Node-level Storage Free
Least (min) Node-level Storage Free %
Least (min) Node-level Storage Used
Least (min) Node-level Storage Used %
Most (max) Node-level non-DFS Storage Used
Most (max) Node-level Storage Configured
Most (max) Node-level Storage Free
Most (max) Node-level Storage Free %
Most (max) Node-level Storage Used
Most (max) Node-level Storage Used %
Node-level Storage Unit of Measure
Node with Least (min) Node-level non-DFS Storage Used
Node with Least (min) Node-level Storage Configured
Node with Least (min) Node-level Storage Free
Node with Least (min) Node-level Storage Free %
Node with Least (min) Node-level Storage Used
Node with Least (min) Node-level Storage Used %
Node with Most (max) Node-level non-DFS Storage Used
Node with Most (max) Node-level Storage Configured
Node with Most (max) Node-level Storage Free
Node with Most (max) Node-level Storage Free %
Node with Most (max) Node-level Storage Used
Node with Most (max) Node-level Storage Used %

JobTracker监控指标Average Task Capacity Per Node
Hadoop Version
JobTracker Start Time
JobTracker State
Map Task Capacity
Number of Blacklisted Nodes
Number of Excluded Nodes
Number of Jobs Completed
Number of Jobs Failed
Number of Jobs Retired
Number of Jobs Running
Number of Jobs Submitted
Number of Map Tasks Running
Number of Nodes in Hadoop Cluster
Number of Reduce Tasks Running
Occupied Map Slots
Occupied Reduce Slots
Reduce Task Capacity
Reserved Map Slots
Reserved Reduce Slots
Pre-canned NameNode Triggers
Less than 20% free space available on the cluster
NameNode was restarted
No monitoring data received for the last 10 minutes
One or more nodes have become alive or restarted
One or more nodes have become dead
One or more nodes have been added to the decommissioned list
One or more nodes have been removed from the decommissioned list
The number of live nodes has been reduced
The number of live nodes has increased
There has been a reduction in the number of under-replicated blocks
There has been an increase in the number of under-replicated blocks
Less than 20% free space available on one or more nodes in the cluster
Pre-canned JobTracker Triggers
No monitoring data received for the last 10 minutes
One or more jobs have failed
One or more nodes have become blacklisted
One or more nodes have been added to the exclude list
One or more nodes have been added to the Hadoop cluster
One or more nodes have been removed from the blacklisted nodes
One or more nodes have been removed from the exclude list
One or more nodes have been removed from the Hadoop cluster
The JobTracker was restarted