基于Hadoo的日志收集框架---Chukwa的安装部署

chukwa是解决在集群环境中收集各节点增量日志的一种基于hadoop的实现方案，其主要有如下四个组成部分。

1. Agents 运行在每个客户端上，负责发送数据。

2. Collectors 接收Agents发送的数据并写入稳定存储。

3. MapReduce jobs 分析和归档数据。

4. HICC 数据中心，用于显示数据的web界面。

它的系统架构如下图



以单机部署为例，前提已经安装hadoop(0.20.2)，HADOOP_HOME=/usr/local/cloud/hadoop

1. 下载chukwa(0.4.0)

2. 解压到/usr/local/cloud/chukwa/，CHUKWA_HOME=/usr/local/cloud/chukwa

3. 修改$CHUKWA_HOME/conf/agents，部署在本机的话为localhost

4. 修改$CHUKWA_HOME/conf/collectors， 部署在本机的话为localhost

5. 修改$CHUKWA_HOME/conf/chukwa-env.sh，设置如下项

Shell代码


export JAVA_HOME=/usr/java/jdk1.6.0_26
export HADOOP_HOME="/usr/local/cloud/hadoop"
export HADOOP_CONF_DIR="/usr/local/cloud/hadoop/conf"
export HADOOP_JAR=${HADOOP_HOME}/hadoop-core-0.20.2.jar
export chukwaRecordsRepository="/chukwa/repos/"
export CHUKWA_PID_DIR=/usr/local/cloud/tmp/chukwa/pids
export CHUKWA_LOG_DIR=/usr/local/cloud/tmp/chukwa/logs
export CHUKWA_DATA_DIR=/usr/local/cloud/tmp/chukwa/data

6. 修改$CHUKWA_HOME/conf/chukwa-agent-conf.xml

Xml代码


<property>
<name>chukwaAgent.tags</name>
<value>cluster="chukwa"</value>
<description>The cluster's name for this agent</description>
</property>

<property>
<name>chukwaAgent.control.port</name>
<value>9093</value>
<description>The socket port number the agent's control interface can be contacted at.</description>
</property>

<property>
<name>chukwaAgent.hostname</name>
<value>localhost</value>
<description>The hostname of the agent on this node. Usually localhost, this is used by the chukwa instrumentation agent-control interface library</description>
</property>

<property>
<name>chukwaAgent.checkpoint.name</name>
<value>chukwa_agent_checkpoint</value>
<description>the prefix to to prepend to the agent's checkpoint file(s)</description>
</property>

<property>
<name>chukwaAgent.checkpoint.dir</name>
<value>${CHUKWA_LOG_DIR}/</value>
<description>the location to put the agent's checkpoint file(s)</description>
</property>

<property>
<name>chukwaAgent.checkpoint.interval</name>
<value>5000</value>
<description>the frequency interval for the agent to do checkpoints, in milliseconds</description>
</property>

<property>
<name>chukwaAgent.sender.fastRetries</name>
<value>4</value>
<description>the number of post attempts to make to a single collector, before marking it failed</description>
</property>

<property>
<name>chukwaAgent.collector.retries</name>
<value>144000</value>
<description>the number of attempts to find a working collector</description>
</property>

<property>
<name>chukwaAgent.collector.retryInterval</name>
<value>20000</value>
<description>the number of milliseconds to wait between searches for a collector</description>
</property>

7. 修改$CHUKWA_HOME/conf/chukwa-collector-conf.xml

Xml代码


<property>
<name>writer.hdfs.filesystem</name>
<value>hdfs://master:9000/</value>
<description>HDFS to dump to</description>
</property>

<property>
<name>chukwaCollector.outputDir</name>
<value>/chukwa/logs/</value>
<description>Chukwa data sink directory</description>
</property>

<property>
<name>chukwaCollector.rotateInterval</name>
<value>300000</value>
<description>Chukwa rotate interval (ms)</description>
</property>

<property>
<name>chukwaCollector.http.port</name>
<value>8080</value>
<description>The HTTP port number the collector will listen on</description>
</property>

8. 修改$CHUKWA_HOME/conf/initial_adaptors，添加要监测的日志文件，比如

Shell代码


add filetailer.CharFileTailingAdaptorUTF8 TestLog1 0 /home/matrix/Program/project/log/testlog1 0

9. 启动collectors

Shell代码


$CHUKWA_HOME/bin/start-collectors.sh

10. 启动agents

Shell代码


$CHUKWA_HOME/bin/start-agents.sh

11. 启动data-processors

Shell代码


$CHUKWA_HOME/bin/start-data-processors.sh

转至:http://savagegarden.iteye.com/blog/1426853