安装配置kafka集群

以下为生产环境安装文档
1. 下载kafka并安装

wget https://archive.apache.org/dist/kafka/0.8.1/kafka_2.9.2-0.8.1.tgz
tar zxvf kafka_2.9.2-0.8.1.tgz

mv kafka_2.9.2-0.8.1 /usr/local/kafka

2. 修改配置文件

cd /usr/local/kafka

vim config/server.properties

broker.id=1            #每个节点用不同的值 如1/2/3

host.name=172.16.100.21         #绑定kafka本机IP地址

log.dirs=/data/kafka

zookeeper.connect=172.16.100.30:2181,172.16.100.31:2181,172.16.100.32:2181/kafka #配置zookeeper的地址，后面的kafka字符串是和zk里面创建的节点是一致的

3. 复制kafka到另外节点

scp -r /usr/local/kafka 172.16.100.22:/usr/local

mkdir -p /data/kafka

4. 配置环境变量

echo "export KAFKA_HOME=/usr/local/kafka" >> /etc/profile.d/app.sh

echo "export PATH=\$KAFKA_HOME/bin:\$PATH" >> /etc/profile.d/app.sh

source /etc/profile

5. 启动和停止

启动kafka

/usr/local/kafka/bin/kafka-server-start.sh /usr/local/kafka/config/server.properties > /dev/null &

停止kafka

/usr/local/kafka/bin/kafka-server-stop.sh

添加自启动（注意这里不要最后的&）

echo "source /etc/profile && /usr/local/kafka/bin/kafka-server-start.sh /usr/local/kafka/config/server.properties > /dev/null" >> /etc/rc.local

在zookeeper中查看

zkCli.sh

get /kafka/brokers/ids/1

get /kafka/brokers/ids/2

创建Topic

创建Topic(replication-factor是副本数量)

/usr/local/kafka/bin/kafka-topics.sh --create --zookeeper 172.16.100.30:2181/kafka --replication-factor 3 --partitions 2 --topic mykafka

查看Topic

/usr/local/kafka/bin/kafka-topics.sh --list --zookeeper 172.16.100.30:2181/kafka

查看Topic详细

/usr/local/kafka/bin/kafka-topics.sh --describe --zookeeper 172.16.100.30:2181/kafka

控制台测试(分别在两台kafka上面测试一下)

生产者

/usr/local/kafka/bin/kafka-console-producer.sh --broker-list 172.16.100.22:9092 --topic mykafka

消费者

/usr/local/kafka/bin/kafka-console-consumer.sh --zookeeper 172.16.100.30/kafka --topic mykafka --from-beginning

6. 安装kafka monitor

mkdir /usr/local/kafka-offset-console

cd /usr/local/kafka-offset-console

mkdir logs
https://github.com/quantifind/KafkaOffsetMonitor/releases/tag/v0.2.0
wget https://github.com/quantifind/KafkaOffsetMonitor/releases/download/v0.2.0/KafkaOffsetMonitor-assembly-0.2.0.jar
vim monitor-start.sh

添加

#!/bin/bash

export JAVA_HOME=/usr/java/jdk1.7.0_79
export MONITOR_HOME=/usr/local/kafka-offset-console
$JAVA_HOME/bin/java -Xms512M -Xmx512M -Xss1024K -XX:PermSize=256m -XX:MaxPermSize=512m -cp $MONITOR_HOME/KafkaOffsetMonitor-assembly-0.2.0.jar \
com.quantifind.kafka.offsetapp.OffsetGetterWeb \
--zk 172.16.100.30:2181,172.16.100.31:2181,172.16.100.32:2181/kafka\
--port 8086\
--refresh 10.seconds \
--retain 7.days 1>$MONITOR_HOME/logs/stdout.log 2>$MONITOR_HOME/logs/stderr.log &

chmod 755 monitor-start.sh

启动monitor

/usr/local/kafka-offset-console/monitor-start.sh

添加自启动
echo "/usr/local/kafka-offset-console/monitor-start.sh" >> /etc/rc.local

以下的不再使用，仅供参考

1. 安装zookeeper
http://blog.csdn.net/u013619834/article/details/41316957
连接zookeeper并添加节点

zkCli.sh -server 192.168.1.111:2181

create /kafka ""

2. 下载kafka并安装（注意如果kafka和zookeeper不在同一服务器，也要先安装jdk，我这里因为测试少用机器，所以是在一个上面安装的zk和kafka）

cd /usr/local/src

wget http://mirrors.cnnic.cn/apache/kafka/0.8.1.1/kafka_2.10-0.8.1.1.tgz
tar zxvf kafka_2.10-0.8.1.1.tgz

mv kafka_2.10-0.8.1.1 /usr/local/kafka

下载slf4j并复制slf4j-nop-1.7.2.jar

cd /usr/local/src

wget http://www.slf4j.org/dist/slf4j-1.7.2.tar.gz
tar zxvf slf4j-1.7.2.tar.gz

cd slf4j-1.7.2

cp slf4j-nop-1.7.2.jar /usr/local/kafka/libs

3. 修改配置文件

cd /usr/local/kafka

vim config/server.properties

broker.id=1            #每个节点用不同的值 如1/2/3

host.name=192.168.1.111         #绑定kafka本机IP地址

log.dirs=/data/kafka/data

zookeeper.connect=192.168.1.111:2181,192.168.1.112:2181,192.168.1.113:2181/kafka #配置zookeeper的地址，后面的kafka字符串是和zk里面创建的节点是一致的

复制到其他的节点并修改broker.id和host.name

scp -r kafka 192.168.1.112:/usr/local

scp -r kafka 192.168.1.113:/usr/local

4. 启动broker

创建目录

mkdir -p /data/kafka/data

启动kafka broker

/usr/local/kafka/bin/kafka-server-start.sh /usr/local/kafka/config/server.properties &

停止kafka broker

/usr/local/kafka/bin/kafka-server-stop.sh

添加broker自启动

echo "/usr/local/kafka/bin/kafka-server-start.sh /usr/local/kafka/config/server.properties &" >> /etc/rc.local

5. 创建Topic

创建Topic(replication-factor是副本数量)

/usr/local/kafka/bin/kafka-topics.sh --create --zookeeper 192.168.1.111:2181/kafka --replication-factor 3 --partitions 2 --topic mykafka

查看Topic

/usr/local/kafka/bin/kafka-topics.sh --list --zookeeper 192.168.1.111:2181/kafka

查看Topic详细

/usr/local/kafka/bin/kafka-topics.sh --describe --zookeeper 192.168.1.111:2181/kafka

6. 控制台测试(分别在两台kafka上面测试一下)

生产者

/usr/local/kafka/bin/kafka-console-producer.sh --broker-list 192.168.1.111:9092 --topic mykafka

消费者

/usr/local/kafka/bin/kafka-console-consumer.sh --zookeeper 192.168.1.111:2181/kafka --topic mykafka --from-beginning