Kafka详细教程：下载、安装、配置与集群部署

1、下载

[root@node1 ~]# wget http://mirror.bit.edu.cn/apache/kafka/0.11.0.1/kafka_2.11-0.11.0.1.tgz --2017-10-28 09:16:07--  http://mirror.bit.edu.cn/apache/kafka/0.11.0.1/kafka_2.11-0.11.0.1.tgz Resolving mirror.bit.edu.cn (mirror.bit.edu.cn)... 202.204.80.77, 2001:da8:204:2001:250:56ff:fea1:22
Connecting to mirror.bit.edu.cn (mirror.bit.edu.cn)|202.204.80.77|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 42031343 (40M) [application/octet-stream]
Saving to: ‘kafka_2.11-0.11.0.1.tgz’

100%[==============================================================================================>] 42,031,343  1.11MB/s   in 33s

2017-10-28 09:16:40 (1.22 MB/s) - ‘kafka_2.11-0.11.0.1.tgz’ saved [42031343/42031343]

[root@node1 ~]#

2、安装

（1）解压缩

[root@node1 ~]# tar -zxvf kafka_2.11-0.11.0.1.tgz -C /opt

（2）进入根目录

[root@node1 ~]# cd /opt/kafka_2.11-0.11.0.1
[root@node1 kafka_2.11-0.11.0.1]# ls
bin  config  libs  LICENSE  NOTICE  site-docs
[root@node1 kafka_2.11-0.11.0.1]#

Kafka目录介绍

- /bin 操作kafka的可执行脚本，还包含windows下脚本

- /config 配置文件所在目录

- /libs 依赖库目录

- /logs 日志数据目录，目录kafka把server端日志分为5种类型，分为:server,request,state，log-cleaner，controller

3、配置

（1）统一修改配置文件

[root@node1 kafka_2.11-0.11.0.1]# vi config/server.properties

brokerID唯一标示，集群中不能有重复的ID

# The id of the broker. This must be set to a unique integer for each broker.
broker.id=1

打开监听

listeners=PLAINTEXT://:9092

修改日志目录

log.dirs=/var/log/kafka-logs

修改 zookeeper.connect

zookeeper.connect=node1:2181,node2:2181,node3:2181

4、分发

（1）分发软件包

[root@node1 ~]# scp -r /opt/kafka_2.11-0.11.0.1/ node2:/opt
[root@node1 ~]# scp -r /opt/kafka_2.11-0.11.0.1/ node3:/opt

（2）修改node2 的配置

[root@node2 ~]# vi /opt/kafka_2.11-0.11.0.1/config/server.properties

修改内容

broker.id=2

（3）修改node3 的配置

[root@node3 ~]# vi /opt/kafka_2.11-0.11.0.1/config/server.properties

修改内容

broker.id=3

5、启动kafka

（1）先将zookeeper集群启动

（2）再启动kafka

启动node1

[root@node1 kafka_2.11-0.11.0.1]# bin/kafka-server-start.sh config/server.properties &
[1] 2576
[root@node1 kafka_2.11-0.11.0.1]#

启动node2

[root@node2 kafka_2.11-0.11.0.1]# bin/kafka-server-start.sh config/server.properties &
[1] 2526
[root@node2 kafka_2.11-0.11.0.1]#

启动node3

[root@node3 kafka_2.11-0.11.0.1]# bin/kafka-server-start.sh config/server.properties &
[1] 7650
[root@node3 kafka_2.11-0.11.0.1]#

（3）查看进程

[root@node1 kafka_2.11-0.11.0.1]# jps
2576 Kafka
2551 QuorumPeerMain
2874 Jps
[root@node1 kafka_2.11-0.11.0.1]#

[root@node2 kafka_2.11-0.11.0.1]# jps
2499 QuorumPeerMain
2821 Jps
2526 Kafka
[root@node2 kafka_2.11-0.11.0.1]#

[root@node3 kafka_2.11-0.11.0.1]# jps
7617 QuorumPeerMain
7650 Kafka
7945 Jps
[root@node3 kafka_2.11-0.11.0.1]#