zookeeper、kafka的安装与配置以及集群

一、zookeeper安装：

1、创建zookeeper用户：

useradd zookeeper

2、将安装包zookeeper-3.4.9解压到/home/zookeeper目录下

3、在/home/zookeeper/zookeeper-3.4.9/conf目录下

cp zoo_sample.cfg zoo.cfg

4、编辑 zoo.cfg

# The number of milliseconds of each tick

tickTime=2000 #作为 Zookeeper 服务器之间或客户端与服务器之间维持心跳的时间间隔，也就是每个 tickTime 时间就会发送一个心跳。

# The number of ticks that the initial

# synchronization phase can take

initLimit=10 #个配置项是用来配置 Zookeeper 接受客户端（这里所说的客户端不是用户连接 Zookeeper 服务器的客户端，而是 Zookeeper 服务器集群中连接到 Leader 的 Follower 服务器）初始化连接时最长能忍受多少个心跳时间间隔数。当已经超过 10 个心跳的时间（也就是 tickTime）长度后 Zookeeper 服务器还没有收到客户端的返回信息，那么表明这个客户端连接失败。总的时间长度就是 5*2000=10
秒

# The number of ticks that can pass between

# sending a request and getting an acknowledgement

syncLimit=5 #这个配置项标识 Leader 与 Follower 之间发送消息，请求和应答时间长度，最长不能超过多少个 tickTime 的时间长度，总的时间长度就是 5*2000=10秒

# the directory where the snapshot is stored.

# do not use /tmp for storage, /tmp here is just

# example sakes.

dataDir=/home/zookeeper/data #zookeeper的数据目录

dataLogDir=/home/zookeeper/datalog  #zookeeper的数据日志目录

# the port at which the clients will connect

clientPort=2181 #客户端连接端口

# the maximum number of client connections.

# increase this if you need to handle more clients

#maxClientCnxns=60 #单个客户端与单台服务器之间的连接数的限制，是ip级别的，默认是60，如果设置为0，那么表明不作任何限制。
#

# Be sure to read the maintenance section of the

# administrator guide before turning on autopurge.

#

# http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance
#

# The number of snapshots to retain in dataDir

#autopurge.snapRetainCount=3 #这个参数和下面的参数搭配使用，这个参数指定了需要保留的文件数目。默认是保留3个

# Purge task interval in hours

# Set to "0" to disable auto purge feature

#autopurge.purgeInterval=1 #3.4.0及之后版本，ZK提供了自动清理事务日志和快照文件的功能，这个参数指定了清理频率，单位是小时，需要配置一个1或更大的整数，默认是0，表示不开启自动清理功能。

server.1=10.20.13.99:2888:3888 #这里的1是一个数字，与myid文件中的id是一致的。右边可以配置两个端口，第一个端口用于F和L之间的数据同步和其它通信，第二个端口用于Leader选举过程中投票通信。配置集群的服务器。

server.2=10.20.13.153:2888:3888

server.3=10.20.13.231:2888:3888

5、在/home/zookeeper/data创建文件myid，并在里面写如上面文件中配置的server.x,x对应的编号

[zookeeper@IP data]$ cat myid

1

6、启动zookeeper

[zookeeper@IP bin]$ /home/zookeeper/zookeeper-3.4.9/bin/zkServer.sh start

ZooKeeper JMX enabled by default

Using config: /home/zookeeper/zookeeper-3.4.9/bin/../conf/zoo.cfg

Starting zookeeper ... STARTED

7、查看zookeeper进程

8、查看zookeeper状态：

[zookeeper@IP bin]$ /home/zookeeper/zookeeper-3.4.9/bin/zkServer.sh status

ZooKeeper JMX enabled by default

Using config: /home/zookeeper/zookeeper-3.4.9/bin/../conf/zoo.cfg

Mode: follower

8、停止zookeeper

[zookeeper@IP bin]$ /home/zookeeper/zookeeper-3.4.9/bin/zkServer.sh stop

ZooKeeper JMX enabled by default

Using config: /home/zookeeper/zookeeper-3.4.9/bin/../conf/zoo.cfg

Stopping zookeeper ... STOPPED 

9、登录zookeeper

[zookeeper@IP bin]$ /home/zookeeper/zookeeper-3.4.9/bin/zkCli.sh -server 127.0.0.1:2181

Connecting to 127.0.0.1:2181

2017-03-14 16:53:56,594 [myid:] - INFO  [main:Environment@100] - Client environment:zookeeper.version=3.4.9-1757313, built on 08/23/2016 06:50 GMT

2017-03-14 16:53:56,598 [myid:] - INFO  [main:Environment@100] - Client environment:host.name=IP

2017-03-14 16:53:56,598 [myid:] - INFO  [main:Environment@100] - Client environment:java.version=1.7.0_09-icedtea

2017-03-14 16:53:56,600 [myid:] - INFO  [main:Environment@100] - Client environment:java.vendor=Oracle Corporation

2017-03-14 16:53:56,600 [myid:] - INFO  [main:Environment@100] - Client environment:java.home=/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.9.x86_64/jre

2017-03-14 16:53:56,600 [myid:] - INFO  [main:Environment@100] - Client environment:java.class.path=/home/zookeeper/zookeeper-3.4.9/bin/../build/classes:/home/zookeeper/zookeeper-3.4.9/bin/../build/lib/*.jar:/home/zookeeper/zookeeper-3.4.9/bin/../lib/slf4j-log4j12-1.6.1.jar:/home/zookeeper/zookeeper-3.4.9/bin/../lib/slf4j-api-1.6.1.jar:/home/zookeeper/zookeeper-3.4.9/bin/../lib/netty-3.10.5.Final.jar:/home/zookeeper/zookeeper-3.4.9/bin/../lib/log4j-1.2.16.jar:/home/zookeeper/zookeeper-3.4.9/bin/../lib/jline-0.9.94.jar:/home/zookeeper/zookeeper-3.4.9/bin/../zookeeper-3.4.9.jar:/home/zookeeper/zookeeper-3.4.9/bin/../src/java/lib/*.jar:/home/zookeeper/zookeeper-3.4.9/bin/../conf:

2017-03-14 16:53:56,601 [myid:] - INFO  [main:Environment@100] - Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib

2017-03-14 16:53:56,601 [myid:] - INFO  [main:Environment@100] - Client environment:java.io.tmpdir=/tmp

2017-03-14 16:53:56,601 [myid:] - INFO  [main:Environment@100] - Client environment:java.compiler=<NA>

2017-03-14 16:53:56,601 [myid:] - INFO  [main:Environment@100] - Client environment:os.name=Linux

2017-03-14 16:53:56,601 [myid:] - INFO  [main:Environment@100] - Client environment:os.arch=amd64

2017-03-14 16:53:56,601 [myid:] - INFO  [main:Environment@100] - Client environment:os.version=2.6.32-358.el6.x86_64

2017-03-14 16:53:56,601 [myid:] - INFO  [main:Environment@100] - Client environment:user.name=zookeeper

2017-03-14 16:53:56,601 [myid:] - INFO  [main:Environment@100] - Client environment:user.home=/home/zookeeper

2017-03-14 16:53:56,602 [myid:] - INFO  [main:Environment@100] - Client environment:user.dir=/home/zookeeper/zookeeper-3.4.9/bin

2017-03-14 16:53:56,604 [myid:] - INFO  [main:ZooKeeper@438] - Initiating client connection, connectString=127.0.0.1:2181 sessionTimeout=30000

watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@77b0d0ed

Welcome to ZooKeeper!

2017-03-14 16:53:56,640 [myid:] - INFO  [main-SendThread(127.0.0.1:2181):ClientCnxn$SendThread@1032] - Opening socket connection to server 127.0.0.1/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)

JLine support is enabled

2017-03-14 16:53:56,647 [myid:] - INFO  [main-SendThread(127.0.0.1:2181):ClientCnxn$SendThread@876] - Socket connection established to 127.0.0.1/127.0.0.1:2181, initiating session

2017-03-14 16:53:56,665 [myid:] - INFO  [main-SendThread(127.0.0.1:2181):ClientCnxn$SendThread@1299] - Session establishment complete on server 127.0.0.1/127.0.0.1:2181, sessionid = 0x15acc0342d40001, negotiated timeout = 30000

WATCHER::

WatchedEvent state:SyncConnected type:None path:null

[zk: 127.0.0.1:2181(CONNECTED) 0] ls /

[isr_change_notification, zookeeper, admin, consumers, cluster, config, controller, zk, brokers, controller_epoch]

[zk: 127.0.0.1:2181(CONNECTED) 1]

二、kafka的安装

1、安装kafka需要jvm环境，如没有需要先安装jdk

2、创建kafka用户：useradd kafka

3、将kafka的安装包解压到/home/kafka目录下

4、修改配置文件/home/kafka/kafka_2.11-0.10.1.1/config目录下的server.properties文件：

# The id of the broker. This must be set to a unique integer for each broker.

broker.id=1 #kafka的编号，每台必须不重复

port=9092 #kafka的端口号

host.name=10.20.13.99 #主机名或ip

# A comma seperated list of directories under which to store log files

log.dirs=/tmp/kafka-logs #log的目录，根据实际情况指定

############################# Zookeeper #############################

# Zookeeper connection string (see zookeeper docs for details).

# This is a comma separated host:port pairs, each corresponding to a zk

# server. e.g. "127.0.0.1:3000,127.0.0.1:3001,127.0.0.1:3002".

# You can also append an optional chroot string to the urls to specify the

# root directory for all kafka znodes.

zookeeper.connect=10.20.13.99:2181,10.20.13.153:2181,10.20.13.231:2181 #指定集成的zookeeper

# Timeout in ms for connecting to zookeeper

zookeeper.connection.timeout.ms=6000 # 与zookeeper连接的超时时间设置

说明：其他很多项可以默认不修改。

5、启动kafka：

nohup bin/kafka-server-start.sh config/server.properties &

6、创建topic

/kafka-topics.sh --create --zookeeper 10.20.13.99:2181 --replication-factor 3 --partitions 2 --topic test

#zookeeper后面只需要写一台服务器即可。

#test 创建的管道名

#partitions 创建的分区数 不能超过最大的服务数

#replication-factor 创建的副本数

./kafka-topics.sh --zookeeper 10.20.13.99:2181 -topic "test1" --describe #查看创建的topic信息

7、生产着连接

./kafka-console-producer.sh --broker-list 10.20.13.99:9092 --topic test

8、消费着连接：

./kafka-console-consumer.sh --zookeeper 10.20.13.153:2181 --topic test --from-beginning