Apache Kafka源码分析 - kafka controller
2015-11-05 15:14
676 查看
前面已经分析过kafka server的启动过程,以及server所能处理的所有的request,即KafkaApis
剩下的,其实关键就是controller,以及partition和replica的状态机
这里先看看controller在broker server的基础上,多做了哪些初始化和failover的工作
最关键的一句,
kafka.server.ZookeeperLeaderElector
做两件事,
1. 建立watcher,去listen “election path” in ZK, "/controller";
当controller发生变化是,可以做相应的处理
LeaderChangeListener
2. elect, 试图去创建EphemeralPath,从而成为Controller
用createEphemeralPathExpectConflictHandleZKBug,试图去创建EphemeralNode,
如果不成功说明已经被别人抢占
成功就说明我成为了controller,调用onBecomingLeader(),即onControllerFailover
KafkaController.onControllerFailover
这个callback就是controller初始化的关键,
这里最后再讨论一下createEphemeralPathExpectConflictHandleZKBug
这个函数看着比较诡异,handleZKBug,到底是zk的什么bug?
注释给出了,https://issues.apache.org/jira/browse/ZOOKEEPER-1740
这个问题在于“The current behavior of zookeeper for ephemeral nodes is that session expiration and ephemeral node deletion is not an atomic operation.”
即zk的session过期和ephemeral node删除并不是一个原子操作,所以带来的问题的过程如下,
1. client session超时,它会假设在zk,这个ephemeral node已经被删除,但是zk当前由于某种原因hang住了,比如very long fsync operations,所以其实这个ephemeral node并没有被删除
2. 但client是认为ephemeral node已经被删除,所以它会尝试重新创建这个ephemeral node,但得到的结果是NodeExists,因为这个node并没有被删除,那么client想既然有就算了
3. 这个时候,zk从very long fsync operations的hang中恢复,它会继续之前没有完成的操作,把ephemeral node删掉
4. 这样client就会发现ephemeral node不存在了,虽然session并没有超时
所以这个函数就是为了规避这个问题,
方法就是,当我发现NodeExists时,说明zk当前hang住了,这个时候我需要等待,并反复尝试,直到zk把这个node删除后,我再重新创建
这个方式有个问题就是,如果zk hang住,这里的逻辑是while true,这个函数也会一直hang住
剩下的,其实关键就是controller,以及partition和replica的状态机
这里先看看controller在broker server的基础上,多做了哪些初始化和failover的工作
最关键的一句,
private val controllerElector = new ZookeeperLeaderElector(controllerContext, ZkUtils.ControllerPath, onControllerFailover, onControllerResignation, config.brokerId)
kafka.server.ZookeeperLeaderElector
参数的含义比较明显,注意两个callback,是关键
class ZookeeperLeaderElector(controllerContext: ControllerContext, electionPath: String, onBecomingLeader: () => Unit, onResigningAsLeader: () => Unit, brokerId: Int)
做两件事,
1. 建立watcher,去listen “election path” in ZK, "/controller";
当controller发生变化是,可以做相应的处理
controllerContext.zkClient.subscribeDataChanges(electionPath, leaderChangeListener)
LeaderChangeListener
监听这个path的两个行为, handleDataChange,即controller发生了变化,这个比较简单,只需要把local的leaderid更新一下就好
leaderId = KafkaController.parseControllerId(data.toString)
handleDataDeleted,这种情况即,controller被删除了,一般是挂了
挂了就重新做elect,但是多个判断,如果本来我就是leader,那先调用onResigningAsLeader,即onControllerResignation,做些清理的工作,比如deregister watcher,关闭各种状态机
if(amILeader) onResigningAsLeader() elect
2. elect, 试图去创建EphemeralPath,从而成为Controller
用createEphemeralPathExpectConflictHandleZKBug,试图去创建EphemeralNode,
如果不成功说明已经被别人抢占
成功就说明我成为了controller,调用onBecomingLeader(),即onControllerFailover
KafkaController.onControllerFailover
这个callback就是controller初始化的关键,
/** * This callback is invoked by the zookeeper leader elector on electing the current broker as the new controller. * It does the following things on the become-controller state change - * 1. Register controller epoch changed listener * 2. Increments the controller epoch * 3. Initializes the controller's context object that holds cache objects for current topics, live brokers and * leaders for all existing partitions. * 4. Starts the controller's channel manager * 5. Starts the replica state machine * 6. Starts the partition state machine * If it encounters any unexpected exception/error while becoming controller, it resigns as the current controller. * This ensures another controller election will be triggered and there will always be an actively serving controller */ def onControllerFailover() { if(isRunning) { info("Broker %d starting become controller state transition".format(config.brokerId)) //开始日志 //read controller epoch from zk readControllerEpochFromZookeeper() // increment the controller epoch incrementControllerEpoch(zkClient) //增加ControllerEpoch,以区分过期 // before reading source of truth from zookeeper, register the listeners to get broker/topic callbacks registerReassignedPartitionsListener() registerPreferredReplicaElectionListener() partitionStateMachine.registerListeners() replicaStateMachine.registerListeners() initializeControllerContext() //从zk读出broker,topic的信息以初始化controllerContext replicaStateMachine.startup() //启动状态机 partitionStateMachine.startup() // register the partition change listeners for all existing topics on failover controllerContext.allTopics.foreach(topic => partitionStateMachine.registerPartitionChangeListener(topic)) //为每个topic增加partition变化的watcher info("Broker %d is ready to serve as the new controller with epoch %d".format(config.brokerId, epoch)) //完成日志 brokerState.newState(RunningAsController) maybeTriggerPartitionReassignment() maybeTriggerPreferredReplicaElection() /* send partition leadership info to all live brokers */ sendUpdateMetadataRequest(controllerContext.liveOrShuttingDownBrokerIds.toSeq) //如注释说,用于成为controller,需要把partition leadership信息发到所有brokers,大家要以我为准 if (config.autoLeaderRebalanceEnable) { //如果打开outoLeaderRebalance,需要把partiton leader由于dead而发生迁徙的,重新迁徙回去 info("starting the partition rebalance scheduler") autoRebalanceScheduler.startup() autoRebalanceScheduler.schedule("partition-rebalance-thread", checkAndTriggerPartitionRebalance, 5, config.leaderImbalanceCheckIntervalSeconds, TimeUnit.SECONDS) } deleteTopicManager.start() } else info("Controller has been shut down, aborting startup/failover") }
这里最后再讨论一下createEphemeralPathExpectConflictHandleZKBug
这个函数看着比较诡异,handleZKBug,到底是zk的什么bug?
注释给出了,https://issues.apache.org/jira/browse/ZOOKEEPER-1740
这个问题在于“The current behavior of zookeeper for ephemeral nodes is that session expiration and ephemeral node deletion is not an atomic operation.”
即zk的session过期和ephemeral node删除并不是一个原子操作,所以带来的问题的过程如下,
1. client session超时,它会假设在zk,这个ephemeral node已经被删除,但是zk当前由于某种原因hang住了,比如very long fsync operations,所以其实这个ephemeral node并没有被删除
2. 但client是认为ephemeral node已经被删除,所以它会尝试重新创建这个ephemeral node,但得到的结果是NodeExists,因为这个node并没有被删除,那么client想既然有就算了
3. 这个时候,zk从very long fsync operations的hang中恢复,它会继续之前没有完成的操作,把ephemeral node删掉
4. 这样client就会发现ephemeral node不存在了,虽然session并没有超时
所以这个函数就是为了规避这个问题,
方法就是,当我发现NodeExists时,说明zk当前hang住了,这个时候我需要等待,并反复尝试,直到zk把这个node删除后,我再重新创建
def createEphemeralPathExpectConflictHandleZKBug(zkClient: ZkClient, path: String, data: String, expectedCallerData: Any, checker: (String, Any) => Boolean, backoffTime: Int): Unit = { while (true) { try { createEphemeralPathExpectConflict(zkClient, path, data) return } catch { case e: ZkNodeExistsException => { // An ephemeral node may still exist even after its corresponding session has expired // due to a Zookeeper bug, in this case we need to retry writing until the previous node is deleted // and hence the write succeeds without ZkNodeExistsException ZkUtils.readDataMaybeNull(zkClient, path)._1 match { case Some(writtenData) => { if (checker(writtenData, expectedCallerData)) { info("I wrote this conflicted ephemeral node [%s] at %s a while back in a different session, ".format(data, path) + "hence I will backoff for this node to be deleted by Zookeeper and retry") Thread.sleep(backoffTime) } else { throw e } } case None => // the node disappeared; retry creating the ephemeral node immediately } } case e2: Throwable => throw e2 } } }
这个方式有个问题就是,如果zk hang住,这里的逻辑是while true,这个函数也会一直hang住
相关文章推荐
- 错误: 找不到或无法加载主类 org.apache.hadoop.hdfs.server.namenode.NameNode 问题解决
- Apache Kafka源码分析 - ReplicaStateMachine
- Apache Client使用说明第一章(第二部分)
- Linux中将Apache加入到系统服务
- WampServer 2.5局域网不能访问网页
- apache activemq的重连
- Apache配置虚拟目录和多主机头
- linux下安装apache
- apache2.4.17 编译安装
- apache 做集群
- Exception starting filter struts2 java.lang.ClassNotFoundException: org.apache.struts2.dispatcher.ng
- 如何从Apache官网下载windows版apache服务器
- 开源性能测试工具 - Apache ab 介绍
- 安装Apache 2.4.17 遇到的问题
- org.apache.jasper.JasperException: Unable to compile class for JSP异常
- Linux下Apache日志分析工具--AWStats安装使用
- java.lang.NoClassDefFoundError: org.apache.axis.cl
- Apache Commons工具集简介(转)
- Apache POI介绍
- centos中apache配置