kafka.common.ConsumerRebalanceFailedException异常解决
2015-11-02 08:46
701 查看
kafka.common.ConsumerRebalanceFailedException: group_dd-1446432618163-2746a209 can't rebalance after 10 retries
at kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener.syncedRebalance(ZookeeperConsumerConnector.scala:432)
at kafka.consumer.ZookeeperConsumerConnector.kafka$consumer$ZookeeperConsumerConnector$$reinitializeConsumer(ZookeeperConsumerConnector.scala:722)
at kafka.consumer.ZookeeperConsumerConnector.consume(ZookeeperConsumerConnector.scala:212)
at kafka.javaapi.consumer.ZookeeperConsumerConnector.createMessageStreams(ZookeeperConsumerConnector.scala:80)
at kafka.javaapi.consumer.ZookeeperConsumerConnector.createMessageStreams(ZookeeperConsumerConnector.scala:92)
at com.symboltech.mine.ConsumerUtil.start(ConsumerUtil.java:40)
at com.symboltech.mine.KafkaConsumer.sdf(KafkaConsumer.java:58)
解决办法:
1.配置zk问题(kafka的consumer配置)
zookeeper.session.timeout.ms=5000
zookeeper.connection.timeout.ms=10000
rebalance.backoff.ms=2000
rebalance.max.retries=10
官方解释:
consumer rebalancing fails (you will see ConsumerRebalanceFailedException): This is due to conflicts when two consumers are trying to own the same topic partition. The log will show you what caused the conflict (search for "conflict in ").
If your consumer subscribes to many topics and your ZK server is busy, this could be caused by consumers not having enough time to see a consistent view of all consumers in the same group. If this is the case, try Increasing rebalance.max.retries and rebalance.backoff.ms.
Another reason could be that one of the consumers is hard killed. Other consumers during rebalancing won't realize that consumer is gone after zookeeper.session.timeout.ms time. In the case, make sure that rebalance.max.retries * rebalance.backoff.ms > zookeeper.session.timeout.ms.
at kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener.syncedRebalance(ZookeeperConsumerConnector.scala:432)
at kafka.consumer.ZookeeperConsumerConnector.kafka$consumer$ZookeeperConsumerConnector$$reinitializeConsumer(ZookeeperConsumerConnector.scala:722)
at kafka.consumer.ZookeeperConsumerConnector.consume(ZookeeperConsumerConnector.scala:212)
at kafka.javaapi.consumer.ZookeeperConsumerConnector.createMessageStreams(ZookeeperConsumerConnector.scala:80)
at kafka.javaapi.consumer.ZookeeperConsumerConnector.createMessageStreams(ZookeeperConsumerConnector.scala:92)
at com.symboltech.mine.ConsumerUtil.start(ConsumerUtil.java:40)
at com.symboltech.mine.KafkaConsumer.sdf(KafkaConsumer.java:58)
解决办法:
1.配置zk问题(kafka的consumer配置)
zookeeper.session.timeout.ms=5000
zookeeper.connection.timeout.ms=10000
rebalance.backoff.ms=2000
rebalance.max.retries=10
官方解释:
consumer rebalancing fails (you will see ConsumerRebalanceFailedException): This is due to conflicts when two consumers are trying to own the same topic partition. The log will show you what caused the conflict (search for "conflict in ").
If your consumer subscribes to many topics and your ZK server is busy, this could be caused by consumers not having enough time to see a consistent view of all consumers in the same group. If this is the case, try Increasing rebalance.max.retries and rebalance.backoff.ms.
Another reason could be that one of the consumers is hard killed. Other consumers during rebalancing won't realize that consumer is gone after zookeeper.session.timeout.ms time. In the case, make sure that rebalance.max.retries * rebalance.backoff.ms > zookeeper.session.timeout.ms.
相关文章推荐
- [Rails Level 2] Ground up
- Daily Scrum (2015/11/1)
- 如何通过AIDL实现跨应用绑定Service
- Daily Scrumming* 2015.11.1(Day 13)
- Daily Scrumming* 2015.10.31(Day 12)
- 【Alpha】Daily Scrum Meeting第二次
- 【Alpha】Daily Scrum Meeting第二次
- log4j:ERROR Failed to load driver问题
- tail 命令
- 软RAID管理命令mdadm详解
- RAID学习笔记
- DataInputStream和DataOutputStream的简单使用
- Daily Scrum 11.1
- 【Alpha】Daily Scrum Meeting第一次
- 【Alpha】Daily Scrum Meeting第一次
- Daily Scrum 11.1
- VirtualBox Failed to open a session on Ubuntu-14.04
- jetbrain
- (转)Hadoop MapReduce链式实践--ChainReducer
- hdu 1039 Easier Done Than Said?