第117课: Spark Streaming性能优化:如何最大程度的确保Spark Cluster和Kafka链接的稳
2016-06-04 21:03
686 查看
第117课: Spark Streaming性能优化:如何最大程度的确保Spark Cluster和Kafka链接的稳定性
1 Spark Streaming与Kafka连接问题
2 KafkaReceiver
spark2.x弄了一个静态类型检查,python就不能用
ZkUtils.scala
zookeeper.session.timeout.ms 系统默认6000 6s,这个超时zk就资源重新分配,所有数据都不能接收。考虑GC等因素,zookeeper.seesion.timeout.ms这个值在生产环境中设置为30s。30000 ACK确认
ZkUtils
class ZKConfig(props: VerifiableProperties) {
/** ZK host string */
val zkConnect = props.getString("zookeeper.connect")
/** zookeeper session timeout */
val zkSessionTimeoutMs = props.getInt("zookeeper.session.timeout.ms", 6000)
/** the max time that the client waits to establish a connection to zookeeper */
val zkConnectionTimeoutMs = props.getInt("zookeeper.connection.timeout.ms",zkSessionTimeoutMs)
/** how far a ZK follower can be behind a ZK leader */
val zkSyncTimeMs = props.getInt("zookeeper.sync.time.ms", 2000)}
ZookeeperConsumerConnector.scala
1 Spark Streaming与Kafka连接问题
2 KafkaReceiver
spark2.x弄了一个静态类型检查,python就不能用
ZkUtils.scala
zookeeper.session.timeout.ms 系统默认6000 6s,这个超时zk就资源重新分配,所有数据都不能接收。考虑GC等因素,zookeeper.seesion.timeout.ms这个值在生产环境中设置为30s。30000 ACK确认
ZkUtils
class ZKConfig(props: VerifiableProperties) {
/** ZK host string */
val zkConnect = props.getString("zookeeper.connect")
/** zookeeper session timeout */
val zkSessionTimeoutMs = props.getInt("zookeeper.session.timeout.ms", 6000)
/** the max time that the client waits to establish a connection to zookeeper */
val zkConnectionTimeoutMs = props.getInt("zookeeper.connection.timeout.ms",zkSessionTimeoutMs)
/** how far a ZK follower can be behind a ZK leader */
val zkSyncTimeMs = props.getInt("zookeeper.sync.time.ms", 2000)}
ZookeeperConsumerConnector.scala
相关文章推荐
- ElasticSearch(七)--请求体查询
- [platform]linux platform device/driver(三)--Platform Device和Platform_driver注册过程之代码对比
- ViewPager与FragmentPagerAdapter实现Tab
- java如何提取url里的域名
- c++ 复制控制和智能指针实现
- C#利用Timer控件实现图片运动
- “代理XP”组件已作为此服务器安全配置的一部分被关闭的解决办法
- 个人最终总结
- poj 2777
- A Template Engine in 500 lines or less 学习笔记
- STORM_0010_Message passing implementation/消息传递的实现
- zabbix简易图表详解
- nested function in python
- 解决ArcGIS Server “工作站服务没有启动”
- 使用Eclipse创建Maven项目
- JAVA-WEB开发第二讲[2016-06-04]JDK的下载
- 深入理解Linux内核 chap 3 进程
- LightOJ 1100 - Again Array Queries
- Guava base -- Objects
- Android新建项目选择SDK的区别(MiniMum Required SDK; Target SDK; Compile With; Theme )