note of kafka learning (first part, before replication)
2014-11-29 07:11
597 查看
sequential
disk access can in some cases be faster than random memory access!
The memory overhead of objects is very high, often doubling the size of the data stored (or worse).
Java garbage collection becomes increasingly fiddly and slow as the in-heap data increases.
All data is immediately written to a persistent log on the filesystem without necessarily flushing to disk. In effect this just means
that it is transferred into the kernel's pagecache.
kafka use poll instead of push
To avoid this we have parameters in our pull request that allow the consumer request
to block in a "long poll" waiting until data arrives
(Kafka's persistent storage makes me feel like the same as Git ... :-))
So effectively Kafka guarantees at-least-once delivery by default and allows the
user to implement at most once delivery by disabling retries on the producer and committing its offset prior to processing a batch of messages. Exactly-once delivery requires co-operation with the destination storage system but Kafka provides the offset which
makes implementing this straight-forward.
disk access can in some cases be faster than random memory access!
The memory overhead of objects is very high, often doubling the size of the data stored (or worse).
Java garbage collection becomes increasingly fiddly and slow as the in-heap data increases.
All data is immediately written to a persistent log on the filesystem without necessarily flushing to disk. In effect this just means
that it is transferred into the kernel's pagecache.
kafka use poll instead of push
To avoid this we have parameters in our pull request that allow the consumer request
to block in a "long poll" waiting until data arrives
(Kafka's persistent storage makes me feel like the same as Git ... :-))
So effectively Kafka guarantees at-least-once delivery by default and allows the
user to implement at most once delivery by disabling retries on the producer and committing its offset prior to processing a batch of messages. Exactly-once delivery requires co-operation with the destination storage system but Kafka provides the offset which
makes implementing this straight-forward.
相关文章推荐
- English Learning Daily Note of First
- English Learning Daily Note of Second
- The NOTE of learning ASP.NET [1]
- Spring v3.0.2 Learning Note 3 - Initialisation of Beans
- This is the first part of a two p
- The NOTE of learning ASP.NET [2] ASP.NET运行库处理并返回响应的整个流程
- Spring v3.0.2 Learning Note 4 - Scope of Beans
- Linear Regression_Note of Machine Learning_Andrew NG
- 每日一命令(12)head (output the first part of files)
- iOS-Core Bluetooth(API Reference) of first part
- The NOTE of learning ASP.NET [3] [关于ASP.NET配置文件 web.config]
- Spring v3.0.2 Learning Note 5 - Lifecycle of Beans
- The first day of learning Python(第一天学习Python)
- The NOTE of learning ASP.NET [4]
- The NOTE of learning ASP.NET [15] 关于.NET字符串类的使用
- The first day of learning java
- Linear Regression_Note of Machine Learning_Andrew NG
- The NOTE of learning ASP.NET [5] 关于.NET Framework及常用缩略词
- The NOTE of learning ASP.NET [10] 关于.NET整型与浮点型变量使用上的效率和存储问题
- The NOTE of learning ASP.NET [16] 关于类和对象以及封装