Pod memory usage in k8s

Cadvisor内存使用率指标

Cadvisor中有关pod内存使用率的指标

其中

container_memory_max_usage_bytes > container_memory_usage_bytes >= container_memory_working_set_bytes > container_memory_rss

Cadvisor中相关定义

type MemoryStats struct { // Current memory usage, this includes all memory regardless of when it was // accessed. // Units: Bytes. Usage uint64 json:"usage"

// Maximum memory usage recorded.
// Units: Bytes.
MaxUsage uint64 `json:"max_usage"`

// Number of bytes of page cache memory.
// Units: Bytes.
Cache uint64 `json:"cache"`

// The amount of anonymous and swap cache memory (includes transparent
// hugepages).
// Units: Bytes.
RSS uint64 `json:"rss"`

// The amount of swap currently used by the processes in this cgroup
// Units: Bytes.
Swap uint64 `json:"swap"`

// The amount of working set memory, this includes recently accessed memory,
// dirty memory, and kernel memory. Working set is <= "usage".
// Units: Bytes.
WorkingSet uint64 `json:"working_set"`

Failcnt uint64 `json:"failcnt"`

ContainerData    MemoryStatsMemoryData `json:"container_data,omitempty"`
HierarchicalData MemoryStatsMemoryData `json:"hierarchical_data,omitempty"`
}

You might think that memory utilization is easily tracked with container_memory_usage_bytes, however, this metric also includes cached (think filesystem cache) items that can be evicted under memory pressure. The better metric is container_memory_working_set_bytes as this is what the OOM killer is watching for.
To calculate container memory utilization we use: sum(container_memory_working_set_bytes{name!~“POD”}) by (name)

kubelet 通过 watch container_memory_working_set_bytes 来判断是否OOM， 所以用 working set来评价容器内存使用量更科学

Cgroup中关于mem指标

cgroup目录相关文件

memory.stat中包含有的内存信息

active_anon + inactive_anon = anonymous memory + file cache for tmpfs + swap cache = rss + file cache for tmpfs
active_file + inactive_file = cache - size of tmpfs
working set = usage - total_inactive(k8s根据workingset 来判断是否驱逐pod)

mstat看到的active/inactive memory就分别是active list和inactive list中的内存大小。如果inactive list很大，表明在必要时可以回收的页面很多；而如果inactive list很小，说明可以回收的页面不多。
Active/inactive memory是针对用户进程所占用的内存而言的，内核占用的内存（包括slab）不在其中。
至于在源代码中看到的ACTIVE_ANON和ACTIVE_FILE，分别表示anonymous pages和file-backed pages。用户进程的内存页分为两种：与文件关联的内存（比如程序文件、数据文件所对应的内存页）和与文件无关的内存（比如进程的堆栈，用malloc申请的内存），前者称为file-backed pages，后者称为anonymous pages。File-backed pages在发生换页(page-in或page-out)时，是从它对应的文件读入或写出；anonymous pages在发生换页时，是对交换区进行读/写操作。

参考

• https://blog.freshtracks.io/a-deep-dive-into-kubernetes-metrics-part-3-container-resource-metrics-361c5ee46e66
• https://github.com/google/cadvisor/blob/08f0c2397cbca790a4db0f1212cb592cc88f6e26/info/v1/container.go#L338:6

• 点赞
• 收藏
• 分享
• 文章举报