[k8s]prometheus+alertmanager二进制安装实现简单邮件告警

本次任务是用alertmanaer发一个报警邮件

本次环境采用二进制普罗组件

本次准备监控一个节点的内存,当使用率大于2%时候(测试),发邮件报警.

k8s集群使用普罗官方文档

环境准备

下载二进制https://prometheus.io/download/

https://github.com/prometheus/prometheus/releases/download/v2.0.0/prometheus-2.0.0.windows-amd64.tar.gz https://github.com/prometheus/alertmanager/releases/download/v0.12.0/alertmanager-0.12.0.windows-amd64.tar.gz https://github.com/prometheus/node_exporter/releases/download/v0.15.2/node_exporter-0.15.2.linux-amd64.tar.gz

解压

/root/
├── alertmanager -> alertmanager-0.12.0.linux-amd64
├── alertmanager-0.12.0.linux-amd64
├── alertmanager-0.12.0.linux-amd64.tar.gz
├── node_exporter-0.15.2.linux-amd64
├── node_exporter-0.15.2.linux-amd64.tar.gz
├── prometheus -> prometheus-2.0.0.linux-amd64
├── prometheus-2.0.0.linux-amd64
└── prometheus-2.0.0.linux-amd64.tar.gz

实验架构

配置alertmanager

创建 alert.yml

[root@n1 alertmanager]# ls
alertmanager  alert.yml  amtool  data  LICENSE  NOTICE  simple.yml

alert.yml 里面定义下: 谁发送 什么事件 发给谁 怎么发等.

cat alert.yml
global:
smtp_smarthost: 'smtp.163.com:25'
smtp_from: 'maotai@163.com'
smtp_auth_username: 'maotai@163.com'
smtp_auth_password: '123456'

templates:
- '/root/alertmanager/template/*.tmpl'

route:
group_by: ['alertname', 'cluster', 'service']
group_wait: 30s
group_interval: 5m
repeat_interval: 10m
receiver: default-receiver

receivers:
- name: 'default-receiver'
email_configs:
- to: 'maotai@foxmail.com'

- 配置好后启动即可
./alertmanager -config.file=./alert.yml

配置prometheus

报警规则rule.yml配置(将被prometheus.yml调用)

当使用率大于2%时候(测试),发邮件报警

$ cat rule.yml
groups:
- name: test-rule
rules:
- alert: NodeMemoryUsage
expr: (node_memory_MemTotal - (node_memory_MemFree+node_memory_Buffers+node_memory_Cached )) / node_memory_MemTotal * 100 > 2
for: 1m
labels:
severity: warning
annotations:
summary: "{{$labels.instance}}: High Memory usage detected"
description: "{{$labels.instance}}: Memory usage is above 80% (current value is: {{ $value }}"

关键在于这个公式

(node_memory_MemTotal - (node_memory_MemFree+node_memory_Buffers+node_memory_Cached )) / node_memory_MemTotal * 100 > 2

labels 给这个规则打个标签

annotations(报警说明)这部分是报警内容

监控k从哪里获取?(后面有说) node_memory_MemTotal/node_memory_Buffers/node_memory_Cached

prometheus.yml配置

添加node_expolore这个job

添加rule_files的报警规则,rule_files部分调用rule.yml

$ cat prometheus.yml
global:
scrape_interval:     15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.

alerting:
alertmanagers:
- static_configs:
- targets: ["localhost:9093"]

rule_files:
- /root/prometheus/rule.yml

scrape_configs:
- job_name: 'prometheus'
static_configs:
- targets: ['192.168.14.11:9090']
- job_name: linux
static_configs:
- targets: ['192.168.14.11:9100']
labels:
instance: db1

配置好后启动普罗然后访问,可以看到了node target了.



查看node_explore抛出的metric



查看alert,可以看到告警规则发生的状态





这些公式的key从这里可以看到(前提是当你安装了对应的explore),按照这个k来写告警公式

查看收到的邮件

微信报警配置

global:
# The smarthost and SMTP sender used for mail notifications.
resolve_timeout: 6m
smtp_smarthost: '172.16.100.14:25'
smtp_from: 'svnbuild_yf@iflytek.com'
smtp_auth_username: 'svnbuild_yf'
smtp_auth_password: 'tag#write@2015313'
smtp_require_tls: false

# The auth token for Hipchat.
hipchat_auth_token: '1234556789'
# Alternative host for Hipchat.
hipchat_api_url: 'https://hipchat.foobar.org/'
wechat_api_url: "https://qyapi.weixin.qq.com/cgi-bin/"
wechat_api_secret: "4tQroVeB0xUcccccccc65Yfkj2Nkt90a80MH3ayI"
wechat_api_corp_id: "wxaf5acxxxx5f8eb98"

# The directory from which notification templates are read.
templates:
- 'templates/*.tmpl'

# The root route on which each incoming alert enters.
route:
# The labels by which incoming alerts are grouped together. For example,
# multiple alerts coming in for cluster=A and alertname=LatencyHigh would
# be batched into a single group.
group_by: ['alertname']

# When a new group of alerts is created by an incoming alert, wait at
# least 'group_wait' to send the initial notification.
# This way ensures that you get multiple alerts for the same group that start
# firing shortly after another are batched together on the first
# notification.
group_wait: 3s

# When the first notification was sent, wait 'group_interval' to send a batch
# of new alerts that started firing for that group.
group_interval: 5m

# If an alert has successfully been sent, wait 'repeat_interval' to
# resend them.
repeat_interval: 1h

# A default receiver
receiver: ybyang2

routes:
- match:
job: "11"
#service: "node_exporter"
routes:
- match:
status: yellow
receiver: ybyang2
- match:
status: orange
receiver: berlin

# Inhibition rules allow to mute a set of alerts given that another alert is
# firing.
# We use this to mute any warning-level notifications if the same alert is
# already critical.
inhibit_rules:
- source_match:
service: 'up'
target_match:
service: 'mysql'
# Apply inhibition if the alerqtname is the same.
equal: ["instance"]

- source_match:
service: "mysql"
target_match:
service: "mysql-query"
equal: ['instance']

- source_match:
service: "A"
target_match:
service: "B"
equal: ["instance"]

- source_match:
service: "B"
target_match:
service: "C"
equal: ["instance"]

receivers:
- name: 'ybyang2'
email_configs:
- to: 'ybyang2@iflytek.com'
send_resolved: true
html: '{{ template "email.default.html" . }}'
headers: { Subject: "[mail] 测试技术部监控告警邮件" }

- name: "berlin"
wechat_configs:
- send_resolved: true
to_user: "@all"
to_party: ""
to_tag: ""
agent_id: "1"
corp_id: "wxaf5a99ccccc5f8eb98"