08 告警工程

✅ Payment API 错误率 > 5%
✅ 下单成功率低于 99%
✅ P99 延迟超过 2s
✅ 用户登录失败比例 > 1%

❌ CPU > 80%
❌ Redis master pod 不可用
❌ 内存使用 > 90%
❌ Pod 重启次数 > 3

# P0 告警：快速燃烧（14.4x：1 小时即烧掉月度预算的 2%）
- alert: HighErrorBudgetBurnRate
  expr: |
    (
      job:slo_errors_per_request:ratio_rate1h{job="payment"} > (14.4 * 0.001)
    )
    and
    (
      job:slo_errors_per_request:ratio_rate5m{job="payment"} > (14.4 * 0.001)
    )
  for: 2m
  labels:
    severity: critical
    team: payment
  annotations:
    summary: '{{ $labels.job }} Error Budget 快速消耗（1h 燃烧率超过 14.4x 阈值）'
    description: |
      服务 {{ $labels.job }} 当前 1h 燃烧率已达 14.4x 阈值（1 小时烧掉月度预算的 2%）。
      按此速度，月度 Error Budget 约 2 天（≈50 小时）内耗尽。
    runbook_url: "https://wiki.example.com/runbooks/slo-burn-rate"
    grafana_url: "https://grafana.example.com/d/slo?var-job={{ $labels.job }}"

# 错误：OR 太宽松，短窗口抖动频繁触发
expr: |
  ratio_rate1h > 0.0144
  or
  ratio_rate5m > 0.0144

# 正确：AND，两个窗口都超标才告警
expr: |
  ratio_rate1h > 0.0144
  and
  ratio_rate5m > 0.0144

groups:
  - name: slo_recording_rules
    interval: 30s
    rules:
      - record: job:slo_errors_per_request:ratio_rate5m
        expr: |
          sum(rate(http_requests_total{status=~"5.."}[5m])) by (job)
          / sum(rate(http_requests_total[5m])) by (job)
      - record: job:slo_errors_per_request:ratio_rate30m
        expr: |
          sum(rate(http_requests_total{status=~"5.."}[30m])) by (job)
          / sum(rate(http_requests_total[30m])) by (job)
      - record: job:slo_errors_per_request:ratio_rate1h
        expr: |
          sum(rate(http_requests_total{status=~"5.."}[1h])) by (job)
          / sum(rate(http_requests_total[1h])) by (job)
      - record: job:slo_errors_per_request:ratio_rate6h
        expr: |
          sum(rate(http_requests_total{status=~"5.."}[6h])) by (job)
          / sum(rate(http_requests_total[6h])) by (job)
      - record: job:slo_errors_per_request:ratio_rate3d
        expr: |
          sum(rate(http_requests_total{status=~"5.."}[3d])) by (job)
          / sum(rate(http_requests_total[3d])) by (job)

# 延迟燃烧率告警：P99 > 500ms 的请求比例超标
- alert: HighLatencyBurnRate
  expr: |
    (
      job:slo_slow_requests:ratio_rate1h{job="payment"} > (14.4 * 0.01)
    )
    and
    (
      job:slo_slow_requests:ratio_rate5m{job="payment"} > (14.4 * 0.01)
    )
  for: 2m
  labels:
    severity: critical
    slo_type: latency

- record: job:slo_slow_requests:ratio_rate5m
  expr: |
    1 - (
      sum(rate(http_request_duration_seconds_bucket{le="0.5"}[5m])) by (job)
      /
      sum(rate(http_request_duration_seconds_count[5m])) by (job)
    )

噪音告警 > 有效告警 → 告警群被静音 → 工程师生理脱敏
  → 真正的事故发生时响应延迟 → MTTR 飙升

# 每班次告警数（按值班人聚合）
sum by(oncall_user) (increase(alertmanager_notifications_total{receiver="pager"}[7d]))

# 夜间告警占比
sum(increase(alertmanager_notifications_total{hour=~"22|23|00|01|02|03|04|05"}[30d]))
/
sum(increase(alertmanager_notifications_total[30d]))

# MTTA（平均响应时间）
avg(pd_incident_ack_time_seconds)

# 告警真阳率（需要手动标记或从 postmortem 推导）
count_over_time(incident_true_positive[30d])
/
count_over_time(incident_total[30d])

route:
  receiver: default
  group_by: ['alertname', 'cluster', 'namespace']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  routes:
    # Critical → PagerDuty + 钉钉
    - matchers: [severity = "critical"]
      receiver: pagerduty-critical
      group_wait: 10s
      repeat_interval: 30m
      continue: true               # 同时发到钉钉
    
    - matchers: [severity = "critical"]
      receiver: dingtalk-critical
    
    # Warning → 钉钉普通群
    - matchers: [severity = "warning"]
      receiver: dingtalk-warning
      group_wait: 60s
      repeat_interval: 8h

inhibit_rules:
  # Critical 抑制同组的 Warning
  - source_matchers: [severity = "critical"]
    target_matchers: [severity = "warning"]
    equal: ['alertname', 'cluster', 'namespace']
  
  # 节点 NotReady 时抑制其上 Pod 告警
  - source_matchers: [alertname = "NodeNotReady"]
    target_matchers: [alertname =~ "PodCrashLooping|PodNotReady"]
    equal: ['node', 'cluster']
  
  # 集群不可达时抑制所有该集群告警
  - source_matchers: [alertname = "ClusterUnreachable"]
    target_matchers: [cluster != ""]
    equal: ['cluster']

# 错误：用 .+ 会误伤 Watchdog
amtool silence add alertname=~".+" --duration=2h

# 正确：用负匹配排除 Watchdog
amtool silence add alertname!=Watchdog --duration=2h --comment="上线窗口"

Watchdog 告警持续 firing → healthchecks.io 定期收到心跳

如果某天 healthchecks.io 超过一定时间没收到心跳
  → 说明整条告警链路（Prometheus → Alertmanager → 通知）自己挂了
  → 触发反向告警

# PrometheusRule
- alert: Watchdog
  expr: vector(1)
  for: 0s
  labels:
    severity: none
    channel: heartbeat
  annotations:
    summary: "Watchdog heartbeat — 永远 firing，无心跳即链路异常"

# Alertmanager 路由
routes:
  - matchers: [alertname = "Watchdog"]
    receiver: dead-mans-switch
    group_wait: 10s
    group_interval: 1m
    repeat_interval: 1m

receivers:
  - name: dead-mans-switch
    webhook_configs:
      - url: 'https://hc-ping.com/<uuid>'
        send_resolved: false

[CRITICAL] payment-service Error Budget 燃烧告警

服务：payment-service
燃烧率：18.3x（正常基准 1x）
当前 1h 错误率：1.83%
Budget 耗尽预计：~1.6 天（≈39 小时）

→ Runbook：https://wiki.example.com/runbook/payment-burn-rate
→ Grafana：https://grafana.example.com/d/slo?var-job=payment-service
→ 日志：https://loki.example.com/?query={job="payment-service"}

新建 → 试运行（2 周，ticket 级）→ 评估 → 升 page 级 / 调整 / 删除
  → 持续运行 → 月度审计 → 调整或退役