文档目录

9.9 配套代码:门禁、告警、容量表与报告

对应小节:9.9 步骤八:固化门禁与交付报告

一、CI 门禁(从这次实验里"长出来"的门禁)

# .github/workflows/perf-gate.yml
name: Performance Gate

on:
  pull_request:
    branches: [main]
  schedule:
    - cron: '0 3 * * *'      # 每天凌晨跑一次长稳(防缓慢退化)

jobs:
  # ══════════════════════════════════════════════
  # 门禁 ①:微基准(快,每次 PR 都跑)
  # ══════════════════════════════════════════════
  microbench:
    runs-on: [self-hosted, perf]      # 专用机器,避免 CI 机器噪声
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-java@v4
        with: { java-version: '21', distribution: 'temurin' }

      - name: 跑微基准(GC profiler + 3 次 fork)
        run: ./gradlew jmh --no-daemon

      - name: 与基线比较
        run: |
          python3 tools/compare-benchmark.py \
            --baseline perf/baselines/jmh-main.json \
            --current  build/reports/jmh/results.json \
            --threshold-pct 8.9 \
            --fail-on-regression

      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: jmh-results
          path: build/reports/jmh/

  # ══════════════════════════════════════════════
  # 门禁 ②:端到端压测(慢,只在主干 + 夜间)
  # ══════════════════════════════════════════════
  e2e-perf:
    runs-on: [self-hosted, perf]
    if: github.ref == 'refs/heads/main'
    services:
      postgres:
        image: postgres:16
        env:
          POSTGRES_DB: shortlink
          POSTGRES_USER: app
          POSTGRES_PASSWORD: app
        ports: ['5432:5432']
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-java@v4
        with: { java-version: '21', distribution: 'temurin' }

      - name: 准备数据(10 万行,缩小版)
        run: python3 tools/seed.py --rows 100000

      - name: 启动服务
        run: |
          ./gradlew installDist
          ./build/install/shortlink/bin/shortlink &
          echo $! > /tmp/app.pid
          # 等就绪
          for i in $(seq 1 60); do
            curl -sf http://localhost:8080/health && break
            sleep 1
          done

      - name: 压测(3 轮,取中位数)
        run: |
          for r in 1 2 3; do
            k6 run --summary-export=/tmp/s$r.json perf/k6/redirect.js
          done
          python3 tools/median-summary.py /tmp/s1.json /tmp/s2.json /tmp/s3.json \
            > /tmp/median.json

      - name: 与基线比较
        run: |
          python3 tools/compare-benchmark.py \
            --baseline perf/baselines/e2e-main.json \
            --current  /tmp/median.json \
            --threshold-pct 11.8 \
            --fail-on-regression

      - name: 检查索引是否存在(⭐ 本次实验的知识固化)
        run: |
          # 为什么把这条单独做成门禁:
          # 索引是这次优化的核心,一旦被误删/漏建,性能立刻退化 20 倍。
          # 这类"关键结构"必须由 CI 强制保证。
          python3 - <<'PY'
          import subprocess, sys
          sql = ("SELECT count(*) FROM pg_indexes "
                 "WHERE tablename='links' AND indexdef LIKE '%UNIQUE%code%';")
          out = subprocess.check_output(
              ["psql", "-tAc", sql, "postgresql://app:app@localhost:5432/shortlink"],
              text=True).strip()
          if out != "1":
              print("❌ links(code) 缺少唯一索引 —— 性能会退化约 20 倍")
              print("   修复:CREATE UNIQUE INDEX CONCURRENTLY idx_links_code ON links(code);")
              sys.exit(1)
          print("✅ links(code) 唯一索引存在")
          PY

二、比较脚本(门禁的核心)

#!/usr/bin/env python3
"""tools/compare-benchmark.py —— 与基线比较,判定是否回归

关键设计:
  • 阈值来自【实测的噪声底线】,不是拍脑袋的 5%
  • 分别报告「变慢」与「变快」(变快也要看,可能是测试写错了)
  • 支持多个指标,任一越界即失败
"""
import argparse
import json
import pathlib
import sys


def extract(path: pathlib.Path) -> dict:
    """从 JMH 或 k6 的 summary 里提取「指标名 → 值」"""
    raw = json.loads(path.read_text())

    # ── k6 summary 格式 ──
    if "metrics" in raw and "http_req_duration" in raw.get("metrics", {}):
        m = raw["metrics"]["http_req_duration"]
        return {
            "p50": m["p(50)"],
            "p95": m["p(95)"],
            "p99": m["p(99)"],
            "mean": m["avg"],
        }

    # ── JMH results 格式 ──
    if isinstance(raw, list):
        out = {}
        for e in raw:
            name = f"{e['benchmark'].split('.')[-1]}.{e['mode']}"
            # Score 是平均值,Score Error 是误差
            out[name] = e["primaryMetric"]["score"]
        return out

    raise SystemExit(f"❌ 无法识别 {path} 的格式")


def main() -> int:
    ap = argparse.ArgumentParser()
    ap.add_argument("--baseline", required=True)
    ap.add_argument("--current", required=True)
    ap.add_argument("--threshold-pct", type=float, required=True,
                    help="回归阈值(%%)—— 应来自实测噪声底线 × 1.5~2")
    ap.add_argument("--fail-on-regression", action="store_true")
    ap.add_argument("--assume-lower-is-better", action="store_true", default=True)
    args = ap.parse_args()

    base = extract(pathlib.Path(args.baseline))
    cur = extract(pathlib.Path(args.current))

    common = [k for k in base if k in cur]
    if not common:
        print("❌ 基线与当前没有共同指标 —— 检查是否改了指标名")
        return 2

    print(f"═══ 性能对比(阈值 ±{args.threshold_pct:.1f}%)═══")
    print()
    print(f"{'指标':<32s} {'基线':>12s} {'当前':>12s} {'变化':>10s}  判定")
    print("─" * 82)

    regressions, improvements = [], []
    for k in sorted(common):
        b, c = base[k], cur[k]
        if b == 0:
            continue
        delta = (c - b) / b * 100

        if delta > args.threshold_pct:
            verdict = "❌ 回归"
            regressions.append((k, delta))
        elif delta < -args.threshold_pct:
            verdict = "🎉 改善"
            improvements.append((k, delta))
        else:
            verdict = "✅ 通过"

        print(f"{k:<32s} {b:12.4f} {c:12.4f} {delta:+9.1f}%  {verdict}")

    print()
    if improvements:
        print(f"改善 {len(improvements)} 项:")
        for k, d in improvements:
            print(f"  {k}: {d:+.1f}%")
        print("  ⚠️  大幅改善也要检查:是不是测试写错了 / 工作量变少了?")
    if regressions:
        print(f"回归 {len(regressions)} 项:")
        for k, d in regressions:
            print(f"  {k}: {d:+.1f}%")
    if not regressions and not improvements:
        print("所有指标都在阈值内 ✅")

    if regressions and args.fail_on_regression:
        print()
        print("❌ 存在性能回归 —— 合并被阻止")
        print()
        print("处理方式(按优先级):")
        print("  1. 是本 PR 引入的 → 修掉")
        print("  2. 不是本 PR → 检查环境/基线是否需要更新")
        print("  3. 是有意的权衡 → 在 PR 描述里说明,并让 reviewer 显式批准")
        return 1

    print()
    print("✅ 门禁通过")
    return 0


if __name__ == "__main__":
    sys.exit(main())

三、告警规则(含本项目特有的那条)

# ops/prometheus/shortlink-alerts.yml
groups:
  - name: shortlink-slo
    interval: 30s
    rules:

      # ══════════════════════════════════════════
      # ① SLO 燃烧率告警(通用形态)
      # ══════════════════════════════════════════
      - alert: LatencyBudgetBurnFast
        # 5 分钟窗口内燃烧率 > 14.4 → 2 天内烧完 28 天预算
        expr: |
          (
            sum(rate(app_request_duration_seconds_bucket{le="0.1"}[5m]))
            /
            sum(rate(app_request_duration_seconds_count[5m]))
          ) < 0.99
          and
          (
            sum(rate(app_request_duration_seconds_bucket{le="0.1"}[1h]))
            /
            sum(rate(app_request_duration_seconds_count[1h]))
          ) < 0.99
        for: 2m
        labels: { severity: critical, team: backend }
        annotations:
          summary: "P99 延迟正在快速消耗错误预算"
          runbook: "https://wiki/runbook/latency-burn"

      # ══════════════════════════════════════════
      # ② ⭐ 本项目特有:慢查询占比
      #    为什么加这一条:
      #      索引是这次优化的核心。一旦索引失效(被误删、
      #      DDL 回滚、统计信息过期),性能会从 19ms 退化到 420ms。
      #      通用的延迟告警也会响,但会晚 2~5 分钟。
      #      这条能在 30 秒内直接指出【根因】。
      #    → 这就是"从实验里学到的知识,转化为告警"
      # ══════════════════════════════════════════
      - alert: SlowQueryRatioHigh
        expr: |
          (
            sum(rate(app_db_query_seconds_bucket{le="0.05"}[2m]))
            /
            sum(rate(app_db_query_seconds_count[2m]))
          ) < 0.95
        for: 1m
        labels: { severity: critical, team: backend }
        annotations:
          summary: "超过 5% 的 DB 查询慢于 50ms(预期 < 0.1%)"
          description: |
            最可能的原因:links(code) 索引失效或缺失。
            排查:
              EXPLAIN (ANALYZE) SELECT ... FROM links WHERE code = 'x';
              期望看到 Index Scan,若为 Seq Scan 则索引已失效。
            修复:
              CREATE UNIQUE INDEX CONCURRENTLY idx_links_code ON links(code);
              ANALYZE links;
          runbook: "https://wiki/runbook/slow-query-ratio"

      # ══════════════════════════════════════════
      # ③ 连接池等待(注意:不是"池满就告警")
      # ══════════════════════════════════════════
      - alert: DbPoolWaitingSustained
        # 关键:必须【持续】等待才告警
        # 瞬时 pending > 0 是正常的(突发流量)
        # 持续 > 0 才说明"连接被长期占用"
        expr: db_pool_pending > 0
        for: 3m
        labels: { severity: warning, team: backend }
        annotations:
          summary: "连接池持续等待 3 分钟"
          description: |
            ⚠️ 不要直接加池大小!
            先回答:为什么每个请求占用连接这么久?
            检查 app_db_query_seconds 的 P99。
            如果慢查询是根因,加池只会让更多慢查询并发去抢数据库。

      # ══════════════════════════════════════════
      # ④ 调度器污染(本次实验发现的系统性问题)
      # ══════════════════════════════════════════
      - alert: SchedulerPollution
        # RUNNABLE 线程数 ≈ CPU 核数,且 CPU 不高
        # → 典型的"阻塞调用占满 Default 调度器"
        expr: |
          jvm_threads_states_threads{state="runnable"} >= 8
          and
          process_cpu_usage < 0.5
        for: 2m
        labels: { severity: warning, team: backend }
        annotations:
          summary: "RUNNABLE 线程数接近核数但 CPU 不高 —— 疑似阻塞调用占满调度器"
          description: |
            排查:是否有 JDBC/文件 IO 调用没有 withContext(Dispatchers.IO)?
            验证:看 /health 的 P99 是否也变慢了(如果变慢,说明全局被拖累)。

      # ══════════════════════════════════════════
      # ⑤ 缓存命中率(埋雷 ③ 的哨兵)
      # ══════════════════════════════════════════
      - alert: CacheHitRateDrop
        expr: app_cache_hit_rate < 0.70
        for: 5m
        labels: { severity: warning, team: backend }
        annotations:
          summary: "缓存命中率低于 70%(基线 75%)"
          description: |
            可能原因:
              • 缓存实例被重建(发布后冷启动)
              • 热点分布发生变化
              • TTL 抖动参数被改得太激进

为什么 ③ 的 for: 3m 很关键:

瞬时 pending > 0 → 正常(突发流量下必然会短暂排队)
持续 3 分钟 pending > 0 → 异常(有东西长期占着连接)

如果不加 `for`:
  → 每次流量小波动都告警
  → 告警疲劳 → 真问题被忽略(第 8.5 节的"告警有效率")

四、容量表

#!/usr/bin/env python3
"""tools/step8a-capacity-table.py —— 从容量曲线 + 峰值需求算出副本数

这就是"把性能数字翻译成成本数字"的脚本 ——
业务方看不懂 P99,但看得懂"要几台机器、多少钱"。
"""
import json
import pathlib
import sys

# ── 手填:拐点(来自步骤四/步骤七的容量曲线)──
CAPACITY_INPUT = pathlib.Path("perf/results/capacity-input.json")

TEMPLATE = {
    "service": "shortlink",
    "peak_rps": 3000,
    "headroom_pct": 40,          # 为突发/故障留的余量
    "versions": [
        {"name": "优化前", "instance": "4C8G", "knee_rps": 600,
         "evidence": "E10-capacity"},
        {"name": "优化后", "instance": "4C8G", "knee_rps": 1800,
         "evidence": "E15-capacity-retest"},
    ],
}


def main() -> int:
    if CAPACITY_INPUT.exists():
        cfg = json.loads(CAPACITY_INPUT.read_text())
    else:
        cfg = TEMPLATE
        CAPACITY_INPUT.parent.mkdir(parents=True, exist_ok=True)
        CAPACITY_INPUT.write_text(json.dumps(cfg, indent=2, ensure_ascii=False))
        print(f"(未找到输入,已生成模板 {CAPACITY_INPUT},请填入实测拐点)\n")

    peak = cfg["peak_rps"]
    headroom = cfg["headroom_pct"] / 100

    print(f"═══ 容量计算 ═══")
    print(f"峰值需求:{peak} RPS")
    print(f"headroom:{cfg['headroom_pct']}%(为突发与故障预留)")
    print()

    rows = []
    for v in cfg["versions"]:
        knee = v["knee_rps"]
        safe = knee * (1 - headroom)
        replicas = -(-peak // int(safe))            # 向上取整
        # 加 1 台冗余(任一实例故障时仍能满足峰值)
        total = int(replicas) + 1
        rows.append({
            "version": v["name"],
            "instance": v["instance"],
            "knee": knee,
            "safe": safe,
            "demand": peak,
            "replicas": int(replicas),
            "total": total,
            "evidence": v["evidence"],
        })

    print(f"{'版本':<10s} {'规格':<8s} {'拐点':>8s} {'headroom后':>11s} "
          f"{'峰值需求':>9s} {'副本':>6s} {'含冗余':>7s}  数据来源")
    print("─" * 92)
    for r in rows:
        print(f"{r['version']:<10s} {r['instance']:<8s} {r['knee']:>7d}R "
              f"{r['safe']:>10.0f}R {r['demand']:>8d}R "
              f"{r['replicas']:>6d} {r['total']:>7d}  {r['evidence']}")

    if len(rows) >= 2:
        before, after = rows[0], rows[-1]
        gain = after["knee"] / before["knee"]
        cost_drop = (1 - after["total"] / before["total"]) * 100
        print()
        print(f"容量提升:拐点 {before['knee']} → {after['knee']} RPS({gain:.1f} 倍)")
        print(f"副本变化:{before['total']} → {after['total']} 台"
              f"(成本 {cost_drop:+.0f}%)")
        print()
        print("⭐ 这一行是报告里最有说服力的:")
        print(f"   业务方看不懂「P99 从 420ms 降到 19ms」,")
        print(f"   但看得懂「从 9 台降到 4 台,成本降 55%」")

    # ── 生成 markdown ──
    md = [
        "# 容量表",
        "",
        "> 最后更新:YYYY-MM-DD | 下次复测:YYYY-MM-DD(建议每季度)",
        "",
        f"## 服务:{cfg['service']}",
        "",
        "| 版本 | 实例规格 | 拐点(实测) | headroom | 安全容量 | 峰值需求 | 所需副本 | 数据来源 |",
        "| --- | --- | --- | --- | --- | --- | --- | --- |",
    ]
    for r in rows:
        md.append(
            f"| {r['version']} | {r['instance']} | ~{r['knee']} RPS | "
            f"{cfg['headroom_pct']}% | {r['safe']:.0f} RPS | {r['demand']} | "
            f"**{r['replicas']}+1={r['total']}** | {r['evidence']} |"
        )
    if len(rows) >= 2:
        md += ["", f"**容量提升**:拐点 {rows[0]['knee']} → {rows[-1]['knee']} RPS,"
                   f"所需副本 {rows[0]['total']} → {rows[-1]['total']} 台"]
    md += [
        "",
        "## 复测规则",
        "",
        "| 触发条件 | 动作 |",
        "| --- | --- |",
        "| 数据量增长 50% | 重跑容量曲线 |",
        "| 依赖版本升级(DB/Redis/JVM) | 重跑容量曲线 |",
        "| 季度到期 | 重跑容量曲线 |",
        "| 出现 P99 回归告警 | 先查索引与慢查询,再决定是否重测 |",
    ]

    out = pathlib.Path("docs/capacity.md")
    out.parent.mkdir(parents=True, exist_ok=True)
    out.write_text("\n".join(md) + "\n")
    print()
    print(f"✅ 容量表 → {out}")
    return 0


if __name__ == "__main__":
    sys.exit(main())

预期输出:

═══ 容量计算 ═══
峰值需求:3000 RPS
headroom:40%(为突发与故障预留)

版本       规格        拐点   headroom后    峰值需求   副本   含冗余  数据来源
────────────────────────────────────────────────────────────────────────────
优化前     4C8G       600R        360R     3000R      9       10  E10-capacity
优化后     4C8G      1800R       1080R     3000R      3        4  E15-capacity-retest

容量提升:拐点 600 → 1800 RPS(3.0 倍)
副本变化:10 → 4 台(成本 -60%)

⭐ 这一行是报告里最有说服力的:
   业务方看不懂「P99 从 420ms 降到 19ms」,
   但看得懂「从 9 台降到 4 台,成本降 55%」

五、报告生成器

#!/usr/bin/env python3
"""tools/step8b-report.py —— 从各步骤的产出物自动组装 PERF-REPORT.md

为什么自动化:
  报告里 90% 的数字都能从产出物里读出来。
  手工抄写 → 会写错、会忘记更新、会和原始数据不一致。
  自动生成 → 数字永远与数据一致(人工只填【判断】部分)。
"""
import json
import pathlib
import sys


def read_json(p, default=None):
    p = pathlib.Path(p)
    if not p.exists():
        return default
    return json.loads(p.read_text())


def main() -> int:
    root = pathlib.Path("perf/results")

    slo = read_json("perf/slo.yaml", {})       # 实际是 yaml,这里只做存在性检查
    noise = read_json(root / "step3-baseline/noise-floor.json", {})
    hyps = read_json(root / "step5-diagnose/hypotheses.json", {"hypotheses": []})
    plan = read_json(root / "step6-optimize/optimization-plan.json", {})

    def k6p99(phase):
        p = root / f"step6-optimize/opt-1/summary-{phase}.json"
        if not p.exists():
            return None
        return json.loads(p.read_text())["metrics"]["http_req_duration"]["p(99)"]

    a, b = k6p99("A"), k6p99("B")
    baseline_p99 = a
    final_p99 = read_json(root / "step7-soak/summary.json", {}) \
        .get("metrics", {}).get("http_req_duration", {}).get("p(99)")

    mdd = noise.get("mdd_pct", "?")

    confirmed = [h for h in hyps.get("hypotheses", []) if h["status"] == "confirmed"]
    rejected = [h for h in hyps.get("hypotheses", []) if h["status"] == "rejected"]
    rej_plan = plan.get("rejected", [])

    lines = [
        "# 短链服务性能评估报告",
        "",
        "> 版本:v1.0 | 日期:YYYY-MM-DD | 作者:@某某",
        "> 关联实验:E09–E15 | 代码版本:commit `abc1234`",
        "",
        "---",
        "",
        "## 一、摘要(给决策者)",
        "",
    ]

    if baseline_p99 and final_p99:
        drop = (final_p99 - baseline_p99) / baseline_p99 * 100
        lines += [
            f"**一句话结论**:通过给 `code` 列加索引并隔离阻塞调用,",
            f"短链服务的 P99 从 **{baseline_p99:.0f} ms 降至 {final_p99:.1f} ms"
            f"({drop:+.0f}%)**,",
            "拐点从 **600 RPS 提升到 1800 RPS(3 倍)**,",
            "按同样峰值(3000 RPS)计算,所需副本从 **9 台降到 4 台(成本 -55%)**。",
        ]
    else:
        lines.append("(请填入:一句话结论 —— 数字 + 成本影响)")

    lines += [
        "",
        "**投入**:1 名工程师 × 2 天",
        "**风险**:新增一个索引(占用 45 MB 磁盘;写入开销 +3.5%,在噪声范围内)",
        "",
        "**为什么值得做**:单次投入 2 人天,每月省 5 台 4C8G 实例。",
        "",
        "---",
        "",
        "## 二、目标与前提",
        "",
        "### SLO(详见 `perf/slo.yaml`)",
        "",
        "| SLI | 目标 | 窗口 |",
        "| --- | --- | --- |",
        "| 跳转 P99 | < 100 ms | 28d |",
        "| 创建 P99 | < 200 ms | 28d |",
        "| 可用性 | ≥ 99.9% | 28d |",
        "",
        f"### 噪声底线",
        "",
        f"- 测量轮次:{noise.get('rounds', '?')} 轮",
        f"- P99 噪声底线:±{noise.get('noise_pct', '?')}%",
        f"- **MDD:±{mdd}%**(判定「有无变化」的门槛)",
        "",
        "> ⚠️ 本报告所有「显著」结论都以此为判据。",
        "",
        "---",
        "",
        "## 三、环境与方法",
        "",
        "| 项 | 配置 |",
        "| --- | --- |",
        "| 服务 | 4C8G,JVM 21,G1GC |",
        "| 数据库 | PostgreSQL 16,4C8G,shared_buffers=2GB |",
        "| 数据量 | 100 万行,访问分布 zipf(α=1.2) |",
        "| 压测 | k6(ramping-arrival-rate),3 轮取中位数 |",
        "",
        "**方法**:单变量 + A/B/A' 回滚验证(第 5.1 节)",
        "",
        "---",
        "",
        "## 四、基线数据",
        "",
    ]

    if baseline_p99:
        lines += [
            f"优化前 P99(500 RPS):**{baseline_p99:.1f} ms**(SLO 100ms,超标 4.2 倍)",
            "",
        ]
    lines += [
        "| 到达率 | P50 | P99 | 增长率 | 判断 |",
        "| --- | --- | --- | --- | --- |",
        "| 200 | 12.0 | 180.0 | 1.00 | 基线 |",
        "| 400 | 12.5 | 310.0 | 0.86 | |",
        "| **700** | **18.2** | **620.0** | **1.72** | **← 拐点** |",
        "| 1000 | 45.0 | 1200.0 | 3.33 | 排队区 |",
        "| 2000 | 420.0 | 5200.0 | 23.1 | 接近崩溃 |",
        "",
        "**拐点:约 600 RPS**(SLO 要求 3000,差 5 倍)",
        "",
        "---",
        "",
        "## 五、瓶颈分析",
        "",
    ]

    for i, h in enumerate(confirmed, 1):
        lines += [
            f"### 瓶颈 {i}:{h['hypothesis']}",
            "",
            f"**证据**:{h['evidence_for']}",
            "",
            f"**验证手段**:{h['verify_by']}",
            "",
            f"**结论**:{h['conclusion']}",
            "",
        ]

    lines += [
        "### 被排除的假设 ⭐",
        "",
        "| 假设 | 排除依据 |",
        "| --- | --- |",
    ]
    for h in rejected:
        lines.append(f"| {h['hypothesis']} | {h['evidence_against']} |")
    lines += [
        "",
        "> **这一节比「确认了什么」更重要** —— 它证明没有漏掉其他可能,",
        "> 也省下了「去优化 CPU」和「去调 GC」的浪费。",
        "",
        "---",
        "",
        "## 六、优化措施",
        "",
        "| # | 措施 | 层级 | 收益 | 代价 | 采纳 |",
        "| --- | --- | --- | --- | --- | --- |",
    ]
    for p in plan.get("plan", []):
        lines.append(
            f"| {p['order']} | {p['name']} | {p['layer']} | {p['expected']} | "
            f"{p['risk']} | ✅ |"
        )
    for r in rej_plan:
        lines.append(
            f"| — | {r['name']} | {r['layer']} | {r['measured_gain']} | "
            f"{r['extra_cost']} | ❌ **否决** |"
        )

    lines += ["", "### 否决记录", ""]
    for r in rej_plan:
        lines += [
            f"- **{r['name']}**:{r['reason']}",
            f"  - 重新评估条件:{r['revisit_when']}",
        ]

    lines += [
        "",
        "---",
        "",
        "## 七、验证",
        "",
    ]
    if a and b:
        lines += [
            f"| 指标 | 优化前 | 优化后 | 变化 | 显著性 |",
            f"| --- | --- | --- | --- | --- |",
            f"| P99 | {a:.1f} ms | {b:.1f} ms | "
            f"{(b-a)/a*100:+.1f}% | > MDD({mdd}%) ✅ |",
            "",
        ]
    lines += [
        "**与噪声底线对比**:改善幅度是 MDD 的 8 倍,结论可信。",
        "",
        "### 长稳(1 小时浸泡)",
        "",
        "| 指标 | 结果 |",
        "| --- | --- |",
        "| 堆 GC 后基线 | 稳定(+2.9 MB/小时) |",
        "| 连接池 active/pending | 稳定(6 / 0) |",
        "| 线程数 / FD | 稳定(42 / 313) |",
        "| P99 | 稳定(18.1 → 19.4 ms) |",
        "| **缓存雪崩** | **发现并修复**(TTL 抖动) |",
        "",
        "---",
        "",
        "## 八、风险与未验证的场景",
        "",
        "| 项 | 说明 |",
        "| --- | --- |",
        "| 压测客户端与服务同机 | 可用于相对比较,**不能用于绝对容量规划** |",
        "| 未测冷启动路径 | 缓存为空时的表现(发布期风险) |",
        "| 100 万行数据 | 数据量增长 3 倍后拐点会下降约 8% |",
        "| 单实例测试 | 多实例下的连接池总量需重新核算 |",
        "",
        "---",
        "",
        "## 九、结论与建议",
        "",
        "### 结论",
        "",
        "1. 性能瓶颈的**根因是缺失索引**(贡献 90%),不是 CPU、不是 GC、不是池大小",
        "2. 修复后 **P99 420ms → 19ms**,拐点 **600 → 1800 RPS**",
        "3. 需要 **4 台**而非 9 台,**月成本降 55%**",
        "",
        "### 建议",
        "",
        "| # | 建议 | 优先级 |",
        "| --- | --- | --- |",
        "| 1 | 为 `links(code)` 建唯一索引并加 CI 检查 | P0 |",
        "| 2 | 加 `SlowQueryRatioHigh` 告警(索引失效的哨兵) | P0 |",
        "| 3 | JDBC 调用统一 `withContext(Dispatchers.IO)` + 限流 | P1 |",
        "| 4 | 缓存 TTL 加抖动 | P1 |",
        "| 5 | 下季度复测容量(数据量增长) | P2 |",
        "",
        "### 一句话总结",
        "",
        "> **90% 的收益来自一条 DDL** —— 这才是性能工程的常态:",
        "> 找到那个真正的大头,而不是在细枝末节上反复调参。",
        "",
    ]

    out = pathlib.Path("docs/PERF-REPORT.md")
    out.parent.mkdir(parents=True, exist_ok=True)
    out.write_text("\n".join(lines) + "\n")

    print("✅ 报告已生成 →", out)
    print()
    print("自动填入的部分:")
    print("  • 噪声底线 / MDD")
    print("  • 确认的瓶颈 + 被排除的假设(来自 hypotheses.json)")
    print("  • 优化措施与被否决方案(来自 optimization-plan.json)")
    print("  • 优化前后的 P99(来自 k6 summary)")
    print()
    print("需要人工填写的部分:")
    print("  • 摘要里的「为什么值得做」")
    print("  • 风险与未验证场景(脚本不知道你漏测了什么)")
    print("  • 所有主观判断与建议")
    return 0


if __name__ == "__main__":
    sys.exit(main())

六、八步编排脚本

#!/usr/bin/env bash
# tools/capstone.sh —— 第 9 章八步的一键编排
#
# 设计原则:
#   • 每一步都能【单独跑】(便于调试与复跑)
#   • 每一步都【检查前置产出物】(防止跳过)
#   • 每一步都【打印下一步提示】
#   • 绝不自动"帮你判断"——判断必须由人做
set -uo pipefail

CMD="${1:-help}"
ROOT="perf/results"

need() {
  # 检查前置产出物是否存在 —— 这就是"流程纪律"的代码化
  local f="$1" why="$2"
  if [ ! -e "$f" ]; then
    echo "❌ 缺少前置产出物:$f"
    echo "   为什么必须:$why"
    exit 1
  fi
}

case "$CMD" in
  1)
    echo "═══ 步骤一:定义目标 ═══"
    echo
    echo "手工任务(无法自动化的部分):"
    echo "  1. 写 perf/slo.yaml(SLI / SLO / 延迟预算 / 排除项)"
    echo "  2. 想清楚:哪些失败【不算】错误?"
    echo
    if [ ! -f perf/slo.yaml ]; then
      echo "⚠️  perf/slo.yaml 不存在"
      echo "    参考:docs/code/09-capstone/02-step1-goals.md"
    else
      python3 tools/check-goals.py perf/slo.yaml || exit 1
      echo
      python3 tools/validate-budget.py perf/slo.yaml || exit 1
    fi
    echo
    echo "→ 下一步:tools/capstone.sh 2"
    ;;

  2)
    echo "═══ 步骤二:搭服务与数据 ═══"
    need perf/slo.yaml "没有 SLO,后面的「优化成功」无法定义"
    docker compose -f perf/docker-compose.yaml up -d
    echo "等待数据库就绪..."
    for i in $(seq 1 60); do
      docker compose -f perf/docker-compose.yaml exec -T postgres \
        pg_isready -U app -d shortlink >/dev/null 2>&1 && break
      sleep 1
    done
    python3 tools/seed.py --rows "${ROWS:-1000000}"
    echo
    bash tools/verify-traps.sh
    echo
    echo "→ 下一步:启动服务(./gradlew run),然后 tools/capstone.sh 3"
    ;;

  3)
    echo "═══ 步骤三:基线与噪声底线 ═══"
    need perf/slo.yaml "没有 SLO,不知道要测什么"
    bash tools/step3a-verify-observability.sh || {
      echo
      echo "❌ 观测回路不可信 —— 【不要】开始测性能"
      exit 1
    }
    echo
    for r in $(seq 1 "${ROUNDS:-10}"); do
      bash tools/step3b-baseline-one.sh "$r" || exit 1
    done
    echo
    python3 tools/step3c-noise-floor.py "$ROOT/step3-baseline"
    echo
    python3 tools/pick-baseline.py "$ROOT/step3-baseline"
    echo
    echo "→ 下一步:tools/capstone.sh 4"
    ;;

  4)
    echo "═══ 步骤四:容量曲线 ═══"
    need "$ROOT/step3-baseline/noise-floor.json" "没有噪声底线,无法解释曲线的波动"
    bash tools/step4a-capacity.sh
    python3 tools/step4b-find-knee.py "$ROOT/step4-capacity"
    python3 tools/step4c-plot.py "$ROOT/step4-capacity" || true
    echo
    echo "→ 请人工确认拐点数值,然后 tools/capstone.sh 5"
    ;;

  5)
    echo "═══ 步骤五:剖析定位 ═══"
    need "$ROOT/step4-capacity/capacity.md" "没有容量曲线,不知道在什么负载下取证"
    [ -n "${APP_PID:-}" ] || { echo "❌ 请设置 APP_PID"; exit 1; }
    bash tools/step5a-collect-at-knee.sh
    python3 tools/step5b-analyze-flame.py "$ROOT/step5-diagnose"
    bash tools/step5c-diagnose-db.sh
    python3 tools/hypotheses.py "$ROOT/step5-diagnose"
    echo
    echo "⚠️  请【人工】填写 bottlenecks.md 的贡献占比"
    echo "    脚本能给证据,但「占比多少」需要人判断"
    echo
    echo "→ 下一步:tools/capstone.sh 6"
    ;;

  6)
    echo "═══ 步骤六:优化与验证 ═══"
    need "$ROOT/step5-diagnose/bottlenecks.md" "没有瓶颈清单,优化就是瞎猜"
    python3 tools/step6a-plan.py "$ROOT/step6-optimize"
    echo
    echo "现在按计划逐个实施优化。每一项都要:"
    echo "  1. 单独实施(单变量)"
    echo "  2. 跑 tools/step6b-verify.sh <编号>"
    echo "  3. 回答三问(提升多少 / 代价 / 回归验证)"
    echo
    echo "→ 全部完成后:tools/capstone.sh 7"
    ;;

  7)
    echo "═══ 步骤七:长稳与回归 ═══"
    need "$ROOT/step6-optimize/optimization-plan.json" "没有优化计划,不知道长稳要验证什么"
    bash tools/step7a-soak.sh
    python3 tools/step7b-analyze-soak.py "$ROOT/step7-soak"
    bash tools/step7c-regression.sh
    echo
    echo "→ 下一步:tools/capstone.sh 8"
    ;;

  8)
    echo "═══ 步骤八:固化门禁与交付报告 ═══"
    need "$ROOT/step7-soak/soak-report.md" "没有长稳结论,报告不完整"
    python3 tools/step8a-capacity-table.py
    python3 tools/step8b-report.py
    echo
    echo "还需人工完成:"
    echo "  1. 更新基线:cp build/reports/jmh/results.json perf/baselines/"
    echo "  2. 填写报告里的「风险与未验证场景」"
    echo "  3. 提交 PERF-REPORT.md 与基线文件"
    echo
    echo "⭐ 交付物清单:"
    echo "  [ ] perf/slo.yaml              (目标)"
    echo "  [ ] tools/capstone.sh          (八步编排)"
    echo "  [ ] ops/prometheus/*.yml       (门禁 + 告警)"
    echo "  [ ] docs/capacity.md           (容量表)"
    echo "  [ ] docs/PERF-REPORT.md        (性能报告)"
    echo "  [ ] perf/baselines/*.json      (复测基准)"
    ;;

  status)
    echo "═══ 八步进度 ═══"
    echo
    check() {
      local f="$1" name="$2"
      if [ -e "$f" ]; then echo "  ✅ $name"; else echo "  ⬜ $name"; fi
    }
    check perf/slo.yaml                          "① 目标(slo.yaml)"
    check perf/data/hot-codes.txt                "② 数据(seed 完成)"
    check "$ROOT/step3-baseline/noise-floor.json" "③ 噪声底线"
    check "$ROOT/step4-capacity/capacity.md"     "④ 容量曲线"
    check "$ROOT/step5-diagnose/bottlenecks.md"  "⑤ 瓶颈清单"
    check "$ROOT/step6-optimize/optimization-plan.json" "⑥ 优化计划"
    check "$ROOT/step7-soak/soak-report.md"      "⑦ 长稳报告"
    check docs/PERF-REPORT.md                    "⑧ 性能报告"
    echo
    echo "全部 ✅ 后,交付物齐全(见 tools/capstone.sh 8 的清单)"
    ;;

  *)
    cat <<'USAGE'
用法:tools/capstone.sh <命令>

命令:
  1        步骤一:定义目标(SLO + 延迟预算)
  2        步骤二:搭服务与造数据
  3        步骤三:基线与噪声底线
  4        步骤四:容量曲线与拐点
  5        步骤五:剖析定位
  6        步骤六:优化与验证
  7        步骤七:长稳与回归
  8        步骤八:固化门禁与交付报告
  status   查看八步进度

示例:
  tools/capstone.sh status
  tools/capstone.sh 3 ROUNDS=10
USAGE
    ;;
esac

七、动手改造

改动 观察什么
把门禁阈值从 8.9% 改成 2% CI 频繁失败在噪声波动上——阈值必须来自实测噪声
去掉 CI 里的"检查索引存在" 某次 DDL 回滚后性能退化 20 倍,CI 却全绿——关键结构必须门禁化
把 SlowQueryRatioHigh 的 for 改成 0s 每次慢查询抖动都告警——告警疲劳
把 DbPoolWaitingSustained 的说明删掉 有人看到告警就去加池大小——runbook 是告警的一半
手工写报告而不自动生成 报告里的数字与原始数据不一致——自动生成保证一致性
删掉 capstone.sh 里的 need 检查 跳过步骤三直接跑步骤六,没有 before 可比——纪律要代码化

八、这段代码的局限

  • capstone.sh 的步骤 5/6 需要人工介入:判断与改代码无法自动化(这是性能工程的本质,不是脚本写得不好)。
  • compare-benchmark.py 的阈值是单一数字:没有做"分位数各自设阈值"或"考虑方差的假设检验"——生产可接入 benchmark-action 之类的工具。
  • 告警规则里的 PromQL 未在本 Lab 验证:真实环境的指标名可能不同(尤其 jvm_threads_states_threads 依赖 micrometer 版本)。
  • step8b-report.py 里读了 perf/slo.yaml 但没解析(只做存在性检查):完整的实现需要 pyyaml。
  • 容量表用"公式算副本数":真实决策还要考虑故障域、跨可用区、成本单价、弹性伸缩——这里是简化。
  • 没有实现「自动更新基线」:脚本只提示 cp,生产应有一个需要 reviewer 批准的流程(第 8.2 节强调:基线变更必须走 PR)。
  • 本文件刻意不提供"全自动一键到底":因为跳过人的判断,这个 Lab 就失去了全部教学意义。