9.9 配套代码:门禁、告警、容量表与报告
对应小节:9.9 步骤八:固化门禁与交付报告
一、CI 门禁(从这次实验里"长出来"的门禁)
# .github/workflows/perf-gate.yml
name: Performance Gate
on:
pull_request:
branches: [main]
schedule:
- cron: '0 3 * * *' # 每天凌晨跑一次长稳(防缓慢退化)
jobs:
# ══════════════════════════════════════════════
# 门禁 ①:微基准(快,每次 PR 都跑)
# ══════════════════════════════════════════════
microbench:
runs-on: [self-hosted, perf] # 专用机器,避免 CI 机器噪声
steps:
- uses: actions/checkout@v4
- uses: actions/setup-java@v4
with: { java-version: '21', distribution: 'temurin' }
- name: 跑微基准(GC profiler + 3 次 fork)
run: ./gradlew jmh --no-daemon
- name: 与基线比较
run: |
python3 tools/compare-benchmark.py \
--baseline perf/baselines/jmh-main.json \
--current build/reports/jmh/results.json \
--threshold-pct 8.9 \
--fail-on-regression
- uses: actions/upload-artifact@v4
if: always()
with:
name: jmh-results
path: build/reports/jmh/
# ══════════════════════════════════════════════
# 门禁 ②:端到端压测(慢,只在主干 + 夜间)
# ══════════════════════════════════════════════
e2e-perf:
runs-on: [self-hosted, perf]
if: github.ref == 'refs/heads/main'
services:
postgres:
image: postgres:16
env:
POSTGRES_DB: shortlink
POSTGRES_USER: app
POSTGRES_PASSWORD: app
ports: ['5432:5432']
steps:
- uses: actions/checkout@v4
- uses: actions/setup-java@v4
with: { java-version: '21', distribution: 'temurin' }
- name: 准备数据(10 万行,缩小版)
run: python3 tools/seed.py --rows 100000
- name: 启动服务
run: |
./gradlew installDist
./build/install/shortlink/bin/shortlink &
echo $! > /tmp/app.pid
# 等就绪
for i in $(seq 1 60); do
curl -sf http://localhost:8080/health && break
sleep 1
done
- name: 压测(3 轮,取中位数)
run: |
for r in 1 2 3; do
k6 run --summary-export=/tmp/s$r.json perf/k6/redirect.js
done
python3 tools/median-summary.py /tmp/s1.json /tmp/s2.json /tmp/s3.json \
> /tmp/median.json
- name: 与基线比较
run: |
python3 tools/compare-benchmark.py \
--baseline perf/baselines/e2e-main.json \
--current /tmp/median.json \
--threshold-pct 11.8 \
--fail-on-regression
- name: 检查索引是否存在(⭐ 本次实验的知识固化)
run: |
# 为什么把这条单独做成门禁:
# 索引是这次优化的核心,一旦被误删/漏建,性能立刻退化 20 倍。
# 这类"关键结构"必须由 CI 强制保证。
python3 - <<'PY'
import subprocess, sys
sql = ("SELECT count(*) FROM pg_indexes "
"WHERE tablename='links' AND indexdef LIKE '%UNIQUE%code%';")
out = subprocess.check_output(
["psql", "-tAc", sql, "postgresql://app:app@localhost:5432/shortlink"],
text=True).strip()
if out != "1":
print("❌ links(code) 缺少唯一索引 —— 性能会退化约 20 倍")
print(" 修复:CREATE UNIQUE INDEX CONCURRENTLY idx_links_code ON links(code);")
sys.exit(1)
print("✅ links(code) 唯一索引存在")
PY
二、比较脚本(门禁的核心)
#!/usr/bin/env python3
"""tools/compare-benchmark.py —— 与基线比较,判定是否回归
关键设计:
• 阈值来自【实测的噪声底线】,不是拍脑袋的 5%
• 分别报告「变慢」与「变快」(变快也要看,可能是测试写错了)
• 支持多个指标,任一越界即失败
"""
import argparse
import json
import pathlib
import sys
def extract(path: pathlib.Path) -> dict:
"""从 JMH 或 k6 的 summary 里提取「指标名 → 值」"""
raw = json.loads(path.read_text())
# ── k6 summary 格式 ──
if "metrics" in raw and "http_req_duration" in raw.get("metrics", {}):
m = raw["metrics"]["http_req_duration"]
return {
"p50": m["p(50)"],
"p95": m["p(95)"],
"p99": m["p(99)"],
"mean": m["avg"],
}
# ── JMH results 格式 ──
if isinstance(raw, list):
out = {}
for e in raw:
name = f"{e['benchmark'].split('.')[-1]}.{e['mode']}"
# Score 是平均值,Score Error 是误差
out[name] = e["primaryMetric"]["score"]
return out
raise SystemExit(f"❌ 无法识别 {path} 的格式")
def main() -> int:
ap = argparse.ArgumentParser()
ap.add_argument("--baseline", required=True)
ap.add_argument("--current", required=True)
ap.add_argument("--threshold-pct", type=float, required=True,
help="回归阈值(%%)—— 应来自实测噪声底线 × 1.5~2")
ap.add_argument("--fail-on-regression", action="store_true")
ap.add_argument("--assume-lower-is-better", action="store_true", default=True)
args = ap.parse_args()
base = extract(pathlib.Path(args.baseline))
cur = extract(pathlib.Path(args.current))
common = [k for k in base if k in cur]
if not common:
print("❌ 基线与当前没有共同指标 —— 检查是否改了指标名")
return 2
print(f"═══ 性能对比(阈值 ±{args.threshold_pct:.1f}%)═══")
print()
print(f"{'指标':<32s} {'基线':>12s} {'当前':>12s} {'变化':>10s} 判定")
print("─" * 82)
regressions, improvements = [], []
for k in sorted(common):
b, c = base[k], cur[k]
if b == 0:
continue
delta = (c - b) / b * 100
if delta > args.threshold_pct:
verdict = "❌ 回归"
regressions.append((k, delta))
elif delta < -args.threshold_pct:
verdict = "🎉 改善"
improvements.append((k, delta))
else:
verdict = "✅ 通过"
print(f"{k:<32s} {b:12.4f} {c:12.4f} {delta:+9.1f}% {verdict}")
print()
if improvements:
print(f"改善 {len(improvements)} 项:")
for k, d in improvements:
print(f" {k}: {d:+.1f}%")
print(" ⚠️ 大幅改善也要检查:是不是测试写错了 / 工作量变少了?")
if regressions:
print(f"回归 {len(regressions)} 项:")
for k, d in regressions:
print(f" {k}: {d:+.1f}%")
if not regressions and not improvements:
print("所有指标都在阈值内 ✅")
if regressions and args.fail_on_regression:
print()
print("❌ 存在性能回归 —— 合并被阻止")
print()
print("处理方式(按优先级):")
print(" 1. 是本 PR 引入的 → 修掉")
print(" 2. 不是本 PR → 检查环境/基线是否需要更新")
print(" 3. 是有意的权衡 → 在 PR 描述里说明,并让 reviewer 显式批准")
return 1
print()
print("✅ 门禁通过")
return 0
if __name__ == "__main__":
sys.exit(main())
三、告警规则(含本项目特有的那条)
# ops/prometheus/shortlink-alerts.yml
groups:
- name: shortlink-slo
interval: 30s
rules:
# ══════════════════════════════════════════
# ① SLO 燃烧率告警(通用形态)
# ══════════════════════════════════════════
- alert: LatencyBudgetBurnFast
# 5 分钟窗口内燃烧率 > 14.4 → 2 天内烧完 28 天预算
expr: |
(
sum(rate(app_request_duration_seconds_bucket{le="0.1"}[5m]))
/
sum(rate(app_request_duration_seconds_count[5m]))
) < 0.99
and
(
sum(rate(app_request_duration_seconds_bucket{le="0.1"}[1h]))
/
sum(rate(app_request_duration_seconds_count[1h]))
) < 0.99
for: 2m
labels: { severity: critical, team: backend }
annotations:
summary: "P99 延迟正在快速消耗错误预算"
runbook: "https://wiki/runbook/latency-burn"
# ══════════════════════════════════════════
# ② ⭐ 本项目特有:慢查询占比
# 为什么加这一条:
# 索引是这次优化的核心。一旦索引失效(被误删、
# DDL 回滚、统计信息过期),性能会从 19ms 退化到 420ms。
# 通用的延迟告警也会响,但会晚 2~5 分钟。
# 这条能在 30 秒内直接指出【根因】。
# → 这就是"从实验里学到的知识,转化为告警"
# ══════════════════════════════════════════
- alert: SlowQueryRatioHigh
expr: |
(
sum(rate(app_db_query_seconds_bucket{le="0.05"}[2m]))
/
sum(rate(app_db_query_seconds_count[2m]))
) < 0.95
for: 1m
labels: { severity: critical, team: backend }
annotations:
summary: "超过 5% 的 DB 查询慢于 50ms(预期 < 0.1%)"
description: |
最可能的原因:links(code) 索引失效或缺失。
排查:
EXPLAIN (ANALYZE) SELECT ... FROM links WHERE code = 'x';
期望看到 Index Scan,若为 Seq Scan 则索引已失效。
修复:
CREATE UNIQUE INDEX CONCURRENTLY idx_links_code ON links(code);
ANALYZE links;
runbook: "https://wiki/runbook/slow-query-ratio"
# ══════════════════════════════════════════
# ③ 连接池等待(注意:不是"池满就告警")
# ══════════════════════════════════════════
- alert: DbPoolWaitingSustained
# 关键:必须【持续】等待才告警
# 瞬时 pending > 0 是正常的(突发流量)
# 持续 > 0 才说明"连接被长期占用"
expr: db_pool_pending > 0
for: 3m
labels: { severity: warning, team: backend }
annotations:
summary: "连接池持续等待 3 分钟"
description: |
⚠️ 不要直接加池大小!
先回答:为什么每个请求占用连接这么久?
检查 app_db_query_seconds 的 P99。
如果慢查询是根因,加池只会让更多慢查询并发去抢数据库。
# ══════════════════════════════════════════
# ④ 调度器污染(本次实验发现的系统性问题)
# ══════════════════════════════════════════
- alert: SchedulerPollution
# RUNNABLE 线程数 ≈ CPU 核数,且 CPU 不高
# → 典型的"阻塞调用占满 Default 调度器"
expr: |
jvm_threads_states_threads{state="runnable"} >= 8
and
process_cpu_usage < 0.5
for: 2m
labels: { severity: warning, team: backend }
annotations:
summary: "RUNNABLE 线程数接近核数但 CPU 不高 —— 疑似阻塞调用占满调度器"
description: |
排查:是否有 JDBC/文件 IO 调用没有 withContext(Dispatchers.IO)?
验证:看 /health 的 P99 是否也变慢了(如果变慢,说明全局被拖累)。
# ══════════════════════════════════════════
# ⑤ 缓存命中率(埋雷 ③ 的哨兵)
# ══════════════════════════════════════════
- alert: CacheHitRateDrop
expr: app_cache_hit_rate < 0.70
for: 5m
labels: { severity: warning, team: backend }
annotations:
summary: "缓存命中率低于 70%(基线 75%)"
description: |
可能原因:
• 缓存实例被重建(发布后冷启动)
• 热点分布发生变化
• TTL 抖动参数被改得太激进
为什么 ③ 的 for: 3m 很关键:
瞬时 pending > 0 → 正常(突发流量下必然会短暂排队)
持续 3 分钟 pending > 0 → 异常(有东西长期占着连接)
如果不加 `for`:
→ 每次流量小波动都告警
→ 告警疲劳 → 真问题被忽略(第 8.5 节的"告警有效率")
四、容量表
#!/usr/bin/env python3
"""tools/step8a-capacity-table.py —— 从容量曲线 + 峰值需求算出副本数
这就是"把性能数字翻译成成本数字"的脚本 ——
业务方看不懂 P99,但看得懂"要几台机器、多少钱"。
"""
import json
import pathlib
import sys
# ── 手填:拐点(来自步骤四/步骤七的容量曲线)──
CAPACITY_INPUT = pathlib.Path("perf/results/capacity-input.json")
TEMPLATE = {
"service": "shortlink",
"peak_rps": 3000,
"headroom_pct": 40, # 为突发/故障留的余量
"versions": [
{"name": "优化前", "instance": "4C8G", "knee_rps": 600,
"evidence": "E10-capacity"},
{"name": "优化后", "instance": "4C8G", "knee_rps": 1800,
"evidence": "E15-capacity-retest"},
],
}
def main() -> int:
if CAPACITY_INPUT.exists():
cfg = json.loads(CAPACITY_INPUT.read_text())
else:
cfg = TEMPLATE
CAPACITY_INPUT.parent.mkdir(parents=True, exist_ok=True)
CAPACITY_INPUT.write_text(json.dumps(cfg, indent=2, ensure_ascii=False))
print(f"(未找到输入,已生成模板 {CAPACITY_INPUT},请填入实测拐点)\n")
peak = cfg["peak_rps"]
headroom = cfg["headroom_pct"] / 100
print(f"═══ 容量计算 ═══")
print(f"峰值需求:{peak} RPS")
print(f"headroom:{cfg['headroom_pct']}%(为突发与故障预留)")
print()
rows = []
for v in cfg["versions"]:
knee = v["knee_rps"]
safe = knee * (1 - headroom)
replicas = -(-peak // int(safe)) # 向上取整
# 加 1 台冗余(任一实例故障时仍能满足峰值)
total = int(replicas) + 1
rows.append({
"version": v["name"],
"instance": v["instance"],
"knee": knee,
"safe": safe,
"demand": peak,
"replicas": int(replicas),
"total": total,
"evidence": v["evidence"],
})
print(f"{'版本':<10s} {'规格':<8s} {'拐点':>8s} {'headroom后':>11s} "
f"{'峰值需求':>9s} {'副本':>6s} {'含冗余':>7s} 数据来源")
print("─" * 92)
for r in rows:
print(f"{r['version']:<10s} {r['instance']:<8s} {r['knee']:>7d}R "
f"{r['safe']:>10.0f}R {r['demand']:>8d}R "
f"{r['replicas']:>6d} {r['total']:>7d} {r['evidence']}")
if len(rows) >= 2:
before, after = rows[0], rows[-1]
gain = after["knee"] / before["knee"]
cost_drop = (1 - after["total"] / before["total"]) * 100
print()
print(f"容量提升:拐点 {before['knee']} → {after['knee']} RPS({gain:.1f} 倍)")
print(f"副本变化:{before['total']} → {after['total']} 台"
f"(成本 {cost_drop:+.0f}%)")
print()
print("⭐ 这一行是报告里最有说服力的:")
print(f" 业务方看不懂「P99 从 420ms 降到 19ms」,")
print(f" 但看得懂「从 9 台降到 4 台,成本降 55%」")
# ── 生成 markdown ──
md = [
"# 容量表",
"",
"> 最后更新:YYYY-MM-DD | 下次复测:YYYY-MM-DD(建议每季度)",
"",
f"## 服务:{cfg['service']}",
"",
"| 版本 | 实例规格 | 拐点(实测) | headroom | 安全容量 | 峰值需求 | 所需副本 | 数据来源 |",
"| --- | --- | --- | --- | --- | --- | --- | --- |",
]
for r in rows:
md.append(
f"| {r['version']} | {r['instance']} | ~{r['knee']} RPS | "
f"{cfg['headroom_pct']}% | {r['safe']:.0f} RPS | {r['demand']} | "
f"**{r['replicas']}+1={r['total']}** | {r['evidence']} |"
)
if len(rows) >= 2:
md += ["", f"**容量提升**:拐点 {rows[0]['knee']} → {rows[-1]['knee']} RPS,"
f"所需副本 {rows[0]['total']} → {rows[-1]['total']} 台"]
md += [
"",
"## 复测规则",
"",
"| 触发条件 | 动作 |",
"| --- | --- |",
"| 数据量增长 50% | 重跑容量曲线 |",
"| 依赖版本升级(DB/Redis/JVM) | 重跑容量曲线 |",
"| 季度到期 | 重跑容量曲线 |",
"| 出现 P99 回归告警 | 先查索引与慢查询,再决定是否重测 |",
]
out = pathlib.Path("docs/capacity.md")
out.parent.mkdir(parents=True, exist_ok=True)
out.write_text("\n".join(md) + "\n")
print()
print(f"✅ 容量表 → {out}")
return 0
if __name__ == "__main__":
sys.exit(main())
预期输出:
═══ 容量计算 ═══
峰值需求:3000 RPS
headroom:40%(为突发与故障预留)
版本 规格 拐点 headroom后 峰值需求 副本 含冗余 数据来源
────────────────────────────────────────────────────────────────────────────
优化前 4C8G 600R 360R 3000R 9 10 E10-capacity
优化后 4C8G 1800R 1080R 3000R 3 4 E15-capacity-retest
容量提升:拐点 600 → 1800 RPS(3.0 倍)
副本变化:10 → 4 台(成本 -60%)
⭐ 这一行是报告里最有说服力的:
业务方看不懂「P99 从 420ms 降到 19ms」,
但看得懂「从 9 台降到 4 台,成本降 55%」
五、报告生成器
#!/usr/bin/env python3
"""tools/step8b-report.py —— 从各步骤的产出物自动组装 PERF-REPORT.md
为什么自动化:
报告里 90% 的数字都能从产出物里读出来。
手工抄写 → 会写错、会忘记更新、会和原始数据不一致。
自动生成 → 数字永远与数据一致(人工只填【判断】部分)。
"""
import json
import pathlib
import sys
def read_json(p, default=None):
p = pathlib.Path(p)
if not p.exists():
return default
return json.loads(p.read_text())
def main() -> int:
root = pathlib.Path("perf/results")
slo = read_json("perf/slo.yaml", {}) # 实际是 yaml,这里只做存在性检查
noise = read_json(root / "step3-baseline/noise-floor.json", {})
hyps = read_json(root / "step5-diagnose/hypotheses.json", {"hypotheses": []})
plan = read_json(root / "step6-optimize/optimization-plan.json", {})
def k6p99(phase):
p = root / f"step6-optimize/opt-1/summary-{phase}.json"
if not p.exists():
return None
return json.loads(p.read_text())["metrics"]["http_req_duration"]["p(99)"]
a, b = k6p99("A"), k6p99("B")
baseline_p99 = a
final_p99 = read_json(root / "step7-soak/summary.json", {}) \
.get("metrics", {}).get("http_req_duration", {}).get("p(99)")
mdd = noise.get("mdd_pct", "?")
confirmed = [h for h in hyps.get("hypotheses", []) if h["status"] == "confirmed"]
rejected = [h for h in hyps.get("hypotheses", []) if h["status"] == "rejected"]
rej_plan = plan.get("rejected", [])
lines = [
"# 短链服务性能评估报告",
"",
"> 版本:v1.0 | 日期:YYYY-MM-DD | 作者:@某某",
"> 关联实验:E09–E15 | 代码版本:commit `abc1234`",
"",
"---",
"",
"## 一、摘要(给决策者)",
"",
]
if baseline_p99 and final_p99:
drop = (final_p99 - baseline_p99) / baseline_p99 * 100
lines += [
f"**一句话结论**:通过给 `code` 列加索引并隔离阻塞调用,",
f"短链服务的 P99 从 **{baseline_p99:.0f} ms 降至 {final_p99:.1f} ms"
f"({drop:+.0f}%)**,",
"拐点从 **600 RPS 提升到 1800 RPS(3 倍)**,",
"按同样峰值(3000 RPS)计算,所需副本从 **9 台降到 4 台(成本 -55%)**。",
]
else:
lines.append("(请填入:一句话结论 —— 数字 + 成本影响)")
lines += [
"",
"**投入**:1 名工程师 × 2 天",
"**风险**:新增一个索引(占用 45 MB 磁盘;写入开销 +3.5%,在噪声范围内)",
"",
"**为什么值得做**:单次投入 2 人天,每月省 5 台 4C8G 实例。",
"",
"---",
"",
"## 二、目标与前提",
"",
"### SLO(详见 `perf/slo.yaml`)",
"",
"| SLI | 目标 | 窗口 |",
"| --- | --- | --- |",
"| 跳转 P99 | < 100 ms | 28d |",
"| 创建 P99 | < 200 ms | 28d |",
"| 可用性 | ≥ 99.9% | 28d |",
"",
f"### 噪声底线",
"",
f"- 测量轮次:{noise.get('rounds', '?')} 轮",
f"- P99 噪声底线:±{noise.get('noise_pct', '?')}%",
f"- **MDD:±{mdd}%**(判定「有无变化」的门槛)",
"",
"> ⚠️ 本报告所有「显著」结论都以此为判据。",
"",
"---",
"",
"## 三、环境与方法",
"",
"| 项 | 配置 |",
"| --- | --- |",
"| 服务 | 4C8G,JVM 21,G1GC |",
"| 数据库 | PostgreSQL 16,4C8G,shared_buffers=2GB |",
"| 数据量 | 100 万行,访问分布 zipf(α=1.2) |",
"| 压测 | k6(ramping-arrival-rate),3 轮取中位数 |",
"",
"**方法**:单变量 + A/B/A' 回滚验证(第 5.1 节)",
"",
"---",
"",
"## 四、基线数据",
"",
]
if baseline_p99:
lines += [
f"优化前 P99(500 RPS):**{baseline_p99:.1f} ms**(SLO 100ms,超标 4.2 倍)",
"",
]
lines += [
"| 到达率 | P50 | P99 | 增长率 | 判断 |",
"| --- | --- | --- | --- | --- |",
"| 200 | 12.0 | 180.0 | 1.00 | 基线 |",
"| 400 | 12.5 | 310.0 | 0.86 | |",
"| **700** | **18.2** | **620.0** | **1.72** | **← 拐点** |",
"| 1000 | 45.0 | 1200.0 | 3.33 | 排队区 |",
"| 2000 | 420.0 | 5200.0 | 23.1 | 接近崩溃 |",
"",
"**拐点:约 600 RPS**(SLO 要求 3000,差 5 倍)",
"",
"---",
"",
"## 五、瓶颈分析",
"",
]
for i, h in enumerate(confirmed, 1):
lines += [
f"### 瓶颈 {i}:{h['hypothesis']}",
"",
f"**证据**:{h['evidence_for']}",
"",
f"**验证手段**:{h['verify_by']}",
"",
f"**结论**:{h['conclusion']}",
"",
]
lines += [
"### 被排除的假设 ⭐",
"",
"| 假设 | 排除依据 |",
"| --- | --- |",
]
for h in rejected:
lines.append(f"| {h['hypothesis']} | {h['evidence_against']} |")
lines += [
"",
"> **这一节比「确认了什么」更重要** —— 它证明没有漏掉其他可能,",
"> 也省下了「去优化 CPU」和「去调 GC」的浪费。",
"",
"---",
"",
"## 六、优化措施",
"",
"| # | 措施 | 层级 | 收益 | 代价 | 采纳 |",
"| --- | --- | --- | --- | --- | --- |",
]
for p in plan.get("plan", []):
lines.append(
f"| {p['order']} | {p['name']} | {p['layer']} | {p['expected']} | "
f"{p['risk']} | ✅ |"
)
for r in rej_plan:
lines.append(
f"| — | {r['name']} | {r['layer']} | {r['measured_gain']} | "
f"{r['extra_cost']} | ❌ **否决** |"
)
lines += ["", "### 否决记录", ""]
for r in rej_plan:
lines += [
f"- **{r['name']}**:{r['reason']}",
f" - 重新评估条件:{r['revisit_when']}",
]
lines += [
"",
"---",
"",
"## 七、验证",
"",
]
if a and b:
lines += [
f"| 指标 | 优化前 | 优化后 | 变化 | 显著性 |",
f"| --- | --- | --- | --- | --- |",
f"| P99 | {a:.1f} ms | {b:.1f} ms | "
f"{(b-a)/a*100:+.1f}% | > MDD({mdd}%) ✅ |",
"",
]
lines += [
"**与噪声底线对比**:改善幅度是 MDD 的 8 倍,结论可信。",
"",
"### 长稳(1 小时浸泡)",
"",
"| 指标 | 结果 |",
"| --- | --- |",
"| 堆 GC 后基线 | 稳定(+2.9 MB/小时) |",
"| 连接池 active/pending | 稳定(6 / 0) |",
"| 线程数 / FD | 稳定(42 / 313) |",
"| P99 | 稳定(18.1 → 19.4 ms) |",
"| **缓存雪崩** | **发现并修复**(TTL 抖动) |",
"",
"---",
"",
"## 八、风险与未验证的场景",
"",
"| 项 | 说明 |",
"| --- | --- |",
"| 压测客户端与服务同机 | 可用于相对比较,**不能用于绝对容量规划** |",
"| 未测冷启动路径 | 缓存为空时的表现(发布期风险) |",
"| 100 万行数据 | 数据量增长 3 倍后拐点会下降约 8% |",
"| 单实例测试 | 多实例下的连接池总量需重新核算 |",
"",
"---",
"",
"## 九、结论与建议",
"",
"### 结论",
"",
"1. 性能瓶颈的**根因是缺失索引**(贡献 90%),不是 CPU、不是 GC、不是池大小",
"2. 修复后 **P99 420ms → 19ms**,拐点 **600 → 1800 RPS**",
"3. 需要 **4 台**而非 9 台,**月成本降 55%**",
"",
"### 建议",
"",
"| # | 建议 | 优先级 |",
"| --- | --- | --- |",
"| 1 | 为 `links(code)` 建唯一索引并加 CI 检查 | P0 |",
"| 2 | 加 `SlowQueryRatioHigh` 告警(索引失效的哨兵) | P0 |",
"| 3 | JDBC 调用统一 `withContext(Dispatchers.IO)` + 限流 | P1 |",
"| 4 | 缓存 TTL 加抖动 | P1 |",
"| 5 | 下季度复测容量(数据量增长) | P2 |",
"",
"### 一句话总结",
"",
"> **90% 的收益来自一条 DDL** —— 这才是性能工程的常态:",
"> 找到那个真正的大头,而不是在细枝末节上反复调参。",
"",
]
out = pathlib.Path("docs/PERF-REPORT.md")
out.parent.mkdir(parents=True, exist_ok=True)
out.write_text("\n".join(lines) + "\n")
print("✅ 报告已生成 →", out)
print()
print("自动填入的部分:")
print(" • 噪声底线 / MDD")
print(" • 确认的瓶颈 + 被排除的假设(来自 hypotheses.json)")
print(" • 优化措施与被否决方案(来自 optimization-plan.json)")
print(" • 优化前后的 P99(来自 k6 summary)")
print()
print("需要人工填写的部分:")
print(" • 摘要里的「为什么值得做」")
print(" • 风险与未验证场景(脚本不知道你漏测了什么)")
print(" • 所有主观判断与建议")
return 0
if __name__ == "__main__":
sys.exit(main())
六、八步编排脚本
#!/usr/bin/env bash
# tools/capstone.sh —— 第 9 章八步的一键编排
#
# 设计原则:
# • 每一步都能【单独跑】(便于调试与复跑)
# • 每一步都【检查前置产出物】(防止跳过)
# • 每一步都【打印下一步提示】
# • 绝不自动"帮你判断"——判断必须由人做
set -uo pipefail
CMD="${1:-help}"
ROOT="perf/results"
need() {
# 检查前置产出物是否存在 —— 这就是"流程纪律"的代码化
local f="$1" why="$2"
if [ ! -e "$f" ]; then
echo "❌ 缺少前置产出物:$f"
echo " 为什么必须:$why"
exit 1
fi
}
case "$CMD" in
1)
echo "═══ 步骤一:定义目标 ═══"
echo
echo "手工任务(无法自动化的部分):"
echo " 1. 写 perf/slo.yaml(SLI / SLO / 延迟预算 / 排除项)"
echo " 2. 想清楚:哪些失败【不算】错误?"
echo
if [ ! -f perf/slo.yaml ]; then
echo "⚠️ perf/slo.yaml 不存在"
echo " 参考:docs/code/09-capstone/02-step1-goals.md"
else
python3 tools/check-goals.py perf/slo.yaml || exit 1
echo
python3 tools/validate-budget.py perf/slo.yaml || exit 1
fi
echo
echo "→ 下一步:tools/capstone.sh 2"
;;
2)
echo "═══ 步骤二:搭服务与数据 ═══"
need perf/slo.yaml "没有 SLO,后面的「优化成功」无法定义"
docker compose -f perf/docker-compose.yaml up -d
echo "等待数据库就绪..."
for i in $(seq 1 60); do
docker compose -f perf/docker-compose.yaml exec -T postgres \
pg_isready -U app -d shortlink >/dev/null 2>&1 && break
sleep 1
done
python3 tools/seed.py --rows "${ROWS:-1000000}"
echo
bash tools/verify-traps.sh
echo
echo "→ 下一步:启动服务(./gradlew run),然后 tools/capstone.sh 3"
;;
3)
echo "═══ 步骤三:基线与噪声底线 ═══"
need perf/slo.yaml "没有 SLO,不知道要测什么"
bash tools/step3a-verify-observability.sh || {
echo
echo "❌ 观测回路不可信 —— 【不要】开始测性能"
exit 1
}
echo
for r in $(seq 1 "${ROUNDS:-10}"); do
bash tools/step3b-baseline-one.sh "$r" || exit 1
done
echo
python3 tools/step3c-noise-floor.py "$ROOT/step3-baseline"
echo
python3 tools/pick-baseline.py "$ROOT/step3-baseline"
echo
echo "→ 下一步:tools/capstone.sh 4"
;;
4)
echo "═══ 步骤四:容量曲线 ═══"
need "$ROOT/step3-baseline/noise-floor.json" "没有噪声底线,无法解释曲线的波动"
bash tools/step4a-capacity.sh
python3 tools/step4b-find-knee.py "$ROOT/step4-capacity"
python3 tools/step4c-plot.py "$ROOT/step4-capacity" || true
echo
echo "→ 请人工确认拐点数值,然后 tools/capstone.sh 5"
;;
5)
echo "═══ 步骤五:剖析定位 ═══"
need "$ROOT/step4-capacity/capacity.md" "没有容量曲线,不知道在什么负载下取证"
[ -n "${APP_PID:-}" ] || { echo "❌ 请设置 APP_PID"; exit 1; }
bash tools/step5a-collect-at-knee.sh
python3 tools/step5b-analyze-flame.py "$ROOT/step5-diagnose"
bash tools/step5c-diagnose-db.sh
python3 tools/hypotheses.py "$ROOT/step5-diagnose"
echo
echo "⚠️ 请【人工】填写 bottlenecks.md 的贡献占比"
echo " 脚本能给证据,但「占比多少」需要人判断"
echo
echo "→ 下一步:tools/capstone.sh 6"
;;
6)
echo "═══ 步骤六:优化与验证 ═══"
need "$ROOT/step5-diagnose/bottlenecks.md" "没有瓶颈清单,优化就是瞎猜"
python3 tools/step6a-plan.py "$ROOT/step6-optimize"
echo
echo "现在按计划逐个实施优化。每一项都要:"
echo " 1. 单独实施(单变量)"
echo " 2. 跑 tools/step6b-verify.sh <编号>"
echo " 3. 回答三问(提升多少 / 代价 / 回归验证)"
echo
echo "→ 全部完成后:tools/capstone.sh 7"
;;
7)
echo "═══ 步骤七:长稳与回归 ═══"
need "$ROOT/step6-optimize/optimization-plan.json" "没有优化计划,不知道长稳要验证什么"
bash tools/step7a-soak.sh
python3 tools/step7b-analyze-soak.py "$ROOT/step7-soak"
bash tools/step7c-regression.sh
echo
echo "→ 下一步:tools/capstone.sh 8"
;;
8)
echo "═══ 步骤八:固化门禁与交付报告 ═══"
need "$ROOT/step7-soak/soak-report.md" "没有长稳结论,报告不完整"
python3 tools/step8a-capacity-table.py
python3 tools/step8b-report.py
echo
echo "还需人工完成:"
echo " 1. 更新基线:cp build/reports/jmh/results.json perf/baselines/"
echo " 2. 填写报告里的「风险与未验证场景」"
echo " 3. 提交 PERF-REPORT.md 与基线文件"
echo
echo "⭐ 交付物清单:"
echo " [ ] perf/slo.yaml (目标)"
echo " [ ] tools/capstone.sh (八步编排)"
echo " [ ] ops/prometheus/*.yml (门禁 + 告警)"
echo " [ ] docs/capacity.md (容量表)"
echo " [ ] docs/PERF-REPORT.md (性能报告)"
echo " [ ] perf/baselines/*.json (复测基准)"
;;
status)
echo "═══ 八步进度 ═══"
echo
check() {
local f="$1" name="$2"
if [ -e "$f" ]; then echo " ✅ $name"; else echo " ⬜ $name"; fi
}
check perf/slo.yaml "① 目标(slo.yaml)"
check perf/data/hot-codes.txt "② 数据(seed 完成)"
check "$ROOT/step3-baseline/noise-floor.json" "③ 噪声底线"
check "$ROOT/step4-capacity/capacity.md" "④ 容量曲线"
check "$ROOT/step5-diagnose/bottlenecks.md" "⑤ 瓶颈清单"
check "$ROOT/step6-optimize/optimization-plan.json" "⑥ 优化计划"
check "$ROOT/step7-soak/soak-report.md" "⑦ 长稳报告"
check docs/PERF-REPORT.md "⑧ 性能报告"
echo
echo "全部 ✅ 后,交付物齐全(见 tools/capstone.sh 8 的清单)"
;;
*)
cat <<'USAGE'
用法:tools/capstone.sh <命令>
命令:
1 步骤一:定义目标(SLO + 延迟预算)
2 步骤二:搭服务与造数据
3 步骤三:基线与噪声底线
4 步骤四:容量曲线与拐点
5 步骤五:剖析定位
6 步骤六:优化与验证
7 步骤七:长稳与回归
8 步骤八:固化门禁与交付报告
status 查看八步进度
示例:
tools/capstone.sh status
tools/capstone.sh 3 ROUNDS=10
USAGE
;;
esac
七、动手改造
| 改动 | 观察什么 |
|---|---|
| 把门禁阈值从 8.9% 改成 2% | CI 频繁失败在噪声波动上——阈值必须来自实测噪声 |
| 去掉 CI 里的"检查索引存在" | 某次 DDL 回滚后性能退化 20 倍,CI 却全绿——关键结构必须门禁化 |
把 SlowQueryRatioHigh 的 for 改成 0s |
每次慢查询抖动都告警——告警疲劳 |
把 DbPoolWaitingSustained 的说明删掉 |
有人看到告警就去加池大小——runbook 是告警的一半 |
| 手工写报告而不自动生成 | 报告里的数字与原始数据不一致——自动生成保证一致性 |
删掉 capstone.sh 里的 need 检查 |
跳过步骤三直接跑步骤六,没有 before 可比——纪律要代码化 |
八、这段代码的局限
capstone.sh的步骤 5/6 需要人工介入:判断与改代码无法自动化(这是性能工程的本质,不是脚本写得不好)。compare-benchmark.py的阈值是单一数字:没有做"分位数各自设阈值"或"考虑方差的假设检验"——生产可接入benchmark-action之类的工具。- 告警规则里的 PromQL 未在本 Lab 验证:真实环境的指标名可能不同(尤其
jvm_threads_states_threads依赖 micrometer 版本)。 step8b-report.py里读了perf/slo.yaml但没解析(只做存在性检查):完整的实现需要pyyaml。- 容量表用"公式算副本数":真实决策还要考虑故障域、跨可用区、成本单价、弹性伸缩——这里是简化。
- 没有实现「自动更新基线」:脚本只提示
cp,生产应有一个需要 reviewer 批准的流程(第 8.2 节强调:基线变更必须走 PR)。 - 本文件刻意不提供"全自动一键到底":因为跳过人的判断,这个 Lab 就失去了全部教学意义。