混沌测试从入门到精通1

糖果店的幽灵2026-07-09 8:37

🌀 混沌测试从入门到精通

Head First 风格 --- 让你像玩一样学会混沌工程

🎯 欢迎来到混沌测试的世界！

想象一下这个场景：

你是一个大型在线购物网站的运维工程师。今天是大促日，流量是平时的 10 倍。突然，一台服务器宕机了 。

你的第一反应是什么？

A. 惊慌失措，疯狂敲键盘

B. 冷静地拿起咖啡，喝一口，然后优雅地处理它

C. 打电话给经理："系统挂了！"

如果你选 B，恭喜你------你已经有了混沌工程师的潜质。

但问题是：你怎么知道系统在真正的灾难面前会优雅地响应？

答案是：你主动制造灾难，看看系统会不会优雅地响应。

这就是混沌测试（Chaos Testing） 的本质。

📖 这本书怎么读？

复制代码 
┌─────────────────────────────────────────┐
│  这不是一本普通的技术书                │
│  ─────────────────────────             │
│  ✓ 有对话、有故事、有漫画（用文字）    │
│  ✓ 有"头脑风暴"练习                   │
│  ✓ 有"你在这儿"的导航                  │
│  ✓ 有代码可以自己动手做               │
│  ✓ 有图！大量 mermaid 图              │
│  └─────────────────────────────────────────┘

每章结构：

  1. 🧠 概念引入 -- 用故事或比喻讲清核心概念
  2. 📊 图解原理 -- Mermaid 图把抽象变直观
  3. 💻 动手 Demo -- 可运行的代码/命令
  4. 🧪 思考练习 -- 巩固你的理解
  5. ✅ 本章要点 -- 快速回顾

🧭 学习路线图

#mermaid-svg-Jg65XyvxT3I62KbL{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-Jg65XyvxT3I62KbL .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-Jg65XyvxT3I62KbL .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-Jg65XyvxT3I62KbL .error-icon{fill:#552222;}#mermaid-svg-Jg65XyvxT3I62KbL .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-Jg65XyvxT3I62KbL .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-Jg65XyvxT3I62KbL .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-Jg65XyvxT3I62KbL .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-Jg65XyvxT3I62KbL .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-Jg65XyvxT3I62KbL .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-Jg65XyvxT3I62KbL .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-Jg65XyvxT3I62KbL .marker{fill:#333333;stroke:#333333;}#mermaid-svg-Jg65XyvxT3I62KbL .marker.cross{stroke:#333333;}#mermaid-svg-Jg65XyvxT3I62KbL svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-Jg65XyvxT3I62KbL p{margin:0;}#mermaid-svg-Jg65XyvxT3I62KbL .edge{stroke-width:3;}#mermaid-svg-Jg65XyvxT3I62KbL .section--1 rect,#mermaid-svg-Jg65XyvxT3I62KbL .section--1 path,#mermaid-svg-Jg65XyvxT3I62KbL .section--1 circle,#mermaid-svg-Jg65XyvxT3I62KbL .section--1 polygon,#mermaid-svg-Jg65XyvxT3I62KbL .section--1 path{fill:hsl(240, 100%, 76.2745098039%);}#mermaid-svg-Jg65XyvxT3I62KbL .section--1 text{fill:#ffffff;}#mermaid-svg-Jg65XyvxT3I62KbL .node-icon--1{font-size:40px;color:#ffffff;}#mermaid-svg-Jg65XyvxT3I62KbL .section-edge--1{stroke:hsl(240, 100%, 76.2745098039%);}#mermaid-svg-Jg65XyvxT3I62KbL .edge-depth--1{stroke-width:17;}#mermaid-svg-Jg65XyvxT3I62KbL .section--1 line{stroke:hsl(60, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-Jg65XyvxT3I62KbL .disabled,#mermaid-svg-Jg65XyvxT3I62KbL .disabled circle,#mermaid-svg-Jg65XyvxT3I62KbL .disabled text{fill:lightgray;}#mermaid-svg-Jg65XyvxT3I62KbL .disabled text{fill:#efefef;}#mermaid-svg-Jg65XyvxT3I62KbL .section-0 rect,#mermaid-svg-Jg65XyvxT3I62KbL .section-0 path,#mermaid-svg-Jg65XyvxT3I62KbL .section-0 circle,#mermaid-svg-Jg65XyvxT3I62KbL .section-0 polygon,#mermaid-svg-Jg65XyvxT3I62KbL .section-0 path{fill:hsl(60, 100%, 73.5294117647%);}#mermaid-svg-Jg65XyvxT3I62KbL .section-0 text{fill:black;}#mermaid-svg-Jg65XyvxT3I62KbL .node-icon-0{font-size:40px;color:black;}#mermaid-svg-Jg65XyvxT3I62KbL .section-edge-0{stroke:hsl(60, 100%, 73.5294117647%);}#mermaid-svg-Jg65XyvxT3I62KbL .edge-depth-0{stroke-width:14;}#mermaid-svg-Jg65XyvxT3I62KbL .section-0 line{stroke:hsl(240, 100%, 83.5294117647%);stroke-width:3;}#mermaid-svg-Jg65XyvxT3I62KbL .disabled,#mermaid-svg-Jg65XyvxT3I62KbL .disabled circle,#mermaid-svg-Jg65XyvxT3I62KbL .disabled text{fill:lightgray;}#mermaid-svg-Jg65XyvxT3I62KbL .disabled text{fill:#efefef;}#mermaid-svg-Jg65XyvxT3I62KbL .section-1 rect,#mermaid-svg-Jg65XyvxT3I62KbL .section-1 path,#mermaid-svg-Jg65XyvxT3I62KbL .section-1 circle,#mermaid-svg-Jg65XyvxT3I62KbL .section-1 polygon,#mermaid-svg-Jg65XyvxT3I62KbL .section-1 path{fill:hsl(80, 100%, 76.2745098039%);}#mermaid-svg-Jg65XyvxT3I62KbL .section-1 text{fill:black;}#mermaid-svg-Jg65XyvxT3I62KbL .node-icon-1{font-size:40px;color:black;}#mermaid-svg-Jg65XyvxT3I62KbL .section-edge-1{stroke:hsl(80, 100%, 76.2745098039%);}#mermaid-svg-Jg65XyvxT3I62KbL .edge-depth-1{stroke-width:11;}#mermaid-svg-Jg65XyvxT3I62KbL .section-1 line{stroke:hsl(260, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-Jg65XyvxT3I62KbL .disabled,#mermaid-svg-Jg65XyvxT3I62KbL .disabled circle,#mermaid-svg-Jg65XyvxT3I62KbL .disabled text{fill:lightgray;}#mermaid-svg-Jg65XyvxT3I62KbL .disabled text{fill:#efefef;}#mermaid-svg-Jg65XyvxT3I62KbL .section-2 rect,#mermaid-svg-Jg65XyvxT3I62KbL .section-2 path,#mermaid-svg-Jg65XyvxT3I62KbL .section-2 circle,#mermaid-svg-Jg65XyvxT3I62KbL .section-2 polygon,#mermaid-svg-Jg65XyvxT3I62KbL .section-2 path{fill:hsl(270, 100%, 76.2745098039%);}#mermaid-svg-Jg65XyvxT3I62KbL .section-2 text{fill:#ffffff;}#mermaid-svg-Jg65XyvxT3I62KbL .node-icon-2{font-size:40px;color:#ffffff;}#mermaid-svg-Jg65XyvxT3I62KbL .section-edge-2{stroke:hsl(270, 100%, 76.2745098039%);}#mermaid-svg-Jg65XyvxT3I62KbL .edge-depth-2{stroke-width:8;}#mermaid-svg-Jg65XyvxT3I62KbL .section-2 line{stroke:hsl(90, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-Jg65XyvxT3I62KbL .disabled,#mermaid-svg-Jg65XyvxT3I62KbL .disabled circle,#mermaid-svg-Jg65XyvxT3I62KbL .disabled text{fill:lightgray;}#mermaid-svg-Jg65XyvxT3I62KbL .disabled text{fill:#efefef;}#mermaid-svg-Jg65XyvxT3I62KbL .section-3 rect,#mermaid-svg-Jg65XyvxT3I62KbL .section-3 path,#mermaid-svg-Jg65XyvxT3I62KbL .section-3 circle,#mermaid-svg-Jg65XyvxT3I62KbL .section-3 polygon,#mermaid-svg-Jg65XyvxT3I62KbL .section-3 path{fill:hsl(300, 100%, 76.2745098039%);}#mermaid-svg-Jg65XyvxT3I62KbL .section-3 text{fill:black;}#mermaid-svg-Jg65XyvxT3I62KbL .node-icon-3{font-size:40px;color:black;}#mermaid-svg-Jg65XyvxT3I62KbL .section-edge-3{stroke:hsl(300, 100%, 76.2745098039%);}#mermaid-svg-Jg65XyvxT3I62KbL .edge-depth-3{stroke-width:5;}#mermaid-svg-Jg65XyvxT3I62KbL .section-3 line{stroke:hsl(120, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-Jg65XyvxT3I62KbL .disabled,#mermaid-svg-Jg65XyvxT3I62KbL .disabled circle,#mermaid-svg-Jg65XyvxT3I62KbL .disabled text{fill:lightgray;}#mermaid-svg-Jg65XyvxT3I62KbL .disabled text{fill:#efefef;}#mermaid-svg-Jg65XyvxT3I62KbL .section-4 rect,#mermaid-svg-Jg65XyvxT3I62KbL .section-4 path,#mermaid-svg-Jg65XyvxT3I62KbL .section-4 circle,#mermaid-svg-Jg65XyvxT3I62KbL .section-4 polygon,#mermaid-svg-Jg65XyvxT3I62KbL .section-4 path{fill:hsl(330, 100%, 76.2745098039%);}#mermaid-svg-Jg65XyvxT3I62KbL .section-4 text{fill:black;}#mermaid-svg-Jg65XyvxT3I62KbL .node-icon-4{font-size:40px;color:black;}#mermaid-svg-Jg65XyvxT3I62KbL .section-edge-4{stroke:hsl(330, 100%, 76.2745098039%);}#mermaid-svg-Jg65XyvxT3I62KbL .edge-depth-4{stroke-width:2;}#mermaid-svg-Jg65XyvxT3I62KbL .section-4 line{stroke:hsl(150, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-Jg65XyvxT3I62KbL .disabled,#mermaid-svg-Jg65XyvxT3I62KbL .disabled circle,#mermaid-svg-Jg65XyvxT3I62KbL .disabled text{fill:lightgray;}#mermaid-svg-Jg65XyvxT3I62KbL .disabled text{fill:#efefef;}#mermaid-svg-Jg65XyvxT3I62KbL .section-5 rect,#mermaid-svg-Jg65XyvxT3I62KbL .section-5 path,#mermaid-svg-Jg65XyvxT3I62KbL .section-5 circle,#mermaid-svg-Jg65XyvxT3I62KbL .section-5 polygon,#mermaid-svg-Jg65XyvxT3I62KbL .section-5 path{fill:hsl(0, 100%, 76.2745098039%);}#mermaid-svg-Jg65XyvxT3I62KbL .section-5 text{fill:black;}#mermaid-svg-Jg65XyvxT3I62KbL .node-icon-5{font-size:40px;color:black;}#mermaid-svg-Jg65XyvxT3I62KbL .section-edge-5{stroke:hsl(0, 100%, 76.2745098039%);}#mermaid-svg-Jg65XyvxT3I62KbL .edge-depth-5{stroke-width:-1;}#mermaid-svg-Jg65XyvxT3I62KbL .section-5 line{stroke:hsl(180, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-Jg65XyvxT3I62KbL .disabled,#mermaid-svg-Jg65XyvxT3I62KbL .disabled circle,#mermaid-svg-Jg65XyvxT3I62KbL .disabled text{fill:lightgray;}#mermaid-svg-Jg65XyvxT3I62KbL .disabled text{fill:#efefef;}#mermaid-svg-Jg65XyvxT3I62KbL .section-6 rect,#mermaid-svg-Jg65XyvxT3I62KbL .section-6 path,#mermaid-svg-Jg65XyvxT3I62KbL .section-6 circle,#mermaid-svg-Jg65XyvxT3I62KbL .section-6 polygon,#mermaid-svg-Jg65XyvxT3I62KbL .section-6 path{fill:hsl(30, 100%, 76.2745098039%);}#mermaid-svg-Jg65XyvxT3I62KbL .section-6 text{fill:black;}#mermaid-svg-Jg65XyvxT3I62KbL .node-icon-6{font-size:40px;color:black;}#mermaid-svg-Jg65XyvxT3I62KbL .section-edge-6{stroke:hsl(30, 100%, 76.2745098039%);}#mermaid-svg-Jg65XyvxT3I62KbL .edge-depth-6{stroke-width:-4;}#mermaid-svg-Jg65XyvxT3I62KbL .section-6 line{stroke:hsl(210, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-Jg65XyvxT3I62KbL .disabled,#mermaid-svg-Jg65XyvxT3I62KbL .disabled circle,#mermaid-svg-Jg65XyvxT3I62KbL .disabled text{fill:lightgray;}#mermaid-svg-Jg65XyvxT3I62KbL .disabled text{fill:#efefef;}#mermaid-svg-Jg65XyvxT3I62KbL .section-7 rect,#mermaid-svg-Jg65XyvxT3I62KbL .section-7 path,#mermaid-svg-Jg65XyvxT3I62KbL .section-7 circle,#mermaid-svg-Jg65XyvxT3I62KbL .section-7 polygon,#mermaid-svg-Jg65XyvxT3I62KbL .section-7 path{fill:hsl(90, 100%, 76.2745098039%);}#mermaid-svg-Jg65XyvxT3I62KbL .section-7 text{fill:black;}#mermaid-svg-Jg65XyvxT3I62KbL .node-icon-7{font-size:40px;color:black;}#mermaid-svg-Jg65XyvxT3I62KbL .section-edge-7{stroke:hsl(90, 100%, 76.2745098039%);}#mermaid-svg-Jg65XyvxT3I62KbL .edge-depth-7{stroke-width:-7;}#mermaid-svg-Jg65XyvxT3I62KbL .section-7 line{stroke:hsl(270, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-Jg65XyvxT3I62KbL .disabled,#mermaid-svg-Jg65XyvxT3I62KbL .disabled circle,#mermaid-svg-Jg65XyvxT3I62KbL .disabled text{fill:lightgray;}#mermaid-svg-Jg65XyvxT3I62KbL .disabled text{fill:#efefef;}#mermaid-svg-Jg65XyvxT3I62KbL .section-8 rect,#mermaid-svg-Jg65XyvxT3I62KbL .section-8 path,#mermaid-svg-Jg65XyvxT3I62KbL .section-8 circle,#mermaid-svg-Jg65XyvxT3I62KbL .section-8 polygon,#mermaid-svg-Jg65XyvxT3I62KbL .section-8 path{fill:hsl(150, 100%, 76.2745098039%);}#mermaid-svg-Jg65XyvxT3I62KbL .section-8 text{fill:black;}#mermaid-svg-Jg65XyvxT3I62KbL .node-icon-8{font-size:40px;color:black;}#mermaid-svg-Jg65XyvxT3I62KbL .section-edge-8{stroke:hsl(150, 100%, 76.2745098039%);}#mermaid-svg-Jg65XyvxT3I62KbL .edge-depth-8{stroke-width:-10;}#mermaid-svg-Jg65XyvxT3I62KbL .section-8 line{stroke:hsl(330, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-Jg65XyvxT3I62KbL .disabled,#mermaid-svg-Jg65XyvxT3I62KbL .disabled circle,#mermaid-svg-Jg65XyvxT3I62KbL .disabled text{fill:lightgray;}#mermaid-svg-Jg65XyvxT3I62KbL .disabled text{fill:#efefef;}#mermaid-svg-Jg65XyvxT3I62KbL .section-9 rect,#mermaid-svg-Jg65XyvxT3I62KbL .section-9 path,#mermaid-svg-Jg65XyvxT3I62KbL .section-9 circle,#mermaid-svg-Jg65XyvxT3I62KbL .section-9 polygon,#mermaid-svg-Jg65XyvxT3I62KbL .section-9 path{fill:hsl(180, 100%, 76.2745098039%);}#mermaid-svg-Jg65XyvxT3I62KbL .section-9 text{fill:black;}#mermaid-svg-Jg65XyvxT3I62KbL .node-icon-9{font-size:40px;color:black;}#mermaid-svg-Jg65XyvxT3I62KbL .section-edge-9{stroke:hsl(180, 100%, 76.2745098039%);}#mermaid-svg-Jg65XyvxT3I62KbL .edge-depth-9{stroke-width:-13;}#mermaid-svg-Jg65XyvxT3I62KbL .section-9 line{stroke:hsl(0, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-Jg65XyvxT3I62KbL .disabled,#mermaid-svg-Jg65XyvxT3I62KbL .disabled circle,#mermaid-svg-Jg65XyvxT3I62KbL .disabled text{fill:lightgray;}#mermaid-svg-Jg65XyvxT3I62KbL .disabled text{fill:#efefef;}#mermaid-svg-Jg65XyvxT3I62KbL .section-10 rect,#mermaid-svg-Jg65XyvxT3I62KbL .section-10 path,#mermaid-svg-Jg65XyvxT3I62KbL .section-10 circle,#mermaid-svg-Jg65XyvxT3I62KbL .section-10 polygon,#mermaid-svg-Jg65XyvxT3I62KbL .section-10 path{fill:hsl(210, 100%, 76.2745098039%);}#mermaid-svg-Jg65XyvxT3I62KbL .section-10 text{fill:black;}#mermaid-svg-Jg65XyvxT3I62KbL .node-icon-10{font-size:40px;color:black;}#mermaid-svg-Jg65XyvxT3I62KbL .section-edge-10{stroke:hsl(210, 100%, 76.2745098039%);}#mermaid-svg-Jg65XyvxT3I62KbL .edge-depth-10{stroke-width:-16;}#mermaid-svg-Jg65XyvxT3I62KbL .section-10 line{stroke:hsl(30, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-Jg65XyvxT3I62KbL .disabled,#mermaid-svg-Jg65XyvxT3I62KbL .disabled circle,#mermaid-svg-Jg65XyvxT3I62KbL .disabled text{fill:lightgray;}#mermaid-svg-Jg65XyvxT3I62KbL .disabled text{fill:#efefef;}#mermaid-svg-Jg65XyvxT3I62KbL .section-root rect,#mermaid-svg-Jg65XyvxT3I62KbL .section-root path,#mermaid-svg-Jg65XyvxT3I62KbL .section-root circle,#mermaid-svg-Jg65XyvxT3I62KbL .section-root polygon{fill:hsl(240, 100%, 46.2745098039%);}#mermaid-svg-Jg65XyvxT3I62KbL .section-root text{fill:#ffffff;}#mermaid-svg-Jg65XyvxT3I62KbL .section-root span{color:#ffffff;}#mermaid-svg-Jg65XyvxT3I62KbL .section-2 span{color:#ffffff;}#mermaid-svg-Jg65XyvxT3I62KbL .icon-container{height:100%;display:flex;justify-content:center;align-items:center;}#mermaid-svg-Jg65XyvxT3I62KbL .edge{fill:none;}#mermaid-svg-Jg65XyvxT3I62KbL .mindmap-node-label{dy:1em;alignment-baseline:middle;text-anchor:middle;dominant-baseline:middle;text-align:center;}#mermaid-svg-Jg65XyvxT3I62KbL :root{--mermaid-font-family:"trebuchet ms",verdana,arial,sans-serif;} 混沌测试学习路线
入门篇
什么是混沌测试
为什么需要它
Netflix Chaos Monkey 的故事
Chaos Engineering 基本原则
基础篇
稳态假设
爆炸半径
实验设计方法
指标与可观测性
实战篇
基础设施层故障注入
容器与K8s层故障
应用层故障
数据层故障
工具篇
Chaos Mesh
LitmusChaos
ChaosBlade
Gremlin
进阶篇
生产环境混沌测试
自动化实验编排
GameDay 实践
混沌成熟度模型

第一部分：混沌测试的「为什么」和「是什么」

第1章 一个真实的故事------Netflix 的觉醒

🎬 场景：2008 年的 Netflix

2008 年，Netflix 还是一家 DVD 邮寄公司。

2009 年，他们开始转型流媒体。

但问题是：他们的基础设施是单体架构，运行在传统数据中心里。

结果呢？

一天，Netflix 的数据库出了故障，服务挂了整整 3天 。

3天！对于一个流媒体服务来说，3天意味着什么？

意味着用户流失、收入损失、品牌信誉受损。

Netflix 的工程师们痛定思痛，问了自己一个问题：

"如果我们不能防止故障发生，那能不能确保故障发生时，系统仍然能正常工作？"

这个问题的答案，催生了一个叫 Chaos Monkey 的工具------一只会在工作时间随机杀死生产服务器的"混沌猴子"。
#mermaid-svg-mrBKO898gmMc3ofN{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-mrBKO898gmMc3ofN .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-mrBKO898gmMc3ofN .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-mrBKO898gmMc3ofN .error-icon{fill:#552222;}#mermaid-svg-mrBKO898gmMc3ofN .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-mrBKO898gmMc3ofN .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-mrBKO898gmMc3ofN .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-mrBKO898gmMc3ofN .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-mrBKO898gmMc3ofN .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-mrBKO898gmMc3ofN .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-mrBKO898gmMc3ofN .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-mrBKO898gmMc3ofN .marker{fill:#333333;stroke:#333333;}#mermaid-svg-mrBKO898gmMc3ofN .marker.cross{stroke:#333333;}#mermaid-svg-mrBKO898gmMc3ofN svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-mrBKO898gmMc3ofN p{margin:0;}#mermaid-svg-mrBKO898gmMc3ofN .edge{stroke-width:3;}#mermaid-svg-mrBKO898gmMc3ofN .section--1 rect,#mermaid-svg-mrBKO898gmMc3ofN .section--1 path,#mermaid-svg-mrBKO898gmMc3ofN .section--1 circle,#mermaid-svg-mrBKO898gmMc3ofN .section--1 path{fill:hsl(240, 100%, 76.2745098039%);}#mermaid-svg-mrBKO898gmMc3ofN .section--1 text{fill:#ffffff;}#mermaid-svg-mrBKO898gmMc3ofN .node-icon--1{font-size:40px;color:#ffffff;}#mermaid-svg-mrBKO898gmMc3ofN .section-edge--1{stroke:hsl(240, 100%, 76.2745098039%);}#mermaid-svg-mrBKO898gmMc3ofN .edge-depth--1{stroke-width:17;}#mermaid-svg-mrBKO898gmMc3ofN .section--1 line{stroke:hsl(60, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-mrBKO898gmMc3ofN .lineWrapper line{stroke:#ffffff;}#mermaid-svg-mrBKO898gmMc3ofN .disabled,#mermaid-svg-mrBKO898gmMc3ofN .disabled circle,#mermaid-svg-mrBKO898gmMc3ofN .disabled text{fill:lightgray;}#mermaid-svg-mrBKO898gmMc3ofN .disabled text{fill:#efefef;}#mermaid-svg-mrBKO898gmMc3ofN .section-0 rect,#mermaid-svg-mrBKO898gmMc3ofN .section-0 path,#mermaid-svg-mrBKO898gmMc3ofN .section-0 circle,#mermaid-svg-mrBKO898gmMc3ofN .section-0 path{fill:hsl(60, 100%, 73.5294117647%);}#mermaid-svg-mrBKO898gmMc3ofN .section-0 text{fill:black;}#mermaid-svg-mrBKO898gmMc3ofN .node-icon-0{font-size:40px;color:black;}#mermaid-svg-mrBKO898gmMc3ofN .section-edge-0{stroke:hsl(60, 100%, 73.5294117647%);}#mermaid-svg-mrBKO898gmMc3ofN .edge-depth-0{stroke-width:14;}#mermaid-svg-mrBKO898gmMc3ofN .section-0 line{stroke:hsl(240, 100%, 83.5294117647%);stroke-width:3;}#mermaid-svg-mrBKO898gmMc3ofN .lineWrapper line{stroke:black;}#mermaid-svg-mrBKO898gmMc3ofN .disabled,#mermaid-svg-mrBKO898gmMc3ofN .disabled circle,#mermaid-svg-mrBKO898gmMc3ofN .disabled text{fill:lightgray;}#mermaid-svg-mrBKO898gmMc3ofN .disabled text{fill:#efefef;}#mermaid-svg-mrBKO898gmMc3ofN .section-1 rect,#mermaid-svg-mrBKO898gmMc3ofN .section-1 path,#mermaid-svg-mrBKO898gmMc3ofN .section-1 circle,#mermaid-svg-mrBKO898gmMc3ofN .section-1 path{fill:hsl(80, 100%, 76.2745098039%);}#mermaid-svg-mrBKO898gmMc3ofN .section-1 text{fill:black;}#mermaid-svg-mrBKO898gmMc3ofN .node-icon-1{font-size:40px;color:black;}#mermaid-svg-mrBKO898gmMc3ofN .section-edge-1{stroke:hsl(80, 100%, 76.2745098039%);}#mermaid-svg-mrBKO898gmMc3ofN .edge-depth-1{stroke-width:11;}#mermaid-svg-mrBKO898gmMc3ofN .section-1 line{stroke:hsl(260, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-mrBKO898gmMc3ofN .lineWrapper line{stroke:black;}#mermaid-svg-mrBKO898gmMc3ofN .disabled,#mermaid-svg-mrBKO898gmMc3ofN .disabled circle,#mermaid-svg-mrBKO898gmMc3ofN .disabled text{fill:lightgray;}#mermaid-svg-mrBKO898gmMc3ofN .disabled text{fill:#efefef;}#mermaid-svg-mrBKO898gmMc3ofN .section-2 rect,#mermaid-svg-mrBKO898gmMc3ofN .section-2 path,#mermaid-svg-mrBKO898gmMc3ofN .section-2 circle,#mermaid-svg-mrBKO898gmMc3ofN .section-2 path{fill:hsl(270, 100%, 76.2745098039%);}#mermaid-svg-mrBKO898gmMc3ofN .section-2 text{fill:#ffffff;}#mermaid-svg-mrBKO898gmMc3ofN .node-icon-2{font-size:40px;color:#ffffff;}#mermaid-svg-mrBKO898gmMc3ofN .section-edge-2{stroke:hsl(270, 100%, 76.2745098039%);}#mermaid-svg-mrBKO898gmMc3ofN .edge-depth-2{stroke-width:8;}#mermaid-svg-mrBKO898gmMc3ofN .section-2 line{stroke:hsl(90, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-mrBKO898gmMc3ofN .lineWrapper line{stroke:#ffffff;}#mermaid-svg-mrBKO898gmMc3ofN .disabled,#mermaid-svg-mrBKO898gmMc3ofN .disabled circle,#mermaid-svg-mrBKO898gmMc3ofN .disabled text{fill:lightgray;}#mermaid-svg-mrBKO898gmMc3ofN .disabled text{fill:#efefef;}#mermaid-svg-mrBKO898gmMc3ofN .section-3 rect,#mermaid-svg-mrBKO898gmMc3ofN .section-3 path,#mermaid-svg-mrBKO898gmMc3ofN .section-3 circle,#mermaid-svg-mrBKO898gmMc3ofN .section-3 path{fill:hsl(300, 100%, 76.2745098039%);}#mermaid-svg-mrBKO898gmMc3ofN .section-3 text{fill:black;}#mermaid-svg-mrBKO898gmMc3ofN .node-icon-3{font-size:40px;color:black;}#mermaid-svg-mrBKO898gmMc3ofN .section-edge-3{stroke:hsl(300, 100%, 76.2745098039%);}#mermaid-svg-mrBKO898gmMc3ofN .edge-depth-3{stroke-width:5;}#mermaid-svg-mrBKO898gmMc3ofN .section-3 line{stroke:hsl(120, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-mrBKO898gmMc3ofN .lineWrapper line{stroke:black;}#mermaid-svg-mrBKO898gmMc3ofN .disabled,#mermaid-svg-mrBKO898gmMc3ofN .disabled circle,#mermaid-svg-mrBKO898gmMc3ofN .disabled text{fill:lightgray;}#mermaid-svg-mrBKO898gmMc3ofN .disabled text{fill:#efefef;}#mermaid-svg-mrBKO898gmMc3ofN .section-4 rect,#mermaid-svg-mrBKO898gmMc3ofN .section-4 path,#mermaid-svg-mrBKO898gmMc3ofN .section-4 circle,#mermaid-svg-mrBKO898gmMc3ofN .section-4 path{fill:hsl(330, 100%, 76.2745098039%);}#mermaid-svg-mrBKO898gmMc3ofN .section-4 text{fill:black;}#mermaid-svg-mrBKO898gmMc3ofN .node-icon-4{font-size:40px;color:black;}#mermaid-svg-mrBKO898gmMc3ofN .section-edge-4{stroke:hsl(330, 100%, 76.2745098039%);}#mermaid-svg-mrBKO898gmMc3ofN .edge-depth-4{stroke-width:2;}#mermaid-svg-mrBKO898gmMc3ofN .section-4 line{stroke:hsl(150, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-mrBKO898gmMc3ofN .lineWrapper line{stroke:black;}#mermaid-svg-mrBKO898gmMc3ofN .disabled,#mermaid-svg-mrBKO898gmMc3ofN .disabled circle,#mermaid-svg-mrBKO898gmMc3ofN .disabled text{fill:lightgray;}#mermaid-svg-mrBKO898gmMc3ofN .disabled text{fill:#efefef;}#mermaid-svg-mrBKO898gmMc3ofN .section-5 rect,#mermaid-svg-mrBKO898gmMc3ofN .section-5 path,#mermaid-svg-mrBKO898gmMc3ofN .section-5 circle,#mermaid-svg-mrBKO898gmMc3ofN .section-5 path{fill:hsl(0, 100%, 76.2745098039%);}#mermaid-svg-mrBKO898gmMc3ofN .section-5 text{fill:black;}#mermaid-svg-mrBKO898gmMc3ofN .node-icon-5{font-size:40px;color:black;}#mermaid-svg-mrBKO898gmMc3ofN .section-edge-5{stroke:hsl(0, 100%, 76.2745098039%);}#mermaid-svg-mrBKO898gmMc3ofN .edge-depth-5{stroke-width:-1;}#mermaid-svg-mrBKO898gmMc3ofN .section-5 line{stroke:hsl(180, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-mrBKO898gmMc3ofN .lineWrapper line{stroke:black;}#mermaid-svg-mrBKO898gmMc3ofN .disabled,#mermaid-svg-mrBKO898gmMc3ofN .disabled circle,#mermaid-svg-mrBKO898gmMc3ofN .disabled text{fill:lightgray;}#mermaid-svg-mrBKO898gmMc3ofN .disabled text{fill:#efefef;}#mermaid-svg-mrBKO898gmMc3ofN .section-6 rect,#mermaid-svg-mrBKO898gmMc3ofN .section-6 path,#mermaid-svg-mrBKO898gmMc3ofN .section-6 circle,#mermaid-svg-mrBKO898gmMc3ofN .section-6 path{fill:hsl(30, 100%, 76.2745098039%);}#mermaid-svg-mrBKO898gmMc3ofN .section-6 text{fill:black;}#mermaid-svg-mrBKO898gmMc3ofN .node-icon-6{font-size:40px;color:black;}#mermaid-svg-mrBKO898gmMc3ofN .section-edge-6{stroke:hsl(30, 100%, 76.2745098039%);}#mermaid-svg-mrBKO898gmMc3ofN .edge-depth-6{stroke-width:-4;}#mermaid-svg-mrBKO898gmMc3ofN .section-6 line{stroke:hsl(210, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-mrBKO898gmMc3ofN .lineWrapper line{stroke:black;}#mermaid-svg-mrBKO898gmMc3ofN .disabled,#mermaid-svg-mrBKO898gmMc3ofN .disabled circle,#mermaid-svg-mrBKO898gmMc3ofN .disabled text{fill:lightgray;}#mermaid-svg-mrBKO898gmMc3ofN .disabled text{fill:#efefef;}#mermaid-svg-mrBKO898gmMc3ofN .section-7 rect,#mermaid-svg-mrBKO898gmMc3ofN .section-7 path,#mermaid-svg-mrBKO898gmMc3ofN .section-7 circle,#mermaid-svg-mrBKO898gmMc3ofN .section-7 path{fill:hsl(90, 100%, 76.2745098039%);}#mermaid-svg-mrBKO898gmMc3ofN .section-7 text{fill:black;}#mermaid-svg-mrBKO898gmMc3ofN .node-icon-7{font-size:40px;color:black;}#mermaid-svg-mrBKO898gmMc3ofN .section-edge-7{stroke:hsl(90, 100%, 76.2745098039%);}#mermaid-svg-mrBKO898gmMc3ofN .edge-depth-7{stroke-width:-7;}#mermaid-svg-mrBKO898gmMc3ofN .section-7 line{stroke:hsl(270, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-mrBKO898gmMc3ofN .lineWrapper line{stroke:black;}#mermaid-svg-mrBKO898gmMc3ofN .disabled,#mermaid-svg-mrBKO898gmMc3ofN .disabled circle,#mermaid-svg-mrBKO898gmMc3ofN .disabled text{fill:lightgray;}#mermaid-svg-mrBKO898gmMc3ofN .disabled text{fill:#efefef;}#mermaid-svg-mrBKO898gmMc3ofN .section-8 rect,#mermaid-svg-mrBKO898gmMc3ofN .section-8 path,#mermaid-svg-mrBKO898gmMc3ofN .section-8 circle,#mermaid-svg-mrBKO898gmMc3ofN .section-8 path{fill:hsl(150, 100%, 76.2745098039%);}#mermaid-svg-mrBKO898gmMc3ofN .section-8 text{fill:black;}#mermaid-svg-mrBKO898gmMc3ofN .node-icon-8{font-size:40px;color:black;}#mermaid-svg-mrBKO898gmMc3ofN .section-edge-8{stroke:hsl(150, 100%, 76.2745098039%);}#mermaid-svg-mrBKO898gmMc3ofN .edge-depth-8{stroke-width:-10;}#mermaid-svg-mrBKO898gmMc3ofN .section-8 line{stroke:hsl(330, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-mrBKO898gmMc3ofN .lineWrapper line{stroke:black;}#mermaid-svg-mrBKO898gmMc3ofN .disabled,#mermaid-svg-mrBKO898gmMc3ofN .disabled circle,#mermaid-svg-mrBKO898gmMc3ofN .disabled text{fill:lightgray;}#mermaid-svg-mrBKO898gmMc3ofN .disabled text{fill:#efefef;}#mermaid-svg-mrBKO898gmMc3ofN .section-9 rect,#mermaid-svg-mrBKO898gmMc3ofN .section-9 path,#mermaid-svg-mrBKO898gmMc3ofN .section-9 circle,#mermaid-svg-mrBKO898gmMc3ofN .section-9 path{fill:hsl(180, 100%, 76.2745098039%);}#mermaid-svg-mrBKO898gmMc3ofN .section-9 text{fill:black;}#mermaid-svg-mrBKO898gmMc3ofN .node-icon-9{font-size:40px;color:black;}#mermaid-svg-mrBKO898gmMc3ofN .section-edge-9{stroke:hsl(180, 100%, 76.2745098039%);}#mermaid-svg-mrBKO898gmMc3ofN .edge-depth-9{stroke-width:-13;}#mermaid-svg-mrBKO898gmMc3ofN .section-9 line{stroke:hsl(0, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-mrBKO898gmMc3ofN .lineWrapper line{stroke:black;}#mermaid-svg-mrBKO898gmMc3ofN .disabled,#mermaid-svg-mrBKO898gmMc3ofN .disabled circle,#mermaid-svg-mrBKO898gmMc3ofN .disabled text{fill:lightgray;}#mermaid-svg-mrBKO898gmMc3ofN .disabled text{fill:#efefef;}#mermaid-svg-mrBKO898gmMc3ofN .section-10 rect,#mermaid-svg-mrBKO898gmMc3ofN .section-10 path,#mermaid-svg-mrBKO898gmMc3ofN .section-10 circle,#mermaid-svg-mrBKO898gmMc3ofN .section-10 path{fill:hsl(210, 100%, 76.2745098039%);}#mermaid-svg-mrBKO898gmMc3ofN .section-10 text{fill:black;}#mermaid-svg-mrBKO898gmMc3ofN .node-icon-10{font-size:40px;color:black;}#mermaid-svg-mrBKO898gmMc3ofN .section-edge-10{stroke:hsl(210, 100%, 76.2745098039%);}#mermaid-svg-mrBKO898gmMc3ofN .edge-depth-10{stroke-width:-16;}#mermaid-svg-mrBKO898gmMc3ofN .section-10 line{stroke:hsl(30, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-mrBKO898gmMc3ofN .lineWrapper line{stroke:black;}#mermaid-svg-mrBKO898gmMc3ofN .disabled,#mermaid-svg-mrBKO898gmMc3ofN .disabled circle,#mermaid-svg-mrBKO898gmMc3ofN .disabled text{fill:lightgray;}#mermaid-svg-mrBKO898gmMc3ofN .disabled text{fill:#efefef;}#mermaid-svg-mrBKO898gmMc3ofN .section-root rect,#mermaid-svg-mrBKO898gmMc3ofN .section-root path,#mermaid-svg-mrBKO898gmMc3ofN .section-root circle{fill:hsl(240, 100%, 46.2745098039%);}#mermaid-svg-mrBKO898gmMc3ofN .section-root text{fill:#ffffff;}#mermaid-svg-mrBKO898gmMc3ofN .icon-container{height:100%;display:flex;justify-content:center;align-items:center;}#mermaid-svg-mrBKO898gmMc3ofN .edge{fill:none;}#mermaid-svg-mrBKO898gmMc3ofN .eventWrapper{filter:brightness(120%);}#mermaid-svg-mrBKO898gmMc3ofN :root{--mermaid-font-family:"trebuchet ms",verdana,arial,sans-serif;} 2008 数据库故障 服务宕机3天 传统单体架构 2009 开始向AWS迁移 微服务化改造 启动混沌工程探索 2011 Chaos Monkey诞生 Simian Army工具集 随机杀死生产实例 2016 Chaos Monkey开源 ChaosEngineering原则发布 社区蓬勃发展 现在 混沌工程成为标准实践 各种成熟工具 行业最佳实践 Netflix 混沌工程的演化之路

💭 想一想

当你第一次听到"随机杀死生产服务器"时，你的感觉是：

复制代码 
□ 这也太疯狂了！
□ 他们不怕出事故吗？
□ 有点道理...但是我不敢
□ 我要学这个！

不管你的第一反应是什么，读完这本书后，你会明白为什么 Netflix 敢这么做 ，以及为什么你的组织也需要这样做。

第2章 混沌测试到底是什么？

🔍 官方定义

根据混沌工程原理（Principles of Chaos Engineering）：

混沌工程是对分布式系统进行实验的学科，目的是建立对系统在动荡条件下行为能力的信心。

翻译成人话：

故意搞破坏，看看系统会不会崩。如果不会，你就知道它很健壮。

🤯 不是你想的那样

很多人对混沌测试有几个巨大的误解：

❌ 误解 ✅ 真相
混沌测试就是随机搞破坏 是有计划的、科学实验驱动的方法
混沌测试只在大公司才有用 任何分布式系统都需要
混沌测试很危险 危险是因为你没有控制好爆炸半径
混沌测试在生产环境做 可以从开发/测试环境开始
混沌测试就是压力测试 压力测试只施压，混沌测试验证假设

🧪 混沌测试 vs 压力测试 vs 故障注入

#mermaid-svg-rZYkNL0Oyy1N85Q0{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-rZYkNL0Oyy1N85Q0 .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-rZYkNL0Oyy1N85Q0 .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-rZYkNL0Oyy1N85Q0 .error-icon{fill:#552222;}#mermaid-svg-rZYkNL0Oyy1N85Q0 .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-rZYkNL0Oyy1N85Q0 .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-rZYkNL0Oyy1N85Q0 .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-rZYkNL0Oyy1N85Q0 .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-rZYkNL0Oyy1N85Q0 .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-rZYkNL0Oyy1N85Q0 .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-rZYkNL0Oyy1N85Q0 .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-rZYkNL0Oyy1N85Q0 .marker{fill:#333333;stroke:#333333;}#mermaid-svg-rZYkNL0Oyy1N85Q0 .marker.cross{stroke:#333333;}#mermaid-svg-rZYkNL0Oyy1N85Q0 svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-rZYkNL0Oyy1N85Q0 p{margin:0;}#mermaid-svg-rZYkNL0Oyy1N85Q0 .label{font-family:"trebuchet ms",verdana,arial,sans-serif;color:#333;}#mermaid-svg-rZYkNL0Oyy1N85Q0 .cluster-label text{fill:#333;}#mermaid-svg-rZYkNL0Oyy1N85Q0 .cluster-label span{color:#333;}#mermaid-svg-rZYkNL0Oyy1N85Q0 .cluster-label span p{background-color:transparent;}#mermaid-svg-rZYkNL0Oyy1N85Q0 .label text,#mermaid-svg-rZYkNL0Oyy1N85Q0 span{fill:#333;color:#333;}#mermaid-svg-rZYkNL0Oyy1N85Q0 .node rect,#mermaid-svg-rZYkNL0Oyy1N85Q0 .node circle,#mermaid-svg-rZYkNL0Oyy1N85Q0 .node ellipse,#mermaid-svg-rZYkNL0Oyy1N85Q0 .node polygon,#mermaid-svg-rZYkNL0Oyy1N85Q0 .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-rZYkNL0Oyy1N85Q0 .rough-node .label text,#mermaid-svg-rZYkNL0Oyy1N85Q0 .node .label text,#mermaid-svg-rZYkNL0Oyy1N85Q0 .image-shape .label,#mermaid-svg-rZYkNL0Oyy1N85Q0 .icon-shape .label{text-anchor:middle;}#mermaid-svg-rZYkNL0Oyy1N85Q0 .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-rZYkNL0Oyy1N85Q0 .rough-node .label,#mermaid-svg-rZYkNL0Oyy1N85Q0 .node .label,#mermaid-svg-rZYkNL0Oyy1N85Q0 .image-shape .label,#mermaid-svg-rZYkNL0Oyy1N85Q0 .icon-shape .label{text-align:center;}#mermaid-svg-rZYkNL0Oyy1N85Q0 .node.clickable{cursor:pointer;}#mermaid-svg-rZYkNL0Oyy1N85Q0 .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-rZYkNL0Oyy1N85Q0 .arrowheadPath{fill:#333333;}#mermaid-svg-rZYkNL0Oyy1N85Q0 .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-rZYkNL0Oyy1N85Q0 .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-rZYkNL0Oyy1N85Q0 .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-rZYkNL0Oyy1N85Q0 .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-rZYkNL0Oyy1N85Q0 .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-rZYkNL0Oyy1N85Q0 .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-rZYkNL0Oyy1N85Q0 .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-rZYkNL0Oyy1N85Q0 .cluster text{fill:#333;}#mermaid-svg-rZYkNL0Oyy1N85Q0 .cluster span{color:#333;}#mermaid-svg-rZYkNL0Oyy1N85Q0 div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-rZYkNL0Oyy1N85Q0 .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-rZYkNL0Oyy1N85Q0 rect.text{fill:none;stroke-width:0;}#mermaid-svg-rZYkNL0Oyy1N85Q0 .icon-shape,#mermaid-svg-rZYkNL0Oyy1N85Q0 .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-rZYkNL0Oyy1N85Q0 .icon-shape p,#mermaid-svg-rZYkNL0Oyy1N85Q0 .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-rZYkNL0Oyy1N85Q0 .icon-shape .label rect,#mermaid-svg-rZYkNL0Oyy1N85Q0 .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-rZYkNL0Oyy1N85Q0 .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-rZYkNL0Oyy1N85Q0 .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-rZYkNL0Oyy1N85Q0 :root{--mermaid-font-family:"trebuchet ms",verdana,arial,sans-serif;} 混沌工程
传统测试
压力测试

加负载看极限
功能测试

验证功能正确
故障注入

注入特定故障
混沌实验

验证系统韧性假设
可观测性
系统韧性
改进措施

关键区别：

  • 压力测试：问"系统能承受多大的负载？"
  • 故障注入：问"注入这个故障，系统会怎样？"
  • 混沌测试：问"我们相信系统能在这种情况下正常运行，真的吗？"

混沌测试 = 故障注入 + 可验证的假设 + 科学的实验方法

第3章 混沌工程的四大原则

2014年，混沌工程社区正式发布了 混沌工程的四大原则 。

别跳过这一章------这些原则是你的"安全护栏"。

原则一：定义稳态（Define Steady State）

核心思想： 在搞破坏之前，你得知道"正常"是什么样子。

复制代码 
┌──────────────────────────────────┐
│  正常状态指标                        │
│  ─────────────────────             │
│  ✓ 响应时间 < 200ms                │
│  ✓ 错误率 < 0.1%                   │
│  ✓ CPU 使用率 < 80%                │
│  ✓ 订单成功率 > 99.9%              │
│  ✓ ...                             │
└──────────────────────────────────┘
     │
     ▼
┌──────────────────────────────────┐
│  引入故障后，检查这些指标是否变化  │
│  如果没变 → ✅ 系统有韧性          │
│  如果变了 → 🔧 需要改进            │
└──────────────────────────────────┘

原则二：提出假设（Hypothesize）

不要只做"看看会发生什么"。混沌实验应该是：

我们相信：实验条件 不会影响 稳态指标

例子：

  • ❌ "我们来杀一个 Pod 看看会发生什么"
  • ✅ "我们相信，当杀死 1 个 Pod 时，由于 K8s Deployment 有 3 个副本，系统错误率不会超过 1%"

原则三：控制爆炸半径（Blast Radius）

这条最重要！不要在第一天就炸掉生产环境。
#mermaid-svg-0eXZN5ADkcgRdjWs{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-0eXZN5ADkcgRdjWs .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-0eXZN5ADkcgRdjWs .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-0eXZN5ADkcgRdjWs .error-icon{fill:#552222;}#mermaid-svg-0eXZN5ADkcgRdjWs .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-0eXZN5ADkcgRdjWs .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-0eXZN5ADkcgRdjWs .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-0eXZN5ADkcgRdjWs .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-0eXZN5ADkcgRdjWs .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-0eXZN5ADkcgRdjWs .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-0eXZN5ADkcgRdjWs .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-0eXZN5ADkcgRdjWs .marker{fill:#333333;stroke:#333333;}#mermaid-svg-0eXZN5ADkcgRdjWs .marker.cross{stroke:#333333;}#mermaid-svg-0eXZN5ADkcgRdjWs svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-0eXZN5ADkcgRdjWs p{margin:0;}#mermaid-svg-0eXZN5ADkcgRdjWs .label{font-family:"trebuchet ms",verdana,arial,sans-serif;color:#333;}#mermaid-svg-0eXZN5ADkcgRdjWs .cluster-label text{fill:#333;}#mermaid-svg-0eXZN5ADkcgRdjWs .cluster-label span{color:#333;}#mermaid-svg-0eXZN5ADkcgRdjWs .cluster-label span p{background-color:transparent;}#mermaid-svg-0eXZN5ADkcgRdjWs .label text,#mermaid-svg-0eXZN5ADkcgRdjWs span{fill:#333;color:#333;}#mermaid-svg-0eXZN5ADkcgRdjWs .node rect,#mermaid-svg-0eXZN5ADkcgRdjWs .node circle,#mermaid-svg-0eXZN5ADkcgRdjWs .node ellipse,#mermaid-svg-0eXZN5ADkcgRdjWs .node polygon,#mermaid-svg-0eXZN5ADkcgRdjWs .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-0eXZN5ADkcgRdjWs .rough-node .label text,#mermaid-svg-0eXZN5ADkcgRdjWs .node .label text,#mermaid-svg-0eXZN5ADkcgRdjWs .image-shape .label,#mermaid-svg-0eXZN5ADkcgRdjWs .icon-shape .label{text-anchor:middle;}#mermaid-svg-0eXZN5ADkcgRdjWs .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-0eXZN5ADkcgRdjWs .rough-node .label,#mermaid-svg-0eXZN5ADkcgRdjWs .node .label,#mermaid-svg-0eXZN5ADkcgRdjWs .image-shape .label,#mermaid-svg-0eXZN5ADkcgRdjWs .icon-shape .label{text-align:center;}#mermaid-svg-0eXZN5ADkcgRdjWs .node.clickable{cursor:pointer;}#mermaid-svg-0eXZN5ADkcgRdjWs .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-0eXZN5ADkcgRdjWs .arrowheadPath{fill:#333333;}#mermaid-svg-0eXZN5ADkcgRdjWs .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-0eXZN5ADkcgRdjWs .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-0eXZN5ADkcgRdjWs .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-0eXZN5ADkcgRdjWs .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-0eXZN5ADkcgRdjWs .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-0eXZN5ADkcgRdjWs .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-0eXZN5ADkcgRdjWs .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-0eXZN5ADkcgRdjWs .cluster text{fill:#333;}#mermaid-svg-0eXZN5ADkcgRdjWs .cluster span{color:#333;}#mermaid-svg-0eXZN5ADkcgRdjWs div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-0eXZN5ADkcgRdjWs .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-0eXZN5ADkcgRdjWs rect.text{fill:none;stroke-width:0;}#mermaid-svg-0eXZN5ADkcgRdjWs .icon-shape,#mermaid-svg-0eXZN5ADkcgRdjWs .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-0eXZN5ADkcgRdjWs .icon-shape p,#mermaid-svg-0eXZN5ADkcgRdjWs .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-0eXZN5ADkcgRdjWs .icon-shape .label rect,#mermaid-svg-0eXZN5ADkcgRdjWs .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-0eXZN5ADkcgRdjWs .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-0eXZN5ADkcgRdjWs .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-0eXZN5ADkcgRdjWs :root{--mermaid-font-family:"trebuchet ms",verdana,arial,sans-serif;} 爆炸半径从小到大的实验
Stage 1

测试环境

关闭1个实例
Stage 2

预发布

关闭1个可用区
Stage 3

生产环境-影子流量

关闭节点子集
Stage 4

生产环境

全范围战术实验

原则四：自动化运行实验（Automate Experiments）

混沌实验不应该是"手动执行"的。你应该：

  1. 编写实验脚本
  2. 设置定时触发
  3. 自动检测结果
  4. 自动回滚

🧪 小测验

请把下面的"假设"和对应的"稳态指标"配对：

复制代码 
假设                                        稳态指标
─────────────────────────────────────────  ────────────────────────
A. 杀掉1个Pod不会影响用户登录             1. 请求成功率 > 99.9%
B. 数据库主从切换应在30秒内完成            2. 登录页面响应时间 < 2s
C. 增加20%流量不会导致错误率上升            3. 写入延迟 < 1s

答案：A-②, B-③, C-①

第二部分：混沌测试的核心概念

第4章 稳态假设------系统的「健康基线」

🩺 就像体检一样

你去体检，医生会先测量你的正常指标：

  • 体温 36.5°C
  • 血压 120/80
  • 心跳 72 次/分钟

然后医生才能判断："嗯，你今天有点发烧。"

混沌实验也一样：

你需要先知道系统的"正常指标"是什么，才能判断故障注入后系统是否"生病"了。

📊 常见的稳态指标

#mermaid-svg-oa6iKN5wtu9zOEGL{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-oa6iKN5wtu9zOEGL .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-oa6iKN5wtu9zOEGL .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-oa6iKN5wtu9zOEGL .error-icon{fill:#552222;}#mermaid-svg-oa6iKN5wtu9zOEGL .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-oa6iKN5wtu9zOEGL .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-oa6iKN5wtu9zOEGL .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-oa6iKN5wtu9zOEGL .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-oa6iKN5wtu9zOEGL .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-oa6iKN5wtu9zOEGL .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-oa6iKN5wtu9zOEGL .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-oa6iKN5wtu9zOEGL .marker{fill:#333333;stroke:#333333;}#mermaid-svg-oa6iKN5wtu9zOEGL .marker.cross{stroke:#333333;}#mermaid-svg-oa6iKN5wtu9zOEGL svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-oa6iKN5wtu9zOEGL p{margin:0;}#mermaid-svg-oa6iKN5wtu9zOEGL .edge{stroke-width:3;}#mermaid-svg-oa6iKN5wtu9zOEGL .section--1 rect,#mermaid-svg-oa6iKN5wtu9zOEGL .section--1 path,#mermaid-svg-oa6iKN5wtu9zOEGL .section--1 circle,#mermaid-svg-oa6iKN5wtu9zOEGL .section--1 polygon,#mermaid-svg-oa6iKN5wtu9zOEGL .section--1 path{fill:hsl(240, 100%, 76.2745098039%);}#mermaid-svg-oa6iKN5wtu9zOEGL .section--1 text{fill:#ffffff;}#mermaid-svg-oa6iKN5wtu9zOEGL .node-icon--1{font-size:40px;color:#ffffff;}#mermaid-svg-oa6iKN5wtu9zOEGL .section-edge--1{stroke:hsl(240, 100%, 76.2745098039%);}#mermaid-svg-oa6iKN5wtu9zOEGL .edge-depth--1{stroke-width:17;}#mermaid-svg-oa6iKN5wtu9zOEGL .section--1 line{stroke:hsl(60, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-oa6iKN5wtu9zOEGL .disabled,#mermaid-svg-oa6iKN5wtu9zOEGL .disabled circle,#mermaid-svg-oa6iKN5wtu9zOEGL .disabled text{fill:lightgray;}#mermaid-svg-oa6iKN5wtu9zOEGL .disabled text{fill:#efefef;}#mermaid-svg-oa6iKN5wtu9zOEGL .section-0 rect,#mermaid-svg-oa6iKN5wtu9zOEGL .section-0 path,#mermaid-svg-oa6iKN5wtu9zOEGL .section-0 circle,#mermaid-svg-oa6iKN5wtu9zOEGL .section-0 polygon,#mermaid-svg-oa6iKN5wtu9zOEGL .section-0 path{fill:hsl(60, 100%, 73.5294117647%);}#mermaid-svg-oa6iKN5wtu9zOEGL .section-0 text{fill:black;}#mermaid-svg-oa6iKN5wtu9zOEGL .node-icon-0{font-size:40px;color:black;}#mermaid-svg-oa6iKN5wtu9zOEGL .section-edge-0{stroke:hsl(60, 100%, 73.5294117647%);}#mermaid-svg-oa6iKN5wtu9zOEGL .edge-depth-0{stroke-width:14;}#mermaid-svg-oa6iKN5wtu9zOEGL .section-0 line{stroke:hsl(240, 100%, 83.5294117647%);stroke-width:3;}#mermaid-svg-oa6iKN5wtu9zOEGL .disabled,#mermaid-svg-oa6iKN5wtu9zOEGL .disabled circle,#mermaid-svg-oa6iKN5wtu9zOEGL .disabled text{fill:lightgray;}#mermaid-svg-oa6iKN5wtu9zOEGL .disabled text{fill:#efefef;}#mermaid-svg-oa6iKN5wtu9zOEGL .section-1 rect,#mermaid-svg-oa6iKN5wtu9zOEGL .section-1 path,#mermaid-svg-oa6iKN5wtu9zOEGL .section-1 circle,#mermaid-svg-oa6iKN5wtu9zOEGL .section-1 polygon,#mermaid-svg-oa6iKN5wtu9zOEGL .section-1 path{fill:hsl(80, 100%, 76.2745098039%);}#mermaid-svg-oa6iKN5wtu9zOEGL .section-1 text{fill:black;}#mermaid-svg-oa6iKN5wtu9zOEGL .node-icon-1{font-size:40px;color:black;}#mermaid-svg-oa6iKN5wtu9zOEGL .section-edge-1{stroke:hsl(80, 100%, 76.2745098039%);}#mermaid-svg-oa6iKN5wtu9zOEGL .edge-depth-1{stroke-width:11;}#mermaid-svg-oa6iKN5wtu9zOEGL .section-1 line{stroke:hsl(260, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-oa6iKN5wtu9zOEGL .disabled,#mermaid-svg-oa6iKN5wtu9zOEGL .disabled circle,#mermaid-svg-oa6iKN5wtu9zOEGL .disabled text{fill:lightgray;}#mermaid-svg-oa6iKN5wtu9zOEGL .disabled text{fill:#efefef;}#mermaid-svg-oa6iKN5wtu9zOEGL .section-2 rect,#mermaid-svg-oa6iKN5wtu9zOEGL .section-2 path,#mermaid-svg-oa6iKN5wtu9zOEGL .section-2 circle,#mermaid-svg-oa6iKN5wtu9zOEGL .section-2 polygon,#mermaid-svg-oa6iKN5wtu9zOEGL .section-2 path{fill:hsl(270, 100%, 76.2745098039%);}#mermaid-svg-oa6iKN5wtu9zOEGL .section-2 text{fill:#ffffff;}#mermaid-svg-oa6iKN5wtu9zOEGL .node-icon-2{font-size:40px;color:#ffffff;}#mermaid-svg-oa6iKN5wtu9zOEGL .section-edge-2{stroke:hsl(270, 100%, 76.2745098039%);}#mermaid-svg-oa6iKN5wtu9zOEGL .edge-depth-2{stroke-width:8;}#mermaid-svg-oa6iKN5wtu9zOEGL .section-2 line{stroke:hsl(90, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-oa6iKN5wtu9zOEGL .disabled,#mermaid-svg-oa6iKN5wtu9zOEGL .disabled circle,#mermaid-svg-oa6iKN5wtu9zOEGL .disabled text{fill:lightgray;}#mermaid-svg-oa6iKN5wtu9zOEGL .disabled text{fill:#efefef;}#mermaid-svg-oa6iKN5wtu9zOEGL .section-3 rect,#mermaid-svg-oa6iKN5wtu9zOEGL .section-3 path,#mermaid-svg-oa6iKN5wtu9zOEGL .section-3 circle,#mermaid-svg-oa6iKN5wtu9zOEGL .section-3 polygon,#mermaid-svg-oa6iKN5wtu9zOEGL .section-3 path{fill:hsl(300, 100%, 76.2745098039%);}#mermaid-svg-oa6iKN5wtu9zOEGL .section-3 text{fill:black;}#mermaid-svg-oa6iKN5wtu9zOEGL .node-icon-3{font-size:40px;color:black;}#mermaid-svg-oa6iKN5wtu9zOEGL .section-edge-3{stroke:hsl(300, 100%, 76.2745098039%);}#mermaid-svg-oa6iKN5wtu9zOEGL .edge-depth-3{stroke-width:5;}#mermaid-svg-oa6iKN5wtu9zOEGL .section-3 line{stroke:hsl(120, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-oa6iKN5wtu9zOEGL .disabled,#mermaid-svg-oa6iKN5wtu9zOEGL .disabled circle,#mermaid-svg-oa6iKN5wtu9zOEGL .disabled text{fill:lightgray;}#mermaid-svg-oa6iKN5wtu9zOEGL .disabled text{fill:#efefef;}#mermaid-svg-oa6iKN5wtu9zOEGL .section-4 rect,#mermaid-svg-oa6iKN5wtu9zOEGL .section-4 path,#mermaid-svg-oa6iKN5wtu9zOEGL .section-4 circle,#mermaid-svg-oa6iKN5wtu9zOEGL .section-4 polygon,#mermaid-svg-oa6iKN5wtu9zOEGL .section-4 path{fill:hsl(330, 100%, 76.2745098039%);}#mermaid-svg-oa6iKN5wtu9zOEGL .section-4 text{fill:black;}#mermaid-svg-oa6iKN5wtu9zOEGL .node-icon-4{font-size:40px;color:black;}#mermaid-svg-oa6iKN5wtu9zOEGL .section-edge-4{stroke:hsl(330, 100%, 76.2745098039%);}#mermaid-svg-oa6iKN5wtu9zOEGL .edge-depth-4{stroke-width:2;}#mermaid-svg-oa6iKN5wtu9zOEGL .section-4 line{stroke:hsl(150, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-oa6iKN5wtu9zOEGL .disabled,#mermaid-svg-oa6iKN5wtu9zOEGL .disabled circle,#mermaid-svg-oa6iKN5wtu9zOEGL .disabled text{fill:lightgray;}#mermaid-svg-oa6iKN5wtu9zOEGL .disabled text{fill:#efefef;}#mermaid-svg-oa6iKN5wtu9zOEGL .section-5 rect,#mermaid-svg-oa6iKN5wtu9zOEGL .section-5 path,#mermaid-svg-oa6iKN5wtu9zOEGL .section-5 circle,#mermaid-svg-oa6iKN5wtu9zOEGL .section-5 polygon,#mermaid-svg-oa6iKN5wtu9zOEGL .section-5 path{fill:hsl(0, 100%, 76.2745098039%);}#mermaid-svg-oa6iKN5wtu9zOEGL .section-5 text{fill:black;}#mermaid-svg-oa6iKN5wtu9zOEGL .node-icon-5{font-size:40px;color:black;}#mermaid-svg-oa6iKN5wtu9zOEGL .section-edge-5{stroke:hsl(0, 100%, 76.2745098039%);}#mermaid-svg-oa6iKN5wtu9zOEGL .edge-depth-5{stroke-width:-1;}#mermaid-svg-oa6iKN5wtu9zOEGL .section-5 line{stroke:hsl(180, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-oa6iKN5wtu9zOEGL .disabled,#mermaid-svg-oa6iKN5wtu9zOEGL .disabled circle,#mermaid-svg-oa6iKN5wtu9zOEGL .disabled text{fill:lightgray;}#mermaid-svg-oa6iKN5wtu9zOEGL .disabled text{fill:#efefef;}#mermaid-svg-oa6iKN5wtu9zOEGL .section-6 rect,#mermaid-svg-oa6iKN5wtu9zOEGL .section-6 path,#mermaid-svg-oa6iKN5wtu9zOEGL .section-6 circle,#mermaid-svg-oa6iKN5wtu9zOEGL .section-6 polygon,#mermaid-svg-oa6iKN5wtu9zOEGL .section-6 path{fill:hsl(30, 100%, 76.2745098039%);}#mermaid-svg-oa6iKN5wtu9zOEGL .section-6 text{fill:black;}#mermaid-svg-oa6iKN5wtu9zOEGL .node-icon-6{font-size:40px;color:black;}#mermaid-svg-oa6iKN5wtu9zOEGL .section-edge-6{stroke:hsl(30, 100%, 76.2745098039%);}#mermaid-svg-oa6iKN5wtu9zOEGL .edge-depth-6{stroke-width:-4;}#mermaid-svg-oa6iKN5wtu9zOEGL .section-6 line{stroke:hsl(210, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-oa6iKN5wtu9zOEGL .disabled,#mermaid-svg-oa6iKN5wtu9zOEGL .disabled circle,#mermaid-svg-oa6iKN5wtu9zOEGL .disabled text{fill:lightgray;}#mermaid-svg-oa6iKN5wtu9zOEGL .disabled text{fill:#efefef;}#mermaid-svg-oa6iKN5wtu9zOEGL .section-7 rect,#mermaid-svg-oa6iKN5wtu9zOEGL .section-7 path,#mermaid-svg-oa6iKN5wtu9zOEGL .section-7 circle,#mermaid-svg-oa6iKN5wtu9zOEGL .section-7 polygon,#mermaid-svg-oa6iKN5wtu9zOEGL .section-7 path{fill:hsl(90, 100%, 76.2745098039%);}#mermaid-svg-oa6iKN5wtu9zOEGL .section-7 text{fill:black;}#mermaid-svg-oa6iKN5wtu9zOEGL .node-icon-7{font-size:40px;color:black;}#mermaid-svg-oa6iKN5wtu9zOEGL .section-edge-7{stroke:hsl(90, 100%, 76.2745098039%);}#mermaid-svg-oa6iKN5wtu9zOEGL .edge-depth-7{stroke-width:-7;}#mermaid-svg-oa6iKN5wtu9zOEGL .section-7 line{stroke:hsl(270, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-oa6iKN5wtu9zOEGL .disabled,#mermaid-svg-oa6iKN5wtu9zOEGL .disabled circle,#mermaid-svg-oa6iKN5wtu9zOEGL .disabled text{fill:lightgray;}#mermaid-svg-oa6iKN5wtu9zOEGL .disabled text{fill:#efefef;}#mermaid-svg-oa6iKN5wtu9zOEGL .section-8 rect,#mermaid-svg-oa6iKN5wtu9zOEGL .section-8 path,#mermaid-svg-oa6iKN5wtu9zOEGL .section-8 circle,#mermaid-svg-oa6iKN5wtu9zOEGL .section-8 polygon,#mermaid-svg-oa6iKN5wtu9zOEGL .section-8 path{fill:hsl(150, 100%, 76.2745098039%);}#mermaid-svg-oa6iKN5wtu9zOEGL .section-8 text{fill:black;}#mermaid-svg-oa6iKN5wtu9zOEGL .node-icon-8{font-size:40px;color:black;}#mermaid-svg-oa6iKN5wtu9zOEGL .section-edge-8{stroke:hsl(150, 100%, 76.2745098039%);}#mermaid-svg-oa6iKN5wtu9zOEGL .edge-depth-8{stroke-width:-10;}#mermaid-svg-oa6iKN5wtu9zOEGL .section-8 line{stroke:hsl(330, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-oa6iKN5wtu9zOEGL .disabled,#mermaid-svg-oa6iKN5wtu9zOEGL .disabled circle,#mermaid-svg-oa6iKN5wtu9zOEGL .disabled text{fill:lightgray;}#mermaid-svg-oa6iKN5wtu9zOEGL .disabled text{fill:#efefef;}#mermaid-svg-oa6iKN5wtu9zOEGL .section-9 rect,#mermaid-svg-oa6iKN5wtu9zOEGL .section-9 path,#mermaid-svg-oa6iKN5wtu9zOEGL .section-9 circle,#mermaid-svg-oa6iKN5wtu9zOEGL .section-9 polygon,#mermaid-svg-oa6iKN5wtu9zOEGL .section-9 path{fill:hsl(180, 100%, 76.2745098039%);}#mermaid-svg-oa6iKN5wtu9zOEGL .section-9 text{fill:black;}#mermaid-svg-oa6iKN5wtu9zOEGL .node-icon-9{font-size:40px;color:black;}#mermaid-svg-oa6iKN5wtu9zOEGL .section-edge-9{stroke:hsl(180, 100%, 76.2745098039%);}#mermaid-svg-oa6iKN5wtu9zOEGL .edge-depth-9{stroke-width:-13;}#mermaid-svg-oa6iKN5wtu9zOEGL .section-9 line{stroke:hsl(0, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-oa6iKN5wtu9zOEGL .disabled,#mermaid-svg-oa6iKN5wtu9zOEGL .disabled circle,#mermaid-svg-oa6iKN5wtu9zOEGL .disabled text{fill:lightgray;}#mermaid-svg-oa6iKN5wtu9zOEGL .disabled text{fill:#efefef;}#mermaid-svg-oa6iKN5wtu9zOEGL .section-10 rect,#mermaid-svg-oa6iKN5wtu9zOEGL .section-10 path,#mermaid-svg-oa6iKN5wtu9zOEGL .section-10 circle,#mermaid-svg-oa6iKN5wtu9zOEGL .section-10 polygon,#mermaid-svg-oa6iKN5wtu9zOEGL .section-10 path{fill:hsl(210, 100%, 76.2745098039%);}#mermaid-svg-oa6iKN5wtu9zOEGL .section-10 text{fill:black;}#mermaid-svg-oa6iKN5wtu9zOEGL .node-icon-10{font-size:40px;color:black;}#mermaid-svg-oa6iKN5wtu9zOEGL .section-edge-10{stroke:hsl(210, 100%, 76.2745098039%);}#mermaid-svg-oa6iKN5wtu9zOEGL .edge-depth-10{stroke-width:-16;}#mermaid-svg-oa6iKN5wtu9zOEGL .section-10 line{stroke:hsl(30, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-oa6iKN5wtu9zOEGL .disabled,#mermaid-svg-oa6iKN5wtu9zOEGL .disabled circle,#mermaid-svg-oa6iKN5wtu9zOEGL .disabled text{fill:lightgray;}#mermaid-svg-oa6iKN5wtu9zOEGL .disabled text{fill:#efefef;}#mermaid-svg-oa6iKN5wtu9zOEGL .section-root rect,#mermaid-svg-oa6iKN5wtu9zOEGL .section-root path,#mermaid-svg-oa6iKN5wtu9zOEGL .section-root circle,#mermaid-svg-oa6iKN5wtu9zOEGL .section-root polygon{fill:hsl(240, 100%, 46.2745098039%);}#mermaid-svg-oa6iKN5wtu9zOEGL .section-root text{fill:#ffffff;}#mermaid-svg-oa6iKN5wtu9zOEGL .section-root span{color:#ffffff;}#mermaid-svg-oa6iKN5wtu9zOEGL .section-2 span{color:#ffffff;}#mermaid-svg-oa6iKN5wtu9zOEGL .icon-container{height:100%;display:flex;justify-content:center;align-items:center;}#mermaid-svg-oa6iKN5wtu9zOEGL .edge{fill:none;}#mermaid-svg-oa6iKN5wtu9zOEGL .mindmap-node-label{dy:1em;alignment-baseline:middle;text-anchor:middle;dominant-baseline:middle;text-align:center;}#mermaid-svg-oa6iKN5wtu9zOEGL :root{--mermaid-font-family:"trebuchet ms",verdana,arial,sans-serif;} 稳态指标
业务指标
订单成功率
支付成功率
用户登录率
请求完成率
技术指标
P99 响应时间
错误率
CPU使用率
内存使用率
基础设施
服务可用性
连接数
磁盘IO
网络延迟
可观测性指标
日志写入速率
Trace采集率
告警延迟

💡 稳态指标的 SLI/SLO/SLA

#mermaid-svg-SDNB6SvUFkqxmCvK{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-SDNB6SvUFkqxmCvK .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-SDNB6SvUFkqxmCvK .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-SDNB6SvUFkqxmCvK .error-icon{fill:#552222;}#mermaid-svg-SDNB6SvUFkqxmCvK .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-SDNB6SvUFkqxmCvK .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-SDNB6SvUFkqxmCvK .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-SDNB6SvUFkqxmCvK .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-SDNB6SvUFkqxmCvK .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-SDNB6SvUFkqxmCvK .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-SDNB6SvUFkqxmCvK .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-SDNB6SvUFkqxmCvK .marker{fill:#333333;stroke:#333333;}#mermaid-svg-SDNB6SvUFkqxmCvK .marker.cross{stroke:#333333;}#mermaid-svg-SDNB6SvUFkqxmCvK svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-SDNB6SvUFkqxmCvK p{margin:0;}#mermaid-svg-SDNB6SvUFkqxmCvK .label{font-family:"trebuchet ms",verdana,arial,sans-serif;color:#333;}#mermaid-svg-SDNB6SvUFkqxmCvK .cluster-label text{fill:#333;}#mermaid-svg-SDNB6SvUFkqxmCvK .cluster-label span{color:#333;}#mermaid-svg-SDNB6SvUFkqxmCvK .cluster-label span p{background-color:transparent;}#mermaid-svg-SDNB6SvUFkqxmCvK .label text,#mermaid-svg-SDNB6SvUFkqxmCvK span{fill:#333;color:#333;}#mermaid-svg-SDNB6SvUFkqxmCvK .node rect,#mermaid-svg-SDNB6SvUFkqxmCvK .node circle,#mermaid-svg-SDNB6SvUFkqxmCvK .node ellipse,#mermaid-svg-SDNB6SvUFkqxmCvK .node polygon,#mermaid-svg-SDNB6SvUFkqxmCvK .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-SDNB6SvUFkqxmCvK .rough-node .label text,#mermaid-svg-SDNB6SvUFkqxmCvK .node .label text,#mermaid-svg-SDNB6SvUFkqxmCvK .image-shape .label,#mermaid-svg-SDNB6SvUFkqxmCvK .icon-shape .label{text-anchor:middle;}#mermaid-svg-SDNB6SvUFkqxmCvK .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-SDNB6SvUFkqxmCvK .rough-node .label,#mermaid-svg-SDNB6SvUFkqxmCvK .node .label,#mermaid-svg-SDNB6SvUFkqxmCvK .image-shape .label,#mermaid-svg-SDNB6SvUFkqxmCvK .icon-shape .label{text-align:center;}#mermaid-svg-SDNB6SvUFkqxmCvK .node.clickable{cursor:pointer;}#mermaid-svg-SDNB6SvUFkqxmCvK .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-SDNB6SvUFkqxmCvK .arrowheadPath{fill:#333333;}#mermaid-svg-SDNB6SvUFkqxmCvK .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-SDNB6SvUFkqxmCvK .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-SDNB6SvUFkqxmCvK .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-SDNB6SvUFkqxmCvK .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-SDNB6SvUFkqxmCvK .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-SDNB6SvUFkqxmCvK .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-SDNB6SvUFkqxmCvK .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-SDNB6SvUFkqxmCvK .cluster text{fill:#333;}#mermaid-svg-SDNB6SvUFkqxmCvK .cluster span{color:#333;}#mermaid-svg-SDNB6SvUFkqxmCvK div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-SDNB6SvUFkqxmCvK .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-SDNB6SvUFkqxmCvK rect.text{fill:none;stroke-width:0;}#mermaid-svg-SDNB6SvUFkqxmCvK .icon-shape,#mermaid-svg-SDNB6SvUFkqxmCvK .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-SDNB6SvUFkqxmCvK .icon-shape p,#mermaid-svg-SDNB6SvUFkqxmCvK .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-SDNB6SvUFkqxmCvK .icon-shape .label rect,#mermaid-svg-SDNB6SvUFkqxmCvK .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-SDNB6SvUFkqxmCvK .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-SDNB6SvUFkqxmCvK .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-SDNB6SvUFkqxmCvK :root{--mermaid-font-family:"trebuchet ms",verdana,arial,sans-serif;} 实际P99延迟=150ms
P99延迟<200ms
P99延迟<300ms

否则赔偿
SLI

服务等级指标

实际测量的指标值
SLO

服务等级目标

我们设定的目标值
SLA

服务等级协议

对外承诺的协议值

例子：

  • SLI: 过去 5 分钟的平均响应时间是 120ms
  • SLO: 响应时间应低于 200ms（我们内部的目标）
  • SLA: 响应时间应低于 500ms（否则赔钱给客户）

混沌测试的最终目的：确保 SLI 总是满足 SLO，即使系统出了故障。

第5章 爆炸半径------守住你的安全线

💣 为什么叫"爆炸半径"？

想象你在一个雷区里排雷。你要引爆一颗地雷前，你需要知道：

  1. 爆炸范围有多大？
  2. 会不会炸到旁边的弹药库？
  3. 周围的人安全吗？

混沌测试的"爆炸半径"就是这个意思：

故障注入的影响范围。

🔒 控制爆炸半径的实战技巧

#mermaid-svg-pLFQRdhUJa9m19jl{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-pLFQRdhUJa9m19jl .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-pLFQRdhUJa9m19jl .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-pLFQRdhUJa9m19jl .error-icon{fill:#552222;}#mermaid-svg-pLFQRdhUJa9m19jl .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-pLFQRdhUJa9m19jl .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-pLFQRdhUJa9m19jl .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-pLFQRdhUJa9m19jl .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-pLFQRdhUJa9m19jl .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-pLFQRdhUJa9m19jl .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-pLFQRdhUJa9m19jl .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-pLFQRdhUJa9m19jl .marker{fill:#333333;stroke:#333333;}#mermaid-svg-pLFQRdhUJa9m19jl .marker.cross{stroke:#333333;}#mermaid-svg-pLFQRdhUJa9m19jl svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-pLFQRdhUJa9m19jl p{margin:0;}#mermaid-svg-pLFQRdhUJa9m19jl .label{font-family:"trebuchet ms",verdana,arial,sans-serif;color:#333;}#mermaid-svg-pLFQRdhUJa9m19jl .cluster-label text{fill:#333;}#mermaid-svg-pLFQRdhUJa9m19jl .cluster-label span{color:#333;}#mermaid-svg-pLFQRdhUJa9m19jl .cluster-label span p{background-color:transparent;}#mermaid-svg-pLFQRdhUJa9m19jl .label text,#mermaid-svg-pLFQRdhUJa9m19jl span{fill:#333;color:#333;}#mermaid-svg-pLFQRdhUJa9m19jl .node rect,#mermaid-svg-pLFQRdhUJa9m19jl .node circle,#mermaid-svg-pLFQRdhUJa9m19jl .node ellipse,#mermaid-svg-pLFQRdhUJa9m19jl .node polygon,#mermaid-svg-pLFQRdhUJa9m19jl .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-pLFQRdhUJa9m19jl .rough-node .label text,#mermaid-svg-pLFQRdhUJa9m19jl .node .label text,#mermaid-svg-pLFQRdhUJa9m19jl .image-shape .label,#mermaid-svg-pLFQRdhUJa9m19jl .icon-shape .label{text-anchor:middle;}#mermaid-svg-pLFQRdhUJa9m19jl .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-pLFQRdhUJa9m19jl .rough-node .label,#mermaid-svg-pLFQRdhUJa9m19jl .node .label,#mermaid-svg-pLFQRdhUJa9m19jl .image-shape .label,#mermaid-svg-pLFQRdhUJa9m19jl .icon-shape .label{text-align:center;}#mermaid-svg-pLFQRdhUJa9m19jl .node.clickable{cursor:pointer;}#mermaid-svg-pLFQRdhUJa9m19jl .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-pLFQRdhUJa9m19jl .arrowheadPath{fill:#333333;}#mermaid-svg-pLFQRdhUJa9m19jl .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-pLFQRdhUJa9m19jl .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-pLFQRdhUJa9m19jl .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-pLFQRdhUJa9m19jl .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-pLFQRdhUJa9m19jl .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-pLFQRdhUJa9m19jl .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-pLFQRdhUJa9m19jl .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-pLFQRdhUJa9m19jl .cluster text{fill:#333;}#mermaid-svg-pLFQRdhUJa9m19jl .cluster span{color:#333;}#mermaid-svg-pLFQRdhUJa9m19jl div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-pLFQRdhUJa9m19jl .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-pLFQRdhUJa9m19jl rect.text{fill:none;stroke-width:0;}#mermaid-svg-pLFQRdhUJa9m19jl .icon-shape,#mermaid-svg-pLFQRdhUJa9m19jl .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-pLFQRdhUJa9m19jl .icon-shape p,#mermaid-svg-pLFQRdhUJa9m19jl .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-pLFQRdhUJa9m19jl .icon-shape .label rect,#mermaid-svg-pLFQRdhUJa9m19jl .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-pLFQRdhUJa9m19jl .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-pLFQRdhUJa9m19jl .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-pLFQRdhUJa9m19jl :root{--mermaid-font-family:"trebuchet ms",verdana,arial,sans-serif;} 没有
有
开始混沌实验
控制爆炸半径?
❌ 灾难!

整个集群挂了
✅ 安全
限制作用范围

只影响特定服务/实例
限制持续时间

设置自动恢复
限制流量类型

只影响影子流量
灰度发布实验

按比例逐步扩大

爆炸半径控制矩阵：

控制维度 开发环境 测试环境 预发布 生产环境
实例数量 全部 50% 20% 1-5%
持续时间 不限 30min 10min 5min
流量类型 模拟流量 测试流量 影子流量 真实流量
自动回滚 可选 必须 必须 必须

🛡️ 熔断器模式------系统自带的"爆炸半径控制"

#mermaid-svg-T8NGUEkuf5HmOJMv{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-T8NGUEkuf5HmOJMv .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-T8NGUEkuf5HmOJMv .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-T8NGUEkuf5HmOJMv .error-icon{fill:#552222;}#mermaid-svg-T8NGUEkuf5HmOJMv .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-T8NGUEkuf5HmOJMv .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-T8NGUEkuf5HmOJMv .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-T8NGUEkuf5HmOJMv .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-T8NGUEkuf5HmOJMv .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-T8NGUEkuf5HmOJMv .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-T8NGUEkuf5HmOJMv .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-T8NGUEkuf5HmOJMv .marker{fill:#333333;stroke:#333333;}#mermaid-svg-T8NGUEkuf5HmOJMv .marker.cross{stroke:#333333;}#mermaid-svg-T8NGUEkuf5HmOJMv svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-T8NGUEkuf5HmOJMv p{margin:0;}#mermaid-svg-T8NGUEkuf5HmOJMv .label{font-family:"trebuchet ms",verdana,arial,sans-serif;color:#333;}#mermaid-svg-T8NGUEkuf5HmOJMv .cluster-label text{fill:#333;}#mermaid-svg-T8NGUEkuf5HmOJMv .cluster-label span{color:#333;}#mermaid-svg-T8NGUEkuf5HmOJMv .cluster-label span p{background-color:transparent;}#mermaid-svg-T8NGUEkuf5HmOJMv .label text,#mermaid-svg-T8NGUEkuf5HmOJMv span{fill:#333;color:#333;}#mermaid-svg-T8NGUEkuf5HmOJMv .node rect,#mermaid-svg-T8NGUEkuf5HmOJMv .node circle,#mermaid-svg-T8NGUEkuf5HmOJMv .node ellipse,#mermaid-svg-T8NGUEkuf5HmOJMv .node polygon,#mermaid-svg-T8NGUEkuf5HmOJMv .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-T8NGUEkuf5HmOJMv .rough-node .label text,#mermaid-svg-T8NGUEkuf5HmOJMv .node .label text,#mermaid-svg-T8NGUEkuf5HmOJMv .image-shape .label,#mermaid-svg-T8NGUEkuf5HmOJMv .icon-shape .label{text-anchor:middle;}#mermaid-svg-T8NGUEkuf5HmOJMv .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-T8NGUEkuf5HmOJMv .rough-node .label,#mermaid-svg-T8NGUEkuf5HmOJMv .node .label,#mermaid-svg-T8NGUEkuf5HmOJMv .image-shape .label,#mermaid-svg-T8NGUEkuf5HmOJMv .icon-shape .label{text-align:center;}#mermaid-svg-T8NGUEkuf5HmOJMv .node.clickable{cursor:pointer;}#mermaid-svg-T8NGUEkuf5HmOJMv .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-T8NGUEkuf5HmOJMv .arrowheadPath{fill:#333333;}#mermaid-svg-T8NGUEkuf5HmOJMv .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-T8NGUEkuf5HmOJMv .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-T8NGUEkuf5HmOJMv .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-T8NGUEkuf5HmOJMv .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-T8NGUEkuf5HmOJMv .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-T8NGUEkuf5HmOJMv .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-T8NGUEkuf5HmOJMv .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-T8NGUEkuf5HmOJMv .cluster text{fill:#333;}#mermaid-svg-T8NGUEkuf5HmOJMv .cluster span{color:#333;}#mermaid-svg-T8NGUEkuf5HmOJMv div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-T8NGUEkuf5HmOJMv .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-T8NGUEkuf5HmOJMv rect.text{fill:none;stroke-width:0;}#mermaid-svg-T8NGUEkuf5HmOJMv .icon-shape,#mermaid-svg-T8NGUEkuf5HmOJMv .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-T8NGUEkuf5HmOJMv .icon-shape p,#mermaid-svg-T8NGUEkuf5HmOJMv .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-T8NGUEkuf5HmOJMv .icon-shape .label rect,#mermaid-svg-T8NGUEkuf5HmOJMv .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-T8NGUEkuf5HmOJMv .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-T8NGUEkuf5HmOJMv .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-T8NGUEkuf5HmOJMv :root{--mermaid-font-family:"trebuchet ms",verdana,arial,sans-serif;} 半开状态
试探请求
1个请求
成功
失败
客户端
熔断器

半开
服务B
熔断器关闭

恢复正常
熔断器打开

继续保护
熔断状态
请求
直接返回

错误回调
不转发请求
客户端
熔断器

已打开!
服务B

已故障
正常状态
请求
响应
客户端
服务B

第6章 科学实验方法------混沌测试的「实验设计」

🔬 混沌实验 = 科学实验

还记得初中科学课上学过的科学方法吗？

复制代码 
1. 观察现象
2. 提出假设
3. 设计实验
4. 执行实验
5. 分析结果
6. 得出结论

混沌实验完全遵循这套方法！

📋 混沌实验模板

每次混沌实验都应该是这样设计的：

yaml 复制代码 
混沌实验模板
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
实验ID: CE-2024-001
实验名称: 验证订单服务在数据库故障时的韧性

假设:
  杀掉订单数据库的一个只读副本不会影响下订单成功
    
稳态指标:
  - 订单成功率 > 99.9%
  - 下单响应时间 P99 < 2s
  - 错误率 < 0.1%

实验步骤:
  1. 记录稳态指标基线
  2. 注入故障: 关闭数据库只读副本
  3. 保持故障 5 分钟
  4. 持续监控稳态指标
  5. 自动恢复故障
  6. 对比实验前后指标

爆炸半径:
  - 影响范围: 订单服务的只读查询
  - 不影响: 订单写入、支付、库存
  - 自动回滚: 5分钟后自动恢复

预期结果:
  ✅ 假设成立 → 系统正常运行，无影响
  ❌ 假设不成立 → 需要改进系统韧性
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

🎮 动手练习------设计你的第一个混沌实验

场景： 你有一个微服务 "user-service"，部署在 K8s 集群中，有 3 个副本。你想测试如果突然杀掉 1 个 Pod，系统是否还能正常工作。

暂停一下，自己先想 30 秒：

  1. 你的稳态假设是什么？
  2. 你的稳态指标是什么？
  3. 爆炸半径怎么控制？

想好了吗？对比一下我的答案：

假设： 杀死 user-service 的一个 Pod 后，由于还有 2 个副本，用户登录成功率仍 > 99%。

指标： 用户登录成功率、登录页面响应时间 P99、错误率

半径： 只影响 user-service 的这一个 Pod，不影响其他服务

第三部分：基础设施层混沌测试

第7章 CPU 压力------让服务器"跑起来"

🧠 概念理解

问题： 当服务器的 CPU 使用率达到 100% 时，你的应用会怎样？

  • 请求变慢？
  • 超时？
  • 完全不可用？
  • 还是...毫无影响（因为应用服务器和 CPU 密集型任务分离了）？

混沌实验： 人为给服务器增加 CPU 负载，观察系统行为。

💻 Demo: 使用 ChaosBlade 注入 CPU 压力

ChaosBlade 是阿里开源的一款混沌工程工具。让我们看看怎么用它来注入 CPU 故障：

bash 复制代码 
# 1. 下载 ChaosBlade
curl -O -L https://github.com/chaosblade-io/chaosblade/releases/download/v1.7.3/chaosblade-1.7.3-linux-amd64.tar.gz
tar -xzf chaosblade-1.7.3-linux-amd64.tar.gz
cd chaosblade-1.7.3

# 2. 查看 CPU 使用情况
top

# 3. 注入 CPU 满载（占用 2 个核）
./blade create cpu fullload --cpu-percent 100 --cpu-count 2

# 输出示例:
# {"code":200,"success":true,"result":"b0e89f5c5f9e4f1b"}

# 4. 查看 CPU 使用率（你应该会看到 CPU 瞬间飙到 100%）
top

# 5. 销毁实验
./blade destroy b0e89f5c5f9e4f1b

📊 实验前后的效果对比

#mermaid-svg-OCGFIL4aeTOLapLh{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-OCGFIL4aeTOLapLh .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-OCGFIL4aeTOLapLh .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-OCGFIL4aeTOLapLh .error-icon{fill:#552222;}#mermaid-svg-OCGFIL4aeTOLapLh .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-OCGFIL4aeTOLapLh .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-OCGFIL4aeTOLapLh .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-OCGFIL4aeTOLapLh .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-OCGFIL4aeTOLapLh .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-OCGFIL4aeTOLapLh .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-OCGFIL4aeTOLapLh .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-OCGFIL4aeTOLapLh .marker{fill:#333333;stroke:#333333;}#mermaid-svg-OCGFIL4aeTOLapLh .marker.cross{stroke:#333333;}#mermaid-svg-OCGFIL4aeTOLapLh svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-OCGFIL4aeTOLapLh p{margin:0;}#mermaid-svg-OCGFIL4aeTOLapLh :root{--mermaid-font-family:"trebuchet ms",verdana,arial,sans-serif;} CPU 故障注入前后系统表现 实验前实验中恢复后 500450400350300250200150100500 请求延迟(ms)

🧪 思考练习

你的订单服务部署在 4 核 8G 的服务器上。你想测试如果某个 Pod 的 CPU 被打满，订单服务会怎样？

  1. 你的稳态指标是什么？
  2. 除了 CPU，还需要监控什么？
  3. 如果响应时间超标，你有什么应对措施？

第8章 内存压力------内存泄漏的"疫苗"

🧠 概念理解

内存泄漏是生产环境最常见的故障之一。但通常只有在流量高峰期才会暴露。

混沌测试帮你提前发现内存问题：主动给系统增加内存压力，看看应用是否会 OOM（内存溢出），以及 K8s 的自动恢复机制是否正常工作。

💻 Demo: 使用 ChaosBlade 注入内存压力

bash 复制代码 
# 注入内存压力（占用 2GB 内存）
./blade create mem load --mem-percent 80 --mode ram

# 监控内存使用
free -h
# 输出:
#               total        used        free      shared  buff/cache   available
# Mem:           15G         12G          1G         1G          2G          2G

# 检查应用是否受影响
curl http://your-service/health

# 恢复
./blade destroy [UID]

📊 内存故障的危害链

#mermaid-svg-xTOgOjqPwca42O5I{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-xTOgOjqPwca42O5I .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-xTOgOjqPwca42O5I .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-xTOgOjqPwca42O5I .error-icon{fill:#552222;}#mermaid-svg-xTOgOjqPwca42O5I .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-xTOgOjqPwca42O5I .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-xTOgOjqPwca42O5I .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-xTOgOjqPwca42O5I .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-xTOgOjqPwca42O5I .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-xTOgOjqPwca42O5I .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-xTOgOjqPwca42O5I .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-xTOgOjqPwca42O5I .marker{fill:#333333;stroke:#333333;}#mermaid-svg-xTOgOjqPwca42O5I .marker.cross{stroke:#333333;}#mermaid-svg-xTOgOjqPwca42O5I svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-xTOgOjqPwca42O5I p{margin:0;}#mermaid-svg-xTOgOjqPwca42O5I .label{font-family:"trebuchet ms",verdana,arial,sans-serif;color:#333;}#mermaid-svg-xTOgOjqPwca42O5I .cluster-label text{fill:#333;}#mermaid-svg-xTOgOjqPwca42O5I .cluster-label span{color:#333;}#mermaid-svg-xTOgOjqPwca42O5I .cluster-label span p{background-color:transparent;}#mermaid-svg-xTOgOjqPwca42O5I .label text,#mermaid-svg-xTOgOjqPwca42O5I span{fill:#333;color:#333;}#mermaid-svg-xTOgOjqPwca42O5I .node rect,#mermaid-svg-xTOgOjqPwca42O5I .node circle,#mermaid-svg-xTOgOjqPwca42O5I .node ellipse,#mermaid-svg-xTOgOjqPwca42O5I .node polygon,#mermaid-svg-xTOgOjqPwca42O5I .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-xTOgOjqPwca42O5I .rough-node .label text,#mermaid-svg-xTOgOjqPwca42O5I .node .label text,#mermaid-svg-xTOgOjqPwca42O5I .image-shape .label,#mermaid-svg-xTOgOjqPwca42O5I .icon-shape .label{text-anchor:middle;}#mermaid-svg-xTOgOjqPwca42O5I .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-xTOgOjqPwca42O5I .rough-node .label,#mermaid-svg-xTOgOjqPwca42O5I .node .label,#mermaid-svg-xTOgOjqPwca42O5I .image-shape .label,#mermaid-svg-xTOgOjqPwca42O5I .icon-shape .label{text-align:center;}#mermaid-svg-xTOgOjqPwca42O5I .node.clickable{cursor:pointer;}#mermaid-svg-xTOgOjqPwca42O5I .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-xTOgOjqPwca42O5I .arrowheadPath{fill:#333333;}#mermaid-svg-xTOgOjqPwca42O5I .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-xTOgOjqPwca42O5I .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-xTOgOjqPwca42O5I .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-xTOgOjqPwca42O5I .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-xTOgOjqPwca42O5I .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-xTOgOjqPwca42O5I .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-xTOgOjqPwca42O5I .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-xTOgOjqPwca42O5I .cluster text{fill:#333;}#mermaid-svg-xTOgOjqPwca42O5I .cluster span{color:#333;}#mermaid-svg-xTOgOjqPwca42O5I div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-xTOgOjqPwca42O5I .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-xTOgOjqPwca42O5I rect.text{fill:none;stroke-width:0;}#mermaid-svg-xTOgOjqPwca42O5I .icon-shape,#mermaid-svg-xTOgOjqPwca42O5I .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-xTOgOjqPwca42O5I .icon-shape p,#mermaid-svg-xTOgOjqPwca42O5I .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-xTOgOjqPwca42O5I .icon-shape .label rect,#mermaid-svg-xTOgOjqPwca42O5I .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-xTOgOjqPwca42O5I .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-xTOgOjqPwca42O5I .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-xTOgOjqPwca42O5I :root{--mermaid-font-family:"trebuchet ms",verdana,arial,sans-serif;} 内存泄漏

应用不断消耗内存
GC 频繁

Full GC 时间变长
CPU 飙升

GC 占用大量CPU
响应变慢

请求超时
OOM

进程被 Kill
服务不可用

流量丢失

💻 Demo2: 使用 Python 脚本模拟内存泄漏

这个小脚本可以模拟一个"内存泄漏"的应用：

python 复制代码 
# memory_leak_simulator.py
# 警告：不要在生产环境运行这个脚本！
import time
import os

memory_hog = []

def leak_memory():
    """模拟内存泄漏：不断分配大对象，但"忘记"释放"""
    chunk_size = 10 * 1024 * 1024  # 10MB
    chunk_count = 0
    
    try:
        while True:
            chunk = bytearray(chunk_size)
            memory_hog.append(chunk)
            chunk_count += 1
            memory_mb = chunk_count * 10
            print(f"已分配 {memory_mb}MB, PID: {os.getpid()}")
            time.sleep(0.5)
    except MemoryError:
        print(f"❌ OOM! 系统无法分配更多内存")
        print(f"最终分配: {chunk_count * 10}MB")
    except KeyboardInterrupt:
        print(f"\n手动停止, 已分配: {chunk_count * 10}MB")

if __name__ == "__main__":
    print("🚀 内存泄漏模拟器启动...")
    print(f"当前进程 PID: {os.getpid()}")
    print("提示: 运行 'top' 或 'htop' 查看内存变化")
    print("-" * 40)
    leak_memory()

你能用这个脚本做什么？

  1. 在测试环境中运行
  2. 观察 K8s 的自动恢复机制
  3. 看看你的监控告警是否及时触发
  4. 验证限流和降级策略是否生效

第9章 网络故障------让分布式系统"失联"

🧠 概念理解

分布式系统最大的敌人不是 CPU 或内存，而是网络。网络可以以无数种方式出问题：
#mermaid-svg-2Q1d7a2dxhYZ5Vj5{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .error-icon{fill:#552222;}#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .marker{fill:#333333;stroke:#333333;}#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .marker.cross{stroke:#333333;}#mermaid-svg-2Q1d7a2dxhYZ5Vj5 svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-2Q1d7a2dxhYZ5Vj5 p{margin:0;}#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .edge{stroke-width:3;}#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .section--1 rect,#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .section--1 path,#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .section--1 circle,#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .section--1 polygon,#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .section--1 path{fill:hsl(240, 100%, 76.2745098039%);}#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .section--1 text{fill:#ffffff;}#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .node-icon--1{font-size:40px;color:#ffffff;}#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .section-edge--1{stroke:hsl(240, 100%, 76.2745098039%);}#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .edge-depth--1{stroke-width:17;}#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .section--1 line{stroke:hsl(60, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .disabled,#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .disabled circle,#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .disabled text{fill:lightgray;}#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .disabled text{fill:#efefef;}#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .section-0 rect,#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .section-0 path,#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .section-0 circle,#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .section-0 polygon,#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .section-0 path{fill:hsl(60, 100%, 73.5294117647%);}#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .section-0 text{fill:black;}#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .node-icon-0{font-size:40px;color:black;}#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .section-edge-0{stroke:hsl(60, 100%, 73.5294117647%);}#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .edge-depth-0{stroke-width:14;}#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .section-0 line{stroke:hsl(240, 100%, 83.5294117647%);stroke-width:3;}#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .disabled,#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .disabled circle,#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .disabled text{fill:lightgray;}#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .disabled text{fill:#efefef;}#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .section-1 rect,#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .section-1 path,#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .section-1 circle,#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .section-1 polygon,#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .section-1 path{fill:hsl(80, 100%, 76.2745098039%);}#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .section-1 text{fill:black;}#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .node-icon-1{font-size:40px;color:black;}#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .section-edge-1{stroke:hsl(80, 100%, 76.2745098039%);}#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .edge-depth-1{stroke-width:11;}#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .section-1 line{stroke:hsl(260, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .disabled,#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .disabled circle,#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .disabled text{fill:lightgray;}#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .disabled text{fill:#efefef;}#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .section-2 rect,#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .section-2 path,#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .section-2 circle,#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .section-2 polygon,#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .section-2 path{fill:hsl(270, 100%, 76.2745098039%);}#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .section-2 text{fill:#ffffff;}#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .node-icon-2{font-size:40px;color:#ffffff;}#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .section-edge-2{stroke:hsl(270, 100%, 76.2745098039%);}#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .edge-depth-2{stroke-width:8;}#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .section-2 line{stroke:hsl(90, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .disabled,#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .disabled circle,#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .disabled text{fill:lightgray;}#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .disabled text{fill:#efefef;}#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .section-3 rect,#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .section-3 path,#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .section-3 circle,#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .section-3 polygon,#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .section-3 path{fill:hsl(300, 100%, 76.2745098039%);}#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .section-3 text{fill:black;}#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .node-icon-3{font-size:40px;color:black;}#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .section-edge-3{stroke:hsl(300, 100%, 76.2745098039%);}#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .edge-depth-3{stroke-width:5;}#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .section-3 line{stroke:hsl(120, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .disabled,#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .disabled circle,#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .disabled text{fill:lightgray;}#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .disabled text{fill:#efefef;}#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .section-4 rect,#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .section-4 path,#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .section-4 circle,#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .section-4 polygon,#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .section-4 path{fill:hsl(330, 100%, 76.2745098039%);}#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .section-4 text{fill:black;}#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .node-icon-4{font-size:40px;color:black;}#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .section-edge-4{stroke:hsl(330, 100%, 76.2745098039%);}#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .edge-depth-4{stroke-width:2;}#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .section-4 line{stroke:hsl(150, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .disabled,#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .disabled circle,#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .disabled text{fill:lightgray;}#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .disabled text{fill:#efefef;}#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .section-5 rect,#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .section-5 path,#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .section-5 circle,#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .section-5 polygon,#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .section-5 path{fill:hsl(0, 100%, 76.2745098039%);}#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .section-5 text{fill:black;}#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .node-icon-5{font-size:40px;color:black;}#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .section-edge-5{stroke:hsl(0, 100%, 76.2745098039%);}#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .edge-depth-5{stroke-width:-1;}#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .section-5 line{stroke:hsl(180, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .disabled,#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .disabled circle,#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .disabled text{fill:lightgray;}#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .disabled text{fill:#efefef;}#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .section-6 rect,#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .section-6 path,#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .section-6 circle,#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .section-6 polygon,#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .section-6 path{fill:hsl(30, 100%, 76.2745098039%);}#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .section-6 text{fill:black;}#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .node-icon-6{font-size:40px;color:black;}#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .section-edge-6{stroke:hsl(30, 100%, 76.2745098039%);}#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .edge-depth-6{stroke-width:-4;}#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .section-6 line{stroke:hsl(210, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .disabled,#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .disabled circle,#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .disabled text{fill:lightgray;}#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .disabled text{fill:#efefef;}#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .section-7 rect,#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .section-7 path,#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .section-7 circle,#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .section-7 polygon,#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .section-7 path{fill:hsl(90, 100%, 76.2745098039%);}#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .section-7 text{fill:black;}#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .node-icon-7{font-size:40px;color:black;}#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .section-edge-7{stroke:hsl(90, 100%, 76.2745098039%);}#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .edge-depth-7{stroke-width:-7;}#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .section-7 line{stroke:hsl(270, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .disabled,#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .disabled circle,#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .disabled text{fill:lightgray;}#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .disabled text{fill:#efefef;}#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .section-8 rect,#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .section-8 path,#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .section-8 circle,#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .section-8 polygon,#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .section-8 path{fill:hsl(150, 100%, 76.2745098039%);}#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .section-8 text{fill:black;}#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .node-icon-8{font-size:40px;color:black;}#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .section-edge-8{stroke:hsl(150, 100%, 76.2745098039%);}#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .edge-depth-8{stroke-width:-10;}#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .section-8 line{stroke:hsl(330, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .disabled,#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .disabled circle,#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .disabled text{fill:lightgray;}#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .disabled text{fill:#efefef;}#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .section-9 rect,#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .section-9 path,#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .section-9 circle,#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .section-9 polygon,#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .section-9 path{fill:hsl(180, 100%, 76.2745098039%);}#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .section-9 text{fill:black;}#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .node-icon-9{font-size:40px;color:black;}#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .section-edge-9{stroke:hsl(180, 100%, 76.2745098039%);}#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .edge-depth-9{stroke-width:-13;}#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .section-9 line{stroke:hsl(0, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .disabled,#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .disabled circle,#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .disabled text{fill:lightgray;}#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .disabled text{fill:#efefef;}#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .section-10 rect,#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .section-10 path,#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .section-10 circle,#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .section-10 polygon,#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .section-10 path{fill:hsl(210, 100%, 76.2745098039%);}#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .section-10 text{fill:black;}#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .node-icon-10{font-size:40px;color:black;}#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .section-edge-10{stroke:hsl(210, 100%, 76.2745098039%);}#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .edge-depth-10{stroke-width:-16;}#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .section-10 line{stroke:hsl(30, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .disabled,#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .disabled circle,#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .disabled text{fill:lightgray;}#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .disabled text{fill:#efefef;}#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .section-root rect,#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .section-root path,#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .section-root circle,#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .section-root polygon{fill:hsl(240, 100%, 46.2745098039%);}#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .section-root text{fill:#ffffff;}#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .section-root span{color:#ffffff;}#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .section-2 span{color:#ffffff;}#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .icon-container{height:100%;display:flex;justify-content:center;align-items:center;}#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .edge{fill:none;}#mermaid-svg-2Q1d7a2dxhYZ5Vj5 .mindmap-node-label{dy:1em;alignment-baseline:middle;text-anchor:middle;dominant-baseline:middle;text-align:center;}#mermaid-svg-2Q1d7a2dxhYZ5Vj5 :root{--mermaid-font-family:"trebuchet ms",verdana,arial,sans-serif;} 网络故障类型
延迟
高延迟网络
不对称延迟
随机延迟抖动
丢包
部分丢包
全部丢包
间歇性丢包
分区
节点隔离
网络分区
防火墙阻断
乱序
数据包乱序
重传包
带宽限制
带宽降级
带宽饱和

💻 Demo: 使用 tc 注入网络故障

tc（Traffic Control）是 Linux 原生的网络控制工具，零安装成本：

bash 复制代码 
# █████ 网络延迟注入 █████
# 给 eth0 网卡增加 100ms 延迟（±20ms 抖动）
tc qdisc add dev eth0 root netem delay 100ms 20ms

# 验证延迟
ping your-service.com
# 输出: time=120.345ms  ← 看到没？多了100ms

# 恢复
tc qdisc del dev eth0 root netem

# █████ 丢包注入 █████
# 模拟 10% 的丢包率
tc qdisc add dev eth0 root netem loss 10%

# 验证丢包
ping -c 100 your-service.com
# 输出: 100 packets transmitted, 90 received, 10% packet loss

# 恢复
tc qdisc del dev eth0 root netem

# █████ 综合场景 █████
# 同时模拟：延迟 + 丢包 + 抖动
tc qdisc add dev eth0 root netem delay 100ms 20ms distribution normal loss 5% 25%

⚠️ tc 命令快速参考卡

复制代码 
┌────────────────────────────────────────────────────────────┐
│  tc 网络故障注入速查表                                      │
├────────────────────────────────────────────────────────────┤
│  功能              命令                                     │
│  ───────────────────────────────────────────────────────── │
│  增加延迟           tc qdisc add dev eth0 root netem delay 100ms │
│  丢包               tc qdisc add dev eth0 root netem loss 10% │
│  包重复             tc qdisc add dev eth0 root netem duplicate 1% │
│  包损坏             tc qdisc add dev eth0 root netem corrupt 0.1% │
│  包乱序             tc qdisc add dev eth0 root netem reorder 25% │
│  带宽限制           tc qdisc add dev eth0 root tbf rate 1mbit  │
│  查看当前规则       tc qdisc show dev eth0                 │
│  清除所有规则       tc qdisc del dev eth0 root             │
└────────────────────────────────────────────────────────────┘

🔍 网络故障注入的典型发现

故障类型 发现的问题 修复措施
100ms延迟 数据库查询超时 添加连接池超时重试
10%丢包 请求成功率下降 实现客户端重试+指数退避
网络分区 服务发现失败 增加本地缓存+多区域部署
带宽限制 日志传输阻塞 异步非阻塞日志传输
相关推荐
汤愈韬2 小时前
EVPN协议概述
网络
kk的vmware虚拟机安装ubuntu13 小时前
鱼皮 yu-rpc：从 0 到 1 手写 RPC 框架的实践教程
网络·网络协议·其他·rpc
ShineWinsu13 小时前
对于Linux：网络基础的解析
linux·网络·面试·udp·笔试·ip·tcp
wenzhangli717 小时前
oodAgent 4.0 Skills分布式调度 — 从Code Agent实战看自主维护的Agent网络
运维·网络·人工智能·自动化
做个文艺程序员17 小时前
Linux第12篇：性能监控与瓶颈分析——CPU/内存/IO/网络全维度
linux·运维·网络
WZF-Sang17 小时前
网络基础——2
服务器·网络·c++·学习·网络编程·php
智慧光迅AINOPOL18 小时前
校园光网络POL方案优劣势分析
网络·全光网解决方案·全光网·校园全光网·校园全光网解决方案
wang_shu_mo_ran18 小时前
网络编程（一）：网络编程初识
运维·服务器·网络
网络研究院19 小时前
Brave浏览器放大招引入容器功能，实现多账户安全隔离
网络·安全·容器·浏览器·数据·隐私
热门推荐
012026 AI 编程工具终极实战指南：Cursor vs Claude Code vs Copilot，开发者该怎么选？022026 年 AI 编程工具终极横评：Cursor vs Claude Code vs Copilot vs Windsurf03AI 编程 IDE 全景解析 2026：Agent 全面接管开发链路042026 年 AI 大模型 & AI 编程工具实战全总结05AI科技热点日报 | 2026年07月01日06GitHub 镜像站点07【AI】2026 年具身智能模型和世界模型总结082026 AI 编程工具选型横评：Cursor / Claude Code / Trae / Copilot 到底选谁（建议收藏·避坑版）09Claude Code、Codex、Cursor三分天下：2026年AI编程Agent生态全景剖析102026年7月AI圈大地震：GPT-5.6被政府限制、Claude入驻Slack、Anthropic自研芯片