凌晨三点，告警群被打爆。订单系统响应时间从 30ms 飙升到 30 秒，数据库连接池告警，下游支付服务超时雪崩。事后复盘发现根因平淡得令人无奈：云厂商某可用区网络抖动 200ms 。这个数字在开发环境从未出现过，开发者的笔记本到本地数据库通常 < 1ms，CI 流水线的容器之间通常 < 5ms。我们写代码时假设的"网络是可靠的"，这是分布式计算 8 大谬误之首。
- The Network is Reliable.（网络是可靠的）
- Latency is Zero.（延迟为零）
- Bandwidth is Infinite.（带宽无限）
- The Network is Secure.（网络是安全的）
- Topology Doesn't Change.（拓扑不变）
- There is One Administrator.（只有一个管理员）
- Transport Cost is Zero.（传输成本为零）
- The Network is Homogeneous.（网络是同构的）
--- L. Peter Deutsch & James Gosling, Sun Microsystems, 1994
Toxiproxy 的存在意义，就是让这些"谬误"在你的开发笔记本上提前发生，对系统进行混沌工程的实践。Netflix《Principles of Chaos Engineering》定义了什么是混动工程：
Chaos Engineering is the discipline of experimenting on a system in order to build confidence in the system's capability to withstand turbulent conditions in production.
注意几个关键词：
- experimenting（实验）--- 不是测试，是科学实验，要有假设和验证
- build confidence（建立信心）--- 目的不是发现 bug，而是产生可信度
- turbulent conditions（动荡条件）--- 接受混乱无法消除，只能验证容忍
混沌工程的终极目标不是构建"鲁棒"系统（能扛住故障），而是构建反脆弱系统每次故障都让系统变得更强（被识别的弱点 → 自动化测试 → 永久免疫）。
在混乱不可避免的世界，目标不是预测危机，而是构建能从危机中获益的系统。--- Nassim Nicholas Taleb
Toxiproxy 内部源码原理
理解工具的内部机制是用好工具的前提。Toxiproxy 用 Go 编写，源码精炼，值得深入。整体架构如下：
#mermaid-svg-4bM5DyW6LQaVnO73{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-4bM5DyW6LQaVnO73 .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-4bM5DyW6LQaVnO73 .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-4bM5DyW6LQaVnO73 .error-icon{fill:#552222;}#mermaid-svg-4bM5DyW6LQaVnO73 .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-4bM5DyW6LQaVnO73 .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-4bM5DyW6LQaVnO73 .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-4bM5DyW6LQaVnO73 .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-4bM5DyW6LQaVnO73 .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-4bM5DyW6LQaVnO73 .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-4bM5DyW6LQaVnO73 .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-4bM5DyW6LQaVnO73 .marker{fill:#333333;stroke:#333333;}#mermaid-svg-4bM5DyW6LQaVnO73 .marker.cross{stroke:#333333;}#mermaid-svg-4bM5DyW6LQaVnO73 svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-4bM5DyW6LQaVnO73 p{margin:0;}#mermaid-svg-4bM5DyW6LQaVnO73 .label{font-family:"trebuchet ms",verdana,arial,sans-serif;color:#333;}#mermaid-svg-4bM5DyW6LQaVnO73 .cluster-label text{fill:#333;}#mermaid-svg-4bM5DyW6LQaVnO73 .cluster-label span{color:#333;}#mermaid-svg-4bM5DyW6LQaVnO73 .cluster-label span p{background-color:transparent;}#mermaid-svg-4bM5DyW6LQaVnO73 .label text,#mermaid-svg-4bM5DyW6LQaVnO73 span{fill:#333;color:#333;}#mermaid-svg-4bM5DyW6LQaVnO73 .node rect,#mermaid-svg-4bM5DyW6LQaVnO73 .node circle,#mermaid-svg-4bM5DyW6LQaVnO73 .node ellipse,#mermaid-svg-4bM5DyW6LQaVnO73 .node polygon,#mermaid-svg-4bM5DyW6LQaVnO73 .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-4bM5DyW6LQaVnO73 .rough-node .label text,#mermaid-svg-4bM5DyW6LQaVnO73 .node .label text,#mermaid-svg-4bM5DyW6LQaVnO73 .image-shape .label,#mermaid-svg-4bM5DyW6LQaVnO73 .icon-shape .label{text-anchor:middle;}#mermaid-svg-4bM5DyW6LQaVnO73 .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-4bM5DyW6LQaVnO73 .rough-node .label,#mermaid-svg-4bM5DyW6LQaVnO73 .node .label,#mermaid-svg-4bM5DyW6LQaVnO73 .image-shape .label,#mermaid-svg-4bM5DyW6LQaVnO73 .icon-shape .label{text-align:center;}#mermaid-svg-4bM5DyW6LQaVnO73 .node.clickable{cursor:pointer;}#mermaid-svg-4bM5DyW6LQaVnO73 .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-4bM5DyW6LQaVnO73 .arrowheadPath{fill:#333333;}#mermaid-svg-4bM5DyW6LQaVnO73 .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-4bM5DyW6LQaVnO73 .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-4bM5DyW6LQaVnO73 .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-4bM5DyW6LQaVnO73 .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-4bM5DyW6LQaVnO73 .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-4bM5DyW6LQaVnO73 .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-4bM5DyW6LQaVnO73 .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-4bM5DyW6LQaVnO73 .cluster text{fill:#333;}#mermaid-svg-4bM5DyW6LQaVnO73 .cluster span{color:#333;}#mermaid-svg-4bM5DyW6LQaVnO73 div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-4bM5DyW6LQaVnO73 .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-4bM5DyW6LQaVnO73 rect.text{fill:none;stroke-width:0;}#mermaid-svg-4bM5DyW6LQaVnO73 .icon-shape,#mermaid-svg-4bM5DyW6LQaVnO73 .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-4bM5DyW6LQaVnO73 .icon-shape p,#mermaid-svg-4bM5DyW6LQaVnO73 .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-4bM5DyW6LQaVnO73 .icon-shape .label rect,#mermaid-svg-4bM5DyW6LQaVnO73 .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-4bM5DyW6LQaVnO73 .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-4bM5DyW6LQaVnO73 .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-4bM5DyW6LQaVnO73 :root{--mermaid-font-family:"trebuchet ms",verdana,arial,sans-serif;} Toxiproxy 进程
响应方向 (Downstream Stream)
请求方向 (Upstream Stream)
① 请求 TCP
② 请求转发
③ 响应 TCP
④ 响应回送
动态注入/修改
客户端App
上游服务
Listener
监听端口
Proxy 实例
ToxicCollection
Upstream Link
Toxic 链
latency → bandwidth → ...
Downstream Link
Toxic 链
latency → slicer → ...
HTTP API :8474
Proxy 核心数据结构源自 proxy.go：
sync.Mutex保证多 goroutine 安全访问tomb.Tomb来自 Canonical 的库，管理 goroutine 生命周期ToxicCollection是 toxic 的容器，支持运行时增删改
go
type Proxy struct {
sync.Mutex
Name string `json:"name"`
Listen string `json:"listen"`
Upstream string `json:"upstream"`
Enabled bool `json:"enabled"`
listener net.Listener
started chan error
tomb tomb.Tomb // 优雅生命周期管理
connections ConnectionList // 活跃连接列表
Toxics *ToxicCollection `json:"-"` // toxic 集合
apiServer *ApiServer
Logger *zerolog.Logger
}每当客户端建立 TCP 连接，Toxiproxy 创建两个独立的 Link ，这就是为什么注入故障时要指定 stream：
stream: upstream--- 影响请求方向（客户端发送时延迟/丢包）stream: downstream--- 影响响应方向（服务器响应时延迟/丢包），通常注入响应方向更接近"服务慢"的语义。
go
// 来自 proxy.go (https://github.com/Shopify/toxiproxy/blob/main/proxy.go#L180-L195)
name := client.RemoteAddr().String()
proxy.connections.Lock()
proxy.connections.list[name+"upstream"] = upstream
proxy.connections.list[name+"downstream"] = client
proxy.connections.Unlock()
proxy.Toxics.StartLink(proxy.apiServer, name+"upstream", client, upstream, stream.Upstream)
proxy.Toxics.StartLink(proxy.apiServer, name+"downstream", upstream, client, stream.Downstream)
graph LR
subgraph "TCP 连接"
Client[客户端]
Server[上游服务]
end
subgraph "Toxiproxy 内部"
UpLink[Upstream Link<br/>req 方向]
DownLink[Downstream Link<br/>resp 方向]
end
Client -->|请求数据| UpLink
UpLink -->|经过 toxic 处理| Server
Server -->|响应数据| DownLink
DownLink -->|经过 toxic 处理| Client
style UpLink fill:#fcf
style DownLink fill:#cffToxiproxy 不是按字节处理，而是按 StreamChunk 处理：
- 时间戳这是延迟计算的核心。
sleep := t.delay() - time.Since(c.Timestamp)定义了已经在 Toxiproxy 内停留的时间。这保证了端到端的精确延迟，无论 toxic 链有多长。
go
// stream/io_chan.go
type StreamChunk struct {
Data []byte
Timestamp time.Time // ← 数据包进入 Toxiproxy 的时刻
}每个 toxic 通过 Go channel 串成流水线：
#mermaid-svg-BTgxdpBZe1tqVCZo{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-BTgxdpBZe1tqVCZo .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-BTgxdpBZe1tqVCZo .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-BTgxdpBZe1tqVCZo .error-icon{fill:#552222;}#mermaid-svg-BTgxdpBZe1tqVCZo .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-BTgxdpBZe1tqVCZo .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-BTgxdpBZe1tqVCZo .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-BTgxdpBZe1tqVCZo .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-BTgxdpBZe1tqVCZo .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-BTgxdpBZe1tqVCZo .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-BTgxdpBZe1tqVCZo .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-BTgxdpBZe1tqVCZo .marker{fill:#333333;stroke:#333333;}#mermaid-svg-BTgxdpBZe1tqVCZo .marker.cross{stroke:#333333;}#mermaid-svg-BTgxdpBZe1tqVCZo svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-BTgxdpBZe1tqVCZo p{margin:0;}#mermaid-svg-BTgxdpBZe1tqVCZo .label{font-family:"trebuchet ms",verdana,arial,sans-serif;color:#333;}#mermaid-svg-BTgxdpBZe1tqVCZo .cluster-label text{fill:#333;}#mermaid-svg-BTgxdpBZe1tqVCZo .cluster-label span{color:#333;}#mermaid-svg-BTgxdpBZe1tqVCZo .cluster-label span p{background-color:transparent;}#mermaid-svg-BTgxdpBZe1tqVCZo .label text,#mermaid-svg-BTgxdpBZe1tqVCZo span{fill:#333;color:#333;}#mermaid-svg-BTgxdpBZe1tqVCZo .node rect,#mermaid-svg-BTgxdpBZe1tqVCZo .node circle,#mermaid-svg-BTgxdpBZe1tqVCZo .node ellipse,#mermaid-svg-BTgxdpBZe1tqVCZo .node polygon,#mermaid-svg-BTgxdpBZe1tqVCZo .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-BTgxdpBZe1tqVCZo .rough-node .label text,#mermaid-svg-BTgxdpBZe1tqVCZo .node .label text,#mermaid-svg-BTgxdpBZe1tqVCZo .image-shape .label,#mermaid-svg-BTgxdpBZe1tqVCZo .icon-shape .label{text-anchor:middle;}#mermaid-svg-BTgxdpBZe1tqVCZo .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-BTgxdpBZe1tqVCZo .rough-node .label,#mermaid-svg-BTgxdpBZe1tqVCZo .node .label,#mermaid-svg-BTgxdpBZe1tqVCZo .image-shape .label,#mermaid-svg-BTgxdpBZe1tqVCZo .icon-shape .label{text-align:center;}#mermaid-svg-BTgxdpBZe1tqVCZo .node.clickable{cursor:pointer;}#mermaid-svg-BTgxdpBZe1tqVCZo .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-BTgxdpBZe1tqVCZo .arrowheadPath{fill:#333333;}#mermaid-svg-BTgxdpBZe1tqVCZo .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-BTgxdpBZe1tqVCZo .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-BTgxdpBZe1tqVCZo .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-BTgxdpBZe1tqVCZo .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-BTgxdpBZe1tqVCZo .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-BTgxdpBZe1tqVCZo .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-BTgxdpBZe1tqVCZo .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-BTgxdpBZe1tqVCZo .cluster text{fill:#333;}#mermaid-svg-BTgxdpBZe1tqVCZo .cluster span{color:#333;}#mermaid-svg-BTgxdpBZe1tqVCZo div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-BTgxdpBZe1tqVCZo .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-BTgxdpBZe1tqVCZo rect.text{fill:none;stroke-width:0;}#mermaid-svg-BTgxdpBZe1tqVCZo .icon-shape,#mermaid-svg-BTgxdpBZe1tqVCZo .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-BTgxdpBZe1tqVCZo .icon-shape p,#mermaid-svg-BTgxdpBZe1tqVCZo .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-BTgxdpBZe1tqVCZo .icon-shape .label rect,#mermaid-svg-BTgxdpBZe1tqVCZo .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-BTgxdpBZe1tqVCZo .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-BTgxdpBZe1tqVCZo .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-BTgxdpBZe1tqVCZo :root{--mermaid-font-family:"trebuchet ms",verdana,arial,sans-serif;} 网络数据
原始字节
StreamChunk
Toxic 1: Latency
等待 1000ms
Toxic 2: Bandwidth
限速 100KB/s
Toxic 3: Slicer
切片传输
输出到下游
Toxic 实现深度解析
Latency Toxic
- Buffered channel ：实现
GetBufferSize() int { return 1024 }，避免延迟限制吞吐量 - 可中断 sleep ：
select { time.After / Interrupt }让动态修改立即生效 - Jitter 实现 ：
rand.Int63n(jitter*2) - jitter是[-jitter, +jitter]均匀分布
go
// 来自 toxics/latency.go
func (t *LatencyToxic) Pipe(stub *ToxicStub) {
for {
select {
case <-stub.Interrupt: // 收到中断信号 (toxic被修改)
return
case c := <-stub.Input:
if c == nil {
stub.Close()
return
}
// 关键：精确延迟计算
sleep := t.delay() - time.Since(c.Timestamp)
select {
case <-time.After(sleep): // 等待计算出的时间
c.Timestamp = c.Timestamp.Add(sleep)
stub.Output <- c // 转发到下一级
case <-stub.Interrupt: // 等待期间收到中断
stub.Output <- c
return
}
}
}
}
func (t *LatencyToxic) delay() time.Duration {
delay := t.Latency
jitter := t.Jitter
if jitter > 0 {
delay += rand.Int63n(jitter*2) - jitter // ±jitter 范围抖动
}
return time.Duration(delay) * time.Millisecond
}Timeout Toxic
timeout=0≠ "无超时"，而是 "接受连接但永远不响应，也不关闭"- 客户端表现：连接保持，但永远收不到任何数据
- 应用必须有自己的客户端超时（如
requests.get(timeout=10)），否则永远卡住
go
// 来自 toxics/timeout.go
func (t *TimeoutToxic) Pipe(stub *ToxicStub) {
timeout := time.Duration(t.Timeout) * time.Millisecond
if timeout > 0 {
for {
select {
case <-time.After(timeout):
stub.Close() // ← 超时后强制关闭连接
return
case <-stub.Interrupt:
return
case c := <-stub.Input:
if c == nil { stub.Close(); return }
// ⚠️ 关键：数据被静默丢弃 ("Drop the data on the ground")
}
}
} else {
// timeout=0：永远丢弃数据，永不关闭
for {
select {
case <-stub.Interrupt: return
case c := <-stub.Input:
if c == nil { stub.Close(); return }
// 静默丢弃
}
}
}
}Bandwidth Toxic
将数据切成 100ms 一片的小包，用 sleep 控制每片之间的间隔，实现近似精确的带宽限制。
go
// 来自 toxics/bandwidth.go
func (t *BandwidthToxic) Pipe(stub *ToxicStub) {
var sleep time.Duration = 0
for {
select {
case <-stub.Interrupt: return
case p := <-stub.Input:
if p == nil { stub.Close(); return }
if t.Rate <= 0 {
sleep = 0
} else {
// 核心公式：传输 N 字节需要的毫秒数 = N / rate(KB/s)
sleep += time.Duration(len(p.Data)) * time.Millisecond / time.Duration(t.Rate)
}
// 大数据包分片：每 100ms 发送 rate*100 字节
for int64(len(p.Data)) > t.Rate*100 {
select {
case <-time.After(100 * time.Millisecond):
stub.Output <- &stream.StreamChunk{
Data: p.Data[:t.Rate*100],
Timestamp: p.Timestamp,
}
p.Data = p.Data[t.Rate*100:]
sleep -= 100 * time.Millisecond
case <-stub.Interrupt:
stub.WriteOutput(p, 5*time.Second)
return
}
}
// 剩余小包发送
start := time.Now()
select {
case <-time.After(sleep):
sleep -= time.Since(start) // 时间补偿，提高精度
stub.Output <- p
case <-stub.Interrupt:
stub.WriteOutput(p, 5*time.Second)
return
}
}
}
}Slicer Toxic
数据包切片模拟 TCP 分片在不同 MTU 网络中的行为，测试应用是否正确处理"半包"问题（HTTP body 分多个 TCP 包到达）。
go
// 来自 toxics/slicer.go
// 递归二分切片算法
func (t *SlicerToxic) chunk(start int, end int) []int {
if (end-start)-t.AverageSize <= t.SizeVariation {
return []int{start, end}
}
mid := start + (end-start)/2
if t.SizeVariation > 0 {
mid += rand.Intn(t.SizeVariation*2) - t.SizeVariation // 随机化分割点
}
left := t.chunk(start, mid)
right := t.chunk(mid, end)
return append(left, right...)
}Limit Data Toxic
实现了 NewState() interface{} 接口，每个连接独立的状态，避免多连接互相干扰。
go
// 来自 toxics/limit_data.go
type LimitDataToxicState struct {
bytesTransmitted int64 // 累计传输字节
}
func (t *LimitDataToxic) Pipe(stub *ToxicStub) {
state := stub.State.(*LimitDataToxicState)
bytesRemaining := t.Bytes - state.bytesTransmitted
for {
select {
case <-stub.Interrupt: return
case c := <-stub.Input:
if c == nil { stub.Close(); return }
// 数据截断
if bytesRemaining < int64(len(c.Data)) {
c = &stream.StreamChunk{
Timestamp: c.Timestamp,
Data: c.Data[0:bytesRemaining], // ← 只保留剩余配额
}
}
stub.Output <- c
state.bytesTransmitted += int64(len(c.Data))
bytesRemaining = t.Bytes - state.bytesTransmitted
if bytesRemaining <= 0 {
stub.Close() // ← 配额用完，关闭连接
return
}
}
}
}Toxicity 概率参数
每个 toxic 都支持 toxicity 参数（0.0 - 1.0），表示故障发生的概率：测试间歇性故障，比"100% 故障"或"100% 正常"都更接近真实生产。
bash
# 30% 的请求会被延迟 2 秒，70% 正常通过
curl -X POST http://localhost:8474/proxies/payment-proxy/toxics \
-d '{
"name":"flaky",
"type":"latency",
"toxicity":0.3,
"attributes":{"latency":2000}
}'tenacity重试库
tenacity 是 Python 生态最流行的重试库，由 Julien Danjou 维护，用于给函数添加重试能力。它的前身是 retrying 库，2016 年因原作者维护中断而 fork 重写。核心特性如下：
- 装饰器语法，零侵入业务代码
- 多种停止条件（次数、总时间、异常类型）
- 多种等待策略（固定、指数退避、随机）
- 异常分类（哪些异常重试，哪些不重试）
- 钩子函数（重试前/后/失败时的回调）
核心三要素如下
py
from tenacity import retry, stop_after_attempt, wait_exponential, retry_if_exception_type
@retry(
stop=stop_after_attempt(3), # ① 何时停止
wait=wait_exponential(multiplier=1, min=1, max=10), # ② 等多久再试
retry=retry_if_exception_type((ConnectionError,)) # ③ 什么异常才重试
)
def call_api():
return requests.get('http://api.example.com')整体的逻辑图图下
#mermaid-svg-AEptuOSXKyGa5iPj{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-AEptuOSXKyGa5iPj .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-AEptuOSXKyGa5iPj .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-AEptuOSXKyGa5iPj .error-icon{fill:#552222;}#mermaid-svg-AEptuOSXKyGa5iPj .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-AEptuOSXKyGa5iPj .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-AEptuOSXKyGa5iPj .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-AEptuOSXKyGa5iPj .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-AEptuOSXKyGa5iPj .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-AEptuOSXKyGa5iPj .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-AEptuOSXKyGa5iPj .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-AEptuOSXKyGa5iPj .marker{fill:#333333;stroke:#333333;}#mermaid-svg-AEptuOSXKyGa5iPj .marker.cross{stroke:#333333;}#mermaid-svg-AEptuOSXKyGa5iPj svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-AEptuOSXKyGa5iPj p{margin:0;}#mermaid-svg-AEptuOSXKyGa5iPj .label{font-family:"trebuchet ms",verdana,arial,sans-serif;color:#333;}#mermaid-svg-AEptuOSXKyGa5iPj .cluster-label text{fill:#333;}#mermaid-svg-AEptuOSXKyGa5iPj .cluster-label span{color:#333;}#mermaid-svg-AEptuOSXKyGa5iPj .cluster-label span p{background-color:transparent;}#mermaid-svg-AEptuOSXKyGa5iPj .label text,#mermaid-svg-AEptuOSXKyGa5iPj span{fill:#333;color:#333;}#mermaid-svg-AEptuOSXKyGa5iPj .node rect,#mermaid-svg-AEptuOSXKyGa5iPj .node circle,#mermaid-svg-AEptuOSXKyGa5iPj .node ellipse,#mermaid-svg-AEptuOSXKyGa5iPj .node polygon,#mermaid-svg-AEptuOSXKyGa5iPj .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-AEptuOSXKyGa5iPj .rough-node .label text,#mermaid-svg-AEptuOSXKyGa5iPj .node .label text,#mermaid-svg-AEptuOSXKyGa5iPj .image-shape .label,#mermaid-svg-AEptuOSXKyGa5iPj .icon-shape .label{text-anchor:middle;}#mermaid-svg-AEptuOSXKyGa5iPj .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-AEptuOSXKyGa5iPj .rough-node .label,#mermaid-svg-AEptuOSXKyGa5iPj .node .label,#mermaid-svg-AEptuOSXKyGa5iPj .image-shape .label,#mermaid-svg-AEptuOSXKyGa5iPj .icon-shape .label{text-align:center;}#mermaid-svg-AEptuOSXKyGa5iPj .node.clickable{cursor:pointer;}#mermaid-svg-AEptuOSXKyGa5iPj .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-AEptuOSXKyGa5iPj .arrowheadPath{fill:#333333;}#mermaid-svg-AEptuOSXKyGa5iPj .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-AEptuOSXKyGa5iPj .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-AEptuOSXKyGa5iPj .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-AEptuOSXKyGa5iPj .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-AEptuOSXKyGa5iPj .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-AEptuOSXKyGa5iPj .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-AEptuOSXKyGa5iPj .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-AEptuOSXKyGa5iPj .cluster text{fill:#333;}#mermaid-svg-AEptuOSXKyGa5iPj .cluster span{color:#333;}#mermaid-svg-AEptuOSXKyGa5iPj div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-AEptuOSXKyGa5iPj .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-AEptuOSXKyGa5iPj rect.text{fill:none;stroke-width:0;}#mermaid-svg-AEptuOSXKyGa5iPj .icon-shape,#mermaid-svg-AEptuOSXKyGa5iPj .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-AEptuOSXKyGa5iPj .icon-shape p,#mermaid-svg-AEptuOSXKyGa5iPj .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-AEptuOSXKyGa5iPj .icon-shape .label rect,#mermaid-svg-AEptuOSXKyGa5iPj .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-AEptuOSXKyGa5iPj .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-AEptuOSXKyGa5iPj .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-AEptuOSXKyGa5iPj :root{--mermaid-font-family:"trebuchet ms",verdana,arial,sans-serif;} 成功
可重试异常
成功
可重试异常
成功
失败
不可重试异常
函数调用
尝试1
返回结果
等待 1s
尝试2
等待 2s
尝试3
抛 RetryError
直接抛异常
等待策略（wait）
python
from tenacity import (
wait_fixed, # 固定等待
wait_random, # 随机等待
wait_exponential, # 指数退避
wait_random_exponential # 指数退避 + 抖动 (推荐)
)
# 指数退避：1s, 2s, 4s, 8s, 16s... 上限 60s
@retry(wait=wait_exponential(multiplier=1, max=60))
def call(): pass
# 指数退避 + 抖动：避免蜂群效应
@retry(wait=wait_random_exponential(multiplier=1, max=60))
def call(): pass为什么需要抖动（jitter）呢？
服务 客户端3 客户端2 客户端1 服务 客户端3 客户端2 客户端1 #mermaid-svg-qphpWwFMbXWeBbCv{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-qphpWwFMbXWeBbCv .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-qphpWwFMbXWeBbCv .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-qphpWwFMbXWeBbCv .error-icon{fill:#552222;}#mermaid-svg-qphpWwFMbXWeBbCv .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-qphpWwFMbXWeBbCv .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-qphpWwFMbXWeBbCv .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-qphpWwFMbXWeBbCv .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-qphpWwFMbXWeBbCv .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-qphpWwFMbXWeBbCv .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-qphpWwFMbXWeBbCv .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-qphpWwFMbXWeBbCv .marker{fill:#333333;stroke:#333333;}#mermaid-svg-qphpWwFMbXWeBbCv .marker.cross{stroke:#333333;}#mermaid-svg-qphpWwFMbXWeBbCv svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-qphpWwFMbXWeBbCv p{margin:0;}#mermaid-svg-qphpWwFMbXWeBbCv .actor{stroke:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);fill:#ECECFF;}#mermaid-svg-qphpWwFMbXWeBbCv text.actor>tspan{fill:black;stroke:none;}#mermaid-svg-qphpWwFMbXWeBbCv .actor-line{stroke:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);}#mermaid-svg-qphpWwFMbXWeBbCv .innerArc{stroke-width:1.5;stroke-dasharray:none;}#mermaid-svg-qphpWwFMbXWeBbCv .messageLine0{stroke-width:1.5;stroke-dasharray:none;stroke:#333;}#mermaid-svg-qphpWwFMbXWeBbCv .messageLine1{stroke-width:1.5;stroke-dasharray:2,2;stroke:#333;}#mermaid-svg-qphpWwFMbXWeBbCv #arrowhead path{fill:#333;stroke:#333;}#mermaid-svg-qphpWwFMbXWeBbCv .sequenceNumber{fill:white;}#mermaid-svg-qphpWwFMbXWeBbCv #sequencenumber{fill:#333;}#mermaid-svg-qphpWwFMbXWeBbCv #crosshead path{fill:#333;stroke:#333;}#mermaid-svg-qphpWwFMbXWeBbCv .messageText{fill:#333;stroke:none;}#mermaid-svg-qphpWwFMbXWeBbCv .labelBox{stroke:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);fill:#ECECFF;}#mermaid-svg-qphpWwFMbXWeBbCv .labelText,#mermaid-svg-qphpWwFMbXWeBbCv .labelText>tspan{fill:black;stroke:none;}#mermaid-svg-qphpWwFMbXWeBbCv .loopText,#mermaid-svg-qphpWwFMbXWeBbCv .loopText>tspan{fill:black;stroke:none;}#mermaid-svg-qphpWwFMbXWeBbCv .loopLine{stroke-width:2px;stroke-dasharray:2,2;stroke:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);fill:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);}#mermaid-svg-qphpWwFMbXWeBbCv .note{stroke:#aaaa33;fill:#fff5ad;}#mermaid-svg-qphpWwFMbXWeBbCv .noteText,#mermaid-svg-qphpWwFMbXWeBbCv .noteText>tspan{fill:black;stroke:none;}#mermaid-svg-qphpWwFMbXWeBbCv .activation0{fill:#f4f4f4;stroke:#666;}#mermaid-svg-qphpWwFMbXWeBbCv .activation1{fill:#f4f4f4;stroke:#666;}#mermaid-svg-qphpWwFMbXWeBbCv .activation2{fill:#f4f4f4;stroke:#666;}#mermaid-svg-qphpWwFMbXWeBbCv .actorPopupMenu{position:absolute;}#mermaid-svg-qphpWwFMbXWeBbCv .actorPopupMenuPanel{position:absolute;fill:#ECECFF;box-shadow:0px 8px 16px 0px rgba(0,0,0,0.2);filter:drop-shadow(3px 5px 2px rgb(0 0 0 / 0.4));}#mermaid-svg-qphpWwFMbXWeBbCv .actor-man line{stroke:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);fill:#ECECFF;}#mermaid-svg-qphpWwFMbXWeBbCv .actor-man circle,#mermaid-svg-qphpWwFMbXWeBbCv line{stroke:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);fill:#ECECFF;stroke-width:2px;}#mermaid-svg-qphpWwFMbXWeBbCv :root{--mermaid-font-family:"trebuchet ms",verdana,arial,sans-serif;} 服务故障 大家都退避 1s par 同时重试 雷鸣群效应 同时打过来,服务再次崩溃 请求失败 请求失败 请求失败 重试 (T+1s) 重试 (T+1s) 重试 (T+1s)
加了抖动后，3 个客户端会在 0.5s, 1.5s 范围内随机分散重试，避免同步冲击。
重试条件（retry）
python
from tenacity import retry, retry_if_exception_type, retry_if_result
# 仅特定异常重试
@retry(retry=retry_if_exception_type((ConnectionError, Timeout)))
def call(): pass
# 根据返回值决定是否重试
@retry(retry=retry_if_result(lambda r: r is None))
def call(): return None # 返回 None 时重试同步异常类型需要区分是否应当重试：
| 异常类型 | 是否应该重试 |
|---|---|
ConnectionError（网络问题） |
应该重试 |
Timeout（超时） |
通常重试 |
404 Not Found |
不要重试（资源不存在） |
400 Bad Request |
不要重试（请求本身错误） |
401 Unauthorized |
不要重试（凭证问题） |
500 Internal Server Error |
视情况（可能是临时问题） |
钩子函数
python
from tenacity import retry, stop_after_attempt, before_sleep_log
import logging
logger = logging.getLogger(__name__)
@retry(
stop=stop_after_attempt(3),
before_sleep=before_sleep_log(logger, logging.WARNING) # 重试前打印日志
)
def call_api():
pass在重试前会输出输出日志：
WARNING:root:Retrying call_api in 1.0 seconds as it raised ConnectionError: ...tenacity 应用
本文实验中订单服务的 process_payment 函数：
timeout=10：单次 HTTP 请求最多等 10 秒stop_after_attempt(2)：失败最多重试 1 次（共 2 次）- 总最坏时间：
10s + 0.5s + 10s = 20.5s
注意：总等待时间 = 单次超时 × 重试次数 + 退避总时间，因此必须确保此值 < 上游调用方的超时，否则你重试还没完，调用方已经超时了。
python
@retry(
stop=stop_after_attempt(2), # 最多 2 次
wait=wait_exponential(multiplier=1, min=0.5, max=2) # 0.5s, 1s, 2s
)
def process_payment(user_id, amount):
try:
r = requests.post(
f"{PAYMENT_SERVICE_URL}/api/payments/charge",
json={'user_id': user_id, 'amount': amount},
timeout=10 # ← 单次超时 10 秒
)
r.raise_for_status()
return True
except requests.Timeout:
logger.error("Payment timeout")
raise # ← 重新抛出，触发 tenacity 重试
except Exception as e:
logger.warning(f"Payment failed: {e}")
return False环境初始化
本次测试的完整架构如下，所有跨服务的 TCP 通信都经过 Toxiproxy。
#mermaid-svg-eDFgzjrwvJIBcCyB{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-eDFgzjrwvJIBcCyB .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-eDFgzjrwvJIBcCyB .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-eDFgzjrwvJIBcCyB .error-icon{fill:#552222;}#mermaid-svg-eDFgzjrwvJIBcCyB .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-eDFgzjrwvJIBcCyB .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-eDFgzjrwvJIBcCyB .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-eDFgzjrwvJIBcCyB .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-eDFgzjrwvJIBcCyB .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-eDFgzjrwvJIBcCyB .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-eDFgzjrwvJIBcCyB .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-eDFgzjrwvJIBcCyB .marker{fill:#333333;stroke:#333333;}#mermaid-svg-eDFgzjrwvJIBcCyB .marker.cross{stroke:#333333;}#mermaid-svg-eDFgzjrwvJIBcCyB svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-eDFgzjrwvJIBcCyB p{margin:0;}#mermaid-svg-eDFgzjrwvJIBcCyB .label{font-family:"trebuchet ms",verdana,arial,sans-serif;color:#333;}#mermaid-svg-eDFgzjrwvJIBcCyB .cluster-label text{fill:#333;}#mermaid-svg-eDFgzjrwvJIBcCyB .cluster-label span{color:#333;}#mermaid-svg-eDFgzjrwvJIBcCyB .cluster-label span p{background-color:transparent;}#mermaid-svg-eDFgzjrwvJIBcCyB .label text,#mermaid-svg-eDFgzjrwvJIBcCyB span{fill:#333;color:#333;}#mermaid-svg-eDFgzjrwvJIBcCyB .node rect,#mermaid-svg-eDFgzjrwvJIBcCyB .node circle,#mermaid-svg-eDFgzjrwvJIBcCyB .node ellipse,#mermaid-svg-eDFgzjrwvJIBcCyB .node polygon,#mermaid-svg-eDFgzjrwvJIBcCyB .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-eDFgzjrwvJIBcCyB .rough-node .label text,#mermaid-svg-eDFgzjrwvJIBcCyB .node .label text,#mermaid-svg-eDFgzjrwvJIBcCyB .image-shape .label,#mermaid-svg-eDFgzjrwvJIBcCyB .icon-shape .label{text-anchor:middle;}#mermaid-svg-eDFgzjrwvJIBcCyB .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-eDFgzjrwvJIBcCyB .rough-node .label,#mermaid-svg-eDFgzjrwvJIBcCyB .node .label,#mermaid-svg-eDFgzjrwvJIBcCyB .image-shape .label,#mermaid-svg-eDFgzjrwvJIBcCyB .icon-shape .label{text-align:center;}#mermaid-svg-eDFgzjrwvJIBcCyB .node.clickable{cursor:pointer;}#mermaid-svg-eDFgzjrwvJIBcCyB .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-eDFgzjrwvJIBcCyB .arrowheadPath{fill:#333333;}#mermaid-svg-eDFgzjrwvJIBcCyB .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-eDFgzjrwvJIBcCyB .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-eDFgzjrwvJIBcCyB .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-eDFgzjrwvJIBcCyB .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-eDFgzjrwvJIBcCyB .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-eDFgzjrwvJIBcCyB .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-eDFgzjrwvJIBcCyB .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-eDFgzjrwvJIBcCyB .cluster text{fill:#333;}#mermaid-svg-eDFgzjrwvJIBcCyB .cluster span{color:#333;}#mermaid-svg-eDFgzjrwvJIBcCyB div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-eDFgzjrwvJIBcCyB .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-eDFgzjrwvJIBcCyB rect.text{fill:none;stroke-width:0;}#mermaid-svg-eDFgzjrwvJIBcCyB .icon-shape,#mermaid-svg-eDFgzjrwvJIBcCyB .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-eDFgzjrwvJIBcCyB .icon-shape p,#mermaid-svg-eDFgzjrwvJIBcCyB .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-eDFgzjrwvJIBcCyB .icon-shape .label rect,#mermaid-svg-eDFgzjrwvJIBcCyB .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-eDFgzjrwvJIBcCyB .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-eDFgzjrwvJIBcCyB .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-eDFgzjrwvJIBcCyB :root{--mermaid-font-family:"trebuchet ms",verdana,arial,sans-serif;} Toxiproxy 代理层 (4个代理)
HTTP :5000
psycopg2
redis client
HTTP
HTTP
psycopg2
动态控制
动态控制
动态控制
动态控制
客户端 curl
Order Service
postgres-proxy
:5433
redis-proxy
:6380
payment-proxy
:9001
inventory-proxy
:8001
PostgreSQL
Redis
Payment Service
Inventory Service
Toxiproxy API :8474
使用docker-compose.yml部署服务，环境变量必须指向 toxiproxy。这是最容易出错的点。应用直连真实服务时，故障注入完全无效。
yaml
services:
order-service:
build: { context: ./webapp, dockerfile: Dockerfile }
ports: ["5000:5000"]
environment:
# ↓ 所有外部依赖都指向 toxiproxy
- DATABASE_URL=postgresql://orderuser:orderpass@toxiproxy:5433/orderdb
- REDIS_URL=redis://toxiproxy:6380/0
- PAYMENT_SERVICE_URL=http://toxiproxy:9001
- INVENTORY_SERVICE_URL=http://toxiproxy:8001
depends_on: [postgres, redis, toxiproxy, payment-service, inventory-service]
networks: [chaos-net]
inventory-service:
build: { context: ./webapp, dockerfile: Dockerfile.inventory }
ports: ["5001:5001"]
environment:
# ↓ 库存服务的数据库调用也走 toxiproxy
- DATABASE_URL=postgresql://invuser:invpass@toxiproxy:5433/inventorydb
depends_on: [postgres, toxiproxy]
networks: [chaos-net]
payment-service:
build: { context: ./webapp, dockerfile: Dockerfile.payment }
ports: ["7001:7001"]
networks: [chaos-net]
postgres:
image: postgres:15
volumes:
- postgres-data:/var/lib/postgresql/data
- ./scripts/init-db.sql:/docker-entrypoint-initdb.d/init.sql
networks: [chaos-net]
healthcheck:
test: ["CMD-SHELL", "pg_isready -U superuser"]
redis:
image: redis:7.2-bookworm
networks: [chaos-net]
toxiproxy:
image: ghcr.io/shopify/toxiproxy:2.5.0
ports:
- "8474:8474" # HTTP API
- "5433:5433" # postgres-proxy
- "6380:6380" # redis-proxy
- "9002:9001" # payment-proxy (9001被占用映射到9002)
- "8001:8001" # inventory-proxy
command: ["-host", "0.0.0.0"]
networks: [chaos-net]
networks:
chaos-net: { driver: bridge }
volumes:
postgres-data:数据库初始化
sql
-- scripts/init-db.sql
CREATE DATABASE orderdb;
CREATE DATABASE inventorydb;
CREATE USER orderuser WITH PASSWORD 'orderpass';
CREATE USER invuser WITH PASSWORD 'invpass';
GRANT ALL PRIVILEGES ON DATABASE orderdb TO orderuser;
GRANT ALL PRIVILEGES ON DATABASE inventorydb TO invuser;
\c orderdb;
CREATE TABLE orders (
id SERIAL PRIMARY KEY,
order_id VARCHAR(50) UNIQUE NOT NULL,
user_id VARCHAR(50) NOT NULL,
total_amount DECIMAL(10, 2) DEFAULT 0,
status VARCHAR(20) DEFAULT 'created',
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);
GRANT ALL PRIVILEGES ON ALL TABLES IN SCHEMA public TO orderuser;
GRANT ALL PRIVILEGES ON ALL SEQUENCES IN SCHEMA public TO orderuser;
\c inventorydb;
CREATE TABLE products (
id VARCHAR(50) PRIMARY KEY,
name VARCHAR(200) NOT NULL,
price DECIMAL(10, 2) NOT NULL,
stock INTEGER DEFAULT 0
);
INSERT INTO products VALUES
('PROD-001', '精品咖啡豆 500g', 128.00, 100),
('PROD-002', '手冲咖啡壶套装', 299.00, 50),
('PROD-003', '进口全脂牛奶 1L', 18.00, 200);
GRANT ALL PRIVILEGES ON ALL TABLES IN SCHEMA public TO invuser;Toxiproxy 代理初始化
服务启动后 Toxiproxy 是空的，没有任何代理，必须手动创建。
bash
# 创建 4 个代理
curl -X POST http://localhost:8474/proxies \
-d '{"name":"postgres-proxy","listen":"0.0.0.0:5433","upstream":"postgres:5432"}'
curl -X POST http://localhost:8474/proxies \
-d '{"name":"redis-proxy","listen":"0.0.0.0:6380","upstream":"redis:6379"}'
curl -X POST http://localhost:8474/proxies \
-d '{"name":"payment-proxy","listen":"0.0.0.0:9001","upstream":"payment-service:7001"}'
curl -X POST http://localhost:8474/proxies \
-d '{"name":"inventory-proxy","listen":"0.0.0.0:8001","upstream":"inventory-service:5001"}'确认创建：
bash
$ curl -s http://localhost:8474/proxies | python3 -m json.tool
{
"inventory-proxy": {"listen": "[::]:8001", "upstream": "inventory-service:5001", ...},
"payment-proxy": {"listen": "[::]:9001", "upstream": "payment-service:7001", ...},
"postgres-proxy": {"listen": "[::]:5433", "upstream": "postgres:5432", ...},
"redis-proxy": {"listen": "[::]:6380", "upstream": "redis:6379", ...}
}应用韧性模式实现
订单服务模拟一个真实的电商下单流程，业务调用链如下，每一步都涉及不同等级的故障容忍策略
#mermaid-svg-Sb5Hs6iClyT9MOVF{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-Sb5Hs6iClyT9MOVF .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-Sb5Hs6iClyT9MOVF .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-Sb5Hs6iClyT9MOVF .error-icon{fill:#552222;}#mermaid-svg-Sb5Hs6iClyT9MOVF .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-Sb5Hs6iClyT9MOVF .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-Sb5Hs6iClyT9MOVF .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-Sb5Hs6iClyT9MOVF .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-Sb5Hs6iClyT9MOVF .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-Sb5Hs6iClyT9MOVF .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-Sb5Hs6iClyT9MOVF .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-Sb5Hs6iClyT9MOVF .marker{fill:#333333;stroke:#333333;}#mermaid-svg-Sb5Hs6iClyT9MOVF .marker.cross{stroke:#333333;}#mermaid-svg-Sb5Hs6iClyT9MOVF svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-Sb5Hs6iClyT9MOVF p{margin:0;}#mermaid-svg-Sb5Hs6iClyT9MOVF .label{font-family:"trebuchet ms",verdana,arial,sans-serif;color:#333;}#mermaid-svg-Sb5Hs6iClyT9MOVF .cluster-label text{fill:#333;}#mermaid-svg-Sb5Hs6iClyT9MOVF .cluster-label span{color:#333;}#mermaid-svg-Sb5Hs6iClyT9MOVF .cluster-label span p{background-color:transparent;}#mermaid-svg-Sb5Hs6iClyT9MOVF .label text,#mermaid-svg-Sb5Hs6iClyT9MOVF span{fill:#333;color:#333;}#mermaid-svg-Sb5Hs6iClyT9MOVF .node rect,#mermaid-svg-Sb5Hs6iClyT9MOVF .node circle,#mermaid-svg-Sb5Hs6iClyT9MOVF .node ellipse,#mermaid-svg-Sb5Hs6iClyT9MOVF .node polygon,#mermaid-svg-Sb5Hs6iClyT9MOVF .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-Sb5Hs6iClyT9MOVF .rough-node .label text,#mermaid-svg-Sb5Hs6iClyT9MOVF .node .label text,#mermaid-svg-Sb5Hs6iClyT9MOVF .image-shape .label,#mermaid-svg-Sb5Hs6iClyT9MOVF .icon-shape .label{text-anchor:middle;}#mermaid-svg-Sb5Hs6iClyT9MOVF .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-Sb5Hs6iClyT9MOVF .rough-node .label,#mermaid-svg-Sb5Hs6iClyT9MOVF .node .label,#mermaid-svg-Sb5Hs6iClyT9MOVF .image-shape .label,#mermaid-svg-Sb5Hs6iClyT9MOVF .icon-shape .label{text-align:center;}#mermaid-svg-Sb5Hs6iClyT9MOVF .node.clickable{cursor:pointer;}#mermaid-svg-Sb5Hs6iClyT9MOVF .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-Sb5Hs6iClyT9MOVF .arrowheadPath{fill:#333333;}#mermaid-svg-Sb5Hs6iClyT9MOVF .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-Sb5Hs6iClyT9MOVF .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-Sb5Hs6iClyT9MOVF .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-Sb5Hs6iClyT9MOVF .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-Sb5Hs6iClyT9MOVF .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-Sb5Hs6iClyT9MOVF .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-Sb5Hs6iClyT9MOVF .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-Sb5Hs6iClyT9MOVF .cluster text{fill:#333;}#mermaid-svg-Sb5Hs6iClyT9MOVF .cluster span{color:#333;}#mermaid-svg-Sb5Hs6iClyT9MOVF div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-Sb5Hs6iClyT9MOVF .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-Sb5Hs6iClyT9MOVF rect.text{fill:none;stroke-width:0;}#mermaid-svg-Sb5Hs6iClyT9MOVF .icon-shape,#mermaid-svg-Sb5Hs6iClyT9MOVF .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-Sb5Hs6iClyT9MOVF .icon-shape p,#mermaid-svg-Sb5Hs6iClyT9MOVF .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-Sb5Hs6iClyT9MOVF .icon-shape .label rect,#mermaid-svg-Sb5Hs6iClyT9MOVF .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-Sb5Hs6iClyT9MOVF .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-Sb5Hs6iClyT9MOVF .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-Sb5Hs6iClyT9MOVF :root{--mermaid-font-family:"trebuchet ms",verdana,arial,sans-serif;} POST /api/orders
用户下单
① 检查库存
Inventory Service
② 处理支付
Payment Service
③ 写入订单
PostgreSQL
④ 缓存订单
Redis
返回订单号
韧性 4 大支柱示意图
#mermaid-svg-kLEbbONRdrjIRwgR{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-kLEbbONRdrjIRwgR .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-kLEbbONRdrjIRwgR .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-kLEbbONRdrjIRwgR .error-icon{fill:#552222;}#mermaid-svg-kLEbbONRdrjIRwgR .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-kLEbbONRdrjIRwgR .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-kLEbbONRdrjIRwgR .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-kLEbbONRdrjIRwgR .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-kLEbbONRdrjIRwgR .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-kLEbbONRdrjIRwgR .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-kLEbbONRdrjIRwgR .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-kLEbbONRdrjIRwgR .marker{fill:#333333;stroke:#333333;}#mermaid-svg-kLEbbONRdrjIRwgR .marker.cross{stroke:#333333;}#mermaid-svg-kLEbbONRdrjIRwgR svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-kLEbbONRdrjIRwgR p{margin:0;}#mermaid-svg-kLEbbONRdrjIRwgR .edge{stroke-width:3;}#mermaid-svg-kLEbbONRdrjIRwgR .section--1 rect,#mermaid-svg-kLEbbONRdrjIRwgR .section--1 path,#mermaid-svg-kLEbbONRdrjIRwgR .section--1 circle,#mermaid-svg-kLEbbONRdrjIRwgR .section--1 polygon,#mermaid-svg-kLEbbONRdrjIRwgR .section--1 path{fill:hsl(240, 100%, 76.2745098039%);}#mermaid-svg-kLEbbONRdrjIRwgR .section--1 text{fill:#ffffff;}#mermaid-svg-kLEbbONRdrjIRwgR .node-icon--1{font-size:40px;color:#ffffff;}#mermaid-svg-kLEbbONRdrjIRwgR .section-edge--1{stroke:hsl(240, 100%, 76.2745098039%);}#mermaid-svg-kLEbbONRdrjIRwgR .edge-depth--1{stroke-width:17;}#mermaid-svg-kLEbbONRdrjIRwgR .section--1 line{stroke:hsl(60, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-kLEbbONRdrjIRwgR .disabled,#mermaid-svg-kLEbbONRdrjIRwgR .disabled circle,#mermaid-svg-kLEbbONRdrjIRwgR .disabled text{fill:lightgray;}#mermaid-svg-kLEbbONRdrjIRwgR .disabled text{fill:#efefef;}#mermaid-svg-kLEbbONRdrjIRwgR .section-0 rect,#mermaid-svg-kLEbbONRdrjIRwgR .section-0 path,#mermaid-svg-kLEbbONRdrjIRwgR .section-0 circle,#mermaid-svg-kLEbbONRdrjIRwgR .section-0 polygon,#mermaid-svg-kLEbbONRdrjIRwgR .section-0 path{fill:hsl(60, 100%, 73.5294117647%);}#mermaid-svg-kLEbbONRdrjIRwgR .section-0 text{fill:black;}#mermaid-svg-kLEbbONRdrjIRwgR .node-icon-0{font-size:40px;color:black;}#mermaid-svg-kLEbbONRdrjIRwgR .section-edge-0{stroke:hsl(60, 100%, 73.5294117647%);}#mermaid-svg-kLEbbONRdrjIRwgR .edge-depth-0{stroke-width:14;}#mermaid-svg-kLEbbONRdrjIRwgR .section-0 line{stroke:hsl(240, 100%, 83.5294117647%);stroke-width:3;}#mermaid-svg-kLEbbONRdrjIRwgR .disabled,#mermaid-svg-kLEbbONRdrjIRwgR .disabled circle,#mermaid-svg-kLEbbONRdrjIRwgR .disabled text{fill:lightgray;}#mermaid-svg-kLEbbONRdrjIRwgR .disabled text{fill:#efefef;}#mermaid-svg-kLEbbONRdrjIRwgR .section-1 rect,#mermaid-svg-kLEbbONRdrjIRwgR .section-1 path,#mermaid-svg-kLEbbONRdrjIRwgR .section-1 circle,#mermaid-svg-kLEbbONRdrjIRwgR .section-1 polygon,#mermaid-svg-kLEbbONRdrjIRwgR .section-1 path{fill:hsl(80, 100%, 76.2745098039%);}#mermaid-svg-kLEbbONRdrjIRwgR .section-1 text{fill:black;}#mermaid-svg-kLEbbONRdrjIRwgR .node-icon-1{font-size:40px;color:black;}#mermaid-svg-kLEbbONRdrjIRwgR .section-edge-1{stroke:hsl(80, 100%, 76.2745098039%);}#mermaid-svg-kLEbbONRdrjIRwgR .edge-depth-1{stroke-width:11;}#mermaid-svg-kLEbbONRdrjIRwgR .section-1 line{stroke:hsl(260, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-kLEbbONRdrjIRwgR .disabled,#mermaid-svg-kLEbbONRdrjIRwgR .disabled circle,#mermaid-svg-kLEbbONRdrjIRwgR .disabled text{fill:lightgray;}#mermaid-svg-kLEbbONRdrjIRwgR .disabled text{fill:#efefef;}#mermaid-svg-kLEbbONRdrjIRwgR .section-2 rect,#mermaid-svg-kLEbbONRdrjIRwgR .section-2 path,#mermaid-svg-kLEbbONRdrjIRwgR .section-2 circle,#mermaid-svg-kLEbbONRdrjIRwgR .section-2 polygon,#mermaid-svg-kLEbbONRdrjIRwgR .section-2 path{fill:hsl(270, 100%, 76.2745098039%);}#mermaid-svg-kLEbbONRdrjIRwgR .section-2 text{fill:#ffffff;}#mermaid-svg-kLEbbONRdrjIRwgR .node-icon-2{font-size:40px;color:#ffffff;}#mermaid-svg-kLEbbONRdrjIRwgR .section-edge-2{stroke:hsl(270, 100%, 76.2745098039%);}#mermaid-svg-kLEbbONRdrjIRwgR .edge-depth-2{stroke-width:8;}#mermaid-svg-kLEbbONRdrjIRwgR .section-2 line{stroke:hsl(90, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-kLEbbONRdrjIRwgR .disabled,#mermaid-svg-kLEbbONRdrjIRwgR .disabled circle,#mermaid-svg-kLEbbONRdrjIRwgR .disabled text{fill:lightgray;}#mermaid-svg-kLEbbONRdrjIRwgR .disabled text{fill:#efefef;}#mermaid-svg-kLEbbONRdrjIRwgR .section-3 rect,#mermaid-svg-kLEbbONRdrjIRwgR .section-3 path,#mermaid-svg-kLEbbONRdrjIRwgR .section-3 circle,#mermaid-svg-kLEbbONRdrjIRwgR .section-3 polygon,#mermaid-svg-kLEbbONRdrjIRwgR .section-3 path{fill:hsl(300, 100%, 76.2745098039%);}#mermaid-svg-kLEbbONRdrjIRwgR .section-3 text{fill:black;}#mermaid-svg-kLEbbONRdrjIRwgR .node-icon-3{font-size:40px;color:black;}#mermaid-svg-kLEbbONRdrjIRwgR .section-edge-3{stroke:hsl(300, 100%, 76.2745098039%);}#mermaid-svg-kLEbbONRdrjIRwgR .edge-depth-3{stroke-width:5;}#mermaid-svg-kLEbbONRdrjIRwgR .section-3 line{stroke:hsl(120, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-kLEbbONRdrjIRwgR .disabled,#mermaid-svg-kLEbbONRdrjIRwgR .disabled circle,#mermaid-svg-kLEbbONRdrjIRwgR .disabled text{fill:lightgray;}#mermaid-svg-kLEbbONRdrjIRwgR .disabled text{fill:#efefef;}#mermaid-svg-kLEbbONRdrjIRwgR .section-4 rect,#mermaid-svg-kLEbbONRdrjIRwgR .section-4 path,#mermaid-svg-kLEbbONRdrjIRwgR .section-4 circle,#mermaid-svg-kLEbbONRdrjIRwgR .section-4 polygon,#mermaid-svg-kLEbbONRdrjIRwgR .section-4 path{fill:hsl(330, 100%, 76.2745098039%);}#mermaid-svg-kLEbbONRdrjIRwgR .section-4 text{fill:black;}#mermaid-svg-kLEbbONRdrjIRwgR .node-icon-4{font-size:40px;color:black;}#mermaid-svg-kLEbbONRdrjIRwgR .section-edge-4{stroke:hsl(330, 100%, 76.2745098039%);}#mermaid-svg-kLEbbONRdrjIRwgR .edge-depth-4{stroke-width:2;}#mermaid-svg-kLEbbONRdrjIRwgR .section-4 line{stroke:hsl(150, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-kLEbbONRdrjIRwgR .disabled,#mermaid-svg-kLEbbONRdrjIRwgR .disabled circle,#mermaid-svg-kLEbbONRdrjIRwgR .disabled text{fill:lightgray;}#mermaid-svg-kLEbbONRdrjIRwgR .disabled text{fill:#efefef;}#mermaid-svg-kLEbbONRdrjIRwgR .section-5 rect,#mermaid-svg-kLEbbONRdrjIRwgR .section-5 path,#mermaid-svg-kLEbbONRdrjIRwgR .section-5 circle,#mermaid-svg-kLEbbONRdrjIRwgR .section-5 polygon,#mermaid-svg-kLEbbONRdrjIRwgR .section-5 path{fill:hsl(0, 100%, 76.2745098039%);}#mermaid-svg-kLEbbONRdrjIRwgR .section-5 text{fill:black;}#mermaid-svg-kLEbbONRdrjIRwgR .node-icon-5{font-size:40px;color:black;}#mermaid-svg-kLEbbONRdrjIRwgR .section-edge-5{stroke:hsl(0, 100%, 76.2745098039%);}#mermaid-svg-kLEbbONRdrjIRwgR .edge-depth-5{stroke-width:-1;}#mermaid-svg-kLEbbONRdrjIRwgR .section-5 line{stroke:hsl(180, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-kLEbbONRdrjIRwgR .disabled,#mermaid-svg-kLEbbONRdrjIRwgR .disabled circle,#mermaid-svg-kLEbbONRdrjIRwgR .disabled text{fill:lightgray;}#mermaid-svg-kLEbbONRdrjIRwgR .disabled text{fill:#efefef;}#mermaid-svg-kLEbbONRdrjIRwgR .section-6 rect,#mermaid-svg-kLEbbONRdrjIRwgR .section-6 path,#mermaid-svg-kLEbbONRdrjIRwgR .section-6 circle,#mermaid-svg-kLEbbONRdrjIRwgR .section-6 polygon,#mermaid-svg-kLEbbONRdrjIRwgR .section-6 path{fill:hsl(30, 100%, 76.2745098039%);}#mermaid-svg-kLEbbONRdrjIRwgR .section-6 text{fill:black;}#mermaid-svg-kLEbbONRdrjIRwgR .node-icon-6{font-size:40px;color:black;}#mermaid-svg-kLEbbONRdrjIRwgR .section-edge-6{stroke:hsl(30, 100%, 76.2745098039%);}#mermaid-svg-kLEbbONRdrjIRwgR .edge-depth-6{stroke-width:-4;}#mermaid-svg-kLEbbONRdrjIRwgR .section-6 line{stroke:hsl(210, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-kLEbbONRdrjIRwgR .disabled,#mermaid-svg-kLEbbONRdrjIRwgR .disabled circle,#mermaid-svg-kLEbbONRdrjIRwgR .disabled text{fill:lightgray;}#mermaid-svg-kLEbbONRdrjIRwgR .disabled text{fill:#efefef;}#mermaid-svg-kLEbbONRdrjIRwgR .section-7 rect,#mermaid-svg-kLEbbONRdrjIRwgR .section-7 path,#mermaid-svg-kLEbbONRdrjIRwgR .section-7 circle,#mermaid-svg-kLEbbONRdrjIRwgR .section-7 polygon,#mermaid-svg-kLEbbONRdrjIRwgR .section-7 path{fill:hsl(90, 100%, 76.2745098039%);}#mermaid-svg-kLEbbONRdrjIRwgR .section-7 text{fill:black;}#mermaid-svg-kLEbbONRdrjIRwgR .node-icon-7{font-size:40px;color:black;}#mermaid-svg-kLEbbONRdrjIRwgR .section-edge-7{stroke:hsl(90, 100%, 76.2745098039%);}#mermaid-svg-kLEbbONRdrjIRwgR .edge-depth-7{stroke-width:-7;}#mermaid-svg-kLEbbONRdrjIRwgR .section-7 line{stroke:hsl(270, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-kLEbbONRdrjIRwgR .disabled,#mermaid-svg-kLEbbONRdrjIRwgR .disabled circle,#mermaid-svg-kLEbbONRdrjIRwgR .disabled text{fill:lightgray;}#mermaid-svg-kLEbbONRdrjIRwgR .disabled text{fill:#efefef;}#mermaid-svg-kLEbbONRdrjIRwgR .section-8 rect,#mermaid-svg-kLEbbONRdrjIRwgR .section-8 path,#mermaid-svg-kLEbbONRdrjIRwgR .section-8 circle,#mermaid-svg-kLEbbONRdrjIRwgR .section-8 polygon,#mermaid-svg-kLEbbONRdrjIRwgR .section-8 path{fill:hsl(150, 100%, 76.2745098039%);}#mermaid-svg-kLEbbONRdrjIRwgR .section-8 text{fill:black;}#mermaid-svg-kLEbbONRdrjIRwgR .node-icon-8{font-size:40px;color:black;}#mermaid-svg-kLEbbONRdrjIRwgR .section-edge-8{stroke:hsl(150, 100%, 76.2745098039%);}#mermaid-svg-kLEbbONRdrjIRwgR .edge-depth-8{stroke-width:-10;}#mermaid-svg-kLEbbONRdrjIRwgR .section-8 line{stroke:hsl(330, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-kLEbbONRdrjIRwgR .disabled,#mermaid-svg-kLEbbONRdrjIRwgR .disabled circle,#mermaid-svg-kLEbbONRdrjIRwgR .disabled text{fill:lightgray;}#mermaid-svg-kLEbbONRdrjIRwgR .disabled text{fill:#efefef;}#mermaid-svg-kLEbbONRdrjIRwgR .section-9 rect,#mermaid-svg-kLEbbONRdrjIRwgR .section-9 path,#mermaid-svg-kLEbbONRdrjIRwgR .section-9 circle,#mermaid-svg-kLEbbONRdrjIRwgR .section-9 polygon,#mermaid-svg-kLEbbONRdrjIRwgR .section-9 path{fill:hsl(180, 100%, 76.2745098039%);}#mermaid-svg-kLEbbONRdrjIRwgR .section-9 text{fill:black;}#mermaid-svg-kLEbbONRdrjIRwgR .node-icon-9{font-size:40px;color:black;}#mermaid-svg-kLEbbONRdrjIRwgR .section-edge-9{stroke:hsl(180, 100%, 76.2745098039%);}#mermaid-svg-kLEbbONRdrjIRwgR .edge-depth-9{stroke-width:-13;}#mermaid-svg-kLEbbONRdrjIRwgR .section-9 line{stroke:hsl(0, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-kLEbbONRdrjIRwgR .disabled,#mermaid-svg-kLEbbONRdrjIRwgR .disabled circle,#mermaid-svg-kLEbbONRdrjIRwgR .disabled text{fill:lightgray;}#mermaid-svg-kLEbbONRdrjIRwgR .disabled text{fill:#efefef;}#mermaid-svg-kLEbbONRdrjIRwgR .section-10 rect,#mermaid-svg-kLEbbONRdrjIRwgR .section-10 path,#mermaid-svg-kLEbbONRdrjIRwgR .section-10 circle,#mermaid-svg-kLEbbONRdrjIRwgR .section-10 polygon,#mermaid-svg-kLEbbONRdrjIRwgR .section-10 path{fill:hsl(210, 100%, 76.2745098039%);}#mermaid-svg-kLEbbONRdrjIRwgR .section-10 text{fill:black;}#mermaid-svg-kLEbbONRdrjIRwgR .node-icon-10{font-size:40px;color:black;}#mermaid-svg-kLEbbONRdrjIRwgR .section-edge-10{stroke:hsl(210, 100%, 76.2745098039%);}#mermaid-svg-kLEbbONRdrjIRwgR .edge-depth-10{stroke-width:-16;}#mermaid-svg-kLEbbONRdrjIRwgR .section-10 line{stroke:hsl(30, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-kLEbbONRdrjIRwgR .disabled,#mermaid-svg-kLEbbONRdrjIRwgR .disabled circle,#mermaid-svg-kLEbbONRdrjIRwgR .disabled text{fill:lightgray;}#mermaid-svg-kLEbbONRdrjIRwgR .disabled text{fill:#efefef;}#mermaid-svg-kLEbbONRdrjIRwgR .section-root rect,#mermaid-svg-kLEbbONRdrjIRwgR .section-root path,#mermaid-svg-kLEbbONRdrjIRwgR .section-root circle,#mermaid-svg-kLEbbONRdrjIRwgR .section-root polygon{fill:hsl(240, 100%, 46.2745098039%);}#mermaid-svg-kLEbbONRdrjIRwgR .section-root text{fill:#ffffff;}#mermaid-svg-kLEbbONRdrjIRwgR .section-root span{color:#ffffff;}#mermaid-svg-kLEbbONRdrjIRwgR .section-2 span{color:#ffffff;}#mermaid-svg-kLEbbONRdrjIRwgR .icon-container{height:100%;display:flex;justify-content:center;align-items:center;}#mermaid-svg-kLEbbONRdrjIRwgR .edge{fill:none;}#mermaid-svg-kLEbbONRdrjIRwgR .mindmap-node-label{dy:1em;alignment-baseline:middle;text-anchor:middle;dominant-baseline:middle;text-align:center;}#mermaid-svg-kLEbbONRdrjIRwgR :root{--mermaid-font-family:"trebuchet ms",verdana,arial,sans-serif;} 韧性系统
Timeout
P99 × 3
连接超时 ≠ 读超时
上下文传播
Retry
指数退避 + 抖动
最大次数限制
幂等性前提
仅瞬态错误
Circuit Breaker
快速失败
避免雪崩
自动半开试探
Fallback
静态默认值
缓存兜底
降级响应
配置层：让流量都走 Toxiproxy，这样 Toxiproxy 才能作为"中间人"拦截到流量。
python
# webapp/app.py
import os
import psycopg2
import redis
import requests
from tenacity import retry, stop_after_attempt, wait_exponential, retry_if_exception_type
DATABASE_URL = os.getenv('DATABASE_URL', 'postgresql://orderuser:orderpass@toxiproxy:5433/orderdb')
REDIS_URL = os.getenv('REDIS_URL', 'redis://toxiproxy:6380/0')
PAYMENT_SERVICE_URL = os.getenv('PAYMENT_SERVICE_URL', 'http://toxiproxy:9001')
INVENTORY_SERVICE_URL = os.getenv('INVENTORY_SERVICE_URL', 'http://toxiproxy:8001')用户点击"提交订单"后，最忌讳的是白屏等待 。这里 connect_timeout=3 意味着，数据库超时设计的考量，如果 3 秒内连不上数据库，立即放弃，向用户报错而不是让浏览器转圈 30 秒。
python
def get_db_connection():
return psycopg2.connect(DATABASE_URL, connect_timeout=3)常见的数据库连接延迟如下，因此3 秒是同机房场景下"绝对不可能正常但还没到雪崩"的阈值。
| 场景 | 数据库正常连接耗时 | 推荐 connect_timeout |
|---|---|---|
| 同机房内网 | < 5ms | 1-3 秒 |
| 跨可用区 | 5-20ms | 3-5 秒 |
| 跨地域 | 50-200ms | 5-10 秒 |
库存检查：为什么"失败也算成功"
这段代码体现了业务驱动的容错决策，即库存检查在订单流程中是"软依赖"。
stop_after_attempt(3)：库存服务允许偶发抖动，给 3 次机会wait_exponential(min=1, max=10)：1s → 2s → 4s，避免重试雪崩retry_if_exception_type((Timeout, ConnectionError))：只对网络问题重试，HTTP 4xx 不重试（4xx 是请求本身有问题，重试也是错）
python
@retry(
stop=stop_after_attempt(3),
wait=wait_exponential(multiplier=1, min=1, max=10),
retry=retry_if_exception_type((requests.Timeout, requests.ConnectionError))
)
def check_inventory(items):
try:
r = requests.get(f"{INVENTORY_SERVICE_URL}/api/inventory", timeout=5)
r.raise_for_status()
return True
except Exception as e:
logger.warning(f"Inventory check degraded: {e}")
return True # ← 注意这里！失败也返回 True**为什么库存可以降级？**因为
- 超卖损失：100 个订单中可能 1-2 单超卖，造成几百元额外发货成本
- 拒绝下单损失：100 个订单全部失败，损失数万元 GMV + 客户体验
绝大多数电商选择"放行 + 事后对账"。这就是为什么代码里 except 分支也返回 True。
#mermaid-svg-GnHCvpg3mfq6GKhL{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-GnHCvpg3mfq6GKhL .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-GnHCvpg3mfq6GKhL .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-GnHCvpg3mfq6GKhL .error-icon{fill:#552222;}#mermaid-svg-GnHCvpg3mfq6GKhL .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-GnHCvpg3mfq6GKhL .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-GnHCvpg3mfq6GKhL .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-GnHCvpg3mfq6GKhL .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-GnHCvpg3mfq6GKhL .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-GnHCvpg3mfq6GKhL .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-GnHCvpg3mfq6GKhL .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-GnHCvpg3mfq6GKhL .marker{fill:#333333;stroke:#333333;}#mermaid-svg-GnHCvpg3mfq6GKhL .marker.cross{stroke:#333333;}#mermaid-svg-GnHCvpg3mfq6GKhL svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-GnHCvpg3mfq6GKhL p{margin:0;}#mermaid-svg-GnHCvpg3mfq6GKhL .label{font-family:"trebuchet ms",verdana,arial,sans-serif;color:#333;}#mermaid-svg-GnHCvpg3mfq6GKhL .cluster-label text{fill:#333;}#mermaid-svg-GnHCvpg3mfq6GKhL .cluster-label span{color:#333;}#mermaid-svg-GnHCvpg3mfq6GKhL .cluster-label span p{background-color:transparent;}#mermaid-svg-GnHCvpg3mfq6GKhL .label text,#mermaid-svg-GnHCvpg3mfq6GKhL span{fill:#333;color:#333;}#mermaid-svg-GnHCvpg3mfq6GKhL .node rect,#mermaid-svg-GnHCvpg3mfq6GKhL .node circle,#mermaid-svg-GnHCvpg3mfq6GKhL .node ellipse,#mermaid-svg-GnHCvpg3mfq6GKhL .node polygon,#mermaid-svg-GnHCvpg3mfq6GKhL .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-GnHCvpg3mfq6GKhL .rough-node .label text,#mermaid-svg-GnHCvpg3mfq6GKhL .node .label text,#mermaid-svg-GnHCvpg3mfq6GKhL .image-shape .label,#mermaid-svg-GnHCvpg3mfq6GKhL .icon-shape .label{text-anchor:middle;}#mermaid-svg-GnHCvpg3mfq6GKhL .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-GnHCvpg3mfq6GKhL .rough-node .label,#mermaid-svg-GnHCvpg3mfq6GKhL .node .label,#mermaid-svg-GnHCvpg3mfq6GKhL .image-shape .label,#mermaid-svg-GnHCvpg3mfq6GKhL .icon-shape .label{text-align:center;}#mermaid-svg-GnHCvpg3mfq6GKhL .node.clickable{cursor:pointer;}#mermaid-svg-GnHCvpg3mfq6GKhL .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-GnHCvpg3mfq6GKhL .arrowheadPath{fill:#333333;}#mermaid-svg-GnHCvpg3mfq6GKhL .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-GnHCvpg3mfq6GKhL .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-GnHCvpg3mfq6GKhL .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-GnHCvpg3mfq6GKhL .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-GnHCvpg3mfq6GKhL .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-GnHCvpg3mfq6GKhL .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-GnHCvpg3mfq6GKhL .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-GnHCvpg3mfq6GKhL .cluster text{fill:#333;}#mermaid-svg-GnHCvpg3mfq6GKhL .cluster span{color:#333;}#mermaid-svg-GnHCvpg3mfq6GKhL div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-GnHCvpg3mfq6GKhL .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-GnHCvpg3mfq6GKhL rect.text{fill:none;stroke-width:0;}#mermaid-svg-GnHCvpg3mfq6GKhL .icon-shape,#mermaid-svg-GnHCvpg3mfq6GKhL .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-GnHCvpg3mfq6GKhL .icon-shape p,#mermaid-svg-GnHCvpg3mfq6GKhL .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-GnHCvpg3mfq6GKhL .icon-shape .label rect,#mermaid-svg-GnHCvpg3mfq6GKhL .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-GnHCvpg3mfq6GKhL .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-GnHCvpg3mfq6GKhL .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-GnHCvpg3mfq6GKhL :root{--mermaid-font-family:"trebuchet ms",verdana,arial,sans-serif;} 拒绝下单
拒绝下单
拒绝下单
放行下单
放行下单
放行下单
库存服务挂了
该怎么办?
用户流失
GMV损失
库存服务变成单点
订单照常完成
⚠️ 可能少量超卖
后续异步对账可补救
支付处理：为什么"失败必须是失败"
支付是订单流程的"关键路径"，任何不确定都必须报错让用户知情。
python
@retry(stop=stop_after_attempt(2), wait=wait_exponential(min=0.5, max=2))
def process_payment(user_id, amount):
try:
r = requests.post(
f"{PAYMENT_SERVICE_URL}/api/payments/charge",
json={'user_id': user_id, 'amount': amount},
timeout=10
)
r.raise_for_status()
return True
except requests.Timeout:
raise # ← 抛出，让 tenacity 接管重试
except Exception as e:
return False # ← 返回失败,不能假装成功对比库存检查的关键差异：
| 维度 | check_inventory | process_payment |
|---|---|---|
| 失败时返回 | True（降级） |
False（如实报告） |
| 重试次数 | 3 次（更宽容） | 2 次（更保守） |
| 退避时间 | 1-10s（慢） | 0.5-2s（快） |
| 业务影响 | 少量超卖 | 用户重复扣款风险 |
**为什么支付重试要保守？**超时不等于失败，有可能上游已经成功，只是响应丢了。重试就会变成重复操作。
银行 Payment Service Order Service 用户 银行 Payment Service Order Service 用户 #mermaid-svg-l4mvRK3zv8JjYnfp{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-l4mvRK3zv8JjYnfp .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-l4mvRK3zv8JjYnfp .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-l4mvRK3zv8JjYnfp .error-icon{fill:#552222;}#mermaid-svg-l4mvRK3zv8JjYnfp .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-l4mvRK3zv8JjYnfp .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-l4mvRK3zv8JjYnfp .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-l4mvRK3zv8JjYnfp .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-l4mvRK3zv8JjYnfp .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-l4mvRK3zv8JjYnfp .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-l4mvRK3zv8JjYnfp .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-l4mvRK3zv8JjYnfp .marker{fill:#333333;stroke:#333333;}#mermaid-svg-l4mvRK3zv8JjYnfp .marker.cross{stroke:#333333;}#mermaid-svg-l4mvRK3zv8JjYnfp svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-l4mvRK3zv8JjYnfp p{margin:0;}#mermaid-svg-l4mvRK3zv8JjYnfp .actor{stroke:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);fill:#ECECFF;}#mermaid-svg-l4mvRK3zv8JjYnfp text.actor>tspan{fill:black;stroke:none;}#mermaid-svg-l4mvRK3zv8JjYnfp .actor-line{stroke:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);}#mermaid-svg-l4mvRK3zv8JjYnfp .innerArc{stroke-width:1.5;stroke-dasharray:none;}#mermaid-svg-l4mvRK3zv8JjYnfp .messageLine0{stroke-width:1.5;stroke-dasharray:none;stroke:#333;}#mermaid-svg-l4mvRK3zv8JjYnfp .messageLine1{stroke-width:1.5;stroke-dasharray:2,2;stroke:#333;}#mermaid-svg-l4mvRK3zv8JjYnfp #arrowhead path{fill:#333;stroke:#333;}#mermaid-svg-l4mvRK3zv8JjYnfp .sequenceNumber{fill:white;}#mermaid-svg-l4mvRK3zv8JjYnfp #sequencenumber{fill:#333;}#mermaid-svg-l4mvRK3zv8JjYnfp #crosshead path{fill:#333;stroke:#333;}#mermaid-svg-l4mvRK3zv8JjYnfp .messageText{fill:#333;stroke:none;}#mermaid-svg-l4mvRK3zv8JjYnfp .labelBox{stroke:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);fill:#ECECFF;}#mermaid-svg-l4mvRK3zv8JjYnfp .labelText,#mermaid-svg-l4mvRK3zv8JjYnfp .labelText>tspan{fill:black;stroke:none;}#mermaid-svg-l4mvRK3zv8JjYnfp .loopText,#mermaid-svg-l4mvRK3zv8JjYnfp .loopText>tspan{fill:black;stroke:none;}#mermaid-svg-l4mvRK3zv8JjYnfp .loopLine{stroke-width:2px;stroke-dasharray:2,2;stroke:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);fill:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);}#mermaid-svg-l4mvRK3zv8JjYnfp .note{stroke:#aaaa33;fill:#fff5ad;}#mermaid-svg-l4mvRK3zv8JjYnfp .noteText,#mermaid-svg-l4mvRK3zv8JjYnfp .noteText>tspan{fill:black;stroke:none;}#mermaid-svg-l4mvRK3zv8JjYnfp .activation0{fill:#f4f4f4;stroke:#666;}#mermaid-svg-l4mvRK3zv8JjYnfp .activation1{fill:#f4f4f4;stroke:#666;}#mermaid-svg-l4mvRK3zv8JjYnfp .activation2{fill:#f4f4f4;stroke:#666;}#mermaid-svg-l4mvRK3zv8JjYnfp .actorPopupMenu{position:absolute;}#mermaid-svg-l4mvRK3zv8JjYnfp .actorPopupMenuPanel{position:absolute;fill:#ECECFF;box-shadow:0px 8px 16px 0px rgba(0,0,0,0.2);filter:drop-shadow(3px 5px 2px rgb(0 0 0 / 0.4));}#mermaid-svg-l4mvRK3zv8JjYnfp .actor-man line{stroke:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);fill:#ECECFF;}#mermaid-svg-l4mvRK3zv8JjYnfp .actor-man circle,#mermaid-svg-l4mvRK3zv8JjYnfp line{stroke:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);fill:#ECECFF;stroke-width:2px;}#mermaid-svg-l4mvRK3zv8JjYnfp :root{--mermaid-font-family:"trebuchet ms",verdana,arial,sans-serif;} 网络抖动,响应丢失 tenacity 触发重试 用户被扣款 ¥598! 下单 (¥299) 扣款 (¥299) 扣款指令 扣款成功 ⏱ Timeout 扣款 (¥299) 重试 扣款指令 ⚠️ 重复! 扣款成功
正确的做法是配合幂等性 ，支付服务收到相同的 key 时直接返回上次结果，不会重复扣款。
python
import uuid
idempotency_key = str(uuid.uuid4()) # 客户端生成,本次订单全程不变
r = requests.post(
f"{PAYMENT_SERVICE_URL}/api/payments/charge",
json={'user_id': user_id, 'amount': amount},
headers={'X-Idempotency-Key': idempotency_key}, # 关键
timeout=10
)订单查询：缓存降级的优雅退化
订单查询是高频读操作（用户反复刷新订单页面），用 Redis 缓存提升性能。但 Redis 不是权威数据源，最终数据在 PostgreSQL。
python
@app.route('/api/orders/<order_id>')
def get_order(order_id):
if redis_client:
try:
cached = redis_client.get(f"order:{order_id}")
if cached:
return jsonify({'data': str(cached), 'source': 'cache'})
except Exception as e:
logger.warning(f"Cache read failed, falling back to DB: {e}")
# 注意:这里没有 raise,流程继续往下走
try:
conn = get_db_connection()
# ... 从 DB 查询并返回
except Exception:
return jsonify({'error': 'database unavailable'}), 503降级路径的业务图景如下
#mermaid-svg-JMqlBHTIihU1MWmu{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-JMqlBHTIihU1MWmu .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-JMqlBHTIihU1MWmu .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-JMqlBHTIihU1MWmu .error-icon{fill:#552222;}#mermaid-svg-JMqlBHTIihU1MWmu .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-JMqlBHTIihU1MWmu .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-JMqlBHTIihU1MWmu .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-JMqlBHTIihU1MWmu .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-JMqlBHTIihU1MWmu .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-JMqlBHTIihU1MWmu .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-JMqlBHTIihU1MWmu .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-JMqlBHTIihU1MWmu .marker{fill:#333333;stroke:#333333;}#mermaid-svg-JMqlBHTIihU1MWmu .marker.cross{stroke:#333333;}#mermaid-svg-JMqlBHTIihU1MWmu svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-JMqlBHTIihU1MWmu p{margin:0;}#mermaid-svg-JMqlBHTIihU1MWmu .label{font-family:"trebuchet ms",verdana,arial,sans-serif;color:#333;}#mermaid-svg-JMqlBHTIihU1MWmu .cluster-label text{fill:#333;}#mermaid-svg-JMqlBHTIihU1MWmu .cluster-label span{color:#333;}#mermaid-svg-JMqlBHTIihU1MWmu .cluster-label span p{background-color:transparent;}#mermaid-svg-JMqlBHTIihU1MWmu .label text,#mermaid-svg-JMqlBHTIihU1MWmu span{fill:#333;color:#333;}#mermaid-svg-JMqlBHTIihU1MWmu .node rect,#mermaid-svg-JMqlBHTIihU1MWmu .node circle,#mermaid-svg-JMqlBHTIihU1MWmu .node ellipse,#mermaid-svg-JMqlBHTIihU1MWmu .node polygon,#mermaid-svg-JMqlBHTIihU1MWmu .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-JMqlBHTIihU1MWmu .rough-node .label text,#mermaid-svg-JMqlBHTIihU1MWmu .node .label text,#mermaid-svg-JMqlBHTIihU1MWmu .image-shape .label,#mermaid-svg-JMqlBHTIihU1MWmu .icon-shape .label{text-anchor:middle;}#mermaid-svg-JMqlBHTIihU1MWmu .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-JMqlBHTIihU1MWmu .rough-node .label,#mermaid-svg-JMqlBHTIihU1MWmu .node .label,#mermaid-svg-JMqlBHTIihU1MWmu .image-shape .label,#mermaid-svg-JMqlBHTIihU1MWmu .icon-shape .label{text-align:center;}#mermaid-svg-JMqlBHTIihU1MWmu .node.clickable{cursor:pointer;}#mermaid-svg-JMqlBHTIihU1MWmu .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-JMqlBHTIihU1MWmu .arrowheadPath{fill:#333333;}#mermaid-svg-JMqlBHTIihU1MWmu .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-JMqlBHTIihU1MWmu .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-JMqlBHTIihU1MWmu .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-JMqlBHTIihU1MWmu .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-JMqlBHTIihU1MWmu .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-JMqlBHTIihU1MWmu .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-JMqlBHTIihU1MWmu .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-JMqlBHTIihU1MWmu .cluster text{fill:#333;}#mermaid-svg-JMqlBHTIihU1MWmu .cluster span{color:#333;}#mermaid-svg-JMqlBHTIihU1MWmu div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-JMqlBHTIihU1MWmu .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-JMqlBHTIihU1MWmu rect.text{fill:none;stroke-width:0;}#mermaid-svg-JMqlBHTIihU1MWmu .icon-shape,#mermaid-svg-JMqlBHTIihU1MWmu .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-JMqlBHTIihU1MWmu .icon-shape p,#mermaid-svg-JMqlBHTIihU1MWmu .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-JMqlBHTIihU1MWmu .icon-shape .label rect,#mermaid-svg-JMqlBHTIihU1MWmu .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-JMqlBHTIihU1MWmu .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-JMqlBHTIihU1MWmu .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-JMqlBHTIihU1MWmu :root{--mermaid-font-family:"trebuchet ms",verdana,arial,sans-serif;} 是
是
否
否
是
否
GET /api/orders/123
Redis 可用?
读取缓存
命中?
返回 source:cache
响应时间 ~5ms
回源数据库
记录警告日志
DB 可用?
返回 source:database
响应时间 ~50ms
503 服务不可用
真正的失败
层次化的故障耐受度：
| 故障组合 | 用户体验 | 实现 |
|---|---|---|
| Redis 正常 + DB 正常 | 极快（5ms） | 缓存命中 |
| Redis 异常 + DB 正常 | 稍慢（50ms） | 自动降级到 DB，用户无感知 |
| Redis 正常 + DB 异常 | 缓存命中可用，未命中失败 | 部分功能 |
| Redis 异常 + DB 异常 | 503 错误 | 真正的故障 |
关键设计原则 ：永远不要让缓存的故障变成业务的故障。
代码里 except Exception as e: logger.warning(...) 这一行没有 raise，因为：
- 缓存读失败 ≠ 业务失败
- 主流程继续走 DB 查询，用户可能感知到"慢了一点"但不会看到错误页
韧性设计的三层防御
#mermaid-svg-FEf773SnGEHoyoac{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-FEf773SnGEHoyoac .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-FEf773SnGEHoyoac .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-FEf773SnGEHoyoac .error-icon{fill:#552222;}#mermaid-svg-FEf773SnGEHoyoac .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-FEf773SnGEHoyoac .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-FEf773SnGEHoyoac .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-FEf773SnGEHoyoac .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-FEf773SnGEHoyoac .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-FEf773SnGEHoyoac .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-FEf773SnGEHoyoac .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-FEf773SnGEHoyoac .marker{fill:#333333;stroke:#333333;}#mermaid-svg-FEf773SnGEHoyoac .marker.cross{stroke:#333333;}#mermaid-svg-FEf773SnGEHoyoac svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-FEf773SnGEHoyoac p{margin:0;}#mermaid-svg-FEf773SnGEHoyoac .label{font-family:"trebuchet ms",verdana,arial,sans-serif;color:#333;}#mermaid-svg-FEf773SnGEHoyoac .cluster-label text{fill:#333;}#mermaid-svg-FEf773SnGEHoyoac .cluster-label span{color:#333;}#mermaid-svg-FEf773SnGEHoyoac .cluster-label span p{background-color:transparent;}#mermaid-svg-FEf773SnGEHoyoac .label text,#mermaid-svg-FEf773SnGEHoyoac span{fill:#333;color:#333;}#mermaid-svg-FEf773SnGEHoyoac .node rect,#mermaid-svg-FEf773SnGEHoyoac .node circle,#mermaid-svg-FEf773SnGEHoyoac .node ellipse,#mermaid-svg-FEf773SnGEHoyoac .node polygon,#mermaid-svg-FEf773SnGEHoyoac .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-FEf773SnGEHoyoac .rough-node .label text,#mermaid-svg-FEf773SnGEHoyoac .node .label text,#mermaid-svg-FEf773SnGEHoyoac .image-shape .label,#mermaid-svg-FEf773SnGEHoyoac .icon-shape .label{text-anchor:middle;}#mermaid-svg-FEf773SnGEHoyoac .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-FEf773SnGEHoyoac .rough-node .label,#mermaid-svg-FEf773SnGEHoyoac .node .label,#mermaid-svg-FEf773SnGEHoyoac .image-shape .label,#mermaid-svg-FEf773SnGEHoyoac .icon-shape .label{text-align:center;}#mermaid-svg-FEf773SnGEHoyoac .node.clickable{cursor:pointer;}#mermaid-svg-FEf773SnGEHoyoac .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-FEf773SnGEHoyoac .arrowheadPath{fill:#333333;}#mermaid-svg-FEf773SnGEHoyoac .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-FEf773SnGEHoyoac .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-FEf773SnGEHoyoac .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-FEf773SnGEHoyoac .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-FEf773SnGEHoyoac .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-FEf773SnGEHoyoac .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-FEf773SnGEHoyoac .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-FEf773SnGEHoyoac .cluster text{fill:#333;}#mermaid-svg-FEf773SnGEHoyoac .cluster span{color:#333;}#mermaid-svg-FEf773SnGEHoyoac div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-FEf773SnGEHoyoac .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-FEf773SnGEHoyoac rect.text{fill:none;stroke-width:0;}#mermaid-svg-FEf773SnGEHoyoac .icon-shape,#mermaid-svg-FEf773SnGEHoyoac .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-FEf773SnGEHoyoac .icon-shape p,#mermaid-svg-FEf773SnGEHoyoac .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-FEf773SnGEHoyoac .icon-shape .label rect,#mermaid-svg-FEf773SnGEHoyoac .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-FEf773SnGEHoyoac .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-FEf773SnGEHoyoac .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-FEf773SnGEHoyoac :root{--mermaid-font-family:"trebuchet ms",verdana,arial,sans-serif;} 超时
重试耗尽
降级失败
成功
成功
降级响应
请求
第1层: 超时
第2层: 重试 + 退避
第3层: 降级 / 熔断
返回错误
记录告警
返回结果
故障注入
每个测试前后需要保持环境洁净，避免上一个实验残留的 toxic 干扰下一个
bash
# 实验开始前: 清除某个代理的所有 toxic
$ curl -X DELETE http://localhost:8474/proxies/payment-proxy/toxics/<toxic_name>
# 查看所有当前注入的 toxic
$ curl -s http://localhost:8474/proxies | python3 -m json.tool | grep -A2 toxics故障注入的通用模板如下
- 每次注入会立即返回 toxic 配置 JSON，可作为成功凭证。注入是即时生效的，下一个 TCP 数据包就会被影响。
bash
curl -X POST http://localhost:8474/proxies/<代理名>/toxics \
-H "Content-Type: application/json" \
-d '{
"name": "<toxic唯一名>", # 用于后续引用/删除
"type": "<latency|timeout|bandwidth|...>", # toxic 类型
"stream": "<upstream|downstream>", # 默认 downstream
"toxicity": <0.0-1.0>, # 默认 1.0,即 100% 触发
"attributes": { ... } # 类型特定参数
}'基线测试
建立"无故障"性能基准，作为后续所有实验的对照组。基线保持代理完全透明。
bash
$ time curl -X POST http://localhost:5000/api/orders \
-H "Content-Type: application/json" \
-d '{"user_id":"user_baseline","items":[{"product_id":"PROD-001","quantity":2,"price":128}]}'响应（70ms）：
json
{
"order_id": "ORD-1780121661-4456",
"user_id": "user_baseline",
"items": [{"price":128, "product_id":"PROD-001", "quantity":2}],
"total": 256,
"status": "created",
"created_at": "2026-05-30T06:14:21.193508"
}应用日志：
2026-05-30 06:14:21,123 [INFO] Creating order ORD-1780121661-4456 for user user_baseline, total=256
2026-05-30 06:14:21,124 [INFO] Checking inventory via toxiproxy...
2026-05-30 06:14:21,140 [INFO] Processing payment via toxiproxy for user_baseline: 256
2026-05-30 06:14:21,180 [INFO] Order ORD-1780121661-4456 saved to PostgreSQL via toxiproxy
2026-05-30 06:14:21,193 [INFO] Order ORD-1780121661-4456 cached
2026-05-30 06:14:21,193 [INFO] Order ORD-1780121661-4456 created successfully链路分解（共 70ms）：
- Inventory 检查：~16ms（HTTP → toxiproxy → inventory → toxiproxy → postgres）
- Payment 处理：~40ms（HTTP → toxiproxy → payment）
- DB 写入：~13ms（psycopg2 → toxiproxy → postgres）
- Redis 写入：~1ms
Inventory 服务注入延迟 2 秒
模拟库存服务因 GC 暂停或慢查询导致响应慢 2 秒的真实场景，验证应用是否能在容忍范围内正常完成下单。
注入命令
bash
$ curl -X POST http://localhost:8474/proxies/inventory-proxy/toxics \
-d '{"name":"inv_lat","type":"latency","stream":"downstream","attributes":{"latency":2000}}'
{"attributes":{"latency":2000,"jitter":0},"name":"inv_lat",
"type":"latency","stream":"downstream","toxicity":1}注入了什么
| 字段 | 值 | 含义 |
|---|---|---|
name |
inv_lat |
toxic 唯一标识，用于后续删除 |
type |
latency |
toxic 类型------网络延迟 |
stream |
downstream |
方向：响应方向（库存服务 → 订单服务） |
latency |
2000 |
每个数据包延迟 2000 毫秒（2秒） |
jitter |
0（默认） |
无抖动，固定延迟 |
toxicity |
1（默认） |
100% 触发率，每次请求都被延迟 |
底层数据流详情
-
**选
downstream**是因为我们要模拟"服务响应慢"，而不是"请求发送慢"。下行延迟模拟的是"上游处理 + 网络回程"的总耗时。订单服务请求 → toxiproxy:8001 (inventory-proxy)
↓
① toxiproxy 立刻转发请求到 inventory-service:5001
② inventory-service 正常处理(~10ms)并返回响应
③ 响应数据进入 downstream toxic 链
④ LatencyToxic 让每个数据包等待 2000ms 后才输出
⑤ 订单服务最终收到响应,总耗时 ≈ 2000ms + 正常耗时
注入后的状态查询：
bash
$ curl -s http://localhost:8474/proxies/inventory-proxy
{
"name": "inventory-proxy",
"listen": "[::]:8001",
"upstream": "inventory-service:5001",
"enabled": true,
"toxics": [
{
"name": "inv_lat",
"type": "latency",
"stream": "downstream",
"toxicity": 1,
"attributes": {"latency": 2000, "jitter": 0}
}
]
}发起订单：
bash
$ time curl -X POST http://localhost:5000/api/orders \
-d '{"user_id":"user_inv","items":[{"product_id":"PROD-002","quantity":1,"price":299}]}'
{"order_id":"ORD-1780121685-8614","status":"created", ...}响应时间：2060ms（基线 70ms + 注入 2000ms）
关键观察：
| 指标 | 基线 | 注入后 | 变化 |
|---|---|---|---|
| 响应时间 | 70ms | 2060ms | +2000ms ≈ 注入值 |
| 业务结果 | 成功 | 成功 | 应用容忍了延迟 |
| 错误日志 | 无 | 无 | 5s timeout > 2s latency |
这印证了 tenacity 配置 ：timeout=5 大于注入延迟 2000ms，所以单次成功，不触发重试。
PostgreSQL 连接累计超时
模拟"数据库网络抖动延迟 1.5 秒"，验证 connect_timeout=3 在协议握手累积下被触发的真实场景------这是云环境中最常见的故障模式之一。
注入命令与参数解读：
bash
$ curl -X POST http://localhost:8474/proxies/postgres-proxy/toxics \
-d '{"name":"db_lat","type":"latency","stream":"downstream","attributes":{"latency":1500}}'注入了什么：
| 字段 | 值 | 含义 |
|---|---|---|
name |
db_lat |
toxic 标识 |
type |
latency |
延迟类型 |
stream |
downstream |
响应方向（PostgreSQL → 应用） |
latency |
1500 |
每个 TCP 包延迟 1.5 秒 |
查询订单：
bash
$ time curl http://localhost:5000/api/orders/ORD-1780121661-4456
{"error":"database unavailable"}
real 0m3.015s1.5 秒乍看"还能接受"，但 PostgreSQL 协议不是单次往返 。它的连接建立是一个多阶段的状态机。为什么 1500ms 延迟会导致失败？ 让我们看 psycopg2 的连接过程：
注意：Toxiproxy 的 toxic 工作在 TCP 连接建立之后的数据流上 。TCP 三次握手（SYN/SYN-ACK/ACK）是网络层操作，由 net.Listener.Accept() → net.Dial() 完成，不经过 toxic pipeline 。stream: downstream 只影响连接内从 PostgreSQL 返回的协议响应数据。
PostgreSQL Toxiproxy (downstream 延迟1500ms) psycopg2 PostgreSQL Toxiproxy (downstream 延迟1500ms) psycopg2 #mermaid-svg-jhGeITD964VQ8tnU{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-jhGeITD964VQ8tnU .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-jhGeITD964VQ8tnU .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-jhGeITD964VQ8tnU .error-icon{fill:#552222;}#mermaid-svg-jhGeITD964VQ8tnU .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-jhGeITD964VQ8tnU .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-jhGeITD964VQ8tnU .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-jhGeITD964VQ8tnU .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-jhGeITD964VQ8tnU .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-jhGeITD964VQ8tnU .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-jhGeITD964VQ8tnU .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-jhGeITD964VQ8tnU .marker{fill:#333333;stroke:#333333;}#mermaid-svg-jhGeITD964VQ8tnU .marker.cross{stroke:#333333;}#mermaid-svg-jhGeITD964VQ8tnU svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-jhGeITD964VQ8tnU p{margin:0;}#mermaid-svg-jhGeITD964VQ8tnU .actor{stroke:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);fill:#ECECFF;}#mermaid-svg-jhGeITD964VQ8tnU text.actor>tspan{fill:black;stroke:none;}#mermaid-svg-jhGeITD964VQ8tnU .actor-line{stroke:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);}#mermaid-svg-jhGeITD964VQ8tnU .innerArc{stroke-width:1.5;stroke-dasharray:none;}#mermaid-svg-jhGeITD964VQ8tnU .messageLine0{stroke-width:1.5;stroke-dasharray:none;stroke:#333;}#mermaid-svg-jhGeITD964VQ8tnU .messageLine1{stroke-width:1.5;stroke-dasharray:2,2;stroke:#333;}#mermaid-svg-jhGeITD964VQ8tnU #arrowhead path{fill:#333;stroke:#333;}#mermaid-svg-jhGeITD964VQ8tnU .sequenceNumber{fill:white;}#mermaid-svg-jhGeITD964VQ8tnU #sequencenumber{fill:#333;}#mermaid-svg-jhGeITD964VQ8tnU #crosshead path{fill:#333;stroke:#333;}#mermaid-svg-jhGeITD964VQ8tnU .messageText{fill:#333;stroke:none;}#mermaid-svg-jhGeITD964VQ8tnU .labelBox{stroke:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);fill:#ECECFF;}#mermaid-svg-jhGeITD964VQ8tnU .labelText,#mermaid-svg-jhGeITD964VQ8tnU .labelText>tspan{fill:black;stroke:none;}#mermaid-svg-jhGeITD964VQ8tnU .loopText,#mermaid-svg-jhGeITD964VQ8tnU .loopText>tspan{fill:black;stroke:none;}#mermaid-svg-jhGeITD964VQ8tnU .loopLine{stroke-width:2px;stroke-dasharray:2,2;stroke:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);fill:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);}#mermaid-svg-jhGeITD964VQ8tnU .note{stroke:#aaaa33;fill:#fff5ad;}#mermaid-svg-jhGeITD964VQ8tnU .noteText,#mermaid-svg-jhGeITD964VQ8tnU .noteText>tspan{fill:black;stroke:none;}#mermaid-svg-jhGeITD964VQ8tnU .activation0{fill:#f4f4f4;stroke:#666;}#mermaid-svg-jhGeITD964VQ8tnU .activation1{fill:#f4f4f4;stroke:#666;}#mermaid-svg-jhGeITD964VQ8tnU .activation2{fill:#f4f4f4;stroke:#666;}#mermaid-svg-jhGeITD964VQ8tnU .actorPopupMenu{position:absolute;}#mermaid-svg-jhGeITD964VQ8tnU .actorPopupMenuPanel{position:absolute;fill:#ECECFF;box-shadow:0px 8px 16px 0px rgba(0,0,0,0.2);filter:drop-shadow(3px 5px 2px rgb(0 0 0 / 0.4));}#mermaid-svg-jhGeITD964VQ8tnU .actor-man line{stroke:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);fill:#ECECFF;}#mermaid-svg-jhGeITD964VQ8tnU .actor-man circle,#mermaid-svg-jhGeITD964VQ8tnU line{stroke:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);fill:#ECECFF;stroke-width:2px;}#mermaid-svg-jhGeITD964VQ8tnU :root{--mermaid-font-family:"trebuchet ms",verdana,arial,sans-serif;} ① TCP 三次握手（网络层，不经过 toxic） 握手完成，~1ms，toxic 未介入 ② PostgreSQL 协议握手（downstream toxic 生效） → upstream 方向，无 toxic，瞬间到达 ⏱ downstream toxic: 延迟 1500ms 累计: 1500ms ⏱ downstream toxic: 延迟 1500ms 累计: 3000ms ⏱ 下行数据继续延迟... connect_timeout=3 已触发 psycopg2 放弃连接 TCP SYN TCP SYN TCP SYN-ACK TCP SYN-ACK StartupMessage (user=orderuser, db=orderdb) AuthenticationOk AuthenticationOk (+1500ms) PasswordMessage (md5) AuthenticationOk AuthenticationOk (+1500ms) ParameterStatus × N
根因分析：
| 阶段 | 方向 | 是否被 toxic 影响 | 延迟 |
|---|---|---|---|
| TCP 三次握手 | 双向 | 不经过 toxic | ~1ms |
| StartupMessage | upstream (App→PG) | 无 upstream toxic | ~1ms |
| AuthenticationOk | downstream (PG→App) | 被延迟 1500ms | +1500ms |
| PasswordMessage | upstream (App→PG) | 无 upstream toxic | ~1ms |
| AuthenticationOk | downstream (PG→App) | 被延迟 1500ms | +1500ms |
| ParameterStatus 等 | downstream (PG→App) | 继续被延迟 | 没等到就超时了 |
connect_timeout=3 计的是从开始连接到收到 ReadyForQuery 的总时间。每个 PostgreSQL 返回的协议消息都 +1500ms，两次认证响应就已达 3000ms，刚好卡在超时阈值上。单次 downstream 延迟看起来"还能接受"，但在 PostgreSQL 协议的多次认证往返中累积后变成致命。
Payment 服务严重延迟
注入超过应用层超时（timeout=10s）的延迟，精确触发 tenacity 重试逻辑，观察重试时间线和最终失败行为。这复现了"下游服务卡死，重试反而让用户等更久"的痛点场景。
注入命令与参数解读：
bash
$ curl -X POST http://localhost:8474/proxies/payment-proxy/toxics \
-d '{"name":"payment_long","type":"latency","attributes":{"latency":15000}}'注入了什么：
| 字段 | 值 | 含义 |
|---|---|---|
name |
payment_long |
toxic 标识 |
type |
latency |
延迟类型 |
stream |
（未指定，默认 downstream） |
响应方向 |
latency |
15000 |
延迟 15 秒 |
为什么选 15 秒？这是精心设计的实验值：
应用配置: timeout=10s (process_payment 函数)
注入延迟: 15s
关系: 15s > 10s
结果: 必然触发 ReadTimeout,然后被 tenacity 接住底层数据流示意图如下：
Payment Service Toxiproxy (latency:15000) Order Service Payment Service Toxiproxy (latency:15000) Order Service #mermaid-svg-mOkX5BJ1rBGkhu85{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-mOkX5BJ1rBGkhu85 .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-mOkX5BJ1rBGkhu85 .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-mOkX5BJ1rBGkhu85 .error-icon{fill:#552222;}#mermaid-svg-mOkX5BJ1rBGkhu85 .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-mOkX5BJ1rBGkhu85 .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-mOkX5BJ1rBGkhu85 .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-mOkX5BJ1rBGkhu85 .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-mOkX5BJ1rBGkhu85 .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-mOkX5BJ1rBGkhu85 .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-mOkX5BJ1rBGkhu85 .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-mOkX5BJ1rBGkhu85 .marker{fill:#333333;stroke:#333333;}#mermaid-svg-mOkX5BJ1rBGkhu85 .marker.cross{stroke:#333333;}#mermaid-svg-mOkX5BJ1rBGkhu85 svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-mOkX5BJ1rBGkhu85 p{margin:0;}#mermaid-svg-mOkX5BJ1rBGkhu85 .actor{stroke:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);fill:#ECECFF;}#mermaid-svg-mOkX5BJ1rBGkhu85 text.actor>tspan{fill:black;stroke:none;}#mermaid-svg-mOkX5BJ1rBGkhu85 .actor-line{stroke:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);}#mermaid-svg-mOkX5BJ1rBGkhu85 .innerArc{stroke-width:1.5;stroke-dasharray:none;}#mermaid-svg-mOkX5BJ1rBGkhu85 .messageLine0{stroke-width:1.5;stroke-dasharray:none;stroke:#333;}#mermaid-svg-mOkX5BJ1rBGkhu85 .messageLine1{stroke-width:1.5;stroke-dasharray:2,2;stroke:#333;}#mermaid-svg-mOkX5BJ1rBGkhu85 #arrowhead path{fill:#333;stroke:#333;}#mermaid-svg-mOkX5BJ1rBGkhu85 .sequenceNumber{fill:white;}#mermaid-svg-mOkX5BJ1rBGkhu85 #sequencenumber{fill:#333;}#mermaid-svg-mOkX5BJ1rBGkhu85 #crosshead path{fill:#333;stroke:#333;}#mermaid-svg-mOkX5BJ1rBGkhu85 .messageText{fill:#333;stroke:none;}#mermaid-svg-mOkX5BJ1rBGkhu85 .labelBox{stroke:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);fill:#ECECFF;}#mermaid-svg-mOkX5BJ1rBGkhu85 .labelText,#mermaid-svg-mOkX5BJ1rBGkhu85 .labelText>tspan{fill:black;stroke:none;}#mermaid-svg-mOkX5BJ1rBGkhu85 .loopText,#mermaid-svg-mOkX5BJ1rBGkhu85 .loopText>tspan{fill:black;stroke:none;}#mermaid-svg-mOkX5BJ1rBGkhu85 .loopLine{stroke-width:2px;stroke-dasharray:2,2;stroke:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);fill:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);}#mermaid-svg-mOkX5BJ1rBGkhu85 .note{stroke:#aaaa33;fill:#fff5ad;}#mermaid-svg-mOkX5BJ1rBGkhu85 .noteText,#mermaid-svg-mOkX5BJ1rBGkhu85 .noteText>tspan{fill:black;stroke:none;}#mermaid-svg-mOkX5BJ1rBGkhu85 .activation0{fill:#f4f4f4;stroke:#666;}#mermaid-svg-mOkX5BJ1rBGkhu85 .activation1{fill:#f4f4f4;stroke:#666;}#mermaid-svg-mOkX5BJ1rBGkhu85 .activation2{fill:#f4f4f4;stroke:#666;}#mermaid-svg-mOkX5BJ1rBGkhu85 .actorPopupMenu{position:absolute;}#mermaid-svg-mOkX5BJ1rBGkhu85 .actorPopupMenuPanel{position:absolute;fill:#ECECFF;box-shadow:0px 8px 16px 0px rgba(0,0,0,0.2);filter:drop-shadow(3px 5px 2px rgb(0 0 0 / 0.4));}#mermaid-svg-mOkX5BJ1rBGkhu85 .actor-man line{stroke:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);fill:#ECECFF;}#mermaid-svg-mOkX5BJ1rBGkhu85 .actor-man circle,#mermaid-svg-mOkX5BJ1rBGkhu85 line{stroke:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);fill:#ECECFF;stroke-width:2px;}#mermaid-svg-mOkX5BJ1rBGkhu85 :root{--mermaid-font-family:"trebuchet ms",verdana,arial,sans-serif;} ⏱ LatencyToxic 持有响应数据 等待 15000ms 10秒后,requests库 ReadTimeout tenacity 接住异常,触发重试 5秒后才会真正放行响应 但客户端已断开,数据被丢弃 POST /charge (建立TCP连接) 转发请求 响应(<10ms 完成) ⏱ 等待中... 主动关闭 TCP
发起订单，请求失败，实践为21 秒
bash
$ time curl -X POST http://localhost:5000/api/orders \
-d '{"user_id":"user_retry","items":[{"product_id":"PROD-001","quantity":1,"price":128}]}'
{"error":"RetryError[<Future at 0x7f8641affe30 state=finished raised ReadTimeout>]"}
real 0m21.032s完整重试时间线（应用日志）：
-
tenacity 配置
stop_after_attempt(2)表示总共 2 次（不是"额外重试 2 次"） -
总耗时 = 单次超时 × 尝试次数 + 退避总和
-
用户感知：21 秒后才看到失败------这往往比快速失败更糟糕
06:14:00.000 [INFO] Creating order ORD-... for user user_retry
06:14:00.005 [INFO] Checking inventory via toxiproxy... [16ms - inventory正常]
06:14:00.021 [INFO] Processing payment via toxiproxy for user_retry: 128
06:14:10.025 [ERROR] Payment timeout ← ① 第1次超时(10s)
06:14:10.526 [INFO] Processing payment via toxiproxy for user_retry: 128 ← ② 退避0.5s后重试
06:14:20.530 [ERROR] Payment timeout ← ③ 第2次超时(10s)
06:14:20.531 [ERROR] Order creation failed: RetryError[...] ← ④ 重试耗尽
Timeout Toxic vs Latency Toxic
timeout toxic 不是"无限延迟"，而是完全不同的故障语义。理解它能避免混沌实验设计错误。
注入命令与参数解读：
bash
$ curl -X POST http://localhost:8474/proxies/payment-proxy/toxics \
-d '{"name":"payment_to","type":"timeout","attributes":{"timeout":0}}'注入了什么：
| 字段 | 值 | 含义 |
|---|---|---|
name |
payment_to |
toxic 标识 |
type |
timeout |
超时丢包类型（注意：不是 latency） |
timeout |
0 |
特殊值：永远不触发主动关闭 |
关键源码回顾 （来自 3.6 节 timeout.go）：
go
if timeout > 0 {
// 模式A: 等 timeout 毫秒后强制关闭连接
} else {
// 模式B: timeout=0,进入"永远丢弃数据"模式
for {
select {
case <-stub.Interrupt: return
case c := <-stub.Input:
if c == nil { stub.Close(); return }
// 静默丢弃所有数据,永不响应,永不主动关闭
}
}
}timeout=0 实际行为：
- 建立TCP连接不受影响，是因为
Toxics.StartLink()在 TCP 握手之后 才被调用。Toxic Pipeline 工作在已建立的 TCP 连接之上 ，处理的是连接内的数据流，而不是连接建立过程本身。
Payment Service Toxiproxy (timeout:0) App Payment Service Toxiproxy (timeout:0) App #mermaid-svg-KnMLZXRSbqlB8qLb{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-KnMLZXRSbqlB8qLb .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-KnMLZXRSbqlB8qLb .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-KnMLZXRSbqlB8qLb .error-icon{fill:#552222;}#mermaid-svg-KnMLZXRSbqlB8qLb .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-KnMLZXRSbqlB8qLb .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-KnMLZXRSbqlB8qLb .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-KnMLZXRSbqlB8qLb .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-KnMLZXRSbqlB8qLb .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-KnMLZXRSbqlB8qLb .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-KnMLZXRSbqlB8qLb .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-KnMLZXRSbqlB8qLb .marker{fill:#333333;stroke:#333333;}#mermaid-svg-KnMLZXRSbqlB8qLb .marker.cross{stroke:#333333;}#mermaid-svg-KnMLZXRSbqlB8qLb svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-KnMLZXRSbqlB8qLb p{margin:0;}#mermaid-svg-KnMLZXRSbqlB8qLb .actor{stroke:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);fill:#ECECFF;}#mermaid-svg-KnMLZXRSbqlB8qLb text.actor>tspan{fill:black;stroke:none;}#mermaid-svg-KnMLZXRSbqlB8qLb .actor-line{stroke:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);}#mermaid-svg-KnMLZXRSbqlB8qLb .innerArc{stroke-width:1.5;stroke-dasharray:none;}#mermaid-svg-KnMLZXRSbqlB8qLb .messageLine0{stroke-width:1.5;stroke-dasharray:none;stroke:#333;}#mermaid-svg-KnMLZXRSbqlB8qLb .messageLine1{stroke-width:1.5;stroke-dasharray:2,2;stroke:#333;}#mermaid-svg-KnMLZXRSbqlB8qLb #arrowhead path{fill:#333;stroke:#333;}#mermaid-svg-KnMLZXRSbqlB8qLb .sequenceNumber{fill:white;}#mermaid-svg-KnMLZXRSbqlB8qLb #sequencenumber{fill:#333;}#mermaid-svg-KnMLZXRSbqlB8qLb #crosshead path{fill:#333;stroke:#333;}#mermaid-svg-KnMLZXRSbqlB8qLb .messageText{fill:#333;stroke:none;}#mermaid-svg-KnMLZXRSbqlB8qLb .labelBox{stroke:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);fill:#ECECFF;}#mermaid-svg-KnMLZXRSbqlB8qLb .labelText,#mermaid-svg-KnMLZXRSbqlB8qLb .labelText>tspan{fill:black;stroke:none;}#mermaid-svg-KnMLZXRSbqlB8qLb .loopText,#mermaid-svg-KnMLZXRSbqlB8qLb .loopText>tspan{fill:black;stroke:none;}#mermaid-svg-KnMLZXRSbqlB8qLb .loopLine{stroke-width:2px;stroke-dasharray:2,2;stroke:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);fill:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);}#mermaid-svg-KnMLZXRSbqlB8qLb .note{stroke:#aaaa33;fill:#fff5ad;}#mermaid-svg-KnMLZXRSbqlB8qLb .noteText,#mermaid-svg-KnMLZXRSbqlB8qLb .noteText>tspan{fill:black;stroke:none;}#mermaid-svg-KnMLZXRSbqlB8qLb .activation0{fill:#f4f4f4;stroke:#666;}#mermaid-svg-KnMLZXRSbqlB8qLb .activation1{fill:#f4f4f4;stroke:#666;}#mermaid-svg-KnMLZXRSbqlB8qLb .activation2{fill:#f4f4f4;stroke:#666;}#mermaid-svg-KnMLZXRSbqlB8qLb .actorPopupMenu{position:absolute;}#mermaid-svg-KnMLZXRSbqlB8qLb .actorPopupMenuPanel{position:absolute;fill:#ECECFF;box-shadow:0px 8px 16px 0px rgba(0,0,0,0.2);filter:drop-shadow(3px 5px 2px rgb(0 0 0 / 0.4));}#mermaid-svg-KnMLZXRSbqlB8qLb .actor-man line{stroke:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);fill:#ECECFF;}#mermaid-svg-KnMLZXRSbqlB8qLb .actor-man circle,#mermaid-svg-KnMLZXRSbqlB8qLb line{stroke:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);fill:#ECECFF;stroke-width:2px;}#mermaid-svg-KnMLZXRSbqlB8qLb :root{--mermaid-font-family:"trebuchet ms",verdana,arial,sans-serif;} 连接建立成功(toxic不影响TCP握手) 🗑️ 数据被丢弃 (Drop on the ground) 请求永远不会到达 Payment Service 应用等待响应... TCP 层因双方都不发数据 OS 触发 RST 或 keepalive 超时 ⚡ 立即收到 RemoteDisconnected 不到 30ms ① 建立 TCP 连接 ✅ ② 发送 HTTP 请求数据 TCP RST / Connection Reset
对比三种"超时"故障模式：
| 注入方式 | 应用看到的 | 触发时间 | 应用日志特征 |
|---|---|---|---|
latency: 99999999（接近无限延迟） |
ReadTimeout |
应用层 timeout 触发（如 10s） | "Timeout while reading" |
timeout: 0（本实验） |
ConnectionError |
TCP RST 立即返回（~30ms） | "RemoteDisconnected" |
timeout: 5000 |
先发数据，5s 后被强制断 | 5 秒 | "Connection aborted" |
生产场景对应：
latency极大值 → 模拟"服务僵死，慢得像挂了"timeout: 0→ 模拟"防火墙黑洞、负载均衡器丢包"timeout: N→ 模拟"主动断连，如 Nginx upstream timeout"
结果：
bash
$ time curl -X POST http://localhost:5000/api/orders ...
{"error":"payment failed"}
real 0m0.036s应用日志：
-
timeout=0让 toxic 进入"永远丢弃数据"模式 -
但 toxic 自己不主动关闭连接
-
客户端发请求 → 数据被丢 → TCP 层最终因为 keepalive 或 buffer 满而 reset
-
表现为
RemoteDisconnected，不触发 ReadTimeout06:15:00.438 [INFO] Creating order ORD-...
06:15:00.460 [WARNING] Payment failed:
('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))
对比表：
| Toxic | 应用看到的错误 | 触发时间 | tenacity 是否重试 |
|---|---|---|---|
timeout |
ConnectionError / RemoteDisconnected |
立即 (~30ms) | 是（ConnectionError 在重试列表） |
latency: 15000 |
Timeout / ReadTimeout |
单次超时 (10s) | 是（Timeout 在重试列表） |
Toxiproxy API 速查
bash
# 代理管理
GET /proxies # 列出所有代理
POST /proxies # 创建代理
GET /proxies/{name} # 查看代理
POST /proxies/{name}/enable # 启用
POST /proxies/{name}/disable # 禁用
DELETE /proxies/{name} # 删除
# Toxic 管理
POST /proxies/{name}/toxics # 添加 toxic
GET /proxies/{name}/toxics # 列出 toxics
POST /proxies/{name}/toxics/{tname} # 修改 toxic
DELETE /proxies/{name}/toxics/{tname} # 删除 toxic
# Toxic 通用参数
{
"name": "my_toxic", # 唯一标识
"type": "latency", # toxic 类型
"stream": "downstream", # 方向 upstream/downstream
"toxicity": 1.0, # 触发概率 0.0-1.0
"attributes": {...} # toxic 特定参数
}