Transformer 与大语言模型：第10章 Residual （残差连接）

richard_first2026-08-27 16:04

第10章 Residual（残差连接）

本章目标：

理解残差连接为什么能防止梯度消失，以及它在 Transformer 中的作用。

10.1 深层网络的问题

假设我们有一个 100 层的神经网络：
#mermaid-svg-lUPpeyYgbR0Tc4kd{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-lUPpeyYgbR0Tc4kd .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-lUPpeyYgbR0Tc4kd .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-lUPpeyYgbR0Tc4kd .error-icon{fill:#552222;}#mermaid-svg-lUPpeyYgbR0Tc4kd .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-lUPpeyYgbR0Tc4kd .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-lUPpeyYgbR0Tc4kd .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-lUPpeyYgbR0Tc4kd .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-lUPpeyYgbR0Tc4kd .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-lUPpeyYgbR0Tc4kd .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-lUPpeyYgbR0Tc4kd .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-lUPpeyYgbR0Tc4kd .marker{fill:#333333;stroke:#333333;}#mermaid-svg-lUPpeyYgbR0Tc4kd .marker.cross{stroke:#333333;}#mermaid-svg-lUPpeyYgbR0Tc4kd svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-lUPpeyYgbR0Tc4kd p{margin:0;}#mermaid-svg-lUPpeyYgbR0Tc4kd .label{font-family:"trebuchet ms",verdana,arial,sans-serif;color:#333;}#mermaid-svg-lUPpeyYgbR0Tc4kd .cluster-label text{fill:#333;}#mermaid-svg-lUPpeyYgbR0Tc4kd .cluster-label span{color:#333;}#mermaid-svg-lUPpeyYgbR0Tc4kd .cluster-label span p{background-color:transparent;}#mermaid-svg-lUPpeyYgbR0Tc4kd .label text,#mermaid-svg-lUPpeyYgbR0Tc4kd span{fill:#333;color:#333;}#mermaid-svg-lUPpeyYgbR0Tc4kd .node rect,#mermaid-svg-lUPpeyYgbR0Tc4kd .node circle,#mermaid-svg-lUPpeyYgbR0Tc4kd .node ellipse,#mermaid-svg-lUPpeyYgbR0Tc4kd .node polygon,#mermaid-svg-lUPpeyYgbR0Tc4kd .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-lUPpeyYgbR0Tc4kd .rough-node .label text,#mermaid-svg-lUPpeyYgbR0Tc4kd .node .label text,#mermaid-svg-lUPpeyYgbR0Tc4kd .image-shape .label,#mermaid-svg-lUPpeyYgbR0Tc4kd .icon-shape .label{text-anchor:middle;}#mermaid-svg-lUPpeyYgbR0Tc4kd .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-lUPpeyYgbR0Tc4kd .rough-node .label,#mermaid-svg-lUPpeyYgbR0Tc4kd .node .label,#mermaid-svg-lUPpeyYgbR0Tc4kd .image-shape .label,#mermaid-svg-lUPpeyYgbR0Tc4kd .icon-shape .label{text-align:center;}#mermaid-svg-lUPpeyYgbR0Tc4kd .node.clickable{cursor:pointer;}#mermaid-svg-lUPpeyYgbR0Tc4kd .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-lUPpeyYgbR0Tc4kd .arrowheadPath{fill:#333333;}#mermaid-svg-lUPpeyYgbR0Tc4kd .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-lUPpeyYgbR0Tc4kd .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-lUPpeyYgbR0Tc4kd .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-lUPpeyYgbR0Tc4kd .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-lUPpeyYgbR0Tc4kd .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-lUPpeyYgbR0Tc4kd .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-lUPpeyYgbR0Tc4kd .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-lUPpeyYgbR0Tc4kd .cluster text{fill:#333;}#mermaid-svg-lUPpeyYgbR0Tc4kd .cluster span{color:#333;}#mermaid-svg-lUPpeyYgbR0Tc4kd div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-lUPpeyYgbR0Tc4kd .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-lUPpeyYgbR0Tc4kd rect.text{fill:none;stroke-width:0;}#mermaid-svg-lUPpeyYgbR0Tc4kd .icon-shape,#mermaid-svg-lUPpeyYgbR0Tc4kd .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-lUPpeyYgbR0Tc4kd .icon-shape p,#mermaid-svg-lUPpeyYgbR0Tc4kd .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-lUPpeyYgbR0Tc4kd .icon-shape .label rect,#mermaid-svg-lUPpeyYgbR0Tc4kd .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-lUPpeyYgbR0Tc4kd .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-lUPpeyYgbR0Tc4kd .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-lUPpeyYgbR0Tc4kd :root{--mermaid-font-family:"trebuchet ms",verdana,arial,sans-serif;} 输入
Layer 1
Layer 2
...
Layer 100
输出

理论上，层数越多，表达能力越强。

但实际上，层数太多会出现两个问题：

  1. 梯度消失：反向传播时，梯度越来越小，前面的层几乎不更新
  2. 退化问题：更深的网络反而比浅层网络效果差

10.2 梯度消失的原因

反向传播时，梯度通过链式法则传递：

∂L∂W1=∂L∂y⋅∂y∂h100⋅∂h100∂h99⋯∂h2∂h1⋅∂h1∂W1\frac{\partial L}{\partial W_1} = \frac{\partial L}{\partial y} \cdot \frac{\partial y}{\partial h_{100}} \cdot \frac{\partial h_{100}}{\partial h_{99}} \cdots \frac{\partial h_2}{\partial h_1} \cdot \frac{\partial h_1}{\partial W_1}∂W1∂L=∂y∂L⋅∂h100∂y⋅∂h99∂h100⋯∂h1∂h2⋅∂W1∂h1

如果每一层的梯度都小于 1（例如 0.9）：

0.9100≈0.0000270.9^{100} \approx 0.0000270.9100≈0.000027

梯度几乎为 0，第一层几乎无法更新。

10.3 残差连接的思想

ResNet（2015年）提出了残差连接：
不学习 F(x)，而是学习 F(x) + x。
#mermaid-svg-r9A4OcJFdXsJYInB{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-r9A4OcJFdXsJYInB .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-r9A4OcJFdXsJYInB .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-r9A4OcJFdXsJYInB .error-icon{fill:#552222;}#mermaid-svg-r9A4OcJFdXsJYInB .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-r9A4OcJFdXsJYInB .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-r9A4OcJFdXsJYInB .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-r9A4OcJFdXsJYInB .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-r9A4OcJFdXsJYInB .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-r9A4OcJFdXsJYInB .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-r9A4OcJFdXsJYInB .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-r9A4OcJFdXsJYInB .marker{fill:#333333;stroke:#333333;}#mermaid-svg-r9A4OcJFdXsJYInB .marker.cross{stroke:#333333;}#mermaid-svg-r9A4OcJFdXsJYInB svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-r9A4OcJFdXsJYInB p{margin:0;}#mermaid-svg-r9A4OcJFdXsJYInB .label{font-family:"trebuchet ms",verdana,arial,sans-serif;color:#333;}#mermaid-svg-r9A4OcJFdXsJYInB .cluster-label text{fill:#333;}#mermaid-svg-r9A4OcJFdXsJYInB .cluster-label span{color:#333;}#mermaid-svg-r9A4OcJFdXsJYInB .cluster-label span p{background-color:transparent;}#mermaid-svg-r9A4OcJFdXsJYInB .label text,#mermaid-svg-r9A4OcJFdXsJYInB span{fill:#333;color:#333;}#mermaid-svg-r9A4OcJFdXsJYInB .node rect,#mermaid-svg-r9A4OcJFdXsJYInB .node circle,#mermaid-svg-r9A4OcJFdXsJYInB .node ellipse,#mermaid-svg-r9A4OcJFdXsJYInB .node polygon,#mermaid-svg-r9A4OcJFdXsJYInB .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-r9A4OcJFdXsJYInB .rough-node .label text,#mermaid-svg-r9A4OcJFdXsJYInB .node .label text,#mermaid-svg-r9A4OcJFdXsJYInB .image-shape .label,#mermaid-svg-r9A4OcJFdXsJYInB .icon-shape .label{text-anchor:middle;}#mermaid-svg-r9A4OcJFdXsJYInB .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-r9A4OcJFdXsJYInB .rough-node .label,#mermaid-svg-r9A4OcJFdXsJYInB .node .label,#mermaid-svg-r9A4OcJFdXsJYInB .image-shape .label,#mermaid-svg-r9A4OcJFdXsJYInB .icon-shape .label{text-align:center;}#mermaid-svg-r9A4OcJFdXsJYInB .node.clickable{cursor:pointer;}#mermaid-svg-r9A4OcJFdXsJYInB .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-r9A4OcJFdXsJYInB .arrowheadPath{fill:#333333;}#mermaid-svg-r9A4OcJFdXsJYInB .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-r9A4OcJFdXsJYInB .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-r9A4OcJFdXsJYInB .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-r9A4OcJFdXsJYInB .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-r9A4OcJFdXsJYInB .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-r9A4OcJFdXsJYInB .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-r9A4OcJFdXsJYInB .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-r9A4OcJFdXsJYInB .cluster text{fill:#333;}#mermaid-svg-r9A4OcJFdXsJYInB .cluster span{color:#333;}#mermaid-svg-r9A4OcJFdXsJYInB div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-r9A4OcJFdXsJYInB .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-r9A4OcJFdXsJYInB rect.text{fill:none;stroke-width:0;}#mermaid-svg-r9A4OcJFdXsJYInB .icon-shape,#mermaid-svg-r9A4OcJFdXsJYInB .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-r9A4OcJFdXsJYInB .icon-shape p,#mermaid-svg-r9A4OcJFdXsJYInB .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-r9A4OcJFdXsJYInB .icon-shape .label rect,#mermaid-svg-r9A4OcJFdXsJYInB .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-r9A4OcJFdXsJYInB .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-r9A4OcJFdXsJYInB .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-r9A4OcJFdXsJYInB :root{--mermaid-font-family:"trebuchet ms",verdana,arial,sans-serif;} 输入 x
子层 F
+
输出 F(x) + x

数学表示：

y=F(x)+xy = F(x) + xy=F(x)+x

用 bank 的例子理解这个设计的深意：

假设当前 bank 的向量表示"可能是河岸"。Attention 从上下文收集信息后，认为"可能是银行"。

如果没有残差连接：

text 复制代码 
bank → Attention → 直接输出"银行"（原始的"河岸"信息被覆盖了）

如果 Attention 算错了呢？原始信息彻底丢失，无法挽回。

如果有残差连接：

text 复制代码 
bank → Attention 输出 + 原始 bank → "河岸信息 + 银行信息 都保留"

即使 Attention 犯了错，原始信息也还在------后面的层仍然有机会纠正。

💡 核心直觉 ：残差连接意味着每一层的输出是建议（suggestion） ，不是命令（override）。子层说"我觉得应该这样修改"，然后把修改量叠加到原有信息上。如果修改是错的，至少原始信息还在。

这很像人类学习的过程：

你一直认为 bank 是"河岸"。有一天老师告诉你，在金融语境下 bank 是"银行"。你不会立刻把"河岸"这个含义从脑子里删掉------你会保留原有认知，同时叠加新的理解。这就是：

新认知=旧认知+老师教的新东西\text{新认知} = \text{旧认知} + \text{老师教的新东西}新认知=旧认知+老师教的新东西

10.4 为什么残差连接能防止梯度消失？

反向传播时：

∂y∂x=∂F(x)∂x+1\frac{\partial y}{\partial x} = \frac{\partial F(x)}{\partial x} + 1∂x∂y=∂x∂F(x)+1

关键在于这个 +1。

即使 ∂F(x)∂x\frac{\partial F(x)}{\partial x}∂x∂F(x) 很小（接近 0），梯度也至少是 1。

梯度不会消失！
#mermaid-svg-ApN79wlsCS2zmPWw{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-ApN79wlsCS2zmPWw .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-ApN79wlsCS2zmPWw .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-ApN79wlsCS2zmPWw .error-icon{fill:#552222;}#mermaid-svg-ApN79wlsCS2zmPWw .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-ApN79wlsCS2zmPWw .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-ApN79wlsCS2zmPWw .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-ApN79wlsCS2zmPWw .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-ApN79wlsCS2zmPWw .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-ApN79wlsCS2zmPWw .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-ApN79wlsCS2zmPWw .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-ApN79wlsCS2zmPWw .marker{fill:#333333;stroke:#333333;}#mermaid-svg-ApN79wlsCS2zmPWw .marker.cross{stroke:#333333;}#mermaid-svg-ApN79wlsCS2zmPWw svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-ApN79wlsCS2zmPWw p{margin:0;}#mermaid-svg-ApN79wlsCS2zmPWw .label{font-family:"trebuchet ms",verdana,arial,sans-serif;color:#333;}#mermaid-svg-ApN79wlsCS2zmPWw .cluster-label text{fill:#333;}#mermaid-svg-ApN79wlsCS2zmPWw .cluster-label span{color:#333;}#mermaid-svg-ApN79wlsCS2zmPWw .cluster-label span p{background-color:transparent;}#mermaid-svg-ApN79wlsCS2zmPWw .label text,#mermaid-svg-ApN79wlsCS2zmPWw span{fill:#333;color:#333;}#mermaid-svg-ApN79wlsCS2zmPWw .node rect,#mermaid-svg-ApN79wlsCS2zmPWw .node circle,#mermaid-svg-ApN79wlsCS2zmPWw .node ellipse,#mermaid-svg-ApN79wlsCS2zmPWw .node polygon,#mermaid-svg-ApN79wlsCS2zmPWw .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-ApN79wlsCS2zmPWw .rough-node .label text,#mermaid-svg-ApN79wlsCS2zmPWw .node .label text,#mermaid-svg-ApN79wlsCS2zmPWw .image-shape .label,#mermaid-svg-ApN79wlsCS2zmPWw .icon-shape .label{text-anchor:middle;}#mermaid-svg-ApN79wlsCS2zmPWw .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-ApN79wlsCS2zmPWw .rough-node .label,#mermaid-svg-ApN79wlsCS2zmPWw .node .label,#mermaid-svg-ApN79wlsCS2zmPWw .image-shape .label,#mermaid-svg-ApN79wlsCS2zmPWw .icon-shape .label{text-align:center;}#mermaid-svg-ApN79wlsCS2zmPWw .node.clickable{cursor:pointer;}#mermaid-svg-ApN79wlsCS2zmPWw .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-ApN79wlsCS2zmPWw .arrowheadPath{fill:#333333;}#mermaid-svg-ApN79wlsCS2zmPWw .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-ApN79wlsCS2zmPWw .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-ApN79wlsCS2zmPWw .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-ApN79wlsCS2zmPWw .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-ApN79wlsCS2zmPWw .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-ApN79wlsCS2zmPWw .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-ApN79wlsCS2zmPWw .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-ApN79wlsCS2zmPWw .cluster text{fill:#333;}#mermaid-svg-ApN79wlsCS2zmPWw .cluster span{color:#333;}#mermaid-svg-ApN79wlsCS2zmPWw div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-ApN79wlsCS2zmPWw .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-ApN79wlsCS2zmPWw rect.text{fill:none;stroke-width:0;}#mermaid-svg-ApN79wlsCS2zmPWw .icon-shape,#mermaid-svg-ApN79wlsCS2zmPWw .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-ApN79wlsCS2zmPWw .icon-shape p,#mermaid-svg-ApN79wlsCS2zmPWw .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-ApN79wlsCS2zmPWw .icon-shape .label rect,#mermaid-svg-ApN79wlsCS2zmPWw .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-ApN79wlsCS2zmPWw .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-ApN79wlsCS2zmPWw .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-ApN79wlsCS2zmPWw :root{--mermaid-font-family:"trebuchet ms",verdana,arial,sans-serif;} 有残差连接
梯度 0.9
+1 = 1.9
+1 = 2.9
+1 = 3.9
梯度不消失
没有残差连接
梯度 0.9
×0.9
×0.9
×0.9
0.9^100 ≈ 0

10.5 残差连接的另一个好处

残差连接提供了一条信息高速公路：
#mermaid-svg-PRwyAkXzABVS1kne{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-PRwyAkXzABVS1kne .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-PRwyAkXzABVS1kne .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-PRwyAkXzABVS1kne .error-icon{fill:#552222;}#mermaid-svg-PRwyAkXzABVS1kne .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-PRwyAkXzABVS1kne .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-PRwyAkXzABVS1kne .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-PRwyAkXzABVS1kne .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-PRwyAkXzABVS1kne .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-PRwyAkXzABVS1kne .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-PRwyAkXzABVS1kne .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-PRwyAkXzABVS1kne .marker{fill:#333333;stroke:#333333;}#mermaid-svg-PRwyAkXzABVS1kne .marker.cross{stroke:#333333;}#mermaid-svg-PRwyAkXzABVS1kne svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-PRwyAkXzABVS1kne p{margin:0;}#mermaid-svg-PRwyAkXzABVS1kne .label{font-family:"trebuchet ms",verdana,arial,sans-serif;color:#333;}#mermaid-svg-PRwyAkXzABVS1kne .cluster-label text{fill:#333;}#mermaid-svg-PRwyAkXzABVS1kne .cluster-label span{color:#333;}#mermaid-svg-PRwyAkXzABVS1kne .cluster-label span p{background-color:transparent;}#mermaid-svg-PRwyAkXzABVS1kne .label text,#mermaid-svg-PRwyAkXzABVS1kne span{fill:#333;color:#333;}#mermaid-svg-PRwyAkXzABVS1kne .node rect,#mermaid-svg-PRwyAkXzABVS1kne .node circle,#mermaid-svg-PRwyAkXzABVS1kne .node ellipse,#mermaid-svg-PRwyAkXzABVS1kne .node polygon,#mermaid-svg-PRwyAkXzABVS1kne .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-PRwyAkXzABVS1kne .rough-node .label text,#mermaid-svg-PRwyAkXzABVS1kne .node .label text,#mermaid-svg-PRwyAkXzABVS1kne .image-shape .label,#mermaid-svg-PRwyAkXzABVS1kne .icon-shape .label{text-anchor:middle;}#mermaid-svg-PRwyAkXzABVS1kne .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-PRwyAkXzABVS1kne .rough-node .label,#mermaid-svg-PRwyAkXzABVS1kne .node .label,#mermaid-svg-PRwyAkXzABVS1kne .image-shape .label,#mermaid-svg-PRwyAkXzABVS1kne .icon-shape .label{text-align:center;}#mermaid-svg-PRwyAkXzABVS1kne .node.clickable{cursor:pointer;}#mermaid-svg-PRwyAkXzABVS1kne .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-PRwyAkXzABVS1kne .arrowheadPath{fill:#333333;}#mermaid-svg-PRwyAkXzABVS1kne .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-PRwyAkXzABVS1kne .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-PRwyAkXzABVS1kne .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-PRwyAkXzABVS1kne .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-PRwyAkXzABVS1kne .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-PRwyAkXzABVS1kne .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-PRwyAkXzABVS1kne .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-PRwyAkXzABVS1kne .cluster text{fill:#333;}#mermaid-svg-PRwyAkXzABVS1kne .cluster span{color:#333;}#mermaid-svg-PRwyAkXzABVS1kne div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-PRwyAkXzABVS1kne .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-PRwyAkXzABVS1kne rect.text{fill:none;stroke-width:0;}#mermaid-svg-PRwyAkXzABVS1kne .icon-shape,#mermaid-svg-PRwyAkXzABVS1kne .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-PRwyAkXzABVS1kne .icon-shape p,#mermaid-svg-PRwyAkXzABVS1kne .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-PRwyAkXzABVS1kne .icon-shape .label rect,#mermaid-svg-PRwyAkXzABVS1kne .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-PRwyAkXzABVS1kne .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-PRwyAkXzABVS1kne .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-PRwyAkXzABVS1kne :root{--mermaid-font-family:"trebuchet ms",verdana,arial,sans-serif;} 输入 x
Attention
直接跳过

（高速公路）
+
LayerNorm
FFN
直接跳过

（高速公路）
+
LayerNorm
输出

原始输入 x 可以直接流到任何深度，不需要经过所有层的变换。

这意味着：

即使某些层学到了错误的东西，原始信息也不会丢失。

10.6 那梯度爆炸呢？

前面说残差连接让梯度"至少为1"，解决了梯度消失。但一个自然的疑问随之而来：

如果梯度至少为1，堆96层（如GPT-3），梯度不会越来越大吗？

答案是：残差连接确实引入了梯度爆炸的风险 ，但 Transformer 通过多种机制组合来防止它：

10.6.1 梯度的真实路径

残差网络的梯度展开后是：

∂L∂x0=∂L∂xL∏i=0L−1(I+∂Fi∂xi)\frac{\partial L}{\partial x_0} = \frac{\partial L}{\partial x_L} \prod_{i=0}^{L-1} \left(I + \frac{\partial F_i}{\partial x_i}\right)∂x0∂L=∂xL∂Li=0∏L−1(I+∂xi∂Fi)

展开这个乘积，会得到 2L2^L2L 条不同的梯度路径（类似 DenseNet 的效果）：有的路径经过所有层的 F，有的跳过一些层。最终梯度是所有路径的叠加，而非简单的连乘。

10.6.2 防止梯度爆炸的完整武器库

机制 作用
LayerNorm / RMSNorm 限制每层输出的范围，防止数值爆炸
梯度裁剪（Gradient Clipping） 当梯度范数超过阈值时直接截断
学习率 Warmup 训练初期用极小学习率，等参数稳定后再增大
Pre-Norm 结构 残差连接"干净"（不经过 Norm），梯度传播更平稳
初始化策略 用较小的初始权重，使 F(x) 初始时接近 0
Weight Decay 防止权重过大

10.6.3 为什么 GPT-3（96层）能工作？

深层 Transformer 能工作，不是单靠残差连接，而是上述所有技术的组合效应：
#mermaid-svg-2FjoAw2kgzJt4gQt{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-2FjoAw2kgzJt4gQt .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-2FjoAw2kgzJt4gQt .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-2FjoAw2kgzJt4gQt .error-icon{fill:#552222;}#mermaid-svg-2FjoAw2kgzJt4gQt .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-2FjoAw2kgzJt4gQt .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-2FjoAw2kgzJt4gQt .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-2FjoAw2kgzJt4gQt .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-2FjoAw2kgzJt4gQt .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-2FjoAw2kgzJt4gQt .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-2FjoAw2kgzJt4gQt .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-2FjoAw2kgzJt4gQt .marker{fill:#333333;stroke:#333333;}#mermaid-svg-2FjoAw2kgzJt4gQt .marker.cross{stroke:#333333;}#mermaid-svg-2FjoAw2kgzJt4gQt svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-2FjoAw2kgzJt4gQt p{margin:0;}#mermaid-svg-2FjoAw2kgzJt4gQt .label{font-family:"trebuchet ms",verdana,arial,sans-serif;color:#333;}#mermaid-svg-2FjoAw2kgzJt4gQt .cluster-label text{fill:#333;}#mermaid-svg-2FjoAw2kgzJt4gQt .cluster-label span{color:#333;}#mermaid-svg-2FjoAw2kgzJt4gQt .cluster-label span p{background-color:transparent;}#mermaid-svg-2FjoAw2kgzJt4gQt .label text,#mermaid-svg-2FjoAw2kgzJt4gQt span{fill:#333;color:#333;}#mermaid-svg-2FjoAw2kgzJt4gQt .node rect,#mermaid-svg-2FjoAw2kgzJt4gQt .node circle,#mermaid-svg-2FjoAw2kgzJt4gQt .node ellipse,#mermaid-svg-2FjoAw2kgzJt4gQt .node polygon,#mermaid-svg-2FjoAw2kgzJt4gQt .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-2FjoAw2kgzJt4gQt .rough-node .label text,#mermaid-svg-2FjoAw2kgzJt4gQt .node .label text,#mermaid-svg-2FjoAw2kgzJt4gQt .image-shape .label,#mermaid-svg-2FjoAw2kgzJt4gQt .icon-shape .label{text-anchor:middle;}#mermaid-svg-2FjoAw2kgzJt4gQt .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-2FjoAw2kgzJt4gQt .rough-node .label,#mermaid-svg-2FjoAw2kgzJt4gQt .node .label,#mermaid-svg-2FjoAw2kgzJt4gQt .image-shape .label,#mermaid-svg-2FjoAw2kgzJt4gQt .icon-shape .label{text-align:center;}#mermaid-svg-2FjoAw2kgzJt4gQt .node.clickable{cursor:pointer;}#mermaid-svg-2FjoAw2kgzJt4gQt .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-2FjoAw2kgzJt4gQt .arrowheadPath{fill:#333333;}#mermaid-svg-2FjoAw2kgzJt4gQt .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-2FjoAw2kgzJt4gQt .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-2FjoAw2kgzJt4gQt .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-2FjoAw2kgzJt4gQt .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-2FjoAw2kgzJt4gQt .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-2FjoAw2kgzJt4gQt .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-2FjoAw2kgzJt4gQt .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-2FjoAw2kgzJt4gQt .cluster text{fill:#333;}#mermaid-svg-2FjoAw2kgzJt4gQt .cluster span{color:#333;}#mermaid-svg-2FjoAw2kgzJt4gQt div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-2FjoAw2kgzJt4gQt .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-2FjoAw2kgzJt4gQt rect.text{fill:none;stroke-width:0;}#mermaid-svg-2FjoAw2kgzJt4gQt .icon-shape,#mermaid-svg-2FjoAw2kgzJt4gQt .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-2FjoAw2kgzJt4gQt .icon-shape p,#mermaid-svg-2FjoAw2kgzJt4gQt .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-2FjoAw2kgzJt4gQt .icon-shape .label rect,#mermaid-svg-2FjoAw2kgzJt4gQt .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-2FjoAw2kgzJt4gQt .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-2FjoAw2kgzJt4gQt .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-2FjoAw2kgzJt4gQt :root{--mermaid-font-family:"trebuchet ms",verdana,arial,sans-serif;} 残差连接

解决梯度消失
深层网络

能稳定训练
Pre-Norm + LayerNorm

控制数值范围
梯度裁剪 + Warmup

防止梯度爆炸
适当的初始化

确保训练起步平稳

💡 总结：残差连接解决了梯度消失，但它不是万能的。需要 LayerNorm 控制范围 + 梯度裁剪兜底 + Warmup 稳定起步，三者配合才能训练出 96 层的模型。

10.7 退化问题的解决

残差连接还解决了退化问题：

如果某一层是多余的，它可以学习 F(x)=0F(x) = 0F(x)=0，这样输出就是 y=0+x=xy = 0 + x = xy=0+x=x（恒等映射）。

这比学习一个恒等映射 F(x)=xF(x) = xF(x)=x 更容易。

10.8 在 Transformer 中的应用

Transformer 中每个子层 都有残差连接------一个 Block 里有两次 Add：

text 复制代码 
H_i（输入 Hidden State）
 │
 ├──────────────────────────┐
 ▼                          │
Multi-Head Attention        │
 │                          │
 ▼                          │
第一次 Add：H_i + Attention(H_i)
 │
 ▼
LayerNorm
 │
 ├──────────────────────────┐
 ▼                          │
FFN                         │
 │                          │
 ▼                          │
第二次 Add：上面的结果 + FFN(...)
 │
 ▼
LayerNorm
 │
 ▼
H_{i+1}（输出 Hidden State）

代码实现：

python 复制代码 
# Attention 子层
x = x + self.attention(self.norm1(x))

# FFN 子层
x = x + self.ffn(self.norm2(x))

10.8.1 用 Shape 追踪整个过程

以 seq_len=4, d_model=512 为例：

text 复制代码 
输入 X:             [4, 512]
                       │
Multi-Head Attention:  [4, 512]  （Attention 输出，和 X 的 Shape 相同）
                       │
第一次 Add:  X + A =  [4, 512] + [4, 512] = [4, 512]  ✅ Shape 不变
                       │
LayerNorm:            [4, 512]
                       │
FFN:                  [4, 512]  （内部先扩到 [4, 2048] 再压回 [4, 512]）
                       │
第二次 Add:           [4, 512] + [4, 512] = [4, 512]  ✅ Shape 不变
                       │
LayerNorm:            [4, 512]
                       │
输出:                 [4, 512]

📌 残差连接要求输入和输出的 Shape 必须相同 ，否则无法逐元素相加。这就是为什么 Transformer Block 的输入输出 Shape 始终是 [batch, seq, d_model]，也是为什么 WOW_OWO 必须把 Concat 后的结果投影回 d_model 维。

10.8.2 为什么需要两次 Residual？

很多人疑惑：为什么不把两次 Add 合并成一次，在 Block 最后一次性加？

因为 Attention 和 FFN 做的是两件完全不同的事：

阶段 做什么 可能犯的错 Residual 的作用
Attention 收集别人的信息 关注了错误的 Token 保留原始表示，Attention 的错不会覆盖旧认知
FFN 对收集到的信息独立加工 推理出错误的结论 保留 Attention 后的中间结果，FFN 的错不会覆盖已收集的信息

如果只在最后做一次 Residual：

text 复制代码 
❌  H_new = H_old + Attention(H_old) + FFN(Attention(H_old))

FFN 接收的输入是 Attention 的"裸输出"------没经过 Add 稳定化，也没经过 Norm 归一化，数值可能不稳定。而且 Attention 如果出错，错误会直接传给 FFN，没有任何缓冲。

两次独立的 Residual 确保了：

  1. FFN 收到的是已经稳定化的中间结果（经过了第一次 Add + Norm）
  2. 每个阶段的错误都被残差"兜底"------无论 Attention 还是 FFN 犯错，原始信息都不会丢失

💡 类比 ：五个人开会。第一阶段是讨论（Attention），讨论完每个人先把听到的和自己原有的想法合并记录 （第一次 Add）。第二阶段是个人思考（FFN），思考完再把结论和之前的记录合并（第二次 Add）。如果跳过中间的合并，讨论中的误解会直接污染后面的独立思考。

10.8.3 残差连接的本质：增量修改，而非重写

综合来看，残差连接意味着每一层不是"重写"Token 的表示，而是在原有表示上"增量补充"新信息：

text 复制代码 
没有残差连接：
  bank → Attention → 完全替换为新向量（原始信息可能丢失）

有残差连接：
  bank → Attention → 原始bank + Attention补充的上下文信息

类比：

  • 没有残差：每一层像是把黑板擦干净重新写------层数多了，最早写的东西早就没了
  • 有残差：每一层像是在原有笔记旁边添加批注------无论经过多少层，原始内容始终保留

所以 Transformer 经过 12 层后，每个 Token 的表示是：

最终表示=原始Embedding+第1层补充+第2层补充+⋯+第12层补充\text{最终表示} = \text{原始Embedding} + \text{第1层补充} + \text{第2层补充} + \cdots + \text{第12层补充}最终表示=原始Embedding+第1层补充+第2层补充+⋯+第12层补充

这也解释了为什么 Transformer 可以堆很多层：每一层只需要学习"在现有理解基础上做什么修正"，而不是"从头理解整个句子"。

10.9 与 ResNet 的对比

特性 ResNet Transformer
残差连接 ✅ ✅
解决梯度消失 ✅ ✅
解决退化问题 ✅ ✅
应用场景 图像 序列
子层类型 卷积层 Attention + FFN

10.10 TensorFlow 实现

python 复制代码 
import tensorflow as tf

class ResidualBlock(tf.keras.layers.Layer):
    def __init__(self, sublayer):
        super().__init__()
        self.sublayer = sublayer
        self.norm = tf.keras.layers.LayerNormalization(epsilon=1e-6)

    def call(self, x, **kwargs):
        # Pre-Norm + 残差
        return x + self.sublayer(self.norm(x), **kwargs)

# 验证梯度不消失
import numpy as np

# 模拟 100 层，有残差连接
x = tf.Variable(tf.ones([1, 10]))
with tf.GradientTape() as tape:
    h = x
    for _ in range(100):
        h = h + tf.zeros_like(h)  # F(x) = 0，模拟残差
    loss = tf.reduce_sum(h)

grad = tape.gradient(loss, x)
print("有残差连接，梯度:", grad.numpy())  # 梯度为 1，不消失

# 模拟 100 层，没有残差连接
x = tf.Variable(tf.ones([1, 10]))
with tf.GradientTape() as tape:
    h = x
    for _ in range(100):
        h = h * 0.9  # 每层梯度 0.9
    loss = tf.reduce_sum(h)

grad = tape.gradient(loss, x)
print("无残差连接，梯度:", grad.numpy())  # 梯度约为 0.9^100 ≈ 0

10.11 深度与残差连接

模型 层数 残差连接
原始 Transformer 6 ✅
BERT-Base 12 ✅
BERT-Large 24 ✅
GPT-3 96 ✅
Llama 3 70B 80 ✅

没有残差连接，这些深层模型根本无法训练。

本章总结

残差连接的三个作用：
#mermaid-svg-Rf3Y2AROcWFIWwRM{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-Rf3Y2AROcWFIWwRM .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-Rf3Y2AROcWFIWwRM .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-Rf3Y2AROcWFIWwRM .error-icon{fill:#552222;}#mermaid-svg-Rf3Y2AROcWFIWwRM .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-Rf3Y2AROcWFIWwRM .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-Rf3Y2AROcWFIWwRM .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-Rf3Y2AROcWFIWwRM .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-Rf3Y2AROcWFIWwRM .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-Rf3Y2AROcWFIWwRM .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-Rf3Y2AROcWFIWwRM .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-Rf3Y2AROcWFIWwRM .marker{fill:#333333;stroke:#333333;}#mermaid-svg-Rf3Y2AROcWFIWwRM .marker.cross{stroke:#333333;}#mermaid-svg-Rf3Y2AROcWFIWwRM svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-Rf3Y2AROcWFIWwRM p{margin:0;}#mermaid-svg-Rf3Y2AROcWFIWwRM .edge{stroke-width:3;}#mermaid-svg-Rf3Y2AROcWFIWwRM .section--1 rect,#mermaid-svg-Rf3Y2AROcWFIWwRM .section--1 path,#mermaid-svg-Rf3Y2AROcWFIWwRM .section--1 circle,#mermaid-svg-Rf3Y2AROcWFIWwRM .section--1 polygon,#mermaid-svg-Rf3Y2AROcWFIWwRM .section--1 path{fill:hsl(240, 100%, 76.2745098039%);}#mermaid-svg-Rf3Y2AROcWFIWwRM .section--1 text{fill:#ffffff;}#mermaid-svg-Rf3Y2AROcWFIWwRM .node-icon--1{font-size:40px;color:#ffffff;}#mermaid-svg-Rf3Y2AROcWFIWwRM .section-edge--1{stroke:hsl(240, 100%, 76.2745098039%);}#mermaid-svg-Rf3Y2AROcWFIWwRM .edge-depth--1{stroke-width:17;}#mermaid-svg-Rf3Y2AROcWFIWwRM .section--1 line{stroke:hsl(60, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-Rf3Y2AROcWFIWwRM .disabled,#mermaid-svg-Rf3Y2AROcWFIWwRM .disabled circle,#mermaid-svg-Rf3Y2AROcWFIWwRM .disabled text{fill:lightgray;}#mermaid-svg-Rf3Y2AROcWFIWwRM .disabled text{fill:#efefef;}#mermaid-svg-Rf3Y2AROcWFIWwRM .section-0 rect,#mermaid-svg-Rf3Y2AROcWFIWwRM .section-0 path,#mermaid-svg-Rf3Y2AROcWFIWwRM .section-0 circle,#mermaid-svg-Rf3Y2AROcWFIWwRM .section-0 polygon,#mermaid-svg-Rf3Y2AROcWFIWwRM .section-0 path{fill:hsl(60, 100%, 73.5294117647%);}#mermaid-svg-Rf3Y2AROcWFIWwRM .section-0 text{fill:black;}#mermaid-svg-Rf3Y2AROcWFIWwRM .node-icon-0{font-size:40px;color:black;}#mermaid-svg-Rf3Y2AROcWFIWwRM .section-edge-0{stroke:hsl(60, 100%, 73.5294117647%);}#mermaid-svg-Rf3Y2AROcWFIWwRM .edge-depth-0{stroke-width:14;}#mermaid-svg-Rf3Y2AROcWFIWwRM .section-0 line{stroke:hsl(240, 100%, 83.5294117647%);stroke-width:3;}#mermaid-svg-Rf3Y2AROcWFIWwRM .disabled,#mermaid-svg-Rf3Y2AROcWFIWwRM .disabled circle,#mermaid-svg-Rf3Y2AROcWFIWwRM .disabled text{fill:lightgray;}#mermaid-svg-Rf3Y2AROcWFIWwRM .disabled text{fill:#efefef;}#mermaid-svg-Rf3Y2AROcWFIWwRM .section-1 rect,#mermaid-svg-Rf3Y2AROcWFIWwRM .section-1 path,#mermaid-svg-Rf3Y2AROcWFIWwRM .section-1 circle,#mermaid-svg-Rf3Y2AROcWFIWwRM .section-1 polygon,#mermaid-svg-Rf3Y2AROcWFIWwRM .section-1 path{fill:hsl(80, 100%, 76.2745098039%);}#mermaid-svg-Rf3Y2AROcWFIWwRM .section-1 text{fill:black;}#mermaid-svg-Rf3Y2AROcWFIWwRM .node-icon-1{font-size:40px;color:black;}#mermaid-svg-Rf3Y2AROcWFIWwRM .section-edge-1{stroke:hsl(80, 100%, 76.2745098039%);}#mermaid-svg-Rf3Y2AROcWFIWwRM .edge-depth-1{stroke-width:11;}#mermaid-svg-Rf3Y2AROcWFIWwRM .section-1 line{stroke:hsl(260, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-Rf3Y2AROcWFIWwRM .disabled,#mermaid-svg-Rf3Y2AROcWFIWwRM .disabled circle,#mermaid-svg-Rf3Y2AROcWFIWwRM .disabled text{fill:lightgray;}#mermaid-svg-Rf3Y2AROcWFIWwRM .disabled text{fill:#efefef;}#mermaid-svg-Rf3Y2AROcWFIWwRM .section-2 rect,#mermaid-svg-Rf3Y2AROcWFIWwRM .section-2 path,#mermaid-svg-Rf3Y2AROcWFIWwRM .section-2 circle,#mermaid-svg-Rf3Y2AROcWFIWwRM .section-2 polygon,#mermaid-svg-Rf3Y2AROcWFIWwRM .section-2 path{fill:hsl(270, 100%, 76.2745098039%);}#mermaid-svg-Rf3Y2AROcWFIWwRM .section-2 text{fill:#ffffff;}#mermaid-svg-Rf3Y2AROcWFIWwRM .node-icon-2{font-size:40px;color:#ffffff;}#mermaid-svg-Rf3Y2AROcWFIWwRM .section-edge-2{stroke:hsl(270, 100%, 76.2745098039%);}#mermaid-svg-Rf3Y2AROcWFIWwRM .edge-depth-2{stroke-width:8;}#mermaid-svg-Rf3Y2AROcWFIWwRM .section-2 line{stroke:hsl(90, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-Rf3Y2AROcWFIWwRM .disabled,#mermaid-svg-Rf3Y2AROcWFIWwRM .disabled circle,#mermaid-svg-Rf3Y2AROcWFIWwRM .disabled text{fill:lightgray;}#mermaid-svg-Rf3Y2AROcWFIWwRM .disabled text{fill:#efefef;}#mermaid-svg-Rf3Y2AROcWFIWwRM .section-3 rect,#mermaid-svg-Rf3Y2AROcWFIWwRM .section-3 path,#mermaid-svg-Rf3Y2AROcWFIWwRM .section-3 circle,#mermaid-svg-Rf3Y2AROcWFIWwRM .section-3 polygon,#mermaid-svg-Rf3Y2AROcWFIWwRM .section-3 path{fill:hsl(300, 100%, 76.2745098039%);}#mermaid-svg-Rf3Y2AROcWFIWwRM .section-3 text{fill:black;}#mermaid-svg-Rf3Y2AROcWFIWwRM .node-icon-3{font-size:40px;color:black;}#mermaid-svg-Rf3Y2AROcWFIWwRM .section-edge-3{stroke:hsl(300, 100%, 76.2745098039%);}#mermaid-svg-Rf3Y2AROcWFIWwRM .edge-depth-3{stroke-width:5;}#mermaid-svg-Rf3Y2AROcWFIWwRM .section-3 line{stroke:hsl(120, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-Rf3Y2AROcWFIWwRM .disabled,#mermaid-svg-Rf3Y2AROcWFIWwRM .disabled circle,#mermaid-svg-Rf3Y2AROcWFIWwRM .disabled text{fill:lightgray;}#mermaid-svg-Rf3Y2AROcWFIWwRM .disabled text{fill:#efefef;}#mermaid-svg-Rf3Y2AROcWFIWwRM .section-4 rect,#mermaid-svg-Rf3Y2AROcWFIWwRM .section-4 path,#mermaid-svg-Rf3Y2AROcWFIWwRM .section-4 circle,#mermaid-svg-Rf3Y2AROcWFIWwRM .section-4 polygon,#mermaid-svg-Rf3Y2AROcWFIWwRM .section-4 path{fill:hsl(330, 100%, 76.2745098039%);}#mermaid-svg-Rf3Y2AROcWFIWwRM .section-4 text{fill:black;}#mermaid-svg-Rf3Y2AROcWFIWwRM .node-icon-4{font-size:40px;color:black;}#mermaid-svg-Rf3Y2AROcWFIWwRM .section-edge-4{stroke:hsl(330, 100%, 76.2745098039%);}#mermaid-svg-Rf3Y2AROcWFIWwRM .edge-depth-4{stroke-width:2;}#mermaid-svg-Rf3Y2AROcWFIWwRM .section-4 line{stroke:hsl(150, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-Rf3Y2AROcWFIWwRM .disabled,#mermaid-svg-Rf3Y2AROcWFIWwRM .disabled circle,#mermaid-svg-Rf3Y2AROcWFIWwRM .disabled text{fill:lightgray;}#mermaid-svg-Rf3Y2AROcWFIWwRM .disabled text{fill:#efefef;}#mermaid-svg-Rf3Y2AROcWFIWwRM .section-5 rect,#mermaid-svg-Rf3Y2AROcWFIWwRM .section-5 path,#mermaid-svg-Rf3Y2AROcWFIWwRM .section-5 circle,#mermaid-svg-Rf3Y2AROcWFIWwRM .section-5 polygon,#mermaid-svg-Rf3Y2AROcWFIWwRM .section-5 path{fill:hsl(0, 100%, 76.2745098039%);}#mermaid-svg-Rf3Y2AROcWFIWwRM .section-5 text{fill:black;}#mermaid-svg-Rf3Y2AROcWFIWwRM .node-icon-5{font-size:40px;color:black;}#mermaid-svg-Rf3Y2AROcWFIWwRM .section-edge-5{stroke:hsl(0, 100%, 76.2745098039%);}#mermaid-svg-Rf3Y2AROcWFIWwRM .edge-depth-5{stroke-width:-1;}#mermaid-svg-Rf3Y2AROcWFIWwRM .section-5 line{stroke:hsl(180, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-Rf3Y2AROcWFIWwRM .disabled,#mermaid-svg-Rf3Y2AROcWFIWwRM .disabled circle,#mermaid-svg-Rf3Y2AROcWFIWwRM .disabled text{fill:lightgray;}#mermaid-svg-Rf3Y2AROcWFIWwRM .disabled text{fill:#efefef;}#mermaid-svg-Rf3Y2AROcWFIWwRM .section-6 rect,#mermaid-svg-Rf3Y2AROcWFIWwRM .section-6 path,#mermaid-svg-Rf3Y2AROcWFIWwRM .section-6 circle,#mermaid-svg-Rf3Y2AROcWFIWwRM .section-6 polygon,#mermaid-svg-Rf3Y2AROcWFIWwRM .section-6 path{fill:hsl(30, 100%, 76.2745098039%);}#mermaid-svg-Rf3Y2AROcWFIWwRM .section-6 text{fill:black;}#mermaid-svg-Rf3Y2AROcWFIWwRM .node-icon-6{font-size:40px;color:black;}#mermaid-svg-Rf3Y2AROcWFIWwRM .section-edge-6{stroke:hsl(30, 100%, 76.2745098039%);}#mermaid-svg-Rf3Y2AROcWFIWwRM .edge-depth-6{stroke-width:-4;}#mermaid-svg-Rf3Y2AROcWFIWwRM .section-6 line{stroke:hsl(210, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-Rf3Y2AROcWFIWwRM .disabled,#mermaid-svg-Rf3Y2AROcWFIWwRM .disabled circle,#mermaid-svg-Rf3Y2AROcWFIWwRM .disabled text{fill:lightgray;}#mermaid-svg-Rf3Y2AROcWFIWwRM .disabled text{fill:#efefef;}#mermaid-svg-Rf3Y2AROcWFIWwRM .section-7 rect,#mermaid-svg-Rf3Y2AROcWFIWwRM .section-7 path,#mermaid-svg-Rf3Y2AROcWFIWwRM .section-7 circle,#mermaid-svg-Rf3Y2AROcWFIWwRM .section-7 polygon,#mermaid-svg-Rf3Y2AROcWFIWwRM .section-7 path{fill:hsl(90, 100%, 76.2745098039%);}#mermaid-svg-Rf3Y2AROcWFIWwRM .section-7 text{fill:black;}#mermaid-svg-Rf3Y2AROcWFIWwRM .node-icon-7{font-size:40px;color:black;}#mermaid-svg-Rf3Y2AROcWFIWwRM .section-edge-7{stroke:hsl(90, 100%, 76.2745098039%);}#mermaid-svg-Rf3Y2AROcWFIWwRM .edge-depth-7{stroke-width:-7;}#mermaid-svg-Rf3Y2AROcWFIWwRM .section-7 line{stroke:hsl(270, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-Rf3Y2AROcWFIWwRM .disabled,#mermaid-svg-Rf3Y2AROcWFIWwRM .disabled circle,#mermaid-svg-Rf3Y2AROcWFIWwRM .disabled text{fill:lightgray;}#mermaid-svg-Rf3Y2AROcWFIWwRM .disabled text{fill:#efefef;}#mermaid-svg-Rf3Y2AROcWFIWwRM .section-8 rect,#mermaid-svg-Rf3Y2AROcWFIWwRM .section-8 path,#mermaid-svg-Rf3Y2AROcWFIWwRM .section-8 circle,#mermaid-svg-Rf3Y2AROcWFIWwRM .section-8 polygon,#mermaid-svg-Rf3Y2AROcWFIWwRM .section-8 path{fill:hsl(150, 100%, 76.2745098039%);}#mermaid-svg-Rf3Y2AROcWFIWwRM .section-8 text{fill:black;}#mermaid-svg-Rf3Y2AROcWFIWwRM .node-icon-8{font-size:40px;color:black;}#mermaid-svg-Rf3Y2AROcWFIWwRM .section-edge-8{stroke:hsl(150, 100%, 76.2745098039%);}#mermaid-svg-Rf3Y2AROcWFIWwRM .edge-depth-8{stroke-width:-10;}#mermaid-svg-Rf3Y2AROcWFIWwRM .section-8 line{stroke:hsl(330, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-Rf3Y2AROcWFIWwRM .disabled,#mermaid-svg-Rf3Y2AROcWFIWwRM .disabled circle,#mermaid-svg-Rf3Y2AROcWFIWwRM .disabled text{fill:lightgray;}#mermaid-svg-Rf3Y2AROcWFIWwRM .disabled text{fill:#efefef;}#mermaid-svg-Rf3Y2AROcWFIWwRM .section-9 rect,#mermaid-svg-Rf3Y2AROcWFIWwRM .section-9 path,#mermaid-svg-Rf3Y2AROcWFIWwRM .section-9 circle,#mermaid-svg-Rf3Y2AROcWFIWwRM .section-9 polygon,#mermaid-svg-Rf3Y2AROcWFIWwRM .section-9 path{fill:hsl(180, 100%, 76.2745098039%);}#mermaid-svg-Rf3Y2AROcWFIWwRM .section-9 text{fill:black;}#mermaid-svg-Rf3Y2AROcWFIWwRM .node-icon-9{font-size:40px;color:black;}#mermaid-svg-Rf3Y2AROcWFIWwRM .section-edge-9{stroke:hsl(180, 100%, 76.2745098039%);}#mermaid-svg-Rf3Y2AROcWFIWwRM .edge-depth-9{stroke-width:-13;}#mermaid-svg-Rf3Y2AROcWFIWwRM .section-9 line{stroke:hsl(0, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-Rf3Y2AROcWFIWwRM .disabled,#mermaid-svg-Rf3Y2AROcWFIWwRM .disabled circle,#mermaid-svg-Rf3Y2AROcWFIWwRM .disabled text{fill:lightgray;}#mermaid-svg-Rf3Y2AROcWFIWwRM .disabled text{fill:#efefef;}#mermaid-svg-Rf3Y2AROcWFIWwRM .section-10 rect,#mermaid-svg-Rf3Y2AROcWFIWwRM .section-10 path,#mermaid-svg-Rf3Y2AROcWFIWwRM .section-10 circle,#mermaid-svg-Rf3Y2AROcWFIWwRM .section-10 polygon,#mermaid-svg-Rf3Y2AROcWFIWwRM .section-10 path{fill:hsl(210, 100%, 76.2745098039%);}#mermaid-svg-Rf3Y2AROcWFIWwRM .section-10 text{fill:black;}#mermaid-svg-Rf3Y2AROcWFIWwRM .node-icon-10{font-size:40px;color:black;}#mermaid-svg-Rf3Y2AROcWFIWwRM .section-edge-10{stroke:hsl(210, 100%, 76.2745098039%);}#mermaid-svg-Rf3Y2AROcWFIWwRM .edge-depth-10{stroke-width:-16;}#mermaid-svg-Rf3Y2AROcWFIWwRM .section-10 line{stroke:hsl(30, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-Rf3Y2AROcWFIWwRM .disabled,#mermaid-svg-Rf3Y2AROcWFIWwRM .disabled circle,#mermaid-svg-Rf3Y2AROcWFIWwRM .disabled text{fill:lightgray;}#mermaid-svg-Rf3Y2AROcWFIWwRM .disabled text{fill:#efefef;}#mermaid-svg-Rf3Y2AROcWFIWwRM .section-root rect,#mermaid-svg-Rf3Y2AROcWFIWwRM .section-root path,#mermaid-svg-Rf3Y2AROcWFIWwRM .section-root circle,#mermaid-svg-Rf3Y2AROcWFIWwRM .section-root polygon{fill:hsl(240, 100%, 46.2745098039%);}#mermaid-svg-Rf3Y2AROcWFIWwRM .section-root text{fill:#ffffff;}#mermaid-svg-Rf3Y2AROcWFIWwRM .section-root span{color:#ffffff;}#mermaid-svg-Rf3Y2AROcWFIWwRM .section-2 span{color:#ffffff;}#mermaid-svg-Rf3Y2AROcWFIWwRM .icon-container{height:100%;display:flex;justify-content:center;align-items:center;}#mermaid-svg-Rf3Y2AROcWFIWwRM .edge{fill:none;}#mermaid-svg-Rf3Y2AROcWFIWwRM .mindmap-node-label{dy:1em;alignment-baseline:middle;text-anchor:middle;dominant-baseline:middle;text-align:center;}#mermaid-svg-Rf3Y2AROcWFIWwRM :root{--mermaid-font-family:"trebuchet ms",verdana,arial,sans-serif;} 残差连接
防止梯度消失
梯度至少为 1
不会连乘趋近于 0
信息高速公路
原始信息直接传递
不会丢失
解决退化问题
可以学习恒等映射
子层输出为 0 即可

核心公式：y=F(x)+xy = F(x) + xy=F(x)+x

本章思考题

  1. 为什么残差连接要求输入和输出的 Shape 相同？如果不同怎么办？
  2. 如果 F(x) 学到了一个很大的值，残差连接会有什么问题？
  3. 残差连接和 LSTM 的 Cell State 有什么相似之处？
  4. 为什么说残差连接让"学习恒等映射"变得更容易？

下一章预告

下一章我们进入 第三篇：Encoder。

我们将看到：

  • Encoder 是如何一层一层理解句子的？
  • CLS Token 是什么？
  • 为什么 Encoder 的输出可以作为 Sentence Embedding？
相关推荐
zhongerzixunshi1 小时前
深耕绿色建材认证 赋能建筑行业低碳高质量发展
人工智能
Wang's Blog1 小时前
Vibe Coding一人即团队系列10: 在 Claude 与 Codex 中接入 DeepSeek 模型
人工智能
lisw051 小时前
计算与科学哲学（Philosophy of Computing and Science）
java·开发语言·人工智能
天天进步20151 小时前
Pixelle-Video 源码解析 #10：多模型适配：GPT、通义千问、DeepSeek、Ollama 如何统一调用？
人工智能·gpt
@atweiwei1 小时前
用 Rust 构建 Agent 应用的高性能框架:langchainrust 架构全景
人工智能·架构·rust·langchain·llm·agent·ai编程
欧特克_Glodon1 小时前
OpenCV计算机视觉开发入门与实践＜二十二＞：图像平滑之线性滤波
c++·人工智能·opencv·计算机视觉
ShiXZ2131 小时前
ChatGPT Plus用户注意：5小时限制今日恢复，额度每5小时自动刷新（且额度又又又重置了）
人工智能·chatgpt
阿童木写作1 小时前
跨马翻译：AI批量图片翻译工具，视频字幕翻译与智能抠图一站式解决
大数据·人工智能·python·音视频
chongzhe1 小时前
AgentConnect：多 Agent 协作的开源版 Claude Tag
人工智能·github·源码
热门推荐
01国内可直接用、免费额度/永久免费的大模型API清单（含 SiliconFlow、火山、阿里、智谱、百度、Kimi、DeepSeek、DMXAPI 等）02GitHub 镜像站点032026 国产 AI 大模型横评：DeepSeek、通义千问、Kimi、文心一言、星火、豆包谁更能打？04历年考研数学一、数学二、数学三真题试卷及答案PDF05微信历史版本含下载地址（ Windows PC | 安卓 | MAC ）及设置微信不更新06Agnes AI 免费 API 接入指南：文本、生图、生视频，一套接口全免费07AI 编程 IDE 全景解析 2026：Agent 全面接管开发链路08VSCode+CodeX扩展，无法启动092026年7月AI圈大地震：GPT-5.6被政府限制、Claude入驻Slack、Anthropic自研芯片10如何新建文件夹？ 电脑新建文件夹的4种方法