第13章 Decoder
本章目标：
理解 Decoder 的 Mask 机制，为什么不能偷看未来，以及 Causal Attention 的实现。
13.1 为什么 Decoder 不能偷看未来？
假设我们在训练一个翻译模型：
text
输入（英文）: I love AI
目标（中文）: 我 爱 AI训练时，Decoder 的输入是目标序列（右移一位）：
text
Decoder 输入: [START] 我 爱
Decoder 输出: 我 爱 AI如果 Decoder 可以看到未来的 Token：
text
预测"我"时，看到了"爱"和"AI" → 作弊！这样训练出来的模型，推理时（没有未来 Token）会完全失效。
所以必须用 Mask 遮住未来的 Token。
13.2 Causal Mask（因果掩码）
Causal Mask 是一个上三角矩阵，用于遮住未来的 Token：
text
序列长度 = 4
Mask:
t=1 t=2 t=3 t=4
t=1 [ 0 -∞ -∞ -∞ ]
t=2 [ 0 0 -∞ -∞ ]
t=3 [ 0 0 0 -∞ ]
t=4 [ 0 0 0 0 ]0：可以看到（不遮住）-∞：不能看到（遮住，Softmax 后变为 0）
#mermaid-svg-UmtLj2J3FZYZ5JVR{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-UmtLj2J3FZYZ5JVR .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-UmtLj2J3FZYZ5JVR .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-UmtLj2J3FZYZ5JVR .error-icon{fill:#552222;}#mermaid-svg-UmtLj2J3FZYZ5JVR .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-UmtLj2J3FZYZ5JVR .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-UmtLj2J3FZYZ5JVR .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-UmtLj2J3FZYZ5JVR .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-UmtLj2J3FZYZ5JVR .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-UmtLj2J3FZYZ5JVR .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-UmtLj2J3FZYZ5JVR .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-UmtLj2J3FZYZ5JVR .marker{fill:#333333;stroke:#333333;}#mermaid-svg-UmtLj2J3FZYZ5JVR .marker.cross{stroke:#333333;}#mermaid-svg-UmtLj2J3FZYZ5JVR svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-UmtLj2J3FZYZ5JVR p{margin:0;}#mermaid-svg-UmtLj2J3FZYZ5JVR .label{font-family:"trebuchet ms",verdana,arial,sans-serif;color:#333;}#mermaid-svg-UmtLj2J3FZYZ5JVR .cluster-label text{fill:#333;}#mermaid-svg-UmtLj2J3FZYZ5JVR .cluster-label span{color:#333;}#mermaid-svg-UmtLj2J3FZYZ5JVR .cluster-label span p{background-color:transparent;}#mermaid-svg-UmtLj2J3FZYZ5JVR .label text,#mermaid-svg-UmtLj2J3FZYZ5JVR span{fill:#333;color:#333;}#mermaid-svg-UmtLj2J3FZYZ5JVR .node rect,#mermaid-svg-UmtLj2J3FZYZ5JVR .node circle,#mermaid-svg-UmtLj2J3FZYZ5JVR .node ellipse,#mermaid-svg-UmtLj2J3FZYZ5JVR .node polygon,#mermaid-svg-UmtLj2J3FZYZ5JVR .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-UmtLj2J3FZYZ5JVR .rough-node .label text,#mermaid-svg-UmtLj2J3FZYZ5JVR .node .label text,#mermaid-svg-UmtLj2J3FZYZ5JVR .image-shape .label,#mermaid-svg-UmtLj2J3FZYZ5JVR .icon-shape .label{text-anchor:middle;}#mermaid-svg-UmtLj2J3FZYZ5JVR .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-UmtLj2J3FZYZ5JVR .rough-node .label,#mermaid-svg-UmtLj2J3FZYZ5JVR .node .label,#mermaid-svg-UmtLj2J3FZYZ5JVR .image-shape .label,#mermaid-svg-UmtLj2J3FZYZ5JVR .icon-shape .label{text-align:center;}#mermaid-svg-UmtLj2J3FZYZ5JVR .node.clickable{cursor:pointer;}#mermaid-svg-UmtLj2J3FZYZ5JVR .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-UmtLj2J3FZYZ5JVR .arrowheadPath{fill:#333333;}#mermaid-svg-UmtLj2J3FZYZ5JVR .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-UmtLj2J3FZYZ5JVR .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-UmtLj2J3FZYZ5JVR .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-UmtLj2J3FZYZ5JVR .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-UmtLj2J3FZYZ5JVR .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-UmtLj2J3FZYZ5JVR .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-UmtLj2J3FZYZ5JVR .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-UmtLj2J3FZYZ5JVR .cluster text{fill:#333;}#mermaid-svg-UmtLj2J3FZYZ5JVR .cluster span{color:#333;}#mermaid-svg-UmtLj2J3FZYZ5JVR div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-UmtLj2J3FZYZ5JVR .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-UmtLj2J3FZYZ5JVR rect.text{fill:none;stroke-width:0;}#mermaid-svg-UmtLj2J3FZYZ5JVR .icon-shape,#mermaid-svg-UmtLj2J3FZYZ5JVR .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-UmtLj2J3FZYZ5JVR .icon-shape p,#mermaid-svg-UmtLj2J3FZYZ5JVR .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-UmtLj2J3FZYZ5JVR .icon-shape .label rect,#mermaid-svg-UmtLj2J3FZYZ5JVR .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-UmtLj2J3FZYZ5JVR .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-UmtLj2J3FZYZ5JVR .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-UmtLj2J3FZYZ5JVR :root{--mermaid-font-family:"trebuchet ms",verdana,arial,sans-serif;} Attention Score
seq, seq
加上 Causal Mask
上三角为 -∞
Softmax
-∞ 变为 0
Masked Attention Weight
只看当前和之前的Token
13.3 Causal Mask 的效果
以序列 [我, 爱, AI] 为例：
没有 Mask（BERT 风格）：
我 爱 AI
我 [0.5 0.3 0.2] ← 我可以看到"爱"和"AI"
爱 [0.2 0.6 0.2] ← 爱可以看到"我"和"AI"
AI [0.3 0.3 0.4]有 Causal Mask（GPT 风格）：
我 爱 AI
我 [1.0 0.0 0.0] ← 我只能看到自己
爱 [0.4 0.6 0.0] ← 爱只能看到"我"和自己
AI [0.3 0.3 0.4] ← AI可以看到所有13.3.1 为什么 Mask 值是 -∞ 而不是 0？
这是初学者最常问的问题。答案和 Softmax 的数学性质有关。
如果把未来位置的 Score 设为 0：
text
Score: [3, 0, 0]
Softmax: [0.91, 0.045, 0.045] ← 未来位置仍然分到了4.5%的权重！因为 e0=1e^0 = 1e0=1，所以 Score=0 在 Softmax 中不代表概率为零，它仍然会分到权重。模型还是"偷看"了未来。
如果把未来位置的 Score 设为 -∞：
text
Score: [3, -∞, -∞]
Softmax: [1.0, 0, 0] ← 未来位置权重严格为零因为 e−∞=0e^{-\infty} = 0e−∞=0，Softmax 后这些位置的概率精确为零。
13.3.2 为什么用加法而不是乘法？
另一个常见疑问：为什么不直接把 Score 乘以 0？
text
❌ 乘法方案：Score × Mask = [3, 0, 0]
→ Softmax 后：[0.91, 0.045, 0.045] ← 没有真正遮住！
✅ 加法方案：Score + Mask = [3, -∞, -∞]
→ Softmax 后：[1.0, 0, 0] ← 完全遮住加法的优势：
- 允许看的位置加 0（不改变原始 Score）
- 需要遮住的位置加 -∞（Softmax 后精确为 0）
- 不会像乘法那样遇到"负数×(-∞)=+∞"的数学问题
💡 实际代码中 ，-∞ 通常用
-1e9或torch.finfo(dtype).min来近似，保证 Softmax 后的值足够接近 0。
13.3.3 Mask 没有删除 Token
一个重要认识：Mask 不会删除任何 Token、Embedding 或 Value。它只修改 Attention Score：
text
Token、Q、K、V → 全部正常计算，一个不少
Score = QK^T → 正常算出完整的 [seq, seq] 矩阵
Score + Mask → 只在这一步把未来位置变成 -∞
Softmax → 未来位置的权重自然变成 0
Weight × V → 未来位置对输出的贡献为 0所以 Mask 的本质是：让未来 Token 在 Attention 权重竞争中失去资格，而不是物理删除它们。
13.4 Causal Mask 的实现
python
import tensorflow as tf
def create_causal_mask(seq_len):
"""
创建因果掩码（上三角矩阵）
返回: [1, 1, seq_len, seq_len]，1表示遮住，0表示不遮住
"""
mask = 1 - tf.linalg.band_part(tf.ones((seq_len, seq_len)), -1, 0)
return mask[tf.newaxis, tf.newaxis, :, :] # [1, 1, seq, seq]
# 测试
mask = create_causal_mask(4)
print(mask[0, 0])
# [[0. 1. 1. 1.]
# [0. 0. 1. 1.]
# [0. 0. 0. 1.]
# [0. 0. 0. 0.]]
# 在 Attention 中使用
def masked_attention(Q, K, V, mask=None):
d_k = tf.cast(tf.shape(K)[-1], tf.float32)
scores = tf.matmul(Q, K, transpose_b=True) / tf.math.sqrt(d_k)
if mask is not None:
scores += (mask * -1e9) # 遮住的位置加 -∞
weights = tf.nn.softmax(scores, axis=-1)
return tf.matmul(weights, V)13.5 Decoder 的完整结构
原始 Transformer 的 Decoder Block 有三个子层：
#mermaid-svg-g4muQmGMygvEppEK{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-g4muQmGMygvEppEK .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-g4muQmGMygvEppEK .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-g4muQmGMygvEppEK .error-icon{fill:#552222;}#mermaid-svg-g4muQmGMygvEppEK .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-g4muQmGMygvEppEK .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-g4muQmGMygvEppEK .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-g4muQmGMygvEppEK .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-g4muQmGMygvEppEK .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-g4muQmGMygvEppEK .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-g4muQmGMygvEppEK .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-g4muQmGMygvEppEK .marker{fill:#333333;stroke:#333333;}#mermaid-svg-g4muQmGMygvEppEK .marker.cross{stroke:#333333;}#mermaid-svg-g4muQmGMygvEppEK svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-g4muQmGMygvEppEK p{margin:0;}#mermaid-svg-g4muQmGMygvEppEK .label{font-family:"trebuchet ms",verdana,arial,sans-serif;color:#333;}#mermaid-svg-g4muQmGMygvEppEK .cluster-label text{fill:#333;}#mermaid-svg-g4muQmGMygvEppEK .cluster-label span{color:#333;}#mermaid-svg-g4muQmGMygvEppEK .cluster-label span p{background-color:transparent;}#mermaid-svg-g4muQmGMygvEppEK .label text,#mermaid-svg-g4muQmGMygvEppEK span{fill:#333;color:#333;}#mermaid-svg-g4muQmGMygvEppEK .node rect,#mermaid-svg-g4muQmGMygvEppEK .node circle,#mermaid-svg-g4muQmGMygvEppEK .node ellipse,#mermaid-svg-g4muQmGMygvEppEK .node polygon,#mermaid-svg-g4muQmGMygvEppEK .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-g4muQmGMygvEppEK .rough-node .label text,#mermaid-svg-g4muQmGMygvEppEK .node .label text,#mermaid-svg-g4muQmGMygvEppEK .image-shape .label,#mermaid-svg-g4muQmGMygvEppEK .icon-shape .label{text-anchor:middle;}#mermaid-svg-g4muQmGMygvEppEK .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-g4muQmGMygvEppEK .rough-node .label,#mermaid-svg-g4muQmGMygvEppEK .node .label,#mermaid-svg-g4muQmGMygvEppEK .image-shape .label,#mermaid-svg-g4muQmGMygvEppEK .icon-shape .label{text-align:center;}#mermaid-svg-g4muQmGMygvEppEK .node.clickable{cursor:pointer;}#mermaid-svg-g4muQmGMygvEppEK .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-g4muQmGMygvEppEK .arrowheadPath{fill:#333333;}#mermaid-svg-g4muQmGMygvEppEK .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-g4muQmGMygvEppEK .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-g4muQmGMygvEppEK .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-g4muQmGMygvEppEK .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-g4muQmGMygvEppEK .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-g4muQmGMygvEppEK .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-g4muQmGMygvEppEK .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-g4muQmGMygvEppEK .cluster text{fill:#333;}#mermaid-svg-g4muQmGMygvEppEK .cluster span{color:#333;}#mermaid-svg-g4muQmGMygvEppEK div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-g4muQmGMygvEppEK .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-g4muQmGMygvEppEK rect.text{fill:none;stroke-width:0;}#mermaid-svg-g4muQmGMygvEppEK .icon-shape,#mermaid-svg-g4muQmGMygvEppEK .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-g4muQmGMygvEppEK .icon-shape p,#mermaid-svg-g4muQmGMygvEppEK .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-g4muQmGMygvEppEK .icon-shape .label rect,#mermaid-svg-g4muQmGMygvEppEK .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-g4muQmGMygvEppEK .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-g4muQmGMygvEppEK .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-g4muQmGMygvEppEK :root{--mermaid-font-family:"trebuchet ms",verdana,arial,sans-serif;} 目标序列输入
batch, tgt_seq, d_model
Masked Self-Attention
只看当前和之前的Token
残差 +
LayerNorm
Cross-Attention
与Encoder输出交互
Encoder 输出
batch, src_seq, d_model
残差 +
LayerNorm
Feed Forward
残差 +
LayerNorm
输出
batch, tgt_seq, d_model
13.6 Cross-Attention
Cross-Attention 是 Decoder 特有的模块，用于与 Encoder 的输出交互：
CrossAttention(Q,K,V)=Softmax(QdecKencTdk)Venc\text{CrossAttention}(Q, K, V) = \text{Softmax}\left(\frac{Q_{dec} K_{enc}^T}{\sqrt{d_k}}\right) V_{enc}CrossAttention(Q,K,V)=Softmax(dk QdecKencT)Venc
- Q：来自 Decoder（当前生成的 Token 在问：我需要什么信息？）
- K, V：来自 Encoder（源句子的所有 Token 在回答）
#mermaid-svg-7EI3AWN72Yf1VNzZ{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-7EI3AWN72Yf1VNzZ .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-7EI3AWN72Yf1VNzZ .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-7EI3AWN72Yf1VNzZ .error-icon{fill:#552222;}#mermaid-svg-7EI3AWN72Yf1VNzZ .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-7EI3AWN72Yf1VNzZ .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-7EI3AWN72Yf1VNzZ .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-7EI3AWN72Yf1VNzZ .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-7EI3AWN72Yf1VNzZ .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-7EI3AWN72Yf1VNzZ .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-7EI3AWN72Yf1VNzZ .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-7EI3AWN72Yf1VNzZ .marker{fill:#333333;stroke:#333333;}#mermaid-svg-7EI3AWN72Yf1VNzZ .marker.cross{stroke:#333333;}#mermaid-svg-7EI3AWN72Yf1VNzZ svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-7EI3AWN72Yf1VNzZ p{margin:0;}#mermaid-svg-7EI3AWN72Yf1VNzZ .label{font-family:"trebuchet ms",verdana,arial,sans-serif;color:#333;}#mermaid-svg-7EI3AWN72Yf1VNzZ .cluster-label text{fill:#333;}#mermaid-svg-7EI3AWN72Yf1VNzZ .cluster-label span{color:#333;}#mermaid-svg-7EI3AWN72Yf1VNzZ .cluster-label span p{background-color:transparent;}#mermaid-svg-7EI3AWN72Yf1VNzZ .label text,#mermaid-svg-7EI3AWN72Yf1VNzZ span{fill:#333;color:#333;}#mermaid-svg-7EI3AWN72Yf1VNzZ .node rect,#mermaid-svg-7EI3AWN72Yf1VNzZ .node circle,#mermaid-svg-7EI3AWN72Yf1VNzZ .node ellipse,#mermaid-svg-7EI3AWN72Yf1VNzZ .node polygon,#mermaid-svg-7EI3AWN72Yf1VNzZ .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-7EI3AWN72Yf1VNzZ .rough-node .label text,#mermaid-svg-7EI3AWN72Yf1VNzZ .node .label text,#mermaid-svg-7EI3AWN72Yf1VNzZ .image-shape .label,#mermaid-svg-7EI3AWN72Yf1VNzZ .icon-shape .label{text-anchor:middle;}#mermaid-svg-7EI3AWN72Yf1VNzZ .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-7EI3AWN72Yf1VNzZ .rough-node .label,#mermaid-svg-7EI3AWN72Yf1VNzZ .node .label,#mermaid-svg-7EI3AWN72Yf1VNzZ .image-shape .label,#mermaid-svg-7EI3AWN72Yf1VNzZ .icon-shape .label{text-align:center;}#mermaid-svg-7EI3AWN72Yf1VNzZ .node.clickable{cursor:pointer;}#mermaid-svg-7EI3AWN72Yf1VNzZ .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-7EI3AWN72Yf1VNzZ .arrowheadPath{fill:#333333;}#mermaid-svg-7EI3AWN72Yf1VNzZ .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-7EI3AWN72Yf1VNzZ .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-7EI3AWN72Yf1VNzZ .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-7EI3AWN72Yf1VNzZ .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-7EI3AWN72Yf1VNzZ .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-7EI3AWN72Yf1VNzZ .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-7EI3AWN72Yf1VNzZ .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-7EI3AWN72Yf1VNzZ .cluster text{fill:#333;}#mermaid-svg-7EI3AWN72Yf1VNzZ .cluster span{color:#333;}#mermaid-svg-7EI3AWN72Yf1VNzZ div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-7EI3AWN72Yf1VNzZ .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-7EI3AWN72Yf1VNzZ rect.text{fill:none;stroke-width:0;}#mermaid-svg-7EI3AWN72Yf1VNzZ .icon-shape,#mermaid-svg-7EI3AWN72Yf1VNzZ .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-7EI3AWN72Yf1VNzZ .icon-shape p,#mermaid-svg-7EI3AWN72Yf1VNzZ .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-7EI3AWN72Yf1VNzZ .icon-shape .label rect,#mermaid-svg-7EI3AWN72Yf1VNzZ .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-7EI3AWN72Yf1VNzZ .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-7EI3AWN72Yf1VNzZ .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-7EI3AWN72Yf1VNzZ :root{--mermaid-font-family:"trebuchet ms",verdana,arial,sans-serif;} Q
K, V
Decoder Token
'爱'
Cross-Attention
Encoder 输出
'I love AI'
融合了源句子信息的
'爱'向量
13.6.1 Cross-Attention 中 Q、K、V 的角色
理解 Cross-Attention 的关键是理解三个矩阵的来源 和含义：
| 角色 | 来源 | 含义 | 类比 |
|---|---|---|---|
| Q（Query） | Decoder 当前层的输出 | "我正在生成'爱'这个字，我需要知道源句子中哪些词和我相关" | 学生提问 |
| K（Key） | Encoder 的最终输出 | "这是源句子每个 Token 的'可被查询的索引'" | 老师列出的知识点标题 |
| V（Value） | Encoder 的最终输出 | "这是源句子每个 Token 的'实际内容'" | 知识点的具体内容 |
为什么 K 和 V 来自同一个 Encoder 输出？
因为 Encoder 输出的每个向量同时扮演了"索引"和"内容"两个角色。WKW_KWK 和 WVW_VWV 这两个不同的投影矩阵负责从同一个向量中提取不同的信息：WKW_KWK 提取"容易被匹配"的特征，WVW_VWV 提取"对生成有用"的内容。
对比 Self-Attention ：Self-Attention 的 Q、K、V 都来自同一个输入（自己问自己）。Cross-Attention 的 Q 来自 Decoder（提问者），K/V 来自 Encoder（信息源），是一种跨序列的信息拉取。
13.7 GPT 的 Decoder 与原始 Decoder 的区别
GPT 是 Decoder-Only 架构，去掉了 Cross-Attention：
| 组件 | 原始 Decoder | GPT Decoder |
|---|---|---|
| Masked Self-Attention | ✅ | ✅ |
| Cross-Attention | ✅ | ❌（没有Encoder） |
| Feed Forward | ✅ | ✅ |
GPT 的 Decoder Block：
#mermaid-svg-ZqwQ7STHavx1E9RE{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-ZqwQ7STHavx1E9RE .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-ZqwQ7STHavx1E9RE .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-ZqwQ7STHavx1E9RE .error-icon{fill:#552222;}#mermaid-svg-ZqwQ7STHavx1E9RE .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-ZqwQ7STHavx1E9RE .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-ZqwQ7STHavx1E9RE .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-ZqwQ7STHavx1E9RE .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-ZqwQ7STHavx1E9RE .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-ZqwQ7STHavx1E9RE .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-ZqwQ7STHavx1E9RE .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-ZqwQ7STHavx1E9RE .marker{fill:#333333;stroke:#333333;}#mermaid-svg-ZqwQ7STHavx1E9RE .marker.cross{stroke:#333333;}#mermaid-svg-ZqwQ7STHavx1E9RE svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-ZqwQ7STHavx1E9RE p{margin:0;}#mermaid-svg-ZqwQ7STHavx1E9RE .label{font-family:"trebuchet ms",verdana,arial,sans-serif;color:#333;}#mermaid-svg-ZqwQ7STHavx1E9RE .cluster-label text{fill:#333;}#mermaid-svg-ZqwQ7STHavx1E9RE .cluster-label span{color:#333;}#mermaid-svg-ZqwQ7STHavx1E9RE .cluster-label span p{background-color:transparent;}#mermaid-svg-ZqwQ7STHavx1E9RE .label text,#mermaid-svg-ZqwQ7STHavx1E9RE span{fill:#333;color:#333;}#mermaid-svg-ZqwQ7STHavx1E9RE .node rect,#mermaid-svg-ZqwQ7STHavx1E9RE .node circle,#mermaid-svg-ZqwQ7STHavx1E9RE .node ellipse,#mermaid-svg-ZqwQ7STHavx1E9RE .node polygon,#mermaid-svg-ZqwQ7STHavx1E9RE .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-ZqwQ7STHavx1E9RE .rough-node .label text,#mermaid-svg-ZqwQ7STHavx1E9RE .node .label text,#mermaid-svg-ZqwQ7STHavx1E9RE .image-shape .label,#mermaid-svg-ZqwQ7STHavx1E9RE .icon-shape .label{text-anchor:middle;}#mermaid-svg-ZqwQ7STHavx1E9RE .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-ZqwQ7STHavx1E9RE .rough-node .label,#mermaid-svg-ZqwQ7STHavx1E9RE .node .label,#mermaid-svg-ZqwQ7STHavx1E9RE .image-shape .label,#mermaid-svg-ZqwQ7STHavx1E9RE .icon-shape .label{text-align:center;}#mermaid-svg-ZqwQ7STHavx1E9RE .node.clickable{cursor:pointer;}#mermaid-svg-ZqwQ7STHavx1E9RE .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-ZqwQ7STHavx1E9RE .arrowheadPath{fill:#333333;}#mermaid-svg-ZqwQ7STHavx1E9RE .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-ZqwQ7STHavx1E9RE .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-ZqwQ7STHavx1E9RE .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-ZqwQ7STHavx1E9RE .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-ZqwQ7STHavx1E9RE .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-ZqwQ7STHavx1E9RE .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-ZqwQ7STHavx1E9RE .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-ZqwQ7STHavx1E9RE .cluster text{fill:#333;}#mermaid-svg-ZqwQ7STHavx1E9RE .cluster span{color:#333;}#mermaid-svg-ZqwQ7STHavx1E9RE div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-ZqwQ7STHavx1E9RE .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-ZqwQ7STHavx1E9RE rect.text{fill:none;stroke-width:0;}#mermaid-svg-ZqwQ7STHavx1E9RE .icon-shape,#mermaid-svg-ZqwQ7STHavx1E9RE .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-ZqwQ7STHavx1E9RE .icon-shape p,#mermaid-svg-ZqwQ7STHavx1E9RE .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-ZqwQ7STHavx1E9RE .icon-shape .label rect,#mermaid-svg-ZqwQ7STHavx1E9RE .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-ZqwQ7STHavx1E9RE .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-ZqwQ7STHavx1E9RE .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-ZqwQ7STHavx1E9RE :root{--mermaid-font-family:"trebuchet ms",verdana,arial,sans-serif;} 输入 x
Masked Self-Attention
残差 +
LayerNorm
Feed Forward
残差 +
LayerNorm
输出
13.7.1 Causal Mask 的隐藏功能：让训练可以并行
Causal Mask 不仅是为了"不偷看未来"，它还有一个极其重要的工程价值：
训练时，整句话一次性输入，利用 Mask 同时训练所有位置的预测。
| 训练 | 推理 | |
|---|---|---|
| 输入 | 整句话一次性输入（如5个Token） | 逐个Token输入 |
| Mask | 用 Causal Mask 遮住未来 | 天然只有过去的Token |
| 并行度 | 所有位置同时计算（高效） | 必须串行，一次生成一个（慢） |
| 一次Forward | 同时得到5个位置的预测结果 | 只得到1个新Token |
text
训练时（并行）：
输入: [The, bank, approved, the, loan] ← 一次全部输入
Mask 保证: 位置3只能看位置0,1,2,3
一次Forward → 同时得到5个预测
推理时（串行）：
第1步: 输入 [The] → 预测 bank
第2步: 输入 [The, bank] → 预测 approved
第3步: 输入 [The, bank, approved] → 预测 the
...这就是为什么 Transformer 训练比 RNN 快得多------训练时完全并行。而推理时不得不串行，是因为每一步的输出要作为下一步的输入（自回归）。
第15章的 KV Cache 正是为了加速这个串行推理过程------缓存已经计算过的 K 和 V，避免重复计算。
13.8 Padding Mask
除了 Causal Mask，还有 Padding Mask：
不同长度的序列需要 Padding（填充）到相同长度，Padding 的位置不应该参与 Attention 计算。
text
序列1: [I, love, AI, PAD, PAD]
序列2: [The, cat, sat, on, mat]
Padding Mask（序列1）:
[0, 0, 0, 1, 1] ← 1表示遮住PAD位置
python
def create_padding_mask(token_ids, pad_id=0):
"""
token_ids: [batch, seq_len]
返回: [batch, 1, 1, seq_len]
"""
mask = tf.cast(tf.equal(token_ids, pad_id), tf.float32)
return mask[:, tf.newaxis, tf.newaxis, :]13.9 TensorFlow 实现（完整 Decoder Block）
python
import tensorflow as tf
class DecoderBlock(tf.keras.layers.Layer):
def __init__(self, d_model, num_heads, dff, dropout_rate=0.1):
super().__init__()
self.self_attention = tf.keras.layers.MultiHeadAttention(
num_heads=num_heads, key_dim=d_model // num_heads
)
self.cross_attention = tf.keras.layers.MultiHeadAttention(
num_heads=num_heads, key_dim=d_model // num_heads
)
self.ffn = tf.keras.Sequential([
tf.keras.layers.Dense(dff, activation='relu'),
tf.keras.layers.Dense(d_model)
])
self.norm1 = tf.keras.layers.LayerNormalization(epsilon=1e-6)
self.norm2 = tf.keras.layers.LayerNormalization(epsilon=1e-6)
self.norm3 = tf.keras.layers.LayerNormalization(epsilon=1e-6)
def call(self, x, encoder_output, causal_mask=None, padding_mask=None, training=False):
# Masked Self-Attention
attn1 = self.self_attention(
self.norm1(x), self.norm1(x),
attention_mask=causal_mask
)
x = x + attn1
# Cross-Attention（如果有 Encoder 输出）
if encoder_output is not None:
attn2 = self.cross_attention(
self.norm2(x), encoder_output,
attention_mask=padding_mask
)
x = x + attn2
# FFN
ffn_out = self.ffn(self.norm3(x))
x = x + ffn_out
return x13.10 Decoder 的两种工作模式：训练 vs 推理
13.10.1 训练阶段：Teacher Forcing
训练时，我们已经有标准答案（平行语料）：
text
英文：I love AI
中文：我 爱 AIDecoder 的任务不是"翻译整句话"，而是逐个预测下一个 Token：
| Decoder 输入 | Decoder 应该预测 |
|---|---|
<BOS> |
我 |
<BOS> 我 |
爱 |
<BOS> 我 爱 |
AI |
<BOS> 我 爱 AI |
<EOS> |
<BOS>= Begin of Sequence（序列开始标记）
<EOS>= End of Sequence（序列结束标记）
为什么输入正确答案而不是模型自己的预测？
如果用模型自己的输出作为下一步输入：第一步预测错了（比如预测成"他"），后面全部跟着错------越错越远，训练根本收敛不了。
所以训练时直接喂正确答案：
不管模型预测了什么，下一步的输入永远是正确的前缀。
这就是 Teacher Forcing（教师强制）------老师直接告诉学生正确答案，而不是让学生从自己的错误中摸索。
13.10.2 推理阶段：自回归生成
真正使用模型时，没有正确答案。Decoder 只能用自己生成的 Token 作为下一步输入：
text
第1步: 输入 [<BOS>] → 生成: 我
第2步: 输入 [<BOS>, 我] → 生成: 爱
第3步: 输入 [<BOS>, 我, 爱] → 生成: AI
第4步: 输入 [<BOS>, 我, 爱, AI] → 生成: <EOS> ← 停止每次只生成一个 Token，追加到输入中，再生成下一个，直到输出 <EOS>。
13.10.3 训练 vs 推理 对比
| 训练（Teacher Forcing） | 推理（Auto-Regressive） | |
|---|---|---|
| Decoder 输入 | 正确答案的前缀 | 模型自己生成的前缀 |
| 速度 | 快（所有位置并行计算） | 慢（逐Token串行生成） |
| 是否知道答案 | ✅ 知道 | ❌ 不知道 |
| 可能出错吗 | 训练时不会偏离 | 可能"雪崩"（错误累积） |
Causal Mask 的作用：训练时整句话一次输入，但 Mask 保证每个位置只能看到过去------模拟了推理时"只有已生成Token"的约束。这样训练可以并行，又不会作弊。
本章总结
#mermaid-svg-f82LwRtkNWBnSwcI{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-f82LwRtkNWBnSwcI .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-f82LwRtkNWBnSwcI .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-f82LwRtkNWBnSwcI .error-icon{fill:#552222;}#mermaid-svg-f82LwRtkNWBnSwcI .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-f82LwRtkNWBnSwcI .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-f82LwRtkNWBnSwcI .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-f82LwRtkNWBnSwcI .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-f82LwRtkNWBnSwcI .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-f82LwRtkNWBnSwcI .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-f82LwRtkNWBnSwcI .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-f82LwRtkNWBnSwcI .marker{fill:#333333;stroke:#333333;}#mermaid-svg-f82LwRtkNWBnSwcI .marker.cross{stroke:#333333;}#mermaid-svg-f82LwRtkNWBnSwcI svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-f82LwRtkNWBnSwcI p{margin:0;}#mermaid-svg-f82LwRtkNWBnSwcI .edge{stroke-width:3;}#mermaid-svg-f82LwRtkNWBnSwcI .section--1 rect,#mermaid-svg-f82LwRtkNWBnSwcI .section--1 path,#mermaid-svg-f82LwRtkNWBnSwcI .section--1 circle,#mermaid-svg-f82LwRtkNWBnSwcI .section--1 polygon,#mermaid-svg-f82LwRtkNWBnSwcI .section--1 path{fill:hsl(240, 100%, 76.2745098039%);}#mermaid-svg-f82LwRtkNWBnSwcI .section--1 text{fill:#ffffff;}#mermaid-svg-f82LwRtkNWBnSwcI .node-icon--1{font-size:40px;color:#ffffff;}#mermaid-svg-f82LwRtkNWBnSwcI .section-edge--1{stroke:hsl(240, 100%, 76.2745098039%);}#mermaid-svg-f82LwRtkNWBnSwcI .edge-depth--1{stroke-width:17;}#mermaid-svg-f82LwRtkNWBnSwcI .section--1 line{stroke:hsl(60, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-f82LwRtkNWBnSwcI .disabled,#mermaid-svg-f82LwRtkNWBnSwcI .disabled circle,#mermaid-svg-f82LwRtkNWBnSwcI .disabled text{fill:lightgray;}#mermaid-svg-f82LwRtkNWBnSwcI .disabled text{fill:#efefef;}#mermaid-svg-f82LwRtkNWBnSwcI .section-0 rect,#mermaid-svg-f82LwRtkNWBnSwcI .section-0 path,#mermaid-svg-f82LwRtkNWBnSwcI .section-0 circle,#mermaid-svg-f82LwRtkNWBnSwcI .section-0 polygon,#mermaid-svg-f82LwRtkNWBnSwcI .section-0 path{fill:hsl(60, 100%, 73.5294117647%);}#mermaid-svg-f82LwRtkNWBnSwcI .section-0 text{fill:black;}#mermaid-svg-f82LwRtkNWBnSwcI .node-icon-0{font-size:40px;color:black;}#mermaid-svg-f82LwRtkNWBnSwcI .section-edge-0{stroke:hsl(60, 100%, 73.5294117647%);}#mermaid-svg-f82LwRtkNWBnSwcI .edge-depth-0{stroke-width:14;}#mermaid-svg-f82LwRtkNWBnSwcI .section-0 line{stroke:hsl(240, 100%, 83.5294117647%);stroke-width:3;}#mermaid-svg-f82LwRtkNWBnSwcI .disabled,#mermaid-svg-f82LwRtkNWBnSwcI .disabled circle,#mermaid-svg-f82LwRtkNWBnSwcI .disabled text{fill:lightgray;}#mermaid-svg-f82LwRtkNWBnSwcI .disabled text{fill:#efefef;}#mermaid-svg-f82LwRtkNWBnSwcI .section-1 rect,#mermaid-svg-f82LwRtkNWBnSwcI .section-1 path,#mermaid-svg-f82LwRtkNWBnSwcI .section-1 circle,#mermaid-svg-f82LwRtkNWBnSwcI .section-1 polygon,#mermaid-svg-f82LwRtkNWBnSwcI .section-1 path{fill:hsl(80, 100%, 76.2745098039%);}#mermaid-svg-f82LwRtkNWBnSwcI .section-1 text{fill:black;}#mermaid-svg-f82LwRtkNWBnSwcI .node-icon-1{font-size:40px;color:black;}#mermaid-svg-f82LwRtkNWBnSwcI .section-edge-1{stroke:hsl(80, 100%, 76.2745098039%);}#mermaid-svg-f82LwRtkNWBnSwcI .edge-depth-1{stroke-width:11;}#mermaid-svg-f82LwRtkNWBnSwcI .section-1 line{stroke:hsl(260, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-f82LwRtkNWBnSwcI .disabled,#mermaid-svg-f82LwRtkNWBnSwcI .disabled circle,#mermaid-svg-f82LwRtkNWBnSwcI .disabled text{fill:lightgray;}#mermaid-svg-f82LwRtkNWBnSwcI .disabled text{fill:#efefef;}#mermaid-svg-f82LwRtkNWBnSwcI .section-2 rect,#mermaid-svg-f82LwRtkNWBnSwcI .section-2 path,#mermaid-svg-f82LwRtkNWBnSwcI .section-2 circle,#mermaid-svg-f82LwRtkNWBnSwcI .section-2 polygon,#mermaid-svg-f82LwRtkNWBnSwcI .section-2 path{fill:hsl(270, 100%, 76.2745098039%);}#mermaid-svg-f82LwRtkNWBnSwcI .section-2 text{fill:#ffffff;}#mermaid-svg-f82LwRtkNWBnSwcI .node-icon-2{font-size:40px;color:#ffffff;}#mermaid-svg-f82LwRtkNWBnSwcI .section-edge-2{stroke:hsl(270, 100%, 76.2745098039%);}#mermaid-svg-f82LwRtkNWBnSwcI .edge-depth-2{stroke-width:8;}#mermaid-svg-f82LwRtkNWBnSwcI .section-2 line{stroke:hsl(90, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-f82LwRtkNWBnSwcI .disabled,#mermaid-svg-f82LwRtkNWBnSwcI .disabled circle,#mermaid-svg-f82LwRtkNWBnSwcI .disabled text{fill:lightgray;}#mermaid-svg-f82LwRtkNWBnSwcI .disabled text{fill:#efefef;}#mermaid-svg-f82LwRtkNWBnSwcI .section-3 rect,#mermaid-svg-f82LwRtkNWBnSwcI .section-3 path,#mermaid-svg-f82LwRtkNWBnSwcI .section-3 circle,#mermaid-svg-f82LwRtkNWBnSwcI .section-3 polygon,#mermaid-svg-f82LwRtkNWBnSwcI .section-3 path{fill:hsl(300, 100%, 76.2745098039%);}#mermaid-svg-f82LwRtkNWBnSwcI .section-3 text{fill:black;}#mermaid-svg-f82LwRtkNWBnSwcI .node-icon-3{font-size:40px;color:black;}#mermaid-svg-f82LwRtkNWBnSwcI .section-edge-3{stroke:hsl(300, 100%, 76.2745098039%);}#mermaid-svg-f82LwRtkNWBnSwcI .edge-depth-3{stroke-width:5;}#mermaid-svg-f82LwRtkNWBnSwcI .section-3 line{stroke:hsl(120, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-f82LwRtkNWBnSwcI .disabled,#mermaid-svg-f82LwRtkNWBnSwcI .disabled circle,#mermaid-svg-f82LwRtkNWBnSwcI .disabled text{fill:lightgray;}#mermaid-svg-f82LwRtkNWBnSwcI .disabled text{fill:#efefef;}#mermaid-svg-f82LwRtkNWBnSwcI .section-4 rect,#mermaid-svg-f82LwRtkNWBnSwcI .section-4 path,#mermaid-svg-f82LwRtkNWBnSwcI .section-4 circle,#mermaid-svg-f82LwRtkNWBnSwcI .section-4 polygon,#mermaid-svg-f82LwRtkNWBnSwcI .section-4 path{fill:hsl(330, 100%, 76.2745098039%);}#mermaid-svg-f82LwRtkNWBnSwcI .section-4 text{fill:black;}#mermaid-svg-f82LwRtkNWBnSwcI .node-icon-4{font-size:40px;color:black;}#mermaid-svg-f82LwRtkNWBnSwcI .section-edge-4{stroke:hsl(330, 100%, 76.2745098039%);}#mermaid-svg-f82LwRtkNWBnSwcI .edge-depth-4{stroke-width:2;}#mermaid-svg-f82LwRtkNWBnSwcI .section-4 line{stroke:hsl(150, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-f82LwRtkNWBnSwcI .disabled,#mermaid-svg-f82LwRtkNWBnSwcI .disabled circle,#mermaid-svg-f82LwRtkNWBnSwcI .disabled text{fill:lightgray;}#mermaid-svg-f82LwRtkNWBnSwcI .disabled text{fill:#efefef;}#mermaid-svg-f82LwRtkNWBnSwcI .section-5 rect,#mermaid-svg-f82LwRtkNWBnSwcI .section-5 path,#mermaid-svg-f82LwRtkNWBnSwcI .section-5 circle,#mermaid-svg-f82LwRtkNWBnSwcI .section-5 polygon,#mermaid-svg-f82LwRtkNWBnSwcI .section-5 path{fill:hsl(0, 100%, 76.2745098039%);}#mermaid-svg-f82LwRtkNWBnSwcI .section-5 text{fill:black;}#mermaid-svg-f82LwRtkNWBnSwcI .node-icon-5{font-size:40px;color:black;}#mermaid-svg-f82LwRtkNWBnSwcI .section-edge-5{stroke:hsl(0, 100%, 76.2745098039%);}#mermaid-svg-f82LwRtkNWBnSwcI .edge-depth-5{stroke-width:-1;}#mermaid-svg-f82LwRtkNWBnSwcI .section-5 line{stroke:hsl(180, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-f82LwRtkNWBnSwcI .disabled,#mermaid-svg-f82LwRtkNWBnSwcI .disabled circle,#mermaid-svg-f82LwRtkNWBnSwcI .disabled text{fill:lightgray;}#mermaid-svg-f82LwRtkNWBnSwcI .disabled text{fill:#efefef;}#mermaid-svg-f82LwRtkNWBnSwcI .section-6 rect,#mermaid-svg-f82LwRtkNWBnSwcI .section-6 path,#mermaid-svg-f82LwRtkNWBnSwcI .section-6 circle,#mermaid-svg-f82LwRtkNWBnSwcI .section-6 polygon,#mermaid-svg-f82LwRtkNWBnSwcI .section-6 path{fill:hsl(30, 100%, 76.2745098039%);}#mermaid-svg-f82LwRtkNWBnSwcI .section-6 text{fill:black;}#mermaid-svg-f82LwRtkNWBnSwcI .node-icon-6{font-size:40px;color:black;}#mermaid-svg-f82LwRtkNWBnSwcI .section-edge-6{stroke:hsl(30, 100%, 76.2745098039%);}#mermaid-svg-f82LwRtkNWBnSwcI .edge-depth-6{stroke-width:-4;}#mermaid-svg-f82LwRtkNWBnSwcI .section-6 line{stroke:hsl(210, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-f82LwRtkNWBnSwcI .disabled,#mermaid-svg-f82LwRtkNWBnSwcI .disabled circle,#mermaid-svg-f82LwRtkNWBnSwcI .disabled text{fill:lightgray;}#mermaid-svg-f82LwRtkNWBnSwcI .disabled text{fill:#efefef;}#mermaid-svg-f82LwRtkNWBnSwcI .section-7 rect,#mermaid-svg-f82LwRtkNWBnSwcI .section-7 path,#mermaid-svg-f82LwRtkNWBnSwcI .section-7 circle,#mermaid-svg-f82LwRtkNWBnSwcI .section-7 polygon,#mermaid-svg-f82LwRtkNWBnSwcI .section-7 path{fill:hsl(90, 100%, 76.2745098039%);}#mermaid-svg-f82LwRtkNWBnSwcI .section-7 text{fill:black;}#mermaid-svg-f82LwRtkNWBnSwcI .node-icon-7{font-size:40px;color:black;}#mermaid-svg-f82LwRtkNWBnSwcI .section-edge-7{stroke:hsl(90, 100%, 76.2745098039%);}#mermaid-svg-f82LwRtkNWBnSwcI .edge-depth-7{stroke-width:-7;}#mermaid-svg-f82LwRtkNWBnSwcI .section-7 line{stroke:hsl(270, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-f82LwRtkNWBnSwcI .disabled,#mermaid-svg-f82LwRtkNWBnSwcI .disabled circle,#mermaid-svg-f82LwRtkNWBnSwcI .disabled text{fill:lightgray;}#mermaid-svg-f82LwRtkNWBnSwcI .disabled text{fill:#efefef;}#mermaid-svg-f82LwRtkNWBnSwcI .section-8 rect,#mermaid-svg-f82LwRtkNWBnSwcI .section-8 path,#mermaid-svg-f82LwRtkNWBnSwcI .section-8 circle,#mermaid-svg-f82LwRtkNWBnSwcI .section-8 polygon,#mermaid-svg-f82LwRtkNWBnSwcI .section-8 path{fill:hsl(150, 100%, 76.2745098039%);}#mermaid-svg-f82LwRtkNWBnSwcI .section-8 text{fill:black;}#mermaid-svg-f82LwRtkNWBnSwcI .node-icon-8{font-size:40px;color:black;}#mermaid-svg-f82LwRtkNWBnSwcI .section-edge-8{stroke:hsl(150, 100%, 76.2745098039%);}#mermaid-svg-f82LwRtkNWBnSwcI .edge-depth-8{stroke-width:-10;}#mermaid-svg-f82LwRtkNWBnSwcI .section-8 line{stroke:hsl(330, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-f82LwRtkNWBnSwcI .disabled,#mermaid-svg-f82LwRtkNWBnSwcI .disabled circle,#mermaid-svg-f82LwRtkNWBnSwcI .disabled text{fill:lightgray;}#mermaid-svg-f82LwRtkNWBnSwcI .disabled text{fill:#efefef;}#mermaid-svg-f82LwRtkNWBnSwcI .section-9 rect,#mermaid-svg-f82LwRtkNWBnSwcI .section-9 path,#mermaid-svg-f82LwRtkNWBnSwcI .section-9 circle,#mermaid-svg-f82LwRtkNWBnSwcI .section-9 polygon,#mermaid-svg-f82LwRtkNWBnSwcI .section-9 path{fill:hsl(180, 100%, 76.2745098039%);}#mermaid-svg-f82LwRtkNWBnSwcI .section-9 text{fill:black;}#mermaid-svg-f82LwRtkNWBnSwcI .node-icon-9{font-size:40px;color:black;}#mermaid-svg-f82LwRtkNWBnSwcI .section-edge-9{stroke:hsl(180, 100%, 76.2745098039%);}#mermaid-svg-f82LwRtkNWBnSwcI .edge-depth-9{stroke-width:-13;}#mermaid-svg-f82LwRtkNWBnSwcI .section-9 line{stroke:hsl(0, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-f82LwRtkNWBnSwcI .disabled,#mermaid-svg-f82LwRtkNWBnSwcI .disabled circle,#mermaid-svg-f82LwRtkNWBnSwcI .disabled text{fill:lightgray;}#mermaid-svg-f82LwRtkNWBnSwcI .disabled text{fill:#efefef;}#mermaid-svg-f82LwRtkNWBnSwcI .section-10 rect,#mermaid-svg-f82LwRtkNWBnSwcI .section-10 path,#mermaid-svg-f82LwRtkNWBnSwcI .section-10 circle,#mermaid-svg-f82LwRtkNWBnSwcI .section-10 polygon,#mermaid-svg-f82LwRtkNWBnSwcI .section-10 path{fill:hsl(210, 100%, 76.2745098039%);}#mermaid-svg-f82LwRtkNWBnSwcI .section-10 text{fill:black;}#mermaid-svg-f82LwRtkNWBnSwcI .node-icon-10{font-size:40px;color:black;}#mermaid-svg-f82LwRtkNWBnSwcI .section-edge-10{stroke:hsl(210, 100%, 76.2745098039%);}#mermaid-svg-f82LwRtkNWBnSwcI .edge-depth-10{stroke-width:-16;}#mermaid-svg-f82LwRtkNWBnSwcI .section-10 line{stroke:hsl(30, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-f82LwRtkNWBnSwcI .disabled,#mermaid-svg-f82LwRtkNWBnSwcI .disabled circle,#mermaid-svg-f82LwRtkNWBnSwcI .disabled text{fill:lightgray;}#mermaid-svg-f82LwRtkNWBnSwcI .disabled text{fill:#efefef;}#mermaid-svg-f82LwRtkNWBnSwcI .section-root rect,#mermaid-svg-f82LwRtkNWBnSwcI .section-root path,#mermaid-svg-f82LwRtkNWBnSwcI .section-root circle,#mermaid-svg-f82LwRtkNWBnSwcI .section-root polygon{fill:hsl(240, 100%, 46.2745098039%);}#mermaid-svg-f82LwRtkNWBnSwcI .section-root text{fill:#ffffff;}#mermaid-svg-f82LwRtkNWBnSwcI .section-root span{color:#ffffff;}#mermaid-svg-f82LwRtkNWBnSwcI .section-2 span{color:#ffffff;}#mermaid-svg-f82LwRtkNWBnSwcI .icon-container{height:100%;display:flex;justify-content:center;align-items:center;}#mermaid-svg-f82LwRtkNWBnSwcI .edge{fill:none;}#mermaid-svg-f82LwRtkNWBnSwcI .mindmap-node-label{dy:1em;alignment-baseline:middle;text-anchor:middle;dominant-baseline:middle;text-align:center;}#mermaid-svg-f82LwRtkNWBnSwcI :root{--mermaid-font-family:"trebuchet ms",verdana,arial,sans-serif;} Decoder
Causal Mask
上三角矩阵
遮住未来Token
防止作弊
Masked Self-Attention
只看当前和之前
自回归生成
Cross-Attention
Q来自Decoder
K,V来自Encoder
获取源句子信息
GPT Decoder
去掉Cross-Attention
只有Masked Self-Attention
本章思考题
- 为什么 Causal Mask 是上三角矩阵，而不是下三角？
- 如果训练时不用 Mask，推理时会发生什么？
- Cross-Attention 中，Q 来自 Decoder，K 和 V 来自 Encoder，这样设计的原因是什么？
- GPT 去掉了 Cross-Attention，那它是如何"理解"输入的？
下一章预告
下一章我们讲 GPT。
为什么 GPT 能聊天？Next Token Prediction 是如何工作的？