Transformer与大语言模型：第11章 Encoder 是如何理解句子的

第11章 Encoder 是如何理解句子的

本章目标：

理解 Encoder 如何逐层建立对句子的理解，CLS Token 的作用，以及如何得到 Sentence Embedding。

---

11.1 Encoder 的整体流程

#mermaid-svg-LZHeA1gqV21FChYm{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-LZHeA1gqV21FChYm .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-LZHeA1gqV21FChYm .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-LZHeA1gqV21FChYm .error-icon{fill:#552222;}#mermaid-svg-LZHeA1gqV21FChYm .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-LZHeA1gqV21FChYm .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-LZHeA1gqV21FChYm .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-LZHeA1gqV21FChYm .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-LZHeA1gqV21FChYm .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-LZHeA1gqV21FChYm .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-LZHeA1gqV21FChYm .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-LZHeA1gqV21FChYm .marker{fill:#333333;stroke:#333333;}#mermaid-svg-LZHeA1gqV21FChYm .marker.cross{stroke:#333333;}#mermaid-svg-LZHeA1gqV21FChYm svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-LZHeA1gqV21FChYm p{margin:0;}#mermaid-svg-LZHeA1gqV21FChYm .label{font-family:"trebuchet ms",verdana,arial,sans-serif;color:#333;}#mermaid-svg-LZHeA1gqV21FChYm .cluster-label text{fill:#333;}#mermaid-svg-LZHeA1gqV21FChYm .cluster-label span{color:#333;}#mermaid-svg-LZHeA1gqV21FChYm .cluster-label span p{background-color:transparent;}#mermaid-svg-LZHeA1gqV21FChYm .label text,#mermaid-svg-LZHeA1gqV21FChYm span{fill:#333;color:#333;}#mermaid-svg-LZHeA1gqV21FChYm .node rect,#mermaid-svg-LZHeA1gqV21FChYm .node circle,#mermaid-svg-LZHeA1gqV21FChYm .node ellipse,#mermaid-svg-LZHeA1gqV21FChYm .node polygon,#mermaid-svg-LZHeA1gqV21FChYm .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-LZHeA1gqV21FChYm .rough-node .label text,#mermaid-svg-LZHeA1gqV21FChYm .node .label text,#mermaid-svg-LZHeA1gqV21FChYm .image-shape .label,#mermaid-svg-LZHeA1gqV21FChYm .icon-shape .label{text-anchor:middle;}#mermaid-svg-LZHeA1gqV21FChYm .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-LZHeA1gqV21FChYm .rough-node .label,#mermaid-svg-LZHeA1gqV21FChYm .node .label,#mermaid-svg-LZHeA1gqV21FChYm .image-shape .label,#mermaid-svg-LZHeA1gqV21FChYm .icon-shape .label{text-align:center;}#mermaid-svg-LZHeA1gqV21FChYm .node.clickable{cursor:pointer;}#mermaid-svg-LZHeA1gqV21FChYm .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-LZHeA1gqV21FChYm .arrowheadPath{fill:#333333;}#mermaid-svg-LZHeA1gqV21FChYm .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-LZHeA1gqV21FChYm .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-LZHeA1gqV21FChYm .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-LZHeA1gqV21FChYm .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-LZHeA1gqV21FChYm .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-LZHeA1gqV21FChYm .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-LZHeA1gqV21FChYm .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-LZHeA1gqV21FChYm .cluster text{fill:#333;}#mermaid-svg-LZHeA1gqV21FChYm .cluster span{color:#333;}#mermaid-svg-LZHeA1gqV21FChYm div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-LZHeA1gqV21FChYm .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-LZHeA1gqV21FChYm rect.text{fill:none;stroke-width:0;}#mermaid-svg-LZHeA1gqV21FChYm .icon-shape,#mermaid-svg-LZHeA1gqV21FChYm .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-LZHeA1gqV21FChYm .icon-shape p,#mermaid-svg-LZHeA1gqV21FChYm .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-LZHeA1gqV21FChYm .icon-shape .label rect,#mermaid-svg-LZHeA1gqV21FChYm .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-LZHeA1gqV21FChYm .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-LZHeA1gqV21FChYm .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-LZHeA1gqV21FChYm :root{--mermaid-font-family:"trebuchet ms",verdana,arial,sans-serif;} 输入: I love deep learning
Tokenizer

CLS\] I love deep learning \[SEP

Token IDs
Embedding + Position Encoding

batch, seq, d_model

Encoder Block 1

浅层: 语法特征
Encoder Block 2

中层: 语义特征
Encoder Block N

深层: 抽象特征
输出: 每个Token的上下文向量

batch, seq, d_model

---

11.2 每一层学到了什么？

研究表明，Encoder 的不同层学到了不同层次的特征：

层次	学到的特征	例子
浅层（1-3层）	词法、语法	词性、句法结构
中层（4-8层）	语义	词义消歧、实体识别
深层（9-12层）	抽象语义	语义角色、推理

---

11.3 Attention Matrix 的可视化

每一层的 Attention Matrix 可以可视化，展示 Token 之间的关注关系：

第1层（语法层）：

         The  animal  didn't  cross  the  street  because  it  was  tired
The     [0.8   0.1    0.0    0.0   0.1   0.0     0.0     0.0  0.0   0.0]
animal  [0.1   0.7    0.1    0.0   0.0   0.0     0.0     0.1  0.0   0.0]
it      [0.0   0.6    0.0    0.0   0.0   0.1     0.0     0.2  0.0   0.1]

it 在第1层就开始关注 animal（0.6）。

深层（语义层）：

it → animal: 0.9  （强烈指代关系）

---

11.4 Token 如何逐渐理解上下文

以 bank 为例：

句子1: I went to the bank to deposit money.
句子2: We sat by the river bank.

初始 Embedding ：两个句子中的 bank 向量完全相同（查表得到）。

经过 Encoder 后：
#mermaid-svg-QmnxufnWHn56swp9{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-QmnxufnWHn56swp9 .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-QmnxufnWHn56swp9 .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-QmnxufnWHn56swp9 .error-icon{fill:#552222;}#mermaid-svg-QmnxufnWHn56swp9 .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-QmnxufnWHn56swp9 .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-QmnxufnWHn56swp9 .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-QmnxufnWHn56swp9 .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-QmnxufnWHn56swp9 .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-QmnxufnWHn56swp9 .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-QmnxufnWHn56swp9 .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-QmnxufnWHn56swp9 .marker{fill:#333333;stroke:#333333;}#mermaid-svg-QmnxufnWHn56swp9 .marker.cross{stroke:#333333;}#mermaid-svg-QmnxufnWHn56swp9 svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-QmnxufnWHn56swp9 p{margin:0;}#mermaid-svg-QmnxufnWHn56swp9 .label{font-family:"trebuchet ms",verdana,arial,sans-serif;color:#333;}#mermaid-svg-QmnxufnWHn56swp9 .cluster-label text{fill:#333;}#mermaid-svg-QmnxufnWHn56swp9 .cluster-label span{color:#333;}#mermaid-svg-QmnxufnWHn56swp9 .cluster-label span p{background-color:transparent;}#mermaid-svg-QmnxufnWHn56swp9 .label text,#mermaid-svg-QmnxufnWHn56swp9 span{fill:#333;color:#333;}#mermaid-svg-QmnxufnWHn56swp9 .node rect,#mermaid-svg-QmnxufnWHn56swp9 .node circle,#mermaid-svg-QmnxufnWHn56swp9 .node ellipse,#mermaid-svg-QmnxufnWHn56swp9 .node polygon,#mermaid-svg-QmnxufnWHn56swp9 .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-QmnxufnWHn56swp9 .rough-node .label text,#mermaid-svg-QmnxufnWHn56swp9 .node .label text,#mermaid-svg-QmnxufnWHn56swp9 .image-shape .label,#mermaid-svg-QmnxufnWHn56swp9 .icon-shape .label{text-anchor:middle;}#mermaid-svg-QmnxufnWHn56swp9 .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-QmnxufnWHn56swp9 .rough-node .label,#mermaid-svg-QmnxufnWHn56swp9 .node .label,#mermaid-svg-QmnxufnWHn56swp9 .image-shape .label,#mermaid-svg-QmnxufnWHn56swp9 .icon-shape .label{text-align:center;}#mermaid-svg-QmnxufnWHn56swp9 .node.clickable{cursor:pointer;}#mermaid-svg-QmnxufnWHn56swp9 .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-QmnxufnWHn56swp9 .arrowheadPath{fill:#333333;}#mermaid-svg-QmnxufnWHn56swp9 .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-QmnxufnWHn56swp9 .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-QmnxufnWHn56swp9 .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-QmnxufnWHn56swp9 .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-QmnxufnWHn56swp9 .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-QmnxufnWHn56swp9 .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-QmnxufnWHn56swp9 .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-QmnxufnWHn56swp9 .cluster text{fill:#333;}#mermaid-svg-QmnxufnWHn56swp9 .cluster span{color:#333;}#mermaid-svg-QmnxufnWHn56swp9 div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-QmnxufnWHn56swp9 .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-QmnxufnWHn56swp9 rect.text{fill:none;stroke-width:0;}#mermaid-svg-QmnxufnWHn56swp9 .icon-shape,#mermaid-svg-QmnxufnWHn56swp9 .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-QmnxufnWHn56swp9 .icon-shape p,#mermaid-svg-QmnxufnWHn56swp9 .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-QmnxufnWHn56swp9 .icon-shape .label rect,#mermaid-svg-QmnxufnWHn56swp9 .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-QmnxufnWHn56swp9 .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-QmnxufnWHn56swp9 .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-QmnxufnWHn56swp9 :root{--mermaid-font-family:"trebuchet ms",verdana,arial,sans-serif;} bank (初始)

同一个向量
Encoder Block 1
bank (句子1)

开始偏向金融语义
bank (句子2)

开始偏向地理语义
Encoder Block N
bank (句子1)

完全是银行的语义
bank (句子2)

完全是河岸的语义

这就是 上下文相关的词向量（Contextual Embedding）。

11.4.1 Encoder 输出的是"语义"，不是"英文"

这是一个容易被忽略但极其重要的观点：

Encoder 的输出不是"英文向量"，而是"语言无关的语义表示"。

以翻译 I love AI → 我爱AI 为例：

经过6层 Encoder 后，I 的向量已经不再代表"英文单词 I"，而是代表第一人称、主语、句子起始这些与语言无关的语义信息。

"I" 的 Encoder 输出 ≈ "第一人称"（语义）
"我" 如果送进同样的 Encoder ≈ 也接近"第一人称"（语义）

这解释了 Cross-Attention 为什么能工作：

Encoder 输出: 语义空间（语言无关）
    ↕ Cross-Attention
Decoder: 用中文表达这些语义

Decoder 通过 Cross-Attention 读取的不是"英文"，而是"意义"。然后用目标语言（中文）的词汇来表达这些意义。

💡 这也是为什么现代多语言模型（如 mBERT、mT5）能够只用一个模型处理几十种语言------不同语言的相同含义，在训练后会自然聚集到语义空间的相近位置。

---

11.5 CLS Token

BERT 在输入序列的开头加了一个特殊 Token：[CLS]。

输入: [CLS] I love deep learning [SEP]

很多教材会直接说：

"经过 Encoder 后，CLS 向量代表整个句子。"

但这句话背后的机制值得仔细展开------[CLS] 到底是怎么"一步步变成整句话的代表"的？

11.5.1 一开始，CLS 什么都不知道

初始时，[CLS] 只是一个特殊 token 的 Embedding 向量------它并不包含任何句子信息：

[CLS]      → h_cls  （只是一个随机初始化的向量）
I          → h_1
love       → h_2
deep       → h_3
learning   → h_4

那它最终为什么能代表整个句子？

11.5.2 Self-Attention 让 CLS 逐层"读取"所有 Token

在每一层 Encoder Block 中，[CLS] 都会通过 Self-Attention 对所有 Token 计算注意力权重：

QCLS=hCLSWQQ_{CLS} = h_{CLS} W_QQCLS=hCLSWQ

αCLS,j=softmax(QCLS⋅KjTdk)\alpha_{CLS,j} = \text{softmax}\left(\frac{Q_{CLS} \cdot K_j^T}{\sqrt{d_k}}\right)αCLS,j=softmax(dk QCLS⋅KjT)

hCLSnew=∑jαCLS,j⋅Vjh_{CLS}^{new} = \sum_{j} \alpha_{CLS,j} \cdot V_jhCLSnew=j∑αCLS,j⋅Vj

也就是说，[CLS] 会根据自己的 Query 去"询问"每一个 Token，然后按注意力权重把它们的信息融合到自己的向量中：

              ┌── I         (α = 0.05)
              │
              ├── love      (α = 0.20)
              │
[CLS] ────────┼── deep      (α = 0.15)
              │
              ├── learning  (α = 0.55)
              │
              └── [SEP]     (α = 0.05)

hCLSnew=0.05⋅VI+0.20⋅Vlove+0.15⋅Vdeep+0.55⋅Vlearning+⋯h_{CLS}^{new} = 0.05 \cdot V_I + 0.20 \cdot V_{love} + 0.15 \cdot V_{deep} + 0.55 \cdot V_{learning} + \cdotshCLSnew=0.05⋅VI+0.20⋅Vlove+0.15⋅Vdeep+0.55⋅Vlearning+⋯

11.5.3 多层堆叠：信息越来越丰富

这个过程在 12 层 Encoder 中反复进行：

Layer 1:   CLS 读取所有 Token → 获得浅层信息（词法）
Layer 2:   CLS 读取已被上下文化的 Token → 获得更丰富的信息
Layer 3:   CLS 进一步融合 → 形成语义理解
...
Layer 12:  CLS 已包含高度抽象的全局语义表示

每一层 [CLS] 读取的不再是原始 Token，而是已经被前面所有层加工过的 Token（它们自己也越来越"聪明"了）。所以信息是逐层递进、越来越丰富的。

11.5.4 CLS 的特殊之处：没有自己的词汇语义

这是 [CLS] 设计的精妙之处。对比：

Token	自身语义	Attention 时的角色
I	人称代词	既要理解别人，也要表达"我"的含义
love	情感/动作	既要理解别人，也要表达"爱"的含义
[CLS]	无自然语言语义	不需要表达任何具体含义，专心汇聚全局信息

💡 [CLS] 像一个没有立场的会议记录员 ------其他 Token 都是发言者，既要说自己的观点又要听别人的；而 [CLS] 的唯一任务就是倾听所有人，形成一份会议纪要。

11.5.5 CLS 为什么能学会"代表整句话"？

[CLS] 不是天生就有全局表示能力------这是训练目标逼出来的。

例如 BERT 做情感分类时：

[CLS] 我 喜欢 学习 Python [SEP]
  │
  ▼
最终 [CLS] 向量
  │
  ▼
分类器 → 正面/负面

反向传播时，Loss 会传回 [CLS]，迫使它的向量学会包含判断情感所需的信息。如果不去读取"喜欢""学习"等关键词，分类器就无法做出正确判断。

所以是训练目标 驱动了 [CLS] 成为全局表示，而不是它的位置或某种特殊机制。

11.5.6 CLS 和其他 Token 在机制上是平等的

一个容易产生的误解是"[CLS] 有某种特殊的 Attention 权限"。

实际上：[CLS] 和其他 Token 使用完全相同的 WQW_QWQ、WKW_KWK、WVW_VWV。它没有任何特殊的计算路径。

区别只在于：

• 它的初始 Embedding 不对应任何自然语言词汇
• 训练目标（分类 Loss）直接作用于它的输出

📌 总结 ：CLS 和其他 Token 在数学机制上完全平等；区别不在 Attention 的计算方式，而在它的语义角色 （无具体词义）和训练目标（被逼着学会汇聚全局信息）。

---

11.6 CLS Token 的用途

[CLS] 向量通常用于：

1. 文本分类 ：在 [CLS] 向量上加一个分类头
2. 句子相似度 ：比较两个句子的 [CLS] 向量
3. Sentence Embedding ：用 [CLS] 向量表示整个句子

#mermaid-svg-sTutckbsFL7hG2dK{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-sTutckbsFL7hG2dK .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-sTutckbsFL7hG2dK .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-sTutckbsFL7hG2dK .error-icon{fill:#552222;}#mermaid-svg-sTutckbsFL7hG2dK .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-sTutckbsFL7hG2dK .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-sTutckbsFL7hG2dK .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-sTutckbsFL7hG2dK .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-sTutckbsFL7hG2dK .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-sTutckbsFL7hG2dK .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-sTutckbsFL7hG2dK .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-sTutckbsFL7hG2dK .marker{fill:#333333;stroke:#333333;}#mermaid-svg-sTutckbsFL7hG2dK .marker.cross{stroke:#333333;}#mermaid-svg-sTutckbsFL7hG2dK svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-sTutckbsFL7hG2dK p{margin:0;}#mermaid-svg-sTutckbsFL7hG2dK .label{font-family:"trebuchet ms",verdana,arial,sans-serif;color:#333;}#mermaid-svg-sTutckbsFL7hG2dK .cluster-label text{fill:#333;}#mermaid-svg-sTutckbsFL7hG2dK .cluster-label span{color:#333;}#mermaid-svg-sTutckbsFL7hG2dK .cluster-label span p{background-color:transparent;}#mermaid-svg-sTutckbsFL7hG2dK .label text,#mermaid-svg-sTutckbsFL7hG2dK span{fill:#333;color:#333;}#mermaid-svg-sTutckbsFL7hG2dK .node rect,#mermaid-svg-sTutckbsFL7hG2dK .node circle,#mermaid-svg-sTutckbsFL7hG2dK .node ellipse,#mermaid-svg-sTutckbsFL7hG2dK .node polygon,#mermaid-svg-sTutckbsFL7hG2dK .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-sTutckbsFL7hG2dK .rough-node .label text,#mermaid-svg-sTutckbsFL7hG2dK .node .label text,#mermaid-svg-sTutckbsFL7hG2dK .image-shape .label,#mermaid-svg-sTutckbsFL7hG2dK .icon-shape .label{text-anchor:middle;}#mermaid-svg-sTutckbsFL7hG2dK .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-sTutckbsFL7hG2dK .rough-node .label,#mermaid-svg-sTutckbsFL7hG2dK .node .label,#mermaid-svg-sTutckbsFL7hG2dK .image-shape .label,#mermaid-svg-sTutckbsFL7hG2dK .icon-shape .label{text-align:center;}#mermaid-svg-sTutckbsFL7hG2dK .node.clickable{cursor:pointer;}#mermaid-svg-sTutckbsFL7hG2dK .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-sTutckbsFL7hG2dK .arrowheadPath{fill:#333333;}#mermaid-svg-sTutckbsFL7hG2dK .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-sTutckbsFL7hG2dK .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-sTutckbsFL7hG2dK .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-sTutckbsFL7hG2dK .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-sTutckbsFL7hG2dK .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-sTutckbsFL7hG2dK .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-sTutckbsFL7hG2dK .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-sTutckbsFL7hG2dK .cluster text{fill:#333;}#mermaid-svg-sTutckbsFL7hG2dK .cluster span{color:#333;}#mermaid-svg-sTutckbsFL7hG2dK div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-sTutckbsFL7hG2dK .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-sTutckbsFL7hG2dK rect.text{fill:none;stroke-width:0;}#mermaid-svg-sTutckbsFL7hG2dK .icon-shape,#mermaid-svg-sTutckbsFL7hG2dK .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-sTutckbsFL7hG2dK .icon-shape p,#mermaid-svg-sTutckbsFL7hG2dK .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-sTutckbsFL7hG2dK .icon-shape .label rect,#mermaid-svg-sTutckbsFL7hG2dK .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-sTutckbsFL7hG2dK .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-sTutckbsFL7hG2dK .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-sTutckbsFL7hG2dK :root{--mermaid-font-family:"trebuchet ms",verdana,arial,sans-serif;} 输入句子
BERT Encoder
CLS 向量

768维

分类头

→ 情感分析
相似度计算

→ 语义搜索
直接作为

Sentence Embedding

11.6.1 为什么不直接用其他方式聚合？

既然每个 Token 经过 Encoder 后都包含了上下文信息，为什么不直接取平均，而要专门设计一个 [CLS]？

平均池化（Mean Pooling）：

hsentence=1N∑i=1Nhih_{sentence} = \frac{1}{N} \sum_{i=1}^{N} h_ihsentence=N1i=1∑Nhi

每个 Token 权重固定为 1N\frac{1}{N}N1------"学习"和"Python"和"的"的权重完全一样。

CLS 通过 Attention：

hCLS=∑jαj⋅Vjh_{CLS} = \sum_{j} \alpha_j \cdot V_jhCLS=j∑αj⋅Vj

每个 Token 的权重 αj\alpha_jαj 是模型自己学出来的------可能"学习"和"Python"权重很高，而"的"权重很低。

所以 [CLS] 相当于给模型提供了一个可学习的信息汇聚方式，比固定的平均池化更灵活。

💡 不过后续研究发现，原始 BERT 的 [CLS] 质量并不总是最优（因为 MLM 预训练没有专门优化句子级表示）。这就是为什么后来的 Sentence-BERT 等模型重新微调了 [CLS]，或者干脆用 Mean Pooling。

---

11.7 Mean Pooling

除了 [CLS]，另一种常用的方法是 Mean Pooling：

对所有 Token 的向量取平均，得到句子向量。

# [CLS] 方法
sentence_embedding = encoder_output[:, 0, :]  # 取第0个Token（[CLS]）

# Mean Pooling 方法
sentence_embedding = tf.reduce_mean(encoder_output, axis=1)  # 对seq维度取平均

研究表明，对于 Sentence Embedding 任务，Mean Pooling 通常比 [CLS] 效果更好。

方法	优点	缺点
CLS Token	简单，BERT 原生支持	需要专门训练
Mean Pooling	利用所有Token信息	忽略了Token的重要性差异
Weighted Mean	考虑重要性	需要额外计算权重

---

11.8 Sentence Embedding 的质量

原始 BERT 的 [CLS] 向量质量并不好，因为 BERT 的预训练任务（MLM）没有专门优化句子级别的表示。

改进方案：
#mermaid-svg-lAQpEcvpS1UG4EAl{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-lAQpEcvpS1UG4EAl .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-lAQpEcvpS1UG4EAl .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-lAQpEcvpS1UG4EAl .error-icon{fill:#552222;}#mermaid-svg-lAQpEcvpS1UG4EAl .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-lAQpEcvpS1UG4EAl .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-lAQpEcvpS1UG4EAl .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-lAQpEcvpS1UG4EAl .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-lAQpEcvpS1UG4EAl .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-lAQpEcvpS1UG4EAl .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-lAQpEcvpS1UG4EAl .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-lAQpEcvpS1UG4EAl .marker{fill:#333333;stroke:#333333;}#mermaid-svg-lAQpEcvpS1UG4EAl .marker.cross{stroke:#333333;}#mermaid-svg-lAQpEcvpS1UG4EAl svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-lAQpEcvpS1UG4EAl p{margin:0;}#mermaid-svg-lAQpEcvpS1UG4EAl .label{font-family:"trebuchet ms",verdana,arial,sans-serif;color:#333;}#mermaid-svg-lAQpEcvpS1UG4EAl .cluster-label text{fill:#333;}#mermaid-svg-lAQpEcvpS1UG4EAl .cluster-label span{color:#333;}#mermaid-svg-lAQpEcvpS1UG4EAl .cluster-label span p{background-color:transparent;}#mermaid-svg-lAQpEcvpS1UG4EAl .label text,#mermaid-svg-lAQpEcvpS1UG4EAl span{fill:#333;color:#333;}#mermaid-svg-lAQpEcvpS1UG4EAl .node rect,#mermaid-svg-lAQpEcvpS1UG4EAl .node circle,#mermaid-svg-lAQpEcvpS1UG4EAl .node ellipse,#mermaid-svg-lAQpEcvpS1UG4EAl .node polygon,#mermaid-svg-lAQpEcvpS1UG4EAl .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-lAQpEcvpS1UG4EAl .rough-node .label text,#mermaid-svg-lAQpEcvpS1UG4EAl .node .label text,#mermaid-svg-lAQpEcvpS1UG4EAl .image-shape .label,#mermaid-svg-lAQpEcvpS1UG4EAl .icon-shape .label{text-anchor:middle;}#mermaid-svg-lAQpEcvpS1UG4EAl .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-lAQpEcvpS1UG4EAl .rough-node .label,#mermaid-svg-lAQpEcvpS1UG4EAl .node .label,#mermaid-svg-lAQpEcvpS1UG4EAl .image-shape .label,#mermaid-svg-lAQpEcvpS1UG4EAl .icon-shape .label{text-align:center;}#mermaid-svg-lAQpEcvpS1UG4EAl .node.clickable{cursor:pointer;}#mermaid-svg-lAQpEcvpS1UG4EAl .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-lAQpEcvpS1UG4EAl .arrowheadPath{fill:#333333;}#mermaid-svg-lAQpEcvpS1UG4EAl .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-lAQpEcvpS1UG4EAl .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-lAQpEcvpS1UG4EAl .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-lAQpEcvpS1UG4EAl .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-lAQpEcvpS1UG4EAl .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-lAQpEcvpS1UG4EAl .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-lAQpEcvpS1UG4EAl .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-lAQpEcvpS1UG4EAl .cluster text{fill:#333;}#mermaid-svg-lAQpEcvpS1UG4EAl .cluster span{color:#333;}#mermaid-svg-lAQpEcvpS1UG4EAl div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-lAQpEcvpS1UG4EAl .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-lAQpEcvpS1UG4EAl rect.text{fill:none;stroke-width:0;}#mermaid-svg-lAQpEcvpS1UG4EAl .icon-shape,#mermaid-svg-lAQpEcvpS1UG4EAl .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-lAQpEcvpS1UG4EAl .icon-shape p,#mermaid-svg-lAQpEcvpS1UG4EAl .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-lAQpEcvpS1UG4EAl .icon-shape .label rect,#mermaid-svg-lAQpEcvpS1UG4EAl .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-lAQpEcvpS1UG4EAl .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-lAQpEcvpS1UG4EAl .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-lAQpEcvpS1UG4EAl :root{--mermaid-font-family:"trebuchet ms",verdana,arial,sans-serif;} BERT
Sentence-BERT

对比学习微调
高质量

Sentence Embedding
语义搜索
文本聚类
RAG检索

现代 Embedding Model（BGE、E5、GTE）都是在 BERT 基础上，通过对比学习（Contrastive Learning）微调得到的。

---

11.9 TensorFlow 实现

import tensorflow as tf

class BERTEncoder(tf.keras.Model):
    def __init__(self, vocab_size, d_model, num_heads, dff, num_layers, max_seq_len):
        super().__init__()
        self.token_embedding = tf.keras.layers.Embedding(vocab_size, d_model)
        self.position_embedding = tf.keras.layers.Embedding(max_seq_len, d_model)
        self.blocks = [
            TransformerBlock(d_model, num_heads, dff)
            for _ in range(num_layers)
        ]
        self.norm = tf.keras.layers.LayerNormalization(epsilon=1e-6)

    def call(self, token_ids, training=False):
        seq_len = tf.shape(token_ids)[1]
        positions = tf.range(seq_len)[tf.newaxis, :]

        x = self.token_embedding(token_ids) + self.position_embedding(positions)

        for block in self.blocks:
            x = block(x, training=training)

        x = self.norm(x)
        return x

    def get_cls_embedding(self, token_ids):
        output = self(token_ids)
        return output[:, 0, :]  # [CLS] Token

    def get_mean_pooling(self, token_ids):
        output = self(token_ids)
        return tf.reduce_mean(output, axis=1)  # Mean Pooling

---

11.10 Encoder 输出的 Shape

#mermaid-svg-sa3DPhMBwDHdpaON{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-sa3DPhMBwDHdpaON .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-sa3DPhMBwDHdpaON .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-sa3DPhMBwDHdpaON .error-icon{fill:#552222;}#mermaid-svg-sa3DPhMBwDHdpaON .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-sa3DPhMBwDHdpaON .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-sa3DPhMBwDHdpaON .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-sa3DPhMBwDHdpaON .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-sa3DPhMBwDHdpaON .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-sa3DPhMBwDHdpaON .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-sa3DPhMBwDHdpaON .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-sa3DPhMBwDHdpaON .marker{fill:#333333;stroke:#333333;}#mermaid-svg-sa3DPhMBwDHdpaON .marker.cross{stroke:#333333;}#mermaid-svg-sa3DPhMBwDHdpaON svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-sa3DPhMBwDHdpaON p{margin:0;}#mermaid-svg-sa3DPhMBwDHdpaON .label{font-family:"trebuchet ms",verdana,arial,sans-serif;color:#333;}#mermaid-svg-sa3DPhMBwDHdpaON .cluster-label text{fill:#333;}#mermaid-svg-sa3DPhMBwDHdpaON .cluster-label span{color:#333;}#mermaid-svg-sa3DPhMBwDHdpaON .cluster-label span p{background-color:transparent;}#mermaid-svg-sa3DPhMBwDHdpaON .label text,#mermaid-svg-sa3DPhMBwDHdpaON span{fill:#333;color:#333;}#mermaid-svg-sa3DPhMBwDHdpaON .node rect,#mermaid-svg-sa3DPhMBwDHdpaON .node circle,#mermaid-svg-sa3DPhMBwDHdpaON .node ellipse,#mermaid-svg-sa3DPhMBwDHdpaON .node polygon,#mermaid-svg-sa3DPhMBwDHdpaON .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-sa3DPhMBwDHdpaON .rough-node .label text,#mermaid-svg-sa3DPhMBwDHdpaON .node .label text,#mermaid-svg-sa3DPhMBwDHdpaON .image-shape .label,#mermaid-svg-sa3DPhMBwDHdpaON .icon-shape .label{text-anchor:middle;}#mermaid-svg-sa3DPhMBwDHdpaON .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-sa3DPhMBwDHdpaON .rough-node .label,#mermaid-svg-sa3DPhMBwDHdpaON .node .label,#mermaid-svg-sa3DPhMBwDHdpaON .image-shape .label,#mermaid-svg-sa3DPhMBwDHdpaON .icon-shape .label{text-align:center;}#mermaid-svg-sa3DPhMBwDHdpaON .node.clickable{cursor:pointer;}#mermaid-svg-sa3DPhMBwDHdpaON .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-sa3DPhMBwDHdpaON .arrowheadPath{fill:#333333;}#mermaid-svg-sa3DPhMBwDHdpaON .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-sa3DPhMBwDHdpaON .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-sa3DPhMBwDHdpaON .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-sa3DPhMBwDHdpaON .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-sa3DPhMBwDHdpaON .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-sa3DPhMBwDHdpaON .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-sa3DPhMBwDHdpaON .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-sa3DPhMBwDHdpaON .cluster text{fill:#333;}#mermaid-svg-sa3DPhMBwDHdpaON .cluster span{color:#333;}#mermaid-svg-sa3DPhMBwDHdpaON div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-sa3DPhMBwDHdpaON .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-sa3DPhMBwDHdpaON rect.text{fill:none;stroke-width:0;}#mermaid-svg-sa3DPhMBwDHdpaON .icon-shape,#mermaid-svg-sa3DPhMBwDHdpaON .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-sa3DPhMBwDHdpaON .icon-shape p,#mermaid-svg-sa3DPhMBwDHdpaON .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-sa3DPhMBwDHdpaON .icon-shape .label rect,#mermaid-svg-sa3DPhMBwDHdpaON .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-sa3DPhMBwDHdpaON .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-sa3DPhMBwDHdpaON .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-sa3DPhMBwDHdpaON :root{--mermaid-font-family:"trebuchet ms",verdana,arial,sans-serif;} 输入 Token IDs

batch, seq_len

Encoder
所有Token的上下文向量

batch, seq_len, d_model

取 CLS

batch, d_model

Mean Pooling

batch, d_model

用于序列标注

batch, seq_len, num_classes

---

本章总结

#mermaid-svg-LnzNMnJsSQaw9rs0{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-LnzNMnJsSQaw9rs0 .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-LnzNMnJsSQaw9rs0 .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-LnzNMnJsSQaw9rs0 .error-icon{fill:#552222;}#mermaid-svg-LnzNMnJsSQaw9rs0 .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-LnzNMnJsSQaw9rs0 .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-LnzNMnJsSQaw9rs0 .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-LnzNMnJsSQaw9rs0 .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-LnzNMnJsSQaw9rs0 .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-LnzNMnJsSQaw9rs0 .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-LnzNMnJsSQaw9rs0 .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-LnzNMnJsSQaw9rs0 .marker{fill:#333333;stroke:#333333;}#mermaid-svg-LnzNMnJsSQaw9rs0 .marker.cross{stroke:#333333;}#mermaid-svg-LnzNMnJsSQaw9rs0 svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-LnzNMnJsSQaw9rs0 p{margin:0;}#mermaid-svg-LnzNMnJsSQaw9rs0 .edge{stroke-width:3;}#mermaid-svg-LnzNMnJsSQaw9rs0 .section--1 rect,#mermaid-svg-LnzNMnJsSQaw9rs0 .section--1 path,#mermaid-svg-LnzNMnJsSQaw9rs0 .section--1 circle,#mermaid-svg-LnzNMnJsSQaw9rs0 .section--1 polygon,#mermaid-svg-LnzNMnJsSQaw9rs0 .section--1 path{fill:hsl(240, 100%, 76.2745098039%);}#mermaid-svg-LnzNMnJsSQaw9rs0 .section--1 text{fill:#ffffff;}#mermaid-svg-LnzNMnJsSQaw9rs0 .node-icon--1{font-size:40px;color:#ffffff;}#mermaid-svg-LnzNMnJsSQaw9rs0 .section-edge--1{stroke:hsl(240, 100%, 76.2745098039%);}#mermaid-svg-LnzNMnJsSQaw9rs0 .edge-depth--1{stroke-width:17;}#mermaid-svg-LnzNMnJsSQaw9rs0 .section--1 line{stroke:hsl(60, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-LnzNMnJsSQaw9rs0 .disabled,#mermaid-svg-LnzNMnJsSQaw9rs0 .disabled circle,#mermaid-svg-LnzNMnJsSQaw9rs0 .disabled text{fill:lightgray;}#mermaid-svg-LnzNMnJsSQaw9rs0 .disabled text{fill:#efefef;}#mermaid-svg-LnzNMnJsSQaw9rs0 .section-0 rect,#mermaid-svg-LnzNMnJsSQaw9rs0 .section-0 path,#mermaid-svg-LnzNMnJsSQaw9rs0 .section-0 circle,#mermaid-svg-LnzNMnJsSQaw9rs0 .section-0 polygon,#mermaid-svg-LnzNMnJsSQaw9rs0 .section-0 path{fill:hsl(60, 100%, 73.5294117647%);}#mermaid-svg-LnzNMnJsSQaw9rs0 .section-0 text{fill:black;}#mermaid-svg-LnzNMnJsSQaw9rs0 .node-icon-0{font-size:40px;color:black;}#mermaid-svg-LnzNMnJsSQaw9rs0 .section-edge-0{stroke:hsl(60, 100%, 73.5294117647%);}#mermaid-svg-LnzNMnJsSQaw9rs0 .edge-depth-0{stroke-width:14;}#mermaid-svg-LnzNMnJsSQaw9rs0 .section-0 line{stroke:hsl(240, 100%, 83.5294117647%);stroke-width:3;}#mermaid-svg-LnzNMnJsSQaw9rs0 .disabled,#mermaid-svg-LnzNMnJsSQaw9rs0 .disabled circle,#mermaid-svg-LnzNMnJsSQaw9rs0 .disabled text{fill:lightgray;}#mermaid-svg-LnzNMnJsSQaw9rs0 .disabled text{fill:#efefef;}#mermaid-svg-LnzNMnJsSQaw9rs0 .section-1 rect,#mermaid-svg-LnzNMnJsSQaw9rs0 .section-1 path,#mermaid-svg-LnzNMnJsSQaw9rs0 .section-1 circle,#mermaid-svg-LnzNMnJsSQaw9rs0 .section-1 polygon,#mermaid-svg-LnzNMnJsSQaw9rs0 .section-1 path{fill:hsl(80, 100%, 76.2745098039%);}#mermaid-svg-LnzNMnJsSQaw9rs0 .section-1 text{fill:black;}#mermaid-svg-LnzNMnJsSQaw9rs0 .node-icon-1{font-size:40px;color:black;}#mermaid-svg-LnzNMnJsSQaw9rs0 .section-edge-1{stroke:hsl(80, 100%, 76.2745098039%);}#mermaid-svg-LnzNMnJsSQaw9rs0 .edge-depth-1{stroke-width:11;}#mermaid-svg-LnzNMnJsSQaw9rs0 .section-1 line{stroke:hsl(260, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-LnzNMnJsSQaw9rs0 .disabled,#mermaid-svg-LnzNMnJsSQaw9rs0 .disabled circle,#mermaid-svg-LnzNMnJsSQaw9rs0 .disabled text{fill:lightgray;}#mermaid-svg-LnzNMnJsSQaw9rs0 .disabled text{fill:#efefef;}#mermaid-svg-LnzNMnJsSQaw9rs0 .section-2 rect,#mermaid-svg-LnzNMnJsSQaw9rs0 .section-2 path,#mermaid-svg-LnzNMnJsSQaw9rs0 .section-2 circle,#mermaid-svg-LnzNMnJsSQaw9rs0 .section-2 polygon,#mermaid-svg-LnzNMnJsSQaw9rs0 .section-2 path{fill:hsl(270, 100%, 76.2745098039%);}#mermaid-svg-LnzNMnJsSQaw9rs0 .section-2 text{fill:#ffffff;}#mermaid-svg-LnzNMnJsSQaw9rs0 .node-icon-2{font-size:40px;color:#ffffff;}#mermaid-svg-LnzNMnJsSQaw9rs0 .section-edge-2{stroke:hsl(270, 100%, 76.2745098039%);}#mermaid-svg-LnzNMnJsSQaw9rs0 .edge-depth-2{stroke-width:8;}#mermaid-svg-LnzNMnJsSQaw9rs0 .section-2 line{stroke:hsl(90, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-LnzNMnJsSQaw9rs0 .disabled,#mermaid-svg-LnzNMnJsSQaw9rs0 .disabled circle,#mermaid-svg-LnzNMnJsSQaw9rs0 .disabled text{fill:lightgray;}#mermaid-svg-LnzNMnJsSQaw9rs0 .disabled text{fill:#efefef;}#mermaid-svg-LnzNMnJsSQaw9rs0 .section-3 rect,#mermaid-svg-LnzNMnJsSQaw9rs0 .section-3 path,#mermaid-svg-LnzNMnJsSQaw9rs0 .section-3 circle,#mermaid-svg-LnzNMnJsSQaw9rs0 .section-3 polygon,#mermaid-svg-LnzNMnJsSQaw9rs0 .section-3 path{fill:hsl(300, 100%, 76.2745098039%);}#mermaid-svg-LnzNMnJsSQaw9rs0 .section-3 text{fill:black;}#mermaid-svg-LnzNMnJsSQaw9rs0 .node-icon-3{font-size:40px;color:black;}#mermaid-svg-LnzNMnJsSQaw9rs0 .section-edge-3{stroke:hsl(300, 100%, 76.2745098039%);}#mermaid-svg-LnzNMnJsSQaw9rs0 .edge-depth-3{stroke-width:5;}#mermaid-svg-LnzNMnJsSQaw9rs0 .section-3 line{stroke:hsl(120, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-LnzNMnJsSQaw9rs0 .disabled,#mermaid-svg-LnzNMnJsSQaw9rs0 .disabled circle,#mermaid-svg-LnzNMnJsSQaw9rs0 .disabled text{fill:lightgray;}#mermaid-svg-LnzNMnJsSQaw9rs0 .disabled text{fill:#efefef;}#mermaid-svg-LnzNMnJsSQaw9rs0 .section-4 rect,#mermaid-svg-LnzNMnJsSQaw9rs0 .section-4 path,#mermaid-svg-LnzNMnJsSQaw9rs0 .section-4 circle,#mermaid-svg-LnzNMnJsSQaw9rs0 .section-4 polygon,#mermaid-svg-LnzNMnJsSQaw9rs0 .section-4 path{fill:hsl(330, 100%, 76.2745098039%);}#mermaid-svg-LnzNMnJsSQaw9rs0 .section-4 text{fill:black;}#mermaid-svg-LnzNMnJsSQaw9rs0 .node-icon-4{font-size:40px;color:black;}#mermaid-svg-LnzNMnJsSQaw9rs0 .section-edge-4{stroke:hsl(330, 100%, 76.2745098039%);}#mermaid-svg-LnzNMnJsSQaw9rs0 .edge-depth-4{stroke-width:2;}#mermaid-svg-LnzNMnJsSQaw9rs0 .section-4 line{stroke:hsl(150, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-LnzNMnJsSQaw9rs0 .disabled,#mermaid-svg-LnzNMnJsSQaw9rs0 .disabled circle,#mermaid-svg-LnzNMnJsSQaw9rs0 .disabled text{fill:lightgray;}#mermaid-svg-LnzNMnJsSQaw9rs0 .disabled text{fill:#efefef;}#mermaid-svg-LnzNMnJsSQaw9rs0 .section-5 rect,#mermaid-svg-LnzNMnJsSQaw9rs0 .section-5 path,#mermaid-svg-LnzNMnJsSQaw9rs0 .section-5 circle,#mermaid-svg-LnzNMnJsSQaw9rs0 .section-5 polygon,#mermaid-svg-LnzNMnJsSQaw9rs0 .section-5 path{fill:hsl(0, 100%, 76.2745098039%);}#mermaid-svg-LnzNMnJsSQaw9rs0 .section-5 text{fill:black;}#mermaid-svg-LnzNMnJsSQaw9rs0 .node-icon-5{font-size:40px;color:black;}#mermaid-svg-LnzNMnJsSQaw9rs0 .section-edge-5{stroke:hsl(0, 100%, 76.2745098039%);}#mermaid-svg-LnzNMnJsSQaw9rs0 .edge-depth-5{stroke-width:-1;}#mermaid-svg-LnzNMnJsSQaw9rs0 .section-5 line{stroke:hsl(180, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-LnzNMnJsSQaw9rs0 .disabled,#mermaid-svg-LnzNMnJsSQaw9rs0 .disabled circle,#mermaid-svg-LnzNMnJsSQaw9rs0 .disabled text{fill:lightgray;}#mermaid-svg-LnzNMnJsSQaw9rs0 .disabled text{fill:#efefef;}#mermaid-svg-LnzNMnJsSQaw9rs0 .section-6 rect,#mermaid-svg-LnzNMnJsSQaw9rs0 .section-6 path,#mermaid-svg-LnzNMnJsSQaw9rs0 .section-6 circle,#mermaid-svg-LnzNMnJsSQaw9rs0 .section-6 polygon,#mermaid-svg-LnzNMnJsSQaw9rs0 .section-6 path{fill:hsl(30, 100%, 76.2745098039%);}#mermaid-svg-LnzNMnJsSQaw9rs0 .section-6 text{fill:black;}#mermaid-svg-LnzNMnJsSQaw9rs0 .node-icon-6{font-size:40px;color:black;}#mermaid-svg-LnzNMnJsSQaw9rs0 .section-edge-6{stroke:hsl(30, 100%, 76.2745098039%);}#mermaid-svg-LnzNMnJsSQaw9rs0 .edge-depth-6{stroke-width:-4;}#mermaid-svg-LnzNMnJsSQaw9rs0 .section-6 line{stroke:hsl(210, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-LnzNMnJsSQaw9rs0 .disabled,#mermaid-svg-LnzNMnJsSQaw9rs0 .disabled circle,#mermaid-svg-LnzNMnJsSQaw9rs0 .disabled text{fill:lightgray;}#mermaid-svg-LnzNMnJsSQaw9rs0 .disabled text{fill:#efefef;}#mermaid-svg-LnzNMnJsSQaw9rs0 .section-7 rect,#mermaid-svg-LnzNMnJsSQaw9rs0 .section-7 path,#mermaid-svg-LnzNMnJsSQaw9rs0 .section-7 circle,#mermaid-svg-LnzNMnJsSQaw9rs0 .section-7 polygon,#mermaid-svg-LnzNMnJsSQaw9rs0 .section-7 path{fill:hsl(90, 100%, 76.2745098039%);}#mermaid-svg-LnzNMnJsSQaw9rs0 .section-7 text{fill:black;}#mermaid-svg-LnzNMnJsSQaw9rs0 .node-icon-7{font-size:40px;color:black;}#mermaid-svg-LnzNMnJsSQaw9rs0 .section-edge-7{stroke:hsl(90, 100%, 76.2745098039%);}#mermaid-svg-LnzNMnJsSQaw9rs0 .edge-depth-7{stroke-width:-7;}#mermaid-svg-LnzNMnJsSQaw9rs0 .section-7 line{stroke:hsl(270, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-LnzNMnJsSQaw9rs0 .disabled,#mermaid-svg-LnzNMnJsSQaw9rs0 .disabled circle,#mermaid-svg-LnzNMnJsSQaw9rs0 .disabled text{fill:lightgray;}#mermaid-svg-LnzNMnJsSQaw9rs0 .disabled text{fill:#efefef;}#mermaid-svg-LnzNMnJsSQaw9rs0 .section-8 rect,#mermaid-svg-LnzNMnJsSQaw9rs0 .section-8 path,#mermaid-svg-LnzNMnJsSQaw9rs0 .section-8 circle,#mermaid-svg-LnzNMnJsSQaw9rs0 .section-8 polygon,#mermaid-svg-LnzNMnJsSQaw9rs0 .section-8 path{fill:hsl(150, 100%, 76.2745098039%);}#mermaid-svg-LnzNMnJsSQaw9rs0 .section-8 text{fill:black;}#mermaid-svg-LnzNMnJsSQaw9rs0 .node-icon-8{font-size:40px;color:black;}#mermaid-svg-LnzNMnJsSQaw9rs0 .section-edge-8{stroke:hsl(150, 100%, 76.2745098039%);}#mermaid-svg-LnzNMnJsSQaw9rs0 .edge-depth-8{stroke-width:-10;}#mermaid-svg-LnzNMnJsSQaw9rs0 .section-8 line{stroke:hsl(330, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-LnzNMnJsSQaw9rs0 .disabled,#mermaid-svg-LnzNMnJsSQaw9rs0 .disabled circle,#mermaid-svg-LnzNMnJsSQaw9rs0 .disabled text{fill:lightgray;}#mermaid-svg-LnzNMnJsSQaw9rs0 .disabled text{fill:#efefef;}#mermaid-svg-LnzNMnJsSQaw9rs0 .section-9 rect,#mermaid-svg-LnzNMnJsSQaw9rs0 .section-9 path,#mermaid-svg-LnzNMnJsSQaw9rs0 .section-9 circle,#mermaid-svg-LnzNMnJsSQaw9rs0 .section-9 polygon,#mermaid-svg-LnzNMnJsSQaw9rs0 .section-9 path{fill:hsl(180, 100%, 76.2745098039%);}#mermaid-svg-LnzNMnJsSQaw9rs0 .section-9 text{fill:black;}#mermaid-svg-LnzNMnJsSQaw9rs0 .node-icon-9{font-size:40px;color:black;}#mermaid-svg-LnzNMnJsSQaw9rs0 .section-edge-9{stroke:hsl(180, 100%, 76.2745098039%);}#mermaid-svg-LnzNMnJsSQaw9rs0 .edge-depth-9{stroke-width:-13;}#mermaid-svg-LnzNMnJsSQaw9rs0 .section-9 line{stroke:hsl(0, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-LnzNMnJsSQaw9rs0 .disabled,#mermaid-svg-LnzNMnJsSQaw9rs0 .disabled circle,#mermaid-svg-LnzNMnJsSQaw9rs0 .disabled text{fill:lightgray;}#mermaid-svg-LnzNMnJsSQaw9rs0 .disabled text{fill:#efefef;}#mermaid-svg-LnzNMnJsSQaw9rs0 .section-10 rect,#mermaid-svg-LnzNMnJsSQaw9rs0 .section-10 path,#mermaid-svg-LnzNMnJsSQaw9rs0 .section-10 circle,#mermaid-svg-LnzNMnJsSQaw9rs0 .section-10 polygon,#mermaid-svg-LnzNMnJsSQaw9rs0 .section-10 path{fill:hsl(210, 100%, 76.2745098039%);}#mermaid-svg-LnzNMnJsSQaw9rs0 .section-10 text{fill:black;}#mermaid-svg-LnzNMnJsSQaw9rs0 .node-icon-10{font-size:40px;color:black;}#mermaid-svg-LnzNMnJsSQaw9rs0 .section-edge-10{stroke:hsl(210, 100%, 76.2745098039%);}#mermaid-svg-LnzNMnJsSQaw9rs0 .edge-depth-10{stroke-width:-16;}#mermaid-svg-LnzNMnJsSQaw9rs0 .section-10 line{stroke:hsl(30, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-LnzNMnJsSQaw9rs0 .disabled,#mermaid-svg-LnzNMnJsSQaw9rs0 .disabled circle,#mermaid-svg-LnzNMnJsSQaw9rs0 .disabled text{fill:lightgray;}#mermaid-svg-LnzNMnJsSQaw9rs0 .disabled text{fill:#efefef;}#mermaid-svg-LnzNMnJsSQaw9rs0 .section-root rect,#mermaid-svg-LnzNMnJsSQaw9rs0 .section-root path,#mermaid-svg-LnzNMnJsSQaw9rs0 .section-root circle,#mermaid-svg-LnzNMnJsSQaw9rs0 .section-root polygon{fill:hsl(240, 100%, 46.2745098039%);}#mermaid-svg-LnzNMnJsSQaw9rs0 .section-root text{fill:#ffffff;}#mermaid-svg-LnzNMnJsSQaw9rs0 .section-root span{color:#ffffff;}#mermaid-svg-LnzNMnJsSQaw9rs0 .section-2 span{color:#ffffff;}#mermaid-svg-LnzNMnJsSQaw9rs0 .icon-container{height:100%;display:flex;justify-content:center;align-items:center;}#mermaid-svg-LnzNMnJsSQaw9rs0 .edge{fill:none;}#mermaid-svg-LnzNMnJsSQaw9rs0 .mindmap-node-label{dy:1em;alignment-baseline:middle;text-anchor:middle;dominant-baseline:middle;text-align:center;}#mermaid-svg-LnzNMnJsSQaw9rs0 :root{--mermaid-font-family:"trebuchet ms",verdana,arial,sans-serif;} Encoder
逐层理解
浅层: 语法
中层: 语义
深层: 抽象
CLS Token
聚合全句信息
用于分类/Embedding
Mean Pooling
平均所有Token
通常效果更好
上下文向量
同一个词不同语境
向量不同

---

本章思考题

1. 为什么 [CLS] Token 能聚合整个句子的信息？
2. Mean Pooling 和 [CLS] 各有什么适用场景？
3. 为什么原始 BERT 的 [CLS] 向量不适合直接做 Sentence Embedding？
4. 如果一个句子有 512 个 Token，Encoder 输出的 Shape 是什么？

---

下一章预告

下一章我们讲 BERT。

为什么 BERT 是双向的？MLM 和 NSP 是什么？为什么 BERT 适合做 Embedding？