【深度学习】序列生成模型(六):评价方法计算实例:计算ROUGE-N得分【理论到程序】

文章目录

  • [一、BLEU-N得分(Bilingual Evaluation Understudy)](#一、BLEU-N得分(Bilingual Evaluation Understudy))
  • [二、ROUGE-N得分(Recall-Oriented Understudy for Gisting Evaluation)](#二、ROUGE-N得分(Recall-Oriented Understudy for Gisting Evaluation))
    • [1. 定义](#1. 定义)
    • [2. 计算](#2. 计算)
    • [3. 程序](#3. 程序)

给定一个生成序列"The cat sat on the mat"和两个参考序列"The cat is on the mat""The bird sat on the bush"分别计算BLEU-N和ROUGE-N得分(N=1或N =2时).

  • 生成序列 x = the cat sat on the mat \mathbf{x}=\text{the cat sat on the mat} x=the cat sat on the mat
  • 参考序列
    • s ( 1 ) = the cat is on the mat \mathbf{s}^{(1)}=\text{the cat is on the mat} s(1)=the cat is on the mat
    • s ( 2 ) = the bird sat on the bush \mathbf{s}^{(2)}=\text{the bird sat on the bush} s(2)=the bird sat on the bush

一、BLEU-N得分(Bilingual Evaluation Understudy)

【深度学习】序列生成模型(五):评价方法计算实例:计算BLEU-N得分

二、ROUGE-N得分(Recall-Oriented Understudy for Gisting Evaluation)

1. 定义

设 x \mathbf{x} x 为从模型分布 p θ p_{\theta} pθ 中生成的一个候选序列, s ( 1 ) , ⋯ , s ( K ) \mathbf{s^{(1)}}, ⋯ , \mathbf{s^{(K)}} s(1),⋯,s(K) 为从真实数据分布中采样得到的一组参考序列, W \mathcal{W} W 为从参考序列中提取N元组合的集合,ROUGE-N算法的定义为:

ROUGE-N ( x ) = ∑ k = 1 K ∑ w ∈ W min ⁡ ( c w ( x ) , c w ( s ( k ) ) ) ∑ k = 1 K ∑ w ∈ W c w ( s ( k ) ) \text{ROUGE-N}(\mathbf{x}) = \frac{\sum_{k=1}^{K} \sum_{w \in \mathcal{W}} \min(c_w(\mathbf{x}), c_w(\mathbf{s}^{(k)}))}{\sum_{k=1}^{K} \sum_{w \in \mathcal{W}} c_w(\mathbf{s}^{(k))}} ROUGE-N(x)=∑k=1K∑w∈Wcw(s(k))∑k=1K∑w∈Wmin(cw(x),cw(s(k)))

其中 c w ( x ) c_w(\mathbf{x}) cw(x) 是N元组合 w w w 在生成序列 x \mathbf{x} x 中出现的次数, c w ( s ( k ) ) ) c_w(\mathbf{s}^{(k))}) cw(s(k))) 是N元组合 w w w 在参考序列 s ( k ) \mathbf{s}^{(k)} s(k) 中出现的次数。

2. 计算

N=1

  • 生成序列 x = the cat sat on the mat \mathbf{x}=\text{the cat sat on the mat} x=the cat sat on the mat
  • 参考序列
    • s ( 1 ) = the cat is on the mat \mathbf{s}^{(1)}=\text{the cat is on the mat} s(1)=the cat is on the mat
    • s ( 2 ) = the bird sat on the bush \mathbf{s}^{(2)}=\text{the bird sat on the bush} s(2)=the bird sat on the bush
  • W = the, cat, is, on, mat, bird, sat, bush \mathcal{W}=\text{ {the, cat, is, on, mat, bird, sat, bush }} W= the, cat, is, on, mat, bird, sat, bush
w w w c w ( x ) c_w(\mathbf{x}) cw(x) c w ( s ( 1 ) ) c_w(\mathbf{s^{(1)}}) cw(s(1)) c w ( s ( 2 ) ) c_w(\mathbf{s^{(2)}}) cw(s(2)) min ⁡ ( c w ( x ) , c w ( s ( 1 ) ) \min(c_w(\mathbf{x}), c_w(\mathbf{s}^{(1)}) min(cw(x),cw(s(1)) min ⁡ ( c w ( x ) , c w ( s ( 2 ) ) \min(c_w(\mathbf{x}), c_w(\mathbf{s}^{(2)}) min(cw(x),cw(s(2))
the 2 2 2 2 2
cat 1 1 0 1 0
is 0 1 0 0 0
on 1 1 1 1 1
mat 1 1 0 1 0
bird 0 0 1 0 0
sat 1 0 1 0 1
bush 0 0 1 0 0
  • 分子 ∑ k = 1 K ∑ w ∈ W min ⁡ ( c w ( x ) , c w ( s ( k ) ) ) \sum_{k=1}^{K} \sum_{w \in \mathcal{W}} \min(c_w(\mathbf{x}), c_w(\mathbf{s}^{(k)})) ∑k=1K∑w∈Wmin(cw(x),cw(s(k)))
    • ∑ w ∈ W min ⁡ ( c w ( x ) , c w ( s ( 1 ) ) = 2 + 1 + 0 + 1 + 1 + 0 + 0 + 0 = 5 \sum_{w \in \mathcal{W}} \min(c_w(\mathbf{x}), c_w(\mathbf{s}^{(1)})=2+1+0+1+1+0+0+0=5 ∑w∈Wmin(cw(x),cw(s(1))=2+1+0+1+1+0+0+0=5
    • ∑ w ∈ W min ⁡ ( c w ( x ) , c w ( s ( 2 ) ) = 2 + 0 + 0 + 1 + 0 + 0 + 1 + 0 = 4 \sum_{w \in \mathcal{W}} \min(c_w(\mathbf{x}), c_w(\mathbf{s}^{(2)})=2+0+0+1+0+0+1+0=4 ∑w∈Wmin(cw(x),cw(s(2))=2+0+0+1+0+0+1+0=4
    • ∑ k = 1 2 ∑ w ∈ W min ⁡ ( c w ( x ) , c w ( s ( k ) ) ) = ∑ w ∈ W min ⁡ ( c w ( x ) , c w ( s ( 1 ) ) ) + ∑ w ∈ W min ⁡ ( c w ( x ) , c w ( s ( 2 ) ) ) = 5 + 4 = 9 \sum_{k=1}^{2} \sum_{w \in \mathcal{W}} \min(c_w(\mathbf{x}), c_w(\mathbf{s}^{(k)}))=\sum_{w \in \mathcal{W}} \min(c_w(\mathbf{x}), c_w(\mathbf{s}^{(1)}))+\sum_{w \in \mathcal{W}} \min(c_w(\mathbf{x}), c_w(\mathbf{s}^{(2)}))=5+4=9 ∑k=12∑w∈Wmin(cw(x),cw(s(k)))=∑w∈Wmin(cw(x),cw(s(1)))+∑w∈Wmin(cw(x),cw(s(2)))=5+4=9
  • 分母 ∑ k = 1 K ∑ w ∈ W c w ( s ( k ) ) \sum_{k=1}^{K} \sum_{w \in \mathcal{W}} c_w(\mathbf{s}^{(k)}) ∑k=1K∑w∈Wcw(s(k))
    • ∑ w ∈ W c w ( s ( 1 ) ) = 6 \sum_{w \in \mathcal{W}} c_w(\mathbf{s}^{(1))}=6 ∑w∈Wcw(s(1))=6
    • ∑ w ∈ W c w ( s ( 2 ) ) = 6 \sum_{w \in \mathcal{W}} c_w(\mathbf{s}^{(2)})=6 ∑w∈Wcw(s(2))=6
    • ∑ k = 1 2 ∑ w ∈ W c w ( s ( k ) ) = ∑ w ∈ W c w ( s ( 1 ) ) + ∑ w ∈ W c w ( s ( 2 ) ) = 12 \sum_{k=1}^{2} \sum_{w \in \mathcal{W}} c_w(\mathbf{s}^{(k)})= \sum_{w \in \mathcal{W}} c_w(\mathbf{s}^{(1)})+ \sum_{w \in \mathcal{W}} c_w(\mathbf{s}^{(2)})=12 ∑k=12∑w∈Wcw(s(k))=∑w∈Wcw(s(1))+∑w∈Wcw(s(2))=12
  • ROUGE-N ( x ) = ∑ k = 1 K ∑ w ∈ W min ⁡ ( c w ( x ) , c w ( s ( k ) ) ) ∑ k = 1 K ∑ w ∈ W c w ( s ( k ) ) = 5 + 4 6 + 6 = 9 12 = 0.75 \text{ROUGE-N}(\mathbf{x}) = \frac{\sum_{k=1}^{K} \sum_{w \in \mathcal{W}} \min(c_w(\mathbf{x}), c_w(\mathbf{s}^{(k)}))}{\sum_{k=1}^{K} \sum_{w \in \mathcal{W}} c_w(\mathbf{s}^{(k))}}=\frac{5+4}{6+6}=\frac{9}{12}=0.75 ROUGE-N(x)=∑k=1K∑w∈Wcw(s(k))∑k=1K∑w∈Wmin(cw(x),cw(s(k)))=6+65+4=129=0.75

N=2

  • 生成序列 x = the cat sat on the mat \mathbf{x}=\text{the cat sat on the mat} x=the cat sat on the mat
  • 参考序列
    • s ( 1 ) = the cat is on the mat \mathbf{s}^{(1)}=\text{the cat is on the mat} s(1)=the cat is on the mat
    • s ( 2 ) = the bird sat on the bush \mathbf{s}^{(2)}=\text{the bird sat on the bush} s(2)=the bird sat on the bush
  • W = the cat, cat is, is on, on the, the mat, the bird, bird sat, sat on, the bush \mathcal{W}=\text{ {the cat, cat is, is on, on the, the mat, the bird, bird sat, sat on, the bush }} W= the cat, cat is, is on, on the, the mat, the bird, bird sat, sat on, the bush
w w w c w ( x ) c_w(\mathbf{x}) cw(x) c w ( s ( 1 ) ) c_w(\mathbf{s^{(1)}}) cw(s(1)) c w ( s ( 2 ) ) c_w(\mathbf{s^{(2)}}) cw(s(2)) min ⁡ ( c w ( x ) , c w ( s ( 1 ) ) \min(c_w(\mathbf{x}), c_w(\mathbf{s}^{(1)}) min(cw(x),cw(s(1)) min ⁡ ( c w ( x ) , c w ( s ( 2 ) ) \min(c_w(\mathbf{x}), c_w(\mathbf{s}^{(2)}) min(cw(x),cw(s(2))
the cat 1 1 0 1 0
cat is 0 1 0 0 0
is on 0 1 0 0 0
on the 1 1 1 1 1
the mat 1 1 0 0 0
the bird 0 0 1 0 0
bird sat 0 0 1 0 0
sat on 1 0 1 1 1
the bush 0 0 1 0 0
  • 分子 ∑ k = 1 K ∑ w ∈ W min ⁡ ( c w ( x ) , c w ( s ( k ) ) ) \sum_{k=1}^{K} \sum_{w \in \mathcal{W}} \min(c_w(\mathbf{x}), c_w(\mathbf{s}^{(k)})) ∑k=1K∑w∈Wmin(cw(x),cw(s(k)))
    • ∑ w ∈ W min ⁡ ( c w ( x ) , c w ( s ( 1 ) ) = 3 \sum_{w \in \mathcal{W}} \min(c_w(\mathbf{x}), c_w(\mathbf{s}^{(1)})=3 ∑w∈Wmin(cw(x),cw(s(1))=3
    • ∑ w ∈ W min ⁡ ( c w ( x ) , c w ( s ( 2 ) ) = 2 \sum_{w \in \mathcal{W}} \min(c_w(\mathbf{x}), c_w(\mathbf{s}^{(2)})=2 ∑w∈Wmin(cw(x),cw(s(2))=2
    • ∑ k = 1 2 ∑ w ∈ W min ⁡ ( c w ( x ) , c w ( s ( k ) ) ) = ∑ w ∈ W min ⁡ ( c w ( x ) , c w ( s ( 1 ) ) ) + ∑ w ∈ W min ⁡ ( c w ( x ) , c w ( s ( 2 ) ) ) = 3 + 2 = 5 \sum_{k=1}^{2} \sum_{w \in \mathcal{W}} \min(c_w(\mathbf{x}), c_w(\mathbf{s}^{(k)}))=\sum_{w \in \mathcal{W}} \min(c_w(\mathbf{x}), c_w(\mathbf{s}^{(1)}))+\sum_{w \in \mathcal{W}} \min(c_w(\mathbf{x}), c_w(\mathbf{s}^{(2)}))=3+2=5 ∑k=12∑w∈Wmin(cw(x),cw(s(k)))=∑w∈Wmin(cw(x),cw(s(1)))+∑w∈Wmin(cw(x),cw(s(2)))=3+2=5
  • 分母 ∑ k = 1 K ∑ w ∈ W c w ( s ( k ) ) \sum_{k=1}^{K} \sum_{w \in \mathcal{W}} c_w(\mathbf{s}^{(k)}) ∑k=1K∑w∈Wcw(s(k))
    • ∑ w ∈ W c w ( s ( 1 ) ) = 5 \sum_{w \in \mathcal{W}} c_w(\mathbf{s}^{(1))}=5 ∑w∈Wcw(s(1))=5
    • ∑ w ∈ W c w ( s ( 2 ) ) = 5 \sum_{w \in \mathcal{W}} c_w(\mathbf{s}^{(2)})=5 ∑w∈Wcw(s(2))=5
    • ∑ k = 1 2 ∑ w ∈ W c w ( s ( k ) ) = ∑ w ∈ W c w ( s ( 1 ) ) + ∑ w ∈ W c w ( s ( 2 ) ) = 10 \sum_{k=1}^{2} \sum_{w \in \mathcal{W}} c_w(\mathbf{s}^{(k)})= \sum_{w \in \mathcal{W}} c_w(\mathbf{s}^{(1)})+ \sum_{w \in \mathcal{W}} c_w(\mathbf{s}^{(2)})=10 ∑k=12∑w∈Wcw(s(k))=∑w∈Wcw(s(1))+∑w∈Wcw(s(2))=10
  • ROUGE-N ( x ) = ∑ k = 1 K ∑ w ∈ W min ⁡ ( c w ( x ) , c w ( s ( k ) ) ) ∑ k = 1 K ∑ w ∈ W c w ( s ( k ) ) = 3 + 2 5 + 5 = 5 10 = 0.5 \text{ROUGE-N}(\mathbf{x}) = \frac{\sum_{k=1}^{K} \sum_{w \in \mathcal{W}} \min(c_w(\mathbf{x}), c_w(\mathbf{s}^{(k)}))}{\sum_{k=1}^{K} \sum_{w \in \mathcal{W}} c_w(\mathbf{s}^{(k))}}=\frac{3+2}{5+5}=\frac{5}{10}=0.5 ROUGE-N(x)=∑k=1K∑w∈Wcw(s(k))∑k=1K∑w∈Wmin(cw(x),cw(s(k)))=5+53+2=105=0.5

3. 程序

python 复制代码
main_string = 'the cat sat on the mat'
string1 = 'the cat is on the mat'
string2 = 'the bird sat on the bush'

words = list(set(string1.split(' ')+string2.split(' ')))  # 去除重复元素

total_occurrences, matching_occurrences = 0, 0
for word in words:
    matching_occurrences += min(main_string.count(word), string1.count(word)) + min(main_string.count(word), string2.count(word))
    total_occurrences += string1.count(word) + string2.count(word)

print(matching_occurrences / total_occurrences)

bigrams = []
split1 = string1.split(' ')
for i in range(len(split1) - 1):
    bigrams.append(split1[i] + ' ' + split1[i + 1])

split2 = string2.split(' ')
for i in range(len(split2) - 1):
    bigrams.append(split2[i] + ' ' + split2[i + 1])

bigrams = list(set(bigrams))  # 去除重复元素

total_occurrences, matching_occurrences = 0, 0
for bigram in bigrams:
    matching_occurrences += min(main_string.count(bigram), string1.count(bigram)) + min(main_string.count(bigram), string2.count(bigram))
    total_occurrences += string1.count(bigram) + string2.count(bigram)

print(matching_occurrences / total_occurrences)

输出:

bash 复制代码
0.75
0.5
相关推荐
GIOTTO情10 小时前
媒介宣发的技术革命:Infoseek如何用AI重构企业传播全链路
大数据·人工智能·重构
阿里云大数据AI技术10 小时前
云栖实录 | 从多模态数据到 Physical AI,PAI 助力客户快速启动 Physical AI 实践
人工智能
小关会打代码10 小时前
计算机视觉进阶教学之颜色识别
人工智能·计算机视觉
IT小哥哥呀10 小时前
基于深度学习的数字图像分类实验与分析
人工智能·深度学习·分类
机器之心11 小时前
VAE时代终结?谢赛宁团队「RAE」登场,表征自编码器或成DiT训练新基石
人工智能·openai
机器之心11 小时前
Sutton判定「LLM是死胡同」后,新访谈揭示AI困境
人工智能·openai
大模型真好玩11 小时前
低代码Agent开发框架使用指南(四)—Coze大模型和插件参数配置最佳实践
人工智能·agent·coze
jerryinwuhan11 小时前
基于大语言模型(LLM)的城市时间、空间与情感交织分析:面向智能城市的情感动态预测与空间优化
人工智能·语言模型·自然语言处理
落雪财神意11 小时前
股指10月想法
大数据·人工智能·金融·区块链·期股
中杯可乐多加冰11 小时前
无代码开发实践|基于业务流能力快速开发市场监管系统,实现投诉处理快速响应
人工智能·低代码