吴恩达深度学习课程五：自然语言处理 第三周：序列模型与注意力机制 课后习题与代码实践

此分类用于记录吴恩达深度学习课程的学习笔记。

课程相关信息链接如下：

1. 原课程视频链接：[双语字幕]吴恩达深度学习deeplearning.ai
2. github课程资料，含课件与笔记:吴恩达深度学习教学资料
3. 课程配套练习（中英）与答案：吴恩达深度学习课后习题与答案

本篇为第五课第三周的课后习题和代码实践部分。

---

1.理论习题

【中英】【吴恩达课后测验】Course 5 - 序列模型 - 第三周测验

在这一周的习题内容中有一道题的相关内容需要展开，先看一下题面：

网络通过学习注意力分数来决定「把注意力放在哪里」，这些值是由一个小神经网络计算得到的。

这里，我们不能 把 \(s^{<t-1>}\) 替换成 \(s^{<t>}\) 作为这个神经网络的输入，

原因如下：因为 \(s^{<t>}\) 依赖于当前的注意力权重，而当前的注意力权重又依赖于注意力分数，因此在需要计算注意力分数的时刻，\(s^{<t>}\) 尚未被计算出来 ， 此时只能使用前一时刻的隐藏状态 \(s^{<t-1>}\)，而无法使用当前的 \(s^{<t>}\)。
答案：正确

这道题一眼看下来是很迷惑的，这是因为这道题讲的是最早的注意力机制的逻辑，和我们在理论部分介绍的主流逻辑有所不同 。

在理论部分，我们介绍的注意力机制，学习注意力分数的网络的输入就是 \(s^{<t>}\) ，但在最早的注意力机制中，它的输入是 \(s^{<t-1>}\)。

其传播逻辑可以用一句话总结：在当前时间步中，原始注意力机制将由上一时刻解码状态计算得到的上下文向量 \(c^{<t>}\) 作为解码器输入的一部分，用于计算当前解码状态 \(s^{<t>}\)。

我们展开如下：

14 年的原论文使用的就是这种传播方式，具有开创性价值。但在现代，这种将上下文向量直接纳入状态递推的做法更多只具有历史意义，了解即可。

2.代码实践

【中文】【吴恩达课后编程作业】Course 5 - 序列模型 - 第三周作业

还是先摆链接，这篇里博主就机器翻译和触发词检测两部分内容，非常详细地演示了本周的内容，但还是要注意 Keras 的导包更新问题。

我们同样使用框架来演示这部分内容，主要内容如下：

1. 使用编码解码框架进行日期格式翻译
2. 在编码解码框架下对比贪心解码和束搜索性能
3. 使用带注意力机制的编码解码框架进行日期格式翻译

2.1 数据准备

需要说明的是，由于 seq2seq 模型自身的编码--解码结构 以及 序列数据的时序特性 ，这类任务相比许多常规监督学习任务，通常对计算资源有更高的要求 。

以机器翻译任务为例，网络上存在大量公开数据集，如英法翻译、中英翻译等。由于这类数据获取相对容易，同时自然语言本身具有词汇规模大、表达形式多样 的特点，模型在训练过程中往往需要同时维护源语言词典和目标语言词典 ，其规模通常至少达到 数万级别 。这直接导致模型在参数规模、计算量以及显存占用等方面的成本显著提升。

因此，这次我们选用一种相对简化的机器翻译任务------日期格式翻译 ，该类任务的主要特点在于词典规模小、输入输出结构清晰、语义歧义极少，可以较高效率地演示本周内容。

这次我们不使用网上公开的数据集，而是拓展吴恩达老师在编程作业里的逻辑，进行人工合成数据 。

我们通过设计好的脚本，生成 10000 条 以下格式的相关数据和对应标签并保存为文件：

HUMAN_TEMPLATES = [  
    "{day} {month} {year}",  
    "{month} {day}, {year}",  
    "{day} of {month}, {year}",  
    "{month} {day} {year}",  
    "{day}/{month_num}/{year_short}",  
    "{month_num}/{day}/{year_short}",  
    "{weekday}, {day} {month_abbr} {year}",  
    "{month_abbr} {day}th {year_short}",  
    "{year}-{month_num}-{day}",  
    "The {day}{day_suffix} of {month}, {year}",  
    "{day}th {month_abbr}, {year}",  
    "{month} the {day}{day_suffix}, {year}",  
    "Date: {year}/{month_num}/{day}",  
    "{day} in Roman: {roman_day} {month} {year}",  
    "{month_abbr}. {day}, '{year_short}",  
]

完整代码放在附录，打印几条生成数据如下：

[1] The 28th of April, 1975  ==>  1975-04-28
[2] Date: 2017/05/5  ==>  2017-05-05
[3] Jan 10th 44  ==>  1944-01-10
[4] Tuesday, 6 Nov 2096  ==>  2096-11-06
[5] The 26th of August, 2089  ==>  2089-08-26
[6] Date: 2061/08/18  ==>  2061-08-18
[7] 08/4/35  ==>  1935-08-04
[8] 27 of May, 1948  ==>  1948-05-27
[9] 8 of April, 1907  ==>  1907-04-08
[10] November 11, 1937  ==>  1937-11-11

准备好数据后，我们便可以进行下一步内容。

2.2 模型设计

在本周第一篇内容里我们就提到过：编码解码框架下的网络结构实际上是分别设计的两个子网络，在代码逻辑中，就是先分别定义设计编码器和解码器，再在整体网络中调用。我们分点展开如下：

（1）编码器

class Encoder(nn.Module):
    def __init__(self, vocab_size, emb_dim, hidden_dim):
        super().__init__()
        # 嵌入层
        self.embedding = nn.Embedding(vocab_size, emb_dim)
        # 使用双向 LSTM
        self.lstm = nn.LSTM(emb_dim, hidden_dim, batch_first=True, bidirectional=True)
    def forward(self, x):
        embedded = self.embedding(x) 
        outputs, (hidden, cell) = self.lstm(embedded)
        # outputs: LSTM 对每个时间步的输出,用于注意力的相关计算。
        # (hidden,cell): 最后时间步的隐藏状态和细胞状态，是解码器的初始输入。
        return outputs, (hidden, cell)

（2）解码器

class Decoder(nn.Module):
    def __init__(self, vocab_size, emb_dim, hidden_dim, use_attention=False):
        super().__init__()
        # 设置参数，默认不使用注意力机制
        self.use_attention = use_attention
        # 嵌入层
        self.embedding = nn.Embedding(vocab_size, emb_dim)
        # LSTM 
        self.lstm = nn.LSTM(emb_dim, hidden_dim, batch_first=True)
        if use_attention:
            # 注意力评分层，将解码器 hidden 与编码器 outputs 拼接计算注意力分数
            self.attn = nn.Linear(hidden_dim + hidden_dim * 2, 1)
            # 注意力结合层：将解码器 hidden 与上下文向量融合为最终 LSTM 输出
            self.attn_combine = nn.Linear(hidden_dim + hidden_dim * 2, hidden_dim)

        # 输出层：将 LSTM 输出映射到词表维度，预测下一个 token
        self.out = nn.Linear(hidden_dim, vocab_size)

    def forward(self, input_step, hidden_cell, encoder_outputs):
        embedded = self.embedding(input_step)
        # LSTM 前向计算，得到当前时间步输出和新的 hidden/cell
        lstm_out, hidden_cell = self.lstm(embedded, hidden_cell)
        # 去掉时间维度，方便注意力计算
        query = lstm_out.squeeze(1)

        if self.use_attention:
            # 扩展 query，实际上就是把当前解码隐藏状态复制到和编码隐藏状态数相同。
            query_exp = query.unsqueeze(1).repeat(1, encoder_outputs.size(1), 1)
            # 计算注意力分数
            attn_scores = self.attn(torch.cat((query_exp, encoder_outputs), dim=-1)).squeeze(-1)
            # 转为注意力权重
            attn_weights = F.softmax(attn_scores, dim=-1).unsqueeze(1)
            # 计算上下文向量 bmm:批量矩阵乘法
            context = torch.bmm(attn_weights, encoder_outputs).squeeze(1)
            # 将 query 与上下文向量融合
            combined = torch.cat((query, context), dim=1)
            output = torch.tanh(self.attn_combine(combined))
        else:
            # 不使用注意力时，直接用 LSTM 输出预测
            output = query

        # 输出层
        output = self.out(output)
        return output, hidden_cell

（3）整体网络

class Seq2Seq(nn.Module):  
    def __init__(self, encoder, decoder):  
        super().__init__()  
        # 编码器和解码器直接作为初始化参数，在主函数中创建传入
        self.encoder = encoder  
        self.decoder = decoder 
         
    def forward(self, src, tgt):  
        # src:样本 tgt:标签
        # 编码器
        enc_outputs, (hidden, cell) = self.encoder(src)

        # 这里是合并双向 LSTM 的 hidden/cell
        # 将两个方向的 hidden/cell 在层维度上分开，再求和或拼接作为解码器初始状态
        hidden = hidden.view(2, -1, hidden.size(-1)).sum(dim=0, keepdim=True)
        cell   = cell.view(2, -1, cell.size(-1)).sum(dim=0, keepdim=True)
        dec_hidden = (hidden, cell)  # 解码器初始状态

        outputs = []  # 存储每个时间步的输出

        # 解码器
        for t in range(tgt.size(1)):
            # 训练逻辑：每步使用真实标签作为输入
            out, dec_hidden = self.decoder(
                tgt[:, t].unsqueeze(1), dec_hidden, enc_outputs
            )
            # tgt[:, t] 因为批次训练，每次要去取第 t 列
            # out: 当前时间步预测的 token 概率分布 [batch_size, vocab_size]
            # dec_hidden: 更新后的解码器隐藏状态
            outputs.append(out)  # 保存当前输出

        # 将所有时间步输出堆叠为最终结果。
        return torch.stack(outputs, dim=1)

这样我们就完成了模型的设计，继续下一部分内容。

2.3 解码函数

还是理论部分提到的，在推理阶段，解码器的输出其实是一系列的概率分布，我们使用解码函数，就是设计相应的搜索策略来寻找最大的联合概率，得到最终输出。

这里，我们定义完整的解码函数，主逻辑为贪心解码，补充束搜索的相关逻辑并通过参数选择是否使用：

def decode(model, input_tensor, use_beam_search=False, beam_width=3, max_len=20):
    """
    参数：
    - model: 已训练的 Seq2Seq 模型（包含 encoder 和 decoder）
    - input_tensor: 输入序列的 tensor（一个样本）
    - use_beam_search: 是否使用束搜索
    - beam_width: 束搜索时的候选数量
    - max_len: 解码最大长度
    """
    model.eval()  # 评估模式
    with torch.no_grad():  
        input_tensor = input_tensor.unsqueeze(0) 
        # 编码器传播
        enc_outputs, (hidden, cell) = model.encoder(input_tensor)
        hidden = hidden.view(2, -1, hidden.size(-1)).sum(dim=0, keepdim=True)
        cell   = cell.view(2, -1, cell.size(-1)).sum(dim=0, keepdim=True)
        dec_hidden = (hidden, cell)  # 解码器初始状态
        
        # 贪心解码
        if not use_beam_search:
            # 初始解码输入：SOS：初始符，这里是简化为了 \t
            dec_input = torch.tensor([[target_token_index['\t']]], device=device)
            result = []
            for _ in range(max_len):
                # 解码器传播
                out, dec_hidden = model.decoder(dec_input, dec_hidden, enc_outputs)
                # 取当前时间步最大概率的 token
                pred = out.argmax(dim=-1)
                token = pred.item()
                if token == target_token_index['\n']:  # 遇到 EOS 停止，同样简化为\n
                    break
                result.append(reverse_target_char_index[token])  # 保存字符
                dec_input = pred.unsqueeze(0)  # 下一步输入为当前预测，即自回归。

            return ''.join(result)  # 返回解码后的字符串

       
        # 束搜索
        candidates = [([target_token_index['\t']], 0.0, dec_hidden)]  # 初始化候选序列
        for _ in range(max_len):
            new_cands = []
            for seq, score, h in candidates:
                # 如果序列已遇到 EOS，保留候选
                if seq[-1] == target_token_index['\n']:
                    new_cands.append((seq, score, h))
                    continue
                # 当前时间步输入
                inp = torch.tensor([[seq[-1]]], device=device)
                out, new_h = model.decoder(inp, h, enc_outputs)
                # 计算 log 概率
                log_probs = F.log_softmax(out, dim=-1).squeeze(0)
                # 取 top-k 候选
                top_vals, top_idx = log_probs.topk(beam_width)
                for v, idx in zip(top_vals, top_idx):
                    new_seq = seq + [idx.item()]  # 扩展序列
                    new_score = score + v.item()  # 更新累积 log 概率
                    new_cands.append((new_seq, new_score, new_h))
            # 按累积概率排序，保留前 beam_width 个序列
            candidates = sorted(new_cands, key=lambda x: x[1], reverse=True)[:beam_width]
        # 最终选择概率最高的序列
        best = candidates[0][0]
        # 去掉控制符，转换为最终字符
        return ''.join(reverse_target_char_index.get(i, '') 
                       for i in best[1:] if i != target_token_index['\n'])

解码的逻辑也被使用在之后的评估环节中。

2.4 评估指标：BELU 和完全匹配率

自然，在完成训练后，我们要有相应的指标来进行评估，首先就是我们在理论部分介绍的 BELU ，它通过计算输出和标签在不同尺度上的匹配度来评估模型效果。

但是，我们说：BELU 这类语言指标使用的原因是因为存在"同义不同形"的情况。

而在日期翻译里，翻译结果是唯一的：错一个数字就是完全不同的一天。

因此，我们再手工定义一个完全匹配率：输出和标签只有相同和不相同两种情况。

这样，我们使用 BELU 来评估序列模型拟合的大致效果，再用完全匹配率来观察在日期翻译任务中模型的真正性能。

from nltk.translate.bleu_score import sentence_bleu, SmoothingFunction
# BLEU
def evaluate_bleu(model, test_enc_input, test_targets, use_beam_search=False, beam_width=3):
    total_bleu = 0
    # BLEU 平滑方法，避免短句导致分数过低
    smoother = SmoothingFunction().method4

    for i in range(len(test_enc_input)):
        # 使用 decode 函数生成预测序列
        pred = decode(model, test_enc_input[i], use_beam_search, beam_width)
        # 去掉目标序列的起始符和结束符
        true = test_targets[i][1:-1]
        # BLEU 接受列表形式：ref 是参考序列列表，hyp 是预测序列
        ref = [list(true)]
        hyp = list(pred)
        # 计算当前样本 BLEU 分数
        bleu = sentence_bleu(ref, hyp, smoothing_function=smoother)
        total_bleu += bleu

    # 平均 BLEU 分数，并转换为百分比
    avg_bleu = total_bleu / len(test_enc_input) * 100
    print(f"BLEU: {avg_bleu:.2f}")
    return avg_bleu

# 完全匹配率
def evaluate_exact_match(model, test_enc_input, test_targets, use_beam_search=False, beam_width=3):
    correct = 0
    total = len(test_enc_input)
    for i in range(total):
        pred = decode(model, test_enc_input[i], use_beam_search, beam_width)
        true = test_targets[i][1:-1]
        # 判断预测与真实是否完全一致
        if pred == true:
            correct += 1
    acc = correct / total * 100
    print(f"完全匹配率: {acc:.2f}%")
    return acc

这里分开写两个指标方便演示和单独调用，实际上，把二者的前半部分逻辑合在一起是更高效的做法。

到此，需要强调的部分就全部定义完毕，我们开始正式运行。

2.5 运行效果

我们设置参数如下：

EMB_DIM = 64  
HIDDEN_DIM = 256  
EPOCHS = 10  
BATCH_SIZE = 128  
LR = 0.001

下面，就来分情况看看效果：

（1）不使用注意力机制，使用贪心解码

在这个条件下，我们定义主函数如下：

if __name__ == "__main__":  
	EMB_DIM = 64  
	HIDDEN_DIM = 256  
	EPOCHS = 10  
	BATCH_SIZE = 128  
	LR = 0.001
	# 关键设置
    USE_ATTENTION    = False  
    USE_BEAM_SEARCH  = False  
    BEAM_WIDTH       = 3  
  
    print(f"\n=== 配置 ===")  
    print(f"使用注意力: {USE_ATTENTION}")  
    print(f"使用束搜索: {USE_BEAM_SEARCH} (beam width = {BEAM_WIDTH if USE_BEAM_SEARCH else 'No'})")  
  
    encoder = Encoder(num_encoder_tokens, EMB_DIM, HIDDEN_DIM).to(device)  
    decoder = Decoder(num_decoder_tokens, EMB_DIM, HIDDEN_DIM, use_attention=USE_ATTENTION).to(device)  
    model = Seq2Seq(encoder, decoder).to(device)  
  
    print("\n开始训练...")  
    train_time = train(model, epochs=EPOCHS, batch_size=BATCH_SIZE, lr=LR)  
  
    print("\n评估中...")  
    bleu = evaluate_bleu(model, test_encoder_input, test_targets,  
                         use_beam_search=USE_BEAM_SEARCH, beam_width=BEAM_WIDTH)  
    exact_acc = evaluate_exact_match(model, test_encoder_input, test_targets,  
                                     use_beam_search=USE_BEAM_SEARCH, beam_width=BEAM_WIDTH)  
  
	print("\n=== 抽取测试样本输出 ===")  
	sample_indices = random.sample(range(len(test_encoder_input)), 10)  
	for i in sample_indices:  
	    src = test_inputs[i]  
	    true = test_targets[i][1:-1]  
	    pred,_ = decode(model, test_encoder_input[i],  
	                  use_beam_search=USE_BEAM_SEARCH, beam_width=BEAM_WIDTH)  
	    print(f"Human  : {src} | True  : {true} | Predicted  : {pred}")

结果如下：

得益于任务本身和合成数据较为简单，可以发现指标还不错，我们不着急分析 ，继续下一步设置进行对比。

（2）不使用注意力机制，使用束搜索

这次，设置主函数参数如下：

USE_ATTENTION    = False  
USE_BEAM_SEARCH  = True # 使用束搜索 
BEAM_WIDTH       = 3 # 束宽

看看结果：

指标好像有所提升，但也可能是训练中的偶然性，我们再增加束宽看看：

BEAM_WIDTH       = 10 

可以发现，随着束宽的增加，解码时间明显增加，而指标浮动不大 ，这说明限制指标的关键因素可能不是搜索策略。

由此，我们进入下一步：

（3）使用注意力机制

现在，设置参数如下：

USE_ATTENTION    = True # 使用注意力机制  
USE_BEAM_SEARCH  = True  
BEAM_WIDTH       = 3 

看看结果：

好像也没什么用啊？ 你会发现：使用注意力机制后，训练时间因相关计算明显增加，但是指标并没有明显提升。

实际上，这和我们的日期翻译任务本身的特点有关：

1. 输入输出序列较短：每个样本通常只有 5~15 个字符左右，LSTM 很容易在没有注意力的情况下就捕捉到序列整体信息。
2. 信息对齐较简单：日期翻译本质上是字符级的对齐映射，例如 "Jan 5, 2003" → "2003-01-05"，几乎不存在复杂的长距离依赖。

注意力机制最明显的作用是在序列较长或输入输出关系复杂时，帮助解码器聚焦到相关的编码状态。而对于现在这种短序列、规则性强的任务，其计算成本增加，而带来的性能提升很有限。

在这个任务中，想要指标继续增加，有一个很简单的方法就是在脚本中增加我们人工合成的数据量，就不再展示了。

至此，吴恩达深度学习课程五课十五周的内容就全部结束了，之后会出一个完整的目录。

3. 附录

3.1 数据合成

import torch
import random
import numpy as np
from datetime import datetime, timedelta
from sklearn.model_selection import train_test_split

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
PAD_TOKEN = "<PAD>"

MONTHS_FULL = [
    "January", "February", "March", "April", "May", "June",
    "July", "August", "September", "October", "November", "December"
]
MONTHS_ABBR = ["Jan", "Feb", "Mar", "Apr", "May", "Jun",
               "Jul", "Aug", "Sep", "Oct", "Nov", "Dec"]

HUMAN_TEMPLATES = [
    "{day} {month} {year}",
    "{month} {day}, {year}",
    "{day} of {month}, {year}",
    "{month} {day} {year}",
    "{day}/{month_num}/{year_short}",
    "{month_num}/{day}/{year_short}",
    "{weekday}, {day} {month_abbr} {year}",
    "{month_abbr} {day}th {year_short}",
    "{year}-{month_num}-{day}",
    "The {day}{day_suffix} of {month}, {year}",
    "{day}th {month_abbr}, {year}",
    "{month} the {day}{day_suffix}, {year}",
    "Date: {year}/{month_num}/{day}",
    "{day} in Roman: {roman_day} {month} {year}",
    "{month_abbr}. {day}, '{year_short}",
]

def roman_numeral(n):
    vals = (10, 9, 5, 4, 1)
    nums = ('X', 'IX', 'V', 'IV', 'I')
    result = ""
    for v, r in zip(vals, nums):
        while n >= v:
            result += r
            n -= v
    return result

def random_date(start_year=1900, end_year=2100):
    start = datetime(start_year, 1, 1)
    end = datetime(end_year, 12, 31)
    delta = end - start
    return start + timedelta(days=random.randint(0, delta.days))

def human_readable_date(date):
    template = random.choice(HUMAN_TEMPLATES)
    day = date.day
    month_idx = date.month - 1
    month_full = MONTHS_FULL[month_idx]
    month_abbr = MONTHS_ABBR[month_idx]
    year = date.year
    month_num = str(date.month).zfill(2)
    year_short = str(year % 100).zfill(2)
    weekday = date.strftime("%A")

    day_suffix = "th"
    if day % 10 == 1 and day != 11: day_suffix = "st"
    elif day % 10 == 2 and day != 12: day_suffix = "nd"
    elif day % 10 == 3 and day != 13: day_suffix = "rd"

    day_str = str(day)
    roman_day = roman_numeral(day) if random.random() < 0.2 else day_str

    return template.format(
        day=day_str,
        day_suffix=day_suffix,
        month=month_full,
        month_abbr=month_abbr,
        year=year,
        month_num=month_num,
        year_short=year_short,
        weekday=weekday,
        roman_day=roman_day
    )

def machine_readable_date(date):
    return date.strftime("%Y-%m-%d")

def generate_date_pair():
    d = random_date()
    return human_readable_date(d), machine_readable_date(d)


NUM_SAMPLES = 10000
input_texts, target_texts = [], []

for _ in range(NUM_SAMPLES):
    human, machine = generate_date_pair()
    input_texts.append(human)
    target_texts.append('\t' + machine + '\n')

train_inputs, test_inputs, train_targets, test_targets = train_test_split(
    input_texts, target_texts, test_size=0.2, random_state=42
)

all_input_chars = [PAD_TOKEN] + sorted(set(''.join(input_texts)))
all_target_chars = [PAD_TOKEN] + sorted(set(''.join(target_texts)))

input_token_index = {c: i for i, c in enumerate(all_input_chars)}
target_token_index = {c: i for i, c in enumerate(all_target_chars)}

reverse_input_char_index = {i: c for c, i in input_token_index.items()}
reverse_target_char_index = {i: c for c, i in target_token_index.items()}

max_encoder_len = max(len(txt) for txt in input_texts)
max_decoder_len = max(len(txt) for txt in target_texts)

num_encoder_tokens = len(all_input_chars)
num_decoder_tokens = len(all_target_chars)

print(f"max enc len: {max_encoder_len}, max dec len: {max_decoder_len}")
print(f"input vocab size: {num_encoder_tokens}, target vocab size: {num_decoder_tokens}")

def texts_to_tensor(texts, token_index, max_len):
    pad_idx = token_index[PAD_TOKEN]
    data = np.full((len(texts), max_len), pad_idx, dtype=np.int64)
    for i, txt in enumerate(texts):
        for t, char in enumerate(txt):
            if t < max_len:
                data[i, t] = token_index[char]
    return torch.tensor(data, dtype=torch.long, device=device)

train_encoder_input = texts_to_tensor(train_inputs, input_token_index, max_encoder_len)
train_decoder_input = texts_to_tensor(train_targets, target_token_index, max_decoder_len)

test_encoder_input = texts_to_tensor(test_inputs, input_token_index, max_encoder_len)
test_decoder_input = texts_to_tensor(test_targets, target_token_index, max_decoder_len)


dataset_pt = {
    'train_encoder_input': train_encoder_input,
    'train_decoder_input': train_decoder_input,
    'test_encoder_input': test_encoder_input,
    'test_decoder_input': test_decoder_input,
    'input_token_index': input_token_index,
    'target_token_index': target_token_index,
    'reverse_input_char_index': reverse_input_char_index,
    'reverse_target_char_index': reverse_target_char_index,
    'max_encoder_len': max_encoder_len,
    'max_decoder_len': max_decoder_len,
    'num_encoder_tokens': num_encoder_tokens,
    'num_decoder_tokens': num_decoder_tokens
}

torch.save(dataset_pt, "date_dataset.pt")
print("已保存：date_dataset.pt")



def tensor_to_numpy(t):
    return t.detach().cpu().numpy()

dataset_npz = {
    "train_encoder_input": tensor_to_numpy(train_encoder_input),
    "train_decoder_input": tensor_to_numpy(train_decoder_input),
    "test_encoder_input": tensor_to_numpy(test_encoder_input),
    "test_decoder_input": tensor_to_numpy(test_decoder_input),

    "input_token_index": np.array(input_token_index, dtype=object),
    "target_token_index": np.array(target_token_index, dtype=object),
    "reverse_input_char_index": np.array(reverse_input_char_index, dtype=object),
    "reverse_target_char_index": np.array(reverse_target_char_index, dtype=object),

    "max_encoder_len": max_encoder_len,
    "max_decoder_len": max_decoder_len,
    "num_encoder_tokens": num_encoder_tokens,
    "num_decoder_tokens": num_decoder_tokens
}

np.savez("date_dataset.npz", **dataset_npz)
print("已保存：date_dataset.npz\n")

def decode(tensor_row, reverse_dict):
    return ''.join(
        reverse_dict[idx.item()]
        for idx in tensor_row
        if reverse_dict[idx.item()] != PAD_TOKEN
    )

print("示例数据：")
for i in range(20):
    src = decode(train_encoder_input[i], reverse_input_char_index)
    tgt = decode(train_decoder_input[i], reverse_target_char_index)
    print(f"[{i+1}] {src}  ==>  {tgt.strip()}")

3.2 日期格式翻译-PyTorch版

import random  
import torch  
import torch.nn as nn  
import torch.nn.functional as F  
from torch import optim  
import time  
from nltk.translate.bleu_score import sentence_bleu, SmoothingFunction  
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")  
  
data = torch.load("date_dataset.pt", map_location=device)  
  
train_encoder_input = data['train_encoder_input']  
train_decoder_input = data['train_decoder_input']  
test_encoder_input  = data['test_encoder_input']  
test_decoder_input  = data['test_decoder_input']  
  
input_token_index = data['input_token_index']  
target_token_index = data['target_token_index']  
reverse_target_char_index = data['reverse_target_char_index']  
  
num_encoder_tokens = data['num_encoder_tokens']  
num_decoder_tokens = data['num_decoder_tokens']  
  
max_encoder_len = data['max_encoder_len']  
max_decoder_len = data['max_decoder_len']  
  
PAD_TOKEN = "<PAD>"  
reverse_input_char_index = {i: c for c, i in input_token_index.items()}  
  
def tensor_to_text(tensor_row, reverse_dict):  
    return ''.join(  
        reverse_dict[idx.item()]  
        for idx in tensor_row  
        if reverse_dict[idx.item()] != PAD_TOKEN  
    )  
  
test_inputs = [tensor_to_text(test_encoder_input[i], reverse_input_char_index)  
               for i in range(len(test_encoder_input))]  
  
test_targets = [tensor_to_text(test_decoder_input[i], reverse_target_char_index)  
                for i in range(len(test_decoder_input))]  
  
class Encoder(nn.Module):  
    def __init__(self, vocab_size, emb_dim, hidden_dim):  
        super().__init__()  
        self.embedding = nn.Embedding(vocab_size, emb_dim)  
        self.lstm = nn.LSTM(emb_dim, hidden_dim, batch_first=True, bidirectional=True)  
  
    def forward(self, x):  
        embedded = self.embedding(x)  
        outputs, (hidden, cell) = self.lstm(embedded)  
        return outputs, (hidden, cell)  
  
class Decoder(nn.Module):  
    def __init__(self, vocab_size, emb_dim, hidden_dim, use_attention=False):  
        super().__init__()  
        self.use_attention = use_attention  
        self.embedding = nn.Embedding(vocab_size, emb_dim)  
        self.lstm = nn.LSTM(emb_dim, hidden_dim, batch_first=True)  
  
        if use_attention:  
            self.attn = nn.Linear(hidden_dim + hidden_dim * 2, 1)  
            self.attn_combine = nn.Linear(hidden_dim + hidden_dim * 2, hidden_dim)  
  
        self.out = nn.Linear(hidden_dim, vocab_size)  
  
    def forward(self, input_step, hidden_cell, encoder_outputs):  
        embedded = self.embedding(input_step)  
        lstm_out, hidden_cell = self.lstm(embedded, hidden_cell)  
  
        query = lstm_out.squeeze(1)  
  
        if self.use_attention:  
            query_exp = query.unsqueeze(1).repeat(1, encoder_outputs.size(1), 1)  
            attn_scores = self.attn(torch.cat((query_exp, encoder_outputs), dim=-1)).squeeze(-1)  
            attn_weights = F.softmax(attn_scores, dim=-1).unsqueeze(1)  
            context = torch.bmm(attn_weights, encoder_outputs).squeeze(1)  
            combined = torch.cat((query, context), dim=1)  
            output = torch.tanh(self.attn_combine(combined))  
        else:  
            output = query  
  
        output = self.out(output)  
        return output, hidden_cell  
  
class Seq2Seq(nn.Module):  
    def __init__(self, encoder, decoder):  
        super().__init__()  
        self.encoder = encoder  
        self.decoder = decoder  
  
    def forward(self, src, tgt):  
        enc_outputs, (hidden, cell) = self.encoder(src)  
        hidden = hidden.view(2, -1, hidden.size(-1)).sum(dim=0, keepdim=True)  
        cell   = cell.view(2, -1, cell.size(-1)).sum(dim=0, keepdim=True)  
        dec_hidden = (hidden, cell)  
        outputs = []  
  
        for t in range(tgt.size(1)):  
            out, dec_hidden = self.decoder(tgt[:, t].unsqueeze(1), dec_hidden, enc_outputs)  
            outputs.append(out)  
  
        return torch.stack(outputs, dim=1)  
  
def train(model, epochs=10, batch_size=128, lr=0.001):  
    enc_optimizer = optim.Adam(model.encoder.parameters(), lr=lr)  
    dec_optimizer = optim.Adam(model.decoder.parameters(), lr=lr)  
    criterion = nn.CrossEntropyLoss(ignore_index=0)  
  
    start_time = time.time()  
  
    for epoch in range(epochs):  
        total_loss = 0  
        num_batches = 0  
        perm = torch.randperm(len(train_encoder_input))  
        for i in range(0, len(train_encoder_input), batch_size):  
            idx = perm[i:i+batch_size]  
            enc_in = train_encoder_input[idx]  
            dec_in_full = train_decoder_input[idx]  
            dec_in = dec_in_full[:, :-1]  
            dec_target = dec_in_full[:, 1:]  
  
            enc_optimizer.zero_grad()  
            dec_optimizer.zero_grad()  
  
            outputs = model(enc_in, dec_in)  
  
            loss = criterion(outputs.reshape(-1, num_decoder_tokens), dec_target.reshape(-1))  
            loss.backward()  
            enc_optimizer.step()  
            dec_optimizer.step()  
  
            total_loss += loss.item()  
            num_batches += 1  
  
        avg_loss = total_loss / num_batches  
        print(f"Epoch [{epoch+1}/{epochs}], Loss: {avg_loss:.4f}")  
  
    train_time = time.time() - start_time  
    print(f"训练总时间: {train_time:.2f} 秒")  
    return train_time  
  
import time  
  
def decode(model, input_tensor, use_beam_search=False, beam_width=3, max_len=20):  
    model.eval()  
    start_time = time.time()    
    with torch.no_grad():  
        input_tensor = input_tensor.unsqueeze(0)  
        enc_outputs, (hidden, cell) = model.encoder(input_tensor)  
        hidden = hidden.view(2, -1, hidden.size(-1)).sum(dim=0, keepdim=True)  
        cell   = cell.view(2, -1, cell.size(-1)).sum(dim=0, keepdim=True)  
        dec_hidden = (hidden, cell)  
  
        if not use_beam_search:  
            dec_input = torch.tensor([[target_token_index['\t']]], device=device)  
            result = []  
            for _ in range(max_len):  
                out, dec_hidden = model.decoder(dec_input, dec_hidden, enc_outputs)  
                pred = out.argmax(dim=-1)  
                token = pred.item()  
                if token == target_token_index['\n']:  
                    break  
                result.append(reverse_target_char_index[token])  
                dec_input = pred.unsqueeze(0)  
            decoded = ''.join(result)  
  
        else:  
            candidates = [([target_token_index['\t']], 0.0, dec_hidden)]  
            for _ in range(max_len):  
                new_cands = []  
                for seq, score, h in candidates:  
                    if seq[-1] == target_token_index['\n']:  
                        new_cands.append((seq, score, h))  
                        continue  
                    inp = torch.tensor([[seq[-1]]], device=device)  
                    out, new_h = model.decoder(inp, h, enc_outputs)  
                    log_probs = F.log_softmax(out, dim=-1).squeeze(0)  
                    k = min(beam_width, log_probs.size(0))    
                    top_vals, top_idx = log_probs.topk(k)  
                    for v, idx in zip(top_vals, top_idx):  
                        new_seq = seq + [idx.item()]  
                        new_score = score + v.item()  
                        new_cands.append((new_seq, new_score, new_h))  
                candidates = sorted(new_cands, key=lambda x: x[1], reverse=True)[:beam_width]  
  
            best = candidates[0][0]  
            decoded = ''.join(reverse_target_char_index.get(i, '') for i in best[1:] if i != target_token_index['\n'])  
  
    decode_time = time.time() - start_time  
    return decoded, decode_time  
  
  
def evaluate_bleu(model, test_enc_input, test_targets, use_beam_search=False, beam_width=3):  
    total_bleu = 0  
    total_time = 0  
    smoother = SmoothingFunction().method4  
  
    for i in range(len(test_enc_input)):  
        pred, dt = decode(model, test_enc_input[i], use_beam_search, beam_width)  
        total_time += dt  
        true = test_targets[i][1:-1]  
        ref = [list(true)]  
        hyp = list(pred)  
        bleu = sentence_bleu(ref, hyp, smoothing_function=smoother)  
        total_bleu += bleu  
  
    avg_bleu = total_bleu / len(test_enc_input) * 100  
    avg_time = total_time / len(test_enc_input)  
    print(f"BLEU: {avg_bleu:.2f} | 平均解码时间: {avg_time:.4f} 秒/样本")  
    return avg_bleu, avg_time  
  
  
def evaluate_exact_match(model, test_enc_input, test_targets, use_beam_search=False, beam_width=3):  
    correct = 0  
    total = len(test_enc_input)  
  
    for i in range(total):  
        pred, _ = decode(model, test_enc_input[i], use_beam_search, beam_width)    
        true = test_targets[i][1:-1]    
        if pred == true:  
            correct += 1  
  
    acc = correct / total * 100  
    print(f"完全匹配率: {acc:.2f}%")  
    return acc  
  
if __name__ == "__main__":  
    EMB_DIM = 64  
    HIDDEN_DIM = 256  
    EPOCHS = 10  
    BATCH_SIZE = 128  
    LR = 0.001  
  
    USE_ATTENTION    = True  
    USE_BEAM_SEARCH  = True  
    BEAM_WIDTH       = 3  
  
    print(f"\n=== 配置 ===")  
    print(f"使用注意力: {USE_ATTENTION}")  
    print(f"使用束搜索: {USE_BEAM_SEARCH} (beam width = {BEAM_WIDTH if USE_BEAM_SEARCH else 'No'})")  
  
    encoder = Encoder(num_encoder_tokens, EMB_DIM, HIDDEN_DIM).to(device)  
    decoder = Decoder(num_decoder_tokens, EMB_DIM, HIDDEN_DIM, use_attention=USE_ATTENTION).to(device)  
    model = Seq2Seq(encoder, decoder).to(device)  
  
    print("\n开始训练...")  
    train_time = train(model, epochs=EPOCHS, batch_size=BATCH_SIZE, lr=LR)  
  
    print("\n评估中...")  
    bleu = evaluate_bleu(model, test_encoder_input, test_targets,  
                         use_beam_search=USE_BEAM_SEARCH, beam_width=BEAM_WIDTH)  
    exact_acc = evaluate_exact_match(model, test_encoder_input, test_targets,  
                                     use_beam_search=USE_BEAM_SEARCH, beam_width=BEAM_WIDTH)  
  
    print("\n=== 抽取测试样本输出 ===")  
    sample_indices = random.sample(range(len(test_encoder_input)), 10)  
    for i in sample_indices:  
        src = test_inputs[i]  
        true = test_targets[i][1:-1]  
        pred,_ = decode(model, test_encoder_input[i],  
                      use_beam_search=USE_BEAM_SEARCH, beam_width=BEAM_WIDTH)  
        print(f"Human  : {src} | True  : {true} | Predicted  : {pred}")

3.3 日期格式翻译-TF版

import numpy as np  
import tensorflow as tf  
from tensorflow.keras.layers import Embedding, LSTM, Dense, Bidirectional  
import random  
import time  
from nltk.translate.bleu_score import sentence_bleu, SmoothingFunction  
  
dataset = np.load("date_dataset.npz", allow_pickle=True)  
  
train_encoder_input = dataset['train_encoder_input']  
train_decoder_input = dataset['train_decoder_input']  
test_encoder_input = dataset['test_encoder_input']  
test_decoder_input = dataset['test_decoder_input']  
  
input_token_index = dataset['input_token_index'].item()  
target_token_index = dataset['target_token_index'].item()  
reverse_target_char_index = dataset['reverse_target_char_index'].item()  
  
num_encoder_tokens = int(dataset['num_encoder_tokens'])  
num_decoder_tokens = int(dataset['num_decoder_tokens'])  
max_encoder_len = int(dataset['max_encoder_len'])  
max_decoder_len = int(dataset['max_decoder_len'])  
  
reverse_input_char_index = {i: c for c, i in input_token_index.items()}  
  
PAD_TOKEN = "<PAD>"    
  
def tensor_to_text(tensor_row, reverse_dict):  
    return ''.join(  
        reverse_dict.get(int(idx), '')  
        for idx in tensor_row  
        if reverse_dict.get(int(idx), '') != PAD_TOKEN  
    )  
  
  
test_inputs = [tensor_to_text(test_encoder_input[i], reverse_input_char_index)  
               for i in range(len(test_encoder_input))]  
  
test_targets = [tensor_to_text(test_decoder_input[i], reverse_target_char_index)  
                for i in range(len(test_decoder_input))]  
  
class Encoder(tf.keras.Model):  
    def __init__(self, vocab_size, emb_dim, hidden_dim):  
        super(Encoder, self).__init__()  
        self.embedding = Embedding(vocab_size, emb_dim, mask_zero=True)  
        self.lstm = Bidirectional(  
            LSTM(hidden_dim, return_sequences=True, return_state=True)  
        )  
  
    def call(self, x):  
        embedded = self.embedding(x)  
        outputs, fw_h, fw_c, bw_h, bw_c = self.lstm(embedded)  
        hidden = tf.concat([fw_h, bw_h], axis=-1)  
        cell = tf.concat([fw_c, bw_c], axis=-1)  
        return outputs, (hidden, cell)  
  
  
class Decoder(tf.keras.layers.Layer):  
    def __init__(self, vocab_size, emb_dim, hidden_dim, use_attention=False):  
        super(Decoder, self).__init__()  
        self.use_attention = use_attention  
        self.embedding = Embedding(vocab_size, emb_dim, mask_zero=True)  
        self.lstm = LSTM(hidden_dim, return_sequences=False, return_state=True)  
        self.out = Dense(vocab_size)  
  
        if use_attention:  
            self.attn_W1 = Dense(hidden_dim)                 
            self.attn_W2 = Dense(hidden_dim)               
            self.attn_V  = Dense(1)  
            self.attn_combine = Dense(hidden_dim)  
  
    def call(self, input_step, hidden_cell, encoder_outputs):  
   
        embedded = self.embedding(input_step)            
        lstm_out, h, c = self.lstm(embedded, initial_state=hidden_cell)  
        query = lstm_out                                  
if self.use_attention:  
            score = self.attn_V(  
                tf.tanh(  
                    self.attn_W1(encoder_outputs) +  
                    self.attn_W2(query[:, tf.newaxis, :])  
                )  
            )                                             
  
            attn_weights = tf.nn.softmax(score, axis=1)   
  
            context = tf.reduce_sum(encoder_outputs * attn_weights, axis=1)  
            combined = tf.concat([query, context], axis=-1)  
            output = tf.tanh(self.attn_combine(combined))  
        else:  
            output = query  
  
        logits = self.out(output)                         
        return logits, (h, c)  
  
  
class Seq2Seq(tf.keras.Model):  
    def __init__(self, encoder, decoder):  
        super(Seq2Seq, self).__init__()  
        self.encoder = encoder  
        self.decoder = decoder  
  
    def call(self, src, tgt, training=True):  
        enc_outputs, (hidden, cell) = self.encoder(src)  
  
        hidden_fw, hidden_bw = tf.split(hidden, num_or_size_splits=2, axis=-1)  
        cell_fw, cell_bw = tf.split(cell, num_or_size_splits=2, axis=-1)  
  
        hidden = hidden_fw + hidden_bw  
        cell = cell_fw + cell_bw  
  
        dec_hidden = (hidden, cell)  
  
        outputs = []  
        for t in range(tgt.shape[1]):  
            step_input = tgt[:, t:t + 1]  
            logits, dec_hidden = self.decoder(  
                step_input, dec_hidden, enc_outputs  
            )  
            outputs.append(logits)  
  
        return tf.stack(outputs, axis=1)  
  
def train(model, epochs=10, batch_size=128, lr=0.001):  
    optimizer = tf.keras.optimizers.Adam(learning_rate=lr)  
    loss_fn = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True)  
  
    start_time = time.time()  
    n_samples = len(train_encoder_input)  
  
    for epoch in range(epochs):  
        total_loss = 0.0  
        num_batches = 0  
  
        indices = np.random.permutation(n_samples)  
        enc = train_encoder_input[indices]  
        dec = train_decoder_input[indices]  
  
        for i in range(0, n_samples, batch_size):  
            enc_batch = enc[i:i + batch_size]  
            dec_batch = dec[i:i + batch_size]  
  
            dec_in = dec_batch[:, :-1]  
            dec_tgt = dec_batch[:, 1:]  
  
            with tf.GradientTape() as tape:  
                logits = model(enc_batch, dec_in, training=True)  
                loss_value = loss_fn(dec_tgt, logits)  
                loss_value += tf.reduce_sum(model.losses)  
  
            grads = tape.gradient(loss_value, model.trainable_variables)  
            optimizer.apply_gradients(zip(grads, model.trainable_variables))  
  
            total_loss += float(loss_value)  
            num_batches += 1  
  
        avg_loss = total_loss / num_batches  
        print(f"Epoch [{epoch + 1}/{epochs}], Loss: {avg_loss:.4f}")  
  
    train_time = time.time() - start_time  
    print(f"训练用时: {train_time:.2f} 秒")  
    return train_time  
  
@tf.function  
def greedy_decode_step(decoder, dec_input, dec_hidden, enc_outputs):  
    logits, dec_hidden = decoder(dec_input, dec_hidden, enc_outputs)  
    return logits, dec_hidden  
  
  
def decode(model, input_tensor, use_beam_search=False, beam_width=3, max_len=20):  
    dummy_src = tf.zeros((1, max_encoder_len), dtype=tf.int32)  
    dummy_tgt = tf.zeros((1, 1), dtype=tf.int32)  
    _ = model(dummy_src, dummy_tgt, training=False)  
  
    start_time = time.time()  
  
    input_tensor = tf.expand_dims(input_tensor, 0)  
    enc_outputs, (hidden, cell) = model.encoder(input_tensor)  
  
    hidden_fw, hidden_bw = tf.split(hidden, num_or_size_splits=2, axis=-1)  
    cell_fw, cell_bw = tf.split(cell, num_or_size_splits=2, axis=-1)  
  
    hidden = hidden_fw + hidden_bw  
    cell = cell_fw + cell_bw  
  
    dec_hidden = (hidden, cell)  
  
    if not use_beam_search:  
        dec_input = tf.constant([[target_token_index['\t']]], dtype=tf.int32)  
        result = []  
        for _ in range(max_len):  
            logits, dec_hidden = greedy_decode_step(  
                model.decoder, dec_input, dec_hidden, enc_outputs  
            )  
            pred = tf.argmax(logits, axis=-1)  
            token = int(pred[0])  
            if token == target_token_index.get('\n', -1):  
                break  
            result.append(reverse_target_char_index.get(token, ''))  
            dec_input = pred[:, tf.newaxis]  
  
        decoded = ''.join(result)  
  
    else:  
        candidates = [([target_token_index['\t']], 0.0, dec_hidden)]  
        for _ in range(max_len):  
            new_cands = []  
            for seq, score, h_c in candidates:  
                if seq[-1] == target_token_index.get('\n', -1):  
                    new_cands.append((seq, score, h_c))  
                    continue  
                inp = tf.constant([[seq[-1]]], dtype=tf.int32)  
                logits, new_h_c = model.decoder(inp, h_c, enc_outputs)  
                log_probs = tf.nn.log_softmax(logits, axis=-1)[0]  
                top_vals, top_idx = tf.math.top_k(log_probs, k=beam_width)  
                for v, idx in zip(top_vals, top_idx):  
                    new_seq = seq + [int(idx)]  
                    new_score = score + float(v)  
                    new_cands.append((new_seq, new_score, new_h_c))  
  
            candidates = sorted(new_cands, key=lambda x: x[1], reverse=True)[:beam_width]  
  
        best_seq = candidates[0][0]  
        decoded = ''.join(  
            reverse_target_char_index.get(i, '') for i in best_seq[1:] if i != target_token_index.get('\n', -1))  
  
    decode_time = time.time() - start_time  
    return decoded, decode_time  
  
  
def evaluate_bleu(model, test_enc_input, test_targets, use_beam_search=False, beam_width=3):  
    total_bleu = 0  
    total_time = 0  
    smoother = SmoothingFunction().method4  
  
    for i in range(len(test_enc_input)):  
        pred, dt = decode(model, test_enc_input[i], use_beam_search, beam_width)  
        total_time += dt  
        true = test_targets[i][1:-1]    
        ref = [list(true)]  
        hyp = list(pred)  
        bleu = sentence_bleu(ref, hyp, smoothing_function=smoother)  
        total_bleu += bleu  
  
    avg_bleu = total_bleu / len(test_enc_input) * 100  
    avg_time = total_time / len(test_enc_input)  
    print(f"BLEU: {avg_bleu:.2f} | 平均解码时间: {avg_time:.4f} 秒/样本")  
    return avg_bleu, avg_time  
  
  
def evaluate_exact_match(model, test_enc_input, test_targets, use_beam_search=False, beam_width=3):  
    correct = 0  
    total = len(test_enc_input)  
  
    for i in range(total):  
        pred, _ = decode(model, test_enc_input[i], use_beam_search, beam_width)  
        true = test_targets[i][1:-1]  
        if pred == true:  
            correct += 1  
  
    acc = correct / total * 100  
    print(f"完全匹配率: {acc:.2f}%")  
    return acc  
  
  
if __name__ == "__main__":  
    EMB_DIM = 64  
    HIDDEN_DIM = 256   
    BATCH_SIZE = 128  
    LR = 0.001  
  
    USE_ATTENTION = True  
    USE_BEAM_SEARCH = True  
    BEAM_WIDTH = 3  
  
    print(f"\n=== 配置 ===")  
    print(f"使用注意力: {USE_ATTENTION}")  
    print(f"使用束搜索: {USE_BEAM_SEARCH} (beam width = {BEAM_WIDTH if USE_BEAM_SEARCH else 'No'})")  
  
    encoder = Encoder(num_encoder_tokens, EMB_DIM, HIDDEN_DIM)  
    decoder = Decoder(num_decoder_tokens, EMB_DIM, HIDDEN_DIM, use_attention=USE_ATTENTION)  
    model = Seq2Seq(encoder, decoder)  
  
    print("\n开始训练...")  
    train_time = train(model, epochs=EPOCHS, batch_size=BATCH_SIZE, lr=LR)  
  
    print("\n评估中...")  
    bleu = evaluate_bleu(model, test_encoder_input, test_targets,  
                         use_beam_search=USE_BEAM_SEARCH, beam_width=BEAM_WIDTH)  
    exact_acc = evaluate_exact_match(model, test_encoder_input, test_targets,  
                                     use_beam_search=USE_BEAM_SEARCH, beam_width=BEAM_WIDTH)  
  
    print("\n=== 抽取测试样本输出 ===")  
    sample_indices = random.sample(range(len(test_encoder_input)), 10)  
    for i in sample_indices:  
        src = test_inputs[i]  
        true = test_targets[i][1:-1]  
        pred, _ = decode(model, test_encoder_input[i],  
                         use_beam_search=USE_BEAM_SEARCH, beam_width=BEAM_WIDTH)  
        print(f"Human  : {src} | True  : {true} | Predicted  : {pred}")