AG-news分类任务

1. 项目背景

本文使用 AG-News 新闻分类数据集，实现一个简单的文本分类模型。

任务目标：

输入一段新闻文本，判断该新闻属于哪个类别。

AG-News包含4个类别：

Label	类别	含义
0	World	世界新闻
1	Sports	体育新闻
2	Business	商业新闻
3	Sci/Tech	科技新闻

---

2. 项目结构

最终项目结构：

NLP-agnews/

├── main.py              # 项目入口
├── dataset.py           # 数据处理
├── train.py             # 模型训练
├── evaluate.py          # 模型评估
│
└── models/
    └── textcnn.py       # TextCNN模型
    │------其他模型。。。

各模块作用：

dataset.py

• 文本分词
• 构建词表
• 文本数字化
• Dataset封装

textcnn.py

• 定义神经网络结构

train.py

• loss计算
• 梯度计算
• 参数更新

evaluate.py

• 测试集评估

main.py

• 组织整个流程

---

3. 数据集加载

使用 HuggingFace datasets 加载 AG-News：

from datasets import load_dataset


dataset = load_dataset(
    "wangrongsheng/ag_news"
)

包含：

train
test

训练集：

120000条数据

测试集：

7600条数据

每条数据：

{
    "text":"新闻文本",
    "label":2
}

---

4. 文本分词

神经网络不能直接处理字符串：

Apple releases new phone

需要转换：

文本
↓
token
↓
数字编号
↓
Tensor

eg：

文本：

apple phone

词表：

apple -> 123
phone -> 456

转换：

[123,456]

模型才能处理。

---

5. 构建词汇表 Vocabulary

词表作用：将单词映射为数字。

eg：

{
    "apple":1,
    "phone":2,
    "market":3
}

文本：

apple phone

转换：

[1,2]

词表只能使用训练集建立：

vocab = build_vocab(
    dataset["train"]
)

不能加入测试集,否则会产生数据泄漏。

---

6. Dataset类

PyTorch中：

Dataset负责：

定义一条数据如何读取和处理。

定义：

class AGNewsDataset(Dataset):

    def __init__(
        self,
        dataset,
        vocab,
        max_len=100
    ):

参数：

参数	作用
dataset	原始数据
vocab	词表
max_len	文本最大长度

处理流程：

文本
↓
tokenizer
↓
数字id
↓
padding
↓
Tensor

例如：

原始文本：

Apple releases new phone

转换：

[123,45,67,89]

padding：

[123,45,67,89,0,0,...]

最终：

Tensor([100])

---

7. DataLoader

Dataset只能获取单条数据。

训练需要batch：

train_loader = DataLoader(
    train_dataset,
    batch_size=64,
    shuffle=True
)

输出：

x.shape

torch.Size([64,100])

指的是

64条新闻

每条新闻100个token

标签：

torch.Size([64])

表示：

64个类别标签。

Dataset.py

from collections import Counter
import torch
from torch.utils.data import Dataset

# 分词器
def tokenizer(text):
    return text.split()

# 建立词汇表
def build_vocab(dataset,max_vacab_size=30000):
    counter = Counter()
    for item in dataset:
        words = tokenizer(item["text"])
        counter.update(words)
    vocab = {
        "<PAD>":0,
        "<UNK>":1
    }
    for word,count in counter.most_common(max_vacab_size):
        vocab[word] = len(vocab)
    return vocab

# 文本数字化 encode
def encode(text,vocab):
    words = tokenizer(text)
    ids = []
    for word in words:
        if word in vocab:
            ids.append(vocab[word])
        else:
            ids.append(vocab["<UNK>"])
    return ids

# padding填充，保证每句话长度一样
def padding(ids,max_len,pad_id=0):
    if len(ids)<max_len:
        ids += [pad_id] * (max_len-len(ids))
    else:
        ids = ids[:max_len]
    return ids


# AGNewsDataset类,继承父类Dataset
class AGNewsDataset(Dataset):

    def __init__(
        self,
        dataset,
        vocab,
        max_len=100
    ):
        self.dataset = dataset
        self.vocab = vocab
        self.max_len = max_len

    def __len__(self):
        return len(self.dataset)

    def __getitem__(self, index):
        item = self.dataset[index]

        text = item["text"]
        label = item["label"]

        ids = encode(
            text,
            self.vocab
        )

        ids = padding(
            ids,
            self.max_len
        )

        return (
            torch.tensor(
                ids,
                dtype=torch.long
            ),

            torch.tensor(
                label,
                dtype=torch.long
            )
        )

---

8. TextCNN模型设计

TextCNN结构：

输入文本
↓
Embedding
↓
Conv1D
↓
MaxPooling
↓
Linear
↓
分类结果

---

8.1 Embedding层

输入：

[batch_size, seq_len]

例如：

[64,100]

Embedding后：

[64,100,128]

表示：

• 64条文本
• 每条100个词
• 每个词128维向量

---

8.2 卷积层

CNN提取局部词语组合。

卷积核：

kernel_size=3

一次查看连续3个词。

---

8.3 分类层

AG-News有4个类别：

因此：

Linear(
    128,
    4
)

输出：

[64,4]

表示：

64条新闻，每条新闻对应4个类别概率。

textcnn.py

import torch
import torch.nn as nn


class TextCNN(nn.Module):

    def __init__(
        self,
        vocab_size,
        embedding_dim,
        num_classes
    ):
        super().__init__()

        self.embedding = nn.Embedding(
            vocab_size,
            embedding_dim
        )

        self.conv = nn.Conv1d(
            in_channels=embedding_dim,
            out_channels=128,
            kernel_size=3
        )

        self.pool = nn.AdaptiveMaxPool1d(1)

        self.fc = nn.Linear(
            128,
            num_classes
        )


    def forward(self,x):

        # x:
        # [batch,seq_len]
        x=self.embedding(x)

        x=x.permute(
            0,
            2,
            1
        )

        x=self.conv(x)
        x=torch.relu(x)
        x=self.pool(x)

        x=x.squeeze(
            -1
        )
        out=self.fc(x)
        return out

---

9. 损失函数 CrossEntropyLoss

AG-News属于多分类任务。

模型输出：

[类别0概率,
 类别1概率,
 类别2概率,
 类别3概率]

真实标签：

Business

目标：

让正确类别概率最大。

使用：

criterion = nn.CrossEntropyLoss()

---

10. Adam优化器

使用Adam更新参数：

optimizer=torch.optim.Adam(
    model.parameters(),
    lr=0.001
)

训练过程：

forward
↓
计算loss
↓
backward
↓
optimizer.step()
↓
更新参数

---

11.textcnn结果

Epoch:1, Loss:0.5032, Train Acc:0.8165, Test Acc:0.8797
Epoch:2, Loss:0.2470, Train Acc:0.9157, Test Acc:0.8912
Epoch:3, Loss:0.1526, Train Acc:0.9491, Test Acc:0.8945
Epoch:4, Loss:0.0878, Train Acc:0.9720, Test Acc:0.8945
Epoch:5, Loss:0.0477, Train Acc:0.9859, Test Acc:0.8899
Epoch:6, Loss:0.0323, Train Acc:0.9914, Test Acc:0.8913
Epoch:7, Loss:0.0272, Train Acc:0.9929, Test Acc:0.8942
Epoch:8, Loss:0.0230, Train Acc:0.9946, Test Acc:0.8950
Epoch:9, Loss:0.0213, Train Acc:0.9951, Test Acc:0.8839
Epoch:10, Loss:0.0196, Train Acc:0.9954, Test Acc:0.8949
Epoch:11, Loss:0.0175, Train Acc:0.9962, Test Acc:0.8954
Epoch:12, Loss:0.0155, Train Acc:0.9968, Test Acc:0.8995
Epoch:13, Loss:0.0159, Train Acc:0.9968, Test Acc:0.9012
Epoch:14, Loss:0.0140, Train Acc:0.9972, Test Acc:0.8949
Epoch:15, Loss:0.0158, Train Acc:0.9971, Test Acc:0.9003
Epoch:16, Loss:0.0130, Train Acc:0.9978, Test Acc:0.8987
Epoch:17, Loss:0.0115, Train Acc:0.9980, Test Acc:0.8908
Epoch:18, Loss:0.0120, Train Acc:0.9979, Test Acc:0.8912
Epoch:19, Loss:0.0116, Train Acc:0.9981, Test Acc:0.9042
Epoch:20, Loss:0.0113, Train Acc:0.9981, Test Acc:0.8974
Epoch:21, Loss:0.0109, Train Acc:0.9983, Test Acc:0.9007
Epoch:22, Loss:0.0103, Train Acc:0.9985, Test Acc:0.9016
Epoch:23, Loss:0.0090, Train Acc:0.9985, Test Acc:0.8989
Epoch:24, Loss:0.0094, Train Acc:0.9985, Test Acc:0.9012
Epoch:25, Loss:0.0092, Train Acc:0.9986, Test Acc:0.9000
Epoch:26, Loss:0.0103, Train Acc:0.9985, Test Acc:0.9000
Epoch:27, Loss:0.0075, Train Acc:0.9989, Test Acc:0.9034
Epoch:28, Loss:0.0083, Train Acc:0.9989, Test Acc:0.8991
Epoch:29, Loss:0.0083, Train Acc:0.9989, Test Acc:0.8945
Epoch:30, Loss:0.0083, Train Acc:0.9989, Test Acc:0.9043
Epoch:31, Loss:0.0091, Train Acc:0.9989, Test Acc:0.8903
Epoch:32, Loss:0.0077, Train Acc:0.9989, Test Acc:0.9014
Epoch:33, Loss:0.0076, Train Acc:0.9991, Test Acc:0.8992
Epoch:34, Loss:0.0069, Train Acc:0.9992, Test Acc:0.8970
Epoch:35, Loss:0.0070, Train Acc:0.9992, Test Acc:0.8976
Epoch:36, Loss:0.0074, Train Acc:0.9991, Test Acc:0.8980
Epoch:37, Loss:0.0079, Train Acc:0.9989, Test Acc:0.9061
Epoch:38, Loss:0.0077, Train Acc:0.9991, Test Acc:0.8937
Epoch:39, Loss:0.0054, Train Acc:0.9994, Test Acc:0.9064
Epoch:40, Loss:0.0074, Train Acc:0.9993, Test Acc:0.9051
Epoch:41, Loss:0.0069, Train Acc:0.9991, Test Acc:0.9008
Epoch:42, Loss:0.0073, Train Acc:0.9991, Test Acc:0.8997
Epoch:43, Loss:0.0066, Train Acc:0.9992, Test Acc:0.9033
Epoch:44, Loss:0.0062, Train Acc:0.9991, Test Acc:0.8857
Epoch:45, Loss:0.0078, Train Acc:0.9991, Test Acc:0.9014
Epoch:46, Loss:0.0058, Train Acc:0.9993, Test Acc:0.9055
Epoch:47, Loss:0.0072, Train Acc:0.9991, Test Acc:0.9030
Epoch:48, Loss:0.0055, Train Acc:0.9993, Test Acc:0.9042
Epoch:49, Loss:0.0064, Train Acc:0.9991, Test Acc:0.8889
Epoch:50, Loss:0.0064, Train Acc:0.9991, Test Acc:0.8972

参数	设置
模型	TextCNN
Embedding维度	128
Batch Size	64
优化器	Adam
学习率	0.001
Epoch	50

---

测试集准确率接近参考的0.9。

---

然后提高一下lr 改成0.01，看看能不能提升acc

效果更不好了，

Epoch:1, Loss:0.4821, Train Acc:0.8464, Test Acc:0.8903
Epoch:2, Loss:0.3090, Train Acc:0.9106, Test Acc:0.8813
Epoch:3, Loss:0.2880, Train Acc:0.9253, Test Acc:0.8820
Epoch:4, Loss:0.2708, Train Acc:0.9384, Test Acc:0.8788
Epoch:5, Loss:0.2553, Train Acc:0.9477, Test Acc:0.8628
Epoch:6, Loss:0.2384, Train Acc:0.9571, Test Acc:0.8749
Epoch:7, Loss:0.2426, Train Acc:0.9619, Test Acc:0.8797
Epoch:8, Loss:0.2453, Train Acc:0.9669, Test Acc:0.8707
Epoch:9, Loss:0.2480, Train Acc:0.9695, Test Acc:0.8778
Epoch:10, Loss:0.2244, Train Acc:0.9745, Test Acc:0.8701
Epoch:11, Loss:0.2616, Train Acc:0.9756, Test Acc:0.8797
Epoch:12, Loss:0.2148, Train Acc:0.9789, Test Acc:0.8784
Epoch:13, Loss:0.2402, Train Acc:0.9801, Test Acc:0.8784
Epoch:14, Loss:0.2612, Train Acc:0.9811, Test Acc:0.8770
Epoch:15, Loss:0.2392, Train Acc:0.9831, Test Acc:0.8801
Epoch:16, Loss:0.2308, Train Acc:0.9848, Test Acc:0.8809
Epoch:17, Loss:0.2660, Train Acc:0.9852, Test Acc:0.8816
Epoch:18, Loss:0.2413, Train Acc:0.9854, Test Acc:0.8824
Epoch:19, Loss:0.2391, Train Acc:0.9871, Test Acc:0.8808
Epoch:20, Loss:0.2602, Train Acc:0.9871, Test Acc:0.8817
Epoch:21, Loss:0.2851, Train Acc:0.9881, Test Acc:0.8849
Epoch:22, Loss:0.2451, Train Acc:0.9894, Test Acc:0.8830
Epoch:23, Loss:0.2530, Train Acc:0.9894, Test Acc:0.8754
Epoch:24, Loss:0.2752, Train Acc:0.9896, Test Acc:0.8764
Epoch:25, Loss:0.2930, Train Acc:0.9902, Test Acc:0.8846
Epoch:26, Loss:0.2702, Train Acc:0.9909, Test Acc:0.8795
Epoch:27, Loss:0.2764, Train Acc:0.9908, Test Acc:0.8779
Epoch:28, Loss:0.2860, Train Acc:0.9917, Test Acc:0.8884
Epoch:29, Loss:0.3010, Train Acc:0.9919, Test Acc:0.8870
Epoch:30, Loss:0.2415, Train Acc:0.9929, Test Acc:0.8824
Epoch:31, Loss:0.3045, Train Acc:0.9923, Test Acc:0.8807
Epoch:32, Loss:0.2476, Train Acc:0.9930, Test Acc:0.8900
Epoch:33, Loss:0.2958, Train Acc:0.9930, Test Acc:0.8849
Epoch:34, Loss:0.2795, Train Acc:0.9930, Test Acc:0.8812
Epoch:35, Loss:0.3147, Train Acc:0.9929, Test Acc:0.8833
Epoch:36, Loss:0.2847, Train Acc:0.9931, Test Acc:0.8897
Epoch:37, Loss:0.3225, Train Acc:0.9937, Test Acc:0.8832
Epoch:38, Loss:0.2719, Train Acc:0.9943, Test Acc:0.8826
Epoch:39, Loss:0.3031, Train Acc:0.9943, Test Acc:0.8862
Epoch:40, Loss:0.2790, Train Acc:0.9947, Test Acc:0.8824
Epoch:41, Loss:0.3208, Train Acc:0.9940, Test Acc:0.8866
Epoch:42, Loss:0.3191, Train Acc:0.9946, Test Acc:0.8883
Epoch:43, Loss:0.2586, Train Acc:0.9950, Test Acc:0.8838
Epoch:44, Loss:0.3107, Train Acc:0.9945, Test Acc:0.8886
Epoch:45, Loss:0.2944, Train Acc:0.9950, Test Acc:0.8904
Epoch:46, Loss:0.2999, Train Acc:0.9953, Test Acc:0.8879
Epoch:47, Loss:0.3530, Train Acc:0.9953, Test Acc:0.8850
Epoch:48, Loss:0.2888, Train Acc:0.9955, Test Acc:0.8875
Epoch:49, Loss:0.2900, Train Acc:0.9956, Test Acc:0.8891
Epoch:50, Loss:0.3184, Train Acc:0.9953, Test Acc:0.8882

而且发生了过拟合

---

把embedding的dim提高试试

效果也不好,过拟合

---

尝试提高一下网络复杂度，捕获更深层语义

查阅资料发现：Multi-Kernel TextCNN（多卷积核 TextCNN），使用不同大小的卷积核，同时提取不同长度的文本局部特征。

其他配置不变，效果不好

---

后面尝试了改变embedding_dim，lr，结果都不会使得acc提升

然后改了batchsize和损失函数

criterion = nn.CrossEntropyLoss(
        label_smoothing=0.2
    )

这次突破了0.92

Epoch:1, Loss:1.0763, Train Acc:0.6576, Test Acc:0.8496
Epoch:2, Loss:0.8660, Train Acc:0.8410, Test Acc:0.8805
Epoch:3, Loss:0.8097, Train Acc:0.8831, Test Acc:0.8946
Epoch:4, Loss:0.7772, Train Acc:0.9053, Test Acc:0.8995
Epoch:5, Loss:0.7549, Train Acc:0.9214, Test Acc:0.9075
Epoch:6, Loss:0.7399, Train Acc:0.9293, Test Acc:0.9122
Epoch:7, Loss:0.7275, Train Acc:0.9374, Test Acc:0.9124
Epoch:8, Loss:0.7172, Train Acc:0.9445, Test Acc:0.9155
Epoch:9, Loss:0.7091, Train Acc:0.9490, Test Acc:0.9151
Epoch:10, Loss:0.7021, Train Acc:0.9545, Test Acc:0.9166
Epoch:11, Loss:0.6959, Train Acc:0.9588, Test Acc:0.9180
Epoch:12, Loss:0.6903, Train Acc:0.9627, Test Acc:0.9200
Epoch:13, Loss:0.6855, Train Acc:0.9666, Test Acc:0.9204
Epoch:14, Loss:0.6815, Train Acc:0.9695, Test Acc:0.9199
Epoch:15, Loss:0.6776, Train Acc:0.9720, Test Acc:0.9207
Epoch:16, Loss:0.6741, Train Acc:0.9738, Test Acc:0.9205
Epoch:17, Loss:0.6707, Train Acc:0.9768, Test Acc:0.9199
Epoch:18, Loss:0.6681, Train Acc:0.9786, Test Acc:0.9197
Epoch:19, Loss:0.6659, Train Acc:0.9796, Test Acc:0.9192
Epoch:20, Loss:0.6634, Train Acc:0.9814, Test Acc:0.9195
Epoch:21, Loss:0.6606, Train Acc:0.9832, Test Acc:0.9163
Epoch:22, Loss:0.6585, Train Acc:0.9845, Test Acc:0.9180
Epoch:23, Loss:0.6566, Train Acc:0.9856, Test Acc:0.9179
Epoch:24, Loss:0.6546, Train Acc:0.9865, Test Acc:0.9208
Epoch:25, Loss:0.6531, Train Acc:0.9879, Test Acc:0.9184
Epoch:26, Loss:0.6515, Train Acc:0.9887, Test Acc:0.9197
Epoch:27, Loss:0.6498, Train Acc:0.9895, Test Acc:0.9184
Epoch:28, Loss:0.6487, Train Acc:0.9895, Test Acc:0.9183
Epoch:29, Loss:0.6469, Train Acc:0.9905, Test Acc:0.9172
Epoch:30, Loss:0.6454, Train Acc:0.9912, Test Acc:0.9182
Epoch:31, Loss:0.6446, Train Acc:0.9914, Test Acc:0.9196
Epoch:32, Loss:0.6433, Train Acc:0.9923, Test Acc:0.9163
Epoch:33, Loss:0.6423, Train Acc:0.9925, Test Acc:0.9179
Epoch:34, Loss:0.6408, Train Acc:0.9930, Test Acc:0.9145
Epoch:35, Loss:0.6400, Train Acc:0.9934, Test Acc:0.9168
Epoch:36, Loss:0.6391, Train Acc:0.9938, Test Acc:0.9159
Epoch:37, Loss:0.6381, Train Acc:0.9937, Test Acc:0.9157
Epoch:38, Loss:0.6372, Train Acc:0.9943, Test Acc:0.9151
Epoch:39, Loss:0.6365, Train Acc:0.9943, Test Acc:0.9157
Epoch:40, Loss:0.6352, Train Acc:0.9950, Test Acc:0.9151
Epoch:41, Loss:0.6344, Train Acc:0.9951, Test Acc:0.9153
Epoch:42, Loss:0.6334, Train Acc:0.9955, Test Acc:0.9150
Epoch:43, Loss:0.6329, Train Acc:0.9956, Test Acc:0.9151
Epoch:44, Loss:0.6323, Train Acc:0.9955, Test Acc:0.9155
Epoch:45, Loss:0.6315, Train Acc:0.9960, Test Acc:0.9158
Epoch:46, Loss:0.6311, Train Acc:0.9960, Test Acc:0.9158
Epoch:47, Loss:0.6301, Train Acc:0.9962, Test Acc:0.9174
Epoch:48, Loss:0.6296, Train Acc:0.9965, Test Acc:0.9132
Epoch:49, Loss:0.6290, Train Acc:0.9964, Test Acc:0.9154
Epoch:50, Loss:0.6285, Train Acc:0.9964, Test Acc:0.9137

batch:128,

lr=0.0005,

weight_decay=1e-2,

embedding_dim = 128

---

multikernel上面这个结果不错，然后用上面这个参数，再跑一遍普通的textcnn，得到的结果更好：

Epoch:1, Loss:1.1842, Train Acc:0.5626, Test Acc:0.8138
Epoch:2, Loss:0.9347, Train Acc:0.7832, Test Acc:0.8636
Epoch:3, Loss:0.8623, Train Acc:0.8438, Test Acc:0.8804
Epoch:4, Loss:0.8247, Train Acc:0.8719, Test Acc:0.8912
Epoch:5, Loss:0.8018, Train Acc:0.8877, Test Acc:0.9004
Epoch:6, Loss:0.7847, Train Acc:0.8994, Test Acc:0.9036
Epoch:7, Loss:0.7711, Train Acc:0.9079, Test Acc:0.9087
Epoch:8, Loss:0.7617, Train Acc:0.9142, Test Acc:0.9117
Epoch:9, Loss:0.7543, Train Acc:0.9180, Test Acc:0.9126
Epoch:10, Loss:0.7474, Train Acc:0.9226, Test Acc:0.9154
Epoch:11, Loss:0.7412, Train Acc:0.9263, Test Acc:0.9192
Epoch:12, Loss:0.7360, Train Acc:0.9302, Test Acc:0.9211
Epoch:13, Loss:0.7318, Train Acc:0.9328, Test Acc:0.9178
Epoch:14, Loss:0.7272, Train Acc:0.9355, Test Acc:0.9216
Epoch:15, Loss:0.7246, Train Acc:0.9360, Test Acc:0.9232
Epoch:16, Loss:0.7205, Train Acc:0.9390, Test Acc:0.9242
Epoch:17, Loss:0.7173, Train Acc:0.9418, Test Acc:0.9238
Epoch:18, Loss:0.7152, Train Acc:0.9431, Test Acc:0.9249
Epoch:19, Loss:0.7121, Train Acc:0.9453, Test Acc:0.9228
Epoch:20, Loss:0.7105, Train Acc:0.9467, Test Acc:0.9236
Epoch:21, Loss:0.7074, Train Acc:0.9478, Test Acc:0.9232
Epoch:22, Loss:0.7052, Train Acc:0.9498, Test Acc:0.9253
Epoch:23, Loss:0.7035, Train Acc:0.9501, Test Acc:0.9232
Epoch:24, Loss:0.7017, Train Acc:0.9518, Test Acc:0.9247
Epoch:25, Loss:0.7003, Train Acc:0.9521, Test Acc:0.9247
Epoch:26, Loss:0.6984, Train Acc:0.9537, Test Acc:0.9250
Epoch:27, Loss:0.6965, Train Acc:0.9554, Test Acc:0.9251
Epoch:28, Loss:0.6943, Train Acc:0.9569, Test Acc:0.9245
Epoch:29, Loss:0.6935, Train Acc:0.9565, Test Acc:0.9247
Epoch:30, Loss:0.6920, Train Acc:0.9591, Test Acc:0.9251
Epoch:31, Loss:0.6902, Train Acc:0.9596, Test Acc:0.9241
Epoch:32, Loss:0.6886, Train Acc:0.9605, Test Acc:0.9247
Epoch:33, Loss:0.6875, Train Acc:0.9619, Test Acc:0.9241
Epoch:34, Loss:0.6860, Train Acc:0.9626, Test Acc:0.9242
Epoch:35, Loss:0.6851, Train Acc:0.9633, Test Acc:0.9254
Epoch:36, Loss:0.6834, Train Acc:0.9647, Test Acc:0.9257
Epoch:37, Loss:0.6829, Train Acc:0.9648, Test Acc:0.9243
Epoch:38, Loss:0.6811, Train Acc:0.9658, Test Acc:0.9238
Epoch:39, Loss:0.6803, Train Acc:0.9666, Test Acc:0.9224
Epoch:40, Loss:0.6794, Train Acc:0.9672, Test Acc:0.9224
Epoch:41, Loss:0.6782, Train Acc:0.9678, Test Acc:0.9228
Epoch:42, Loss:0.6766, Train Acc:0.9697, Test Acc:0.9230
Epoch:43, Loss:0.6757, Train Acc:0.9695, Test Acc:0.9229
Epoch:44, Loss:0.6754, Train Acc:0.9697, Test Acc:0.9229
Epoch:45, Loss:0.6742, Train Acc:0.9711, Test Acc:0.9237
Epoch:46, Loss:0.6734, Train Acc:0.9713, Test Acc:0.9237
Epoch:47, Loss:0.6720, Train Acc:0.9719, Test Acc:0.9232
Epoch:48, Loss:0.6710, Train Acc:0.9730, Test Acc:0.9216
Epoch:49, Loss:0.6705, Train Acc:0.9732, Test Acc:0.9222
Epoch:50, Loss:0.6695, Train Acc:0.9738, Test Acc:0.9214

test acc 最高达到0.9257

---

接着上面的模型，把lr稍微调大一些，改为0.001

得到了test acc 达到0.9271

Epoch:1, Loss:1.0804, Train Acc:0.6588, Test Acc:0.8575
Epoch:2, Loss:0.8602, Train Acc:0.8467, Test Acc:0.8908
Epoch:3, Loss:0.8093, Train Acc:0.8835, Test Acc:0.9020
Epoch:4, Loss:0.7821, Train Acc:0.9019, Test Acc:0.9079
Epoch:5, Loss:0.7658, Train Acc:0.9130, Test Acc:0.9138
Epoch:6, Loss:0.7535, Train Acc:0.9198, Test Acc:0.9216
Epoch:7, Loss:0.7440, Train Acc:0.9259, Test Acc:0.9220
Epoch:8, Loss:0.7372, Train Acc:0.9308, Test Acc:0.9217
Epoch:9, Loss:0.7316, Train Acc:0.9335, Test Acc:0.9243
Epoch:10, Loss:0.7254, Train Acc:0.9377, Test Acc:0.9199
Epoch:11, Loss:0.7216, Train Acc:0.9402, Test Acc:0.9230
Epoch:12, Loss:0.7170, Train Acc:0.9435, Test Acc:0.9250
Epoch:13, Loss:0.7126, Train Acc:0.9465, Test Acc:0.9238
Epoch:14, Loss:0.7100, Train Acc:0.9477, Test Acc:0.9233
Epoch:15, Loss:0.7063, Train Acc:0.9504, Test Acc:0.9241
Epoch:16, Loss:0.7028, Train Acc:0.9528, Test Acc:0.9271
Epoch:17, Loss:0.7000, Train Acc:0.9544, Test Acc:0.9246
Epoch:18, Loss:0.6979, Train Acc:0.9559, Test Acc:0.9255
Epoch:19, Loss:0.6950, Train Acc:0.9581, Test Acc:0.9241
Epoch:20, Loss:0.6936, Train Acc:0.9591, Test Acc:0.9238
Epoch:21, Loss:0.6904, Train Acc:0.9610, Test Acc:0.9228
Epoch:22, Loss:0.6882, Train Acc:0.9625, Test Acc:0.9243
Epoch:23, Loss:0.6859, Train Acc:0.9643, Test Acc:0.9221
Epoch:24, Loss:0.6848, Train Acc:0.9644, Test Acc:0.9195
Epoch:25, Loss:0.6827, Train Acc:0.9656, Test Acc:0.9204
Epoch:26, Loss:0.6805, Train Acc:0.9671, Test Acc:0.9201
Epoch:27, Loss:0.6790, Train Acc:0.9682, Test Acc:0.9189
Epoch:28, Loss:0.6774, Train Acc:0.9693, Test Acc:0.9220
Epoch:29, Loss:0.6753, Train Acc:0.9709, Test Acc:0.9180
Epoch:30, Loss:0.6728, Train Acc:0.9723, Test Acc:0.9205
Epoch:31, Loss:0.6719, Train Acc:0.9727, Test Acc:0.9216
Epoch:32, Loss:0.6708, Train Acc:0.9738, Test Acc:0.9186
Epoch:33, Loss:0.6685, Train Acc:0.9750, Test Acc:0.9212
Epoch:34, Loss:0.6674, Train Acc:0.9760, Test Acc:0.9201
Epoch:35, Loss:0.6656, Train Acc:0.9773, Test Acc:0.9204
Epoch:36, Loss:0.6646, Train Acc:0.9779, Test Acc:0.9179
Epoch:37, Loss:0.6631, Train Acc:0.9786, Test Acc:0.9192
Epoch:38, Loss:0.6618, Train Acc:0.9793, Test Acc:0.9195
Epoch:39, Loss:0.6608, Train Acc:0.9803, Test Acc:0.9196
Epoch:40, Loss:0.6594, Train Acc:0.9811, Test Acc:0.9180
Epoch:41, Loss:0.6577, Train Acc:0.9820, Test Acc:0.9189
Epoch:42, Loss:0.6569, Train Acc:0.9825, Test Acc:0.9179
Epoch:43, Loss:0.6560, Train Acc:0.9833, Test Acc:0.9175
Epoch:44, Loss:0.6547, Train Acc:0.9834, Test Acc:0.9182
Epoch:45, Loss:0.6530, Train Acc:0.9849, Test Acc:0.9172
Epoch:46, Loss:0.6524, Train Acc:0.9849, Test Acc:0.9170
Epoch:47, Loss:0.6515, Train Acc:0.9853, Test Acc:0.9188
Epoch:48, Loss:0.6502, Train Acc:0.9859, Test Acc:0.9196
Epoch:49, Loss:0.6495, Train Acc:0.9865, Test Acc:0.9187
Epoch:50, Loss:0.6481, Train Acc:0.9874, Test Acc:0.9176

---

只使用CNN就超过了参考的0.92，达到0.9271，下面使用更复杂一点的模型试试：

---

12.使用LSTM

import torch
import torch.nn as nn


class LSTMClassifier(nn.Module):

    def __init__(
        self,
        vocab_size,
        embedding_dim,
        hidden_dim,
        num_classes
    ):
        super().__init__()


        self.embedding = nn.Embedding(
            vocab_size,
            embedding_dim
        )

        self.lstm = nn.LSTM(
            input_size=embedding_dim,
            hidden_size=hidden_dim,
            batch_first=True
        )

        self.fc = nn.Linear(
            hidden_dim,
            num_classes
        )



    def forward(self,x):



        x=self.embedding(x)
        output,(hidden,cell)=self.lstm(x)

        hidden=hidden.squeeze(0)

        out=self.fc(hidden)


        return out

用跟上面同样的参数跑了一遍，发现test acc始终是0.25，就是完全平均瞎猜。没学到东西，

排查原因发现是

class AGNewsDataset(Dataset):

    def __init__(
        self,
        dataset,
        vocab,
        max_len=200
    ):

文本经过padding后，补上了大量,导致最终hidden state被大量padding信息影响。

---

效果不好，然后使用bi-LSTM

import torch
import torch.nn as nn

class BiLSTMClassifier(nn.Module):
    def __init__(
        self,
        vocab_size,
        embedding_dim,
        hidden_dim,
        num_classes
    ):
        super().__init__()

        self.embedding = nn.Embedding(
            vocab_size,
            embedding_dim
        )

        self.lstm = nn.LSTM(
            input_size=embedding_dim,
            hidden_size=hidden_dim,
            batch_first=True,
            bidirectional=True,
            dropout=0.5,
            num_layers=2
        )

        self.dropout = nn.Dropout(0.5)


        self.fc = nn.Linear(
            hidden_dim*2,
            num_classes
        )

    def forward(self,x):

        x=self.embedding(x)
        output,(hidden,cell)=self.lstm(x)

        forward_hidden=hidden[-2]
        backward_hidden=hidden[-1]
        hidden=torch.cat(
            (
                forward_hidden,
                backward_hidden
            ),
            dim=1
        )
        hidden=self.dropout(hidden)
        out=self.fc(hidden)
        return out

Epoch:1, Loss:0.7187, Train Acc:0.7089, Test Acc:0.8404
Epoch:2, Loss:0.3382, Train Acc:0.8833, Test Acc:0.8861
Epoch:3, Loss:0.2540, Train Acc:0.9149, Test Acc:0.8961
Epoch:4, Loss:0.2045, Train Acc:0.9326, Test Acc:0.8926
Epoch:5, Loss:0.1684, Train Acc:0.9445, Test Acc:0.9037
Epoch:6, Loss:0.1381, Train Acc:0.9560, Test Acc:0.8988
Epoch:7, Loss:0.1169, Train Acc:0.9633, Test Acc:0.9036
Epoch:8, Loss:0.0958, Train Acc:0.9704, Test Acc:0.9001
Epoch:9, Loss:0.0808, Train Acc:0.9755, Test Acc:0.9021
Epoch:10, Loss:0.0653, Train Acc:0.9806, Test Acc:0.9013
Epoch:11, Loss:0.0582, Train Acc:0.9832, Test Acc:0.9028
Epoch:12, Loss:0.0498, Train Acc:0.9858, Test Acc:0.8988
Epoch:13, Loss:0.0459, Train Acc:0.9866, Test Acc:0.9051
Epoch:14, Loss:0.0391, Train Acc:0.9887, Test Acc:0.9051
Epoch:15, Loss:0.0345, Train Acc:0.9899, Test Acc:0.9050
Epoch:16, Loss:0.0335, Train Acc:0.9903, Test Acc:0.9003
Epoch:17, Loss:0.0275, Train Acc:0.9921, Test Acc:0.9012
Epoch:18, Loss:0.0245, Train Acc:0.9930, Test Acc:0.9033
Epoch:19, Loss:0.0232, Train Acc:0.9933, Test Acc:0.9058
Epoch:20, Loss:0.0234, Train Acc:0.9930, Test Acc:0.9005
Epoch:21, Loss:0.0204, Train Acc:0.9942, Test Acc:0.8988
Epoch:22, Loss:0.0217, Train Acc:0.9935, Test Acc:0.9009
Epoch:23, Loss:0.0148, Train Acc:0.9957, Test Acc:0.8992
Epoch:24, Loss:0.0141, Train Acc:0.9959, Test Acc:0.9017
Epoch:25, Loss:0.0141, Train Acc:0.9959, Test Acc:0.9001
Epoch:26, Loss:0.0168, Train Acc:0.9949, Test Acc:0.9001
Epoch:27, Loss:0.0152, Train Acc:0.9955, Test Acc:0.9054
Epoch:28, Loss:0.0142, Train Acc:0.9957, Test Acc:0.9025
Epoch:29, Loss:0.0109, Train Acc:0.9967, Test Acc:0.9018
Epoch:30, Loss:0.0121, Train Acc:0.9963, Test Acc:0.9057
Epoch:31, Loss:0.0105, Train Acc:0.9969, Test Acc:0.8963
Epoch:32, Loss:0.0107, Train Acc:0.9966, Test Acc:0.9051
Epoch:33, Loss:0.0134, Train Acc:0.9958, Test Acc:0.9046
Epoch:34, Loss:0.0085, Train Acc:0.9974, Test Acc:0.9050
Epoch:35, Loss:0.0085, Train Acc:0.9974, Test Acc:0.9009
Epoch:36, Loss:0.0091, Train Acc:0.9972, Test Acc:0.9067
Epoch:37, Loss:0.0082, Train Acc:0.9974, Test Acc:0.9057
Epoch:38, Loss:0.0071, Train Acc:0.9977, Test Acc:0.9058
Epoch:39, Loss:0.0070, Train Acc:0.9979, Test Acc:0.9004
Epoch:40, Loss:0.0112, Train Acc:0.9965, Test Acc:0.9014
Epoch:41, Loss:0.0077, Train Acc:0.9976, Test Acc:0.9022
Epoch:42, Loss:0.0076, Train Acc:0.9977, Test Acc:0.9026
Epoch:43, Loss:0.0056, Train Acc:0.9984, Test Acc:0.9018
Epoch:44, Loss:0.0077, Train Acc:0.9976, Test Acc:0.9068
Epoch:45, Loss:0.0073, Train Acc:0.9978, Test Acc:0.9032
Epoch:46, Loss:0.0057, Train Acc:0.9982, Test Acc:0.9020
Epoch:47, Loss:0.0061, Train Acc:0.9979, Test Acc:0.9016
Epoch:48, Loss:0.0074, Train Acc:0.9976, Test Acc:0.9012
Epoch:49, Loss:0.0064, Train Acc:0.9980, Test Acc:0.9038
Epoch:50, Loss:0.0063, Train Acc:0.9980, Test Acc:0.9028

效果不好

13.使用Transformer

import torch
import torch.nn as nn
import math


# 位置编码
class PositionalEncoding(nn.Module):

    def __init__(
        self,
        embedding_dim,
        max_len=500
    ):
        super().__init__()

        pe = torch.zeros(
            max_len,
            embedding_dim
        )

        position = torch.arange(
            0,
            max_len,
            dtype=torch.float
        ).unsqueeze(1)

        div_term = torch.exp(
            torch.arange(
                0,
                embedding_dim,
                2
            ).float()
            *
            (-math.log(10000.0) / embedding_dim)
        )

        pe[:,0::2] = torch.sin(
            position * div_term
        )

        pe[:,1::2] = torch.cos(
            position * div_term
        )

        pe = pe.unsqueeze(0)

        # [1,max_len,embedding_dim]

        self.register_buffer(
            "pe",
            pe
        )

    def forward(self,x):

        x = x + self.pe[:,:x.size(1)]

        return x


class TransformerClassifier(nn.Module):

    def __init__(
        self,
        vocab_size,
        embedding_dim,
        num_classes,
        num_heads=4,
        num_layers=2,
        dropout=0.3
    ):
        super().__init__()

        self.embedding = nn.Embedding(
            vocab_size,
            embedding_dim,
            padding_idx=0
        )

        self.position = PositionalEncoding(
            embedding_dim
        )

        encoder_layer = nn.TransformerEncoderLayer(
            d_model=embedding_dim,
            nhead=num_heads,
            dim_feedforward=embedding_dim*4,
            dropout=dropout,
            batch_first=True
        )

        self.encoder = nn.TransformerEncoder(
            encoder_layer,
            num_layers=num_layers
        )

        self.dropout = nn.Dropout(
            dropout
        )

        self.fc = nn.Linear(
            embedding_dim,
            num_classes
        )


    def forward(self,x):
        x=self.embedding(x)
        x=self.position(x)
        x=self.encoder(x)

        x=torch.mean(
            x,
            dim=1
        )
        x=self.dropout(x)
        out=self.fc(x)
        return out

这个用的head和layer简化的，但是效果不好

Epoch:1, Loss:0.7370, Train Acc:0.6918, Test Acc:0.8496
Epoch:2, Loss:0.3479, Train Acc:0.8788, Test Acc:0.8775
Epoch:3, Loss:0.2730, Train Acc:0.9059, Test Acc:0.8896
Epoch:4, Loss:0.2241, Train Acc:0.9235, Test Acc:0.9038
Epoch:5, Loss:0.1925, Train Acc:0.9351, Test Acc:0.9066
Epoch:6, Loss:0.1640, Train Acc:0.9448, Test Acc:0.9068
Epoch:7, Loss:0.1418, Train Acc:0.9524, Test Acc:0.9051
Epoch:8, Loss:0.1239, Train Acc:0.9581, Test Acc:0.9039
Epoch:9, Loss:0.1067, Train Acc:0.9637, Test Acc:0.9042
Epoch:10, Loss:0.0932, Train Acc:0.9683, Test Acc:0.9022
Epoch:11, Loss:0.0800, Train Acc:0.9733, Test Acc:0.9042
Epoch:12, Loss:0.0686, Train Acc:0.9767, Test Acc:0.8980
Epoch:13, Loss:0.0612, Train Acc:0.9791, Test Acc:0.9036
Epoch:14, Loss:0.0561, Train Acc:0.9807, Test Acc:0.8997
Epoch:15, Loss:0.0510, Train Acc:0.9821, Test Acc:0.8997
Epoch:16, Loss:0.0442, Train Acc:0.9847, Test Acc:0.8993
Epoch:17, Loss:0.0395, Train Acc:0.9862, Test Acc:0.8975
Epoch:18, Loss:0.0368, Train Acc:0.9873, Test Acc:0.9039
Epoch:19, Loss:0.0332, Train Acc:0.9884, Test Acc:0.9008
Epoch:20, Loss:0.0310, Train Acc:0.9891, Test Acc:0.8997
Epoch:21, Loss:0.0274, Train Acc:0.9906, Test Acc:0.8975
Epoch:22, Loss:0.0247, Train Acc:0.9914, Test Acc:0.8978
Epoch:23, Loss:0.0248, Train Acc:0.9915, Test Acc:0.9008
Epoch:24, Loss:0.0225, Train Acc:0.9917, Test Acc:0.8997
Epoch:25, Loss:0.0228, Train Acc:0.9923, Test Acc:0.9043
Epoch:26, Loss:0.0205, Train Acc:0.9929, Test Acc:0.9041
Epoch:27, Loss:0.0207, Train Acc:0.9928, Test Acc:0.9000
Epoch:28, Loss:0.0202, Train Acc:0.9930, Test Acc:0.8974
Epoch:29, Loss:0.0180, Train Acc:0.9938, Test Acc:0.9011
Epoch:30, Loss:0.0189, Train Acc:0.9935, Test Acc:0.8975
Epoch:31, Loss:0.0170, Train Acc:0.9941, Test Acc:0.9005
Epoch:32, Loss:0.0188, Train Acc:0.9934, Test Acc:0.9016
Epoch:33, Loss:0.0158, Train Acc:0.9945, Test Acc:0.8991
Epoch:34, Loss:0.0166, Train Acc:0.9940, Test Acc:0.8955
Epoch:35, Loss:0.0139, Train Acc:0.9954, Test Acc:0.8982
Epoch:36, Loss:0.0143, Train Acc:0.9951, Test Acc:0.8961
Epoch:37, Loss:0.0130, Train Acc:0.9956, Test Acc:0.8958
Epoch:38, Loss:0.0159, Train Acc:0.9946, Test Acc:0.8991
Epoch:39, Loss:0.0131, Train Acc:0.9955, Test Acc:0.9005
Epoch:40, Loss:0.0121, Train Acc:0.9958, Test Acc:0.9007
Epoch:41, Loss:0.0129, Train Acc:0.9956, Test Acc:0.9003
Epoch:42, Loss:0.0128, Train Acc:0.9958, Test Acc:0.8949
Epoch:43, Loss:0.0137, Train Acc:0.9953, Test Acc:0.9024
Epoch:44, Loss:0.0117, Train Acc:0.9959, Test Acc:0.8987
Epoch:45, Loss:0.0110, Train Acc:0.9961, Test Acc:0.8992
Epoch:46, Loss:0.0105, Train Acc:0.9965, Test Acc:0.9030
Epoch:47, Loss:0.0127, Train Acc:0.9955, Test Acc:0.8999
Epoch:48, Loss:0.0115, Train Acc:0.9961, Test Acc:0.9021
Epoch:49, Loss:0.0100, Train Acc:0.9965, Test Acc:0.8979
Epoch:50, Loss:0.0113, Train Acc:0.9961, Test Acc:0.9005

然后换成完全的head=8，layer=6

Epoch:1, Loss:0.9644, Train Acc:0.7170, Test Acc:0.8718
Epoch:2, Loss:0.7676, Train Acc:0.8859, Test Acc:0.8817
Epoch:3, Loss:0.7337, Train Acc:0.9089, Test Acc:0.9028
Epoch:4, Loss:0.7126, Train Acc:0.9230, Test Acc:0.9016
Epoch:5, Loss:0.6959, Train Acc:0.9343, Test Acc:0.9020
Epoch:6, Loss:0.6833, Train Acc:0.9425, Test Acc:0.9063
Epoch:7, Loss:0.6741, Train Acc:0.9477, Test Acc:0.9047
Epoch:8, Loss:0.6652, Train Acc:0.9545, Test Acc:0.9033
Epoch:9, Loss:0.6580, Train Acc:0.9590, Test Acc:0.9100
Epoch:10, Loss:0.6525, Train Acc:0.9628, Test Acc:0.9089
Epoch:11, Loss:0.6467, Train Acc:0.9663, Test Acc:0.9087
Epoch:12, Loss:0.6422, Train Acc:0.9697, Test Acc:0.9084
Epoch:13, Loss:0.6393, Train Acc:0.9713, Test Acc:0.9051
Epoch:14, Loss:0.6353, Train Acc:0.9741, Test Acc:0.9092
Epoch:15, Loss:0.6325, Train Acc:0.9760, Test Acc:0.9080
Epoch:16, Loss:0.6302, Train Acc:0.9772, Test Acc:0.8996
Epoch:17, Loss:0.6287, Train Acc:0.9781, Test Acc:0.9097
Epoch:18, Loss:0.6257, Train Acc:0.9801, Test Acc:0.9028
Epoch:19, Loss:0.6240, Train Acc:0.9810, Test Acc:0.9088
Epoch:20, Loss:0.6219, Train Acc:0.9827, Test Acc:0.9061
Epoch:21, Loss:0.6219, Train Acc:0.9824, Test Acc:0.9070
Epoch:22, Loss:0.6200, Train Acc:0.9835, Test Acc:0.9067
Epoch:23, Loss:0.6191, Train Acc:0.9840, Test Acc:0.9047
Epoch:24, Loss:0.6183, Train Acc:0.9848, Test Acc:0.9078
Epoch:25, Loss:0.6168, Train Acc:0.9855, Test Acc:0.9037
Epoch:26, Loss:0.6160, Train Acc:0.9860, Test Acc:0.9039
Epoch:27, Loss:0.6155, Train Acc:0.9865, Test Acc:0.9000
Epoch:28, Loss:0.6144, Train Acc:0.9869, Test Acc:0.9050
Epoch:29, Loss:0.6142, Train Acc:0.9870, Test Acc:0.9005
Epoch:30, Loss:0.6133, Train Acc:0.9875, Test Acc:0.9072
Epoch:31, Loss:0.6129, Train Acc:0.9878, Test Acc:0.9066
Epoch:32, Loss:0.6118, Train Acc:0.9886, Test Acc:0.9054
Epoch:33, Loss:0.6123, Train Acc:0.9879, Test Acc:0.9072
Epoch:34, Loss:0.6120, Train Acc:0.9883, Test Acc:0.9054
Epoch:35, Loss:0.6104, Train Acc:0.9894, Test Acc:0.9062
Epoch:36, Loss:0.6102, Train Acc:0.9895, Test Acc:0.9075
Epoch:37, Loss:0.6102, Train Acc:0.9896, Test Acc:0.9051
Epoch:38, Loss:0.6096, Train Acc:0.9898, Test Acc:0.9055
Epoch:39, Loss:0.6095, Train Acc:0.9899, Test Acc:0.8993
Epoch:40, Loss:0.6087, Train Acc:0.9903, Test Acc:0.9059
Epoch:41, Loss:0.6093, Train Acc:0.9899, Test Acc:0.9030
Epoch:42, Loss:0.6085, Train Acc:0.9905, Test Acc:0.9014
Epoch:43, Loss:0.6079, Train Acc:0.9908, Test Acc:0.9024
Epoch:44, Loss:0.6078, Train Acc:0.9908, Test Acc:0.9003
Epoch:45, Loss:0.6074, Train Acc:0.9910, Test Acc:0.9061
Epoch:46, Loss:0.6069, Train Acc:0.9913, Test Acc:0.8964
Epoch:47, Loss:0.6065, Train Acc:0.9915, Test Acc:0.9030
Epoch:48, Loss:0.6071, Train Acc:0.9911, Test Acc:0.9003
Epoch:49, Loss:0.6066, Train Acc:0.9915, Test Acc:0.9057
Epoch:50, Loss:0.6056, Train Acc:0.9922, Test Acc:0.9026

还是过拟合

---

词汇表大小30000->50000，效果还是不好

---

目前效果最好的模型是cnn