机器学习:使用LSTM训练情感分析模型

背景:电商平台收到了许多关于宝贝的评论,需要自动从大量的数据中分析出各种情感的用户评论,并实时给出预警和提醒。

数据格式:

数据中最重要的两列:Review Text 和Rating,一个是评论内容一个是评分。

第一步:先进行数据处理:

python 复制代码
def load_and_preprocess_data(filepath):
    """加载并预处理数据"""
    df = pd.read_csv(filepath)
    texts = df['Review Text'].values
    labels = df['Rating'].values

    # 标签编码 (1-5 -> 0-4)
    le = LabelEncoder()
    labels = le.fit_transform(labels)

    # 文本序列化
    tokenizer = Tokenizer(num_words=20000, oov_token="<OOV>")
    tokenizer.fit_on_texts(texts)
    sequences = tokenizer.texts_to_sequences(texts)
    padded_sequences = pad_sequences(sequences, maxlen=200, truncating='post')

    return padded_sequences, labels, tokenizer, le

加载数据,并将数据序列化。

第二步:利用LSTM建立模型的结构,设置数据嵌入的向量维度:

python 复制代码
def build_model(vocab_size, max_len, embedding_dim=128):
    """构建LSTM模型"""
    model = Sequential([
        Embedding(vocab_size, embedding_dim, input_length=max_len),
        Bidirectional(LSTM(64, return_sequences=True)),
        Dropout(0.5),
        LSTM(32),
        Dropout(0.5),
        Dense(64, activation='relu'),
        Dropout(0.5),
        Dense(5, activation='softmax')
    ])

    model.compile(
        optimizer=Adam(learning_rate=0.001),
        loss='sparse_categorical_crossentropy',
        metrics=['accuracy']
    )
    return model

第三步:训练并保存模型:

python 复制代码
def main():
    # 加载数据
    X, y, tokenizer, le = load_and_preprocess_data('Clothing_Reviews.csv')
    X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

    # 构建模型
    model = build_model(vocab_size=20000, max_len=200)
    model.summary()

    # 训练模型
    history = model.fit(
        X_train, y_train,
        validation_data=(X_test, y_test),
        epochs=10,
        batch_size=64,
        callbacks=[EarlyStopping(monitor='val_loss', patience=3)],
        verbose=1
    )
    print(f"history: {history}")
    # 评估模型
    # plot_history(history)
    loss, accuracy = model.evaluate(X_test, y_test, verbose=0)
    print(f'测试集准确率: {accuracy:.2f}')

    # 保存模型和tokenizer
    model.save('sentiment_lstm_5class.h5')
    with open('tokenizer.pickle', 'wb') as handle:
        pickle.dump(tokenizer, handle, protocol=pickle.HIGHEST_PROTOCOL)
    print("模型和tokenizer已保存")
    # 保存label encoder
    with open('label_encoder.pickle', 'wb') as handle:
        pickle.dump(le, handle, protocol=pickle.HIGHEST_PROTOCOL)

至此,根据训练集训练的模型保存完毕。要想使用训练保持好的模型进行数据预测,需要新建一个预测的方法:

python 复制代码
import numpy as np
import pickle

from keras.src.saving import load_model
from keras.src.utils import pad_sequences
from sklearn.preprocessing import LabelEncoder


class SentimentPredictor:
    def __init__(self, model_path, tokenizer_path, label_encoder_path=None, max_len=200):
        """初始化预测器"""
        self.model = load_model(model_path)
        with open(tokenizer_path, 'rb') as handle:
            self.tokenizer = pickle.load(handle)
        with open(label_encoder_path, 'rb') as handle:
            self.label_encoder = pickle.load(handle)
        self.max_len = max_len

    def preprocess_text(self, text):
        """预处理文本"""
        sequence = self.tokenizer.texts_to_sequences([text])
        padded = pad_sequences(sequence, maxlen=self.max_len, truncating='post')
        return padded

    def predict_sentiment(self, text, verbose=False):
        """预测情感分值(1-5)"""
        # 预处理
        padded_sequence = self.preprocess_text(text)

        # 预测
        prediction = self.model.predict(padded_sequence, verbose=0)
        predicted_class = np.argmax(prediction, axis=1)

        # 转换回原始标签(1-5)
        predicted_score = self.label_encoder.inverse_transform(predicted_class)[0]

        if verbose:
            print(f"评论: {text}")
            print(f"预测情感分值: {predicted_score}")
            print("各类别概率:")
            for i, prob in enumerate(prediction[0]):
                print(f"{i + 1}分: {prob:.4f}")
            print("-" * 50)

        return predicted_score, prediction[0]

这个类封装了读取加载模型、使用模型进行预测的方法,调用是传入已经保持的模型路径即可。使用实例:

python 复制代码
# 初始化预测器
predictor = SentimentPredictor(
    model_path='sentiment_lstm_5class.h5',
    tokenizer_path='tokenizer.pickle',
    label_encoder_path='label_encoder.pickle'
)

# 测试评论
test_reviews = [
    "This product is absolutely amazing! Best purchase ever!",
    "The item was okay, but not worth the price.",
    "Terrible quality. Would not recommend to anyone.",
    "It's decent for the price, though it has some flaws.",
    "I'm completely satisfied with this purchase. It exceeded all my expectations!"
]

# 批量预测
for review in test_reviews:
    score, probs = predictor.predict_sentiment(review, verbose=True)
    print(f"{review}: 预测情感分值: {score},准确率: {np.max(probs) * 100:.2f}%")

# 预测单个评论
sample_review = "The product was good but the delivery took too long."
score, probs = predictor.predict_sentiment(sample_review, verbose=True)
print(f"{sample_review}: 预测情感分值: {score},准确率: {np.max(probs) * 100:.2f}%")
相关推荐
塑料铲子18 小时前
你的 Skill 越多越蠢?Token 上下文爆炸的求生指南
机器学习·agent
TsingtaoAI20 小时前
VeyForge:构建具身智能的AI 3D资产与Sim-to-Real数据引擎
人工智能·机器学习
罗西的思考3 天前
【轻量化复刻大模型 / KIMI K3】Muon 优化器及 训练系统
人工智能·算法·机器学习
Asher11253 天前
电池数据整理复盘:从 MAT 文件到充电曲线
机器学习
龙腾AI白云3 天前
具身智能的空间导航与未知环境探索
大数据·数据库·人工智能·机器学习·知识图谱
汤姆宝库3 天前
RoPE旋转位置编码的实现与长上下文外推
python·深度学习·机器学习
回眸&啤酒鸭3 天前
【回眸】Hy4 模型效果展示与能力评测大纲
大数据·人工智能·机器学习
AI小怪兽3 天前
你的视觉基础模型会说“植物语“吗?面向目标检测的视觉基础模型植物语法
人工智能·深度学习·机器学习
峯5803 天前
机器学习05,用 scikit-learn 做加州房价预测的完整入门工程
人工智能·机器学习·scikit-learn