NLP 基础：文本预处理/词向量/文本分类

大鱼>2026-07-06 9:54

NLP 基础：文本预处理/词向量/文本分类

1. 文本预处理

python 复制代码

import re
import jieba

# 中文分词
text = "机器学习是人工智能的一个分支"
words = jieba.lcut(text)
print(words)

# 去停用词
stopwords = set(open('stopwords.txt').read().split())
words = [w for w in words if w not in stopwords]

# 文本清洗
def clean_text(text):
    text = re.sub(r'[^\w\s]', '', text)  # 去标点
    text = re.sub(r'\d+', '', text)       # 去数字
    text = text.strip().lower()
    return text

2. 文本向量化

python 复制代码

from sklearn.feature_extraction.text import TfidfVectorizer, CountVectorizer

# 词袋模型
bow = CountVectorizer(max_features=5000)
X_bow = bow.fit_transform(texts)

# TF-IDF
tfidf = TfidfVectorizer(max_features=5000, ngram_range=(1, 2))
X_tfidf = tfidf.fit_transform(texts)

# Word2Vec
from gensim.models import Word2Vec
model = Word2Vec(sentences, vector_size=100, window=5, min_count=1)
vector = model.wv['机器学习']

3. 文本分类

python 复制代码

from sklearn.naive_bayes import MultinomialNB
from sklearn.linear_model import LogisticRegression
from sklearn.svm import SVC

# TF-IDF + 朴素贝叶斯
tfidf = TfidfVectorizer(max_features=5000)
X_train_tfidf = tfidf.fit_transform(X_train_text)
X_test_tfidf = tfidf.transform(X_test_text)

nb = MultinomialNB()
nb.fit(X_train_tfidf, y_train)

# TF-IDF + 逻辑回归
lr = LogisticRegression(max_iter=1000)
lr.fit(X_train_tfidf, y_train)

总结

方法	适用场景	优势
TF-IDF + NB	短文本分类	快速简单
TF-IDF + LR	通用分类	精度较高
Word2Vec	语义相似度	捕捉语义