1. Transformers (Hugging Face)
技术栈
- 核心语言: Python
- 深度学习框架: PyTorch, TensorFlow, JAX
- 依赖: tokenizers, datasets, accelerate
- 模型格式: ONNX, TensorRT支持
优点
- 生态丰富: 超过200,000个预训练模型
- 统一接口: 一套API支持多种任务和模型
- 社区活跃: 快速跟进最新研究成果
- 部署便利: 支持多种推理优化
- 文档完善: 详细的教程和示例
技术特点
- 基于Transformer架构的统一抽象
- 自动混合精度训练支持
- 分布式训练集成
- 模型量化和剪枝支持
项目目录结构
transformers/
├── src/transformers/
│ ├── models/ # 各种模型实现
│ │ ├── bert/
│ │ ├── gpt2/
│ │ └── t5/
│ ├── tokenization_utils.py
│ ├── trainer.py # 训练器
│ ├── pipeline/ # 预定义管道
│ └── optimization.py # 优化器
├── examples/ # 示例代码
├── tests/ # 单元测试
└── docs/ # 文档开发示例
python
from transformers import AutoTokenizer, AutoModelForSequenceClassification
from transformers import TrainingArguments, Trainer
import torch
# 加载预训练模型
tokenizer = AutoTokenizer.from_pretrained('bert-base-chinese')
model = AutoModelForSequenceClassification.from_pretrained(
'bert-base-chinese',
num_labels=2
)
# 数据预处理
def tokenize_function(examples):
return tokenizer(
examples['text'],
padding='max_length',
truncation=True,
max_length=512
)
# 训练配置
training_args = TrainingArguments(
output_dir='./results',
learning_rate=2e-5,
per_device_train_batch_size=16,
num_train_epochs=3,
weight_decay=0.01,
)
trainer = Trainer(
model=model,
args=training_args,
train_dataset=train_dataset,
eval_dataset=eval_dataset,
)
trainer.train()实现难易度: ⭐⭐ (简单)
- 开箱即用的预训练模型
- 丰富的预定义Pipeline
- 详细的文档和教程
2. spaCy
技术栈
- 核心语言: Python + Cython
- 底层: C++ 优化
- 扩展: 支持深度学习组件
- 部署: 支持Docker、REST API
优点
- 工业级性能: 高度优化的C++后端
- 生产就绪: 内置的管道化处理
- 多语言支持: 70+语言的预训练模型
- 可扩展: 自定义组件和插件系统
- 内存高效: 优化的数据结构
技术特点
- 基于依存语法的NLP管道
- 支持自定义神经网络组件
- 实体链接和知识图谱集成
- 规则和机器学习混合方法
项目目录结构
spacy/
├── spacy/
│ ├── lang/ # 语言特定模块
│ │ ├── zh/ # 中文支持
│ │ └── en/ # 英文支持
│ ├── pipeline/ # 处理管道组件
│ │ ├── ner.py # 命名实体识别
│ │ └── tagger.py # 词性标注
│ ├── tokens/ # Token处理
│ ├── matcher/ # 模式匹配
│ └── training/ # 训练工具
├── models/ # 预训练模型
└── examples/ # 示例代码开发示例
python
import spacy
from spacy.training import Example
from spacy.util import minibatch
# 加载预训练模型
nlp = spacy.load("zh_core_web_sm")
# 自定义实体识别器
@spacy.registry.misc("custom_ner_data")
def create_training_data():
TRAIN_DATA = [
("苹果公司发布了新iPhone", {"entities": [(0, 3, "ORG")]}),
("微软收购了GitHub", {"entities": [(0, 2, "ORG"), (5, 11, "ORG")]}),
]
return TRAIN_DATA
# 训练自定义模型
def train_ner_model():
nlp = spacy.blank("zh")
ner = nlp.add_pipe("ner")
# 添加标签
ner.add_label("ORG")
# 训练数据
train_data = create_training_data()
examples = [Example.from_dict(nlp.make_doc(text), annotations)
for text, annotations in train_data]
# 训练
nlp.begin_training()
for epoch in range(100):
spacy.util.fix_random_seed(0)
for batch in minibatch(examples, size=2):
nlp.update(batch)
return nlp
# 使用管道处理
def process_text(text):
doc = nlp(text)
results = {
"tokens": [(token.text, token.pos_, token.dep_)
for token in doc],
"entities": [(ent.text, ent.label_)
for ent in doc.ents],
"sentences": [sent.text for sent in doc.sents]
}
return results实现难易度: ⭐⭐⭐ (中等)
- 需要理解NLP管道概念
- 自定义组件需要一定经验
- 文档详细但概念较多
3. NLTK (Natural Language Toolkit)
技术栈
- 核心语言: 纯Python
- 数据格式: 多种语料库格式支持
- 算法: 经典机器学习算法
- 扩展: 与scikit-learn集成
优点
- 教育友好: 丰富的教学资源
- 算法全面: 包含大量经典NLP算法
- 语料库丰富: 内置多种标准数据集
- 文档详细: 配套书籍《NLTK Book》
- 接口简单: 函数式编程风格
技术特点
- 基于规则和统计的方法
- 丰富的文本预处理工具
- 语法分析和语义分析工具
- 机器学习分类器集成
项目目录结构
nltk/
├── nltk/
│ ├── tokenize/ # 分词工具
│ ├── tag/ # 词性标注
│ ├── chunk/ # 短语识别
│ ├── parse/ # 语法分析
│ ├── sem/ # 语义分析
│ ├── corpus/ # 语料库接口
│ ├── classify/ # 分类器
│ └── metrics/ # 评估指标
├── nltk_data/ # 数据文件
└── examples/ # 示例代码开发示例
python
import nltk
from nltk.tokenize import word_tokenize, sent_tokenize
from nltk.tag import pos_tag
from nltk.chunk import ne_chunk
from nltk.corpus import stopwords
from nltk.stem import PorterStemmer
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.naive_bayes import MultinomialNB
# 下载必要的数据
nltk.download('punkt')
nltk.download('averaged_perceptron_tagger')
nltk.download('maxent_ne_chunker')
nltk.download('words')
nltk.download('stopwords')
class NLTKTextProcessor:
def __init__(self):
self.stemmer = PorterStemmer()
self.stop_words = set(stopwords.words('english'))
def preprocess_text(self, text):
"""完整的文本预处理管道"""
# 分句
sentences = sent_tokenize(text)
processed_sentences = []
for sentence in sentences:
# 分词
tokens = word_tokenize(sentence)
# 词性标注
pos_tags = pos_tag(tokens)
# 命名实体识别
named_entities = ne_chunk(pos_tags)
# 去除停用词和词干提取
clean_tokens = []
for word, tag in pos_tags:
if word.lower() not in self.stop_words:
stemmed = self.stemmer.stem(word.lower())
clean_tokens.append(stemmed)
processed_sentences.append({
'original': sentence,
'tokens': tokens,
'pos_tags': pos_tags,
'entities': named_entities,
'clean_tokens': clean_tokens
})
return processed_sentences
# 文本分类示例
class TextClassifier:
def __init__(self):
self.vectorizer = TfidfVectorizer()
self.classifier = MultinomialNB()
self.processor = NLTKTextProcessor()
def train(self, texts, labels):
# 预处理文本
processed_texts = []
for text in texts:
tokens = self.processor.preprocess_text(text)
clean_text = ' '.join([' '.join(sent['clean_tokens'])
for sent in tokens])
processed_texts.append(clean_text)
# 特征提取
X = self.vectorizer.fit_transform(processed_texts)
# 训练分类器
self.classifier.fit(X, labels)
def predict(self, text):
processed_text = self.processor.preprocess_text(text)
clean_text = ' '.join([' '.join(sent['clean_tokens'])
for sent in processed_text])
X = self.vectorizer.transform([clean_text])
return self.classifier.predict(X)[0]实现难易度: ⭐⭐⭐ (中等)
- 需要理解传统NLP方法
- 丰富的功能需要学习成本
- 适合学习和教学用途
4. Stanford CoreNLP
技术栈
- 核心语言: Java
- 接口: Python, JavaScript, R等多语言绑定
- 部署: REST API服务
- 模型: 基于深度学习和规则混合
优点
- 学术权威: 斯坦福大学开发
- 功能全面: 完整的NLP管道
- 多语言: 支持多种语言
- 高准确度: 学术级别的模型质量
- 服务化: 易于部署为微服务
技术特点
- 基于神经网络的现代NLP方法
- 完整的语义分析能力
- 共指消解和情感分析
- 关系抽取和开放信息抽取
项目目录结构
stanford-corenlp/
├── src/edu/stanford/nlp/
│ ├── pipeline/ # 处理管道
│ ├── ling/ # 语言学结构
│ ├── trees/ # 语法树
│ ├── sentiment/ # 情感分析
│ ├── ie/ # 信息抽取
│ └── coref/ # 共指消解
├── models/ # 预训练模型
├── lib/ # 依赖库
└── scripts/ # 启动脚本开发示例
python
import stanza
import requests
import json
# 方法1: 使用stanza包装器
class StanzaNLPProcessor:
def __init__(self, lang='en'):
# 下载并初始化模型
stanza.download(lang)
self.nlp = stanza.Pipeline(
lang=lang,
processors='tokenize,mwt,pos,lemma,depparse,ner'
)
def process_text(self, text):
doc = self.nlp(text)
results = []
for sentence in doc.sentences:
sent_info = {
'text': sentence.text,
'tokens': [],
'dependencies': [],
'entities': []
}
# 提取token信息
for token in sentence.tokens:
for word in token.words:
sent_info['tokens'].append({
'text': word.text,
'lemma': word.lemma,
'pos': word.pos,
'deprel': word.deprel
})
# 提取依存关系
for word in sentence.words:
if word.head != 0:
sent_info['dependencies'].append({
'dependent': word.text,
'governor': sentence.words[word.head-1].text,
'relation': word.deprel
})
# 提取实体
for ent in sentence.ents:
sent_info['entities'].append({
'text': ent.text,
'type': ent.type,
'start': ent.start_char,
'end': ent.end_char
})
results.append(sent_info)
return results
# 方法2: 使用CoreNLP服务器
class CoreNLPClient:
def __init__(self, server_url='http://localhost:9000'):
self.server_url = server_url
def analyze_text(self, text):
"""调用CoreNLP服务器进行分析"""
properties = {
'annotators': 'tokenize,ssplit,pos,lemma,ner,parse,sentiment',
'outputFormat': 'json'
}
response = requests.post(
self.server_url,
params=properties,
data=text.encode('utf-8'),
headers={'Content-Type': 'application/x-www-form-urlencoded'}
)
return response.json()
# 使用示例
def demo_corenlp():
# 使用Stanza处理器
processor = StanzaNLPProcessor('en')
text = """
Apple Inc. is planning to build a new campus in Austin, Texas.
The CEO Tim Cook announced this exciting development yesterday.
"""
results = processor.process_text(text)
for i, sent in enumerate(results):
print(f"Sentence {i+1}: {sent['text']}")
print("Entities:")
for ent in sent['entities']:
print(f" - {ent['text']}: {ent['type']}")
print("Dependencies:")
for dep in sent['dependencies'][:5]: # 显示前5个
print(f" - {dep['dependent']} --{dep['relation']}--> {dep['governor']}")
print()实现难易度: ⭐⭐⭐⭐ (较难)
- Java环境配置复杂
- 需要理解语言学概念
- 内存消耗较大
5. FastText
技术栈
- 核心语言: C++
- 接口: Python, Java, Go等
- 模型: Word embedding + 分类
- 优化: 高度优化的向量运算
优点
- 速度极快: C++实现，高度优化
- 支持子词: 处理OOV问题
- 多语言: 支持157种语言
- 轻量级: 模型体积小
- 易用: 简单的API接口
技术特点
- 基于字符级n-gram的词向量
- 层次softmax优化
- 支持监督学习分类
- 增量学习能力
项目目录结构
fastText/
├── src/
│ ├── fasttext.cc # 主要接口
│ ├── args.cc # 参数处理
│ ├── dictionary.cc # 词典管理
│ ├── matrix.cc # 矩阵运算
│ ├── model.cc # 模型实现
│ └── vector.cc # 向量运算
├── python/ # Python绑定
├── models/ # 预训练模型
└── tests/ # 测试代码开发示例
python
import fasttext
import numpy as np
from sklearn.metrics import accuracy_score, classification_report
class FastTextClassifier:
def __init__(self):
self.model = None
self.word_vectors = None
def prepare_data(self, texts, labels, output_file):
"""准备FastText格式的训练数据"""
with open(output_file, 'w', encoding='utf-8') as f:
for text, label in zip(texts, labels):
# FastText格式: __label__类别 文本内容
clean_text = text.replace('\n', ' ').strip()
f.write(f"__label__{label} {clean_text}\n")
def train_classifier(self, train_file, **kwargs):
"""训练分类模型"""
default_params = {
'lr': 0.1, # 学习率
'epoch': 100, # 训练轮数
'wordNgrams': 2, # n-gram特征
'dim': 100, # 向量维度
'ws': 5, # 上下文窗口大小
'loss': 'softmax' # 损失函数
}
default_params.update(kwargs)
# 训练模型
self.model = fasttext.train_supervised(train_file, **default_params)
return self.model
def train_word_vectors(self, corpus_file, **kwargs):
"""训练词向量"""
default_params = {
'model': 'skipgram', # skipgram 或 cbow
'lr': 0.05,
'dim': 300,
'ws': 5,
'epoch': 5,
'minCount': 1,
'neg': 5
}
default_params.update(kwargs)
self.word_vectors = fasttext.train_unsupervised(
corpus_file, **default_params
)
return self.word_vectors
def predict(self, texts, k=1):
"""预测分类"""
if self.model is None:
raise ValueError("模型未训练")
predictions = []
for text in texts:
pred_labels, pred_probs = self.model.predict(text, k=k)
predictions.append({
'labels': [label.replace('__label__', '') for label in pred_labels],
'probabilities': pred_probs.tolist()
})
return predictions
def get_word_vector(self, word):
"""获取词向量"""
if self.word_vectors is None:
raise ValueError("词向量模型未训练")
return self.word_vectors.get_word_vector(word)
def find_similar_words(self, word, k=10):
"""查找相似词"""
if self.word_vectors is None:
raise ValueError("词向量模型未训练")
return self.word_vectors.get_nearest_neighbors(word, k)
# 使用示例
def demo_fasttext():
classifier = FastTextClassifier()
# 示例数据
train_texts = [
"这是一个很好的产品",
"质量太差了，不推荐",
"非常满意，会再次购买",
"完全不值这个价钱"
]
train_labels = ["positive", "negative", "positive", "negative"]
# 准备训练数据
classifier.prepare_data(train_texts, train_labels, "train.txt")
# 训练分类器
model = classifier.train_classifier("train.txt", epoch=50, lr=0.1)
# 预测
test_texts = ["这个产品还不错", "质量有问题"]
predictions = classifier.predict(test_texts)
for text, pred in zip(test_texts, predictions):
print(f"文本: {text}")
print(f"预测: {pred['labels'][0]} (置信度: {pred['probabilities'][0]:.3f})")
print()
# 训练词向量
with open("corpus.txt", "w", encoding="utf-8") as f:
f.write("\n".join(train_texts))
word_model = classifier.train_word_vectors("corpus.txt", dim=100)
# 获取相似词
try:
similar_words = classifier.find_similar_words("产品")
print("与'产品'相似的词:")
for word, similarity in similar_words:
print(f" {word}: {similarity:.3f}")
except:
print("词汇表中没有找到'产品'")
# 高级特性示例
class AdvancedFastText:
def __init__(self):
self.model = None
def train_with_validation(self, train_file, valid_file):
"""带验证集的训练"""
self.model = fasttext.train_supervised(
input=train_file,
epoch=100,
lr=0.1,
wordNgrams=2,
verbose=2,
minCount=1,
autotuneValidationFile=valid_file, # 自动调参
autotuneDuration=600 # 调参时间限制(秒)
)
return self.model
def quantize_model(self, model_path):
"""模型量化压缩"""
self.model.quantize(
model_path,
retrain=True,
cutoff=100000
)
return self.model
def evaluate_model(self, test_file):
"""评估模型性能"""
result = self.model.test(test_file)
precision = result[1]
recall = result[2]
f1_score = 2 * (precision * recall) / (precision + recall)
return {
'samples': result[0],
'precision': precision,
'recall': recall,
'f1_score': f1_score
}实现难易度: ⭐⭐ (简单)
- API简单直观
- 训练速度快
- 适合快速原型开发
6. AllenNLP
技术栈
- 核心语言: Python
- 深度学习: PyTorch
- 配置: JSON/YAML配置文件
- 部署: REST API, Docker支持
优点
- 研究导向: 面向NLP研究设计
- 模块化: 高度可配置和可扩展
- 实验管理: 内置实验跟踪
- 可复现: 配置文件确保实验可重复
- 预训练模型: 丰富的预训练模型库
技术特点
- 声明式配置系统
- 自动梯度累积和混合精度
- 内置的数据加载和批处理
- 模型解释和可视化工具
项目目录结构
allennlp/
├── allennlp/
│ ├── models/ # 模型定义
│ ├── modules/ # 神经网络模块
│ ├── data/ # 数据处理
│ │ ├── dataset_readers/
│ │ └── token_indexers/
│ ├── training/ # 训练相关
│ └── predictors/ # 预测器
├── training_config/ # 训练配置文件
├── tests/ # 测试代码
└── models/ # 预训练模型开发示例
python
from allennlp.data import DatasetReader, Instance
from allennlp.data.fields import TextField, LabelField
from allennlp.data.token_indexers import TokenIndexer, SingleIdTokenIndexer
from allennlp.data.tokenizers import Tokenizer, WhitespaceTokenizer
from allennlp.models import Model
from allennlp.modules import TextFieldEmbedder, Seq2VecEncoder
from allennlp.modules.text_field_embedders import BasicTextFieldEmbedder
from allennlp.modules.token_embedders import Embedding
from allennlp.modules.seq2vec_encoders import BagOfEmbeddingsEncoder
from allennlp.training.metrics import CategoricalAccuracy
from allennlp.training import GradientDescentTrainer
from allennlp.training.optimizers import AdamOptimizer
from allennlp.predictors import Predictor
import torch
import torch.nn as nn
from typing import Dict, Any
import json
# 自定义数据读取器
@DatasetReader.register("text_classification")
class TextClassificationDatasetReader(DatasetReader):
def __init__(self,
tokenizer: Tokenizer = None,
token_indexers: Dict[str, TokenIndexer] = None,
max_tokens: int = None):
super().__init__()
self.tokenizer = tokenizer or WhitespaceTokenizer()
self.token_indexers = token_indexers or {
"tokens": SingleIdTokenIndexer()
}
self.max_tokens = max_tokens
def text_to_instance(self, text: str, label: str = None) -> Instance:
tokens = self.tokenizer.tokenize(text)
if self.max_tokens:
tokens = tokens[:self.max_tokens]
text_field = TextField(tokens, self.token_indexers)
fields = {"text": text_field}
if label:
fields["label"] = LabelField(label)
return Instance(fields)
def _read(self, file_path: str):
with open(file_path, 'r', encoding='utf-8') as file:
for line in file:
data = json.loads(line)
yield self.text_to_instance(
data["text"],
data.get("label")
)
# 自定义模型
@Model.register("simple_classifier")
class SimpleClassifier(Model):
def __init__(self,
vocab: Vocabulary,
text_field_embedder: TextFieldEmbedder,
encoder: Seq2VecEncoder,
classifier_feedforward: FeedForward = None,
dropout: float = 0.0):
super().__init__(vocab)
self.text_field_embedder = text_field_embedder
self.encoder = encoder
self.dropout = nn.Dropout(dropout)
if classifier_feedforward:
self.classifier_feedforward = classifier_feedforward
else:
self.classifier_feedforward = nn.Linear(
encoder.get_output_dim(),
vocab.get_vocab_size("labels")
)
self.accuracy = CategoricalAccuracy()
self.loss_function = nn.CrossEntropyLoss()
def forward(self,
text: Dict[str, torch.Tensor],
label: torch.Tensor = None) -> Dict[str, Any]:
# 词嵌入
embedded_text = self.text_field_embedder(text)
# 编码
encoded_text = self.encoder(embedded_text)
# Dropout
encoded_text = self.dropout(encoded_text)
# 分类
logits = self.classifier_feedforward(encoded_text)
probs = torch.softmax(logits, dim=-1)
output = {"logits": logits, "probs": probs}
if label is not None:
loss = self.loss_function(logits, label)
output["loss"] = loss
self.accuracy(logits, label)
return output
def get_metrics(self, reset: bool = False) -> Dict[str, float]:
return {"accuracy": self.accuracy.get_metric(reset)}
# 配置文件定义 (config.json)
config = {
"dataset_reader": {
"type": "text_classification",
"tokenizer": {
"type": "whitespace"
},
"token_indexers": {
"tokens": {
"type": "single_id"
}
},
"max_tokens": 100
},
"train_data_path": "train.jsonl",
"validation_data_path": "valid.jsonl",
"model": {
"type": "simple_classifier",
"text_field_embedder": {
"token_embedders": {
"tokens": {
"type": "embedding",
"embedding_dim": 100,
"vocab_namespace": "tokens"
}
}
},
"encoder": {
"type": "bag_of_embeddings",
"embedding_dim": 100
},
"dropout": 0.2
},
"data_loader": {
"batch_size": 32,
"shuffle": True
},
"trainer": {
"type": "gradient_descent",
"optimizer": {
"type": "adam",
"lr": 0.001
},
"num_epochs": 10,
"patience": 3,
"validation_metric": "+accuracy"
}
}
# 训练和使用示例
class AllenNLPPipeline:
def __init__(self, config_path: str):
self.config_path = config_path
self.model = None
self.predictor = None
def train_model(self):
"""使用配置文件训练模型"""
from allennlp.commands.train import train_model_from_file
# 训练模型
train_model_from_file(
parameter_filename=self.config_path,
serialization_dir="./model_output"
)
# 加载训练好的模型
from allennlp.models.archival import load_archive
archive = load_archive("./model_output/model.tar.gz")
self.model = archive.model
return self.model
def create_predictor(self):
"""创建预测器"""
@Predictor.register("text_classifier_predictor")
class TextClassifierPredictor(Predictor):
def predict(self, text: str) -> Dict[str, Any]:
instance = self._dataset_reader.text_to_instance(text)
output = self.predict_instance(instance)
# 获取预测标签
label_vocab = self.model.vocab.get_index_to_token_vocabulary("labels")
predicted_label = label_vocab[output["logits"].argmax().item()]
return {
"text": text,
"predicted_label": predicted_label,
"probabilities": output["probs"].tolist(),
"logits": output["logits"].tolist()
}
if self.model:
self.predictor = TextClassifierPredictor(
model=self.model,
dataset_reader=TextClassificationDatasetReader()
)
return self.predictor
# 使用示例
def demo_allennlp():
# 准备训练数据
train_data = [
{"text": "I love this movie", "label": "positive"},
{"text": "This film is terrible", "label": "negative"},
{"text": "Great acting and plot", "label": "positive"},
{"text": "Waste of time", "label": "negative"}
]
# 保存为JSONL格式
with open("train.jsonl", "w") as f:
for item in train_data:
f.write(json.dumps(item) + "\n")
with open("valid.jsonl", "w") as f:
for item in train_data: # 简化示例，实际应该用不同的验证集
f.write(json.dumps(item) + "\n")
# 保存配置文件
with open("config.json", "w") as f:
json.dump(config, f, indent=2)
# 训练模型
pipeline = AllenNLPPipeline("config.json")
model = pipeline.train_model()
# 创建预测器
predictor = pipeline.create_predictor()
# 预测
result = predictor.predict("This movie is amazing!")
print(f"Text: {result['text']}")
print(f"Predicted: {result['predicted_label']}")
print(f"Probabilities: {result['probabilities']}")实现难易度: ⭐⭐⭐⭐⭐ (很难)
- 需要深入理解PyTorch
- 配置系统复杂
- 适合有经验的研究人员
7. Flair
技术栈
- 核心语言: Python
- 深度学习: PyTorch
- 词嵌入: 上下文字符串嵌入
- 模型: BiLSTM-CRF, Transformer
优点
- 先进嵌入: 创新的上下文字符串嵌入
- 易于使用: 简洁的API设计
- 多任务: 支持多种NLP任务
- 多语言: 广泛的语言支持
- 社区活跃: 持续更新和改进
技术特点
- 字符级语言模型嵌入
- 堆叠式嵌入组合
- 序列标注优化
- 预训练模型丰富
项目目录结构
flair/
├── flair/
│ ├── data/ # 数据处理
│ ├── datasets/ # 数据集
│ ├── embeddings/ # 词嵌入
│ ├── models/ # 模型实现
│ ├── trainers/ # 训练器
│ └── nn/ # 神经网络层
├── resources/ # 预训练资源
└── tests/ # 测试代码开发示例
python
import flair
from flair.data import Sentence, Corpus, Token
from flair.datasets import ColumnCorpus
from flair.embeddings import (
WordEmbeddings,
FlairEmbeddings,
StackedEmbeddings,
TransformerWordEmbeddings
)
from flair.models import SequenceTagger, TextClassifier
from flair.trainers import ModelTrainer
from flair.nn import Classifier
from typing import List, Dict
# 命名实体识别示例
class FlairNERPipeline:
def __init__(self):
self.model = None
def load_pretrained_model(self, model_name='ner'):
"""加载预训练的NER模型"""
from flair.models import SequenceTagger
# 可选模型: 'ner', 'ner-fast', 'ner-ontonotes', 'ner-ontonotes-fast'
self.model = SequenceTagger.load(model_name)
return self.model
def predict_entities(self, texts: List[str]) -> List[Dict]:
"""预测实体"""
if not self.model:
self.load_pretrained_model()
results = []
for text in texts:
sentence = Sentence(text)
self.model.predict(sentence)
entities = []
for entity in sentence.get_spans('ner'):
entities.append({
'text': entity.text,
'label': entity.tag,
'confidence': entity.score,
'start': entity.start_position,
'end': entity.end_position
})
results.append({
'text': text,
'entities': entities,
'tokens': [token.text for token in sentence.tokens]
})
return results
# 自定义NER模型训练
class CustomNERTrainer:
def __init__(self):
self.corpus = None
self.model = None
def prepare_corpus(self, train_file, dev_file, test_file):
"""准备训练语料"""
# 数据格式: token label 每行一个token，空行分隔句子
columns = {0: 'text', 1: 'ner'}
self.corpus = ColumnCorpus(
data_folder='./data',
column_format=columns,
train_file=train_file,
dev_file=dev_file,
test_file=test_file
)
return self.corpus
def create_model(self, embedding_type='flair'):
"""创建模型"""
# 创建词嵌入
if embedding_type == 'flair':
embeddings = StackedEmbeddings([
WordEmbeddings('glove'),
FlairEmbeddings('news-forward'),
FlairEmbeddings('news-backward'),
])
elif embedding_type == 'transformer':
embeddings = TransformerWordEmbeddings('bert-base-uncased')
else:
embeddings = WordEmbeddings('glove')
# 创建序列标注器
self.model = SequenceTagger(
hidden_size=256,
embeddings=embeddings,
tag_dictionary=self.corpus.make_tag_dictionary(tag_type='ner'),
tag_type='ner'
)
return self.model
def train_model(self, max_epochs=150):
"""训练模型"""
trainer = ModelTrainer(self.model, self.corpus)
trainer.train(
base_path='./models/ner',
learning_rate=0.1,
mini_batch_size=32,
max_epochs=max_epochs,
patience=3,
checkpoint=True,
embeddings_storage_mode='gpu'
)
return self.model
# 文本分类示例
class FlairTextClassifier:
def __init__(self):
self.model = None
def load_sentiment_model(self):
"""加载情感分析模型"""
self.model = TextClassifier.load('en-sentiment')
return self.model
def predict_sentiment(self, texts: List[str]) -> List[Dict]:
"""预测情感"""
if not self.model:
self.load_sentiment_model()
results = []
for text in texts:
sentence = Sentence(text)
self.model.predict(sentence)
results.append({
'text': text,
'sentiment': sentence.labels[0].value,
'confidence': sentence.labels[0].score
})
return results
# 自定义分类器训练
class CustomTextClassifier:
def __init__(self):
self.corpus = None
self.model = None
def prepare_classification_corpus(self, train_data, dev_data, test_data):
"""准备分类语料"""
from flair.data import Corpus
from flair.datasets import ClassificationCorpus
# 保存数据为文本文件
self._save_classification_data(train_data, 'train.txt')
self._save_classification_data(dev_data, 'dev.txt')
self._save_classification_data(test_data, 'test.txt')
# 创建语料库
self.corpus = ClassificationCorpus(
data_folder='./data',
train_file='train.txt',
dev_file='dev.txt',
test_file='test.txt'
)
return self.corpus
def _save_classification_data(self, data, filename):
"""保存分类数据"""
with open(f'./data/{filename}', 'w', encoding='utf-8') as f:
for item in data:
# 格式: __label__类别 文本内容
f.write(f"__label__{item['label']} {item['text']}\n")
def create_classifier(self, embedding_type='flair'):
"""创建分类器"""
if embedding_type == 'flair':
embeddings = StackedEmbeddings([
WordEmbeddings('glove'),
FlairEmbeddings('news-forward-fast'),
FlairEmbeddings('news-backward-fast'),
])
elif embedding_type == 'transformer':
embeddings = TransformerWordEmbeddings('bert-base-uncased')
else:
embeddings = WordEmbeddings('glove')
# 创建文档嵌入
from flair.embeddings import DocumentRNNEmbeddings
document_embeddings = DocumentRNNEmbeddings(
embeddings,
hidden_size=512,
reproject_words=True,
reproject_words_dimension=256,
)
# 创建分类器
self.model = TextClassifier(
document_embeddings,
label_dictionary=self.corpus.make_label_dictionary(),
multi_label=False
)
return self.model
def train_classifier(self, max_epochs=150):
"""训练分类器"""
trainer = ModelTrainer(self.model, self.corpus)
trainer.train(
base_path='./models/classifier',
learning_rate=0.1,
mini_batch_size=16,
anneal_factor=0.5,
patience=3,
max_epochs=max_epochs,
embeddings_storage_mode='gpu'
)
return self.model
# 综合使用示例
def demo_flair_pipeline():
# 1. NER预测
print("=== 命名实体识别 ===")
ner_pipeline = FlairNERPipeline()
texts = [
"Apple Inc. was founded by Steve Jobs in Cupertino.",
"Microsoft is located in Redmond, Washington."
]
ner_results = ner_pipeline.predict_entities(texts)
for result in ner_results:
print(f"Text: {result['text']}")
for entity in result['entities']:
print(f" - {entity['text']}: {entity['label']} ({entity['confidence']:.3f})")
print()
# 2. 情感分析
print("=== 情感分析 ===")
sentiment_classifier = FlairTextClassifier()
sentiment_texts = [
"I love this product!",
"This is the worst experience ever.",
"It's okay, nothing special."
]
sentiment_results = sentiment_classifier.predict_sentiment(sentiment_texts)
for result in sentiment_results:
print(f"Text: {result['text']}")
print(f"Sentiment: {result['sentiment']} ({result['confidence']:.3f})")
print()
# 3. 自定义模型训练示例
print("=== 自定义模型训练 ===")
# 准备训练数据
train_data = [
{"text": "This movie is fantastic!", "label": "positive"},
{"text": "I hate this film.", "label": "negative"},
{"text": "Great story and acting.", "label": "positive"},
{"text": "Boring and predictable.", "label": "negative"}
]
dev_data = train_data[:2] # 简化示例
test_data = train_data[2:]
# 创建自定义分类器
custom_classifier = CustomTextClassifier()
corpus = custom_classifier.prepare_classification_corpus(
train_data, dev_data, test_data
)
print(f"训练集大小: {len(corpus.train)}")
print(f"验证集大小: {len(corpus.dev)}")
print(f"测试集大小: {len(corpus.test)}")
print(f"标签: {corpus.make_label_dictionary().get_items()}")
# 高级特性示例
class FlairAdvancedFeatures:
def __init__(self):
pass
def multi_lingual_ner(self):
"""多语言NER"""
# 加载多语言模型
model = SequenceTagger.load('ner-multi')
# 处理不同语言文本
texts = {
'english': "Apple Inc. is located in California.",
'german': "Die BMW AG hat ihren Sitz in München.",
'spanish': "Real Madrid es un club de fútbol español."
}
results = {}
for lang, text in texts.items():
sentence = Sentence(text)
model.predict(sentence)
entities = [(entity.text, entity.tag, entity.score)
for entity in sentence.get_spans('ner')]
results[lang] = {
'text': text,
'entities': entities
}
return results
def embeddings_comparison(self, text: str):
"""比较不同词嵌入效果"""
from flair.embeddings import (
WordEmbeddings,
FlairEmbeddings,
ELMoEmbeddings,
TransformerWordEmbeddings
)
sentence = Sentence(text)
embedding_types = {
'glove': WordEmbeddings('glove'),
'flair_forward': FlairEmbeddings('news-forward-fast'),
'elmo': ELMoEmbeddings('original'),
'bert': TransformerWordEmbeddings('bert-base-uncased')
}
results = {}
for name, embedding in embedding_types.items():
try:
embedding.embed(sentence)
# 获取第一个token的嵌入向量维度
vector_dim = sentence[0].embedding.shape[0]
results[name] = {
'dimension': vector_dim,
'available': True
}
except Exception as e:
results[name] = {
'dimension': 0,
'available': False,
'error': str(e)
}
return results实现难易度: ⭐⭐⭐ (中等)
- API相对简单
- 预训练模型丰富
- 需要理解深度学习概念
8. Gensim
技术栈
- 核心语言: Python + Cython
- 算法: Word2Vec, Doc2Vec, FastText, LDA
- 优化: 内存高效，支持大规模数据
- 存储: 支持多种模型存储格式
优点
- 内存效率: 流式处理大规模语料
- 算法丰富: 经典的主题建模和词向量算法
- 易于使用: 简洁的API接口
- 可扩展: 支持分布式计算
- 文档详细: 丰富的教程和示例
技术特点
- 无监督学习专长
- 增量学习支持
- 相似性查询优化
- 主题建模工具链
项目目录结构
gensim/
├── gensim/
│ ├── models/ # 模型实现
│ │ ├── word2vec.py
│ │ ├── doc2vec.py
│ │ ├── ldamodel.py
│ │ └── fasttext.py
│ ├── corpora/ # 语料库处理
│ ├── similarities/ # 相似性计算
│ ├── parsing/ # 文本解析
│ └── utils.py # 工具函数
├── docs/ # 文档
└── tutorials/ # 教程开发示例
python
import gensim
from gensim.models import Word2Vec, Doc2Vec, LdaModel, FastText
from gensim.corpora import Dictionary
from gensim import similarities
from gensim.models.doc2vec import TaggedDocument
from gensim.parsing.preprocessing import preprocess_string
import numpy as np
from typing import List, Dict, Any
class GensimNLPPipeline:
def __init__(self):
self.word2vec_model = None
self.doc2vec_model = None
self.lda_model = None
self.dictionary = None
def preprocess_texts(self, texts: List[str]) -> List[List[str]]:
"""文本预处理"""
processed_texts = []
for text in texts:
# 使用gensim内置预处理
processed = preprocess_string(text)
processed_texts.append(processed)
return processed_texts
def train_word2vec(self, sentences: List[List[str]], **kwargs):
"""训练Word2Vec模型"""
default_params = {
'vector_size': 100, # 向量维度
'window': 5, # 上下文窗口大小
'min_count': 1, # 最小词频
'workers': 4, # 并行训练线程数
'sg': 0, # 0=CBOW, 1=Skip-gram
'hs': 0, # 0=negative sampling, 1=hierarchical softmax
'negative': 5, # negative sampling参数
'alpha': 0.025, # 学习率
'epochs': 100 # 训练轮数
}
default_params.update(kwargs)
self.word2vec_model = Word2Vec(sentences, **default_params)
return self.word2vec_model
def train_doc2vec(self, documents: List[str], **kwargs):
"""训练Doc2Vec模型"""
# 创建TaggedDocument
tagged_docs = [TaggedDocument(doc.split(), [i])
for i, doc in enumerate(documents)]
default_params = {
'vector_size': 100,
'window': 5,
'min_count': 1,
'workers': 4,
'dm': 1, # 1=PV-DM, 0=PV-DBOW
'epochs': 100
}
default_params.update(kwargs)
self.doc2vec_model = Doc2Vec(tagged_docs, **default_params)
return self.doc2vec_model
def train_lda(self, texts: List[List[str]], num_topics: int = 10, **kwargs):
"""训练LDA主题模型"""
# 创建词典
self.dictionary = Dictionary(texts)
# 过滤极端词汇
self.dictionary.filter_extremes(
no_below=2, # 过滤出现次数少于2次的词
no_above=0.5 # 过滤出现在超过50%文档中的词
)
# 转换为词袋模型
corpus = [self.dictionary.doc2bow(text) for text in texts]
default_params = {
'num_topics': num_topics,
'id2word': self.dictionary,
'passes': 10,
'alpha': 'auto',
'per_word_topics': True,
'random_state': 42
}
default_params.update(kwargs)
self.lda_model = LdaModel(corpus, **default_params)
return self.lda_model, corpus
def find_similar_words(self, word: str, topn: int = 10):
"""查找相似词"""
if not self.word2vec_model:
raise ValueError("Word2Vec模型未训练")
try:
similar_words = self.word2vec_model.wv.most_similar(word, topn=topn)
return similar_words
except KeyError:
return f"词汇'{word}'不在词典中"
def get_document_similarity(self, doc1: str, doc2: str):
"""计算文档相似度"""
if not self.doc2vec_model:
raise ValueError("Doc2Vec模型未训练")
# 推断文档向量
vec1 = self.doc2vec_model.infer_vector(doc1.split())
vec2 = self.doc2vec_model.infer_vector(doc2.split())
# 计算余弦相似度
similarity = np.dot(vec1, vec2) / (np.linalg.norm(vec1) * np.linalg.norm(vec2))
return similarity
def get_document_topics(self, text: List[str]):
"""获取文档主题分布"""
if not self.lda_model or not self.dictionary:
raise ValueError("LDA模型未训练")
# 转换为词袋格式
bow = self.dictionary.doc2bow(text)
# 获取主题分布
doc_topics = self.lda_model.get_document_topics(bow)
return doc_topics
def print_topics(self, num_words: int = 10):
"""打印主题"""
if not self.lda_model:
raise ValueError("LDA模型未训练")
topics = self.lda_model.print_topics(num_words=num_words)
for topic_id, topic in topics:
print(f"主题 {topic_id}: {topic}")
return topics
# 高级功能类
class AdvancedGensimFeatures:
def __init__(self):
pass
def build_similarity_index(self, corpus, model):
"""构建相似性索引"""
# 获取文档向量
if hasattr(model, 'docvecs'):
# Doc2Vec模型
index = similarities.MatrixSimilarity(
[model.docvecs[i] for i in range(len(corpus))]
)
else:
# LDA模型
index = similarities.MatrixSimilarity(
[model[doc] for doc in corpus]
)
return index
def incremental_training(self, model, new_sentences):
"""增量训练"""
if isinstance(model, Word2Vec):
# 更新词汇表
model.build_vocab(new_sentences, update=True)
# 继续训练
model.train(new_sentences, total_examples=len(new_sentences), epochs=model.epochs)
return model
def model_evaluation(self, model, test_sentences):
"""模型评估"""
if isinstance(model, Word2Vec):
# 词汇类比测试
try:
accuracy = model.wv.evaluate_word_analogies('questions-words.txt')
return accuracy
except FileNotFoundError:
print("需要下载词汇类比测试数据集")
return None
return None
# 使用示例
def demo_gensim_pipeline():
# 示例文档
documents = [
"machine learning is a subset of artificial intelligence",
"deep learning uses neural networks with multiple layers",
"natural language processing deals with text data",
"computer vision focuses on image recognition",
"data mining extracts patterns from large datasets",
"statistical analysis helps understand data distributions",
"python is popular for data science projects",
"tensorflow and pytorch are deep learning frameworks"
]
# 初始化管道
pipeline = GensimNLPPipeline()
# 文本预处理
processed_docs = pipeline.preprocess_texts(documents)
print("预处理后的文档:")
for i, doc in enumerate(processed_docs[:2]):
print(f"文档 {i}: {doc}")
print()
# 1. 训练Word2Vec
print("=== Word2Vec训练 ===")
word2vec_model = pipeline.train_word2vec(
processed_docs,
vector_size=50,
window=3,
min_count=1,
epochs=50
)
# 查找相似词
try:
similar_words = pipeline.find_similar_words("learning", topn=5)
print(f"与'learning'最相似的词: {similar_words}")
except:
print("词汇不在词典中")
# 词汇运算
try:
result = word2vec_model.wv.most_similar(
positive=['machine', 'intelligence'],
negative=['computer'],
topn=3
)
print(f"machine + intelligence - computer = {result}")
except:
print("词汇运算失败")
print()
# 2. 训练Doc2Vec
print("=== Doc2Vec训练 ===")
doc2vec_model = pipeline.train_doc2vec(
documents,
vector_size=50,
epochs=50
)
# 计算文档相似度
similarity = pipeline.get_document_similarity(
documents[0], documents[1]
)
print(f"文档1和文档2的相似度: {similarity:.3f}")
# 查找相似文档
test_doc = "artificial intelligence and machine learning"
inferred_vector = doc2vec_model.infer_vector(test_doc.split())
similar_docs = doc2vec_model.dv.most_similar([inferred_vector], topn=3)
print(f"与'{test_doc}'最相似的文档:")
for doc_id, similarity in similar_docs:
print(f" 文档{doc_id}: {documents[doc_id][:50]}... (相似度: {similarity:.3f})")
print()
# 3. 训练LDA主题模型
print("=== LDA主题建模 ===")
lda_model, corpus = pipeline.train_lda(
processed_docs,
num_topics=3,
passes=50
)
# 打印主题
topics = pipeline.print_topics(num_words=5)
print()
# 获取文档主题分布
doc_topics = pipeline.get_document_topics(processed_docs[0])
print(f"第一个文档的主题分布:")
for topic_id, prob in doc_topics:
print(f" 主题{topic_id}: {prob:.3f}")
print()
# 4. 相似性搜索
print("=== 相似性搜索 ===")
advanced_features = AdvancedGensimFeatures()
# 构建LDA相似性索引
similarity_index = advanced_features.build_similarity_index(corpus, lda_model)
# 查询相似文档
query_doc = processed_docs[0]
query_bow = pipeline.dictionary.doc2bow(query_doc)
query_lda = lda_model[query_bow]
# 计算相似度
sims = similarity_index[query_lda]
sorted_sims = sorted(enumerate(sims), key=lambda x: x[1], reverse=True)
print(f"与查询文档最相似的文档:")
for doc_id, similarity in sorted_sims[:3]:
print(f" 文档{doc_id}: {' '.join(processed_docs[doc_id][:8])}... (相似度: {similarity:.3f})")
print()
# 实际应用示例
class ProductRecommendationSystem:
"""基于Gensim的产品推荐系统"""
def __init__(self):
self.pipeline = GensimNLPPipeline()
self.products = []
self.product_vectors = None
self.similarity_index = None
def add_products(self, products: List[Dict]):
"""添加产品数据"""
self.products = products
# 提取产品描述
descriptions = [p['description'] for p in products]
# 预处理
processed_descriptions = self.pipeline.preprocess_texts(descriptions)
# 训练Doc2Vec模型
self.pipeline.train_doc2vec(descriptions, vector_size=100, epochs=100)
# 构建相似性索引
self.product_vectors = [
self.pipeline.doc2vec_model.infer_vector(desc.split())
for desc in descriptions
]
self.similarity_index = similarities.MatrixSimilarity(self.product_vectors)
def recommend_products(self, user_query: str, topn: int = 5):
"""基于用户查询推荐产品"""
if not self.pipeline.doc2vec_model:
raise ValueError("模型未训练")
# 推断查询向量
query_vector = self.pipeline.doc2vec_model.infer_vector(user_query.split())
# 计算相似度
similarities_scores = self.similarity_index[query_vector]
# 排序并获取推荐
recommendations = sorted(
enumerate(similarities_scores),
key=lambda x: x[1],
reverse=True
)[:topn]
results = []
for product_id, score in recommendations:
results.append({
'product': self.products[product_id],
'similarity_score': score
})
return results
# 新闻分类系统
class NewsClassificationSystem:
"""基于主题模型的新闻分类系统"""
def __init__(self):
self.pipeline = GensimNLPPipeline()
self.topic_labels = {}
def train_on_news_corpus(self, news_articles: List[Dict], num_topics: int = 10):
"""在新闻语料上训练"""
# 提取新闻内容
contents = [article['content'] for article in news_articles]
# 预处理
processed_contents = self.pipeline.preprocess_texts(contents)
# 训练LDA模型
lda_model, corpus = self.pipeline.train_lda(
processed_contents,
num_topics=num_topics,
passes=50,
alpha=0.1,
beta=0.01
)
return lda_model, corpus
def assign_topic_labels(self, topic_labels: Dict[int, str]):
"""为主题分配标签"""
self.topic_labels = topic_labels
def classify_news(self, news_content: str):
"""分类新闻"""
if not self.pipeline.lda_model:
raise ValueError("LDA模型未训练")
# 预处理
processed_content = self.pipeline.preprocess_texts([news_content])[0]
# 获取主题分布
doc_topics = self.pipeline.get_document_topics(processed_content)
# 找到主导主题
dominant_topic = max(doc_topics, key=lambda x: x[1])
topic_id, probability = dominant_topic
result = {
'topic_id': topic_id,
'probability': probability,
'topic_label': self.topic_labels.get(topic_id, f"主题{topic_id}"),
'all_topics': doc_topics
}
return result
# 文档相似性检测系统
class DocumentSimilaritySystem:
"""文档相似性检测系统"""
def __init__(self):
self.pipeline = GensimNLPPipeline()
self.document_database = []
def add_documents(self, documents: List[str]):
"""添加文档到数据库"""
self.document_database.extend(documents)
# 重新训练模型
processed_docs = self.pipeline.preprocess_texts(self.document_database)
# 训练Word2Vec和Doc2Vec
self.pipeline.train_word2vec(processed_docs, vector_size=200, epochs=100)
self.pipeline.train_doc2vec(self.document_database, vector_size=200, epochs=100)
def find_similar_documents(self, query_document: str, threshold: float = 0.7):
"""查找相似文档"""
if not self.pipeline.doc2vec_model:
raise ValueError("模型未训练")
# 推断查询文档向量
query_vector = self.pipeline.doc2vec_model.infer_vector(query_document.split())
similar_docs = []
for i, doc in enumerate(self.document_database):
doc_vector = self.pipeline.doc2vec_model.dv[i]
similarity = np.dot(query_vector, doc_vector) / (
np.linalg.norm(query_vector) * np.linalg.norm(doc_vector)
)
if similarity > threshold:
similar_docs.append({
'document_id': i,
'document': doc,
'similarity': similarity
})
# 按相似度排序
similar_docs.sort(key=lambda x: x['similarity'], reverse=True)
return similar_docs
def detect_plagiarism(self, document: str, threshold: float = 0.8):
"""抄袭检测"""
similar_docs = self.find_similar_documents(document, threshold)
if similar_docs:
return {
'is_plagiarized': True,
'max_similarity': similar_docs[0]['similarity'],
'similar_documents': similar_docs
}
else:
return {
'is_plagiarized': False,
'max_similarity': 0.0,
'similar_documents': []
}
# 综合演示
def comprehensive_gensim_demo():
print("=== Gensim综合应用演示 ===\n")
# 1. 产品推荐系统
print("1. 产品推荐系统")
products = [
{'id': 1, 'name': 'iPhone 14', 'description': 'smartphone with advanced camera system'},
{'id': 2, 'name': 'MacBook Pro', 'description': 'laptop computer for professional work'},
{'id': 3, 'name': 'iPad Air', 'description': 'tablet device for entertainment and productivity'},
{'id': 4, 'name': 'Samsung Galaxy', 'description': 'android smartphone with large display'},
{'id': 5, 'name': 'Dell XPS', 'description': 'windows laptop for business and gaming'}
]
recommendation_system = ProductRecommendationSystem()
recommendation_system.add_products(products)
user_query = "professional laptop for work"
recommendations = recommendation_system.recommend_products(user_query, topn=3)
print(f"用户查询: '{user_query}'")
print("推荐产品:")
for i, rec in enumerate(recommendations, 1):
product = rec['product']
score = rec['similarity_score']
print(f" {i}. {product['name']}: {product['description']} (相似度: {score:.3f})")
print()
# 2. 新闻分类系统
print("2. 新闻分类系统")
news_articles = [
{'content': 'stock market reaches new highs as technology companies surge'},
{'content': 'new breakthrough in artificial intelligence research announced'},
{'content': 'football team wins championship in dramatic final game'},
{'content': 'government announces new economic policy to boost growth'},
{'content': 'scientists discover new species in deep ocean exploration'},
{'content': 'latest smartphone features advanced camera technology'}
]
news_classifier = NewsClassificationSystem()
lda_model, corpus = news_classifier.train_on_news_corpus(news_articles, num_topics=3)
# 手动分配主题标签
topic_labels = {0: '科技', 1: '经济', 2: '体育'}
news_classifier.assign_topic_labels(topic_labels)
test_news = "new artificial intelligence model shows promising results in medical diagnosis"
classification_result = news_classifier.classify_news(test_news)
print(f"测试新闻: '{test_news}'")
print(f"分类结果: {classification_result['topic_label']} (置信度: {classification_result['probability']:.3f})")
print()
# 3. 文档相似性检测
print("3. 文档相似性检测系统")
documents = [
"Machine learning is a subset of artificial intelligence",
"Deep learning uses neural networks with multiple layers",
"Data science involves statistical analysis of large datasets",
"Computer vision focuses on image recognition and processing"
]
similarity_system = DocumentSimilaritySystem()
similarity_system.add_documents(documents)
query_doc = "Artificial intelligence includes machine learning algorithms"
similar_docs = similarity_system.find_similar_documents(query_doc, threshold=0.3)
print(f"查询文档: '{query_doc}'")
print("相似文档:")
for doc in similar_docs:
print(f" 相似度 {doc['similarity']:.3f}: {doc['document']}")
print()
# 抄袭检测示例
potential_plagiarism = "Machine learning algorithms are part of artificial intelligence"
plagiarism_result = similarity_system.detect_plagiarism(potential_plagiarism, threshold=0.5)
print(f"抄袭检测: '{potential_plagiarism}'")
print(f"是否抄袭: {plagiarism_result['is_plagiarized']}")
print(f"最高相似度: {plagiarism_result['max_similarity']:.3f}")
if __name__ == "__main__":
# 运行基础演示
demo_gensim_pipeline()
print("\n" + "="*50 + "\n")
# 运行综合应用演示
comprehensive_gensim_demo()实现难易度: ⭐⭐⭐ (中等)
- 算法概念需要理解
- 参数调优需要经验
- 适合传统NLP任务
9. TextBlob
技术栈
- 核心语言: Python
- 底层: NLTK和pattern库
- 简化接口: 面向对象的简单API
- 扩展: 支持多种语言包
优点
- 极简API: 最简单的NLP库之一
- 快速上手: 几行代码即可完成复杂任务
- 功能全面: 覆盖基础NLP任务
- 多语言: 支持多语言处理
- 教学友好: 适合初学者学习
技术特点
- 统一的字符串处理接口
- 内置的机器学习分类器
- 简化的语言检测
- 直观的API设计
项目目录结构
textblob/
├── textblob/
│ ├── blob.py # 核心Blob类
│ ├── classifiers.py # 分类器
│ ├── sentiments.py # 情感分析
│ ├── taggers.py # 词性标注
│ ├── tokenizers.py # 分词器
│ └── translate.py # 翻译
├── tests/ # 测试代码
└── docs/ # 文档开发示例
python
from textblob import TextBlob, Word, Sentence
from textblob.classifiers import NaiveBayesClassifier, DecisionTreeClassifier
from textblob.sentiments import NaiveBayesAnalyzer
import random
class TextBlobPipeline:
"""TextBlob完整处理管道"""
def __init__(self):
self.classifier = None
def basic_analysis(self, text: str):
"""基础文本分析"""
blob = TextBlob(text)
analysis = {
'original_text': text,
'sentences': [str(sentence) for sentence in blob.sentences],
'words': [str(word) for word in blob.words],
'noun_phrases': list(blob.noun_phrases),
'tags': blob.tags,
'sentiment': {
'polarity': blob.sentiment.polarity,
'subjectivity': blob.sentiment.subjectivity
},
'language': blob.detect_language()
}
return analysis
def advanced_word_analysis(self, text: str):
"""高级词汇分析"""
blob = TextBlob(text)
word_analysis = []
for word in blob.words:
word_obj = Word(word)
word_info = {
'word': str(word),
'lemma': word_obj.lemmatize(),
'stem': word_obj.stem(),
'definitions': word_obj.definitions[:3], # 前3个定义
'synsets': [str(synset) for synset in word_obj.synsets[:3]]
}
word_analysis.append(word_info)
return word_analysis
def sentiment_analysis_detailed(self, text: str):
"""详细情感分析"""
# 使用默认分析器
blob_default = TextBlob(text)
# 使用NaiveBayes分析器
blob_nb = TextBlob(text, analyzer=NaiveBayesAnalyzer())
sentiment_analysis = {
'text': text,
'default_analyzer': {
'polarity': blob_default.sentiment.polarity,
'subjectivity': blob_default.sentiment.subjectivity,
'interpretation': self._interpret_sentiment(
blob_default.sentiment.polarity,
blob_default.sentiment.subjectivity
)
},
'naivebayes_analyzer': {
'classification': blob_nb.sentiment.classification,
'p_pos': blob_nb.sentiment.p_pos,
'p_neg': blob_nb.sentiment.p_neg
}
}
return sentiment_analysis
def _interpret_sentiment(self, polarity: float, subjectivity: float):
"""解释情感分析结果"""
# 极性解释
if polarity > 0.1:
polarity_desc = "积极"
elif polarity < -0.1:
polarity_desc = "消极"
else:
polarity_desc = "中性"
# 主观性解释
if subjectivity > 0.6:
subjectivity_desc = "主观"
elif subjectivity < 0.3:
subjectivity_desc = "客观"
else:
subjectivity_desc = "一般"
return f"{polarity_desc}, {subjectivity_desc}"
def translation_service(self, text: str, target_language: str = 'zh'):
"""翻译服务"""
blob = TextBlob(text)
try:
translated = blob.translate(to=target_language)
translation_result = {
'original': text,
'original_language': blob.detect_language(),
'translated': str(translated),
'target_language': target_language,
'confidence': getattr(translated, 'confidence', None)
}
return translation_result
except Exception as e:
return {
'error': str(e),
'original': text
}
def spelling_correction(self, text: str):
"""拼写纠错"""
blob = TextBlob(text)
correction_result = {
'original': text,
'corrected': str(blob.correct()),
'word_corrections': []
}
# 逐词检查
for word in blob.words:
corrected_word = word.correct()
if str(word) != str(corrected_word):
correction_result['word_corrections'].append({
'original': str(word),
'corrected': str(corrected_word)
})
return correction_result
# 自定义分类器
class CustomTextClassifier:
"""基于TextBlob的自定义文本分类器"""
def __init__(self):
self.classifier = None
def prepare_training_data(self, data_list):
"""准备训练数据"""
# 数据格式: [(text, label), (text, label), ...]
return data_list
def train_naive_bayes(self, training_data):
"""训练朴素贝叶斯分类器"""
self.classifier = NaiveBayesClassifier(training_data)
return self.classifier
def train_decision_tree(self, training_data):
"""训练决策树分类器"""
self.classifier = DecisionTreeClassifier(training_data)
return self.classifier
def classify_text(self, text: str):
"""分类文本"""
if not self.classifier:
raise ValueError("分类器未训练")
blob = TextBlob(text, classifier=self.classifier)
result = {
'text': text,
'classification': blob.classify(),
'probability_distribution': blob.classify_prob()
}
return result
def evaluate_classifier(self, test_data):
"""评估分类器"""
if not self.classifier:
raise ValueError("分类器未训练")
accuracy = self.classifier.accuracy(test_data)
# 详细评估
correct = 0
total = len(test_data)
predictions = []
for text, actual_label in test_data:
predicted_label = self.classifier.classify(text)
predictions.append({
'text': text,
'actual': actual_label,
'predicted': predicted_label,
'correct': actual_label == predicted_label
})
if actual_label == predicted_label:
correct += 1
evaluation_result = {
'accuracy': accuracy,
'correct': correct,
'total': total,
'predictions': predictions[:10] # 显示前10个预测结果
}
return evaluation_result
# 高级特性示例
class TextBlobAdvancedFeatures:
"""TextBlob高级特性"""
def __init__(self):
pass
def n_gram_analysis(self, text: str, n: int = 2):
"""N-gram分析"""
blob = TextBlob(text)
# 获取n-grams
n_grams = list(blob.ngrams(n=n))
# 统计频率
from collections import Counter
n_gram_freq = Counter(n_grams)
result = {
'text': text,
'n': n,
'n_grams': [' '.join(gram) for gram in n_grams],
'most_common': [
(' '.join(gram), freq)
for gram, freq in n_gram_freq.most_common(10)
]
}
return result
def word_inflection(self, word: str):
"""词汇变形"""
word_obj = Word(word)
inflections = {
'word': word,
'plural': word_obj.pluralize(),
'singular': word_obj.singularize(),
'lemma': word_obj.lemmatize(),
'stem': word_obj.stem()
}
# 如果是动词，获取不同时态
try:
inflections.update({
'past_tense': word_obj.lemmatize('v'),
'gerund': word_obj.lemmatize('v') + 'ing' # 简化处理
})
except:
pass
return inflections
def text_summarization_simple(self, text: str, sentence_count: int = 3):
"""简单文本摘要"""
blob = TextBlob(text)
sentences = blob.sentences
if len(sentences) <= sentence_count:
return {
'original_text': text,
'summary': text,
'sentence_count': len(sentences)
}
# 基于句子长度和词汇多样性的简单评分
sentence_scores = []
for i, sentence in enumerate(sentences):
score = len(sentence.words) # 基础分数：词汇数量
score += len(set(sentence.words.lower())) # 词汇多样性
sentence_scores.append((i, score, str(sentence)))
# 选择得分最高的句子
sentence_scores.sort(key=lambda x: x[1], reverse=True)
selected_sentences = sentence_scores[:sentence_count]
selected_sentences.sort(key=lambda x: x[0]) # 按原顺序排列
summary = ' '.join([sentence[2] for sentence in selected_sentences])
result = {
'original_text': text,
'summary': summary,
'original_sentence_count': len(sentences),
'summary_sentence_count': sentence_count,
'selected_sentences': [
{
'index': idx,
'score': score,
'sentence': sent
}
for idx, score, sent in selected_sentences
]
}
return result
# 实际应用示例
def demo_textblob_applications():
print("=== TextBlob应用演示 ===\n")
# 初始化管道
pipeline = TextBlobPipeline()
# 1. 基础文本分析
print("1. 基础文本分析")
sample_text = """
I love this new smartphone! The camera quality is amazing and the battery life is excellent.
However, the price might be a bit too expensive for some people.
"""
basic_result = pipeline.basic_analysis(sample_text.strip())
print(f"原文: {basic_result['original_text']}")
print(f"语言: {basic_result['language']}")
print(f"情感极性: {basic_result['sentiment']['polarity']:.3f}")
print(f"主观性: {basic_result['sentiment']['subjectivity']:.3f}")
print(f"名词短语: {basic_result['noun_phrases']}")
print()
# 2. 详细情感分析
print("2. 详细情感分析")
sentiment_texts = [
"I absolutely love this product!",
"This is the worst experience ever.",
"The weather is sunny today."
]
for text in sentiment_texts:
sentiment_result = pipeline.sentiment_analysis_detailed(text)
default_analysis = sentiment_result['default_analyzer']
print(f"文本: {text}")
print(f"情感: {default_analysis['interpretation']}")
print(f"极性: {default_analysis['polarity']:.3f}, "
f"主观性: {default_analysis['subjectivity']:.3f}")
print()
# 3. 拼写纠错
print("3. 拼写纠错")
misspelled_text = "I havve goood speling abilitiy"
correction_result = pipeline.spelling_correction(misspelled_text)
print(f"原文: {correction_result['original']}")
print(f"纠错: {correction_result['corrected']}")
if correction_result['word_corrections']:
print("单词纠错:")
for correction in correction_result['word_corrections']:
print(f" {correction['original']} -> {correction['corrected']}")
print()
# 4. 自定义分类器
print("4. 自定义文本分类")
# 准备训练数据
training_data = [
('I love this product', 'positive'),
('This is amazing quality', 'positive'),
('Great value for money', 'positive'),
('Excellent customer service', 'positive'),
('I hate this item', 'negative'),
('Poor quality material', 'negative'),
('Terrible experience', 'negative'),
('Complete waste of money', 'negative'),
('It is okay', 'neutral'),
('Average product', 'neutral'),
('Nothing special', 'neutral'),
('Standard quality', 'neutral')
]
# 分割训练和测试数据
random.shuffle(training_data)
split_point = int(len(training_data) * 0.8)
train_data = training_data[:split_point]
test_data = training_data[split_point:]
# 训练分类器
classifier = CustomTextClassifier()
classifier.train_naive_bayes(train_data)
# 评估分类器
evaluation = classifier.evaluate_classifier(test_data)
print(f"分类器准确率: {evaluation['accuracy']:.3f}")
print(f"正确预测: {evaluation['correct']}/{evaluation['total']}")
# 测试分类
test_texts = [
"This product exceeded my expectations",
"Not worth the price at all",
"It works fine but nothing extraordinary"
]
print("\n分类结果:")
for text in test_texts:
classification_result = classifier.classify_text(text)
print(f"文本: {text}")
print(f"分类: {classification_result['classification']}")
prob_dist = classification_result['probability_distribution']
print(f"置信度: {dict(prob_dist)}")
print()
# 5. 高级特性
print("5. 高级特性演示")
advanced_features = TextBlobAdvancedFeatures()
# N-gram分析
ngram_text = "natural language processing is a fascinating field of study"
ngram_result = advanced_features.n_gram_analysis(ngram_text, n=2)
print(f"文本: {ngram_text}")
print(f"2-grams: {ngram_result['n_grams'][:5]}") # 显示前5个
print()
# 词汇变形
words_to_analyze = ['run', 'mouse', 'child', 'go']
print("词汇变形:")
for word in words_to_analyze:
inflection = advanced_features.word_inflection(word)
print(f"{word}: 复数={inflection['plural']}, "
f"词根={inflection['lemma']}, 词干={inflection['stem']}")
print()
# 简单文本摘要
long_text = """
Artificial intelligence is transforming the way we live and work.
Machine learning algorithms can analyze vast amounts of data quickly.
Deep learning networks are inspired by the human brain structure.
Natural language processing helps computers understand human language.
Computer vision enables machines to interpret visual information.
Robotics combines AI with physical systems to create autonomous machines.
The future of AI holds great promise for solving complex problems.
"""
summary_result = advanced_features.text_summarization_simple(
long_text.strip(), sentence_count=3
)
print("文本摘要:")
print(f"原文({summary_result['original_sentence_count']}句) -> "
f"摘要({summary_result['summary_sentence_count']}句)")
print(f"摘要: {summary_result['summary']}")
if __name__ == "__main__":
demo_textblob_applications()实现难易度: ⭐ (非常简单)
- API极其简洁
- 几乎无学习曲线
- 适合快速原型开发
框架对比总结
按使用场景分类
1. 生产环境推荐
- Transformers: 现代NLP任务的首选
- spaCy: 工业级性能和稳定性
- Stanford CoreNLP: 学术级准确度
2. 研究开发推荐
- AllenNLP: 深度学习NLP研究
- Flair: 先进的嵌入技术
- Transformers: 最新模型支持
3. 学习教育推荐
- **TextBl "