一、说明
关于文本分类,文章已经很多,本文这里有实操代码,明确而清晰地表述这种过程,是实战工程师所可以参照和依赖的案例版本。 本文是 2023 年 1 月的 WomenWhoCode 数据科学跟踪活动提供的会议系列文章中的一篇。
之前的文章在这里:第 2 部分(涵盖 NLP 简介)、第 3 部分(涵盖 NLTK 和 SpaCy 库)、第 4 部分(涵盖文本预处理技术)、第 <> 部分(涵盖文本表示技术)。
二、什么是文本分类?
- 文本分类是指将一段文本(例如,客户评论、电子邮件、网页、新闻文章)分类为一些预定义的类别或类。正面、负面或中立的评论评论、垃圾邮件或非垃圾邮件、作为个人或商业页面的网页、有关政治、体育或金融的新闻文章)
- 即,文本分类是为给定文本分配标签或类的任务。例如,将电子邮件归类为垃圾邮件。出于分类的目的,通常从输入文本中识别出一些信息量很大的特征。
- 监督机器学习和无监督学习可用于 NLP 中的文本分类。监督学习涉及在标记的数据集上训练分类器。无监督学习不需要标记的数据集,它使用数据的固有属性(相似性)将数据聚类到组中。高质量的标记数据,通常来自人类注释者,对于监督机器学习非常重要。标记的数据通常分为 3 个部分,训练集、验证集和测试集。分类器的性能使用准确率、精确度、召回率和 F1 分数等指标进行评估。
三、文本分类的重要用例:
- 情绪分析
- POS 标签
- 自然语言推理 --- 推断两段文本之间的关系------前提和假设。这种关系的类型------蕴涵性、矛盾性和中性性。
- 蕴涵:假设由前提支持
- 矛盾:假设被前提否定
- 中性:假设和前提之间没有关系 例如,
- 前提:我现在正在看电影。
- 假设:我现在正在打板球。关系标签:矛盾
- 检查语法正确性:可接受/不可接受。
四、使用 NLTK 进行文本分类
- 对于文本分类的第一个示例,我们将使用 nltk 库中内置的movie_reviews语料库。
- 您可以使用 nltk.download 函数下载 movie_reviews 包:
import nltk nltk.download("movie_reviews")
fileids() 方法允许我们访问 nltk.corpus 中数据集中所有文件的列表。movie_reviews数据集有 2000 个文本文件,每个文件都有一个 fileid。这些文件中的每一个都包含对电影的评论。其中 1000 个文件包含负面评论,1000 个包含正面评论。负面文件位于名为"neg"的文件夹中,所有包含正面评论的文件都位于名为"pos"的文件夹中。
bi
#Required imports
import nltk
import random
from nltk.corpus import movie_reviews
#Total no. of review files in corpus
##There are 1000 negative reviews, and 1000 positive reviews (one review per file)
len(movie_reviews.fileids())
#separating filenames in two lists one for positive reviews, one for negative reviews(based on which folder they exists in corpus)
negative_fileids = movie_reviews.fileids('neg')
positive_fileids = movie_reviews.fileids('pos')
# Now we will load all reviews and their labels (i.e., folder name pos or neg in which the review file is present)
reviewswithcategory = [(list(movie_reviews.words(fileid)), category)
for category in movie_reviews.categories()
for fileid in movie_reviews.fileids(category)]
random.shuffle(reviewswithcategory)
接下来,让我们做一些文本预处理:
bi
# Text pre-processing - lower casing, removing stop words and punctuation marks
import string
from nltk.corpus import stopwords
stop_words = stopwords.words('english')
def text_preprocessing(review):
review = [w.lower() for w in review]
review = [w.translate(str.maketrans('', '', string.punctuation)) for w in review]
review = [w for w in review if w not in stop_words]
review = list(filter(None, review)) #Remove empty strings
return review
cleaned_reviewswithcategory = []
for review, cat in reviewswithcategory:
cleanreview = text_preprocessing(review)
cleaned_reviewswithcategory.append((cleanreview, cat))
# Our variable cleaned_reviewswithcategory is a list of tuples,
# each tuple in it has a list of words and category label
# here we all getting all words from all tuples into a single iterable
allcleanwords = list(itertools.chain(*cleaned_reviewswithcategory))
allcleanwordslist = []
for m in range(len(allcleanwords)):
# traversing the inner lists
for n in range (len(allcleanwords[m])):
# Add each element to the result list
allcleanwordslist.append(allcleanwords[m][n])
接下来,我们从电影评论数据中清理过的单词列表中确定 5000 个最常见的单词
bi
# Using NLTK FreqDist for computing word frequencies
freqd = nltk.FreqDist(allcleanwordslist)
# Identifying 5000 most frequent words from Frequency Distribution
frequent_words = list(freqd.keys())[:5000]
现在,我们将只使用这 5000 个单词。对于正面和负面类别中的每条评论,特征向量将包含这些常用词和一个布尔值 True(如果该词存在于该评论中),否则为 False。
bi
# Identify the presence of these most frequent words in out positive and negative reviews.
# This function returns word and True if word is present, else it returns word and False.
def extract_frequentwordfeatures(text):
words = set(text) # computing all unique words (vocabulary) in input text
features = {}
for w in frequent_words:
features[w] = (w in words)
return features
review_features = [(extract_frequentwordfeatures(review), category) for (review, category) in cleaned_reviewswithcategory]
现在,每个评论都由其特征表示,该特征是一个单词列表和一个布尔值 True 或 False,指示该单词是否存在于评论中。列表中共有 2000 条评论。接下来,我们将评审功能拆分为训练和测试部分。在 2000 条评论中,我们将使用 1800 条来训练来自 NLTK 的朴素贝叶斯分类器,并使用 200 条来测试其性能。
bi
# Splitting the documents into training and test portions
train_data = review_features[:1800]
# set that we'll test against.
test_data = review_features[1800:]
# use Naive Bayes classifier from NLTK to train
clf_nb = nltk.NaiveBayesClassifier.train(train_data)
# After training, let us see the accuracy
print(" Accuracy:",(nltk.classify.accuracy(clf_nb, test_data)))
五、使用 sklearn 分类器对上述评论数据进行分类
bi
from nltk.classify.scikitlearn import SklearnClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.naive_bayes import MultinomialNB
from sklearn.svm import SVC
from sklearn.metrics import classification_report, accuracy_score, confusion_matrix
names = ['Logistic Regression','Multinomial Naive Bayes', 'Support Vector Machine']
classifiers = [LogisticRegression(),MultinomialNB(),SVC(kernel='linear')]
models = zip(names, classifiers)
text_features, labels = zip(*test_data)
for name, model in models:
nltk_clf = SklearnClassifier(model)
nltk_clf.train(train_data)
accuracy = nltk.classify.accuracy(nltk_clf, test_data)
print("\n{} Classifier Accuracy: {}".format(name, accuracy))
六、使用 Keras 进行文本分类
在这部分,我们将使用来自 UCI 存储库的 Sentiment Labelled Sentences 数据集。此数据集包含标有正面或负面情绪的句子。情绪是分数的形式,分数是 1 表示积极情绪,0 表示消极情绪。这些句子来自三个不同的网站:imdb.com、amazon.com yelp.com 对于每个网站,存在 500 个正面句子和 500 个负面句子。在这里的示例中,我们将仅使用文件amazon_cells_labelled.txt中的亚马逊评论。
首先,我们将使用 pandas 将数据读入 pandas 数据帧。此数据有两列 --- 句子和标签。句子是产品评论,标签是 0 或 1。标签 1 表示积极情绪,0 表示消极情绪。
bi
import pandas as pd
df = pd.read_csv('amazon_cells_labelled.txt', names=['sentence', 'label'], sep='\t')
df.head()
接下来,我们对句子列中的文本进行预处理
bi
# This process_text() function returns list of cleaned tokens of the text
import numpy
import re
import string
import unicodedata
from nltk.corpus import stopwords
from nltk.stem import WordNetLemmatizer
stop_words = stopwords.words('english')
lemmatizer = WordNetLemmatizer()
def process_text(text):
text = unicodedata.normalize('NFKD', text).encode('ascii', 'ignore').decode('utf-8', 'ignore')
text = re.sub(r'[^a-zA-Z\s]', '', text)
text = text.translate(str.maketrans('', '', string.punctuation))
text = text.lower()
text = " ".join([word for word in str(text).split() if word not in stop_words])
text = " ".join([lemmatizer.lemmatize(word) for word in text.split()])
return text
df['sentence'] = df['sentence'].apply(process_text)
df['sentence']
现在我们将数据分为训练和测试部分,在此之前,让我们将"句子"列中的文本和"标签"列中的标签分为两个pandas系列------"句子"和"标签"。
bi
# Taking cleaned text and sentiment labels in two separate variables
sentences = df['sentence']
labels = df['label']
# Splitting into train-test portions
from sklearn.model_selection import train_test_split
sentences_train, sentences_test, labels_train, labels_test = train_test_split(sentences, labels, test_size=0.25, random_state=1000)
接下来,我们使用 sklearn 中带有 CountVectorizer 的词袋模型以向量形式表示句子中的文本
bi
# Converting sentences to vectors using Bag of words model with CountVectorizer
from sklearn.feature_extraction.text import CountVectorizer
cv = CountVectorizer()
cv.fit(sentences_train)
vc_traindata = cv.transform(sentences_train)
vc_testdata = cv.transform(sentences_test)
vc_traindata
现在我们将使用 Keras 进行分类,因此应该在我们的计算机上安装 TensorFlow 和 Keras 库。可以使用 pip 命令从 Jupyter Notebook 中安装它们
bi
!pip install tensorflow
!pip install keras
首先,我们将使用 Keras Tokenizer 来计算文本的单词嵌入
bi
# Using Keras Tokenizer to compute embeddings for text
from keras.preprocessing.text import Tokenizer
tokenizer = Tokenizer(num_words=5000)
tokenizer.fit_on_texts(df['sentence'])
# Take the sentence text in a variable X and labels in y.
X = df['sentence']
y = df['label']
#Splitting X and y into train and test portions
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=1000)
#converting sentences into vector sequences
X_train = tokenizer.texts_to_sequences(X_train)
X_test = tokenizer.texts_to_sequences(X_test)
# Total number of unique words in our sentences
vocab_size = len(tokenizer.word_index) + 1 # Adding 1 because of reserved 0 index
print('Vocabulary size:' , vocab_size)
然后,我们将填充所有向量(代表产品评论的文本)的长度相同 --- 50
bi
# padding vector sequences to make them all of same length
from keras_preprocessing.sequence import pad_sequences
maxlen = 50
X_train = pad_sequences(X_train, padding='post', maxlen=maxlen)
X_test = pad_sequences(X_test, padding='post', maxlen=maxlen)
print(X_train[4, :])
七、使用 Keras 的嵌入层
我们现在将使用 Keras 的嵌入层,它采用先前计算的整数并将它们映射到嵌入的密集向量。它需要以下参数:
- input_dim:词汇量的大小
- output_dim: the size of the dense vector
- input_length: the length of the sequence
我们可以获取嵌入层的输出并将其插入密集层。为此,我们需要在中间添加一个 Flatten 层,为 Dense 层准备顺序输入。
在训练期间,要训练的参数数vacab_size乘以embedding_dim。嵌入层的权重是随机初始化的,然后在训练期间使用反向传播进行微调。该模型将按句子顺序出现的单词作为输入向量
在下面的代码中,我们使用嵌入层 GlobalMaxPool1D 进行扁平化,以及由 15 个神经元和一个输出层组成的密集层。我们编译了 keras 模型。
bi
from keras.models import Sequential
from keras import layers
embedding_dim = 100
maxlen = 50
model = Sequential()
model.add(layers.Embedding(input_dim=vocab_size, output_dim=embedding_dim, input_length=maxlen))
model.add(layers.GlobalMaxPool1D())
model.add(layers.Dense(15, activation='relu'))
model.add(layers.Dense(1, activation='sigmoid'))
model.compile(optimizer='adam',loss='binary_crossentropy',metrics=['accuracy'])
model.summary()
接下来,我们将模型拟合在 50 个 epoch 的训练数据上,并评估其性能。使用 matplotlib 绘制模型的准确性
bi
import matplotlib.pyplot as plt
plt.figure(figsize=(9, 5))
history = model.fit(X_train, y_train,epochs=50, verbose=False,validation_data=(X_test, y_test), batch_size=10)
loss, accuracy = model.evaluate(X_train, y_train, verbose=False)
print("Training Accuracy: {:.4f}".format(accuracy))
loss, accuracy = model.evaluate(X_test, y_test, verbose=False)
print("Testing Accuracy: {:.4f}".format(accuracy))
acc = history.history['accuracy']
val_acc = history.history['val_accuracy']
loss = history.history['loss']
val_loss = history.history['val_loss']
x = range(1, len(acc) + 1)
plt.plot(x, acc, 'b', label='Training acc')
plt.plot(x, val_acc, 'r', label='Validation acc')
plt.legend()
plt.title('Training and validation accuracy')
plt.show()
在下一篇文章中,我们将看看变形金刚。代码出处尼姆里塔·库尔
参考和引用:
- https://developers.google.com/machine-learning/guides/text-classification
- Text Classification with Python and Scikit-Learn
- https://towardsdatascience.com/a-practitioners-guide-to-natural-language-processing-part-i-processing-understanding-text-9f4abfd13e72
- Text Classification using NLTK | Foundations of AI & ML
- Practical Text Classification With Python and Keras -- Real Python
- Text Classification with NLTK | Chan`s Jupyter
- https://www.analyticsvidhya.com/blog/2018/04/a-comprehensive-guide-to-understand-and-implement-text-classification-in-python/
- https://medium.com/analytics-vidhya/nlp-tutorial-for-text-classification-in-python-8f19cd17b49e
- Practical Text Classification With Python and Keras -- Real Python