NLP---文本前期预处理的几个步骤

1、读取文本

python 复制代码
text1 ="""
Football is a family of team sports that involve, to varying degrees, kicking a ball to score a goal. 
Unqualified, the word football is understood to refer to whichever form of football is the most popular 
in the regional context in which the word appears. Sports commonly called football in certain places 
include association football (known as soccer in some countries); gridiron football (specifically American 
football or Canadian football); Australian rules football; rugby football (either rugby league or rugby union); 
and Gaelic football. These different variations of football are known as football codes.
"""
print("原文:\n", text1)

2、去除换行符

python 复制代码
text = text1.replace("\n", "")
print("去除原文中的换行符:\n", text)

3、分句

python 复制代码
import nltk
sents = nltk.sent_tokenize(text)
print("将文本进行分句:\n", sents)

4、分词

python 复制代码
import string
punctuation_tokens = []
for sent in sents:
    for word in nltk.word_tokenize(sent):
        punctuation_tokens.append(word)
print("将每个句子进行分词:\n", punctuation_tokens)

5、过滤标点符号

python 复制代码
tokens = []
for word in punctuation_tokens:
    if word not in string.punctuation:
        tokens.append(word)
print("将分词结果去除标点符号:\n", tokens)

6、过滤停用词

python 复制代码
from nltk.corpus import stopwords
fltered = [w for w in tokens if w not in stopwords.words("english")]
print("过滤完停用词之后:\n", fltered)

7、剩下有用的单词进行计数

python 复制代码
from collections import Counter
count = Counter(fltered)
print("对最终清洗好的单词进行计数:\n", count)
相关推荐
强哥之神2 小时前
Meta AI 推出 Multi - SpatialMLLM:借助多模态大语言模型实现多帧空间理解
人工智能·深度学习·计算机视觉·语言模型·自然语言处理·llama
不会计算机的g_c__b4 小时前
跨越NLP的三重曲线:从词法到叙事的进化之路
人工智能·自然语言处理
陈敬雷-充电了么-CEO兼CTO5 小时前
复杂任务攻坚:多模态大模型推理技术从 CoT 数据到 RL 优化的突破之路
人工智能·python·神经网络·自然语言处理·chatgpt·aigc·智能体
heart000_117 小时前
128K 长文本处理实战:腾讯混元 + 云函数 SCF 构建 PDF 摘要生成器
人工智能·自然语言处理·pdf
kebijuelun20 小时前
百度文心 4.5 大模型详解:ERNIE 4.5 Technical Report
人工智能·深度学习·百度·语言模型·自然语言处理·aigc
MUTA️1 天前
ELMo——Embeddings from Language Models原理速学
人工智能·语言模型·自然语言处理
小叮当爱咖啡1 天前
Seq2seq+Attention 机器翻译
人工智能·自然语言处理·机器翻译
FF-Studio1 天前
大语言模型(LLM)课程学习(Curriculum Learning)、数据课程(data curriculum)指南:从原理到实践
人工智能·python·深度学习·神经网络·机器学习·语言模型·自然语言处理
uncle_ll2 天前
李宏毅NLP-8-语音模型
人工智能·自然语言处理·语音识别·语音模型·lm
Liudef062 天前
FLUX.1-Kontext 高效训练 LoRA:释放大语言模型定制化潜能的完整指南
人工智能·语言模型·自然语言处理·ai作画·aigc