NLP---文本前期预处理的几个步骤

1、读取文本

python 复制代码
text1 ="""
Football is a family of team sports that involve, to varying degrees, kicking a ball to score a goal. 
Unqualified, the word football is understood to refer to whichever form of football is the most popular 
in the regional context in which the word appears. Sports commonly called football in certain places 
include association football (known as soccer in some countries); gridiron football (specifically American 
football or Canadian football); Australian rules football; rugby football (either rugby league or rugby union); 
and Gaelic football. These different variations of football are known as football codes.
"""
print("原文:\n", text1)

2、去除换行符

python 复制代码
text = text1.replace("\n", "")
print("去除原文中的换行符:\n", text)

3、分句

python 复制代码
import nltk
sents = nltk.sent_tokenize(text)
print("将文本进行分句:\n", sents)

4、分词

python 复制代码
import string
punctuation_tokens = []
for sent in sents:
    for word in nltk.word_tokenize(sent):
        punctuation_tokens.append(word)
print("将每个句子进行分词:\n", punctuation_tokens)

5、过滤标点符号

python 复制代码
tokens = []
for word in punctuation_tokens:
    if word not in string.punctuation:
        tokens.append(word)
print("将分词结果去除标点符号:\n", tokens)

6、过滤停用词

python 复制代码
from nltk.corpus import stopwords
fltered = [w for w in tokens if w not in stopwords.words("english")]
print("过滤完停用词之后:\n", fltered)

7、剩下有用的单词进行计数

python 复制代码
from collections import Counter
count = Counter(fltered)
print("对最终清洗好的单词进行计数:\n", count)
相关推荐
第七序章3 天前
【C++STL】list的详细用法和底层实现
c语言·c++·自然语言处理·list
大千AI助手3 天前
TruthfulQA:衡量语言模型真实性的基准
人工智能·语言模型·自然语言处理·llm·模型评估·truthfulqa·事实性基准
什么都想学的阿超3 天前
【大语言模型 58】分布式文件系统:训练数据高效存储
人工智能·语言模型·自然语言处理
金井PRATHAMA3 天前
认知语义学隐喻理论对人工智能自然语言处理中深层语义分析的赋能与挑战
人工智能·自然语言处理·知识图谱
J_Xiong01173 天前
【VLMs篇】07:Open-Qwen2VL:在学术资源上对完全开放的多模态大语言模型进行计算高效的预训练
人工智能·语言模型·自然语言处理
苏苏susuus3 天前
NLP:Transformer之self-attention(特别分享3)
人工智能·自然语言处理·transformer
这张生成的图像能检测吗4 天前
(综述)视觉任务的视觉语言模型
人工智能·计算机视觉·语言模型·自然语言处理·视觉语言模型
GRITJW4 天前
注意力机制:从核心原理到前沿应用
自然语言处理
fanstuck4 天前
Prompt提示工程上手指南(六):AI避免“幻觉”(Hallucination)策略下的Prompt
人工智能·语言模型·自然语言处理·nlp·prompt
亚里随笔4 天前
小型语言模型:智能体AI的未来?
人工智能·语言模型·自然语言处理·llm·rlhf·agentic