Data Mining数据挖掘—4. Text Mining文本挖掘

5. Text Mining

The extraction of implicit, previously unknown and potentially useful information from a large amount of textual resources.

Applications: Email and news filtering, spam detection, fake review classification, sentiment analysis, query suggestion, auto-complete, search result organization, information extraction(named entity recognition and disambiguation, relation extraction, fact extraction) ...
Text Mining Process

5.1 Text Similarity and Feature Creation

1. Levenshtein / edit distance

Compute total number of character insertions/deletions/changes, then divide by longer text length. Levenshtein距离(也称为编辑距离)定义为通过插入、删除或替换一个字符,将一个字符串转换为另一个字符串所需的最小操作次数。

Example: kitten & sitting -> Levenshtein distance = 3
2. Jaro-Winkler

Based on the number of common characters within a specific distance (Optimized for comparing person names)
3. n-gram

split string into set of trigrams, then measure overlap of trigrams. e.g. Jaccard: |common trigrams| / |all trigrams|

Example: "iPhone5 Apple" vs. "Apple iPhone 5"

common trigrams: "iPh", "Pho", "hon", "one", "App", "ppl", "ple"

other trigrams: "ne5", "e5 ", "5 A", " Ap" (1), "le ", "e i", " iP", "e 5" (2)

Jaccard: 7/15 = 0.47

4. word2vec Distance

represent a word by a numerical vector (sematic similarity)

5. Term-Document Matrix

compare long texts

Document is treated as a bag of words (each word or term becomes a feature; order of words/terms is ignored)

Each document is represented by a vector
Different techniques for vector creation:
(1)Binary Term Occurrence : Boolean attributes describe whether or not a term appears in the document.
Asymmetric binary attributes : If one of the states is more important or more valuable than the other. By convention, state 1 represents the more important state, rare or infrequent state.
Jaccard Coefficient for Text Similarity

Interpretation of Jaccard: Ratio of common words to all words in both texts

Drawback: All words are considered equal
(2)Term Occurrence : Number of occurences of a term in the document (problematic if documents have different length).
(3) Terms frequency : Attributes represent the frequency in which a term appears in the document (Number of occurrences / Number of words in document)
(4) TF-IDF

TF-IDF weight (term frequency--inverse document frequency) is used to evaluate how important a word is to a corpus of documents
Similarity of documents -- Cosine Similarity


Words are now weighted

-- Words that are important for a text: high TF

-- Words that are too unspecific: low IDF

5.2 Text Preprocessing

5.2.1 Syntactic / Linguistic Text Analysis

Simple Syntactic Analysis: Text Cleanup (remove punctuation, HTML tags), Normalize case, Tokenization (break text into single words or N-grams)

Advanced Linguistic Analysis: Word Sense Disambiguation, Part Of Speech (POS) tagging

Synonym Normalization & Spelling Correction: usually using catalogs, such as WordNet. Or anchor texts of links to that page

POS Tagging: determining word classes and syntactic functions, finding the structure of a sentence. Supervised approach (Use an annotated corpus of text). Simple algorithm for key phrase extraction-- e.g., annotation of text corpora.

Stop Words Removal: Many of the most frequent words (stop words) are likely to be useless -> remove -> Reduce data set size + Improve efficiency and effectiveness

Stemming: Techniques to find out the root/stem of a word -> improve effectiveness, reduce term vector size

Lookup-based Stemming Rule-based Stemming
accurate, exceptions are handled easily, consume much space low space consumption, works for emerging words without update

5.3 Feature Selection

Not all features are helpful. Transformation approaches tend to create lots of features (dimensionality problem).
Pruning methods

Specify if and how too frequent or too infrequent words should be ignored [percentual, absolute, by rank]
POS Tagging

POS tags may help with feature selection
Named Entity Recognition and Linking

The classes of Named Entity Recognition (NER) may be useful features

Named Entity Linking (Identify named entities in a knowledge base) May be used to create additional features

5.4 Pattern Discovery

Classification Definition

Given: A collection of labeled documents (training set)

Find: A model for the class as a function of the values of the features.

Goal: Previously unseen documents should be assigned a class as accurately as possible.

Classification methods commonly used for text: Naive Bayes, SVMs, Neural Networks, Random Forests...
Sentiment Analysis

A specific classification task.

Given: a text

Target: a class of sentiments( e.g., positive, neutral, negative; sad, happy, angry, surprised)

Alternative: numerical score (e.g., -5...+5)

Can be implemented as supervised classification/regression task

Reasonable features for sentiment analysis might include punctuation: use of "!", "?", "?!"; smileys (usually encoded using punctuation: 😉); use of visual markup, where available (red color, bold face, ...); amount of capitalization ("screaming")
Identifying Fake Reviews
Alternative Document Representations
[a] Topic Modeling

[b] doc2vec

an extension of word2vec, an extension of word2vec

[c] BERT

5.5 Processing Text from Social Media

Respect special characters

Normalizing

相关推荐
CappuccinoRose20 小时前
MATLAB学习文档(二十三)
matlab·信息可视化·数据挖掘·数据分析
MoRanzhi12031 天前
11. Pandas 数据分类与区间分组(cut 与 qcut)
人工智能·python·机器学习·数学建模·分类·数据挖掘·pandas
你好~每一天1 天前
2025 中小企业 AI 转型:核心岗技能 “怎么证、怎么用”?
人工智能·百度·数据挖掘·数据分析·职业·转行
唐兴通个人1 天前
清华大学AI领导力AI时代领导力AI变革领导力培训师培训讲师专家唐兴通讲授数字化转型人工智能组织创新实践领导力国央企国有企业金融运营商制造业
人工智能·数据挖掘
码上地球1 天前
大数据成矿预测系列(三) | 从统计模型到机器学习:为何机器学习是成矿预测的新前沿?
大数据·机器学习·数据挖掘
F_D_Z2 天前
【Python】家庭用电数据分析Prophet预测
python·数据挖掘·数据分析·prophet
好开心啊没烦恼2 天前
Python数据分析:使用爬虫从网页、社交媒体平台、论坛等公开资源提取中文和英文人名。
开发语言·爬虫·python·数据挖掘·数据分析
计算机源码社3 天前
基于Hadoop的车辆二氧化碳排放量分析与可视化系统|基于Spark的车辆排放量实时监控与预测系统|基于数据挖掘的汽车排放源识别与减排策略系统
大数据·hadoop·机器学习·数据挖掘·spark·毕业设计·课程设计
罗小罗同学3 天前
覆盖9个癌种,基于11671张病理切片训练的模型登上Nature子刊,可精准“读出”分子标志物,突破传统分类局限
人工智能·深度学习·分类·数据挖掘·病理组学·医学人工智能·医工交叉
nju_spy4 天前
2023 美赛C Predicting Wordle Results(上)
人工智能·机器学习·数学建模·数据挖掘·arima·时间序列预测·相关性分析