使用ChatGPT自动构建知识图谱

1.概述

本文将探讨利用OpenAI的gpt-3.5-turbo从原始文本构建知识图谱，通过LLM和RAG技术实现文本生成、问答和特定领域知识的高效提取，以获得有价值的洞察。在开始前，我们需要明确一些关键概念。

2.内容

2.1 什么是知识图谱？

知识图谱是一种语义网络，它表示和连接现实世界中的实体，如人物、组织、物体、事件和概念。知识图谱由具有以下结构的三元组组成：知识图谱由"头实体 → 关系 → 尾实体"或语义网术语"主语 → 谓语 → 宾语"的三元组构成，用于提取和分析实体间的复杂关系。它通常包含一个定义概念、关系及其属性的本体，作为目标领域中概念和关系的正式规范，为网络提供语义。搜索引擎等自动化代理使用本体来理解网页内容，以正确索引和显示。

2.2 案例

2.2.1 准备依赖

使用 OpenAI 的 gpt-3.5-turbo 根据产品数据集中的产品描述创建知识图。Python依赖如下：

pip install pandas openai sentence-transformers networkx

2.2.2 读取数据

读取数据集，代码如下所示：

import json
import logging
import matplotlib.pyplot as plt
import networkx as nx
from networkx import connected_components
from openai import OpenAI
import pandas as pd
from sentence_transformers import SentenceTransformer, util
data = pd.read_csv("products.csv")

数据集包含"PRODUCT_ID"、"TITLE"、"BULLET_POINTS"、"DESCRIPTION"、"PRODUCT_TYPE_ID"和"PRODUCT_LENGTH"列。我们将合并"TITLE"、"BULLET_POINTS"和"DESCRIPTION"列成"text"列，用于提示ChatGPT从中提取实体和关系的商品规格。

实现代码如下：

data['text'] = data['TITLE'] + data['BULLET_POINTS'] + data['DESCRIPTION']

2.2.3 特征提取

我们将指导ChatGPT从提供的商品规格中提取实体和关系，并以JSON对象数组的形式返回结果。JSON对象必须包含以下键：'head'、'head_type'、'relation'、'tail'和'tail_type'。

'head'键必须包含从用户提示提供的列表中提取的实体文本。'head_type'键必须包含从用户提供的列表中提取的头实体类型。'relation'键必须包含'head'和'tail'之间的关系类型，'tail'键必须表示提取的实体文本，该实体是三元组中的对象，而'tail_type'键必须包含尾实体的类型。

我们将使用下面列出的实体类型和关系类型来提示ChatGPT进行实体关系提取。我们将把这些实体和关系映射到Schema.org本体中对应的实体和关系。映射中的键表示提供给ChatGPT的实体和关系类型，值表示Schema.org中的对象和属性的URL。

# ENTITY TYPES:
entity_types = {
  "product": "https://schema.org/Product", 
  "rating": "https://schema.org/AggregateRating",
  "price": "https://schema.org/Offer", 
  "characteristic": "https://schema.org/PropertyValue", 
  "material": "https://schema.org/Text",
  "manufacturer": "https://schema.org/Organization", 
  "brand": "https://schema.org/Brand", 
  "measurement": "https://schema.org/QuantitativeValue", 
  "organization": "https://schema.org/Organization",  
  "color": "https://schema.org/Text",
}

# RELATION TYPES:
relation_types = {
  "hasCharacteristic": "https://schema.org/additionalProperty",
  "hasColor": "https://schema.org/color", 
  "hasBrand": "https://schema.org/brand", 
  "isProducedBy": "https://schema.org/manufacturer", 
  "hasColor": "https://schema.org/color",
  "hasMeasurement": "https://schema.org/hasMeasurement", 
  "isSimilarTo": "https://schema.org/isSimilarTo", 
  "madeOfMaterial": "https://schema.org/material", 
  "hasPrice": "https://schema.org/offers", 
  "hasRating": "https://schema.org/aggregateRating", 
  "relatedTo": "https://schema.org/isRelatedTo"
 }

为使用ChatGPT进行信息提取，我们创建了OpenAI客户端，利用聊天完成API，为每个识别到的关系生成JSON对象输出数组。选择gpt-3.5-turbo作为默认模型，因其性能已足够满足此简单演示需求。

client = OpenAI(api_key="<YOUR_API_KEY>")

定义提取函数：

def extract_information(text, model="gpt-3.5-turbo"):
   completion = client.chat.completions.create(
        model=model,
        temperature=0,
        messages=[
        {
            "role": "system",
            "content": system_prompt
        },
        {
            "role": "user",
            "content": user_prompt.format(
              entity_types=entity_types,
              relation_types=relation_types,
              specification=text
            )
        }
        ]
    )

   return completion.choices[0].message.content

2.2.4 编写Prompt

system_prompt变量包含了指导ChatGPT从原始文本中提取实体和关系，并将结果以JSON对象数组形式返回的指令，每个JSON对象包含以下键：'head'、'head_type'、'relation'、'tail'和'tail_type'。

system_prompt = """You are an expert agent specialized in analyzing product specifications in an online retail store.
Your task is to identify the entities and relations requested with the user prompt, from a given product specification.
You must generate the output in a JSON containing a list with JOSN objects having the following keys: "head", "head_type", "relation", "tail", and "tail_type".
The "head" key must contain the text of the extracted entity with one of the types from the provided list in the user prompt, the "head_type"
key must contain the type of the extracted head entity which must be one of the types from the provided user list,
the "relation" key must contain the type of relation between the "head" and the "tail", the "tail" key must represent the text of an
extracted entity which is the tail of the relation, and the "tail_type" key must contain the type of the tail entity. Attempt to extract as
many entities and relations as you can.
"""

user_prompt变量包含来自数据集单个规范所需的输出示例，并提示ChatGPT以相同的方式从提供的规范中提取实体和关系。这是ChatGPT单次学习的一个示例。

user_prompt = """Based on the following example, extract entities and relations from the provided text.
Use the following entity types:

# ENTITY TYPES:
{entity_types}

Use the following relation types:
{relation_types}

--> Beginning of example

# Specification
"YUVORA 3D Brick Wall Stickers | PE Foam Fancy Wallpaper for Walls,
 Waterproof & Self Adhesive, White Color 3D Latest Unique Design Wallpaper for Home (70*70 CMT) -40 Tiles
 [Made of soft PE foam,Anti Children's Collision,take care of your family.Waterproof, moist-proof and sound insulated. Easy clean and maintenance with wet cloth,economic wall covering material.,Self adhesive peel and stick wallpaper,Easy paste And removement .Easy To cut DIY the shape according to your room area,The embossed 3d wall sticker offers stunning visual impact. the tiles are light, water proof, anti-collision, they can be installed in minutes over a clean and sleek surface without any mess or specialized tools, and never crack with time.,Peel and stick 3d wallpaper is also an economic wall covering material, they will remain on your walls for as long as you wish them to be. The tiles can also be easily installed directly over existing panels or smooth surface.,Usage range: Featured walls,Kitchen,bedroom,living room, dinning room,TV walls,sofa background,office wall decoration,etc. Don't use in shower and rugged wall surface]
Provide high quality foam 3D wall panels self adhesive peel and stick wallpaper, made of soft PE foam,children's collision, waterproof, moist-proof and sound insulated,easy cleaning and maintenance with wet cloth,economic wall covering material, the material of 3D foam wallpaper is SAFE, easy to paste and remove . Easy to cut DIY the shape according to your decor area. Offers best quality products. This wallpaper we are is a real wallpaper with factory done self adhesive backing. You would be glad that you it. Product features High-density foaming technology Total Three production processes Can be use of up to 10 years Surface Treatment: 3D Deep Embossing Damask Pattern."

################

# Output
[
  {{
    "head": "YUVORA 3D Brick Wall Stickers",
    "head_type": "product",
    "relation": "isProducedBy",
    "tail": "YUVORA",
    "tail_type": "manufacturer"
  }},
  {{
    "head": "YUVORA 3D Brick Wall Stickers",
    "head_type": "product",
    "relation": "hasCharacteristic",
    "tail": "Waterproof",
    "tail_type": "characteristic"
  }},
  {{
    "head": "YUVORA 3D Brick Wall Stickers",
    "head_type": "product",
    "relation": "hasCharacteristic",
    "tail": "Self Adhesive",
    "tail_type": "characteristic"
  }},
  {{
    "head": "YUVORA 3D Brick Wall Stickers",
    "head_type": "product",
    "relation": "hasColor",
    "tail": "White",
    "tail_type": "color"
  }},
  {{
    "head": "YUVORA 3D Brick Wall Stickers",
    "head_type": "product",
    "relation": "hasMeasurement",
    "tail": "70*70 CMT",
    "tail_type": "measurement"
  }},
  {{
    "head": "YUVORA 3D Brick Wall Stickers",
    "head_type": "product",
    "relation": "hasMeasurement",
    "tail": "40 tiles",
    "tail_type": "measurement"
  }},
  {{
    "head": "YUVORA 3D Brick Wall Stickers",
    "head_type": "product",
    "relation": "hasMeasurement",
    "tail": "40 tiles",
    "tail_type": "measurement"
  }}
]

--> End of example

For the following specification, generate extract entitites and relations as in the provided example.

# Specification
{specification}
################

# Output

"""

View Code

现在，我们对数据集中的每个规范调用extract_information函数，并创建一个包含所有提取的三元组的列表，这将代表我们的知识图谱。为了演示，我们将使用仅包含100个产品规范的子集来生成知识图谱。

kg = []
for content in data['text'].values[:100]:
  try:
    extracted_relations = extract_information(content)
    extracted_relations = json.loads(extracted_relations)
    kg.extend(extracted_relations)
  except Exception as e:
    logging.error(e)

kg_relations = pd.DataFrame(kg)

信息提取的结果显示在下面的图中。

2.2.5 实体关系

实体解析（ER）是消除与现实世界概念对应的实体歧义的过程。在这种情况下，我们将尝试对数据集中的头实体和尾实体进行基本的实体解析。这样做的原因是使文本中存在的实体具有更简洁的表示。

我们将使用NLP技术进行实体解析，更具体地说，我们将使用sentence-transformers库为每个头实体创建嵌入，并计算头实体之间的余弦相似性。

我们将使用'all-MiniLM-L6-v2'句子转换器来创建嵌入，因为它是一个快速且相对准确的模型，适用于这种情况。对于每对头实体，我们将检查相似性是否大于0.95，如果是，我们将认为这些实体是相同的实体，并将它们的文本值标准化为相等。对于尾实体也是同样的道理。

这个过程将帮助我们实现以下结果。如果我们有两个实体，一个的值为'Microsoft'，另一个为'Microsoft Inc.'，那么这两个实体将被合并为一个。

我们以以下方式加载和使用嵌入模型来计算第一个和第二个头实体之间的相似性。

heads = kg_relations['head'].values
embedding_model = SentenceTransformer('all-MiniLM-L6-v2')
embeddings = embedding_model.encode(heads)
similarity = util.cos_sim(embeddings[0], embeddings[1])

为了可视化实体解析后提取的知识图谱，我们使用Python的networkx库。首先，我们创建一个空图，然后将每个提取的关系添加到图中。

G = nx.Graph()
for _, row in kg_relations.iterrows():
  G.add_edge(row['head'], row['tail'], label=row['relation'])

要绘制图表，我们可以使用以下代码：

pos = nx.spring_layout(G, seed=47, k=0.9)
labels = nx.get_edge_attributes(G, 'label')
plt.figure(figsize=(15, 15))
nx.draw(G, pos, with_labels=True, font_size=10, node_size=700, node_color='lightblue', edge_color='gray', alpha=0.6)
nx.draw_networkx_edge_labels(G, pos, edge_labels=labels, font_size=8, label_pos=0.3, verticalalignment='baseline')
plt.title('Product Knowledge Graph')
plt.show()

下面的图中显示了生成的知识图谱的一个子图：

我们可以看到，通过这种方式，我们可以基于共享的特征将多个不同的产品连接起来。这对于学习产品之间的共同属性、标准化产品规格、使用通用模式（如Schema.org）描述网络资源，甚至基于产品规格进行产品推荐都是有用的。

3.总结

大多数公司有大量未被利用的非结构化数据存储在数据湖中。创建知识图谱以从这些未使用的数据中提取洞察的方法将有助于从未经处理和非结构化的文本语料库中获取信息，并利用这些信息做出更明智的决策。