ContextGem：简化文档数据提取的利器

ContextGem 是一个免费、开源的框架，旨在简化从文档中提取结构化数据和洞察力的过程。它通过强大的抽象化，减少了开发时间和复杂性，使得开发者可以轻松地从文档中提取有价值的信息。

ContextGem 的优势

简化开发流程：ContextGem 提供了一个灵活且直观的框架，通过最少的代码即可从文档中提取结构化数据和洞察力。
自动化功能：支持自动动态提示、数据建模和验证、精确的段落和句子映射、理由支持、神经分割（SaT）等功能。
多语言支持：无需提示即可支持多语言输入输出。
统一的提取管道：提供单一、统一的提取管道，支持声明式、可序列化的数据处理。
多LLM支持：支持多个LLM提供商，并提供简单的API接口用于切换。

核心组件

ContextGem 的核心组件包括：

文档模型（Document Model）：包含文本或视觉内容的文档。
方面模型（Aspect Model）：定义文档中的特定区域或主题。
概念模型（Concept Model）：从方面或文档中提取的信息单元或实体。

安装和快速开始

安装

复制代码

bash
pip install -U contextgem

快速开始：方面提取

ini 复制代码

python
import os
from contextgem import Aspect, Document, DocumentLLM

# 示例文档
doc = Document(
    raw_text=(
        "SERVICE AGREEMENT\n"
        "SERVICES. Provider agrees to provide the following services to Client: "
        "Cloud-based data analytics platform access and maintenance...\n"
        "PAYMENT. Client agrees to pay $5,000 per month for the services. "
        "Payment is due on the 1st of each month. Late payments will incur a 2% fee per month...\n"
        "CONFIDENTIALITY. Both parties agree to keep all proprietary information confidential "
        "for a period of 5 years following termination of this Agreement..."
    ),
)

# 定义方面
doc.aspects = [
    Aspect(
        name="Payment Terms",
        description="Payment terms and conditions in the contract",
    ),
]

# 定义LLM
llm = DocumentLLM(
    model="openai/gpt-4o-mini",
    api_key=os.environ.get("CONTEXTGEM_OPENAI_API_KEY"),
)

# 提取信息
doc = llm.extract_all(doc)

# 访问提取的信息
for item in doc.aspects[0].extracted_items:
    print(f"• {item.value}")

快速开始：概念提取

ini 复制代码

python
import os
from contextgem import Document, DocumentLLM, StringConcept

# 示例文档
doc = Document(
    raw_text=(
        "Consultancy Agreement\n"
        "This agreement between Company A (Supplier) and Company B (Customer)...\n"
        "The term of the agreement is 1 year from the Effective Date...\n"
        "The Supplier shall provide consultancy services as described in Annex 2...\n"
        "The Customer shall pay the Supplier within 30 calendar days of receiving an invoice...\n"
        "The purple elephant danced gracefully on the moon while eating ice cream.\n"
        "This agreement is governed by the laws of Norway...\n"
    ),
)

# 定义概念
doc.concepts = [
    StringConcept(
        name="Anomalies",
        description="Anomalies in the document",
        add_references=True,
        reference_depth="sentences",
        add_justifications=True,
        justification_depth="brief",
    )
]

# 定义LLM
llm = DocumentLLM(
    model="openai/gpt-4o-mini",
    api_key=os.environ.get("CONTEXTGEM_OPENAI_API_KEY"),
)

# 提取信息
doc = llm.extract_all(doc)

# 访问提取的信息
print(doc.concepts[0].extracted_items)

优化策略

ContextGem 提供了多种优化策略，以提高提取准确率、速度和降低成本。这些策略包括选择合适的LLM、优化长文档处理等。更多信息请参考 ContextGem 文档。