【Langchain大语言模型开发教程】评估

🔗 LangChain for LLM Application Development - DeepLearning.AI

学习目标

1、Example generation

2、Manual evaluation and debug

3、LLM-assisted evaluation

4、LangChain evaluation platform

1、引包、加载环境变量;

python 复制代码
import os

from dotenv import load_dotenv, find_dotenv
_ = load_dotenv(find_dotenv()) # read local .env file

from langchain.chains import RetrievalQA
from langchain_openai import ChatOpenAI
from langchain.document_loaders import CSVLoader
from langchain.indexes import VectorstoreIndexCreator
from langchain.vectorstores import DocArrayInMemorySearch

2、加载数据;

python 复制代码
file = 'OutdoorClothingCatalog_1000.csv'
loader = CSVLoader(file_path=file, encoding='utf-8')
data = loader.load()

3、创建向量数据库(内存警告⚠);

python 复制代码
model_name = "bge-large-en-v1.5"
embeddings = HuggingFaceEmbeddings(
    model_name=model_name,
)

db = DocArrayInMemorySearch.from_documents(data, embeddings)
retriever = db.as_retriever()

4、初始化一个LLM并创建一个RetrievalQ链;

python 复制代码
llm = ChatOpenAI(api_key=os.environ.get('ZHIPUAI_API_KEY'),
                         base_url=os.environ.get('ZHIPUAI_API_URL'),
                         model="glm-4",
                         temperature=0.98)

qa = RetrievalQA.from_chain_type(
    llm=llm, 
    chain_type="stuff", 
    retriever=retriever,
    verbose=True,
    chain_type_kwargs = {
        "document_separator": "<<<<>>>>>"
    }
)

Example generation

python 复制代码
from langchain.evaluation.qa import QAGenerateChain

example_gen_chain = QAGenerateChain.from_llm(llm)

new_examples = example_gen_chain.apply_and_parse(
    [{"doc": t} for t in data[:5]]
)

这里我们打印一下这个生成的example,发现是一个列表长下面这个样子;

python 复制代码
[{'qa_pairs': {'query': "What is the unique feature of the innersole in the Women's Campside Oxfords?", 'answer': 'The innersole has a vintage hunt, fish, and camping motif.'}}, {'qa_pairs': {'query': 'What is the name of the dog mat that is ruggedly constructed from recycled plastic materials, helping to keep dirt and water off the floors and plastic out of landfills?', 'answer': 'The name of the dog mat is Recycled Waterhog Dog Mat, Chevron Weave.'}}, {'qa_pairs': {'query': 'What is the name of the product described in the document that is suitable for Infant and Toddler Girls?', 'answer': "The product is called 'Infant and Toddler Girls' Coastal Chill Swimsuit, Two-Piece'."}}, {'qa_pairs': {'query': 'What is the primary material used in the construction of the Refresh Swimwear V-Neck Tankini, and what percentage of it is recycled?', 'answer': 'The primary material is nylon, with 82% of it being recycled nylon.'}}, {'qa_pairs': {'query': 'What is the material used for the EcoFlex 3L Storm Pants, according to the document?', 'answer': 'The EcoFlex 3L Storm Pants are made of 100% nylon, exclusive of trim.'}}]

所以这里我们需要进行一步提取;

python 复制代码
for example in new_examples:
    examples.append(example["qa_pairs"])

print(examples)

qa.invoke(examples[0]["query"])

Manual Evaluation

python 复制代码
import langchain
langchain.debug = True #开始debug模式,查看chain中的详细步骤

我们再次执行来查看chain中的细节;

LLM-assisted evaluation

那我们是不是可以使用语言模型来评估呢;

python 复制代码
langchain.debug = False #关闭debug模式

from langchain.evaluation.qa import QAEvalChain

让大语言模型来为我们每个example来生成答案;

python 复制代码
predictions = qa.apply(examples)

我们初始化一个评估链;

python 复制代码
eval_chain = QAEvalChain.from_llm(llm)

让大语言模型对实际答案和预测答案进行对比并给出一个评分;

python 复制代码
graded_outputs = eval_chain.evaluate(examples, predictions)

最后,我们可以打印一下看看结果;

python 复制代码
for i, eg in enumerate(examples):
    print(f"Example {i}:")
    print("Question: " + predictions[i]['query'])
    print("Real Answer: " + predictions[i]['answer'])
    print("Predicted Answer: " + predictions[i]['result'])
    print("Predicted Grade: " + graded_outputs[i]['results'])
    print()
相关推荐
芯盾时代41 分钟前
安全大模型智驱网络和数据安全效能跃迁
网络·人工智能·安全·网络安全
彩讯股份3006341 小时前
打造多模态交互新范式|彩讯股份中标2025年中国移动和留言平台AI智能体研发项目
人工智能
思通数科大数据舆情2 小时前
工业安全零事故的智能守护者:一体化AI智能安防平台
人工智能·安全·目标检测·计算机视觉·目标跟踪·数据挖掘·知识图谱
AI360labs_atyun2 小时前
2025 高考:AI 都在哪些地方发挥了作用
人工智能·科技·ai·高考
Yxh181377845543 小时前
短视频矩阵系统技术saas源头6年开发构架
人工智能·矩阵
m0_634448894 小时前
图上合成:用于大型语言模型持续预训练的知识合成数据生成
人工智能·语言模型·自然语言处理
张较瘦_5 小时前
[论文阅读] 人工智能 | 利用负信号蒸馏:用REDI框架提升LLM推理能力
论文阅读·人工智能
1296004525 小时前
机器学习的可解释性
人工智能·深度学习·自然语言处理·transformer
何中应5 小时前
第一个人工智能(AI)问答Demo
java·人工智能·语言模型
InternLM5 小时前
论文分类打榜赛Baseline(2):InternLM昇腾硬件微调实践
人工智能·分类·大模型·internlm·书生大模型