【论文复现】QuestEval:《QuestEval: Summarization Asks for Fact-based Evaluation》

以下是复现论文《QuestEval: Summarization Asks for Fact-based Evaluation》(NAACL 2021)代码https://github.com/ThomasScialom/QuestEval/的流程记录:

  1. 在服务器上conda创建虚拟环境questeval(python版本于readme保持一致,為3.9)

    bash 复制代码
    conda create -n questeval python=3.9
  2. git clone下载项目代码于本地,用pycharm打开并远程连接到服务器的该环境中。

  3. 服务器上进入该项目目录,安裝本项目需要的库。

    bash 复制代码
    pip install -e .
  4. 这个项目作者开源的其实是功能 ,真正的主程序要自己创建,不过作者封装得很漂亮,只需要在项目根目录下新建一个python文件,如run.py,然后拷贝以下内容:(大体作者都在README.md中给出了,这里我是想用QuestEval模型去做摘要事实一致性检测

    python 复制代码
    from questeval.questeval_metric import QuestEval
    questeval = QuestEval(no_cuda=False, task="summarization", do_weighter=True)
    
    source_1 = "Since 2000, the recipient of the Kate Greenaway medal has also been presented with the Colin Mears award to the value of 35000."
    prediction_1 = "Since 2000, the winner of the Kate Greenaway medal has also been given to the Colin Mears award of the Kate Greenaway medal."
    references_1 = [
        "Since 2000, the recipient of the Kate Greenaway Medal will also receive the Colin Mears Awad which worth 5000 pounds",
        "Since 2000, the recipient of the Kate Greenaway Medal has also been given the Colin Mears Award."
    ]
    
    source_2 = "He is also a member of another Jungiery boyband 183 Club."
    prediction_2 = "He also has another Jungiery Boyband 183 club."
    references_2 = [
        "He's also a member of another Jungiery boyband, 183 Club.",
        "He belonged to the Jungiery boyband 183 Club."
    ]
    
    
    if __name__ == "__main__":
        score = questeval.corpus_questeval(
            hypothesis=[prediction_1, prediction_2],
            sources=[source_1, source_2],
            list_references=[references_1, references_2]
        )
    
        print(score)
  5. 如果服务器能够顺利连接huggingface,那么直接执行就跑通了,作者的代码没有任何bug。然而对于服务器访问不了huggingface的朋友们(比如我qwq),那么就需要把++所有涉及远程加载模型的代码修改成本地加载的逻辑++。

    1. 先在huggingface把需要的模型给传进服务器里。我个人把下载好的模型文件会放在/dev_data_2/zkyao/pretrain_model/下。这里需要下载的模型有:t5-qa_squad2neg-en,t5-qg_squad1-en,t5-weighter_cnndm-en,bert-base-multilingual-cased

    2. 首先修改questeval/questeval_metric.py。作者把加载QuestEval框架 所涉及到的模型的逻辑全部写在了_load_all_models()方法中。修改这几个部分:

      python 复制代码
      # models['hyp']['QA'] = f'{HF_ORGANIZATION}/t5-qa_squad2neg-en'
      models['hyp']['QA'] = "/dev_data_2/zkyao/pretrain_model/t5-qa_squad2neg-en"
      # models['hyp']['QG'] = f'{HF_ORGANIZATION}/t5-qg_squad1-en'
      models['hyp']['QG'] = "/dev_data_2/zkyao/pretrain_model/t5-qg_squad1-en"
      python 复制代码
      # models['Weighter'] = self.get_model(model_name=f'{HF_ORGANIZATION}/t5-weighter_cnndm-en')
      models['Weighter'] = self.get_model(model_name="/dev_data_2/zkyao/pretrain_model/t5-weighter_cnndm-en")
    3. 接下來就是特别隐蔽的库源码了,因为huggingface提供的metrics组件内部实现逻辑,是要加载模型的。然而正不巧的是,这里用到的metric------bert_score,源码的开发者显然不会考虑到服务器访问不了huggingface的我们。

      bert_score库的scorer.py代码的这部分,将模型类型和模型路径同时用self.model_type属性指代,导致把逻辑写死了必须远程加载模型。

      为了能本地加载模型,不得不这样了。打开/{path_to_your_env}/lib/python3.9/site-packages/bert_score/scorer.py,作出如下修改:

  6. 接下来整个测试程序就能顺利执行了!

相关推荐
charlee442 天前
PandasAI连接LLM对MySQL数据库进行数据分析
mysql·数据分析·nlp·pandasai·deepseek
热心不起来的市民小周3 天前
True or False? 基于 BERT 学生数学问题误解检测
深度学习·nlp·bert
xiaoli23277 天前
课题学习笔记3——SBERT
笔记·学习·nlp·bert
java1234_小锋8 天前
[免费]【NLP舆情分析】基于python微博舆情分析可视化系统(flask+pandas+echarts)【论文+源码+SQL脚本】
python·flask·nlp·舆情分析·微博舆情分析
Easy数模8 天前
使用llm进行高级主题建模:通过利用 BERTopic 的表示模型和生成式 AI 深入探讨主题建模
人工智能·llm·nlp
在未来等你10 天前
RAG实战指南 Day 24:上下文构建与提示工程
nlp·rag·retrieval-augmented generation·prompt engineering·context management
AI大模型12 天前
基于 RAG 和 Claude 的智能文档聊天系统实战指南
程序员·llm·nlp
Easy数模12 天前
ModernBERT如何突破BERT局限?情感分析全流程解析
人工智能·深度学习·nlp·bert
Himon14 天前
LLM参数有效性学习综述
人工智能·算法·nlp