ChatGLM3报错:No chat template is defined for this tokenizer

使用官方提供的脚本创建ChatGLM3的DEMO:

bash 复制代码
cd basic_demo
python web_demo_gradio.py

出现效果异常问题:

====conversation====

[{'role': 'user', 'content': '你好'}, {'role': 'assistant', 'content': '你好,有什么我可以帮助你的吗?\n\n<|im_end|>'}, {'role': 'user', 'content': '你好'}]

No chat template is defined for this tokenizer - using a default chat template that implements the ChatML format (without BOS/EOS tokens!). If the default is not appropriate for your model, please set `tokenizer.chat_template` to an appropriate template. See https://huggingface.co/docs/transformers/main/chat_templating for more information.

原因分析:

  • 模型版本与代码不匹配,tokenizer_config.json配置文件中缺少prompt模板
  • 官方代码存在问题,尚不支持本地模型使用apply_chat_template方法

解决方案:修改tokenizer方式,不要使用apply_chat_template方法,单轮对话可以改用build_chat_input方法

python 复制代码
def predict(history, max_length, top_p, temperature):
    stop = StopOnTokens()
    messages = []
    for idx, (user_msg, model_msg) in enumerate(history):
        if idx == len(history) - 1 and not model_msg:
            messages.append({"role": "user", "content": user_msg})
            break
        if user_msg:
            messages.append({"role": "user", "content": user_msg})
        if model_msg:
            messages.append({"role": "assistant", "content": model_msg})

    print("\n\n====conversation====\n", messages)
    model_inputs = tokenizer.apply_chat_template(messages,
                                                 add_generation_prompt=True,
                                                 tokenize=True,
                                                 return_tensors="pt").to(next(model.parameters()).device)
    print('debug: old: model_inputs: {}'.format(model_inputs))
    model_inputs = tokenizer.build_chat_input(messages[-1]['content'], history=None, role="user").input_ids.to(model.device)
    print('debug: new: model_inputs: {}'.format(model_inputs))
    streamer = TextIteratorStreamer(tokenizer, timeout=60, skip_prompt=True, skip_special_tokens=True)
    generate_kwargs = {
        "input_ids": model_inputs,
        "streamer": streamer,
        "max_new_tokens": max_length,
        "do_sample": True,
        "top_p": top_p,
        "temperature": temperature,
        "stopping_criteria": StoppingCriteriaList([stop]),
        "repetition_penalty": 1.2,
    }
    t = Thread(target=model.generate, kwargs=generate_kwargs)
    t.start()

    for new_token in streamer:
        if new_token != '':
            history[-1][1] += new_token
            yield history

tokenizer.chat_template介绍

Next time you use apply_chat_template(), it will use your new template! This attribute will be saved in the tokenizer_config.json file, so you can use push_to_hub() to upload your new template to the Hub and make sure everyone's using the right template for your model!

设置tokenizer.chat_template属性后,下次使用apply_chat_template()时,将使用您的新模板!此属性保存在tokenizer_config.json文件中,因此您可以用push_to_hub()将新模板上传到Hub,确保大家都能使用正确的模板!
If a model does not have a chat template set, but there is a default template for its model class, the ConversationalPipeline class and methods like apply_chat_template will use the class template instead. You can find out what the default template for your tokenizer is by checking the tokenizer.default_chat_template attribute.

如果模型没有设置聊天模板,但有其模型类的默认模板,则ConversationalPipeline类和apply_chat_template等方法将使用类模板代替。你可以通过检查tokenizer.default_chat_template属性来了解你的tokenizer的默认模板是什么。

python 复制代码
def predict(history, max_length, top_p, temperature):
    stop = StopOnTokens()
    messages = []
    for idx, (user_msg, model_msg) in enumerate(history):
        if idx == len(history) - 1 and not model_msg:
            messages.append({"role": "user", "content": user_msg})
            break
        if user_msg:
            messages.append({"role": "user", "content": user_msg})
        if model_msg:
            messages.append({"role": "assistant", "content": model_msg})

    print("\n\n====conversation====\n", messages)
    print('debug: tokenizer.chat_template:\n{}'.format(tokenizer.chat_template))
    print('debug: tokenizer.default_chat_template:\n{}'.format(tokenizer.default_chat_template))
    model_inputs = tokenizer.apply_chat_template(messages,
                                                 add_generation_prompt=True,
                                                 tokenize=True,
                                                 return_tensors="pt").to(next(model.parameters()).device)
    streamer = TextIteratorStreamer(tokenizer, timeout=600, skip_prompt=True, skip_special_tokens=True)
    generate_kwargs = {
        "input_ids": model_inputs,
        "streamer": streamer,
        "max_new_tokens": max_length,
        "do_sample": True,
        "top_p": top_p,
        "temperature": temperature,
        "stopping_criteria": StoppingCriteriaList([stop]),
        "repetition_penalty": 1.2,
    }
    t = Thread(target=model.generate, kwargs=generate_kwargs)
    t.start()

    for new_token in streamer:
        if new_token != '':
            history[-1][1] += new_token
            yield history
相关推荐
数字化营销工兵4 小时前
大语言模型(LLM)安全:十大风险、影响和防御措施
llm·大语言模型·数据安全·隐私保护
gz7seven13 小时前
BLIP-2模型的详解与思考
大模型·llm·多模态·blip·多模态大模型·blip-2·q-former
不爱说话郭德纲17 小时前
探索LLM前沿,共话科技未来
人工智能·算法·llm
我爱学Python!1 天前
解决复杂查询难题:如何通过 Self-querying Prompting 提高 RAG 系统效率?
人工智能·程序人生·自然语言处理·大模型·llm·大语言模型·rag
任某某01162 天前
第四期书生大模型实战营 - 基础岛闯关作业3 - 浦语提示词工程实践
llm
知来者逆3 天前
DrugLLM——利用大规模语言模型通过 Few-Shot 生成生物制药小分子
人工智能·语言模型·自然语言处理·llm·大语言模型·生物制药
waiting不是违停3 天前
MetaGPT实现多动作Agent
langchain·llm
HuggingFace4 天前
通用辅助生成: 使用任意辅助模型加速解码
llm
用户3157476081354 天前
带你在前端上玩转机器学习,从零去构建,训练一个模型!!
前端·机器学习·llm