VLLM 格式化LLM输出

文章目录

前言

vllm OpenAI Compatible Server 提供了格式化LLM输出的能力，默认的格式化解码后端应该是outlines

目前提供了四个参数来控制格式化输出，分别是：

bash 复制代码

guided_json: 按照给定的json schema输出
guided_choice: 从给定的选项里面选一个
guided_regex: 按照给定的正则表达式输出
guided_grammar: 按照给定的 扩展巴科斯范式（EBNF）格式 的上下文无关语法输出（我也不懂）

下面我们直接看看如何使用这四个参数，控制LLM的输出

python 复制代码

import json
from openai import OpenAI

def chatgpt_base(system_prompt, user_prompt):
    api_key = "empty"
    base_url = "http://localhost:8000/v1"

    model = "Qwen1.5-14B-Chat-AWQ"

    client = OpenAI(api_key=api_key, base_url=base_url)

    completion = client.chat.completions.create(
        model=model,
        temperature=0,
        messages=[
            {"role": "system", "content": system_prompt},
            {"role": "user", "content": user_prompt},
        ],
    )
    return completion.choices[0].message.content

guided_json

python 复制代码

from pydantic import BaseModel


class Topic(BaseModel):
    问题: str
    答案: str


def chatgpt_guide_json(system_prompt, user_prompt):
    api_key = "empty"
    base_url = "http://localhost:8000/v1"

    model = "Qwen1.5-14B-Chat-AWQ"

    client = OpenAI(api_key=api_key, base_url=base_url)

    completion = client.chat.completions.create(
        model=model,
        temperature=0,
        messages=[
            {"role": "system", "content": system_prompt},
            {"role": "user", "content": user_prompt},
        ],
        extra_body={"guided_json": Topic.model_json_schema()},
    )

    return completion.choices[0].message.content


system_prompt = "You are a helpful assistant."
user_prompt = """
请你生成一对和python相关的问题和答案
"""
response = chatgpt_base(system_prompt, user_prompt)
print("base response: ", response)
print("----" * 5)
guide_reponse = chatgpt_guide_json(system_prompt, user_prompt)
print("guide json reponse: ", guide_reponse)

输出：

bash 复制代码

base response:  问题：如何在Python中安装一个新的库？

答案：在Python中，你可以使用pip工具来安装新的库。首先，你需要确保pip已安装。然后，打开命令行或终端，输入以下命令来安装所需的库：

    ```
    pip install 库名
    ```

    例如，如果你想安装requests库，你可以输入：

    ```
    pip install requests
    ```

这将从Python Package Index (PyPI)下载并安装requests库及其依赖项。
--------------------
guide json reponse:  { "问题": "如何在Python中安装一个新的库？", "答案": "在Python中，你可以使用pip工具来安装新的库。例如，如果你想安装requests库，你可以在命令行中输入：\">> pip install requests\"。这将会从Python Package Index (PyPI) 下载并安装requests库。" }

可以看到，即使我们不显式地在prompt中告诉LLM要返回JSON格式，我们拿到的响应竟还是JSON，并且符合我们给的格式。

我们还能给每个字段添加解释，如：

python 复制代码

class Topic(BaseModel):
    问题: str = Field(description="问题")
    答案: str = Field(description="答案")

guided_choice

python 复制代码

def chatgpt_guide_choice(system_prompt, user_prompt):
    api_key = "empty"
    base_url = "http://localhost:8000/v1"

    model = "Qwen1.5-14B-Chat-AWQ"

    client = OpenAI(api_key=api_key, base_url=base_url)

    completion = client.chat.completions.create(
        model=model,
        temperature=0,
        messages=[
            {"role": "system", "content": system_prompt},
            {"role": "user", "content": user_prompt},
        ],
        extra_body={"guided_choice": ["Positive", "Negative"]},
    )

    return completion.choices[0].message.content


system_prompt = "You are a helpful assistant."
user_prompt = """
Is the following review positive or negative?

Review: 今天天气真不错，好想出去吃大餐
"""
response = chatgpt_base(system_prompt, user_prompt)
print("base response: ", response)
print("----" * 5)
guide_reponse = chatgpt_guide_choice(system_prompt, user_prompt)
print("guide choice reponse: ", guide_reponse)

输出：

bash 复制代码

base response:  这条评论并不是在评价某个产品或服务，而是在描述天气并表达了想出去吃大餐的愿望。不过，如果要从情感色彩来看，这条评论是积极的，因为它提到了好天气，并且表达了积极的愿望。
--------------------
guide choice reponse:  Positive

注意：guide_choice 无法输出选项中的多个答案，即无法处理多标签任务

guided_regex

python 复制代码

def chatgpt_guide_regex(system_prompt, user_prompt):
    api_key = "empty"
    base_url = "http://localhost:8000/v1"

    model = "Qwen1.5-14B-Chat-AWQ"

    client = OpenAI(api_key=api_key, base_url=base_url)

    completion = client.chat.completions.create(
        model=model,
        temperature=0,
        messages=[
            {"role": "system", "content": system_prompt},
            {"role": "user", "content": user_prompt},
        ],
        extra_body={
            "guided_regex": r"((25[0-5]|2[0-4]\d|[01]?\d\d?)\.){3}(25[0-5]|2[0-4]\d|[01]?\d\d?)"
        },
    )

    return completion.choices[0].message.content


system_prompt = "You are a helpful assistant."
user_prompt = """
What is the IP address of the Google DNS servers? 
"""
response = chatgpt_base(system_prompt, user_prompt)
print("base response: ", response)
print("----" * 5)
guide_reponse = chatgpt_guide_regex(system_prompt, user_prompt)
print("guide regex reponse: ", guide_reponse)

输出：

bash 复制代码

base response:  The IP addresses for the Google Public DNS servers are as follows:

- 8.8.8.8 (primary server)
- 8.8.4.4 (secondary server)

These can be used as your DNS servers to take advantage of Google's DNS service.
--------------------
guide regex reponse:  1.8.8.88

从这个输出结果来看，使用格式化输出似乎会导致LLM效果下降？

guided_grammar

python 复制代码

arithmetic_grammar = r"""
    ?start: expression

    ?expression: term (("+" | "-") term)*

    ?term: factor (("*" | "/") factor)*

    ?factor: NUMBER
           | "-" factor
           | "(" expression ")"

    %import common.NUMBER
"""

def chatgpt_guide_grammar(system_prompt, user_prompt):
    api_key = "empty"
    base_url = "http://localhost:8000/v1"

    model = "Qwen1.5-14B-Chat-AWQ"

    client = OpenAI(api_key=api_key, base_url=base_url)

    completion = client.chat.completions.create(
        model=model,
        temperature=0,
        messages=[
            {"role": "system", "content": system_prompt},
            {"role": "user", "content": user_prompt},
        ],
        extra_body={"guided_grammar": arithmetic_grammar},
    )

    return completion.choices[0].message.content


system_prompt = "You are a helpful assistant."
user_prompt = """
Alice had 4 apples and Bob ate 2. Write an expression for Alice's apples:
"""
response = chatgpt_base(system_prompt, user_prompt)
print("base response: ", response)
print("----" * 5)
guide_reponse = chatgpt_guide_grammar(system_prompt, user_prompt)
print("guide grammar reponse: ", guide_reponse)

输出：

bash 复制代码

base response:  If Alice originally had 4 apples and Bob ate 2 of them, the expression for the number of apples Alice has left would be:

\[ 4 - 2 \]

So, Alice now has:

\[ 4 - 2 = 2 \]

Therefore, the expression for the number of apples Alice has left is \( 4 - 2 \).
--------------------
guide grammar reponse: (4-2)

这个咱也不懂，就不乱讲了，各位同学可以自行探索

感兴趣的同学可以看看：EBNF

总结

输出JSON还可以通过 response_format 控制，具体介绍可以查看vllm官方文档

这几个例子，也可以通过 outlines 仓库学习具体的用法