没有NVIDIA显卡，基于 OpenVINO_GenAI 框架在Intel集显GPU上也能流畅本地运行14B大模型

一、OpenVINO_GenAI 框架介绍

在本专栏的上篇文章中，介绍了使用 OpenVINO 分别在 CPU、NPU、核显 GPU 上加速过 YOLO 系列模型的推理，对 OpenVINO 的推理能力已经有了比较直观的感受。

基于 Intel Ultra 处理器中的 NPU、核显 GPU 加速 YOLO12 模型推理运算

而大模型的推理流程要复杂得多，一次完整的生成包含分词、多轮自回归解码、KV cache 管理、采样策略、停止条件判断等环节，如果继续用 OpenVINO Runtime 的原生 API 手工实现这些逻辑，代码量会迅速膨胀。

OpenVINO GenAI 正是为了解决这个问题而推出的，它是构建在 OpenVINO Runtime 之上的一层生成式 AI 高层库，专门封装了大模型的完整流水线，如：自动处理 tokenizer、循环解码、KV Cache、流式输出、采样策略等，主打本地推理、轻量、低依赖、以及结合 Intel 硬件加速本地推理场景。

其中流水线预置多种常用流水线，覆盖文本生成 LLMPipeline、多模态模型VLMPipeline 等，其中 LLMPipeline 是本文下面实验使用到的，初始化之后调用 generate 就能完成一轮对话。

本次实验模型层面本文选取 Qwen3-14B 模型，在推理能力、指令遵循和多语言能力之间取得了不错的平衡。实验会在核显 GPU 上完成对话推理、工具调用，并封装为 OpenAI 协议接口供上层应用调用。

本次实验使用的硬件配置：

设备类型	型号
CPU	Intel® Core™ Ultra X7 358H
核显 GPU	Intel® Arc™ B390 GPU (iGPU)

二、下载 Qwen3-14B 模型

下载模型，这里使用 modelscope 加速下载：

shell 复制代码

modelscope download --model="Qwen/Qwen3-14B" --local_dir Qwen3-14B

下载完成后的内容：

三、模型转化为OV模型

LLMPipeline 需要使用 OpenVINO IR 格式的模型，首先第一步需要做格式的转换，optimum-intel 依赖包已经实现了相关的转换功能，具体可以参考如下逻辑，本质是执行了 optimum_cli 指令，这里导出的过程做了量化处理，转化为了 Int4 格式，可以极大的减少内存的占用情况。

注意将 Qwen3-14B 模型的地址换成你电脑的路径：

python 复制代码

import os
import subprocess
import sys

# 待导出的 HuggingFace 模型
HF_MODEL = "F:/model/Qwen3-14B"

# 导出目标目录
OV_MODEL_DIR = "ov_qwen3_14b"
TASK = "text-generation-with-past"


def validate_ir(out_dir):
    """
    校验导出的 IR 是否带 KV cache（LLMPipeline 必须）

    :param out_dir: OpenVINO IR 模型目录
    :return: (是否可用, 输入名列表, 输出名列表)
    """
    import openvino as ov

    model = ov.Core().read_model(os.path.join(out_dir, "openvino_model.xml"))
    inputs = [i.get_any_name() for i in model.inputs]
    outputs = [o.get_any_name() for o in model.outputs]

    # 带 KV cache 的 decoder 模型：状态（KV cache）被隐藏进 model 内部，
    # 所以 outputs 只有 logits，要通过 get_sinks() 查内部状态，并检查 beam_idx 输入
    num_states = len(model.get_sinks())
    has_beam_idx = any("beam_idx" in n for n in inputs)
    has_past = any("present" in n for n in outputs)

    print(f"  输入: {inputs}")
    print(f"  输出: {outputs}")
    print(f"  内部状态数 (KV cache): {num_states}")

    # stateful 导出：KV cache 是内部状态 + 有 beam_idx 输入
    # 非 stateful 导出：KV cache 以 present.* 暴露在 outputs 里
    if not has_beam_idx and not has_past:
        print("\n  [失败] IR 校验未通过：没有 KV cache。")
        return False, inputs, outputs

    print("  [通过] IR 校验通过（带 KV cache）")
    return True, inputs, outputs


def export_int4(hf_model=HF_MODEL, out_dir=OV_MODEL_DIR, weight_format="int4",
                task=TASK):

    # 用 -m 调模块而不是直接调 optimum-cli，保证用的是同一个解释器
    cmd = [
        sys.executable, "-m", "optimum.commands.optimum_cli", "export", "openvino",
        "--model", hf_model,
        "--task", task,
        "--weight-format", weight_format,
        "--trust-remote-code",
        out_dir,
    ]
    print("执行导出命令：")
    print(" ".join(cmd))
    print()
    subprocess.run(cmd, check=True)
    print(f"\n导出完成: {out_dir}")
    print("校验 IR：")
    validate_ir(out_dir)

    return out_dir


if __name__ == "__main__":
    export_int4()

执行过程：

转换模型内容：

四、运行模型

模型转换完成后就可以加载运行了。使用 ov_genai.LLMPipeline 构建 LLM 流水线，把前面导出的 IR 目录和推理设备传进去即可。

为了让运行结果便于观察，下面把推理过程封装成了一个 OpenVINOChat 类，还通过 Pipeline 的性能指标接口采集了首 token 延迟和吞吐数据：

python 复制代码

from dataclasses import dataclass
from pathlib import Path
import time

import openvino_genai as ov_genai

# OpenVINO IR 模型目录
MODEL_DIR = "ov_qwen3_14b"
# 使用设备 GPU / CPU
DEVICE = "GPU"
# 最大生成 token 数
MAX_NEW_TOKENS = 1024
SYSTEM_PROMPT = "你叫小毕超，是一个超级人工智能助手"


@dataclass
class ChatResult:
    text: str
    token_count: int
    first_token_time: float | None
    total_time: float
    throughput: float | None


class OpenVINOChat:
    def __init__(self, model_dir=MODEL_DIR, device=DEVICE,
                 max_new_tokens=MAX_NEW_TOKENS, system_prompt=SYSTEM_PROMPT):
        if max_new_tokens < 1:
            raise ValueError("max_new_tokens 必须大于 0")
        model_path = Path(model_dir)
        if not model_path.is_absolute():
            model_path = Path(__file__).resolve().parent / model_path
        self.device = device
        self.max_new_tokens = max_new_tokens
        self.system_prompt = system_prompt
        self.pipe = ov_genai.LLMPipeline(str(model_path), device)
        self.tokenizer = self.pipe.get_tokenizer()
        self.reset()

    def reset(self):
        self.messages = [{"role": "system", "content": self.system_prompt}]

    @staticmethod
    def _streamer(fragment):
        print(fragment, end="", flush=True)
        return False

    def chat(self, question: str, stream: bool = True, *, extra_context: dict | None = None) -> ChatResult:
        if not question.strip():
            raise ValueError("问题不能为空")
        user_message = {"role": "user", "content": question}
        prompt = self.tokenizer.apply_chat_template(
            self.messages + [user_message],
            add_generation_prompt=True,
            extra_context=extra_context,
        )
        config = ov_genai.GenerationConfig()
        config.max_new_tokens = self.max_new_tokens
        # prompt 已应用聊天模板，生成时不再重复应用。
        config.apply_chat_template = False

        start = time.perf_counter()
        result = self.pipe.generate([prompt], config, self._streamer if stream else None)
        total_time = time.perf_counter() - start
        metrics = result.perf_metrics
        token_count = metrics.get_num_generated_tokens()
        response = ChatResult(
            text=result.texts[0],
            token_count=token_count,
            first_token_time=metrics.get_ttft().mean / 1000 if token_count else None,
            total_time=total_time,
            throughput=metrics.get_throughput().mean if token_count > 1 else None,
        )
        # 成功生成后再更新历史，失败的请求不留在对话中。
        self.messages.extend([user_message, {"role": "assistant", "content": response.text}])
        return response


def print_metrics(result: ChatResult, device: str):
    print(f"\n{'-' * 50}")
    print(f"设备: {device}")
    print(f"生成 token 数: {result.token_count}")
    ttft = f"{result.first_token_time:.3f}s" if result.first_token_time is not None else "N/A"
    throughput = f"{result.throughput:.2f} token/s" if result.throughput is not None else "N/A"
    print(f"首 token 延迟: {ttft}")
    print(f"总耗时: {result.total_time:.2f}s")
    print(f"吞吐: {throughput}")
    print(f"{'-' * 50}\n")


def main():
    print(f"加载模型: {MODEL_DIR} @ {DEVICE}...")
    start = time.perf_counter()
    chat = OpenVINOChat()
    print(f"编译完成，耗时 {time.perf_counter() - start:.2f}s\n")
    try:
        while True:
            question = input("请输入：")
            if not question.strip():
                continue
            if question == "q":
                break
            result = chat.chat(question)
            print_metrics(result, chat.device)
    except (EOFError, KeyboardInterrupt):
        print()


if __name__ == "__main__":
    main()

运行效果如下，执行期间可以随意提出对话问题：

推理速度平均在 14 token/s，这个速度谈不上很快，但已经超过了人正常的阅读速度，本地使用的话速度还是可以接受的，而且整个过程中 CPU 占用很低，浏览网页、写代码等其他操作基本不受影响。

以下是运行期间 CPU、GPU占用情况：

Qwen3 默认是开启思考的，如果需要关闭思考可在 extra_context 中添加：{"enable_thinking": False}，例如：

python 复制代码

def main():
    print(f"加载模型: {MODEL_DIR} @ {DEVICE}...")
    start = time.perf_counter()
    chat = OpenVINOChat()
    print(f"编译完成，耗时 {time.perf_counter() - start:.2f}s\n")
    try:
        while True:
            question = input("请输入：")
            if not question.strip():
                continue
            if question == "q":
                break
            result = chat.chat(question, extra_context={"enable_thinking": False})
            print_metrics(result, chat.device)
    except (EOFError, KeyboardInterrupt):
        print()

再次运行提问：

五、工具调用示例

只要模型支持工具调用，使用 LLMPipeline 同样也支持工具调用，以下是对上述调用过程增加了工具调用的支持的逻辑，并且增加了一个网络搜索的工具作为测试使用：

注意：网络搜索采用的 Tavily ，可以将你的 API KEY 放到环境变量 TAVILY_API_KEY 中，或者换成其他的测试工具

python 复制代码

from dataclasses import dataclass, field
import json
import os
from pathlib import Path
import time
import uuid
from typing import Literal

import openvino_genai as ov_genai
from tavily import TavilyClient

MODEL_DIR = "ov_qwen3_14b"
DEVICE = "GPU"
MAX_NEW_TOKENS = 1024
SYSTEM_PROMPT = "你是一个超级人工智能助手。需要最新或不确定的信息时，先调用 web_search 搜索，回答时引用来源链接。"
MAX_TOOL_ROUNDS = 4

TOOLS = [{"type": "function", "function": {
    "name": "web_search",
    "description": "搜索互联网，获取最新信息、新闻及相关来源链接。",
    "parameters": {"type": "object", "properties": {
        "query": {"type": "string", "description": "搜索关键词或需要查询的问题。"},
        "max_results": {"type": "integer", "description": "最多返回的搜索结果数量，范围为 1 至 20，默认返回 5 条。", "minimum": 1, "maximum": 20, "default": 5},
        "topic": {"type": "string", "description": "搜索主题：general 表示通用搜索，news 表示新闻搜索，finance 表示财经搜索；默认为通用搜索。", "enum": ["general", "news", "finance"], "default": "general"},
        "include_raw_content": {"type": "boolean", "description": "是否返回网页原始内容，默认不返回。", "default": False},
    }, "required": ["query"], "additionalProperties": False},
}}]


def web_search(query: str, max_results: int = 5,
               topic: Literal["general", "news", "finance"] = "general",
               include_raw_content: bool = False):
    api_key = os.environ.get("TAVILY_API_KEY")
    if not api_key:
        raise ValueError("Set TAVILY_API_KEY before using web_search")
    return TavilyClient(api_key=api_key, timeout=30).search(
        query, max_results=max_results, topic=topic, include_raw_content=include_raw_content,
    )


@dataclass
class ChatResult:
    text: str
    token_count: int
    first_token_time: float | None
    total_time: float
    throughput: float | None
    tool_calls: list[dict] = field(default_factory=list)


def parse_tool_calls(text: str):
    # Tool markup inside thinking is not an actual tool request.
    thinking, separator, answer = text.partition("</think>")
    if separator:
        prefix, remaining = thinking + separator, answer
    elif text.lstrip().startswith("<think>"):
        return text, []
    else:
        prefix, remaining = "", text
    parts, calls = [prefix], []
    while "<tool_call>" in remaining:
        before, _, remaining = remaining.partition("<tool_call>")
        payload, end, remaining = remaining.partition("</tool_call>")
        if not end:
            raise ValueError("Incomplete tool call; increase max_new_tokens")
        call = json.loads(payload)
        if not isinstance(call, dict) or not isinstance(call.get("name"), str) or not call["name"]:
            raise ValueError("Invalid tool name")
        arguments = call.get("arguments")
        if isinstance(arguments, str):
            arguments = json.loads(arguments)
        if not isinstance(arguments, dict):
            raise ValueError("Tool arguments must be a JSON object")
        calls.append({"id": "call_" + uuid.uuid4().hex, "type": "function", "function": {
            "name": call["name"], "arguments": json.dumps(arguments, ensure_ascii=False),
        }})
        parts.append(before)
    return "".join(parts) + remaining, calls


class OpenVINOChat:
    def __init__(self, model_dir=MODEL_DIR, device=DEVICE,
                 max_new_tokens=MAX_NEW_TOKENS, system_prompt=SYSTEM_PROMPT):
        if max_new_tokens < 1:
            raise ValueError("max_new_tokens must be positive")
        model_path = Path(model_dir)
        if not model_path.is_absolute():
            model_path = Path(__file__).resolve().parent / model_path
        self.device = device
        self.max_new_tokens = max_new_tokens
        self.system_prompt = system_prompt
        self.pipe = ov_genai.LLMPipeline(str(model_path), device)
        self.tokenizer = self.pipe.get_tokenizer()
        self.reset()

    def reset(self):
        self.messages = [{"role": "system", "content": self.system_prompt}]
        self._pending_tool_calls = set()

    @staticmethod
    def _streamer(fragment):
        print(fragment, end="", flush=True)
        return False

    def add_tool_result(self, tool_call_id: str, content: str):
        if tool_call_id not in self._pending_tool_calls or not isinstance(content, str):
            raise ValueError("Supply an unanswered tool_call_id and string result")
        self.messages.append({"role": "tool", "tool_call_id": tool_call_id, "content": content})
        self._pending_tool_calls.remove(tool_call_id)

    def chat(self, question: str | None = None, stream: bool = True, *,
             tools: list[dict] | None = None, extra_context: dict | None = None) -> ChatResult:
        if self._pending_tool_calls:
            raise ValueError("Submit all pending tool results first")
        if question is None and self.messages[-1]["role"] != "tool":
            raise ValueError("Supply a question or return tool results before continuing")
        if question is not None and (not isinstance(question, str) or not question.strip()):
            raise ValueError("question must not be empty")
        messages = self.messages.copy()
        if question is not None:
            messages.append({"role": "user", "content": question})
        prompt = self.tokenizer.apply_chat_template(
            messages, add_generation_prompt=True, tools=tools, extra_context=extra_context,
        )
        config = ov_genai.GenerationConfig()
        config.max_new_tokens = self.max_new_tokens
        config.apply_chat_template = False
        start = time.perf_counter()
        result = self.pipe.generate([prompt], config, self._streamer if stream else None)
        total_time = time.perf_counter() - start
        text, calls = parse_tool_calls(result.texts[0])
        metrics = result.perf_metrics
        token_count = metrics.get_num_generated_tokens()
        response = ChatResult(
            text=text, token_count=token_count,
            first_token_time=metrics.get_ttft().mean / 1000 if token_count else None,
            total_time=total_time,
            throughput=metrics.get_throughput().mean if token_count > 1 else None,
            tool_calls=calls,
        )
        assistant = {"role": "assistant", "content": text}
        if calls:
            assistant["tool_calls"] = calls
        self.messages = messages + [assistant]
        self._pending_tool_calls = {call["id"] for call in calls}
        return response


def print_metrics(result: ChatResult, device: str):
    ttft = f"{result.first_token_time:.3f}s" if result.first_token_time is not None else "N/A"
    throughput = f"{result.throughput:.2f} token/s" if result.throughput is not None else "N/A"
    print(f"\n{'-' * 50}")
    print(f"Device: {device}; generated tokens: {result.token_count}")
    print(f"Time to first token: {ttft}; total time: {result.total_time:.2f}s")
    print(f"Throughput: {throughput}")
    print(f"{'-' * 50}\n")


def main():
    print(f"加载模型: {MODEL_DIR} @ {DEVICE}...")
    start = time.perf_counter()
    chat = OpenVINOChat()
    print(f"编译完成，耗时 {time.perf_counter() - start:.2f}s\n")
    options = {"tools": TOOLS, "extra_context": {"enable_thinking": False}}
    try:
        while True:
            question = input("请输入：")
            if not question.strip():
                continue
            if question == "q":
                break
            result = chat.chat(question, **options)
            print_metrics(result, chat.device)
            for _ in range(MAX_TOOL_ROUNDS):
                if not result.tool_calls:
                    break
                for call in result.tool_calls:
                    function = call["function"]
                    print(f"\n[tool] {function['name']}: {function['arguments']}")
                    # Tool execution belongs to the caller, not OpenVINOToolChat.
                    try:
                        if function["name"] != "web_search":
                            raise ValueError(f"Unknown tool: {function['name']}")
                        output = web_search(**json.loads(function["arguments"]))
                    except Exception as exc:
                        output = {"error": str(exc)}
                    chat.add_tool_result(call["id"], json.dumps(output, ensure_ascii=False))
                result = chat.chat(**options)
                print_metrics(result, chat.device)
            if result.tool_calls:
                print("Tool round limit reached; resetting the conversation.")
                chat.reset()
    except (EOFError, KeyboardInterrupt):
        print()


if __name__ == "__main__":
    main()

测试运行效果：

六、封装为 OpenAI 格式供上层应用使用

如果想接入 Cherry Studio 这类现成的客户端，或者让其他程序统一调用，就需要把功能封装成通用的接口。目前各类 AI 客户端和开发框架基本都支持 OpenAI 的 Chat Completions 协议，因此下面把 LLMPipeline 封装成一个兼容该协议的 HTTP 服务。

6.1 将上述调用过程封装为 OpenAI 协议接口

python 复制代码

import argparse
import asyncio
from concurrent.futures import TimeoutError as FutureTimeout
from contextlib import asynccontextmanager
from dataclasses import dataclass
from importlib.machinery import EXTENSION_SUFFIXES
import json
import logging
import os
from pathlib import Path
import secrets
import sys
import threading
import time
from typing import Any, Literal
import uuid
from fastapi import Depends, FastAPI, Request
from fastapi.exceptions import RequestValidationError
from fastapi.responses import JSONResponse, StreamingResponse
import openvino_genai as ov_genai
from pydantic import BaseModel, ConfigDict, Field
from starlette.exceptions import HTTPException
import uvicorn

logger = logging.getLogger("openvino_api")


class APIError(Exception):
    def __init__(self, message, status=400, param=None, code=None):
        self.status = status
        self.body = {"error": {
            "message": message,
            "type": "server_error" if status >= 500 else "invalid_request_error",
            "param": param,
            "code": code,
        }}


class Message(BaseModel):
    model_config = ConfigDict(extra="allow")
    role: Literal["system", "developer", "user", "assistant", "tool"]
    content: str | list[dict[str, Any]] | None = None
    tool_calls: list[dict[str, Any]] | None = None
    tool_call_id: str | None = None
    reasoning_content: str | None = None


class ChatRequest(BaseModel):
    model_config = ConfigDict(extra="allow", allow_inf_nan=False)
    model: str
    messages: list[Message] = Field(min_length=1)
    stream: bool = False
    stream_options: dict[str, Any] | None = None
    max_tokens: int | None = Field(default=None, ge=1)
    max_completion_tokens: int | None = Field(default=None, ge=1)
    temperature: float | None = Field(default=None, ge=0, le=2)
    top_p: float | None = Field(default=None, gt=0, le=1)
    frequency_penalty: float = Field(default=0, ge=-2, le=2)
    presence_penalty: float = Field(default=0, ge=-2, le=2)
    seed: int | None = Field(default=None, ge=0, le=2**32 - 1)
    stop: str | list[str] | None = None
    n: int = Field(default=1, ge=1)
    tools: list[dict[str, Any]] | None = None
    tool_choice: str | dict[str, Any] | None = None
    parallel_tool_calls: bool = True
    enable_thinking: bool = False
    response_format: dict[str, Any] | None = None


@dataclass
class Settings:
    model_dir: Path = Path(__file__).resolve().parent / "ov_qwen3_14b"
    model_name: str = "qwen3-14b"
    device: str = "GPU"
    api_key: str | None = None
    default_max_tokens: int = 1024
    max_output_tokens: int = 8192
    context_length: int = 8192


def normalize_request(body: ChatRequest, settings: Settings):
    if body.model != settings.model_name:
        raise APIError(f"Model '{body.model}' is not available.", 404, "model", "model_not_found")
    if body.n != 1:
        raise APIError("Only n=1 is supported.", param="n")
    if body.response_format and body.response_format.get("type") != "text":
        raise APIError("Only response_format.type='text' is supported.", param="response_format")
    extra = body.model_extra or {}
    for key in ("functions", "function_call", "logit_bias", "prediction", "audio"):
        if extra.get(key):
            raise APIError(f"'{key}' is not supported.", param=key)
    if extra.get("logprobs") or extra.get("top_logprobs"):
        raise APIError("Log probabilities are not supported.", param="logprobs")
    if extra.get("modalities", ["text"]) != ["text"]:
        raise APIError("Only text input and output are supported.", param="modalities")
    if body.max_tokens is not None and body.max_completion_tokens is not None:
        raise APIError("Specify max_tokens or max_completion_tokens, not both.")
    limit = body.max_completion_tokens or body.max_tokens or settings.default_max_tokens
    if limit > settings.max_output_tokens:
        raise APIError(f"Output limit exceeds {settings.max_output_tokens} tokens.", param="max_completion_tokens")
    stops = [body.stop] if isinstance(body.stop, str) else body.stop or []
    if len(stops) > 4 or any(not isinstance(s, str) or not s for s in stops):
        raise APIError("stop must contain one to four nonempty strings.", param="stop")

    tools = body.tools or []
    names = set()
    for tool in tools:
        function = tool.get("function")
        if tool.get("type") != "function" or not isinstance(function, dict):
            raise APIError("Only tools of type 'function' are supported.", param="tools")
        name = function.get("name")
        if not isinstance(name, str) or not name or name in names:
            raise APIError("Tool names must be nonempty and unique.", param="tools")
        if not isinstance(function.get("parameters", {}), dict):
            raise APIError("Tool parameters must be a JSON schema object.", param="tools")
        if function.get("strict"):
            raise APIError("Strict tool schema enforcement is not supported.", param="tools")
        names.add(name)
    choice = body.tool_choice or ("auto" if tools else "none")
    forced_name = None
    if isinstance(choice, dict):
        function = choice.get("function")
        forced_name = function.get("name") if isinstance(function, dict) else None
        if choice.get("type") != "function" or forced_name not in names:
            raise APIError("tool_choice must name an available function.", param="tool_choice")
        tools = [tool for tool in tools if tool["function"]["name"] == forced_name]
    elif choice not in ("auto", "none", "required"):
        raise APIError("Invalid tool_choice.", param="tool_choice")
    if choice != "none" and not tools:
        raise APIError("tool_choice requires tools.", param="tool_choice")
    if choice == "none":
        tools = []

    messages = []
    pending_calls = set()
    for message in body.messages:
        item = message.model_dump(exclude_none=True)
        content = message.content
        if isinstance(content, list):
            if any(part.get("type") != "text" or not isinstance(part.get("text"), str) for part in content):
                raise APIError("Only text content parts are supported.", param="messages")
            content = "".join(part["text"] for part in content)
        if message.role == "developer":
            item["role"] = "system"
        if message.tool_calls:
            if message.role != "assistant":
                raise APIError("tool_calls require the assistant role.", param="messages")
            for call in message.tool_calls:
                function = call.get("function")
                if (call.get("type") != "function" or not isinstance(call.get("id"), str)
                        or not isinstance(function, dict) or not isinstance(function.get("name"), str)
                        or not isinstance(function.get("arguments"), str)):
                    raise APIError("Invalid assistant tool call.", param="messages")
                try:
                    arguments = json.loads(function["arguments"])
                except (ValueError, TypeError):
                    raise APIError("Tool arguments must be a JSON string.", param="messages") from None
                if not isinstance(arguments, dict):
                    raise APIError("Tool arguments must describe an object.", param="messages")
                if call["id"] in pending_calls:
                    raise APIError("Assistant tool call IDs must be unique.", param="messages")
                pending_calls.add(call["id"])
        if message.role == "tool":
            if message.tool_call_id not in pending_calls:
                raise APIError("tool_call_id must refer to an unanswered assistant tool call.", param="messages")
            pending_calls.remove(message.tool_call_id)
        elif pending_calls and not message.tool_calls:
            raise APIError("Return tool results before continuing the conversation.", param="messages")
        if content is None and not (message.role == "assistant" and message.tool_calls):
            raise APIError("Message content is required.", param="messages")
        item["content"] = content or ""
        messages.append(item)
    if pending_calls:
        raise APIError("The conversation has unanswered tool calls.", param="messages")
    if choice == "required" or forced_name:
        instruction = "You must call a provided function in this response."
        if forced_name:
            instruction += f" Call the function named {forced_name}."
        if messages[0]["role"] == "system":
            messages[0]["content"] += "\n" + instruction
        else:
            messages.insert(0, {"role": "system", "content": instruction})
    return messages, tools, limit, stops, choice


class OutputParser:
    """Frame Qwen3 delimiters across chunks; parse function bodies with JSON."""

    def __init__(self, tool_names, enable_thinking=False, parallel=True):
        self.tool_names = set(tool_names)
        self.parallel = parallel
        self.mode = "initial"
        self.buffer = ""
        self.content = ""
        self.reasoning = ""
        self.tool_calls = []

    @staticmethod
    def split_partial(text, marker):
        for size in range(min(len(text), len(marker) - 1), 0, -1):
            if text.endswith(marker[:size]):
                return text[:-size], text[-size:]
        return text, ""

    def feed(self, text, final=False):
        self.buffer += text
        deltas = []
        while self.buffer:
            if self.mode == "initial":
                candidate = self.buffer.lstrip()
                if not final and (not candidate or "<think>".startswith(candidate)):
                    break
                if candidate.startswith("<think>"):
                    self.buffer = candidate[len("<think>"):]
                    self.mode = "reasoning_content"
                else:
                    self.mode = "content"
                continue
            if self.mode == "tool":
                end = self.buffer.find("</tool_call>")
                if end < 0:
                    if final:
                        raise APIError("Model returned an incomplete tool call; increase max_tokens.", 502)
                    break
                payload, self.buffer = self.buffer[:end], self.buffer[end + len("</tool_call>"):]
                try:
                    call = json.loads(payload)
                    name, arguments = call["name"], call["arguments"]
                    if isinstance(arguments, str):
                        arguments = json.loads(arguments)
                    if name not in self.tool_names or not isinstance(arguments, dict):
                        raise ValueError("Invalid tool name or arguments")
                except (ValueError, TypeError, KeyError):
                    raise APIError("Model returned invalid tool call JSON.", 502) from None
                if self.tool_calls and not self.parallel:
                    raise APIError("Model returned multiple calls with parallel_tool_calls=false.", 502)
                call = {"id": "call_" + uuid.uuid4().hex[:24], "type": "function",
                        "function": {"name": name, "arguments": json.dumps(arguments, ensure_ascii=False)}}
                deltas.append({"tool_calls": [{"index": len(self.tool_calls), **call}]})
                self.tool_calls.append(call)
                self.mode = "content"
                continue
            marker = "</think>" if self.mode == "reasoning_content" else "<tool_call>"
            parse_marker = self.mode == "reasoning_content" or bool(self.tool_names)
            index = self.buffer.find(marker) if parse_marker else -1
            if index >= 0:
                fragment = self.buffer[:index]
                self.buffer = self.buffer[index + len(marker):]
                next_mode = "content" if self.mode == "reasoning_content" else "tool"
            else:
                if final or not parse_marker:
                    fragment, self.buffer = self.buffer, ""
                else:
                    fragment, self.buffer = self.split_partial(self.buffer, marker)
                next_mode = self.mode
            if fragment:
                if self.mode == "reasoning_content":
                    self.reasoning += fragment
                else:
                    self.content += fragment
                deltas.append({self.mode: fragment})
            self.mode = next_mode
            if index < 0:
                break
        if final and self.mode == "tool":
            raise APIError("Model returned an incomplete tool call; increase max_tokens.", 502)
        return deltas

    def message(self):
        message = {"role": "assistant", "content": self.content or (None if self.tool_calls else "")}
        if self.reasoning:
            message["reasoning_content"] = self.reasoning
        if self.tool_calls:
            message["tool_calls"] = self.tool_calls
        return message


class OpenVINOEngine:
    def __init__(self, settings):
        self.settings = settings
        self.inference_lock = threading.Lock()
        logger.info("Loading %s on %s", settings.model_dir, settings.device)
        self.pipe = ov_genai.LLMPipeline(str(settings.model_dir), settings.device)
        self.tokenizer = self.pipe.get_tokenizer()
        model_config = json.loads((settings.model_dir / "config.json").read_text(encoding="utf-8"))
        self.context_length = min(settings.context_length, model_config.get("max_position_embeddings", settings.context_length))
        logger.info("Model ready; context limit: %d tokens", self.context_length)

    def generate(self, body, prepared, emit, cancelled):
        # A single pipeline has mutable inference state shared by all requests.
        with self.inference_lock:
            self._generate(body, prepared, emit, cancelled)

    def _generate(self, body, prepared, emit, cancelled):
        messages, tools, limit, stops, choice = prepared
        if cancelled.is_set():
            return
        prompt = self.tokenizer.apply_chat_template(
            messages, add_generation_prompt=True, tools=tools,
            extra_context={"enable_thinking": body.enable_thinking},
        )
        inputs = self.tokenizer.encode(prompt, add_special_tokens=False)
        input_count = inputs.input_ids.shape[-1]
        if input_count + limit > self.context_length:
            raise APIError(f"Input ({input_count}) plus output limit ({limit}) exceeds context limit ({self.context_length}).", param="messages", code="context_length_exceeded")
        config = self.pipe.get_generation_config()
        config.max_new_tokens = limit
        config.apply_chat_template = False
        config.num_return_sequences = 1
        config.num_beams = 1
        config.frequency_penalty = body.frequency_penalty
        config.presence_penalty = body.presence_penalty
        if body.temperature is not None:
            config.do_sample = body.temperature > 0
            if config.do_sample:
                config.temperature = body.temperature
        if body.top_p is not None:
            config.top_p = body.top_p
        if body.seed is not None:
            config.rng_seed = body.seed
        config.stop_strings = set(stops)
        config.include_stop_str_in_output = False
        config.validate()
        parser = OutputParser([tool["function"]["name"] for tool in tools], body.enable_thinking, body.parallel_tool_calls)
        emit("ready", None)

        def streamer(fragment):
            if cancelled.is_set():
                return ov_genai.StreamingStatus.CANCEL
            for delta in parser.feed(fragment):
                if body.stream:
                    emit("delta", delta)
            return ov_genai.StreamingStatus.RUNNING

        result = self.pipe.generate(inputs, config, streamer)
        if cancelled.is_set():
            return
        for delta in parser.feed("", final=True):
            if body.stream:
                emit("delta", delta)
        if (choice == "required" or isinstance(choice, dict)) and not parser.tool_calls:
            raise APIError("Model did not produce the required tool call.", 502)
        metrics = result.perf_metrics
        prompt_tokens = metrics.get_num_input_tokens()
        completion_tokens = metrics.get_num_generated_tokens()
        reasons = result.finish_reasons
        length = bool(reasons and reasons[0] == ov_genai.GenerationFinishReason.LENGTH)
        if not reasons:
            length = completion_tokens >= limit
        finish_reason = "length" if length else "tool_calls" if parser.tool_calls else "stop"
        logger.info("Generated %d tokens; TTFT %.1f ms; %.2f tokens/s", completion_tokens,
                    metrics.get_ttft().mean, metrics.get_throughput().mean)
        emit("done", {"message": parser.message(), "finish_reason": finish_reason,
                      "usage": {"prompt_tokens": prompt_tokens, "completion_tokens": completion_tokens,
                                "total_tokens": prompt_tokens + completion_tokens}})


def emit_event(events, loop, cancelled, kind, payload):
    if cancelled.is_set():
        return
    # This buffer transfers stream fragments, not pending inference requests.
    pending = asyncio.run_coroutine_threadsafe(events.put((kind, payload)), loop)
    while True:
        try:
            pending.result(timeout=0.2)
            return
        except FutureTimeout:
            if cancelled.is_set():
                pending.cancel()
                return


async def next_event(events, cancelled, request):
    while True:
        if await request.is_disconnected():
            cancelled.set()
            raise APIError("Client disconnected.", 499)
        try:
            return await asyncio.wait_for(events.get(), timeout=0.25)
        except asyncio.TimeoutError:
            pass


def sse(payload):
    return "data: " + json.dumps(payload, ensure_ascii=False) + "\n\n"


def create_app(settings=None, engine_factory=OpenVINOEngine):
    settings = settings or Settings()

    @asynccontextmanager
    async def lifespan(app):
        generations = {}
        app.state.generations = generations
        try:
            app.state.engine = await asyncio.to_thread(engine_factory, settings)
            yield
        finally:
            active_generations = list(generations.items())
            for _, cancelled in active_generations:
                cancelled.set()
            for thread, _ in active_generations:
                await asyncio.to_thread(thread.join)

    app = FastAPI(title="OpenVINO Chat Completions API", lifespan=lifespan)

    @app.exception_handler(APIError)
    async def api_error(request, exc):
        return JSONResponse(exc.body, status_code=exc.status)

    @app.exception_handler(RequestValidationError)
    async def validation_error(request, exc):
        error = exc.errors()[0]
        param = ".".join(str(part) for part in error["loc"] if part != "body")
        return JSONResponse(APIError(error["msg"], param=param or None).body, status_code=400)

    @app.exception_handler(HTTPException)
    async def http_error(request, exc):
        return JSONResponse(APIError(str(exc.detail), exc.status_code).body, status_code=exc.status_code)

    async def authenticate(request: Request):
        if settings.api_key:
            supplied = request.headers.get("authorization", "")
            if not secrets.compare_digest(supplied.encode(), ("Bearer " + settings.api_key).encode()):
                raise APIError("Invalid API key.", 401, code="invalid_api_key")

    @app.get("/health")
    async def health():
        return {"status": "ok", "model": settings.model_name, "device": settings.device}

    def model_info():
        return {"id": settings.model_name, "object": "model", "created": 0, "owned_by": "local"}

    @app.get("/v1/models", dependencies=[Depends(authenticate)])
    async def models():
        return {"object": "list", "data": [model_info()]}

    @app.get("/v1/models/{model_id}", dependencies=[Depends(authenticate)])
    async def model(model_id: str):
        if model_id != settings.model_name:
            raise APIError("Model not found.", 404, "model", "model_not_found")
        return model_info()

    @app.post("/v1/chat/completions", dependencies=[Depends(authenticate)])
    async def completions(body: ChatRequest, request: Request):
        prepared = normalize_request(body, settings)
        loop = asyncio.get_running_loop()
        events = asyncio.Queue(maxsize=128)
        cancelled = threading.Event()

        def emit(kind, payload):
            emit_event(events, loop, cancelled, kind, payload)

        def run():
            try:
                app.state.engine.generate(body, prepared, emit, cancelled)
            except APIError as exc:
                emit("error", exc)
            except Exception:
                logger.exception("Inference failed")
                emit("error", APIError("OpenVINO inference failed. See server log.", 500))
            finally:
                loop.call_soon_threadsafe(app.state.generations.pop, thread, None)

        thread = threading.Thread(target=run, name="openvino-request")
        app.state.generations[thread] = cancelled
        thread.start()
        completion_id = "chatcmpl-" + uuid.uuid4().hex
        created = int(time.time())

        def chunk(delta, finish_reason=None):
            payload = {"id": completion_id, "object": "chat.completion.chunk", "created": created,
                       "model": settings.model_name, "choices": [{"index": 0, "delta": delta,
                       "logprobs": None, "finish_reason": finish_reason}]}
            if body.stream_options and body.stream_options.get("include_usage"):
                payload["usage"] = None
            return payload

        try:
            kind, payload = await next_event(events, cancelled, request)
            if kind == "error":
                raise payload
            if not body.stream:
                kind, payload = await next_event(events, cancelled, request)
                if kind == "error":
                    raise payload
                return {"id": completion_id, "object": "chat.completion", "created": created,
                        "model": settings.model_name, "choices": [{"index": 0, "message": payload["message"],
                        "logprobs": None, "finish_reason": payload["finish_reason"]}], "usage": payload["usage"]}
        except BaseException:
            cancelled.set()
            raise

        async def stream():
            try:
                yield sse(chunk({"role": "assistant", "content": ""}))
                while True:
                    kind, payload = await next_event(events, cancelled, request)
                    if kind == "error":
                        yield sse(payload.body)
                        break
                    if kind == "done":
                        yield sse(chunk({}, payload["finish_reason"]))
                        if body.stream_options and body.stream_options.get("include_usage"):
                            usage_chunk = chunk({})
                            usage_chunk["choices"] = []
                            usage_chunk["usage"] = payload["usage"]
                            yield sse(usage_chunk)
                        break
                    yield sse(chunk(payload))
                yield "data: [DONE]\n\n"
            except APIError:
                pass
            finally:
                cancelled.set()

        return StreamingResponse(stream(), media_type="text/event-stream",
                                 headers={"Cache-Control": "no-cache", "X-Accel-Buffering": "no"})

    return app


def main():
    parser = argparse.ArgumentParser(description=__doc__)
    parser.add_argument("--model-dir", type=Path, default=Settings.model_dir)
    parser.add_argument("--model-name", default="qwen3-14b")
    parser.add_argument("--device", default="GPU")
    parser.add_argument("--host", default="127.0.0.1")
    parser.add_argument("--port", type=int, default=8000)
    parser.add_argument("--context-length", type=int, default=8192)
    parser.add_argument("--max-output-tokens", type=int, default=8192)
    parser.add_argument("--default-max-tokens", type=int, default=1024)
    args = parser.parse_args()
    if min(args.context_length, args.max_output_tokens, args.default_max_tokens) < 1:
        parser.error("Token limits must be positive.")
    if args.default_max_tokens > args.max_output_tokens:
        parser.error("default-max-tokens must not exceed max-output-tokens.")
    logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(name)s: %(message)s")
    settings = Settings(model_dir=args.model_dir.resolve(), model_name=args.model_name, device=args.device,
                        api_key=os.environ.get("OPENVINO_API_KEY"), context_length=args.context_length,
                        max_output_tokens=args.max_output_tokens, default_max_tokens=args.default_max_tokens)
    uvicorn.run(create_app(settings), host=args.host, port=args.port, workers=1)


if __name__ == "__main__":
    main()

启动服务：

6.2 使用 Cherry Studio 调用测试

添加一个自定义提供商，填写如下内容，其中 API 秘钥可以随便写，上面实现的 API 接口没有做相关的校验：

使获取模型列表，添加模型：

添加 Qwen3-14B 模型：

调用测试：