LiteLLM + vLLM模型调用引擎架构

susu10830189112026-03-19 19:40

二、Docker 安装 vLLM

docker-compose.yml

复制代码

version: '3.7'

services:

  vllm-qwen:
    image: vllm/vllm-openai:latest
    container_name: vllm-qwen
    runtime: nvidia
    environment:
      - NVIDIA_VISIBLE_DEVICES=all
    volumes:
      - ./models:/models
    command: >
      --model /models/qwen/Qwen2.5-0.5B-Instruct
      --host 0.0.0.0
      --port 8000
      --gpu-memory-utilization 0.5
      --max-model-len 1024
    ports:
      - "8000:8000"

  litellm:
    image: ghcr.io/berriai/litellm:main-latest
    container_name: litellm
    volumes:
      - ./config.yaml:/app/config.yaml
    command: --config /app/config.yaml
    ports:
      - "4000:4000"
    depends_on:
      - vllm-qwen

把模型放到models

LiteLLM 配置config.yaml

复制代码

model_list:
  - model_name: qwen
    litellm_params:
      model: openai//models/qwen/Qwen2.5-0.5B-Instruct  # 使用 vLLM 返回的完整模型 ID
      api_base: http://vllm-qwen:8000/v1
      api_key: none

启动服务

docker compose up -d

此过程比较慢，因为下载的比较大。

测试 vLLM

curl http://localhost:8000/v1/models

测试 LiteLLM

curl http://localhost:4000/v1/models

整体测试：

curl http://localhost:4000/v1/chat/completions -H "Content-Type: application/json" -d "{\"model\":\"qwen\",\"messages\": ${\\"role\\":\\"user\\",\\"content\\":\\"你好\\"}$ }"

python代码测试：

复制代码

from openai import OpenAI

client = OpenAI(
    api_key="anything",
    base_url="http://10.61.104.181:4000/v1"
)

response = client.chat.completions.create(
    model="qwen",
    messages=[
        {"role": "user", "content": "你好，讲个笑话"}
    ]
)
 
print(response.choices[0].message.content)

增加多个模型（暂未尝试）