先介绍一下系统环境
大模型搭建在局域网的一台ubuntu 24.04.3上
LLM的机器配置:
4块A100-40Gb的GPU卡
Ollama 0.12.6
litellm 1.78.7
客户机配置:
windows11
ollama配置
ollama安装略了,网上一堆,这里只说明重要的配置 主要是监听IP开放,并追加了日志已方便调试,调试没问题可以将监听地址改为局域网
bash
# cat /etc/systemd/system/ollama.service
# 这是ollama的系统服务脚本
[Unit]
Description=Ollama Service
After=network-online.target
[Service]
ExecStart=/usr/local/bin/ollama serve
Environment="OLLAMA_HOST=0.0.0.0"
Environment="OLLAMA_ORIGINS=*"
Environment="STREAMING=true"
User=ollama
Group=ollama
Restart=always
RestartSec=3
# 标准输出和错误输出重定向到文件
StandardOutput=file:/var/log/ollama/ollama.log
StandardError=file:/var/log/ollama/ollama-error.log
# 或者使用追加模式
# StandardOutput=append:/var/log/ollama/ollama.log
# StandardError=append:/var/log/ollama/ollama-error.log
# 日志文件限制
LogRateLimitIntervalSec=30
LogRateLimitBurst=1000
Environment="PATH=/root/.nvm/versions/node/v22.20.0/bin:/root/anaconda3/bin:/root/anaconda3/condabin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/bin/gcc:/root/anaconda3/bin:/usr/local/matlab/bin"
[Install]
WantedBy=default.target
litellm 配置
这里只说明重要的配置
bash
# cat config.yaml
# 为什么这么配置,主要是claude code会调用自家的模型,这里都做了转换为qwen3-code
model_list:
- model_name: "qwen3-code"
litellm_params:
model: "ollama/qwen3-coder:latest"
api_base: "http://127.0.0.1:11434"
api_key: "none"
# 添加 Anthropic 兼容的模型别名
- model_name: "claude-3-5-sonnet-20241022"
litellm_params:
model: "ollama/qwen3-coder:latest"
api_base: "http://127.0.0.1:11434"
api_key: "none"
- model_name: "claude-3-5-haiku-20241022"
litellm_params:
model: "ollama/qwen3-coder:latest"
api_base: "http://127.0.0.1:11434"
api_key: "none"
- model_name: "claude-3-opus-20240229"
litellm_params:
model: "ollama/qwen3-coder:latest"
api_base: "http://127.0.0.1:11434"
api_key: "none"
- model_name: "claude-haiku-4-5-20251001"
litellm_params:
model: "ollama/qwen3-coder:latest"
api_base: "http://127.0.0.1:11434"
api_key: "none"
bash
# cat cat /etc/systemd/system/litellm.service
# 这是litellm的系统服务脚本
[Unit]
Description=LiteLLM Proxy Server
After=network.target
[Service]
Type=simple
User=pan
WorkingDirectory=/home/pan/LiteLLM
ExecStart=/home/pan/LiteLLM/start_litellm.sh
Restart=always
RestartSec=10
[Install]
WantedBy=multi-user.target
bash
# cat /home/pan/LiteLLM/start_litellm.sh
#!/bin/bash
# LiteLLM 启动脚本
# 这里监听本机IP并指定8000端口为服务端口
cd /home/pan/LiteLLM
source .venv/bin/activate
exec litellm --config config.yaml --port 8000 --host 192.168.84.251
好了,上面介绍了ollama+litellm的组合配置,这里ollama对外服务是开放的,只是为了测试ollama的服务是否正常,投入使用后,请将0.0.0.0改为127.0.0.1,只供litellm访问。
防火墙配置
bash
# 检查防火墙状态
sudo ufw status
# 如果启用,确保端口开放
sudo ufw allow 11434/tcp
sudo ufw allow 8000/tcp
测试服务是否正常(在ubuntu机器上进行测试)
ollama
bash
curl -v http://192.168.84.251:11434/api/tags
# 返回:模型列表,说明ollama服务正常
* Trying 192.168.84.251:11434...
* Connected to 192.168.84.251 (192.168.84.251) port 11434
> GET /api/tags HTTP/1.1
> Host: 192.168.84.251:11434
> User-Agent: curl/8.9.1
> Accept: */*
>
* Request completely sent off
< HTTP/1.1 200 OK
< Content-Type: application/json; charset=utf-8
< Date: Tue, 04 Nov 2025 10:02:07 GMT
< Transfer-Encoding: chunked
<
{"models":[{"name":"qwen3-coder:latest","model":"qwen3-coder:latest".....
litellm
bash
curl http://192.168.84.251:8000/v1/models
# 返回:模型列表,说明litellm服务正常
{"data":[{"id":"qwen3-code","object":"model","created":1677610602,"owned_by":"openai"}......
在windows机器上安装claude code
安装claude code 用PowerShell打开系统管理员终端:
powershell
# 需要node.js 18+
npm install -g @anthropic-ai/claude-code
验证安装,
powershell
mkdir claude_test ; cd claude_test
claude
提示连接anthropic.com官网失败:
powershell
Welcome to Claude Code v2.0.32
..............................................................................................................................................................................
* █████▓▓░
* ███▓░ ░░
░░░░░░ ███▓░
░░░ ░░░░░░░░░░ ███▓░
░░░░░░░░░░░░░░░░░░░ * ██▓░░ ▓
░▓▓███▓▓░
* ░░░░
░░░░░░░░
░░░░░░░░░░░░░░░░
█████████ *
██▄█████▄██ *
█████████ *
.....................█ █ █ █..............................................................................................................................
Unable to connect to Anthropic services
Failed to connect to api.anthropic.com: ERR_BAD_REQUEST
Please check your internet connection and network settings.
Note: Claude Code might not be available in your country. Check supported countries at
https://anthropic.com/supported-countries
PS C:\Users\MSI\claude_test>
给claude code添加本地环境:
说明:创建全局配置文件,这里指定模型为litellm的config.yaml配置的模型,前提ollama下载了该模型,同时指定了base_url的地址。
PowerShell
echo '` {
"env": {
"ANTHROPIC_MODEL": "qwen3-code",
"ANTHROPIC_SMALL_FAST_MODEL": "qwen3-code",
"ANTHROPIC_BASE_URL": "http://192.168.84.251:8000",
"ANTHROPIC_AUTH_TOKEN": "litellm"
}
}' > C:\Users\MSI\.claude\settings.json
再次运行claude
PowerShell
Welcome to Claude Code v2.0.32
..............................................................................................................................................................................
* █████▓▓░
* ███▓░ ░░
░░░░░░ ███▓░
░░░ ░░░░░░░░░░ ███▓░
░░░░░░░░░░░░░░░░░░░ * ██▓░░ ▓
░▓▓███▓▓░
* ░░░░
░░░░░░░░
░░░░░░░░░░░░░░░░
█████████ *
██▄█████▄██ *
█████████ *
.....................█ █ █ █..............................................................................................................................
Let's get started.
Choose the text style that looks best with your terminal
To change this later, run /theme
❯ 1. Dark mode ✔
2. Light mode
3. Dark mode (colorblind-friendly)
4. Light mode (colorblind-friendly)
5. Dark mode (ANSI colors only)
6. Light mode (ANSI colors only)
╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌
1 function greet() {
2 - console.log("Hello, World!");
2 + console.log("Hello, Claude!");
3 }
╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌
可以使用了。