本文整理自一次真实的部署与排错日志(2026-10-09 ~ 2026-10-10),命令与输出尽量保持原样。日志中过长的部分用
...标注了省略;文中标注"补充"的内容为整理时根据日志推断或补全,非原始日志。
总览
部署意图
把一个本地 LLM 推理服务拆成两个独立组件,并最终跑在 Kubernetes 上:
- FastAPI 服务(
llm-serving) :对外提供 OpenAI 风格的/v1/chat/completions和/health接口,内部转发请求给 Ollama。 - Ollama 服务(
llm-serving-ollama) :负责实际推理,模型qwen2:0.5b提前打进镜像,做到"离线可用"------容器启动后不需要再联网拉模型。
两个组件各自一个镜像、一个 Deployment、一个 Service,通过集群内 DNS 名互相访问。
目标架构
Ingress
│
▼
llm-serving Service :8000
│
▼
FastAPI Pod
│
│ http://llm-serving-ollama:11434
▼
llm-serving-ollama Service
│
▼
Ollama Pod
│
▼
qwen2:0.5b过程概览
| 阶段 | 内容 | 产物 |
|---|---|---|
| 一 | 构建 FastAPI 镜像 | llm-serving:0.2.0 |
| 二 | 通过国内镜像源拉取 Ollama,本地下载模型,打包成离线 Ollama 镜像 | llm-serving-ollama:0.1.0 |
| 三 | 确认两个镜像都已就绪 | docker images 输出 |
| 四 | 用 Docker 专用网络联调两个容器,验证 /health 与聊天接口 |
联调通过 |
| 五 | 在 Kubernetes(Helm 部署)中验收功能与资源,对照配置完善清单 | Deployment / Service / Ingress 运行正常 |
验证结果
- Ollama 离线镜像单独运行后,
/api/tags能直接列出qwen2:0.5b,证明模型已在镜像内。 - Docker 专用网络中,
/health返回{"status":"ok","service":"llm-serving","ollama":"ok"},说明 FastAPI 能通过容器名访问 Ollama。 /v1/chat/completions返回完整的chat.completion响应,Ollama 日志中可以看到/api/chat被 FastAPI 容器(172.20.0.3)调用。- Kubernetes 中两个 Deployment 均
1/1Ready,两个 PodRunning,Service 为 ClusterIP,Ingress 已分配地址,接口调用成功。
环境信息(来自日志)
- 宿主机:MacBook,Apple M4,Docker Desktop(
docker:desktop-linux) - 项目目录:
/Users/igwanglyang/Documents/GitHub/llm-serving - 宿主机 Ollama 版本
0.40.1(Metal GPU);镜像内 Ollama 版本0.40.2(纯 CPU,约 7.7 GiB 内存) - Kubernetes 命名空间:
llm-serving,通过 Helm 管理(日志中出现sh.helm.release.v1.llm-serving.v2 ~ v11)
一、构建 FastAPI 镜像
bash
(.venv) (base) igwanglyang@MacbookProdahouzicn llm-serving % docker build -t llm-serving:0.2.0 .
[+] Building 6.5s (10/10) FINISHED docker:desktop-linux
=> [internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 354B 0.0s
=> [internal] load metadata for docker.io/library/python:3.12-slim 6.4s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [1/5] FROM docker.io/library/python:3.12-slim@sha256:a6e34c598f2467ed0e9a8d349809fcd8b5c603269512df273a0bb1784edc11b1 0.0s
=> => resolve docker.io/library/python:3.12-slim@sha256:a6e34c598f2467ed0e9a8d349809fcd8b5c603269512df273a0bb1784edc11b1 0.0s
=> [internal] load build context 0.0s
=> => transferring context: 562B 0.0s
=> CACHED [2/5] WORKDIR /app 0.0s
=> CACHED [3/5] COPY requirements.txt . 0.0s
=> CACHED [4/5] RUN pip install --no-cache-dir -r requirements.txt 0.0s
=> CACHED [5/5] COPY app ./app 0.0s
=> exporting to image 0.0s
=> => exporting layers 0.0s
=> => writing image sha256:1f08145802dec9055418502ecf278f4bd9d054c490b6ba4bf468234fc4c7fdf4 0.0s
=> => naming to docker.io/library/llm-serving:0.2.0 0.0s
View build details: docker-desktop://dashboard/build/desktop-linux/desktop-linux/rfnhj7ohfrgdnp0ofh0yxcwnw要点:
- 基础镜像
python:3.12-slim,依赖安装与代码复制都命中了缓存(CACHED),所以构建只花了几秒。 - 把
COPY requirements.txt和pip install放在COPY app之前,是为了让改代码时不用重装依赖。
二、构建离线 Ollama 镜像
思路:先在宿主机准备好模型文件,再把模型目录 COPY 进基于官方 Ollama 的镜像。这样镜像自带模型,容器启动后无需联网。
2.1 下载 Ollama 基础镜像(使用 DaoCloud 镜像源)
先确认镜像源可达(返回 401 是正常的,说明 registry 在线,只是需要 token):
bash
(.venv) (base) igwanglyang@MacbookProdahouzicn llm-serving % curl -I --max-time 10 https://docker.m.daocloud.io/v2/
HTTP/2 401
server: nginx
date: Fri, 09 Oct 2026 17:05:15 GMT
content-type: application/json; charset=utf-8
content-length: 73
www-authenticate: Bearer realm="https://m.daocloud.io/auth/token",service="docker.m.daocloud.io"
docker-distribution-api-version: registry/2.0拉取镜像:
bash
(.venv) (base) igwanglyang@MacbookProdahouzicn llm-serving % docker pull docker.m.daocloud.io/ollama/ollama:latest
latest: Pulling from ollama/ollama
90812e242c93: Pull complete
2a5871fefe49: Pull complete
ecbacac73fe2: Pull complete
cf86a6b81b3e: Pull complete
Digest: sha256:b86366bb528bbf7f1424435d165028497a5b69bf6ddb4fa5a87102e2b79f44fb
Status: Downloaded newer image for docker.m.daocloud.io/ollama/ollama:latest
docker.m.daocloud.io/ollama/ollama:latest补充 :后面打包时 Dockerfile 用的是
FROM ollama/ollama:latest(见构建日志里的docker.io/ollama/ollama:latest)。如果本地没有这个名字的镜像,需要先给镜像源拉下来的镜像打个同名标签,例如:
bashdocker tag docker.m.daocloud.io/ollama/ollama:latest ollama/ollama:latest原始日志中没有记录这一步,请按你的实际情况处理。
2.2 下载离线模型 qwen2:0.5b
① 在宿主机上启动一个专用的 Ollama 服务
用独立端口 11435(避开可能已在运行的默认实例),并把模型目录指到项目内的 ollama/models,这样下载的模型文件会直接落在项目目录里,后面 COPY 就能用:
bash
(.venv) (base) igwanglyang@MacbookProdahouzicn llm-serving % cd /Users/igwanglyang/Documents/GitHub/llm-serving
OLLAMA_HOST=127.0.0.1:11435 \
OLLAMA_MODELS="$PWD/ollama/models" \
ollama serve
time=2026-10-10T01:30:18.626+08:00 level=INFO source=routes.go:2373 msg="server config" env="map[... OLLAMA_HOST:http://127.0.0.1:11435 ... OLLAMA_MODELS:/Users/igwanglyang/Documents/GitHub/llm-serving/ollama/models ...]"
time=2026-10-10T01:30:18.628+08:00 level=INFO source=images.go:989 msg="total blobs: 0"
time=2026-10-10T01:30:18.628+08:00 level=INFO source=images.go:998 msg="total unused blobs removed: 0"
time=2026-10-10T01:30:18.630+08:00 level=INFO source=routes.go:2435 msg="Listening on 127.0.0.1:11435 (version 0.40.1)"
time=2026-10-10T01:30:18.634+08:00 level=INFO source=runner.go:60 msg="discovering available GPUs..."
time=2026-10-10T01:30:19.255+08:00 level=INFO source=types.go:32 msg="inference compute" id=0 filter_id=0 library=Metal compute=0.0 name=MTL0 description="Apple M4" libdirs="" driver=0.0 pci_id="" type=iGPU total="17.8 GiB" available="17.8 GiB"
time=2026-10-10T01:30:19.255+08:00 level=INFO source=routes.go:2485 msg="vram-based default context" total_vram="17.8 GiB" default_num_ctx=4096
...② 另开一个终端,拉取模型
bash
(.venv) (base) igwanglyang@MacbookProdahouzicn llm-serving % cd /Users/igwanglyang/Documents/GitHub/llm-serving
OLLAMA_HOST=127.0.0.1:11435 ollama pull qwen2:0.5b
pulling manifest
pulling 8de95da68dc4: 100% ▕██████████████████████████████████████████████████████████████████████████████ ▏ 352 MB/352 MB 9.3 MB/s 0s
pulling 2184ab82477b: 100% ▕███████████████████████████████████████████████████████████████████████████████▏ 488 B
verifying sha256 digest
writing manifest
success 服务端对应的日志(多次重复拉取时,已下载的部分会很快完成):
text
[GIN] 2026/10/10 - 01:30:40 | 200 | 403.75µs | 127.0.0.1 | HEAD "/"
time=2026-10-10T01:30:44.771+08:00 level=INFO source=download.go:182 msg="downloading 8de95da68dc4 in 4 100 MB part(s)"
[GIN] 2026/10/10 - 01:30:45 | 200 | 4.989851666s | 127.0.0.1 | POST "/api/pull"
...
[GIN] 2026/10/10 - 01:32:46 | 200 | 14.932196208s | 127.0.0.1 | POST "/api/pull"③ 确认模型和文件
bash
(.venv) (base) igwanglyang@MacbookProdahouzicn llm-serving % OLLAMA_HOST=127.0.0.1:11435 ollama list
du -sh ollama/models
find ollama/models -maxdepth 3 -type f
NAME ID SIZE MODIFIED
qwen2:0.5b 6f48b936a09f 352 MB 3 seconds ago
336M ollama/models
ollama/models/blobs/sha256-6f48b936a09f7743c7dd30e72fdb14cba296bc5861902e4d0c387e8fb5050b39
...
ollama/models/blobs/sha256-2184ab82477bc33a5e08fa209df88f0631a19e686320cce2cfe9e00695b2f0e6
ollama/models/metadata/sha256-8de95da68dc485c0889c205384c24642f83ca18d089559c977ffc6a3972a71a8.json中间省略的是 4 个 blob 文件(
sha256-62fbfd9e...、sha256-f02dd72b...、sha256-c156170b...、sha256-8de95da6...),其中8de95da6...是约 352 MB 的模型权重本体。ollama/models总计 336M。
2.3 打包为离线镜像
bash
(.venv) (base) igwanglyang@MacbookProdahouzicn llm-serving % cd /Users/igwanglyang/Documents/GitHub/llm-serving
docker build \
-f ollama/Dockerfile \
-t llm-serving-ollama:0.1.0 \
ollama
[+] Building 5.7s (7/7) FINISHED docker:desktop-linux
=> [internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 174B 0.0s
=> [internal] load metadata for docker.io/ollama/ollama:latest 0.0s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [internal] load build context 4.1s
=> => transferring context: 352.25MB 4.1s
=> [1/2] FROM docker.io/ollama/ollama:latest 0.1s
=> [2/2] COPY models/ /models/ 0.7s
=> exporting to image 0.8s
=> => exporting layers 0.8s
=> => writing image sha256:8c5617c32f48c6efe1d184a51dbb99c6f8b421303bc4462967907c1a29deec8b 0.0s
=> => naming to docker.io/library/llm-serving-ollama:0.1.0 0.0s
View build details: docker-desktop://dashboard/build/desktop-linux/desktop-linux/jg8r3bqh96t1c7fpzccbbk0j8补充 :日志里没有贴出
ollama/Dockerfile的内容。从构建步骤(两步:FROM+COPY models/ /models/,174B)以及后面容器日志里的OLLAMA_MODELS:/models推断,它大致是下面这样,仅供参考:
dockerfileFROM ollama/ollama:latest ENV OLLAMA_MODELS=/models COPY models/ /models/构建上下文传输了
352.25MB,正好对应模型文件体积。另外,第一次运行这个构建命令时,输出停在了Building 5.3s (1/2)(日志被截断,看起来是构建被打断或终端输出未结束),随后重新执行才得到上面的完整成功结果。
2.4 单独测试并删除测试容器
bash
(.venv) (base) igwanglyang@MacbookProdahouzicn llm-serving % docker run -d \
--name ollama-offline-test \
-p 11436:11434 \
llm-serving-ollama:0.1.0
02a791767c15ee281f5e590c5156deb1d91ed9d408616d6a042eb2abf74d4957
(.venv) (base) igwanglyang@MacbookProdahouzicn llm-serving % curl http://localhost:11436/api/tags
{"models":[{"name":"qwen2:0.5b","model":"qwen2:0.5b","modified_at":"2026-10-09T17:36:31.470145772Z","size":352164041,"digest":"6f48b936a09f7743c7dd30e72fdb14cba296bc5861902e4d0c387e8fb5050b39","details":{"parent_model":"","format":"gguf","family":"qwen2","families":["qwen2"],"parameter_size":"494.03M","quantization_level":"Q4_0","context_length":32768,"embedding_length":896,"runner":"ggml"},"capabilities":["completion"]}%
(.venv) (base) igwanglyang@MacbookProdahouzicn llm-serving % docker rm -f ollama-offline-test
ollama-offline-test/api/tags 直接列出了 qwen2:0.5b(量化 Q4_0、参数量 494.03M),说明模型确实已经在镜像里,没有联网下载。
三、确认两个镜像
再次构建 FastAPI 镜像(依然全部命中缓存),并列出镜像:
bash
(.venv) (base) igwanglyang@MacbookProdahouzicn llm-serving % docker build -t llm-serving:0.2.0 .
[+] Building 4.4s (10/10) FINISHED docker:desktop-linux
...
=> CACHED [5/5] COPY app ./app 0.0s
=> exporting to image 0.0s
=> => exporting layers 0.0s
=> => writing image sha256:7b8f009cae8cf775f72d9e74237e1a627ec22d1b428ff066ebafa2e1c74a1b10 0.0s
=> => naming to docker.io/library/llm-serving:0.2.0 0.0s
bash
(.venv) (base) igwanglyang@MacbookProdahouzicn llm-serving % docker images | grep llm-serving
llm-serving-ollama 0.1.0 8c5617c32f48 17 minutes ago 4.58GB
llm-serving 0.2.0 7b8f009cae8c 26 hours ago 200MB
llm-serving 0.1.1 6f5b68138c97 26 hours ago 200MB
llm-serving 0.1.0 ...原始输出的列间距很宽,这里压缩了空白。注意两点:
- FastAPI 镜像约 200MB ,Ollama 离线镜像约 4.58GB(官方 Ollama 基础镜像本身就很大,再加上 352MB 模型)。
- 第二次构建后
llm-serving:0.2.0的镜像 ID 从1f08145802de...变成了7b8f009cae8c...,但内容层全部是CACHED,通常是镜像配置元数据(如构建时间等)变化所致。
四、用 Docker 专用网络联调
在上 Kubernetes 之前,先用 Docker 网络模拟"两个 Service 之间用名字互相访问"的场景。
4.1 创建专用网络
bash
(.venv) (base) igwanglyang@MacbookProdahouzicn llm-serving % docker network create llm-serving-net
38e47cbd349775901f8c241156685be5645f3a9d79902a342d6f6f66c504cb07自定义 bridge 网络自带 DNS,容器之间可以直接用容器名互相访问。
4.2 启动 Ollama 容器
容器名取 llm-serving-ollama,与后面 Kubernetes 中的 Service 名保持一致:
bash
(.venv) (base) igwanglyang@MacbookProdahouzicn llm-serving % docker run -d \
--name llm-serving-ollama \
--network llm-serving-net \
-p 11436:11434 \
llm-serving-ollama:0.1.0
900a3b083555760786fa68159f40dd8e8bdc7c5d2c223bed5d5b0345435bf48b4.3 启动 FastAPI 容器(含一次命名冲突)
通过环境变量告诉 FastAPI:Ollama 在哪里、默认用哪个模型。
bash
(.venv) (base) igwanglyang@MacbookProdahouzicn llm-serving % docker run -d \
--name llm-serving \
--network llm-serving-net \
-p 8000:8000 \
-e OLLAMA_BASE_URL=http://llm-serving-ollama:11434 \
-e DEFAULT_MODEL=qwen2:0.5b \
llm-serving:0.2.0
1c073af4bc1132b8637bd5da86eede1d0666fcfbdd4713034bcdc87613fb0983随后同样的命令又执行了两次,报了容器名冲突(此前的容器已经占用了 llm-serving 这个名字):
bash
(.venv) (base) igwanglyang@MacbookProdahouzicn llm-serving % docker run -d \
--name llm-serving \
--network llm-serving-net \
-p 8000:8000 \
-e OLLAMA_BASE_URL=http://llm-serving-ollama:11434 \
-e DEFAULT_MODEL=qwen2:0.5b \
llm-serving:0.2.0
docker: Error response from daemon: Conflict. The container name "/llm-serving" is already in use by container "1c073af4bc1132b8637bd5da86eede1d0666fcfbdd4713034bcdc87613fb0983". You have to remove (or rename) that container to be able to reuse that name.
Run 'docker run --help' for more information尝试用 docker remove 删除,但这不是合法命令:
bash
(.venv) (base) igwanglyang@MacbookProdahouzicn llm-serving % docker remove llm-serving
docker: unknown command: docker remove
Run 'docker --help' for more information正确做法是 docker rm -f,删除后重新启动:
bash
(.venv) (base) igwanglyang@MacbookProdahouzicn llm-serving % docker rm -f llm-serving
llm-serving
(.venv) (base) igwanglyang@MacbookProdahouzicn llm-serving % docker run -d \
--name llm-serving \
--network llm-serving-net \
-p 8000:8000 \
-e OLLAMA_BASE_URL=http://llm-serving-ollama:11434 \
-e DEFAULT_MODEL=qwen2:0.5b \
llm-serving:0.2.0
ebbca645f028454391e8de65486fc58e9a0c60f22612bc99b11f9ff0c9ea0922小结 :Docker 容器名全局唯一,重复
docker run --name xxx会冲突;删除容器用docker rm(强制删除运行中的用-f),没有docker remove这个子命令。
4.4 健康检查与聊天接口验证
健康检查 :ollama: ok 表示 FastAPI 已经能通过 http://llm-serving-ollama:11434 访问到 Ollama。
bash
(.venv) (base) igwanglyang@MacbookProdahouzicn llm-serving % curl http://localhost:8000/health
{"status":"ok","service":"llm-serving","ollama":"ok"}%聊天补全:
bash
(.venv) (base) igwanglyang@MacbookProdahouzicn llm-serving % curl http://localhost:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "qwen2:0.5b",
"messages": [
{
"role": "user",
"content": "100 word Explain Kubernetes Service."
}
]
}'
{"id":"chatcmpl-0f46884e531b","object":"chat.completion","created":1791568826,"model":"qwen2:0.5b","choices":[{"index":0,"message":{"role":"assistant","content":"Kubernetes Service, as its name implies, allows applications to run in containers, creating a highly flexible platform that can scale horizontally to handle increased load or expand in response to changing requirements. It is the most widely adopted approach for managing containerized applications, allowing developers to easily add new components or modify existing ones without having to manually modify the codebase. Kubernetes Service allows developers to easily deploy, scale, and manage applications, providing a powerful tool for managing complex applications with dynamic behavior and scalability. By leveraging Kubernetes Service, organizations can enhance performance, scalability, and flexibility of their applications, enabling them to handle increasing traffic load and adapt to changing requirements."},"finish_reason":"stop"}],"usage":{"prompt_tokens":16,"completion_tokens":131,"total_tokens":147}}%0.5B 的小模型回答流畅但内容不够准确(比如把 Service 说成"运行容器的平台"),这里的目的是验证链路通畅,而不是模型质量。
4.5 查看两个容器的日志
bash
(.venv) (base) igwanglyang@MacbookProdahouzicn llm-serving % docker logs llm-serving
docker logs llm-serving-ollamaFastAPI 日志:
text
INFO: Started server process [1]
INFO: Waiting for application startup.
INFO: Application startup complete.
INFO: Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)
INFO: 192.168.65.1:62195 - "GET /health HTTP/1.1" 200 OK
INFO: 192.168.65.1:44825 - "POST /v1/chat/completions HTTP/1.1" 200 OKOllama 日志 (节选,完整日志中还有大量 llama_model_loader / print_info 模型参数输出,已省略):
text
Couldn't find '/root/.ollama/id_ed25519'. Generating new private key.
Your new public key is:
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGhECwQbywKPHXwbKlBE77t3+naUbZKTFFllZar2Zpny
time=2026-10-09T17:57:52.849Z level=INFO source=routes.go:2373 msg="server config" env="map[... OLLAMA_HOST:http://0.0.0.0:11434 ... OLLAMA_MODELS:/models ... OLLAMA_NUM_PARALLEL:1 ...]"
time=2026-10-09T17:57:52.849Z level=INFO source=routes.go:2375 msg="Ollama cloud disabled: false"
time=2026-10-09T17:57:52.853Z level=INFO source=images.go:989 msg="total blobs: 0"
time=2026-10-09T17:57:52.854Z level=INFO source=images.go:998 msg="total unused blobs removed: 0"
time=2026-10-09T17:57:52.855Z level=INFO source=routes.go:2435 msg="Listening on [::]:11434 (version 0.40.2)"
time=2026-10-09T17:57:52.861Z level=INFO source=runner.go:60 msg="discovering available GPUs..."
time=2026-10-09T17:57:52.978Z level=INFO source=types.go:50 msg="inference compute" id=cpu library=cpu compute="" name=cpu description=cpu libdirs=ollama driver="" pci_id="" type="" total="7.7 GiB" available="7.6 GiB"
time=2026-10-09T17:57:52.978Z level=INFO source=routes.go:2485 msg="vram-based default context" total_vram="0 B" default_num_ctx=4096
time=2026-10-09T17:57:54.316Z level=INFO source=model_recommendations.go:177 msg="model recommendations cache sleep scheduled" wait=4h31m18.884575993s consecutive_failures=0
time=2026-10-09T17:57:59.458Z level=WARN source=model_show_cache.go:327 msg="failed to hydrate cloud model show cache" model=nemotron-3-ultra error="Post \"https://ollama.com:443/api/show?ts=1791568676\": context deadline exceeded"
time=2026-10-09T17:57:59.721Z level=WARN source=model_show_cache.go:327 msg="failed to hydrate cloud model show cache" model=kimi-k3 error="Post \"https://ollama.com:443/api/show?ts=1791568676\": context deadline exceeded"
time=2026-10-09T17:57:59.894Z level=WARN source=model_show_cache.go:327 msg="failed to hydrate cloud model show cache" model=minimax-m3 error="Post \"https://ollama.com:443/api/show?ts=1791568676\": context deadline exceeded"
[GIN] 2026/10/09 - 18:00:05 | 200 | 6.494209ms | 172.20.0.3 | GET "/api/tags"
time=2026-10-09T18:00:24.197Z level=INFO source=sched.go:1166 msg="disabling mmap for llama-server load by default" model=/models/blobs/sha256-8de95da68dc485c0889c205384c24642f83ca18d089559c977ffc6a3972a71a8 reason=cpu
time=2026-10-09T18:00:24.197Z level=INFO source=server.go:107 msg="using llama-server for model" model=/models/blobs/sha256-8de95da68dc485c0889c205384c24642f83ca18d089559c977ffc6a3972a71a8
...
llama_model_loader: - kv 0: general.architecture str = qwen2
llama_model_loader: - kv 1: general.name str = Qwen2-0.5B-Instruct
llama_model_loader: - kv 2: qwen2.block_count u32 = 24
llama_model_loader: - kv 3: qwen2.context_length u32 = 32768
...
print_info: file format = GGUF V3 (latest)
print_info: file type = Q4_0
print_info: file size = 330.17 MiB (5.61 BPW)
...
print_info: model params = 494.03 M
print_info: general.name = Qwen2-0.5B-Instruct
...
llama_context: n_ctx = 4096
...
srv llama_server: model loaded
srv llama_server: listening on http://127.0.0.1:40247
srv update_slots: all slots are idle
time=2026-10-09T18:00:25.231Z level=INFO source=llama_server.go:1374 msg="llama-server started in 1.03 seconds"
...
slot print_timing: id 0 | task 0 | prompt eval time = 34.70 ms / 16 tokens ( 2.17 ms per token, 461.13 tokens per second)
slot print_timing: id 0 | task 0 | eval time = 1449.23 ms / 131 tokens ( 11.15 ms per token, 89.70 tokens per second)
slot print_timing: id 0 | task 0 | total time = 1483.93 ms / 147 tokens
slot print_timing: id 0 | task 0 | graphs reused = 130
slot release: id 0 | task 0 | stop processing: n_tokens = 146, truncated = 0
srv update_slots: all slots are idle
[GIN] 2026/10/09 - 18:00:26 | 200 | 2.574404376s | 172.20.0.3 | POST "/api/chat"日志里值得注意的几点:
- 调用来源 :
GET /api/tags(健康检查)和POST /api/chat(聊天)都来自172.20.0.3,即 FastAPI 容器在专用网络中的地址,证明容器间是通过网络名互通的。 - 模型从镜像内加载 :模型路径是
/models/blobs/sha256-8de95da6...,与镜像里OLLAMA_MODELS=/models对应。 - 纯 CPU 推理 :Docker Desktop 的 Linux 容器里用不到 Apple 的 Metal GPU,日志显示
library=cpu。即便如此,0.5B 小模型也能跑到约 89.7 tokens/s ,整个请求2.57s(含首次加载模型1.03s)。 - 一些无害的告警 :
failed to hydrate cloud model show cache ... context deadline exceeded是 Ollama 尝试访问ollama.com的云模型信息超时,不影响本地推理;Couldn't find '/root/.ollama/id_ed25519'. Generating new private key.只是容器首次启动自动生成身份密钥,也属正常。
4.6 清理现场
bash
(.venv) (base) igwanglyang@MacbookProdahouzicn llm-serving % docker rm -f llm-serving llm-serving-ollama
docker network rm llm-serving-net
llm-serving
llm-serving-ollama
llm-serving-net五、Kubernetes 阶段:功能验收与配置完善
Docker 联调通过后,进入阶段 2:在集群里验收功能,并对照"生产化配置清单"检查还缺什么。
5.1 接口功能验证
通过 Ingress / 本地端口访问 /v1/chat/completions,显式带上 "stream": false:
bash
(.venv) (base) igwanglyang@MacbookProdahouzicn llm-serving % curl -sS http://localhost:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "qwen2:0.5b",
"messages": [
{
"role": "user",
"content": "Explain Kubernetes Service in 3 sentences."
}
],
"stream": false
}'
{"id":"chatcmpl-bfc6a4630554","object":"chat.completion","created":1791576033,"model":"qwen2:0.5b","choices":[{"index":0,"message":{"role":"assistant","content":"Kubernetes Service enables deployment of a pod as an application service across all nodes in Kubernetes cluster. It allows applications to be deployed across multiple nodes in a Kubernetes cluster."},"finish_reason":"stop"}],"usage":{"prompt_tokens":17,"completion_tokens":34,"total_tokens":51}}%5.2 验收 Kubernetes 资源
bash
(.venv) (base) igwanglyang@MacbookProdahouzicn llm-serving % kubectl get deploy,pods,svc,ingress -n llm-serving
kubectl get pvc,sa,secret -n llm-serving
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/llm-serving 1/1 1 1 27h
deployment.apps/llm-serving-ollama 1/1 1 1 103m
NAME READY STATUS RESTARTS AGE
pod/llm-serving-64856b46ff-qztkx 1/1 Running 0 8m30s
pod/llm-serving-ollama-df84fbbfc-xqdjn 1/1 Running 0 103m
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/llm-serving ClusterIP 10.104.107.206 <none> 8000/TCP 27h
service/llm-serving-ollama ClusterIP 10.104.228.49 <none> 11434/TCP 103m
NAME CLASS HOSTS ADDRESS PORTS AGE
ingress.networking.k8s.io/llm-serving nginx llm-serving.local 10.104.86.242 80 27h
NAME SECRETS AGE
serviceaccount/default 0 27h
serviceaccount/llm-serving 0 95m
NAME TYPE DATA AGE
secret/sh.helm.release.v1.llm-serving.v10 helm.sh/release.v1 1 13m
secret/sh.helm.release.v1.llm-serving.v11 helm.sh/release.v1 1 8m30s
secret/sh.helm.release.v1.llm-serving.v2 helm.sh/release.v1 1 27h
secret/sh.helm.release.v1.llm-serving.v3 helm.sh/release.v1 1 27h
secret/sh.helm.release.v1.llm-serving.v4 helm.sh/release.v1 1 27h
secret/sh.helm.release.v1.llm-serving.v5 helm.sh/release.v1 1 27h
secret/sh.helm.release.v1.llm-serving.v6 helm.sh/release.v1 1 103m
secret/sh.helm.release.v1.llm-serving.v7 helm.sh/release.v1 1 95m
secret/sh.helm.release.v1.llm-serving.v8 helm.sh/release.v1 1 90m
secret/sh.helm.release.v1.llm-serving.v9 helm.sh/release.v1 1 24m从输出可以读出:
- 两个 Deployment 都是
1/1:llm-serving(FastAPI,已运行 27h)和llm-serving-ollama(Ollama,103 分钟前新加入)。 - Service 均为 ClusterIP :
llm-serving:8000和llm-serving-ollama:11434,与架构图一致,只有 FastAPI 通过 Ingress 对外暴露(nginx类,Host 为llm-serving.local)。 - ServiceAccount :除
default外多了一个专用的llm-serving(95 分钟前创建),对应清单中"明确 Pod 使用的身份"。 - Helm 发布历史 :
sh.helm.release.v1.llm-serving.v2 ~ v11是 Helm 每次upgrade留下的 release 记录,当前已到第 11 个版本,说明这套配置是通过 Helm 反复迭代出来的。 - PVC :
kubectl get pvc在输出里没有对应的资源行,说明当前还没有创建 PVC。这是合理的------模型已经打进镜像,不依赖持久卷,所以清单里写的是"按实际数据持久化需求配置"。
5.3 配置完善清单
下表是阶段 2 要对照检查的资源。"对应文件"一列在原始笔记中为空,以下为补充 ,按常见 Helm Chart 目录结构填写,请按你项目的实际文件名调整;"当前状态"一列依据上面的 kubectl 输出判断。
| 资源 | 目的 | 对应文件(补充,示例) | 当前状态 |
|---|---|---|---|
| Namespace | 隔离应用资源 | templates/namespace.yaml(或 helm --create-namespace) |
已存在 llm-serving |
| ServiceAccount | 明确 Pod 使用的身份 | templates/serviceaccount.yaml |
已创建 llm-serving |
| ConfigMap / Secret | 分离普通配置和敏感配置 | templates/configmap.yaml、templates/secret.yaml |
输出中仅见 Helm release Secret,应用级配置待确认 |
| PVC | 按实际数据持久化需求配置 | templates/pvc.yaml |
未创建(模型已内置镜像) |
| Deployment / Service | 分别管理 FastAPI 和 Ollama | templates/deployment.yaml、templates/service.yaml、templates/ollama-deployment.yaml、templates/ollama-service.yaml |
已就绪 |
| Ingress | 对外暴露 FastAPI API | templates/ingress.yaml |
已就绪(llm-serving.local) |
| requests / limits | 控制 CPU、内存使用 | 写在 Deployment 的 resources 字段 |
待配置/待确认 |
| HPA | 按指标扩缩容;需具备可用指标源 | templates/hpa.yaml |
待配置(需先有 metrics-server 等指标源) |
| NetworkPolicy | 限制不必要的 Pod 网络访问 | templates/networkpolicy.yaml |
待配置 |
补充建议:
- requests / limits:Ollama 即使是 0.5B 模型,日志显示容器内占用也就几百 MiB(模型 + KV cache + 计算缓冲约 225 MiB),可以据此给 Ollama 设置内存 requests/limits,并为 FastAPI 设置较小的值。
- HPA :Ollama 的并行度默认是
OLLAMA_NUM_PARALLEL:1,扩容 FastAPI 副本前要先想清楚瓶颈其实在 Ollama,盲目扩 FastAPI 意义不大。- NetworkPolicy:合理的最小策略是------只允许 Ingress Controller 访问 FastAPI:8000,只允许 FastAPI 访问 Ollama:11434。
总结
这次部署的关键点:
- 离线镜像思路 :在宿主机用
OLLAMA_MODELS指向项目目录下载模型,再COPY进基于官方 Ollama 的镜像,一次打包,处处离线可跑。 - 先用 Docker 专用网络联调 :容器名与未来 Kubernetes 中的 Service 名保持一致(
llm-serving-ollama),同一套OLLAMA_BASE_URL=http://llm-serving-ollama:11434配置在 Docker 和 K8s 里都能用。 - 验证看三处 :
/api/tags证明模型在镜像里;/health里的ollama: ok证明服务间通路;Ollama 日志里的来源 IP 与/api/chat证明调用链真实走通。 - 踩坑记录 :
docker run --name重名冲突需docker rm -f(没有docker remove);重复下载模型时 Ollama 会复用已有 blob。 - 下一步:补齐 requests/limits、HPA 与 NetworkPolicy,并确认 ConfigMap / Secret 的拆分,让清单中"待配置"的项目全部落地。