基于TencentOS的PVMCubeSandBox模板构建失败

cubesandbox创建模版报错

模板构建失败。

该模板构建失败,无法查看详情,但你可以将其删除

CubeSandbox 模板构建失败(FAILED),页面只显示:

复制代码
模板构建失败。

该模板构建失败,无法查看详情
Job ID:
ceaa39fc-aa7b-43d4-974a-1ef430a074f7

这里需要从 CubeSandbox 后端 Job 日志定位,前端不会展示具体原因

结合你的环境(v0.7.1 + PVM + MinIO + Harbor 镜像),优先排查以下几个方向


1. 根据 Job ID 查构建日志(第一步)

你的 Job ID:

复制代码
ceaa39fc-aa7b-43d4-974a-1ef430a074f7

在 CubeSandbox 节点执行:

复制代码
grep -R "ceaa39fc-aa7b-43d4-974a-1ef430a074f7" /var/log 2>/dev/null

或者:

复制代码
journalctl -xe | grep ceaa39fc

如果 CubeSandbox 是容器:

复制代码
docker ps | grep cube

然后:

复制代码
docker logs <cube-container> | grep ceaa39fc

2. 查看 CubeSandbox 服务日志

先找服务:

复制代码
systemctl list-units | grep cube

常见:

复制代码
cube-sandbox.service
cube-server.service
cube-agent.service

查看:

复制代码
journalctl -u cube-sandbox -n 200 --no-pager

重点搜索:

复制代码
journalctl -u cube-sandbox -n 500 | egrep "ERROR|failed|ceaa39fc"

3. 你的截图里有一个重要信息

模板来源:

复制代码
镜像:
harbor-intranet.allintechinc.com/llm-planning/llm-planning-sandb...

说明模板创建流程大概率:

复制代码
Harbor 镜像
      |
      v
CubeSandbox Builder
      |
      v
生成 CFS/MinIO 快照
      |
      v
Template

所以重点检查 Harbor 拉取


4. 检查 Harbor 镜像访问

在 CubeSandbox 主机:

复制代码
docker login harbor-intranet.allintechinc.com

测试:

复制代码
docker pull harbor-intranet.allintechinc.com/llm-planning/llm-planning-sandbox:xxx

如果失败:

常见:

认证失败

复制代码
unauthorized: authentication required

需要配置 Harbor 凭证

TLS问题

例如:

复制代码
x509: certificate signed by unknown authority

需要导入 CA:

复制代码
/etc/docker/certs.d/harbor-intranet.allintechinc.com/ca.crt

5. 检查 MinIO/CFS

你安装时:

复制代码
CUBE_S3_ENDPOINT=http://192.168.0.6:9000
bucket=cube-volumes

模板生成需要写入:

复制代码
mc ls cube/cube-volumes

检查空间:

复制代码
df -h

重点:

复制代码
/var
/tmp
/mnt

模板构建可能需要几十 GB 临时空间


6. 检查 PVM/KVM

你的安装:

复制代码
PVM host detected
kvm_pvm loaded

检查:

复制代码
lsmod | grep kvm

应该:

复制代码
kvm_pvm
kvm

检查:

复制代码
/dev/kvm

权限:

复制代码
ls -l /dev/kvm

7. 删除失败模板

页面删除失败模板即可

如果 UI 删除失败:

查模板:

复制代码
kubectl get template -A

或者:

复制代码
docker exec -it <cube-container> sh

查看数据库:

复制代码
grep -R "tpl-464f1b5538764bd7afa240ca" /

8. 建议你现在执行这几个命令

按顺序:

① 找 Job

复制代码
grep -R "ceaa39fc-aa7b-43d4-974a-1ef430a074f7" /var/log 2>/dev/null

② 看服务

复制代码
systemctl list-units | grep cube

③ 看最近错误

复制代码
journalctl -xe -n 200 | grep -i cube

④ 看磁盘

复制代码
df -h

root@VM-0-6-tencentos \~# systemctl list-units | grep -i cube sys-devices-virtual-net-cube\x2ddev.device loaded active plugged /sys/devices/virtual/net/cube-dev sys-devices-virtual-net-cube\x2ddns0.device loaded active plugged /sys/devices/virtual/net/cube-dns0 sys-subsystem-net-devices-cube\x2ddev.device loaded active plugged /sys/subsystem/net/devices/cube-dev sys-subsystem-net-devices-cube\x2ddns0.device loaded active plugged /sys/subsystem/net/devices/cube-dns0 usr-local-services-cubetoolbox-cubeletmnt-mnt.mount loaded active mounted /usr/local/services/cubetoolbox/cubeletmnt/mnt usr-local-services-cubetoolbox-cubeletmnt.mount loaded active mounted /usr/local/services/cubetoolbox/cubeletmnt cube-sandbox-coredns.service loaded active running Cube Sandbox CoreDNS container cube-sandbox-cube-api.service loaded active running Cube Sandbox Cube API cube-sandbox-cube-egress-net.service loaded active exited Cube Sandbox cube-egress host network rules (TPROXY + ip rule) cube-sandbox-cube-egress.service loaded active running Cube Sandbox cube-egress (transparent MITM proxy) cube-sandbox-cube-lifecycle-manager.service loaded active running Cube Sandbox cube-lifecycle-manager container cube-sandbox-cube-proxy.service loaded active running Cube Sandbox cube-proxy container cube-sandbox-cube-templatecenter.service loaded active running Cube Sandbox CubeTemplateCenter cube-sandbox-cubelet.service loaded active running Cube Sandbox Cubelet cube-sandbox-cubemaster.service loaded active running Cube Sandbox CubeMaster cube-sandbox-cubeops.service loaded active running Cube Sandbox CubeOps (admin/ops API) cube-sandbox-dns.service loaded active exited Cube Sandbox host DNS routing cube-sandbox-minio.service loaded active running Cube Sandbox MinIO container cube-sandbox-mysql.service loaded active running Cube Sandbox MySQL container cube-sandbox-redis.service loaded active running Cube Sandbox Redis container cube-sandbox-webui.service loaded active running Cube Sandbox Web UI container cube-sandbox-control.target loaded active active Cube Sandbox control-plane stack root@VM-0-6-tencentos \~# journalctl -xe -n 300 | grep -Ei "cube|template|job|error|failed" 9月 20 17:47:44 VM-0-6-tencentos bash5538: 0.450ms rows:0 SELECT * FROM t_cube_template_definition WHERE template_id = 'tpl-464f1b5538764bd7afa240ca' AND t_cube_template_definition.deleted_at IS NULL ORDER BY t_cube_template_definition.id LIMIT 1 9月 20 17:47:44 VM-0-6-tencentos d5c0c2c48c921915: 1.119.147.226 - - 20/Sep/2026:09:47:44 +0000 "GET /templates/tpl-464f1b5538764bd7afa240ca HTTP/1.1" 404 61 "http://152.136.174.29:12088/templates/tpl-464f1b5538764bd7afa240ca" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/152.0.0.0 Safari/537.36" 9月 20 17:47:44 VM-0-6-tencentos bash6300: cube-webui | 1.119.147.226 - - 20/Sep/2026:09:47:44 +0000 "GET /templates/tpl-464f1b5538764bd7afa240ca HTTP/1.1" 404 61 "http://152.136.174.29:12088/templates/tpl-464f1b5538764bd7afa240ca" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/152.0.0.0 Safari/537.36" 9月 20 17:47:44 VM-0-6-tencentos d5c0c2c48c921915: 1.119.147.226 - - 20/Sep/2026:09:47:44 +0000 "GET /templates/compat HTTP/1.1" 200 112 "http://152.136.174.29:12088/templates/tpl-464f1b5538764bd7afa240ca" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/152.0.0.0 Safari/537.36" 9月 20 17:47:44 VM-0-6-tencentos bash6300: cube-webui | 1.119.147.226 - - 20/Sep/2026:09:47:44 +0000 "GET /templates/compat HTTP/1.1" 200 112 "http://152.136.174.29:12088/templates/tpl-464f1b5538764bd7afa240ca" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/152.0.0.0 Safari/537.36" 9月 20 17:47:45 VM-0-6-tencentos bash6300: cube-webui | 127.0.0.1 - - 20/Sep/2026:09:47:45 +0000 "GET / HTTP/1.1" 200 1120 "-" "Wget/1.21.3" 9月 20 17:47:45 VM-0-6-tencentos bash5538: 2026/09/20 17:47:45 /workspace/CubeMaster/pkg/templatecenter/snapshot_table.go:29 record not found 9月 20 17:47:45 VM-0-6-tencentos bash5538: 0.683ms rows:0 SELECT * FROM t_cube_snapshot WHERE snapshot_id = 'tpl-464f1b5538764bd7afa240ca' AND t_cube_snapshot.deleted_at IS NULL ORDER BY t_cube_snapshot.id LIMIT 1 9月 20 17:47:45 VM-0-6-tencentos bash5538: 2026/09/20 17:47:45 /workspace/CubeMaster/pkg/templatecenter/store.go:1118 record not found 9月 20 17:47:45 VM-0-6-tencentos bash5538: 0.397ms rows:0 SELECT * FROM t_cube_template_definition WHERE template_id = 'tpl-464f1b5538764bd7afa240ca' AND t_cube_template_definition.deleted_at IS NULL ORDER BY t_cube_template_definition.id LIMIT 1 9月 20 17:47:45 VM-0-6-tencentos d5c0c2c48c921915: 1.119.147.226 - - 20/Sep/2026:09:47:45 +0000 "GET /templates/tpl-464f1b5538764bd7afa240ca HTTP/1.1" 404 61 "http://152.136.174.29:12088/templates/tpl-464f1b5538764bd7afa240ca" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/152.0.0.0 Safari/537.36" 9月 20 17:47:45 VM-0-6-tencentos bash6300: cube-webui | 1.119.147.226 - - 20/Sep/2026:09:47:45 +0000 "GET /templates/tpl-464f1b5538764bd7afa240ca HTTP/1.1" 404 61 "http://152.136.174.29:12088/templates/tpl-464f1b5538764bd7afa240ca" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/152.0.0.0 Safari/537.36" 9月 20 17:47:55 VM-0-6-tencentos bash6300: cube-webui | 127.0.0.1 - - 20/Sep/2026:09:47:55 +0000 "GET / HTTP/1.1" 200 1120 "-" "Wget/1.21.3" 9月 20 17:48:05 VM-0-6-tencentos bash6300: cube-webui | 127.0.0.1 - - 20/Sep/2026:09:48:05 +0000 "GET / HTTP/1.1" 200 1120 "-" "Wget/1.21.3" 9月 20 17:48:12 VM-0-6-tencentos bash5538: 2026/09/20 17:48:12 /workspace/CubeMaster/pkg/templatecenter/store.go:1118 record not found 9月 20 17:48:12 VM-0-6-tencentos bash5538: 0.411ms rows:0 SELECT * FROM t_cube_template_definition WHERE template_id = 'tpl-464f1b5538764bd7afa240ca' AND t_cube_template_definition.deleted_at IS NULL ORDER BY t_cube_template_definition.id LIMIT 1 9月 20 17:48:12 VM-0-6-tencentos bash5538: 2026/09/20 17:48:12 /workspace/CubeMaster/pkg/templatecenter/snapshot_table.go:29 record not found 9月 20 17:48:12 VM-0-6-tencentos bash5538: 0.387ms rows:0 SELECT * FROM t_cube_snapshot WHERE snapshot_id = 'tpl-464f1b5538764bd7afa240ca' AND t_cube_snapshot.deleted_at IS NULL ORDER BY t_cube_snapshot.id LIMIT 1 9月 20 17:48:15 VM-0-6-tencentos bash6300: cube-webui | 127.0.0.1 - - 20/Sep/2026:09:48:15 +0000 "GET / HTTP/1.1" 200 1120 "-" "Wget/1.21.3" 9月 20 17:48:25 VM-0-6-tencentos bash6300: cube-webui | 127.0.0.1 - - 20/Sep/2026:09:48:25 +0000 "GET / HTTP/1.1" 200 1120 "-" "Wget/1.21.3" 9月 20 17:48:35 VM-0-6-tencentos bash6300: cube-webui | 127.0.0.1 - - 20/Sep/2026:09:48:35 +0000 "GET / HTTP/1.1" 200 1120 "-" "Wget/1.21.3" 9月 20 17:48:42 VM-0-6-tencentos bash5538: 2026/09/20 17:48:42 /workspace/CubeMaster/pkg/templatecenter/store.go:1118 record not found 9月 20 17:48:42 VM-0-6-tencentos bash5538: 0.371ms rows:0 SELECT * FROM t_cube_template_definition WHERE template_id = 'tpl-464f1b5538764bd7afa240ca' AND t_cube_template_definition.deleted_at IS NULL ORDER BY t_cube_template_definition.id LIMIT 1 9月 20 17:48:42 VM-0-6-tencentos bash5538: 2026/09/20 17:48:42 /workspace/CubeMaster/pkg/templatecenter/snapshot_table.go:29 record not found 9月 20 17:48:42 VM-0-6-tencentos bash5538: 0.363ms rows:0 SELECT * FROM t_cube_snapshot WHERE snapshot_id = 'tpl-464f1b5538764bd7afa240ca' AND t_cube_snapshot.deleted_at IS NULL ORDER BY t_cube_snapshot.id LIMIT 1 9月 20 17:48:45 VM-0-6-tencentos bash6300: cube-webui | 127.0.0.1 - - 20/Sep/2026:09:48:45 +0000 "GET / HTTP/1.1" 200 1120 "-" "Wget/1.21.3" 9月 20 17:48:55 VM-0-6-tencentos bash6300: cube-webui | 127.0.0.1 - - 20/Sep/2026:09:48:55 +0000 "GET / HTTP/1.1" 200 1120 "-" "Wget/1.21.3" 9月 20 17:49:05 VM-0-6-tencentos bash6300: cube-webui | 127.0.0.1 - - 20/Sep/2026:09:49:05 +0000 "GET / HTTP/1.1" 200 1120 "-" "Wget/1.21.3" 9月 20 17:49:12 VM-0-6-tencentos bash5538: 2026/09/20 17:49:12 /workspace/CubeMaster/pkg/templatecenter/store.go:1118 record not found 9月 20 17:49:12 VM-0-6-tencentos bash5538: 0.361ms rows:0 SELECT * FROM t_cube_template_definition WHERE template_id = 'tpl-464f1b5538764bd7afa240ca' AND t_cube_template_definition.deleted_at IS NULL ORDER BY t_cube_template_definition.id LIMIT 1 9月 20 17:49:12 VM-0-6-tencentos bash5538: 2026/09/20 17:49:12 /workspace/CubeMaster/pkg/templatecenter/snapshot_table.go:29 record not found 9月 20 17:49:12 VM-0-6-tencentos bash5538: 0.356ms rows:0 SELECT * FROM t_cube_snapshot WHERE snapshot_id = 'tpl-464f1b5538764bd7afa240ca' AND t_cube_snapshot.deleted_at IS NULL ORDER BY t_cube_snapshot.id LIMIT 1 9月 20 17:49:15 VM-0-6-tencentos bash6300: cube-webui | 127.0.0.1 - - 20/Sep/2026:09:49:15 +0000 "GET / HTTP/1.1" 200 1120 "-" "Wget/1.21.3" 9月 20 17:49:25 VM-0-6-tencentos bash6300: cube-webui | 127.0.0.1 - - 20/Sep/2026:09:49:25 +0000 "GET / HTTP/1.1" 200 1120 "-" "Wget/1.21.3" 9月 20 17:49:35 VM-0-6-tencentos bash6300: cube-webui | 127.0.0.1 - - 20/Sep/2026:09:49:35 +0000 "GET / HTTP/1.1" 200 1120 "-" "Wget/1.21.3" 9月 20 17:49:42 VM-0-6-tencentos bash5538: 2026/09/20 17:49:42 /workspace/CubeMaster/pkg/templatecenter/store.go:1118 record not found 9月 20 17:49:42 VM-0-6-tencentos bash5538: 0.426ms rows:0 SELECT * FROM t_cube_template_definition WHERE template_id = 'tpl-464f1b5538764bd7afa240ca' AND t_cube_template_definition.deleted_at IS NULL ORDER BY t_cube_template_definition.id LIMIT 1 9月 20 17:49:42 VM-0-6-tencentos bash5538: 2026/09/20 17:49:42 /workspace/CubeMaster/pkg/templatecenter/snapshot_table.go:29 record not found 9月 20 17:49:42 VM-0-6-tencentos bash5538: 0.403ms rows:0 SELECT * FROM t_cube_snapshot WHERE snapshot_id = 'tpl-464f1b5538764bd7afa240ca' AND t_cube_snapshot.deleted_at IS NULL ORDER BY t_cube_snapshot.id LIMIT 1 9月 20 17:49:45 VM-0-6-tencentos bash6300: cube-webui | 127.0.0.1 - - 20/Sep/2026:09:49:45 +0000 "GET / HTTP/1.1" 200 1120 "-" "Wget/1.21.3" 9月 20 17:49:54 VM-0-6-tencentos bash3723: cube-sandbox-redis | 1:M 20 Sep 2026 09:49:54.070 * 100 changes in 300 seconds. Saving... 9月 20 17:49:54 VM-0-6-tencentos bash3723: cube-sandbox-redis | 1:M 20 Sep 2026 09:49:54.070 * Background saving started by pid 13477 9月 20 17:49:54 VM-0-6-tencentos bash3723: cube-sandbox-redis | 13477:C 20 Sep 2026 09:49:54.073 * DB saved on disk 9月 20 17:49:54 VM-0-6-tencentos bash3723: cube-sandbox-redis | 13477:C 20 Sep 2026 09:49:54.073 * Fork CoW for RDB: current 0 MB, peak 0 MB, average 0 MB 9月 20 17:49:54 VM-0-6-tencentos bash3723: cube-sandbox-redis | 1:M 20 Sep 2026 09:49:54.171 * Background saving terminated with success 9月 20 17:49:56 VM-0-6-tencentos bash6300: cube-webui | 127.0.0.1 - - 20/Sep/2026:09:49:56 +0000 "GET / HTTP/1.1" 200 1120 "-" "Wget/1.21.3" 9月 20 17:50:06 VM-0-6-tencentos bash6300: cube-webui | 127.0.0.1 - - 20/Sep/2026:09:50:06 +0000 "GET / HTTP/1.1" 200 1120 "-" "Wget/1.21.3" 9月 20 17:50:12 VM-0-6-tencentos bash5538: 2026/09/20 17:50:12 /workspace/CubeMaster/pkg/templatecenter/store.go:1118 record not found 9月 20 17:50:12 VM-0-6-tencentos bash5538: 0.374ms rows:0 SELECT * FROM t_cube_template_definition WHERE template_id = 'tpl-464f1b5538764bd7afa240ca' AND t_cube_template_definition.deleted_at IS NULL ORDER BY t_cube_template_definition.id LIMIT 1 9月 20 17:50:12 VM-0-6-tencentos bash5538: 2026/09/20 17:50:12 /workspace/CubeMaster/pkg/templatecenter/snapshot_table.go:29 record not found 9月 20 17:50:12 VM-0-6-tencentos bash5538: 0.366ms rows:0 SELECT * FROM t_cube_snapshot WHERE snapshot_id = 'tpl-464f1b5538764bd7afa240ca' AND t_cube_snapshot.deleted_at IS NULL ORDER BY t_cube_snapshot.id LIMIT 1 9月 20 17:50:16 VM-0-6-tencentos bash6300: cube-webui | 127.0.0.1 - - 20/Sep/2026:09:50:16 +0000 "GET / HTTP/1.1" 200 1120 "-" "Wget/1.21.3" 9月 20 17:50:26 VM-0-6-tencentos bash6300: cube-webui | 127.0.0.1 - - 20/Sep/2026:09:50:26 +0000 "GET / HTTP/1.1" 200 1120 "-" "Wget/1.21.3" 9月 20 17:50:36 VM-0-6-tencentos bash6300: cube-webui | 127.0.0.1 - - 20/Sep/2026:09:50:36 +0000 "GET / HTTP/1.1" 200 1120 "-" "Wget/1.21.3" 9月 20 17:50:42 VM-0-6-tencentos bash5538: 2026/09/20 17:50:42 /workspace/CubeMaster/pkg/templatecenter/store.go:1118 record not found 9月 20 17:50:42 VM-0-6-tencentos bash5538: 0.350ms rows:0 SELECT * FROM t_cube_template_definition WHERE template_id = 'tpl-464f1b5538764bd7afa240ca' AND t_cube_template_definition.deleted_at IS NULL ORDER BY t_cube_template_definition.id LIMIT 1 9月 20 17:50:42 VM-0-6-tencentos bash5538: 2026/09/20 17:50:42 /workspace/CubeMaster/pkg/templatecenter/snapshot_table.go:29 record not found 9月 20 17:50:42 VM-0-6-tencentos bash5538: 0.329ms rows:0 SELECT * FROM t_cube_snapshot WHERE snapshot_id = 'tpl-464f1b5538764bd7afa240ca' AND t_cube_snapshot.deleted_at IS NULL ORDER BY t_cube_snapshot.id LIMIT 1 9月 20 17:50:46 VM-0-6-tencentos bash6300: cube-webui | 127.0.0.1 - - 20/Sep/2026:09:50:46 +0000 "GET / HTTP/1.1" 200 1120 "-" "Wget/1.21.3" 9月 20 17:50:56 VM-0-6-tencentos bash6300: cube-webui | 127.0.0.1 - - 20/Sep/2026:09:50:56 +0000 "GET / HTTP/1.1" 200 1120 "-" "Wget/1.21.3" 9月 20 17:51:06 VM-0-6-tencentos bash6300: cube-webui | 127.0.0.1 - - 20/Sep/2026:09:51:06 +0000 "GET / HTTP/1.1" 200 1120 "-" "Wget/1.21.3" 9月 20 17:51:12 VM-0-6-tencentos bash5538: 2026/09/20 17:51:12 /workspace/CubeMaster/pkg/templatecenter/store.go:1118 record not found 9月 20 17:51:12 VM-0-6-tencentos bash5538: 0.421ms rows:0 SELECT * FROM t_cube_template_definition WHERE template_id = 'tpl-464f1b5538764bd7afa240ca' AND t_cube_template_definition.deleted_at IS NULL ORDER BY t_cube_template_definition.id LIMIT 1 9月 20 17:51:12 VM-0-6-tencentos bash5538: 2026/09/20 17:51:12 /workspace/CubeMaster/pkg/templatecenter/snapshot_table.go:29 record not found 9月 20 17:51:12 VM-0-6-tencentos bash5538: 0.366ms rows:0 SELECT * FROM t_cube_snapshot WHERE snapshot_id = 'tpl-464f1b5538764bd7afa240ca' AND t_cube_snapshot.deleted_at IS NULL ORDER BY t_cube_snapshot.id LIMIT 1 9月 20 17:51:16 VM-0-6-tencentos bash6300: cube-webui | 127.0.0.1 - - 20/Sep/2026:09:51:16 +0000 "GET / HTTP/1.1" 200 1120 "-" "Wget/1.21.3" 9月 20 17:51:26 VM-0-6-tencentos bash6300: cube-webui | 127.0.0.1 - - 20/Sep/2026:09:51:26 +0000 "GET / HTTP/1.1" 200 1120 "-" "Wget/1.21.3" 9月 20 17:51:36 VM-0-6-tencentos bash6300: cube-webui | 127.0.0.1 - - 20/Sep/2026:09:51:36 +0000 "GET / HTTP/1.1" 200 1120 "-" "Wget/1.21.3" 9月 20 17:51:42 VM-0-6-tencentos bash5538: 2026/09/20 17:51:42 /workspace/CubeMaster/pkg/templatecenter/store.go:1118 record not found 9月 20 17:51:42 VM-0-6-tencentos bash5538: 0.386ms rows:0 SELECT * FROM t_cube_template_definition WHERE template_id = 'tpl-464f1b5538764bd7afa240ca' AND t_cube_template_definition.deleted_at IS NULL ORDER BY t_cube_template_definition.id LIMIT 1 9月 20 17:51:42 VM-0-6-tencentos bash5538: 2026/09/20 17:51:42 /workspace/CubeMaster/pkg/templatecenter/snapshot_table.go:29 record not found 9月 20 17:51:42 VM-0-6-tencentos bash5538: 0.384ms rows:0 SELECT * FROM t_cube_snapshot WHERE snapshot_id = 'tpl-464f1b5538764bd7afa240ca' AND t_cube_snapshot.deleted_at IS NULL ORDER BY t_cube_snapshot.id LIMIT 1 9月 20 17:51:46 VM-0-6-tencentos bash6300: cube-webui | 127.0.0.1 - - 20/Sep/2026:09:51:46 +0000 "GET / HTTP/1.1" 200 1120 "-" "Wget/1.21.3" 9月 20 17:51:56 VM-0-6-tencentos bash6300: cube-webui | 127.0.0.1 - - 20/Sep/2026:09:51:56 +0000 "GET / HTTP/1.1" 200 1120 "-" "Wget/1.21.3" 9月 20 17:52:06 VM-0-6-tencentos bash6300: cube-webui | 127.0.0.1 - - 20/Sep/2026:09:52:06 +0000 "GET / HTTP/1.1" 200 1120 "-" "Wget/1.21.3" 9月 20 17:52:12 VM-0-6-tencentos bash5538: 2026/09/20 17:52:12 /workspace/CubeMaster/pkg/templatecenter/store.go:1118 record not found 9月 20 17:52:12 VM-0-6-tencentos bash5538: 0.369ms rows:0 SELECT * FROM t_cube_template_definition WHERE template_id = 'tpl-464f1b5538764bd7afa240ca' AND t_cube_template_definition.deleted_at IS NULL ORDER BY t_cube_template_definition.id LIMIT 1 9月 20 17:52:12 VM-0-6-tencentos bash5538: 2026/09/20 17:52:12 /workspace/CubeMaster/pkg/templatecenter/snapshot_table.go:29 record not found 9月 20 17:52:12 VM-0-6-tencentos bash5538: 0.375ms rows:0 SELECT * FROM t_cube_snapshot WHERE snapshot_id = 'tpl-464f1b5538764bd7afa240ca' AND t_cube_snapshot.deleted_at IS NULL ORDER BY t_cube_snapshot.id LIMIT 1 9月 20 17:52:16 VM-0-6-tencentos bash6300: cube-webui | 127.0.0.1 - - 20/Sep/2026:09:52:16 +0000 "GET / HTTP/1.1" 200 1120 "-" "Wget/1.21.3" 9月 20 17:52:26 VM-0-6-tencentos bash6300: cube-webui | 127.0.0.1 - - 20/Sep/2026:09:52:26 +0000 "GET / HTTP/1.1" 200 1120 "-" "Wget/1.21.3" 9月 20 17:52:36 VM-0-6-tencentos bash6300: cube-webui | 127.0.0.1 - - 20/Sep/2026:09:52:36 +0000 "GET / HTTP/1.1" 200 1120 "-" "Wget/1.21.3" 9月 20 17:52:42 VM-0-6-tencentos bash5538: 2026/09/20 17:52:42 /workspace/CubeMaster/pkg/templatecenter/store.go:1118 record not found 9月 20 17:52:42 VM-0-6-tencentos bash5538: 0.341ms rows:0 SELECT * FROM t_cube_template_definition WHERE template_id = 'tpl-464f1b5538764bd7afa240ca' AND t_cube_template_definition.deleted_at IS NULL ORDER BY t_cube_template_definition.id LIMIT 1 9月 20 17:52:42 VM-0-6-tencentos bash5538: 2026/09/20 17:52:42 /workspace/CubeMaster/pkg/templatecenter/snapshot_table.go:29 record not found 9月 20 17:52:42 VM-0-6-tencentos bash5538: 0.325ms rows:0 SELECT * FROM t_cube_snapshot WHERE snapshot_id = 'tpl-464f1b5538764bd7afa240ca' AND t_cube_snapshot.deleted_at IS NULL ORDER BY t_cube_snapshot.id LIMIT 1 9月 20 17:52:46 VM-0-6-tencentos bash6300: cube-webui | 127.0.0.1 - - 20/Sep/2026:09:52:46 +0000 "GET / HTTP/1.1" 200 1120 "-" "Wget/1.21.3" 9月 20 17:52:56 VM-0-6-tencentos bash6300: cube-webui | 127.0.0.1 - - 20/Sep/2026:09:52:56 +0000 "GET / HTTP/1.1" 200 1120 "-" "Wget/1.21.3" 9月 20 17:53:06 VM-0-6-tencentos bash6300: cube-webui | 127.0.0.1 - - 20/Sep/2026:09:53:06 +0000 "GET / HTTP/1.1" 200 1120 "-" "Wget/1.21.3" 9月 20 17:53:12 VM-0-6-tencentos bash5538: 2026/09/20 17:53:12 /workspace/CubeMaster/pkg/templatecenter/store.go:1118 record not found 9月 20 17:53:12 VM-0-6-tencentos bash5538: 0.368ms rows:0 SELECT * FROM t_cube_template_definition WHERE template_id = 'tpl-464f1b5538764bd7afa240ca' AND t_cube_template_definition.deleted_at IS NULL ORDER BY t_cube_template_definition.id LIMIT 1 9月 20 17:53:12 VM-0-6-tencentos bash5538: 2026/09/20 17:53:12 /workspace/CubeMaster/pkg/templatecenter/snapshot_table.go:29 record not found 9月 20 17:53:12 VM-0-6-tencentos bash5538: 0.382ms rows:0 SELECT * FROM t_cube_snapshot WHERE snapshot_id = 'tpl-464f1b5538764bd7afa240ca' AND t_cube_snapshot.deleted_at IS NULL ORDER BY t_cube_snapshot.id LIMIT 1 9月 20 17:53:16 VM-0-6-tencentos bash6300: cube-webui | 127.0.0.1 - - 20/Sep/2026:09:53:16 +0000 "GET / HTTP/1.1" 200 1120 "-" "Wget/1.21.3" 9月 20 17:53:26 VM-0-6-tencentos bash6300: cube-webui | 127.0.0.1 - - 20/Sep/2026:09:53:26 +0000 "GET / HTTP/1.1" 200 1120 "-" "Wget/1.21.3" root@VM-0-6-tencentos \~# docker ps | grep cube 5d9ef03bf1ed cube-sandbox-cn.tencentcloudcr.com/cube-sandbox/cube-egress:v0.7.1 "/usr/local/openrest..." About an hour ago Up About an hour cube-egress d5c0c2c48c92 cube-sandbox-image.tencentcloudcr.com/opensource/openresty:1.21.4.1-6-alpine-fat "/usr/local/openrest..." About an hour ago Up About an hour (healthy) 0.0.0.0:12088->80/tcp cube-webui 5041bbb48a0f cube-sandbox-cn.tencentcloudcr.com/cube-sandbox/cube-proxy:v0.7.1 "/usr/local/openrest..." About an hour ago Up About an hour cube-proxy 459a081ffa37 cube-sandbox-cn.tencentcloudcr.com/cube-sandbox/cube-lifecycle-manager:v0.7.1 "/usr/local/bin/cube..." About an hour ago Up About an hour cube-lifecycle-manager 8d12b0ccbb0b cube-sandbox-image.tencentcloudcr.com/opensource/redis:7-alpine "docker-entrypoint.s..." About an hour ago Up About an hour (healthy) 127.0.0.1:6379->6379/tcp cube-sandbox-redis 76096c049704 cube-sandbox-image.tencentcloudcr.com/opensource/mysql:8.0 "docker-entrypoint.s..." About an hour ago Up About an hour (healthy) 127.0.0.1:3306->3306/tcp, 33060/tcp cube-sandbox-mysql 861435ecd634 cube-sandbox-cn.tencentcloudcr.com/cube-sandbox/minio:RELEASE.2025-09-07T16-13-09Z "/usr/bin/docker-ent..." About an hour ago Up About an hour (healthy) 192.168.0.6:9000->9000/tcp, 127.0.0.1:9001->9001/tcp cube-sandbox-minio a0dd3c78f388 cube-sandbox-image.tencentcloudcr.com/opensource/coredns/coredns:1.14.2 "/coredns -conf /etc..." About an hour ago Up About an hour cube-proxy-coredns root@VM-0-6-tencentos \~# grep -R "ceaa39fc-aa7b-43d4-974a-1ef430a074f7" /var/log 2>/dev/null root@VM-0-6-tencentos \~#

根据贴出的日志,已经可以缩小范围:

CubeSandbox 控制面正常运行,但模板构建任务记录已经丢失,没有留下失败原因

关键日志:

复制代码
SELECT * FROM `t_cube_template_definition`
WHERE template_id = 'tpl-464f1b5538764bd7afa240ca'
...
record not found

以及:

复制代码
SELECT * FROM `t_cube_snapshot`
WHERE snapshot_id = 'tpl-464f1b5538764bd7afa240ca'
...
record not found

说明:

  • WebUI 请求模板详情
  • CubeMaster 查询模板定义
  • MySQL 中已经没有该模板记录
  • snapshot 记录也不存在

所以现在看到的:

复制代码
模板构建失败
无法查看详情

不是当前实时错误,而是构建任务失败后清理了数据库记录


当前重点:找真正的 builder 日志

你的 systemd 列表里有:

复制代码
cube-sandbox-cubelet.service
cube-sandbox-cubemaster.service
cube-sandbox-cube-templatecenter.service
cube-sandbox-cube-api.service
cube-sandbox-cubeops.service

模板构建通常不是 WebUI/CubeMaster 报错,而是在:

  • cubelet
  • lifecycle-manager
  • templatecenter

里面


1. 查看 cubelet 日志(优先)

执行:

复制代码
journalctl -u cube-sandbox-cubelet -n 500 --no-pager

过滤:

复制代码
journalctl -u cube-sandbox-cubelet --no-pager | \
grep -Ei "template|snapshot|build|error|failed|harbor|image"

2. 查看 lifecycle-manager

模板创建任务通常由它执行:

复制代码
journalctl -u cube-sandbox-cube-lifecycle-manager -n 500 --no-pager

过滤:

复制代码
journalctl -u cube-sandbox-cube-lifecycle-manager --no-pager | \
grep -Ei "template|snapshot|error|failed"

3. 查看 TemplateCenter

复制代码
journalctl -u cube-sandbox-cube-templatecenter -n 500 --no-pager

4. Docker 容器日志

你的容器:

复制代码
459a081ffa37 cube-lifecycle-manager

查看:

复制代码
docker logs 459a081ffa37 --tail 500

重点:

复制代码
docker logs 459a081ffa37 2>&1 | \
grep -Ei "template|snapshot|error|failed"

5. 另外发现一个可能问题

你的模板来源之前截图:

复制代码
harbor-intranet.allintechinc.com/llm-planning/...

但当前:

复制代码
docker ps

没有看到任何 builder 容器

正常模板构建期间应该可能出现:

  • sandbox builder
  • qemu
  • guest
  • converter

现在全部没有

可能情况:

情况A

构建任务启动后立即失败

常见原因:

  • Harbor 拉取失败
  • 镜像权限失败
  • 镜像格式不支持

情况B

PVM 创建失败

你之前安装:

复制代码
CUBE_PVM_ENABLE=1

重点检查:

复制代码
lsmod | grep kvm

和:

复制代码
ls -l /dev/kvm

6. 检查 MinIO 是否有模板残留

因为模板 snapshot 存 MinIO:

复制代码
docker exec -it cube-sandbox-minio sh

或者:

复制代码
mc alias ls
mc ls <alias>/cube-volumes

看看是否有:

复制代码
tpl-464f1b5538764bd7afa240ca

7. 建议下一步直接执行这几个

一次贴回来:

复制代码
journalctl -u cube-sandbox-cubelet -n 200 --no-pager

journalctl -u cube-sandbox-cube-lifecycle-manager -n 200 --no-pager

docker logs 459a081ffa37 --tail 200

我预计错误会在这里出现

目前可以排除:

✅ CubeSandbox 主服务启动问题

✅ MySQL/Redis异常

✅ WebUI异常

重点怀疑:

  1. Harbor 镜像拉取失败
  2. PVM guest 创建失败
  3. 模板 snapshot 生成失败
  4. MinIO 写入失败

下一批日志基本可以直接定位。

继续排查。你当前贴的日志已经证明:

  • cube-sandbox-* 控制面正常 ✅
  • MySQL/Redis/MinIO 容器正常运行 ✅
  • 失败模板的数据库记录已经被清理 ❗
  • 真正失败点发生在模板构建执行阶段,不在 WebUI

下一步需要抓 构建执行组件日志

请执行下面命令:

1. Cubelet(最重要)

复制代码
journalctl -u cube-sandbox-cubelet -n 500 --no-pager

重点看:

复制代码
template
snapshot
pull
image
qemu
kvm
mount
error
failed

也可以过滤:

复制代码
journalctl -u cube-sandbox-cubelet --no-pager | \
grep -Ei "template|snapshot|image|pull|qemu|kvm|error|failed"

2. Lifecycle Manager

模板生命周期一般由它调度:

复制代码
journalctl -u cube-sandbox-cube-lifecycle-manager -n 500 --no-pager

过滤:

复制代码
journalctl -u cube-sandbox-cube-lifecycle-manager --no-pager | \
grep -Ei "template|snapshot|build|error|failed"

3. TemplateCenter

复制代码
journalctl -u cube-sandbox-cube-templatecenter -n 300 --no-pager

4. 查看是否有临时构建容器

模板失败后可能已经退出:

复制代码
docker ps -a | grep -Ei "cube|template|builder|qemu"

5. 检查 PVM 环境

因为你安装用了:

复制代码
CUBE_PVM_ENABLE=1

执行:

复制代码
lsmod | grep kvm

以及:

复制代码
ls -l /dev/kvm

正常应该类似:

复制代码
/dev/kvm

6. 检查 Harbor 连通性

你的模板镜像来自:

复制代码
harbor-intranet.allintechinc.com/llm-planning/...

执行:

复制代码
curl -k https://harbor-intranet.allintechinc.com/v2/

正常:

复制代码
{}

如果:

复制代码
401 Unauthorized

说明需要配置 Harbor 登录


我目前最怀疑的两个点

可能1:Harbor 私有镜像无法拉取

流程:

复制代码
创建模板
 ↓
CubeSandbox
 ↓
pull Harbor image
 ↓
失败
 ↓
删除 template/snapshot记录

符合你现在现象

可能2:PVM 模板转换失败

因为:

复制代码
CUBE_PVM_ENABLE=1

如果:

  • kernel 不匹配
  • /dev/kvm 不可用
  • guest kernel 创建失败

也会出现模板直接失败

journalctl -u cube-sandbox-cubelet -n 200 --no-pager journalctl -u cube-sandbox-cube-lifecycle-manager -n 200 --no-pager docker ps -a | grep -Ei "cube|template|builder|qemu" 9月 20 15:36:05 VM-0-6-tencentos systemd1: Starting cube-sandbox-cubelet.service - Cube Sandbox Cubelet... 9月 20 15:36:05 VM-0-6-tencentos bash5832: one-click-systemd updated cubelet cubeops_addr=http://127.0.0.1:3010 9月 20 15:36:05 VM-0-6-tencentos bash5922: one-click-systemd updated cubelet cubeops_addr=http://127.0.0.1:3010 9月 20 15:36:06 VM-0-6-tencentos bash5875: one-click-systemd updated cubelet cow.s3 enable=false 9月 20 15:36:06 VM-0-6-tencentos bash5875: one-click-systemd cubelet initial pid=6023 9月 20 15:36:06 VM-0-6-tencentos bash7079: 2026-09-20T15:36:06.584521396+08:00,listen file/usr/local/services/cubetoolbox/Cubelet/dynamicconf/conf.yaml 9月 20 15:36:06 VM-0-6-tencentos bash7079: cfg:{"Common":{"CommonTimeout":10000000000,"LogLevel":"","DescribeAsyncInterval":100000000,"EnablePFMode":false,"DescribeBDFInterval":10000000,"DescribeBDFTimeout":10000000000,"GetBDFByUuidCmd":"/usr/local/services/AdamPlugins-1.0/snhost_snic_sdk/cath/bm/tools/get_bdf_by_uuid","GetBDFByIfNameCmd":"/usr/local/services/AdamPlugins-1.0/snhost_snic_sdk/cath/bm/tools/get_bdf_by_ifname","CommandTimeout":1000000000,"DisableHostCgroup":true,"DisableVmCgroup":false,"EnableSandboxExecCmdBeforeExist":false,"SandboxExecCmdTimeOut":5000000000,"SandboxExecCmdBeforeExist":"sh","/usr/local/cubetools/cube_exec_cmd_before_exist.sh","SandboxExecCmdBeforeExistLogOut":false,"SandboxExecCmdOutMaxLines":2000,"SandboxExecCmdMatchLine":"","SandboxExecCmdAfterMatch":null,"SandboxExecCmdBase64AfterMatch":"","SandboxExecCmdAfterMatchLogOut":false,"CgroupDisableMemoryReparentFile":"","CgroupDisableCpusetList":"","DisableHostNetfile":true,"DefaultDNSServers":"119.29.29.29","DefaultDNSSearches":\[\],"DefaultDNSOptions":\[\],"ReconcileInterval":300000000000,"DisableCubeBoxTemplateBaseFormatPoolOfNumberVer":false},"Tenant":{"Tenants":{},"UseRespectiveNamespace":false,"DefaultNS":"default","ZiyanUinListStr":""},"MetaServerConfig":{"MetaServerEndpoint":"127.0.0.1:3010","NodeStatusMaxImages":40000,"CubeMasterHTTPAddr":"127.0.0.1:8089"},"HostConf":{"Quota":{"Cpu":0,"Mem":"","MvmLimit":0,"CreationConcurrentNum":0,"PausedResourceReleaseRatio":0},"GC":{"CodeExpirationTime":"72h","ImageExpirationTime":"24h"},"SchedulerLabel":"default-cluster"}} 9月 20 15:36:06 VM-0-6-tencentos bash7079: 2026/09/20 15:36:06 mounted state tmpfs /data/cubelet/state size=1024m 9月 20 15:36:06 VM-0-6-tencentos bash7079: 2026/09/20 15:36:06 set /data/cubelet/root as shared mount 9月 20 15:36:07 VM-0-6-tencentos bash5875: /usr/local/services/cubetoolbox/scripts/systemd/cubelet-start.sh: line 57: 6023 Killed ( export CUBE_SANDBOX_NODE_IP="{CUBE_SANDBOX_NODE_IP}"; exec "{CUBELET_BIN}" --config "{CUBELET_CONFIG}" --dynamic-conf-path "{CUBELET_DYNAMICCONF}" ) 9月 20 15:36:07 VM-0-6-tencentos bash5875: one-click-systemd cubelet stable pid=7079 9月 20 15:36:07 VM-0-6-tencentos systemd1: Started cube-sandbox-cubelet.service - Cube Sandbox Cubelet. 9月 20 16:18:54 VM-0-6-tencentos systemd1: Stopping cube-sandbox-cubelet.service - Cube Sandbox Cubelet... 9月 20 16:18:55 VM-0-6-tencentos bash87481: one-click-systemd cubelet stopped pid=7079 9月 20 16:18:55 VM-0-6-tencentos systemd1: cube-sandbox-cubelet.service: Deactivated successfully. 9月 20 16:18:55 VM-0-6-tencentos systemd1: Stopped cube-sandbox-cubelet.service - Cube Sandbox Cubelet. 9月 20 16:18:55 VM-0-6-tencentos systemd1: cube-sandbox-cubelet.service: Consumed 10.594s CPU time, 237.9M memory peak, 0B memory swap peak. -- Boot f6f2fcd1381547d990b4d2b666efa29d -- 9月 20 16:19:13 VM-0-6-tencentos systemd1: Starting cube-sandbox-cubelet.service - Cube Sandbox Cubelet... 9月 20 16:19:13 VM-0-6-tencentos bash1510: one-click-systemd updated cubelet cubeops_addr=http://127.0.0.1:3010 9月 20 16:19:13 VM-0-6-tencentos bash1587: one-click-systemd updated cubelet cubeops_addr=http://127.0.0.1:3010 9月 20 16:19:13 VM-0-6-tencentos bash1566: one-click-systemd updated cubelet cow.s3 enable=false 9月 20 16:19:13 VM-0-6-tencentos bash1566: one-click-systemd cubelet initial pid=1668 9月 20 16:19:14 VM-0-6-tencentos bash2125: 2026-09-20T16:19:14.223712774+08:00,listen file/usr/local/services/cubetoolbox/Cubelet/dynamicconf/conf.yaml 9月 20 16:19:14 VM-0-6-tencentos bash2125: cfg:{"Common":{"CommonTimeout":10000000000,"LogLevel":"","DescribeAsyncInterval":100000000,"EnablePFMode":false,"DescribeBDFInterval":10000000,"DescribeBDFTimeout":10000000000,"GetBDFByUuidCmd":"/usr/local/services/AdamPlugins-1.0/snhost_snic_sdk/cath/bm/tools/get_bdf_by_uuid","GetBDFByIfNameCmd":"/usr/local/services/AdamPlugins-1.0/snhost_snic_sdk/cath/bm/tools/get_bdf_by_ifname","CommandTimeout":1000000000,"DisableHostCgroup":true,"DisableVmCgroup":false,"EnableSandboxExecCmdBeforeExist":false,"SandboxExecCmdTimeOut":5000000000,"SandboxExecCmdBeforeExist":"sh","/usr/local/cubetools/cube_exec_cmd_before_exist.sh","SandboxExecCmdBeforeExistLogOut":false,"SandboxExecCmdOutMaxLines":2000,"SandboxExecCmdMatchLine":"","SandboxExecCmdAfterMatch":null,"SandboxExecCmdBase64AfterMatch":"","SandboxExecCmdAfterMatchLogOut":false,"CgroupDisableMemoryReparentFile":"","CgroupDisableCpusetList":"","DisableHostNetfile":true,"DefaultDNSServers":"119.29.29.29","DefaultDNSSearches":\[\],"DefaultDNSOptions":\[\],"ReconcileInterval":300000000000,"DisableCubeBoxTemplateBaseFormatPoolOfNumberVer":false},"Tenant":{"Tenants":{},"UseRespectiveNamespace":false,"DefaultNS":"default","ZiyanUinListStr":""},"MetaServerConfig":{"MetaServerEndpoint":"127.0.0.1:3010","NodeStatusMaxImages":40000,"CubeMasterHTTPAddr":"127.0.0.1:8089"},"HostConf":{"Quota":{"Cpu":0,"Mem":"","MvmLimit":0,"CreationConcurrentNum":0,"PausedResourceReleaseRatio":0},"GC":{"CodeExpirationTime":"72h","ImageExpirationTime":"24h"},"SchedulerLabel":"default-cluster"}} 9月 20 16:19:14 VM-0-6-tencentos bash2125: 2026/09/20 16:19:14 mounted state tmpfs /data/cubelet/state size=1024m 9月 20 16:19:14 VM-0-6-tencentos bash2125: 2026/09/20 16:19:14 set /data/cubelet/root as shared mount 9月 20 16:19:14 VM-0-6-tencentos bash1566: /usr/local/services/cubetoolbox/scripts/systemd/cubelet-start.sh: line 57: 1668 Killed ( export CUBE_SANDBOX_NODE_IP="{CUBE_SANDBOX_NODE_IP}"; exec "{CUBELET_BIN}" --config "{CUBELET_CONFIG}" --dynamic-conf-path "{CUBELET_DYNAMICCONF}" ) 9月 20 16:19:14 VM-0-6-tencentos bash1566: one-click-systemd cubelet stable pid=2125 9月 20 16:19:14 VM-0-6-tencentos systemd1: Started cube-sandbox-cubelet.service - Cube Sandbox Cubelet. 9月 20 16:24:13 VM-0-6-tencentos systemd1: Stopping cube-sandbox-cubelet.service - Cube Sandbox Cubelet... 9月 20 16:24:14 VM-0-6-tencentos bash18750: one-click-systemd cubelet stopped pid=2125 9月 20 16:24:14 VM-0-6-tencentos systemd1: cube-sandbox-cubelet.service: Deactivated successfully. 9月 20 16:24:14 VM-0-6-tencentos systemd1: Stopped cube-sandbox-cubelet.service - Cube Sandbox Cubelet. 9月 20 16:24:14 VM-0-6-tencentos systemd1: cube-sandbox-cubelet.service: Consumed 11.631s CPU time. -- Boot 0756528841154b8595d93af011d03619 -- 9月 20 16:24:35 VM-0-6-tencentos systemd1: Starting cube-sandbox-cubelet.service - Cube Sandbox Cubelet... 9月 20 16:24:35 VM-0-6-tencentos bash1505: one-click-systemd updated cubelet cubeops_addr=http://127.0.0.1:3010 9月 20 16:24:35 VM-0-6-tencentos bash1608: one-click-systemd updated cubelet cubeops_addr=http://127.0.0.1:3010 9月 20 16:24:35 VM-0-6-tencentos bash1580: one-click-systemd updated cubelet cow.s3 enable=false 9月 20 16:24:35 VM-0-6-tencentos bash1580: one-click-systemd cubelet initial pid=1710 9月 20 16:24:35 VM-0-6-tencentos bash1949: 2026-09-20T16:24:35.552388365+08:00,listen file/usr/local/services/cubetoolbox/Cubelet/dynamicconf/conf.yaml 9月 20 16:24:35 VM-0-6-tencentos bash1949: cfg:{"Common":{"CommonTimeout":10000000000,"LogLevel":"","DescribeAsyncInterval":100000000,"EnablePFMode":false,"DescribeBDFInterval":10000000,"DescribeBDFTimeout":10000000000,"GetBDFByUuidCmd":"/usr/local/services/AdamPlugins-1.0/snhost_snic_sdk/cath/bm/tools/get_bdf_by_uuid","GetBDFByIfNameCmd":"/usr/local/services/AdamPlugins-1.0/snhost_snic_sdk/cath/bm/tools/get_bdf_by_ifname","CommandTimeout":1000000000,"DisableHostCgroup":true,"DisableVmCgroup":false,"EnableSandboxExecCmdBeforeExist":false,"SandboxExecCmdTimeOut":5000000000,"SandboxExecCmdBeforeExist":"sh","/usr/local/cubetools/cube_exec_cmd_before_exist.sh","SandboxExecCmdBeforeExistLogOut":false,"SandboxExecCmdOutMaxLines":2000,"SandboxExecCmdMatchLine":"","SandboxExecCmdAfterMatch":null,"SandboxExecCmdBase64AfterMatch":"","SandboxExecCmdAfterMatchLogOut":false,"CgroupDisableMemoryReparentFile":"","CgroupDisableCpusetList":"","DisableHostNetfile":true,"DefaultDNSServers":"119.29.29.29","DefaultDNSSearches":\[\],"DefaultDNSOptions":\[\],"ReconcileInterval":300000000000,"DisableCubeBoxTemplateBaseFormatPoolOfNumberVer":false},"Tenant":{"Tenants":{},"UseRespectiveNamespace":false,"DefaultNS":"default","ZiyanUinListStr":""},"MetaServerConfig":{"MetaServerEndpoint":"127.0.0.1:3010","NodeStatusMaxImages":40000,"CubeMasterHTTPAddr":"127.0.0.1:8089"},"HostConf":{"Quota":{"Cpu":0,"Mem":"","MvmLimit":0,"CreationConcurrentNum":0,"PausedResourceReleaseRatio":0},"GC":{"CodeExpirationTime":"72h","ImageExpirationTime":"24h"},"SchedulerLabel":"default-cluster"}} 9月 20 16:24:35 VM-0-6-tencentos bash1949: 2026/09/20 16:24:35 mounted state tmpfs /data/cubelet/state size=1024m 9月 20 16:24:35 VM-0-6-tencentos bash1949: 2026/09/20 16:24:35 set /data/cubelet/root as shared mount 9月 20 16:24:36 VM-0-6-tencentos bash1580: /usr/local/services/cubetoolbox/scripts/systemd/cubelet-start.sh: line 57: 1710 Killed ( export CUBE_SANDBOX_NODE_IP="{CUBE_SANDBOX_NODE_IP}"; exec "{CUBELET_BIN}" --config "{CUBELET_CONFIG}" --dynamic-conf-path "{CUBELET_DYNAMICCONF}" ) 9月 20 16:24:36 VM-0-6-tencentos bash1580: one-click-systemd cubelet stable pid=1949 9月 20 16:24:36 VM-0-6-tencentos systemd1: Started cube-sandbox-cubelet.service - Cube Sandbox Cubelet. 9月 20 15:36:26 VM-0-6-tencentos systemd1: Starting cube-sandbox-cube-lifecycle-manager.service - Cube Sandbox cube-lifecycle-manager container... 9月 20 15:36:26 VM-0-6-tencentos bash9297: one-click-runtime cube-lifecycle-manager runtime files prepared under /usr/local/services/cubetoolbox/cube-lifecycle-manager 9月 20 15:36:26 VM-0-6-tencentos systemd1: Started cube-sandbox-cube-lifecycle-manager.service - Cube Sandbox cube-lifecycle-manager container. 9月 20 15:36:26 VM-0-6-tencentos bash9375: one-click-runtime pulling cube-sandbox-cn.tencentcloudcr.com/cube-sandbox/cube-lifecycle-manager:v0.7.1 9月 20 15:36:35 VM-0-6-tencentos bash11195: time="2026-09-20T15:36:35+08:00" level=warning msg="/usr/local/services/cubetoolbox/cube-lifecycle-manager/docker-compose.yaml: the attribute version is obsolete, it will be ignored, please remove it to avoid potential confusion" 9月 20 15:36:35 VM-0-6-tencentos bash11195: Container cube-lifecycle-manager Creating 9月 20 15:36:35 VM-0-6-tencentos bash11195: Container cube-lifecycle-manager Created 9月 20 15:36:35 VM-0-6-tencentos bash11195: Attaching to cube-lifecycle-manager 9月 20 15:36:35 VM-0-6-tencentos bash11195: Container cube-lifecycle-manager Starting 9月 20 15:36:36 VM-0-6-tencentos bash11195: cube-lifecycle-manager | {"level":"info","ts":1789889796.0163305,"caller":"cube-lifecycle-manager/main.go:59","msg":"cube-lifecycle-manager starting","redis_addr":"127.0.0.1:6379","cube_proxy_admin_urls":"http://127.0.0.1:8082","cubemaster_url":"http://192.168.0.6:8089","listen_addr":"0.0.0.0:8083","instance_identity":"VM-0-6-tencentos","leader_election_enabled":false} 9月 20 15:36:36 VM-0-6-tencentos bash11195: cube-lifecycle-manager | {"level":"info","ts":1789889796.0164294,"caller":"cube-lifecycle-manager/main.go:81","msg":"eventbus enabled"} 9月 20 15:36:36 VM-0-6-tencentos bash11195: cube-lifecycle-manager | {"level":"info","ts":1789889796.0179007,"caller":"cube-lifecycle-manager/main.go:641","msg":"bootstrap complete","entries":0} 9月 20 15:36:36 VM-0-6-tencentos bash11195: cube-lifecycle-manager | {"level":"info","ts":1789889796.0205412,"logger":"http","caller":"httpapi/server.go:86","msg":"cube-lifecycle-manager http server listening","addr":"0.0.0.0:8083"} 9月 20 15:36:36 VM-0-6-tencentos bash11195: Container cube-lifecycle-manager Started 9月 20 15:36:48 VM-0-6-tencentos bash11195: cube-lifecycle-manager | {"level":"info","ts":1789889808.0200999,"logger":"discovery","caller":"discovery/redis.go:182","msg":"discovery: proxy joined","proxy_id":"192.168.0.6:8082","admin_url":"http://192.168.0.6:8082"} 9月 20 15:36:48 VM-0-6-tencentos bash11195: cube-lifecycle-manager | {"level":"info","ts":1789889808.0201688,"logger":"replay","caller":"cube-lifecycle-manager/main.go:366","msg":"replay begin","proxy_id":"192.168.0.6:8082","admin_url":"http://192.168.0.6:8082","entries":0} 9月 20 15:36:48 VM-0-6-tencentos bash11195: cube-lifecycle-manager | {"level":"info","ts":1789889808.0202017,"logger":"replay","caller":"cube-lifecycle-manager/main.go:394","msg":"replay done","proxy_id":"192.168.0.6:8082","pushed":0,"failed":0} 9月 20 16:18:54 VM-0-6-tencentos systemd1: Stopping cube-sandbox-cube-lifecycle-manager.service - Cube Sandbox cube-lifecycle-manager container... 9月 20 16:18:54 VM-0-6-tencentos bash11195: 45B blob data 9月 20 16:18:54 VM-0-6-tencentos bash87476: one-click-runtime cube-lifecycle-manager stopped 9月 20 16:18:54 VM-0-6-tencentos systemd1: cube-sandbox-cube-lifecycle-manager.service: Deactivated successfully. 9月 20 16:18:54 VM-0-6-tencentos systemd1: Stopped cube-sandbox-cube-lifecycle-manager.service - Cube Sandbox cube-lifecycle-manager container. 9月 20 16:18:54 VM-0-6-tencentos systemd1: cube-sandbox-cube-lifecycle-manager.service: Consumed 1.073s CPU time, 29.4M memory peak, 0B memory swap peak. -- Boot f6f2fcd1381547d990b4d2b666efa29d -- 9月 20 16:19:20 VM-0-6-tencentos systemd1: Starting cube-sandbox-cube-lifecycle-manager.service - Cube Sandbox cube-lifecycle-manager container... 9月 20 16:19:20 VM-0-6-tencentos bash7085: one-click-runtime cube-lifecycle-manager runtime files prepared under /usr/local/services/cubetoolbox/cube-lifecycle-manager 9月 20 16:19:20 VM-0-6-tencentos systemd1: Started cube-sandbox-cube-lifecycle-manager.service - Cube Sandbox cube-lifecycle-manager container. 9月 20 16:19:20 VM-0-6-tencentos bash7170: one-click-runtime pulling cube-sandbox-cn.tencentcloudcr.com/cube-sandbox/cube-lifecycle-manager:v0.7.1 9月 20 16:19:21 VM-0-6-tencentos bash7978: time="2026-09-20T16:19:21+08:00" level=warning msg="/usr/local/services/cubetoolbox/cube-lifecycle-manager/docker-compose.yaml: the attribute version is obsolete, it will be ignored, please remove it to avoid potential confusion" 9月 20 16:19:21 VM-0-6-tencentos bash7978: Container cube-lifecycle-manager Creating 9月 20 16:19:21 VM-0-6-tencentos bash7978: Container cube-lifecycle-manager Created 9月 20 16:19:21 VM-0-6-tencentos bash7978: Attaching to cube-lifecycle-manager 9月 20 16:19:21 VM-0-6-tencentos bash7978: Container cube-lifecycle-manager Starting 9月 20 16:19:21 VM-0-6-tencentos bash7978: Container cube-lifecycle-manager Started 9月 20 16:19:21 VM-0-6-tencentos bash7978: cube-lifecycle-manager | {"level":"info","ts":1789892361.5976703,"caller":"cube-lifecycle-manager/main.go:59","msg":"cube-lifecycle-manager starting","redis_addr":"127.0.0.1:6379","cube_proxy_admin_urls":"http://127.0.0.1:8082","cubemaster_url":"http://192.168.0.6:8089","listen_addr":"0.0.0.0:8083","instance_identity":"VM-0-6-tencentos","leader_election_enabled":false} 9月 20 16:19:21 VM-0-6-tencentos bash7978: cube-lifecycle-manager | {"level":"info","ts":1789892361.600487,"caller":"cube-lifecycle-manager/main.go:81","msg":"eventbus enabled"} 9月 20 16:19:21 VM-0-6-tencentos bash7978: cube-lifecycle-manager | {"level":"info","ts":1789892361.6083386,"caller":"cube-lifecycle-manager/main.go:641","msg":"bootstrap complete","entries":0} 9月 20 16:19:21 VM-0-6-tencentos bash7978: cube-lifecycle-manager | {"level":"info","ts":1789892361.6112247,"logger":"http","caller":"httpapi/server.go:86","msg":"cube-lifecycle-manager http server listening","addr":"0.0.0.0:8083"} 9月 20 16:19:24 VM-0-6-tencentos bash7978: cube-lifecycle-manager | {"level":"info","ts":1789892364.61076,"logger":"discovery","caller":"discovery/redis.go:182","msg":"discovery: proxy joined","proxy_id":"192.168.0.6:8082","admin_url":"http://192.168.0.6:8082"} 9月 20 16:19:24 VM-0-6-tencentos bash7978: cube-lifecycle-manager | {"level":"info","ts":1789892364.6108475,"logger":"replay","caller":"cube-lifecycle-manager/main.go:366","msg":"replay begin","proxy_id":"192.168.0.6:8082","admin_url":"http://192.168.0.6:8082","entries":0} 9月 20 16:19:24 VM-0-6-tencentos bash7978: cube-lifecycle-manager | {"level":"info","ts":1789892364.610934,"logger":"replay","caller":"cube-lifecycle-manager/main.go:394","msg":"replay done","proxy_id":"192.168.0.6:8082","pushed":0,"failed":0} 9月 20 16:24:13 VM-0-6-tencentos systemd1: Stopping cube-sandbox-cube-lifecycle-manager.service - Cube Sandbox cube-lifecycle-manager container... 9月 20 16:24:18 VM-0-6-tencentos bash7978: 45B blob data 9月 20 16:24:18 VM-0-6-tencentos bash18760: one-click-runtime cube-lifecycle-manager stopped 9月 20 16:24:18 VM-0-6-tencentos systemd1: cube-sandbox-cube-lifecycle-manager.service: Deactivated successfully. 9月 20 16:24:18 VM-0-6-tencentos systemd1: Stopped cube-sandbox-cube-lifecycle-manager.service - Cube Sandbox cube-lifecycle-manager container. -- Boot 0756528841154b8595d93af011d03619 -- 9月 20 16:24:41 VM-0-6-tencentos systemd1: Starting cube-sandbox-cube-lifecycle-manager.service - Cube Sandbox cube-lifecycle-manager container... 9月 20 16:24:41 VM-0-6-tencentos bash4667: one-click-runtime cube-lifecycle-manager runtime files prepared under /usr/local/services/cubetoolbox/cube-lifecycle-manager 9月 20 16:24:41 VM-0-6-tencentos systemd1: Started cube-sandbox-cube-lifecycle-manager.service - Cube Sandbox cube-lifecycle-manager container. 9月 20 16:24:41 VM-0-6-tencentos bash4762: one-click-runtime pulling cube-sandbox-cn.tencentcloudcr.com/cube-sandbox/cube-lifecycle-manager:v0.7.1 9月 20 16:24:42 VM-0-6-tencentos bash5603: time="2026-09-20T16:24:42+08:00" level=warning msg="/usr/local/services/cubetoolbox/cube-lifecycle-manager/docker-compose.yaml: the attribute version is obsolete, it will be ignored, please remove it to avoid potential confusion" 9月 20 16:24:42 VM-0-6-tencentos bash5603: Container cube-lifecycle-manager Creating 9月 20 16:24:42 VM-0-6-tencentos bash5603: Container cube-lifecycle-manager Created 9月 20 16:24:42 VM-0-6-tencentos bash5603: Attaching to cube-lifecycle-manager 9月 20 16:24:42 VM-0-6-tencentos bash5603: Container cube-lifecycle-manager Starting 9月 20 16:24:42 VM-0-6-tencentos bash5603: Container cube-lifecycle-manager Started 9月 20 16:24:42 VM-0-6-tencentos bash5603: cube-lifecycle-manager | {"level":"info","ts":1789892682.4259038,"caller":"cube-lifecycle-manager/main.go:59","msg":"cube-lifecycle-manager starting","redis_addr":"127.0.0.1:6379","cube_proxy_admin_urls":"http://127.0.0.1:8082","cubemaster_url":"http://192.168.0.6:8089","listen_addr":"0.0.0.0:8083","instance_identity":"VM-0-6-tencentos","leader_election_enabled":false} 9月 20 16:24:42 VM-0-6-tencentos bash5603: cube-lifecycle-manager | {"level":"info","ts":1789892682.428625,"caller":"cube-lifecycle-manager/main.go:81","msg":"eventbus enabled"} 9月 20 16:24:42 VM-0-6-tencentos bash5603: cube-lifecycle-manager | {"level":"info","ts":1789892682.4323788,"caller":"cube-lifecycle-manager/main.go:641","msg":"bootstrap complete","entries":0} 9月 20 16:24:42 VM-0-6-tencentos bash5603: cube-lifecycle-manager | {"level":"info","ts":1789892682.4353082,"logger":"http","caller":"httpapi/server.go:86","msg":"cube-lifecycle-manager http server listening","addr":"0.0.0.0:8083"} 9月 20 16:24:45 VM-0-6-tencentos bash5603: cube-lifecycle-manager | {"level":"info","ts":1789892685.4343956,"logger":"discovery","caller":"discovery/redis.go:182","msg":"discovery: proxy joined","proxy_id":"192.168.0.6:8082","admin_url":"http://192.168.0.6:8082"} 9月 20 16:24:45 VM-0-6-tencentos bash5603: cube-lifecycle-manager | {"level":"info","ts":1789892685.434483,"logger":"replay","caller":"cube-lifecycle-manager/main.go:366","msg":"replay begin","proxy_id":"192.168.0.6:8082","admin_url":"http://192.168.0.6:8082","entries":0} 9月 20 16:24:45 VM-0-6-tencentos bash5603: cube-lifecycle-manager | {"level":"info","ts":1789892685.434507,"logger":"replay","caller":"cube-lifecycle-manager/main.go:394","msg":"replay done","proxy_id":"192.168.0.6:8082","pushed":0,"failed":0} 5d9ef03bf1ed cube-sandbox-cn.tencentcloudcr.com/cube-sandbox/cube-egress:v0.7.1 "/usr/local/openrest..." 2 hours ago Up 2 hours cube-egress d5c0c2c48c92 cube-sandbox-image.tencentcloudcr.com/opensource/openresty:1.21.4.1-6-alpine-fat "/usr/local/openrest..." 2 hours ago Up 2 hours (healthy) 0.0.0.0:12088->80/tcp cube-webui 5041bbb48a0f cube-sandbox-cn.tencentcloudcr.com/cube-sandbox/cube-proxy:v0.7.1 "/usr/local/openrest..." 2 hours ago Up 2 hours cube-proxy 459a081ffa37 cube-sandbox-cn.tencentcloudcr.com/cube-sandbox/cube-lifecycle-manager:v0.7.1 "/usr/local/bin/cube..." 2 hours ago Up 2 hours cube-lifecycle-manager 8d12b0ccbb0b cube-sandbox-image.tencentcloudcr.com/opensource/redis:7-alpine "docker-entrypoint.s..." 2 hours ago Up 2 hours (healthy) 127.0.0.1:6379->6379/tcp cube-sandbox-redis 76096c049704 cube-sandbox-image.tencentcloudcr.com/opensource/mysql:8.0 "docker-entrypoint.s..." 2 hours ago Up 2 hours (healthy) 127.0.0.1:3306->3306/tcp, 33060/tcp cube-sandbox-mysql 861435ecd634 cube-sandbox-cn.tencentcloudcr.com/cube-sandbox/minio:RELEASE.2025-09-07T16-13-09Z "/usr/bin/docker-ent..." 2 hours ago Up 2 hours (healthy) 192.168.0.6:9000->9000/tcp, 127.0.0.1:9001->9001/tcp cube-sandbox-minio a0dd3c78f388 cube-sandbox-image.tencentcloudcr.com/opensource/coredns/coredns:1.14.2 "/coredns -conf /etc..." 2 hours ago Up 2 hours cube-proxy-coredns

从日志看,Cube Sandbox v0.7.1 主链路基本启动正常,真正异常点在 cubelet 进程启动阶段被系统 Kill

关键异常:

复制代码
cubelet initial pid=1710

/usr/local/services/cubetoolbox/scripts/systemd/cubelet-start.sh: line 57:
1710 Killed

(exec "${CUBELET_BIN}" --config ... )

但是随后:

复制代码
cubelet stable pid=1949
systemd Started cube-sandbox-cubelet.service

说明:

  • wrapper 脚本认为 cubelet 已稳定
  • 但是第一次启动的真实 cubelet binary 被杀掉
  • 后续可能启动了另一个监听进程/守护逻辑
  • systemd 没有认为失败

1. 最大可能原因:OOM Killer 杀掉 cubelet

日志:

复制代码
Consumed 10.594s CPU time, 237.9M memory peak

虽然 cubelet 自己峰值只有 237MB,但启动瞬间可能:

  • 加载模板
  • 初始化 qemu
  • 扫描 image cache
  • 初始化 container runtime
  • 读取大量 metadata

触发瞬时内存压力

先确认:

复制代码
dmesg -T | grep -Ei "killed process|out of memory|oom"

或者:

复制代码
journalctl -k | grep -Ei "oom|killed"

重点看:

类似:

复制代码
Out of memory: Killed process 1710 (cubelet)
Killed process 1710 (cubelet) total-vm...

如果有,就是 OOM


2. 检查机器内存

执行:

复制代码
free -h

以及:

复制代码
cat /proc/meminfo | head

Cube Sandbox 最低建议:

组件 建议
cubelet >=2G
cube-proxy 512M
redis/mysql/minio >=3G
模板构建 >=4G
运行 sandbox 8G+

如果这是腾讯云轻量机/小规格 VM,很容易发生


3. 检查 systemd 是否限制内存

查看:

复制代码
systemctl cat cube-sandbox-cubelet

重点:

复制代码
MemoryLimit=
MemoryMax=

如果存在:

例如:

复制代码
MemoryMax=256M

会直接 kill

查看:

复制代码
systemctl show cube-sandbox-cubelet | grep -i memory

4. 检查 cgroup OOM

你的配置里面:

复制代码
"DisableHostCgroup":true,
"DisableVmCgroup":false

说明:

host cgroup 禁用了

但是 VM cgroup 开启

查看:

复制代码
cat /sys/fs/cgroup/memory.events

如果看到:

复制代码
oom 1
oom_kill 1

说明 cgroup kill


5. cubelet binary 单独启动测试

找到 binary:

复制代码
grep CUBELET_BIN \
/usr/local/services/cubetoolbox/scripts/systemd/cubelet-start.sh

例如:

复制代码
CUBELET_BIN=/usr/local/services/cubetoolbox/Cubelet/cubelet

直接运行:

复制代码
/usr/local/services/cubetoolbox/Cubelet/cubelet \
--config /xxx/conf.yaml \
--dynamic-conf-path /xxx/conf.yaml

观察:

  • panic
  • segmentation fault
  • permission denied
  • killed

6. 检查 kernel 是否因为权限杀

执行:

复制代码
journalctl -k -n 200

关注:

复制代码
segfault
apparmor
selinux
denied

7. lifecycle-manager 正常

这一部分没有问题:

复制代码
cube-lifecycle-manager starting

redis_addr:
127.0.0.1:6379

cubemaster_url:
http://192.168.0.6:8089

listen:
0.0.0.0:8083

状态:

复制代码
bootstrap complete entries=0

http server listening

proxy joined

说明:

✅ redis OK

✅ mysql OK

✅ minio OK

✅ proxy OK

✅ lifecycle manager OK


8. 当前架构状态

现在:

复制代码
             cube-webui
                |
           cube-proxy
                |
        cube-lifecycle-manager
                |
             redis
                |
            cubelet  ❌
                |
          qemu/container runtime

问题在:

复制代码
cubelet -> sandbox runtime

9. 建议下一步执行

请贴下面输出:

A. 看 OOM

复制代码
dmesg -T | grep -Ei "oom|killed"

B. 看机器资源

复制代码
free -h
df -h

C. 看 cubelet service

复制代码
systemctl cat cube-sandbox-cubelet

D. 看进程

复制代码
ps aux | grep cubelet

E. 看 kernel

复制代码
journalctl -k -n 100 --no-pager

目前判断:

80% 概率是内存/cgroup OOM,15% 是 cubelet 自身启动异常,5% 是权限或内核限制

v0.7.1 其它组件已经健康,先不用动 Redis、MinIO、Proxy。重点查 cubelet 被 Kill 原因

root@VM-0-6-tencentos \~# dmesg -T | grep -Ei "oom|killed" root@VM-0-6-tencentos \~# free -h df -h total used free shared buff/cache available Mem: 125Gi 9.7Gi 112Gi 9.0Mi 4.0Gi 115Gi Swap: 0B 0B 0B 文件系统 大小 已用 可用 已用% 挂载点 /dev/vda1 50G 15G 36G 29% / devtmpfs 4.0M 0 4.0M 0% /dev tmpfs 63G 24K 63G 1% /dev/shm tmpfs 26G 3.4M 26G 1% /run /dev/vdb 500G 9.7G 491G 2% /mnt/allin_data overlay 50G 15G 36G 29% /var/lib/docker/rootfs/overlayfs/a0dd3c78f388d00b5da2b45e55232b14edbe6bd76026713aee3d701c19844282 overlay 50G 15G 36G 29% /var/lib/docker/rootfs/overlayfs/861435ecd634b2db18ec9e127fd778634b7f12f55c52b76dad3185ef796c29fe overlay 50G 15G 36G 29% /var/lib/docker/rootfs/overlayfs/8d12b0ccbb0b18a4c8ae2bc343c7de840c06e3a4a00ccb67215dd9ab73537407 overlay 50G 15G 36G 29% /var/lib/docker/rootfs/overlayfs/76096c0497040f2c04de824409505bbb6a89c82c7e4ebdb5e9119c914a0ed519 overlay 50G 15G 36G 29% /var/lib/docker/rootfs/overlayfs/459a081ffa3731c574f85bdf91424a92f5e703be1dcf7c19f01cb4ba875c82d9 overlay 50G 15G 36G 29% /var/lib/docker/rootfs/overlayfs/5041bbb48a0fe1c365ccef646224fd7037a04145976ba2feeb04f48ff6742706 overlay 50G 15G 36G 29% /var/lib/docker/rootfs/overlayfs/d5c0c2c48c9251acf461358177fca6dbafbfc77e8b0f5439703b9bea9d8a8117 overlay 50G 15G 36G 29% /var/lib/docker/rootfs/overlayfs/5d9ef03bf1eda889418b5487f557960fe62caae604871929dcca5a490011e7df 192.168.0.20:/guqqwbm4/ 10G 42M 10G 1% /mnt/allin_data/allin/pod_log/data_share root@VM-0-6-tencentos \~# systemctl cat cube-sandbox-cubelet # /etc/systemd/system/cube-sandbox-cubelet.service Unit Description=Cube Sandbox Cubelet After=network-online.target Wants=network-online.target ConditionPathExists=/usr/local/services/cubetoolbox/.one-click.env RequiresMountsFor=/data/cubelet /data/cube-shim /data/snapshot_pack PartOf=cube-sandbox-control.target cube-sandbox-compute.target Service Type=forking ExecStartPre=/usr/bin/bash /usr/local/services/cubetoolbox/scripts/systemd/prepare-compute-role.sh User=root WorkingDirectory=/usr/local/services/cubetoolbox EnvironmentFile=-/usr/local/services/cubetoolbox/.one-click.env PIDFile=/run/cube-sandbox-systemd/cubelet.pid Delegate=yes KillMode=process ExecStart=/usr/bin/bash /usr/local/services/cubetoolbox/scripts/systemd/cubelet-start.sh ExecStop=/usr/bin/bash /usr/local/services/cubetoolbox/scripts/systemd/cubelet-stop.sh Restart=on-failure RestartSec=2s TimeoutStartSec=60s TimeoutStopSec=30s Install WantedBy=cube-sandbox-control.target WantedBy=cube-sandbox-compute.target root@VM-0-6-tencentos \~# ps aux | grep cubelet root 1949 0.5 0.1 4164304 221340 ? Sl 16:24 0:31 /usr/local/services/cubetoolbox/Cubelet/bin/cubelet --config /usr/local/services/cubetoolbox/Cubelet/config/config.toml --dynamic-conf-path /usr/local/services/cubetoolbox/Cubelet/dynamicconf/conf.yaml root 178554 0.0 0.0 6696 2060 pts/0 S+ 18:03 0:00 grep --color=auto cubelet root@VM-0-6-tencentos \~# journalctl -k -n 100 --no-pager 9月 20 16:24:32 VM-0-6-tencentos systemd1: Mounting dev-mqueue.mount - POSIX Message Queue File System... 9月 20 16:24:32 VM-0-6-tencentos systemd1: Mounting sys-kernel-debug.mount - Kernel Debug File System... 9月 20 16:24:32 VM-0-6-tencentos systemd1: Mounting sys-kernel-tracing.mount - Kernel Trace File System... 9月 20 16:24:32 VM-0-6-tencentos systemd1: auth-rpcgss-module.service - Kernel Module supporting RPCSEC_GSS was skipped because of an unmet condition check (ConditionPathExists=/etc/krb5.keytab). 9月 20 16:24:32 VM-0-6-tencentos systemd1: Starting kmod-static-nodes.service - Create List of Static Device Nodes... 9月 20 16:24:32 VM-0-6-tencentos systemd1: loadmodules.service - Load legacy module configuration was skipped because no trigger condition checks were met. 9月 20 16:24:32 VM-0-6-tencentos systemd1: Starting lvm2-monitor.service - Monitoring of LVM2 mirrors, snapshots etc. using dmeventd or progress polling... 9月 20 16:24:32 VM-0-6-tencentos systemd1: Starting modprobe@configfs.service - Load Kernel Module configfs... 9月 20 16:24:32 VM-0-6-tencentos systemd1: Starting modprobe@drm.service - Load Kernel Module drm... 9月 20 16:24:32 VM-0-6-tencentos systemd1: Starting modprobe@fuse.service - Load Kernel Module fuse... 9月 20 16:24:32 VM-0-6-tencentos systemd1: Starting systemd-journald.service - Journal Service... 9月 20 16:24:32 VM-0-6-tencentos systemd1: Starting systemd-modules-load.service - Load Kernel Modules... 9月 20 16:24:32 VM-0-6-tencentos systemd1: systemd-pcrmachine.service - TPM2 PCR Machine ID Measurement was skipped because of an unmet condition check (ConditionSecurity=measured-uki). 9月 20 16:24:32 VM-0-6-tencentos systemd1: Starting systemd-remount-fs.service - Remount Root and Kernel File Systems... 9月 20 16:24:32 VM-0-6-tencentos systemd1: systemd-tpm2-setup-early.service - TPM2 SRK Setup (Early) was skipped because of an unmet condition check (ConditionSecurity=measured-uki). 9月 20 16:24:32 VM-0-6-tencentos systemd1: Starting systemd-udev-trigger.service - Coldplug All udev Devices... 9月 20 16:24:32 VM-0-6-tencentos systemd1: Mounted dev-hugepages.mount - Huge Pages File System. 9月 20 16:24:32 VM-0-6-tencentos systemd1: Mounted dev-mqueue.mount - POSIX Message Queue File System. 9月 20 16:24:32 VM-0-6-tencentos systemd1: Mounted sys-kernel-debug.mount - Kernel Debug File System. 9月 20 16:24:32 VM-0-6-tencentos systemd1: Mounted sys-kernel-tracing.mount - Kernel Trace File System. 9月 20 16:24:32 VM-0-6-tencentos systemd1: Finished kmod-static-nodes.service - Create List of Static Device Nodes. 9月 20 16:24:32 VM-0-6-tencentos systemd1: modprobe@configfs.service: Deactivated successfully. 9月 20 16:24:32 VM-0-6-tencentos systemd-journald767: Collecting audit messages is disabled. 9月 20 16:24:32 VM-0-6-tencentos kernel: fuse: init (API version 7.39) 9月 20 16:24:32 VM-0-6-tencentos systemd1: Finished modprobe@configfs.service - Load Kernel Module configfs. 9月 20 16:24:32 VM-0-6-tencentos systemd1: Started systemd-journald.service - Journal Service. 9月 20 16:24:32 VM-0-6-tencentos kernel: ACPI: bus type drm_connector registered 9月 20 16:24:33 VM-0-6-tencentos systemd-journald767: Received client request to flush runtime journal. 9月 20 16:24:33 VM-0-6-tencentos systemd-journald767: /var/log/journal/2fa88b3fbc4d497eaa61f948bcf01cb9/system.journal: Journal file uses a different sequence number ID, rotating. 9月 20 16:24:33 VM-0-6-tencentos systemd-journald767: Rotating system journal. 9月 20 16:24:33 VM-0-6-tencentos kernel: input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input2 9月 20 16:24:33 VM-0-6-tencentos kernel: ACPI: button: Power Button PWRF 9月 20 16:24:33 VM-0-6-tencentos kernel: input: PC Speaker as /devices/platform/pcspkr/input/input3 9月 20 16:24:33 VM-0-6-tencentos kernel: piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 9月 20 16:24:33 VM-0-6-tencentos kernel: XFS (vdb): Mounting V5 Filesystem 2435a2e9-47a9-42c4-b1a9-3492b8ef3232 9月 20 16:24:33 VM-0-6-tencentos kernel: cirrus 0000:00:02.0: vgaarb: deactivate vga console 9月 20 16:24:33 VM-0-6-tencentos kernel: Console: switching to colour dummy device 80x25 9月 20 16:24:33 VM-0-6-tencentos kernel: drm Initialized cirrus 2.0.0 2019 for 0000:00:02.0 on minor 0 9月 20 16:24:33 VM-0-6-tencentos kernel: fbcon: cirrusdrmfb (fb0) is primary device 9月 20 16:24:33 VM-0-6-tencentos kernel: XFS (vdb): Ending clean mount 9月 20 16:24:33 VM-0-6-tencentos kernel: Console: switching to colour frame buffer device 128x48 9月 20 16:24:33 VM-0-6-tencentos kernel: cirrus 0000:00:02.0: drm fb0: cirrusdrmfb frame buffer device 9月 20 16:24:33 VM-0-6-tencentos kernel: RPC: Registered named UNIX socket transport module. 9月 20 16:24:33 VM-0-6-tencentos kernel: RPC: Registered udp transport module. 9月 20 16:24:33 VM-0-6-tencentos kernel: RPC: Registered tcp transport module. 9月 20 16:24:33 VM-0-6-tencentos kernel: RPC: Registered tcp-with-tls transport module. 9月 20 16:24:33 VM-0-6-tencentos kernel: RPC: Registered tcp NFSv4.1 backchannel transport module. 9月 20 16:24:34 VM-0-6-tencentos kernel: ISO 9660 Extensions: Microsoft Joliet Level 3 9月 20 16:24:34 VM-0-6-tencentos kernel: ISO 9660 Extensions: RRIP_1991A 9月 20 16:24:34 VM-0-6-tencentos kernel: ISO 9660 Extensions: Microsoft Joliet Level 3 9月 20 16:24:34 VM-0-6-tencentos kernel: ISO 9660 Extensions: RRIP_1991A 9月 20 16:24:34 VM-0-6-tencentos kernel: 8021q: adding VLAN 0 to HW filter on device eth0 9月 20 16:24:34 VM-0-6-tencentos kernel: IPMI message handler: version 39.2 9月 20 16:24:34 VM-0-6-tencentos kernel: ipmi device interface 9月 20 16:24:34 VM-0-6-tencentos kernel: ipmi_si: IPMI System Interface driver 9月 20 16:24:34 VM-0-6-tencentos kernel: ipmi_si: Unable to find any System Interface(s) 9月 20 16:24:35 VM-0-6-tencentos kernel: block vda: the capability attribute has been deprecated. 9月 20 16:24:35 VM-0-6-tencentos kernel: ISO 9660 Extensions: Microsoft Joliet Level 3 9月 20 16:24:35 VM-0-6-tencentos kernel: ISO 9660 Extensions: RRIP_1991A 9月 20 16:24:37 VM-0-6-tencentos kernel: br-f323d309da84: port 1(veth6c05fad) entered blocking state 9月 20 16:24:37 VM-0-6-tencentos kernel: br-f323d309da84: port 1(veth6c05fad) entered disabled state 9月 20 16:24:37 VM-0-6-tencentos kernel: veth6c05fad: entered allmulticast mode 9月 20 16:24:37 VM-0-6-tencentos kernel: veth6c05fad: entered promiscuous mode 9月 20 16:24:37 VM-0-6-tencentos kernel: br-f323d309da84: port 1(veth6c05fad) entered blocking state 9月 20 16:24:37 VM-0-6-tencentos kernel: br-f323d309da84: port 1(veth6c05fad) entered forwarding state 9月 20 16:24:37 VM-0-6-tencentos kernel: br-f323d309da84: port 1(veth6c05fad) entered disabled state 9月 20 16:24:37 VM-0-6-tencentos kernel: br-f323d309da84: port 2(vethef8fcbc) entered blocking state 9月 20 16:24:37 VM-0-6-tencentos kernel: br-f323d309da84: port 2(vethef8fcbc) entered disabled state 9月 20 16:24:37 VM-0-6-tencentos kernel: vethef8fcbc: entered allmulticast mode 9月 20 16:24:37 VM-0-6-tencentos kernel: vethef8fcbc: entered promiscuous mode 9月 20 16:24:37 VM-0-6-tencentos kernel: br-f323d309da84: port 2(vethef8fcbc) entered blocking state 9月 20 16:24:37 VM-0-6-tencentos kernel: br-f323d309da84: port 2(vethef8fcbc) entered forwarding state 9月 20 16:24:37 VM-0-6-tencentos kernel: br-f323d309da84: port 2(vethef8fcbc) entered disabled state 9月 20 16:24:37 VM-0-6-tencentos kernel: br-f323d309da84: port 3(veth82f3ebc) entered blocking state 9月 20 16:24:37 VM-0-6-tencentos kernel: br-f323d309da84: port 3(veth82f3ebc) entered disabled state 9月 20 16:24:37 VM-0-6-tencentos kernel: veth82f3ebc: entered allmulticast mode 9月 20 16:24:37 VM-0-6-tencentos kernel: veth82f3ebc: entered promiscuous mode 9月 20 16:24:37 VM-0-6-tencentos kernel: br-f323d309da84: port 3(veth82f3ebc) entered blocking state 9月 20 16:24:37 VM-0-6-tencentos kernel: br-f323d309da84: port 3(veth82f3ebc) entered forwarding state 9月 20 16:24:37 VM-0-6-tencentos kernel: eth0: renamed from vetha1cbfcc 9月 20 16:24:37 VM-0-6-tencentos kernel: br-f323d309da84: port 3(veth82f3ebc) entered disabled state 9月 20 16:24:37 VM-0-6-tencentos kernel: br-f323d309da84: port 1(veth6c05fad) entered blocking state 9月 20 16:24:37 VM-0-6-tencentos kernel: br-f323d309da84: port 1(veth6c05fad) entered forwarding state 9月 20 16:24:37 VM-0-6-tencentos kernel: eth0: renamed from veth807343f 9月 20 16:24:37 VM-0-6-tencentos kernel: eth0: renamed from veth11f3145 9月 20 16:24:37 VM-0-6-tencentos kernel: br-f323d309da84: port 2(vethef8fcbc) entered blocking state 9月 20 16:24:37 VM-0-6-tencentos kernel: br-f323d309da84: port 2(vethef8fcbc) entered forwarding state 9月 20 16:24:37 VM-0-6-tencentos kernel: br-f323d309da84: port 3(veth82f3ebc) entered blocking state 9月 20 16:24:37 VM-0-6-tencentos kernel: br-f323d309da84: port 3(veth82f3ebc) entered forwarding state 9月 20 16:24:43 VM-0-6-tencentos kernel: br-cd57826e4de1: port 1(veth61ccaa6) entered blocking state 9月 20 16:24:43 VM-0-6-tencentos kernel: br-cd57826e4de1: port 1(veth61ccaa6) entered disabled state 9月 20 16:24:43 VM-0-6-tencentos kernel: veth61ccaa6: entered allmulticast mode 9月 20 16:24:43 VM-0-6-tencentos kernel: veth61ccaa6: entered promiscuous mode 9月 20 16:24:43 VM-0-6-tencentos kernel: br-cd57826e4de1: port 1(veth61ccaa6) entered blocking state 9月 20 16:24:43 VM-0-6-tencentos kernel: br-cd57826e4de1: port 1(veth61ccaa6) entered forwarding state 9月 20 16:24:43 VM-0-6-tencentos kernel: br-cd57826e4de1: port 1(veth61ccaa6) entered disabled state 9月 20 16:24:43 VM-0-6-tencentos kernel: eth0: renamed from vetha4c7914 9月 20 16:24:43 VM-0-6-tencentos kernel: br-cd57826e4de1: port 1(veth61ccaa6) entered blocking state 9月 20 16:24:43 VM-0-6-tencentos kernel: br-cd57826e4de1: port 1(veth61ccaa6) entered forwarding state 9月 20 16:24:49 VM-0-6-tencentos kernel: FS-Cache: Loaded

根据补充的信息,可以基本排除 宿主机 OOMsystemd Memory 限制

当前结论调整:

cubelet 进程不是因为内存不足被杀,而是 cubelet 自己的启动流程里主动退出/被父脚本处理,或者触发了某个运行时检查


关键证据

1. 宿主机资源完全正常

你的机器:

复制代码
Mem: 125Gi
used: 9.7Gi
available:115Gi
Swap:0

内存非常充足

同时:

复制代码
dmesg | grep -Ei "oom|killed"

无输出

排除:

  • ❌ kernel OOM killer
  • ❌ docker OOM
  • ❌ 内存不足

2. 当前 cubelet 实际运行正常

现在:

复制代码
ps aux | grep cubelet

看到:

复制代码
root 1949
/usr/local/services/cubetoolbox/Cubelet/bin/cubelet

状态:

复制代码
Sl

说明:

  • S = sleeping
  • l = multi-thread

进程正常


3. 真正异常是这个:

启动:

复制代码
cubelet initial pid=1710

马上:

复制代码
1710 Killed

然后:

复制代码
cubelet stable pid=1949

也就是说:

启动脚本:

复制代码
cubelet-start.sh

逻辑类似:

复制代码
start cubelet &
wait

if alive
  use it
else
  restart
fi

第一次 pid 1710 被杀,然后重新拉起 1949

所以不是服务失败


下一步重点查 cubelet-start.sh

执行:

复制代码
sed -n '1,150p' \
/usr/local/services/cubetoolbox/scripts/systemd/cubelet-start.sh

重点看:

第57行附近

你的日志:

复制代码
cubelet-start.sh: line 57:
1710 Killed

需要看:

复制代码
nl -ba \
/usr/local/services/cubetoolbox/scripts/systemd/cubelet-start.sh \
| sed -n '45,70p'

第二个重点:为什么第一次启动被杀?

目前可能性:

可能1:cubelet 自带 watchdog/reexec

很多云沙箱组件启动:

复制代码
bootstrap process
       |
       |
       +--> real cubelet

第一次 pid:

复制代码
1710

只是 bootstrap

然后:

复制代码
exec/reload

换 pid:

复制代码
1949

脚本误判为 killed


可能2:动态配置 reload

日志:

复制代码
listen file[
/usr/local/services/cubetoolbox/Cubelet/dynamicconf/conf.yaml
]

说明 cubelet 有动态配置监听

启动流程:

复制代码
cubelet
 |
加载config
 |
读取dynamicconf
 |
重新fork
 |
旧pid退出
 |
新pid接管

也可能显示:

复制代码
Killed

但实际是正常行为


3. 检查 cubelet 健康接口

先确认:

复制代码
ss -lntp | grep cubelet

或者:

复制代码
netstat -lntp | grep 1949

看监听端口


4. 查看 cubelet 日志目录

找日志:

复制代码
find /usr/local/services/cubetoolbox/Cubelet \
-name "*.log" \
-o -name "*.out"

重点:

复制代码
ls -lh /usr/local/services/cubetoolbox/Cubelet/log/

5. 检查是否有 sandbox runtime

你的 docker:

没有看到:

复制代码
qemu
builder
template

只有:

复制代码
cube-proxy
cube-egress
cube-webui
cube-lifecycle-manager
redis
mysql
minio
coredns

说明:

目前只是 control plane

还没有创建 sandbox

执行:

复制代码
docker ps -a | grep -Ei "qemu|sandbox|builder|template"

如果为空:

正常


6. lifecycle-manager 状态正常

这一块不用处理:

复制代码
bootstrap complete entries=0

proxy joined

replay done

说明:

复制代码
cube-master
cube-proxy
lifecycle-manager
redis

通信正常


我建议现在执行这几个命令

按顺序:

① 看启动脚本第57行

复制代码
nl -ba \
/usr/local/services/cubetoolbox/scripts/systemd/cubelet-start.sh \
| sed -n '45,70p'

② 看 cubelet 日志

复制代码
find /usr/local/services/cubetoolbox/Cubelet \
-type f | grep -Ei "log|err"

③ 看端口

复制代码
ss -lntp | grep cube

④ 看 cubelet完整启动参数

复制代码
cat /proc/1949/cmdline | tr '\0' ' '

目前判断:

项目 状态
宿主机内存 ✅正常
磁盘 ✅正常
Docker组件 ✅正常
Lifecycle Manager ✅正常
Cubelet最终进程 ✅正常
首次pid killed ⚠️启动脚本/自重启机制

下一步主要确认 cubelet-start.sh 第57行到底是什么逻辑

nl -ba \ /usr/local/services/cubetoolbox/scripts/systemd/cubelet-start.sh \ | sed -n '45,70p' 45 46 if \[ -n "${CUBE_SANDBOX_NODE_IP:-}" ]; then 47 ( 48 export CUBE_SANDBOX_NODE_IP="{CUBE_SANDBOX_NODE_IP}" 49 exec "{CUBELET_BIN}" --config "{CUBELET_CONFIG}" --dynamic-conf-path "{CUBELET_DYNAMICCONF}" 50 ) & 51 else 52 "{CUBELET_BIN}" --config "{CUBELET_CONFIG}" --dynamic-conf-path "{CUBELET_DYNAMICCONF}" \& 53 fi 54 initial_pid=! 55 log "cubelet initial pid={initial_pid}" 56 57 for _ in {1..20}; do 58 if ! kill -0 "{initial_pid}" >/dev/null 2>&1; then 59 break 60 fi 61 sleep 1 62 done 63 64 refresh_pidfile_from_pattern "{PID_FILE}" "{PATTERN}" 20 1 || die "failed to determine cubelet child pid" 65 log "cubelet stable pid=(\<"{PID_FILE}")" root@VM-0-6-tencentos \~# find /usr/local/services/cubetoolbox/Cubelet \ -type f | grep -Ei "log|err" root@VM-0-6-tencentos \~# ss -lntp | grep cube LISTEN 0 4096 0.0.0.0:3010 0.0.0.0:* users:(("cubeops",pid=4673,fd=10)) LISTEN 0 4096 0.0.0.0:3000 0.0.0.0:* users:(("cube-api",pid=5903,fd=9)) LISTEN 0 4096 0.0.0.0:8083 0.0.0.0:* users:(("cube-lifecycle-",pid=5708,fd=9)) LISTEN 0 4096 0.0.0.0:8089 0.0.0.0:* users:(("cubemaster",pid=5538,fd=15)) LISTEN 0 4096 0.0.0.0:9999 0.0.0.0:* users:(("cubelet",pid=1949,fd=636)) LISTEN 0 4096 0.0.0.0:9998 0.0.0.0:* users:(("cubelet",pid=1949,fd=633)) LISTEN 0 4096 0.0.0.0:9966 0.0.0.0:* users:(("cubelet",pid=1949,fd=631)) root@VM-0-6-tencentos \~# cat /proc/1949/cmdline | tr '\0' ' ' /usr/local/services/cubetoolbox/Cubelet/bin/cubelet --config /usr/local/services/cubetoolbox/Cubelet/config/config.toml --dynamic-conf-path /usr/local/services/cubetoolbox/Cubelet/dynamicconf/conf.yaml

从目前信息看,cubelet 并没有真正异常退出,之前日志里的:

复制代码
cubelet-start.sh: line 57: 6023 Killed
cubelet-start.sh: line 57: 1668 Killed
cubelet-start.sh: line 57: 1710 Killed

不是 cubelet 服务最终 PID 被杀,而是 启动脚本 fork 出来的初始子进程被替换/退出。现在稳定运行:

复制代码
root 1949 0.5 0.1 4164304 221340 ? Sl 16:24 0:31
/usr/local/services/cubetoolbox/Cubelet/bin/cubelet

并且监听:

复制代码
0.0.0.0:9999 cubelet
0.0.0.0:9998 cubelet
0.0.0.0:9966 cubelet

说明 cubelet 已正常启动


1. 先解释为什么看到 Killed

你的脚本:

复制代码
if [[ -n "${CUBE_SANDBOX_NODE_IP:-}" ]]; then
(
 export CUBE_SANDBOX_NODE_IP="${CUBE_SANDBOX_NODE_IP}"
 exec "${CUBELET_BIN}" ...
) &
fi

initial_pid=$!

这里:

复制代码
6023
1668
1710

是 shell fork 出来的 wrapper pid

启动后:

复制代码
exec cubelet

替换进程。

然后脚本继续:

复制代码
refresh_pidfile_from_pattern

重新通过:

复制代码
PATTERN

找到真正 cubelet PID

所以:

复制代码
Killed 1710
stable pid=1949

实际过程:

复制代码
cubelet-start.sh
        |
        +-- bash subshell PID 1710
                |
                exec cubelet
                |
                PID变成cubelet

最终:

复制代码
PID 1949 cubelet

这个行为类似 Kubernetes kubelet wrapper


2. 但是有一个值得检查的问题

你的日志:

第一次:

复制代码
cubelet initial pid=6023

Killed 6023

cubelet stable pid=7079

第二次:

复制代码
cubelet initial pid=1668

Killed 1668

stable pid=2125

第三次:

复制代码
initial pid=1710

Killed 1710

stable pid=1949

每次都是:

启动后约 1 秒:

复制代码
Killed

说明 wrapper 检测机制可能有问题

建议检查:

复制代码
cat /usr/local/services/cubetoolbox/scripts/systemd/cubelet-start.sh

前面定义:

复制代码
PATTERN=

重点看:

复制代码
refresh_pidfile_from_pattern

匹配规则

执行:

复制代码
grep PATTERN -n \
/usr/local/services/cubetoolbox/scripts/systemd/cubelet-start.sh

3. 当前 cubelet 状态确认

执行:

复制代码
systemctl status cube-sandbox-cubelet

应该类似:

复制代码
Active: active (running)
Main PID: 1949

再看:

复制代码
cat /run/cube-sandbox-systemd/cubelet.pid

应该:

复制代码
1949

确认 systemd 管理的是正确 PID


4. 当前资源没有问题

你的机器:

复制代码
Mem:
125Gi total
9.7Gi used
115Gi available

Swap:
0

不存在 OOM

同时:

复制代码
dmesg -T | grep -Ei "oom|killed"

为空

所以排除:

  • 内存不足
  • kernel OOM killer
  • cgroup memory kill

5. cubelet 自己日志在哪里?

你找:

复制代码
find /usr/local/services/cubetoolbox/Cubelet \
-type f | grep -Ei "log|err"

没有输出

说明它大概率写 stdout

systemd 里查看:

复制代码
journalctl \
-u cube-sandbox-cubelet \
-f

然后:

另开窗口:

复制代码
curl http://127.0.0.1:9999

或者:

复制代码
curl http://127.0.0.1:9966

观察


6. 建议检查 cubelet health

你的端口:

端口 可能用途
9966 metrics
9998 RPC
9999 HTTP API

测试:

复制代码
curl -v http://127.0.0.1:9999/health

或者:

复制代码
curl -v http://127.0.0.1:9999/healthz

查看:

复制代码
curl -s http://127.0.0.1:9966/metrics | head

如果返回 Prometheus 指标:

说明完全正常


7. 我认为当前真正问题点

从你的日志:

复制代码
cube-lifecycle-manager
bootstrap complete
entries:0

proxy joined

replay done pushed:0 failed:0

控制面正常

Docker:

复制代码
cube-proxy
cube-webui
cube-egress
redis
mysql
minio

全部 healthy

cubelet:

复制代码
PID 1949
ports opened

正常

目前唯一异常:

复制代码
cubelet-start.sh wrapper pid killed

但不影响服务

cat /run/cube-sandbox-systemd/cubelet.pid systemctl status cube-sandbox-cubelet grep PATTERN -n \ /usr/local/services/cubetoolbox/scripts/systemd/cubelet-start.sh curl -s http://127.0.0.1:9966/metrics | head -20 curl -v http://127.0.0.1:9999/health 1949 ● cube-sandbox-cubelet.service - Cube Sandbox Cubelet Loaded: loaded (/etc/systemd/system/cube-sandbox-cubelet.service; disabled; preset: disabled) Active: active (running) since Sun 2026-09-20 16:24:36 CST; 1h 46min ago Process: 1505 ExecStartPre=/usr/bin/bash /usr/local/services/cubetoolbox/scripts/systemd/prepare-compute-role.sh (code=exited, status=0/SUCCESS) Process: 1580 ExecStart=/usr/bin/bash /usr/local/services/cubetoolbox/scripts/systemd/cubelet-start.sh (code=exited, status=0/SUCCESS) Main PID: 1949 (cubelet) Tasks: 0 (limit: 153562) Memory: 212.4M (peak: 231.0M) CPU: 10.815s CGroup: /system.slice/cube-sandbox-cubelet.service ‣ 1949 /usr/local/services/cubetoolbox/Cubelet/bin/cubelet --config /usr/local/services/cubetoolbox/Cubelet/config/config.toml --dynamic-conf-path /usr/local/services/cubetoolbox/Cubelet/dynamicconf/conf.yaml 9月 20 16:24:35 VM-0-6-tencentos bash1608: one-click-systemd updated cubelet cubeops_addr=http://127.0.0.1:3010 9月 20 16:24:35 VM-0-6-tencentos bash1580: one-click-systemd updated cubelet cow.s3 enable=false 9月 20 16:24:35 VM-0-6-tencentos bash1580: one-click-systemd cubelet initial pid=1710 9月 20 16:24:35 VM-0-6-tencentos bash1949: 2026-09-20T16:24:35.552388365+08:00,listen file/usr/local/services/cubetoolbox/Cubelet/dynamicconf/conf.yaml 9月 20 16:24:35 VM-0-6-tencentos bash1949: cfg:{"Common":{"CommonTimeout":10000000000,"LogLevel":"","DescribeAsyncInterval":100000000,"EnablePFMode":false,"DescribeBDFInterval":10000000,"DescribeBDFTimeout":10000000000,"GetBDFByUuidCmd":"/usr/local/services/AdamPlugins-1.0/snhost_snic_sdk/cath/bm/tools/get_bdf_by_uuid","GetBDFByIfNameCmd":"/usr/local/services> 9月 20 16:24:35 VM-0-6-tencentos bash1949: 2026/09/20 16:24:35 mounted state tmpfs /data/cubelet/state size=1024m 9月 20 16:24:35 VM-0-6-tencentos bash1949: 2026/09/20 16:24:35 set /data/cubelet/root as shared mount 9月 20 16:24:36 VM-0-6-tencentos bash1580: /usr/local/services/cubetoolbox/scripts/systemd/cubelet-start.sh: line 57: 1710 Killed ( export CUBE_SANDBOX_NODE_IP="{CUBE_SANDBOX_NODE_IP}"; exec "{CUBELET_BIN}" --config "{CUBELET_CONFIG}" --dynamic-conf-path "{CUBELET_DYNAMICCONF}" ) 9月 20 16:24:36 VM-0-6-tencentos bash1580: one-click-systemd cubelet stable pid=1949 9月 20 16:24:36 VM-0-6-tencentos systemd1: Started cube-sandbox-cubelet.service - Cube Sandbox Cubelet. 19:PATTERN="^{CUBELET_BIN} --config" 39: if \[\[ -n "{existing_pid}" ]] && kill -0 "{existing_pid}" \>/dev/null 2\>\&1 \&\& pid_matches_pattern "{existing_pid}" "{PATTERN}"; then 64:refresh_pidfile_from_pattern "{PID_FILE}" "{PATTERN}" 20 1 \|\| die "failed to determine cubelet child pid" 404 page not found \* Trying 127.0.0.1:9999... \* Connected to 127.0.0.1 (127.0.0.1) port 9999 \> GET /health HTTP/1.1 \> Host: 127.0.0.1:9999 \> User-Agent: curl/8.4.0 \> Accept: \*/\* \> \* Received HTTP/0.9 when not allowed \* Closing connection curl: (1) Received HTTP/0.9 when not allowed \[root@VM-0-6-tencentos \~\]# journalctl -u cube-sandbox-cubelet -f docker logs cube-proxy docker logs cube-lifecycle-manager 9月 20 16:24:35 VM-0-6-tencentos bash\[1608\]: \[one-click-systemd\] updated cubelet cubeops_addr=http://127.0.0.1:3010 9月 20 16:24:35 VM-0-6-tencentos bash\[1580\]: \[one-click-systemd\] updated cubelet cow.s3 enable=false 9月 20 16:24:35 VM-0-6-tencentos bash\[1580\]: \[one-click-systemd\] cubelet initial pid=1710 9月 20 16:24:35 VM-0-6-tencentos bash\[1949\]: 2026-09-20T16:24:35.552388365+08:00,listen file\[/usr/local/services/cubetoolbox/Cubelet/dynamicconf/conf.yaml\] 9月 20 16:24:35 VM-0-6-tencentos bash\[1949\]: cfg:{"Common":{"CommonTimeout":10000000000,"LogLevel":"","DescribeAsyncInterval":100000000,"EnablePFMode":false,"DescribeBDFInterval":10000000,"DescribeBDFTimeout":10000000000,"GetBDFByUuidCmd":"/usr/local/services/AdamPlugins-1.0/snhost_snic_sdk/cath/bm/tools/get_bdf_by_uuid","GetBDFByIfNameCmd":"/usr/local/services/AdamPlugins-1.0/snhost_snic_sdk/cath/bm/tools/get_bdf_by_ifname","CommandTimeout":1000000000,"DisableHostCgroup":true,"DisableVmCgroup":false,"EnableSandboxExecCmdBeforeExist":false,"SandboxExecCmdTimeOut":5000000000,"SandboxExecCmdBeforeExist":\["sh","/usr/local/cubetools/cube_exec_cmd_before_exist.sh"\],"SandboxExecCmdBeforeExistLogOut":false,"SandboxExecCmdOutMaxLines":2000,"SandboxExecCmdMatchLine":"","SandboxExecCmdAfterMatch":null,"SandboxExecCmdBase64AfterMatch":"","SandboxExecCmdAfterMatchLogOut":false,"CgroupDisableMemoryReparentFile":"","CgroupDisableCpusetList":"","DisableHostNetfile":true,"DefaultDNSServers":\["119.29.29.29"\],"DefaultDNSSearches":\[\],"DefaultDNSOptions":\[\],"ReconcileInterval":300000000000,"DisableCubeBoxTemplateBaseFormatPoolOfNumberVer":false},"Tenant":{"Tenants":{},"UseRespectiveNamespace":false,"DefaultNS":"default","ZiyanUinListStr":""},"MetaServerConfig":{"MetaServerEndpoint":"127.0.0.1:3010","NodeStatusMaxImages":40000,"CubeMasterHTTPAddr":"127.0.0.1:8089"},"HostConf":{"Quota":{"Cpu":0,"Mem":"","MvmLimit":0,"CreationConcurrentNum":0,"PausedResourceReleaseRatio":0},"GC":{"CodeExpirationTime":"72h","ImageExpirationTime":"24h"},"SchedulerLabel":"default-cluster"}} 9月 20 16:24:35 VM-0-6-tencentos bash\[1949\]: 2026/09/20 16:24:35 mounted state tmpfs /data/cubelet/state size=1024m 9月 20 16:24:35 VM-0-6-tencentos bash\[1949\]: 2026/09/20 16:24:35 set /data/cubelet/root as shared mount 9月 20 16:24:36 VM-0-6-tencentos bash\[1580\]: /usr/local/services/cubetoolbox/scripts/systemd/cubelet-start.sh: line 57: 1710 Killed ( export CUBE_SANDBOX_NODE_IP="{CUBE_SANDBOX_NODE_IP}"; exec "{CUBELET_BIN}" --config "{CUBELET_CONFIG}" --dynamic-conf-path "${CUBELET_DYNAMICCONF}" ) 9月 20 16:24:36 VM-0-6-tencentos bash1580: one-click-systemd cubelet stable pid=1949 9月 20 16:24:36 VM-0-6-tencentos systemd1: Started cube-sandbox-cubelet.service - Cube Sandbox Cubelet. ^C {"level":"info","ts":1789892682.4259038,"caller":"cube-lifecycle-manager/main.go:59","msg":"cube-lifecycle-manager starting","redis_addr":"127.0.0.1:6379","cube_proxy_admin_urls":"http://127.0.0.1:8082","cubemaster_url":"http://192.168.0.6:8089","listen_addr":"0.0.0.0:8083","instance_identity":"VM-0-6-tencentos","leader_election_enabled":false} {"level":"info","ts":1789892682.428625,"caller":"cube-lifecycle-manager/main.go:81","msg":"eventbus enabled"} {"level":"info","ts":1789892682.4323788,"caller":"cube-lifecycle-manager/main.go:641","msg":"bootstrap complete","entries":0} {"level":"info","ts":1789892682.4353082,"logger":"http","caller":"httpapi/server.go:86","msg":"cube-lifecycle-manager http server listening","addr":"0.0.0.0:8083"} {"level":"info","ts":1789892685.4343956,"logger":"discovery","caller":"discovery/redis.go:182","msg":"discovery: proxy joined","proxy_id":"192.168.0.6:8082","admin_url":"http://192.168.0.6:8082"} {"level":"info","ts":1789892685.434483,"logger":"replay","caller":"cube-lifecycle-manager/main.go:366","msg":"replay begin","proxy_id":"192.168.0.6:8082","admin_url":"http://192.168.0.6:8082","entries":0} {"level":"info","ts":1789892685.434507,"logger":"replay","caller":"cube-lifecycle-manager/main.go:394","msg":"replay done","proxy_id":"192.168.0.6:8082","pushed":0,"failed":0} root@VM-0-6-tencentos \~# ls -lah /data/cubelet ls -lah /data/cube-shim ls -lah /data/snapshot_pack 总计 4.0K drwxr-x--- 6 root root 127 9月20日 16:24 . drwxr-xr-x 10 root root 136 9月20日 15:36 .. drwxr-xr-x 3 root root 16 9月20日 15:36 cleanup srw-rw---- 1 root root 0 9月20日 16:24 cubelet.sock srw-rw---- 1 root root 0 9月20日 16:24 cubelet.sock.ttrpc srw-rw---- 1 root root 0 9月20日 16:24 cubetap.sock drwxr-x--- 13 root root 4.0K 9月20日 15:36 root drwx--x--x 2 root root 6 9月20日 15:36 state drwxr-xr-x 4 root root 55 9月20日 15:36 storage 总计 0 drwxr-x--- 3 root root 19 9月20日 15:36 . drwxr-xr-x 10 root root 136 9月20日 15:36 .. drwxr-x--- 2 root root 6 9月20日 15:36 disks 总计 0 drwxr-x--- 3 root root 19 9月20日 15:36 . drwxr-xr-x 10 root root 136 9月20日 15:36 .. drwxr-x--- 2 root root 6 9月20日 15:36 disks

根据补充的结果,Cube Sandbox v0.7.1 控制面和 cubelet 本身已经正常运行。之前怀疑的 OOM、磁盘、systemd PID 管理问题基本可以排除

下面是详细判断


1. cubelet PID 管理正常

现在:

复制代码
cat /run/cube-sandbox-systemd/cubelet.pid

结果:

复制代码
1949

systemd:

复制代码
Main PID: 1949 (cubelet)

完全一致

说明:

复制代码
cubelet-start.sh
        |
        +-- 临时启动PID 1710
        |
        +-- exec cubelet
        |
        +-- 最终PID 1949
        |
        +-- 写入pidfile

这个流程正常


2. Killed 不是异常

关键:

复制代码
cubelet-start.sh: line 57: 1710 Killed
cubelet stable pid=1949

这里的 1710:

复制代码
(
 export CUBE_SANDBOX_NODE_IP
 exec cubelet
) &

产生的 subshell PID

后续:

复制代码
refresh_pidfile_from_pattern

找到:

复制代码
/usr/local/services/cubetoolbox/Cubelet/bin/cubelet --config

真正 PID:

复制代码
1949

所以不用处理


3. PATTERN 匹配也正确

你的:

复制代码
19:PATTERN="^${CUBELET_BIN} --config"

当前:

复制代码
cat /proc/1949/cmdline | tr '\0' ' '

输出:

复制代码
/usr/local/services/cubetoolbox/Cubelet/bin/cubelet --config ...

匹配:

复制代码
^cubelet --config

OK


4. cubelet API 判断

你执行:

复制代码
curl -s http://127.0.0.1:9966/metrics

返回:

复制代码
404 page not found

说明:

9966 有 HTTP server

只是:

复制代码
/metrics

不是暴露路径

不是故障


9999:

复制代码
Received HTTP/0.9 when not allowed

这个非常关键

说明:

9999 不是普通 HTTP/1.1 REST

可能是:

  • gRPC
  • ttrpc
  • 自定义 TCP protocol

所以:

复制代码
curl http://127.0.0.1:9999/health

方式不适用


5. 当前目录状态正常

你的:

复制代码
/data/cubelet

里面:

复制代码
cubelet.sock
cubelet.sock.ttrpc
cubetap.sock
root/
state/
storage/

说明:

cubelet runtime 初始化成功

特别:

复制代码
cubelet.sock.ttrpc

说明它已经进入 sandbox runtime 阶段


现在真正需要排查的是:为什么没有 Sandbox / Template

因为:

控制面:

复制代码
cubeops      3010
cube-api     3000
cubemaster   8089
lifecycle    8083
proxy        8082

全部起来

但是:

复制代码
/data/snapshot_pack/disks

为空:

复制代码
drwxr-x--- 2 root root 6 disks

说明:

目前还没有:

  • template
  • snapshot
  • image
  • qemu disk

下一步重点检查

1. 看 cubelet 注册状态

执行:

复制代码
curl -s http://127.0.0.1:3010/api/v1/nodes | jq

如果不知道 API:

先:

复制代码
ss -lntp | grep 3010

确认 cubeops


2. 查看 cubelet RPC状态

找端口:

复制代码
ss -lntp | grep cubelet

已有:

复制代码
9966
9998
9999

尝试:

复制代码
curl -v http://127.0.0.1:9966/

和:

复制代码
curl -v http://127.0.0.1:9998/

看看返回


3. 检查 qemu / shim

你的:

复制代码
/data/cube-shim/disks

为空

检查:

复制代码
ls -lah /data/cube-shim

以及:

复制代码
find /usr/local/services/cubetoolbox \
-type f \
-name "*qemu*"

4. 看 cube-proxy 日志

你只看了 lifecycle

执行:

复制代码
docker logs --tail=200 cube-proxy

重点找:

复制代码
error
failed
template
sandbox
create
node
cubelet

5. 看 cubemaster

目前没有看

执行:

复制代码
journalctl -u cube-sandbox-cubemaster -n 200 --no-pager

或者:

复制代码
ps aux | grep cubemaster

6. 检查模板目录

执行:

复制代码
find /data -maxdepth 3 \
-type d \
| grep -Ei "template|image|snapshot|box"

正常安装后通常应该有类似:

复制代码
template/
snapshot/
image/
rootfs/

当前状态总结

组件 状态
cube-api
cubeops
cubemaster
cube-proxy
cube-lifecycle-manager
redis
mysql
minio
cubelet
OOM ❌ 排除
磁盘压力 ❌ 排除
cubelet启动失败 ❌ 排除
template/image ⚠️ 未发现

root@VM-0-6-tencentos \~# docker logs --tail=200 cube-proxy root@VM-0-6-tencentos \~# ps aux | grep cubemaster root 5538 1.6 0.0 3566552 42072 ? Ssl 16:24 2:01 /usr/local/services/cubetoolbox/CubeMaster/bin/cubemaster root 214146 0.0 0.0 6696 2060 pts/0 S+ 18:24 0:00 grep --color=auto cubemaster root@VM-0-6-tencentos \~# journalctl -u cube-sandbox-cubemaster -n 200 --no-pager 9月 20 17:54:12 VM-0-6-tencentos bash5538: 2026/09/20 17:54:12 /workspace/CubeMaster/pkg/templatecenter/store.go:1118 record not found 9月 20 17:54:12 VM-0-6-tencentos bash5538: 0.332ms rows:0 SELECT * FROM t_cube_template_definition WHERE template_id = 'tpl-464f1b5538764bd7afa240ca' AND t_cube_template_definition.deleted_at IS NULL ORDER BY t_cube_template_definition.id LIMIT 1 9月 20 17:54:12 VM-0-6-tencentos bash5538: 9月 20 17:54:12 VM-0-6-tencentos bash5538: 2026/09/20 17:54:12 /workspace/CubeMaster/pkg/templatecenter/snapshot_table.go:29 record not found 9月 20 17:54:12 VM-0-6-tencentos bash5538: 0.502ms rows:0 SELECT * FROM t_cube_snapshot WHERE snapshot_id = 'tpl-464f1b5538764bd7afa240ca' AND t_cube_snapshot.deleted_at IS NULL ORDER BY t_cube_snapshot.id LIMIT 1 9月 20 17:54:42 VM-0-6-tencentos bash5538: 9月 20 17:54:42 VM-0-6-tencentos bash5538: 2026/09/20 17:54:42 /workspace/CubeMaster/pkg/templatecenter/store.go:1118 record not found 9月 20 17:54:42 VM-0-6-tencentos bash5538: 0.362ms rows:0 SELECT * FROM t_cube_template_definition WHERE template_id = 'tpl-464f1b5538764bd7afa240ca' AND t_cube_template_definition.deleted_at IS NULL ORDER BY t_cube_template_definition.id LIMIT 1 9月 20 17:54:42 VM-0-6-tencentos bash5538: 9月 20 17:54:42 VM-0-6-tencentos bash5538: 2026/09/20 17:54:42 /workspace/CubeMaster/pkg/templatecenter/snapshot_table.go:29 record not found 9月 20 17:54:42 VM-0-6-tencentos bash5538: 0.419ms rows:0 SELECT * FROM t_cube_snapshot WHERE snapshot_id = 'tpl-464f1b5538764bd7afa240ca' AND t_cube_snapshot.deleted_at IS NULL ORDER BY t_cube_snapshot.id LIMIT 1 9月 20 17:55:12 VM-0-6-tencentos bash5538: 9月 20 17:55:12 VM-0-6-tencentos bash5538: 2026/09/20 17:55:12 /workspace/CubeMaster/pkg/templatecenter/store.go:1118 record not found 9月 20 17:55:12 VM-0-6-tencentos bash5538: 0.421ms rows:0 SELECT * FROM t_cube_template_definition WHERE template_id = 'tpl-464f1b5538764bd7afa240ca' AND t_cube_template_definition.deleted_at IS NULL ORDER BY t_cube_template_definition.id LIMIT 1 9月 20 17:55:12 VM-0-6-tencentos bash5538: 9月 20 17:55:12 VM-0-6-tencentos bash5538: 2026/09/20 17:55:12 /workspace/CubeMaster/pkg/templatecenter/snapshot_table.go:29 record not found 9月 20 17:55:12 VM-0-6-tencentos bash5538: 0.374ms rows:0 SELECT * FROM t_cube_snapshot WHERE snapshot_id = 'tpl-464f1b5538764bd7afa240ca' AND t_cube_snapshot.deleted_at IS NULL ORDER BY t_cube_snapshot.id LIMIT 1 9月 20 17:55:42 VM-0-6-tencentos bash5538: 9月 20 17:55:42 VM-0-6-tencentos bash5538: 2026/09/20 17:55:42 /workspace/CubeMaster/pkg/templatecenter/store.go:1118 record not found 9月 20 17:55:42 VM-0-6-tencentos bash5538: 0.376ms rows:0 SELECT * FROM t_cube_template_definition WHERE template_id = 'tpl-464f1b5538764bd7afa240ca' AND t_cube_template_definition.deleted_at IS NULL ORDER BY t_cube_template_definition.id LIMIT 1 9月 20 17:55:42 VM-0-6-tencentos bash5538: 9月 20 17:55:42 VM-0-6-tencentos bash5538: 2026/09/20 17:55:42 /workspace/CubeMaster/pkg/templatecenter/snapshot_table.go:29 record not found 9月 20 17:55:42 VM-0-6-tencentos bash5538: 0.395ms rows:0 SELECT * FROM t_cube_snapshot WHERE snapshot_id = 'tpl-464f1b5538764bd7afa240ca' AND t_cube_snapshot.deleted_at IS NULL ORDER BY t_cube_snapshot.id LIMIT 1 9月 20 17:56:12 VM-0-6-tencentos bash5538: 9月 20 17:56:12 VM-0-6-tencentos bash5538: 2026/09/20 17:56:12 /workspace/CubeMaster/pkg/templatecenter/store.go:1118 record not found 9月 20 17:56:12 VM-0-6-tencentos bash5538: 0.358ms rows:0 SELECT * FROM t_cube_template_definition WHERE template_id = 'tpl-464f1b5538764bd7afa240ca' AND t_cube_template_definition.deleted_at IS NULL ORDER BY t_cube_template_definition.id LIMIT 1 9月 20 17:56:12 VM-0-6-tencentos bash5538: 9月 20 17:56:12 VM-0-6-tencentos bash5538: 2026/09/20 17:56:12 /workspace/CubeMaster/pkg/templatecenter/snapshot_table.go:29 record not found 9月 20 17:56:12 VM-0-6-tencentos bash5538: 0.401ms rows:0 SELECT * FROM t_cube_snapshot WHERE snapshot_id = 'tpl-464f1b5538764bd7afa240ca' AND t_cube_snapshot.deleted_at IS NULL ORDER BY t_cube_snapshot.id LIMIT 1 9月 20 17:56:42 VM-0-6-tencentos bash5538: 9月 20 17:56:42 VM-0-6-tencentos bash5538: 2026/09/20 17:56:42 /workspace/CubeMaster/pkg/templatecenter/store.go:1118 record not found 9月 20 17:56:42 VM-0-6-tencentos bash5538: 0.384ms rows:0 SELECT * FROM t_cube_template_definition WHERE template_id = 'tpl-464f1b5538764bd7afa240ca' AND t_cube_template_definition.deleted_at IS NULL ORDER BY t_cube_template_definition.id LIMIT 1 9月 20 17:56:42 VM-0-6-tencentos bash5538: 9月 20 17:56:42 VM-0-6-tencentos bash5538: 2026/09/20 17:56:42 /workspace/CubeMaster/pkg/templatecenter/snapshot_table.go:29 record not found 9月 20 17:56:42 VM-0-6-tencentos bash5538: 0.406ms rows:0 SELECT * FROM t_cube_snapshot WHERE snapshot_id = 'tpl-464f1b5538764bd7afa240ca' AND t_cube_snapshot.deleted_at IS NULL ORDER BY t_cube_snapshot.id LIMIT 1 9月 20 17:57:12 VM-0-6-tencentos bash5538: 9月 20 17:57:12 VM-0-6-tencentos bash5538: 2026/09/20 17:57:12 /workspace/CubeMaster/pkg/templatecenter/store.go:1118 record not found 9月 20 17:57:12 VM-0-6-tencentos bash5538: 0.418ms rows:0 SELECT * FROM t_cube_template_definition WHERE template_id = 'tpl-464f1b5538764bd7afa240ca' AND t_cube_template_definition.deleted_at IS NULL ORDER BY t_cube_template_definition.id LIMIT 1 9月 20 17:57:12 VM-0-6-tencentos bash5538: 9月 20 17:57:12 VM-0-6-tencentos bash5538: 2026/09/20 17:57:12 /workspace/CubeMaster/pkg/templatecenter/snapshot_table.go:29 record not found 9月 20 17:57:12 VM-0-6-tencentos bash5538: 0.376ms rows:0 SELECT * FROM t_cube_snapshot WHERE snapshot_id = 'tpl-464f1b5538764bd7afa240ca' AND t_cube_snapshot.deleted_at IS NULL ORDER BY t_cube_snapshot.id LIMIT 1 9月 20 17:57:42 VM-0-6-tencentos bash5538: 9月 20 17:57:42 VM-0-6-tencentos bash5538: 2026/09/20 17:57:42 /workspace/CubeMaster/pkg/templatecenter/store.go:1118 record not found 9月 20 17:57:42 VM-0-6-tencentos bash5538: 0.346ms rows:0 SELECT * FROM t_cube_template_definition WHERE template_id = 'tpl-464f1b5538764bd7afa240ca' AND t_cube_template_definition.deleted_at IS NULL ORDER BY t_cube_template_definition.id LIMIT 1 9月 20 17:57:42 VM-0-6-tencentos bash5538: 9月 20 17:57:42 VM-0-6-tencentos bash5538: 2026/09/20 17:57:42 /workspace/CubeMaster/pkg/templatecenter/snapshot_table.go:29 record not found 9月 20 17:57:42 VM-0-6-tencentos bash5538: 0.351ms rows:0 SELECT * FROM t_cube_snapshot WHERE snapshot_id = 'tpl-464f1b5538764bd7afa240ca' AND t_cube_snapshot.deleted_at IS NULL ORDER BY t_cube_snapshot.id LIMIT 1 9月 20 17:58:12 VM-0-6-tencentos bash5538: 9月 20 17:58:12 VM-0-6-tencentos bash5538: 2026/09/20 17:58:12 /workspace/CubeMaster/pkg/templatecenter/store.go:1118 record not found 9月 20 17:58:12 VM-0-6-tencentos bash5538: 0.360ms rows:0 SELECT * FROM t_cube_template_definition WHERE template_id = 'tpl-464f1b5538764bd7afa240ca' AND t_cube_template_definition.deleted_at IS NULL ORDER BY t_cube_template_definition.id LIMIT 1 9月 20 17:58:12 VM-0-6-tencentos bash5538: 9月 20 17:58:12 VM-0-6-tencentos bash5538: 2026/09/20 17:58:12 /workspace/CubeMaster/pkg/templatecenter/snapshot_table.go:29 record not found 9月 20 17:58:12 VM-0-6-tencentos bash5538: 0.352ms rows:0 SELECT * FROM t_cube_snapshot WHERE snapshot_id = 'tpl-464f1b5538764bd7afa240ca' AND t_cube_snapshot.deleted_at IS NULL ORDER BY t_cube_snapshot.id LIMIT 1 9月 20 17:58:42 VM-0-6-tencentos bash5538: 9月 20 17:58:42 VM-0-6-tencentos bash5538: 2026/09/20 17:58:42 /workspace/CubeMaster/pkg/templatecenter/store.go:1118 record not found 9月 20 17:58:42 VM-0-6-tencentos bash5538: 0.379ms rows:0 SELECT * FROM t_cube_template_definition WHERE template_id = 'tpl-464f1b5538764bd7afa240ca' AND t_cube_template_definition.deleted_at IS NULL ORDER BY t_cube_template_definition.id LIMIT 1 9月 20 17:58:42 VM-0-6-tencentos bash5538: 9月 20 17:58:42 VM-0-6-tencentos bash5538: 2026/09/20 17:58:42 /workspace/CubeMaster/pkg/templatecenter/snapshot_table.go:29 record not found 9月 20 17:58:42 VM-0-6-tencentos bash5538: 0.342ms rows:0 SELECT * FROM t_cube_snapshot WHERE snapshot_id = 'tpl-464f1b5538764bd7afa240ca' AND t_cube_snapshot.deleted_at IS NULL ORDER BY t_cube_snapshot.id LIMIT 1 9月 20 17:59:12 VM-0-6-tencentos bash5538: 9月 20 17:59:12 VM-0-6-tencentos bash5538: 2026/09/20 17:59:12 /workspace/CubeMaster/pkg/templatecenter/store.go:1118 record not found 9月 20 17:59:12 VM-0-6-tencentos bash5538: 0.332ms rows:0 SELECT * FROM t_cube_template_definition WHERE template_id = 'tpl-464f1b5538764bd7afa240ca' AND t_cube_template_definition.deleted_at IS NULL ORDER BY t_cube_template_definition.id LIMIT 1 9月 20 17:59:12 VM-0-6-tencentos bash5538: 9月 20 17:59:12 VM-0-6-tencentos bash5538: 2026/09/20 17:59:12 /workspace/CubeMaster/pkg/templatecenter/snapshot_table.go:29 record not found 9月 20 17:59:12 VM-0-6-tencentos bash5538: 0.341ms rows:0 SELECT * FROM t_cube_snapshot WHERE snapshot_id = 'tpl-464f1b5538764bd7afa240ca' AND t_cube_snapshot.deleted_at IS NULL ORDER BY t_cube_snapshot.id LIMIT 1 9月 20 17:59:42 VM-0-6-tencentos bash5538: 9月 20 17:59:42 VM-0-6-tencentos bash5538: 2026/09/20 17:59:42 /workspace/CubeMaster/pkg/templatecenter/store.go:1118 record not found 9月 20 17:59:42 VM-0-6-tencentos bash5538: 0.394ms rows:0 SELECT * FROM t_cube_template_definition WHERE template_id = 'tpl-464f1b5538764bd7afa240ca' AND t_cube_template_definition.deleted_at IS NULL ORDER BY t_cube_template_definition.id LIMIT 1 9月 20 17:59:42 VM-0-6-tencentos bash5538: 9月 20 17:59:42 VM-0-6-tencentos bash5538: 2026/09/20 17:59:42 /workspace/CubeMaster/pkg/templatecenter/snapshot_table.go:29 record not found 9月 20 17:59:42 VM-0-6-tencentos bash5538: 0.366ms rows:0 SELECT * FROM t_cube_snapshot WHERE snapshot_id = 'tpl-464f1b5538764bd7afa240ca' AND t_cube_snapshot.deleted_at IS NULL ORDER BY t_cube_snapshot.id LIMIT 1 9月 20 18:00:12 VM-0-6-tencentos bash5538: 9月 20 18:00:12 VM-0-6-tencentos bash5538: 2026/09/20 18:00:12 /workspace/CubeMaster/pkg/templatecenter/store.go:1118 record not found 9月 20 18:00:12 VM-0-6-tencentos bash5538: 0.400ms rows:0 SELECT * FROM t_cube_template_definition WHERE template_id = 'tpl-464f1b5538764bd7afa240ca' AND t_cube_template_definition.deleted_at IS NULL ORDER BY t_cube_template_definition.id LIMIT 1 9月 20 18:00:12 VM-0-6-tencentos bash5538: 9月 20 18:00:12 VM-0-6-tencentos bash5538: 2026/09/20 18:00:12 /workspace/CubeMaster/pkg/templatecenter/snapshot_table.go:29 record not found 9月 20 18:00:12 VM-0-6-tencentos bash5538: 0.385ms rows:0 SELECT * FROM t_cube_snapshot WHERE snapshot_id = 'tpl-464f1b5538764bd7afa240ca' AND t_cube_snapshot.deleted_at IS NULL ORDER BY t_cube_snapshot.id LIMIT 1 9月 20 18:00:42 VM-0-6-tencentos bash5538: 9月 20 18:00:42 VM-0-6-tencentos bash5538: 2026/09/20 18:00:42 /workspace/CubeMaster/pkg/templatecenter/store.go:1118 record not found 9月 20 18:00:42 VM-0-6-tencentos bash5538: 0.382ms rows:0 SELECT * FROM t_cube_template_definition WHERE template_id = 'tpl-464f1b5538764bd7afa240ca' AND t_cube_template_definition.deleted_at IS NULL ORDER BY t_cube_template_definition.id LIMIT 1 9月 20 18:00:42 VM-0-6-tencentos bash5538: 9月 20 18:00:42 VM-0-6-tencentos bash5538: 2026/09/20 18:00:42 /workspace/CubeMaster/pkg/templatecenter/snapshot_table.go:29 record not found 9月 20 18:00:42 VM-0-6-tencentos bash5538: 0.407ms rows:0 SELECT * FROM t_cube_snapshot WHERE snapshot_id = 'tpl-464f1b5538764bd7afa240ca' AND t_cube_snapshot.deleted_at IS NULL ORDER BY t_cube_snapshot.id LIMIT 1 9月 20 18:01:12 VM-0-6-tencentos bash5538: 9月 20 18:01:12 VM-0-6-tencentos bash5538: 2026/09/20 18:01:12 /workspace/CubeMaster/pkg/templatecenter/store.go:1118 record not found 9月 20 18:01:12 VM-0-6-tencentos bash5538: 0.360ms rows:0 SELECT * FROM t_cube_template_definition WHERE template_id = 'tpl-464f1b5538764bd7afa240ca' AND t_cube_template_definition.deleted_at IS NULL ORDER BY t_cube_template_definition.id LIMIT 1 9月 20 18:01:12 VM-0-6-tencentos bash5538: 9月 20 18:01:12 VM-0-6-tencentos bash5538: 2026/09/20 18:01:12 /workspace/CubeMaster/pkg/templatecenter/snapshot_table.go:29 record not found 9月 20 18:01:12 VM-0-6-tencentos bash5538: 0.417ms rows:0 SELECT * FROM t_cube_snapshot WHERE snapshot_id = 'tpl-464f1b5538764bd7afa240ca' AND t_cube_snapshot.deleted_at IS NULL ORDER BY t_cube_snapshot.id LIMIT 1 9月 20 18:01:42 VM-0-6-tencentos bash5538: 9月 20 18:01:42 VM-0-6-tencentos bash5538: 2026/09/20 18:01:42 /workspace/CubeMaster/pkg/templatecenter/store.go:1118 record not found 9月 20 18:01:42 VM-0-6-tencentos bash5538: 0.389ms rows:0 SELECT * FROM t_cube_template_definition WHERE template_id = 'tpl-464f1b5538764bd7afa240ca' AND t_cube_template_definition.deleted_at IS NULL ORDER BY t_cube_template_definition.id LIMIT 1 9月 20 18:01:42 VM-0-6-tencentos bash5538: 9月 20 18:01:42 VM-0-6-tencentos bash5538: 2026/09/20 18:01:42 /workspace/CubeMaster/pkg/templatecenter/snapshot_table.go:29 record not found 9月 20 18:01:42 VM-0-6-tencentos bash5538: 0.374ms rows:0 SELECT * FROM t_cube_snapshot WHERE snapshot_id = 'tpl-464f1b5538764bd7afa240ca' AND t_cube_snapshot.deleted_at IS NULL ORDER BY t_cube_snapshot.id LIMIT 1 9月 20 18:02:12 VM-0-6-tencentos bash5538: 9月 20 18:02:12 VM-0-6-tencentos bash5538: 2026/09/20 18:02:12 /workspace/CubeMaster/pkg/templatecenter/store.go:1118 record not found 9月 20 18:02:12 VM-0-6-tencentos bash5538: 0.389ms rows:0 SELECT * FROM t_cube_template_definition WHERE template_id = 'tpl-464f1b5538764bd7afa240ca' AND t_cube_template_definition.deleted_at IS NULL ORDER BY t_cube_template_definition.id LIMIT 1

从目前日志看,Cube Sandbox 主链路已经基本起来了

  • cubelet 已正常运行
  • cubemaster 已正常运行
  • cube-lifecycle-manager 已正常运行
  • cubeops :3010
  • cube-api :3000
  • cube-proxy 可能无日志(需要确认)
  • ❌ 当前异常集中在 Template / Snapshot 数据缺失

重点不是 cubelet 启动问题,而是 模板中心状态不一致


1. cubelet 的 Killed 不是故障

这里:

复制代码
cubelet-start.sh: line 57:
1710 Killed

实际发生:

复制代码
initial pid=1710
...
stable pid=1949

你的脚本逻辑:

复制代码
( 
 export CUBE_SANDBOX_NODE_IP
 exec cubelet ...
) &

先启动一个探测进程:

复制代码
1710

然后:

复制代码
refresh_pidfile_from_pattern

找到真正 daemon:

复制代码
1949

所以:

复制代码
systemctl status
Main PID: 1949

正常


2. cubelet API 端口判断

你测试:

复制代码
curl http://127.0.0.1:9999/health

返回:

复制代码
HTTP/0.9

说明:

9999 不是 HTTP 服务

从监听:

复制代码
9966
9998
9999

看:

大概率:

端口 用途
9966 metrics
9998 ttrpc
9999 cubelet grpc/http2

所以:

复制代码
curl 9999

不适用

测试:

复制代码
curl http://127.0.0.1:9966/metrics | head

返回:

复制代码
404 page not found

说明:

9966 也不是标准 prometheus path

建议查看:

复制代码
cat /proc/1949/net/tcp

或者:

复制代码
strings /usr/local/services/cubetoolbox/Cubelet/bin/cubelet | grep -Ei "metric|health|debug"

3. 当前真正问题:Template 不存在

核心日志:

复制代码
SELECT * FROM t_cube_template_definition
WHERE template_id='tpl-464f1b5538764bd7afa240ca'

结果:

复制代码
rows:0

还有:

复制代码
SELECT * FROM t_cube_snapshot
WHERE snapshot_id='tpl-464f1b5538764bd7afa240ca'

结果:

复制代码
rows:0

说明:

CubeMaster 收到请求:

复制代码
template_id
tpl-464f1b5538764bd7afa240ca

但是数据库:

复制代码
t_cube_template_definition

没有记录


4. 检查 TemplateCenter

你有:

复制代码
/usr/local/services/cubetoolbox/CubeTemplateCenter

执行:

复制代码
systemctl status cube-sandbox-cube-templatecenter

然后:

复制代码
journalctl \
-u cube-sandbox-cube-templatecenter \
-n 200 \
--no-pager

重点看:

有没有:

复制代码
mysql connect failed
redis failed
register template failed

5. 查数据库里的模板

需要找到 CubeMaster 数据库

先找配置:

复制代码
grep -R "mysql\|database\|dsn" \
/usr/local/services/cubetoolbox \
-n

找到 DB:

例如:

复制代码
mysql://user:pass@tcp(...)

进入 mysql:

复制代码
mysql -h xxx -u xxx -p

查看:

复制代码
show tables;

重点:

复制代码
select count(*) 
from t_cube_template_definition;

查看模板:

复制代码
select 
 template_id,
 name,
 status
from t_cube_template_definition;

6. 检查默认镜像

你有:

复制代码
/usr/local/services/cubetoolbox/cube-image/

里面:

复制代码
cube-guest-image-cpu.img
version

但是:

复制代码
/data/snapshot_pack

为空:

复制代码
drwxr-xr-x snapshot_pack
└── disks

没有磁盘

说明:

目前只是安装了组件:

复制代码
Cubelet
CubeMaster
TemplateCenter

但是:

模板镜像没有导入


7. 导入模板流程需要确认

搜索脚本:

复制代码
find /usr/local/services/cubetoolbox \
-type f \
| grep -Ei "init|template|image|import|snapshot"

重点看:

复制代码
cubetemplatecenter-postcheck.sh

内容:

复制代码
cat \
/usr/local/services/cubetoolbox/scripts/systemd/cubetemplatecenter-postcheck.sh

8. 检查 TemplateCenter API

看看监听:

复制代码
ss -lntp | grep template

或者:

复制代码
docker ps

因为:

你现在:

复制代码
CubeTemplateCenter/bin/templatecenter

应该是 systemd

查看:

复制代码
systemctl status cube-sandbox-cube-templatecenter

9. 检查 cube-api 调用链

现在链路:

复制代码
cube-api
 |
 cube-master :8089
 |
 TemplateCenter
 |
 DB
 |
 image storage

你的状态:

复制代码
cube-api        OK
cubemaster      OK
cubelet         OK
TemplateCenter  未确认
DB              有访问
Template数据    缺失
image           缺失

10. 下一步建议执行这组命令

请贴输出:

复制代码
systemctl status cube-sandbox-cube-templatecenter

journalctl \
-u cube-sandbox-cube-templatecenter \
-n 100 \
--no-pager

grep -R "t_cube_template_definition" \
/usr/local/services/cubetoolbox \
-n

以及:

复制代码
ls -lh \
/usr/local/services/cubetoolbox/cube-image/

目前判断:

不是 Cubelet、不是 systemd、不是网络问题

根因概率最高:

Cube Sandbox 安装完成,但默认 Guest Template / Snapshot 没有初始化导入,导致 CubeMaster 周期性查询不存在的 tpl-* 资源
root@VM-0-6-tencentos \~# systemctl status cube-sandbox-cube-templatecenter ● cube-sandbox-cube-templatecenter.service - Cube Sandbox CubeTemplateCenter Loaded: loaded (/etc/systemd/system/cube-sandbox-cube-templatecenter.service; enabled; preset: disabled) Active: active (running) since Sun 2026-09-20 16:24:44 CST; 2h 1min ago Process: 5910 ExecStartPost=/usr/bin/bash /usr/local/services/cubetoolbox/scripts/systemd/cubetemplatecenter-postcheck.sh (code=exited, status=0/SUCCESS) Main PID: 5907 (templatecenter) Tasks: 28 (limit: 153562) Memory: 45.8M (peak: 48.5M) CPU: 9.938s CGroup: /system.slice/cube-sandbox-cube-templatecenter.service └─5907 /usr/local/services/cubetoolbox/CubeTemplateCenter/bin/templatecenter 9月 20 16:24:43 VM-0-6-tencentos systemd1: Starting cube-sandbox-cube-templatecenter.service - Cube Sandbox CubeTemplateCenter... 9月 20 16:24:43 VM-0-6-tencentos bash5907: 2026-09-20T16:24:43.375476072+08:00,listen file/usr/local/services/cubetoolbox/CubeTemplateCenter/conf.yaml 9月 20 16:24:43 VM-0-6-tencentos bash5907: cfg:{"Common":{"MockUpdateAction":false,"DebugDumpHttpBody":false,"MockDebug":false,"MockNodeNum":0,"MockCreateDirect":false,"MockCreateDirectHandle":false,"MockHttpDirect":false,"MockCreateSleep":0,"MockPercents":null,"CubeDestroyCheckFilter":false,"Debug":{"Address":""},"HttpPort":8090,"HttpBind":"127.0.0.1","Write> 9月 20 16:24:43 VM-0-6-tencentos bash5907: 2026/09/20 16:24:43 migrate: 1 applied migration version(s) have no fingerprint baseline and are NOT content-checked (e.g. applied before fingerprinting existed): 0 9月 20 16:24:43 VM-0-6-tencentos bash5907: {"LocalIp":"192.168.0.6","Module":"","@timestamp":"2026-09-20T16:24:43.38896107+08:00","Action":"","InstanceId":"","CostTime":0,"Caller":"","CallerIp":"","LogLevel":"INFO","Callee":"","RetCode":0,"InstanceType":"","FunctionType":"","LogContent":"dao schema migration completed (driver= db=cube_mvp)","Version":"","Call> 9月 20 16:24:43 VM-0-6-tencentos bash5907: {"AppId":0,"Action":"","LocalIp":"192.168.0.6","InstanceType":"","CostTime":0,"Caller":"","InstanceId":"","LogLevel":"ERROR","RequestId":"","ContainerId":"","CodeLine":"","@timestamp":"2026-09-20T16:24:43.392978879+08:00","RetCode":0,"Version":"","Callee":"","CallerIp":"","LogContent":"templatecenter node identity: C> 9月 20 16:24:43 VM-0-6-tencentos bash5907: {"Callee":"","LogContent":"templatecenter listen address: CUBE_TEMPLATE_CENTER_NODE_IP=192.168.0.6 ignored for the listen address: http_bind=\"127.0.0.1\" is set explicitly","InstanceId":"","CallerIp":"","AppId":0,"InstanceType":"","Version":"","Caller":"","CodeLine":"","CalleeAction":"","FunctionType":"","RequestId"> 9月 20 16:24:43 VM-0-6-tencentos bash5907: {"Namespace":"","CostTime":0,"InstanceType":"","InstanceId":"","ContainerId":"","RetCode":0,"CallerIp":"","CalleeEndpoint":"","Version":"","Action":"","LogContent":"templatecenter successfully booted in 0.014476s","LocalIp":"192.168.0.6","Module":"","CodeLine":"","RequestId":"","Caller":"","CalleeAction":"","Function> 9月 20 16:24:44 VM-0-6-tencentos systemd1: Started cube-sandbox-cube-templatecenter.service - Cube Sandbox CubeTemplateCenter. root@VM-0-6-tencentos \~# journalctl \ -u cube-sandbox-cube-templatecenter \ -n 100 \ --no-pager 9月 20 15:36:49 VM-0-6-tencentos systemd1: Starting cube-sandbox-cube-templatecenter.service - Cube Sandbox CubeTemplateCenter... 9月 20 15:36:49 VM-0-6-tencentos bash12505: 2026-09-20T15:36:49.937762237+08:00,listen file/usr/local/services/cubetoolbox/CubeTemplateCenter/conf.yaml 9月 20 15:36:49 VM-0-6-tencentos bash12505: cfg:{"Common":{"MockUpdateAction":false,"DebugDumpHttpBody":false,"MockDebug":false,"MockNodeNum":0,"MockCreateDirect":false,"MockCreateDirectHandle":false,"MockHttpDirect":false,"MockCreateSleep":0,"MockPercents":null,"CubeDestroyCheckFilter":false,"Debug":{"Address":""},"HttpPort":8090,"HttpBind":"127.0.0.1","WriteTimeout":360,"ReadTimeout":120,"IdleTimeout":360,"GraceFullStopTimeoutInSec":120,"CubeOpsAddr":"","MasterAddr":"http://127.0.0.1:8089","CubeOpsBootRetries":5,"CubeOpsBootBackoff":1000000000,"SyncMetaDataInterval":1000000000,"SyncMetricDataInterval":1000000000,"CleanSandboxCacheInterval":7200000000000,"EnabledListRunningSandboxCache":false,"AsyncTaskQueueSize":10000,"AsyncTaskWorkerNum":32,"HeadlessServiceName":"","DefaultHeadlessServiceNodesNum":1,"ListFilterOutLables":null,"CollectMetricInterval":100000000,"ReportLocalCreateNum":false,"ReportStdevMetric":false,"GwCacheExpiredTime":15000000000,"GwCacheEnable":false,"ReportGWRedisGetMetric":false,"EnableGetStatusFromCubelet":false,"DisableHardDelete":false,"TemplateCacheTTL":21600000000000,"TemplateCenterAddrValue":"","CollectSandboxMemoryWhitelist":null,"EnableAllCollectSandboxMemory":false,"FilterErrMsgErrorCode":null,"DescribeInstancesWhiteList":null,"DescribeTaskExpireTime":86400,"EnablePrivateIpQuery":false,"DbMaxRetryCount":5,"DbRetryInterval":5000000,"EnableCheckComNetIDParam":false,"EnableDescribeInstanceFromRedis":false,"MaxNICQueue":4,"DisableCreateImageCluster":null,"EnableAGSColdStartSwitch":false},"AuthConf":{"Enable":false,"SignatureExpireTimeInsec":120,"SecretKeyMap":null},"Log":{"Region":"","Cluster":"","Module":"templatecenter","Path":"/data/log/CubeTemplateCenter","FileSize":100,"FileNum":10,"Level":"info","EnableLogMetric":false},"CubeletConf":{"Grpc":{"GrpcPort":9999,"CleanConnTaskIntervalInMin":60,"CleanConnTaskRoutinePoolSize":64,"ConnExpireTimeInSec":180},"CommonTimeoutInsec":30,"CreateImageTimeoutInSec":300,"AppSnapshotTimeoutInSec":300,"DefaultTimeoutInsec":0,"CreateTimeoutInsec":600,"AsyncFlows":null,"RetryCode":null,"LoopRetryCode":null,"ReuseRetryCode":null,"CircuitBreakCode":null,"ExcludeLoopRetryCode":null,"BackoffRetryCode":null,"MaxRetries":5,"LoopMaxRetries":100,"BufferQueueMinJob":10,"CreateConcurrentLimit":100,"DestroyConcurentLimit":50,"ExposedPortList":null,"EnableExposedPort":false,"DisableRedisProxyPort":false,"MaxDelayInSecond":1,"BackoffRetryDelay":5000000},"InstanceDBConfig":{"Driver":"","Addr":"127.0.0.1:3306","User":"cube","Pwd":"cube_pass","DBName":"cube_mvp","ConnTimeout":5,"ReadTimeout":5,"WriteTimeout":5,"MaxIdleConns":5,"MaxOpenConns":20,"MaxConnLifeTimeSeconds":300,"MigrationLockTimeoutSeconds":0},"RedisConf":{"Password":"ceuhvu123","MaxActive":32,"MaxIdle":8,"IdleTimeout":30,"DbNo":0,"Nodes":"127.0.0.1:6379","MaxRetry":2,"MasterName":"","SentinelNodes":"","SentinelPassword":"","NodeMetricTTLSec":600,"SandboxProxyTTLSec":0},"ExtraConf":{"BlkQos":"{}","BlkQosMap":null,"FsQos":"{}","FsQosMap":null,"NetQosList":"\[\]","AllowedHostMountPrefixes":null},"Scheduler":{"Overhead":{"VmMemoryOverheadBase":"42Mi","VmMemoryOverheadCoefficient":64,"HostMemoryOverheadBase":"24Mi","CubeMsgMemoryOverhead":"16Mi","VmCpuOverhead":"0","HostCpuOverhead":"0.3"},"NodeMaxMvmNum":3000,"NodeMaxMvmNumReserveNumPercent":1,"NodeMaxMemReservedInMB":10240,"NodeMaxCpuUtil":80,"PreSelectNum":-1,"PrioritySelectNum":-1,"LeastSelectName":"random","MetricUpdateTimeout":3600000000000,"LocalMetricUpdateTimeout":3600000000000,"Filter":null,"Score":null,"PostScore":null,"DisableCircuitFilter":false,"InBackoffMode":false,"AffinityConf":null,"NodeMaxMvmNumConf":null,"EnableRunInstanceHostIps":false,"MaxMvmCPU":"","MaxMvmMemory":"","DiskUsageMaxPercent":80,"LargeSizeAffinityConf":null,"NodeMaxMemReservedConf":null,"DisableBackoffFilterInstanceType":null,"ThirtpartyFilterInstanceType":null,"InstanceTypeConf":null,"NodeAffinitySelectorAllowedKeys":null,"IgnoreRedisAllocation":false,"DeprecatedOvercommitRatio":null,"DeprecatedOvercommitRatioByType":null},"ReqTemplateConf":null,"HookWhitelist":null,"CubeEgressConf":null,"CubeProxyConf":{"AdminPort":8082,"AdminToken":"","AdminURLs":null,"HeartbeatTTLMs":15000},"SoftDeletePurge":null,"VolumePlugins":null} 9月 20 15:36:49 VM-0-6-tencentos bash12505: 2026/09/20 15:36:49 migrate: 1 applied migration version(s) have no fingerprint baseline and are NOT content-checked (e.g. applied before fingerprinting existed): 0 9月 20 15:36:49 VM-0-6-tencentos bash12505: {"CallerIp":"","FunctionType":"","InstanceType":"","Action":"","CostTime":0,"AppId":0,"LogContent":"dao schema migration completed (driver= db=cube_mvp)","LocalIp":"192.168.0.6","CodeLine":"","@timestamp":"2026-09-20T15:36:49.95661439+08:00","RequestId":"","Callee":"","RetCode":0,"CalleeAction":"","CalleeEndpoint":"","Module":"","Version":"","Namespace":"","LogLevel":"INFO","ContainerId":"","Caller":"","InstanceId":""} 9月 20 15:36:49 VM-0-6-tencentos bash12505: {"LocalIp":"192.168.0.6","CallerIp":"","AppId":0,"RetCode":0,"InstanceType":"","RequestId":"","InstanceId":"","CostTime":0,"CalleeAction":"","Action":"","CalleeEndpoint":"","ContainerId":"","LogContent":"templatecenter node identity: CUBE_TEMPLATE_CENTER_NODE_IP=192.168.0.6","Module":"","@timestamp":"2026-09-20T15:36:49.963805389+08:00","LogLevel":"ERROR","Version":"","Namespace":"","CodeLine":"","FunctionType":"","Callee":"","Caller":""} 9月 20 15:36:49 VM-0-6-tencentos bash12505:

从你贴出的日志看,CubeTemplateCenter 服务本身启动正常,没有启动失败问题 。当前重点应该是排查 模板定义(t_cube_template_definition)为空 / 镜像模板未注册 / CubeTemplateCenter无法加载模板 这一类问题

我先分析现象


1. CubeTemplateCenter 状态正常

当前:

复制代码
Active: active (running)
Main PID: 5907

启动流程:

复制代码
dao schema migration completed (driver= db=cube_mvp)
templatecenter successfully booted in 0.014476s

说明:

✅ 二进制正常

✅ 配置文件可读取

✅ MySQL schema migration正常

✅ Redis配置没有阻塞

✅ 服务监听成功


2. 注意这个 WARN:监听地址问题

日志:

复制代码
templatecenter listen address:
CUBE_TEMPLATE_CENTER_NODE_IP=192.168.0.6 ignored 
for the listen address:
http_bind="127.0.0.1" is set explicitly

你的配置:

复制代码
HttpPort:8090
HttpBind:"127.0.0.1"

意味着:

复制代码
127.0.0.1:8090

只能本机访问

如果 CubeMaster、CubeOps、其他节点访问 TemplateCenter,需要修改:

文件:

复制代码
/usr/local/services/cubetoolbox/CubeTemplateCenter/conf.yaml

找到:

复制代码
http_bind: 127.0.0.1

改:

复制代码
http_bind: 192.168.0.6

或者:

复制代码
http_bind: 0.0.0.0

然后:

复制代码
systemctl restart cube-sandbox-cube-templatecenter

验证:

复制代码
ss -lntp | grep 8090

应该看到:

正常:

复制代码
LISTEN 0 4096 192.168.0.6:8090

或者:

复制代码
LISTEN 0 4096 0.0.0.0:8090

3. t_cube_template_definition 搜索方式不对

你执行:

复制代码
grep -R "t_cube_template_definition" /usr/local/services/cubetoolbox

结果:

复制代码
templatecenter binary 匹配
cubemaster binary 匹配
cubeops binary 匹配

说明:

这是 Go 二进制内嵌字符串

不是数据库表

应该查 MySQL


4. 查询 Cube 数据库

日志:

复制代码
DBName:"cube_mvp"
Addr:"127.0.0.1:3306"
User:"cube"
Pwd:"cube_pass"

登录:

复制代码
mysql -h127.0.0.1 \
-u cube \
-pcube_pass \
cube_mvp

然后:

查看模板相关表

执行:

复制代码
show tables;

重点找:

复制代码
t_cube_template_definition

或者:

复制代码
show tables like '%template%';

5. 查看模板定义

如果存在:

复制代码
select * from t_cube_template_definition;

重点字段:

例如:

复制代码
id
name
version
image
status
created_at

正常应该有类似:

复制代码
cpu
gpu
ubuntu
centos

模板


6. 你现在 cube-image 只有一个镜像

你这里:

复制代码
/usr/local/services/cubetoolbox/cube-image/

cube-guest-image-cpu.img 232M
version

说明:

当前只有:

复制代码
CPU Guest Image

没有 GPU image

如果你要跑:

  • GPU Sandbox
  • NVIDIA CUDA
  • AI环境

需要额外镜像,例如:

复制代码
cube-guest-image-gpu.img

否则模板创建会失败


7. 检查 CubeTemplateCenter API

本机测试:

复制代码
curl http://127.0.0.1:8090/

或者:

复制代码
curl http://127.0.0.1:8090/health

查看:

复制代码
curl -v http://127.0.0.1:8090/

8. 检查 CubeMaster

你的配置:

复制代码
MasterAddr:"http://127.0.0.1:8089"

说明:

TemplateCenter依赖:

复制代码
CubeMaster
  |
  |
TemplateCenter
  |
  |
Cubelet

检查:

复制代码
systemctl status cube-sandbox-cube-master

以及:

复制代码
ss -lntp | grep 8089

应该:

复制代码
127.0.0.1:8089

9. 检查 CubeOps

配置:

复制代码
CubeOpsAddr:""

但是日志没有报错

检查:

复制代码
systemctl status cube-sandbox-cube-ops

10. 当前最可能问题

根据你现在的信息:

服务层

组件 状态
CubeTemplateCenter ✅正常
MySQL ✅正常
Redis 未知
CubeMaster 需要确认
模板数据 未知
镜像 只有CPU

我建议下一步执行:

① 查看数据库模板

复制代码
mysql -h127.0.0.1 -ucube -pcube_pass cube_mvp

执行:

复制代码
show tables like '%template%';

select count(*) from t_cube_template_definition;

② 查看 TemplateCenter 注册接口

执行:

复制代码
curl -s http://127.0.0.1:8090/metrics | head

③ 查看所有 Cube 服务

复制代码
systemctl list-units \
| grep cube-sandbox

你目前更像是 Cube Sandbox 初始化完成,但模板中心没有加载模板镜像定义 ,不是 TemplateCenter 启动故障。下一步重点看 t_cube_template_definition 数据和 CubeMaster 状态

http_bind改为192.168.0.6后无法启动

root@VM-0-6-tencentos \~#systemctl restart cube-sandbox-cube-templatecenter

Job for cube-sandbox-cube-templatecenter.service failed because the control process exited with error code.

See "systemctl status cube-sandbox-cube-templatecenter.service" and "journalctl -xeu cube-sandbox-cube-templatecenter.service" for details.

从前面的日志看,真正的问题不是 CUBE_TEMPLATE_CENTER_NODE_IP,而是 http_bind 修改后的监听行为

原配置:

复制代码
HttpPort:8090
HttpBind:"127.0.0.1"

启动正常:

复制代码
templatecenter successfully booted in 0.014476s

但是日志已经提示:

复制代码
templatecenter listen address:
CUBE_TEMPLATE_CENTER_NODE_IP=192.168.0.6 ignored for the listen address:
http_bind="127.0.0.1" is set explicitly

说明:

  • CUBE_TEMPLATE_CENTER_NODE_IP 只影响节点身份注册
  • 真正 HTTP 服务监听地址由 http_bind 控制

你改成:

复制代码
http_bind="192.168.0.6"

后启动失败,需要进一步确认原因


1. 先查看具体失败原因

执行:

复制代码
systemctl status cube-sandbox-cube-templatecenter -l

以及:

复制代码
journalctl -xeu cube-sandbox-cube-templatecenter --no-pager

重点看有没有:

情况1:监听失败

例如:

复制代码
bind: cannot assign requested address

说明:

192.168.0.6 并不是本机地址

确认:

复制代码
ip addr | grep 192.168.0.6

应该看到:

复制代码
inet 192.168.0.6/xx

情况2:端口占用

例如:

复制代码
listen tcp 192.168.0.6:8090: bind: address already in use

检查:

复制代码
ss -lntp | grep 8090

2. 检查 conf.yaml 修改是否正确

找到:

复制代码
grep -n "http_bind" \
/usr/local/services/cubetoolbox/CubeTemplateCenter/conf.yaml

应该类似:

复制代码
http_bind: "192.168.0.6"

注意:

不要写:

复制代码
http_bind=192.168.0.6

不要写:

复制代码
http_bind:192.168.0.6

3. 推荐生产配置方式

CubeSandbox 多节点环境不建议直接绑定固定 IP

推荐:

复制代码
http_bind: "0.0.0.0"

即:

复制代码
Common:
  HttpPort: 8090
  HttpBind: "0.0.0.0"

原因:

配置 效果
127.0.0.1 只能本机访问
192.168.0.6 只监听这个网卡
0.0.0.0 监听所有网卡

生产环境通常:

复制代码
TemplateCenter
        |
        |
   192.168.0.6:8090
        |
其它Cube节点访问

应该使用:

复制代码
0.0.0.0:8090

而不是:

复制代码
192.168.0.6:8090

4. 修改方式

编辑:

复制代码
vi /usr/local/services/cubetoolbox/CubeTemplateCenter/conf.yaml

修改:

复制代码
http_bind: "0.0.0.0"

然后:

复制代码
systemctl daemon-reload

systemctl restart cube-sandbox-cube-templatecenter

检查:

复制代码
ss -lntp | grep 8090

正常应该:

复制代码
LISTEN 0 4096 0.0.0.0:8090

5. 同时检查 CubeMaster 配置

你的 TemplateCenter 配置:

复制代码
MasterAddr:"http://127.0.0.1:8089"

说明当前是单机部署

如果 CubeSandbox 是多节点:

需要检查:

复制代码
grep -R "TemplateCenter" \
/usr/local/services/cubetoolbox/CubeMaster/conf.yaml

以及:

复制代码
grep -R "CubeTemplateCenter" \
/usr/local/services/cubetoolbox

应该类似:

复制代码
http://192.168.0.6:8090

否则其它 Cubelet 无法访问模板中心


6. 你的当前环境判断

根据日志:

复制代码
LocalIp":"192.168.0.6"

以及:

复制代码
CUBE_TEMPLATE_CENTER_NODE_IP=192.168.0.6

节点IP没有问题

当前最佳配置:

复制代码
CUBE_TEMPLATE_CENTER_NODE_IP=192.168.0.6

http_bind=0.0.0.0

HttpPort=8090

然后其它组件访问:

复制代码
http://192.168.0.6:8090

root@VM-0-6-tencentos \~# systemctl status cube-sandbox-cube-templatecenter -l ● cube-sandbox-cube-templatecenter.service - Cube Sandbox CubeTemplateCenter Loaded: loaded (/etc/systemd/system/cube-sandbox-cube-templatecenter.service; enabled; preset: disabled) Active: activating (auto-restart) (Result: exit-code) since Sun 2026-09-20 18:38:43 CST; 1s ago Process: 240327 ExecStart=/usr/bin/bash /usr/local/services/cubetoolbox/scripts/systemd/cubetemplatecenter-start.sh (code=exited, status=0/SUCCESS) Process: 240328 ExecStartPost=/usr/bin/bash /usr/local/services/cubetoolbox/scripts/systemd/cubetemplatecenter-postcheck.sh (code=exited, status=1/FAILURE) Main PID: 240327 (code=exited, status=0/SUCCESS) CPU: 281ms root@VM-0-6-tencentos \~# journalctl -xeu cube-sandbox-cube-templatecenter --no-pager 9月 20 16:24:43 VM-0-6-tencentos systemd1: Starting cube-sandbox-cube-templatecenter.service - Cube Sandbox CubeTemplateCenter... ░░ Subject: cube-sandbox-cube-templatecenter.service 单元已开始启动 ░░ Defined-By: systemd ░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel ░░ ░░ cube-sandbox-cube-templatecenter.service 单元已开始启动。 9月 20 16:24:43 VM-0-6-tencentos bash5907: 2026-09-20T16:24:43.375476072+08:00,listen file/usr/local/services/cubetoolbox/CubeTemplateCenter/conf.yaml 9月 20 16:24:43 VM-0-6-tencentos bash5907: cfg:{"Common":{"MockUpdateAction":false,"DebugDumpHttpBody":false,"MockDebug":false,"MockNodeNum":0,"MockCreateDirect":false,"MockCreateDirectHandle":false,"MockHttpDirect":false,"MockCreateSleep":0,"MockPercents":null,"CubeDestroyCheckFilter":false,"Debug":{"Address":""},"HttpPort":8090,"HttpBind":"127.0.0.1","WriteTimeout":360,"ReadTimeout":120,"IdleTimeout":360,"GraceFullStopTimeoutInSec":120,"CubeOpsAddr":"","MasterAddr":"http://127.0.0.1:8089","CubeOpsBootRetries":5,"CubeOpsBootBackoff":1000000000,"SyncMetaDataInterval":1000000000,"SyncMetricDataInterval":1000000000,"CleanSandboxCacheInterval":7200000000000,"EnabledListRunningSandboxCache":false,"AsyncTaskQueueSize":10000,"AsyncTaskWorkerNum":32,"HeadlessServiceName":"","DefaultHeadlessServiceNodesNum":1,"ListFilterOutLables":null,"CollectMetricInterval":100000000,"ReportLocalCreateNum":false,"ReportStdevMetric":false,"GwCacheExpiredTime":15000000000,"GwCacheEnable":false,"ReportGWRedisGetMetric":false,"EnableGetStatusFromCubelet":false,"DisableHardDelete":false,"TemplateCacheTTL":21600000000000,"TemplateCenterAddrValue":"","CollectSandboxMemoryWhitelist":null,"EnableAllCollectSandboxMemory":false,"FilterErrMsgErrorCode":null,"DescribeInstancesWhiteList":null,"DescribeTaskExpireTime":86400,"EnablePrivateIpQuery":false,"DbMaxRetryCount":5,"DbRetryInterval":5000000,"EnableCheckComNetIDParam":false,"EnableDescribeInstanceFromRedis":false,"MaxNICQueue":4,"DisableCreateImageCluster":null,"EnableAGSColdStartSwitch":false},"AuthConf":{"Enable":false,"SignatureExpireTimeInsec":120,"SecretKeyMap":null},"Log":{"Region":"","Cluster":"","Module":"templatecenter","Path":"/data/log/CubeTemplateCenter","FileSize":100,"FileNum":10,"Level":"info","EnableLogMetric":false},"CubeletConf":{"Grpc":{"GrpcPort":9999,"CleanConnTaskIntervalInMin":60,"CleanConnTaskRoutinePoolSize":64,"ConnExpireTimeInSec":180},"CommonTimeoutInsec":30,"CreateImageTimeoutInSec":300,"AppSnapshotTimeoutInSec":300,"DefaultTimeoutInsec":0,"CreateTimeoutInsec":600,"AsyncFlows":null,"RetryCode":null,"LoopRetryCode":null,"ReuseRetryCode":null,"CircuitBreakCode":null,"ExcludeLoopRetryCode":null,"BackoffRetryCode":null,"MaxRetries":5,"LoopMaxRetries":100,"BufferQueueMinJob":10,"CreateConcurrentLimit":100,"DestroyConcurentLimit":50,"ExposedPortList":null,"EnableExposedPort":false,"DisableRedisProxyPort":false,"MaxDelayInSecond":1,"BackoffRetryDelay":5000000},"InstanceDBConfig":{"Driver":"","Addr":"127.0.0.1:3306","User":"cube","Pwd":"cube_pass","DBName":"cube_mvp","ConnTimeout":5,"ReadTimeout":5,"WriteTimeout":5,"MaxIdleConns":5,"MaxOpenConns":20,"MaxConnLifeTimeSeconds":300,"MigrationLockTimeoutSeconds":0},"RedisConf":{"Password":"ceuhvu123","MaxActive":32,"MaxIdle":8,"IdleTimeout":30,"DbNo":0,"Nodes":"127.0.0.1:6379","MaxRetry":2,"MasterName":"","SentinelNodes":"","SentinelPassword":"","NodeMetricTTLSec":600,"SandboxProxyTTLSec":0},"ExtraConf":{"BlkQos":"{}","BlkQosMap":null,"FsQos":"{}","FsQosMap":null,"NetQosList":"\[\]","AllowedHostMountPrefixes":null},"Scheduler":{"Overhead":{"VmMemoryOverheadBase":"42Mi","VmMemoryOverheadCoefficient":64,"HostMemoryOverheadBase":"24Mi","CubeMsgMemoryOverhead":"16Mi","VmCpuOverhead":"0","HostCpuOverhead":"0.3"},"NodeMaxMvmNum":3000,"NodeMaxMvmNumReserveNumPercent":1,"NodeMaxMemReservedInMB":10240,"NodeMaxCpuUtil":80,"PreSelectNum":-1,"PrioritySelectNum":-1,"LeastSelectName":"random","MetricUpdateTimeout":3600000000000,"LocalMetricUpdateTimeout":3600000000000,"Filter":null,"Score":null,"PostScore":null,"DisableCircuitFilter":false,"InBackoffMode":false,"AffinityConf":null,"NodeMaxMvmNumConf":null,"EnableRunInstanceHostIps":false,"MaxMvmCPU":"","MaxMvmMemory":"","DiskUsageMaxPercent":80,"LargeSizeAffinityConf":null,"NodeMaxMemReservedConf":null,"DisableBackoffFilterInstanceType":null,"ThirtpartyFilterInstanceType":null,"InstanceTypeConf":null,"NodeAffinitySelectorAllowedKeys":null,"IgnoreRedisAllocation":false,"DeprecatedOvercommitRatio":null,"DeprecatedOvercommitRatioByType":null},"ReqTemplateConf":null,"HookWhitelist":null,"CubeEgressConf":null,"CubeProxyConf":{"AdminPort":8082,"AdminToken":"","AdminURLs":null,"HeartbeatTTLMs":15000},"SoftDeletePurge":null,"VolumePlugins":null} 9月 20 16:24:43 VM-0-6-tencentos bash5907: 2026/09/20 16:24:43 migrate: 1 applied migration version(s) have no fingerprint baseline and are NOT content-checked (e.g. applied before fingerprinting existed): 0 9月 20 16:24:43 VM-0-6-tencentos bash5907: {"LocalIp":"192.168.0.6","Module":"","@timestamp":"2026-09-20T16:24:43.38896107+08:00","Action":"","InstanceId":"","CostTime":0,"Caller":"","CallerIp":"","LogLevel":"INFO","Callee":"","RetCode":0,"InstanceType":"","FunctionType":"","LogContent":"dao schema migration completed (driver= db=cube_mvp)","Version":"","CalleeEndpoint":"","RequestId":"","AppId":0,"CodeLine":"","Namespace":"","CalleeAction":"","ContainerId":""} 9月 20 16:24:43 VM-0-6-tencentos bash5907: {"AppId":0,"Action":"","LocalIp":"192.168.0.6","InstanceType":"","CostTime":0,"Caller":"","InstanceId":"","LogLevel":"ERROR","RequestId":"","ContainerId":"","CodeLine":"","@timestamp":"2026-09-20T16:24:43.392978879+08:00","RetCode":0,"Version":"","Callee":"","CallerIp":"","LogContent":"templatecenter node identity: CUBE_TEMPLATE_CENTER_NODE_IP=192.168.0.6","Module":"","CalleeAction":"","Namespace":"","FunctionType":"","CalleeEndpoint":""} 9月 20 16:24:43 VM-0-6-tencentos bash5907: {"Callee":"","LogContent":"templatecenter listen address: CUBE_TEMPLATE_CENTER_NODE_IP=192.168.0.6 ignored for the listen address: http_bind=\"127.0.0.1\" is set explicitly","InstanceId":"","CallerIp":"","AppId":0,"InstanceType":"","Version":"","Caller":"","CodeLine":"","CalleeAction":"","FunctionType":"","RequestId":"","CostTime":0,"LocalIp":"192.168.0.6","Module":"","LogLevel":"WARN","ContainerId":"","RetCode":0,"Namespace":"","CalleeEndpoint":"","@timestamp":"2026-09-20T16:24:43.393017301+08:00","Action":""} 9月 20 16:24:43 VM-0-6-tencentos bash5907: {"Namespace":"","CostTime":0,"InstanceType":"","InstanceId":"","ContainerId":"","RetCode":0,"CallerIp":"","CalleeEndpoint":"","Version":"","Action":"","LogContent":"templatecenter successfully booted in 0.014476s","LocalIp":"192.168.0.6","Module":"","CodeLine":"","RequestId":"","Caller":"","CalleeAction":"","FunctionType":"","@timestamp":"2026-09-20T16:24:43.394146817+08:00","LogLevel":"ERROR","AppId":0,"Callee":""} 9月 20 16:24:44 VM-0-6-tencentos systemd1: Started cube-sandbox-cube-templatecenter.service - Cube Sandbox CubeTemplateCenter. ░░ Subject: cube-sandbox-cube-templatecenter.service 单元已结束启动 ░░ Defined-By: systemd ░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel ░░ ░░ cube-sandbox-cube-templatecenter.service 单元已结束启动。 ░░ ░░ 启动结果为"done"。 9月 20 18:28:05 VM-0-6-tencentos bash5907: 2026-09-20T18:28:05.383136309+08:00,OnFileEvent:WRITE "/usr/local/services/cubetoolbox/CubeTemplateCenter/conf.yaml" 9月 20 18:28:05 VM-0-6-tencentos bash5907: setLogLevel:info 9月 20 18:28:05 VM-0-6-tencentos bash5907: 2026-09-20T18:28:05.384444503+08:00,OnFileEvent:CHMOD "/usr/local/services/cubetoolbox/CubeTemplateCenter/conf.yaml" 9月 20 18:28:06 VM-0-6-tencentos bash5907: {"CostTime":0,"LogLevel":"INFO","RetCode":0,"AppId":0,"Module":"","FunctionType":"","InstanceId":"","Caller":"","CalleeAction":"","ContainerId":"","Version":"","LogContent":"templatecenter received signal terminated, shutting down","LocalIp":"192.168.0.6","@timestamp":"2026-09-20T18:28:06.568537334+08:00","Namespace":"","RequestId":"","InstanceType":"","CallerIp":"","Callee":"","CodeLine":"","CalleeEndpoint":"","Action":""} 9月 20 18:28:06 VM-0-6-tencentos bash5907: {"Caller":"","LogLevel":"ERROR","Callee":"","ContainerId":"","AppId":0,"InstanceId":"","CalleeEndpoint":"","CalleeAction":"","LogContent":"templatecenter shutting down: cancelling in-flight builds","FunctionType":"","CostTime":0,"LocalIp":"192.168.0.6","Module":"","@timestamp":"2026-09-20T18:28:06.568596615+08:00","Namespace":"","RequestId":"","Version":"","RetCode":0,"InstanceType":"","CodeLine":"","Action":"","CallerIp":""} 9月 20 18:28:06 VM-0-6-tencentos bash5907: {"@timestamp":"2026-09-20T18:28:06.568607436+08:00","LogLevel":"ERROR","LogContent":"templatecenter stopped gracefully begin, pid 5907","LocalIp":"192.168.0.6","Module":"","CodeLine":""} 9月 20 18:28:06 VM-0-6-tencentos bash5907: {"CodeLine":"","@timestamp":"2026-09-20T18:28:06.56866807+08:00","LogLevel":"ERROR","LogContent":"templatecenter InternalHttp Shutdown succ","LocalIp":"192.168.0.6","Module":""} 9月 20 18:28:06 VM-0-6-tencentos systemd1: Stopping cube-sandbox-cube-templatecenter.service - Cube Sandbox CubeTemplateCenter... ░░ Subject: cube-sandbox-cube-templatecenter.service 单元已开始停止操作 ░░ Defined-By: systemd ░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel ░░ ░░ cube-sandbox-cube-templatecenter.service 单元已开始停止操作。 9月 20 18:28:06 VM-0-6-tencentos systemd1: cube-sandbox-cube-templatecenter.service: Deactivated successfully. ░░ Subject: Unit succeeded ░░ Defined-By: systemd ░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel ░░ ░░ The unit cube-sandbox-cube-templatecenter.service has successfully entered the 'dead' state. 9月 20 18:28:06 VM-0-6-tencentos systemd1: Stopped cube-sandbox-cube-templatecenter.service - Cube Sandbox CubeTemplateCenter. ░░ Subject: cube-sandbox-cube-templatecenter.service 单元已结束停止操作 ░░ Defined-By: systemd ░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel ░░ ░░ cube-sandbox-cube-templatecenter.service 单元已结束停止操作。 9月 20 18:28:06 VM-0-6-tencentos systemd1: cube-sandbox-cube-templatecenter.service: Consumed 10.129s CPU time, 48.5M memory peak, 0B memory swap peak. ░░ Subject: Resources consumed by unit runtime ░░ Defined-By: systemd ░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel ░░ ░░ The unit cube-sandbox-cube-templatecenter.service completed and consumed the indicated resources. 9月 20 18:28:06 VM-0-6-tencentos systemd1: Starting cube-sandbox-cube-templatecenter.service - Cube Sandbox CubeTemplateCenter... ░░ Subject: cube-sandbox-cube-templatecenter.service 单元已开始启动 ░░ Defined-By: systemd ░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel ░░ ░░ cube-sandbox-cube-templatecenter.service 单元已开始启动。 9月 20 18:28:06 VM-0-6-tencentos bash220740: 2026-09-20T18:28:06.627410234+08:00,listen file/usr/local/services/cubetoolbox/CubeTemplateCenter/conf.yaml 9月 20 18:28:06 VM-0-6-tencentos bash220740: cfg:{"Common":{"MockUpdateAction":false,"DebugDumpHttpBody":false,"MockDebug":false,"MockNodeNum":0,"MockCreateDirect":false,"MockCreateDirectHandle":false,"MockHttpDirect":false,"MockCreateSleep":0,"MockPercents":null,"CubeDestroyCheckFilter":false,"Debug":{"Address":""},"HttpPort":8090,"HttpBind":"0.0.0.0","WriteTimeout":360,"ReadTimeout":120,"IdleTimeout":360,"GraceFullStopTimeoutInSec":120,"CubeOpsAddr":"","MasterAddr":"http://192.168.0.6:8089","CubeOpsBootRetries":5,"CubeOpsBootBackoff":1000000000,"SyncMetaDataInterval":1000000000,"SyncMetricDataInterval":1000000000,"CleanSandboxCacheInterval":7200000000000,"EnabledListRunningSandboxCache":false,"AsyncTaskQueueSize":10000,"AsyncTaskWorkerNum":32,"HeadlessServiceName":"","DefaultHeadlessServiceNodesNum":1,"ListFilterOutLables":null,"CollectMetricInterval":100000000,"ReportLocalCreateNum":false,"ReportStdevMetric":false,"GwCacheExpiredTime":15000000000,"GwCacheEnable":false,"ReportGWRedisGetMetric":false,"EnableGetStatusFromCubelet":false,"DisableHardDelete":false,"TemplateCacheTTL":21600000000000,"TemplateCenterAddrValue":"","CollectSandboxMemoryWhitelist":null,"EnableAllCollectSandboxMemory":false,"FilterErrMsgErrorCode":null,"DescribeInstancesWhiteList":null,"DescribeTaskExpireTime":86400,"EnablePrivateIpQuery":false,"DbMaxRetryCount":5,"DbRetryInterval":5000000,"EnableCheckComNetIDParam":false,"EnableDescribeInstanceFromRedis":false,"MaxNICQueue":4,"DisableCreateImageCluster":null,"EnableAGSColdStartSwitch":false},"AuthConf":{"Enable":false,"SignatureExpireTimeInsec":120,"SecretKeyMap":null},"Log":{"Region":"","Cluster":"","Module":"templatecenter","Path":"/data/log/CubeTemplateCenter","FileSize":100,"FileNum":10,"Level":"info","EnableLogMetric":false},"CubeletConf":{"Grpc":{"GrpcPort":9999,"CleanConnTaskIntervalInMin":60,"CleanConnTaskRoutinePoolSize":64,"ConnExpireTimeInSec":180},"CommonTimeoutInsec":30,"CreateImageTimeoutInSec":300,"AppSnapshotTimeoutInSec":300,"DefaultTimeoutInsec":0,"CreateTimeoutInsec":600,"AsyncFlows":null,"RetryCode":null,"LoopRetryCode":null,"ReuseRetryCode":null,"CircuitBreakCode":null,"ExcludeLoopRetryCode":null,"BackoffRetryCode":null,"MaxRetries":5,"LoopMaxRetries":100,"BufferQueueMinJob":10,"CreateConcurrentLimit":100,"DestroyConcurentLimit":50,"ExposedPortList":null,"EnableExposedPort":false,"DisableRedisProxyPort":false,"MaxDelayInSecond":1,"BackoffRetryDelay":5000000},"InstanceDBConfig":{"Driver":"","Addr":"127.0.0.1:3306","User":"cube","Pwd":"cube_pass","DBName":"cube_mvp","ConnTimeout":5,"ReadTimeout":5,"WriteTimeout":5,"MaxIdleConns":5,"MaxOpenConns":20,"MaxConnLifeTimeSeconds":300,"MigrationLockTimeoutSeconds":0},"RedisConf":{"Password":"ceuhvu123","MaxActive":32,"MaxIdle":8,"IdleTimeout":30,"DbNo":0,"Nodes":"127.0.0.1:6379","MaxRetry":2,"MasterName":"","SentinelNodes":"","SentinelPassword":"","NodeMetricTTLSec":600,"SandboxProxyTTLSec":0},"ExtraConf":{"BlkQos":"{}","BlkQosMap":null,"FsQos":"{}","FsQosMap":null,"NetQosList":"\[\]","AllowedHostMountPrefixes":null},"Scheduler":{"Overhead":{"VmMemoryOverheadBase":"42Mi","VmMemoryOverheadCoefficient":64,"HostMemoryOverheadBase":"24Mi","CubeMsgMemoryOverhead":"16Mi","VmCpuOverhead":"0","HostCpuOverhead":"0.3"},"NodeMaxMvmNum":3000,"NodeMaxMvmNumReserveNumPercent":1,"NodeMaxMemReservedInMB":10240,"NodeMaxCpuUtil":80,"PreSelectNum":-1,"PrioritySelectNum":-1,"LeastSelectName":"random","MetricUpdateTimeout":3600000000000,"LocalMetricUpdateTimeout":3600000000000,"Filter":null,"Score":null,"PostScore":null,"DisableCircuitFilter":false,"InBackoffMode":false,"AffinityConf":null,"NodeMaxMvmNumConf":null,"EnableRunInstanceHostIps":false,"MaxMvmCPU":"","MaxMvmMemory":"","DiskUsageMaxPercent":80,"LargeSizeAffinityConf":null,"NodeMaxMemReservedConf":null,"DisableBackoffFilterInstanceType":null,"ThirtpartyFilterInstanceType":null,"InstanceTypeConf":null,"NodeAffinitySelectorAllowedKeys":null,"IgnoreRedisAllocation":false,"DeprecatedOvercommitRatio":null,"DeprecatedOvercommitRatioByType":null},"ReqTemplateConf":null,"HookWhitelist":null,"CubeEgressConf":null,"CubeProxyConf":{"AdminPort":8082,"AdminToken":"","AdminURLs":null,"HeartbeatTTLMs":15000},"SoftDeletePurge":null,"VolumePlugins":null} 9月 20 18:28:06 VM-0-6-tencentos bash220740: 2026/09/20 18:28:06 migrate: 1 applied migration version(s) have no fingerprint baseline and are NOT content-checked (e.g. applied before fingerprinting existed): 0 9月 20 18:28:06 VM-0-6-tencentos bash220740: {"Action":"","Version":"","CallerIp":"","CalleeAction":"","LogContent":"dao schema migration completed (driver= db=cube_mvp)","CodeLine":"","@timestamp":"2026-09-20T18:28:06.63496995+08:00","LogLevel":"INFO","Callee":"","FunctionType":"","AppId":0,"ContainerId":"","InstanceId":"","Module":"","InstanceType":"","RetCode":0,"RequestId":"","LocalIp":"192.168.0.6","CalleeEndpoint":"","CostTime":0,"Namespace":"","Caller":""} 9月 20 18:28:06 VM-0-6-tencentos bash220740: {"Module":"","AppId":0,"Namespace":"","ContainerId":"","Caller":"","CodeLine":"","InstanceId":"","RetCode":0,"CostTime":0,"RequestId":"","Action":"","CalleeEndpoint":"","FunctionType":"","@timestamp":"2026-09-20T18:28:06.637522696+08:00","LogLevel":"ERROR","Version":"","CalleeAction":"","Callee":"","LogContent":"templatecenter node identity: CUBE_TEMPLATE_CENTER_NODE_IP=192.168.0.6","LocalIp":"192.168.0.6","CallerIp":"","InstanceType":""} 9月 20 18:28:06 VM-0-6-tencentos bash220740: {"Callee":"","RequestId":"","Version":"","InstanceType":"","InstanceId":"","Action":"","CostTime":0,"CodeLine":"","@timestamp":"2026-09-20T18:28:06.637677728+08:00","LogLevel":"WARN","AppId":0,"Caller":"","Namespace":"","RetCode":0,"LogContent":"templatecenter listen address: listening on 192.168.0.6 from CUBE_TEMPLATE_CENTER_NODE_IP","LocalIp":"192.168.0.6","Module":"","ContainerId":"","CalleeEndpoint":"","CalleeAction":"","CallerIp":"","FunctionType":""} 9月 20 18:28:06 VM-0-6-tencentos bash220740: {"Module":"","CodeLine":"","@timestamp":"2026-09-20T18:28:06.637922399+08:00","LogLevel":"ERROR","CostTime":0,"Caller":"","FunctionType":"","AppId":0,"LocalIp":"192.168.0.6","Namespace":"","CallerIp":"","CalleeEndpoint":"","Version":"","InstanceId":"","InstanceType":"","RequestId":"","LogContent":"templatecenter successfully booted in 0.009064s","Callee":"","Action":"","RetCode":0,"CalleeAction":"","ContainerId":""} 9月 20 18:28:36 VM-0-6-tencentos bash220741: one-click-systemd ERROR wait_for_http timeout: url=http://127.0.0.1:8090/health waited=30s last_curl_error=curl: (7) Failed to connect to 127.0.0.1 port 8090 after 0 ms: Couldn't connect to server 9月 20 18:28:36 VM-0-6-tencentos bash220741: one-click-systemd ERROR: cubetemplatecenter health not ready 9月 20 18:28:36 VM-0-6-tencentos systemd1: cube-sandbox-cube-templatecenter.service: Control process exited, code=exited, status=1/FAILURE ░░ Subject: Unit process exited ░░ Defined-By: systemd ░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel ░░ ░░ An ExecStartPost= process belonging to unit cube-sandbox-cube-templatecenter.service has exited. ░░ ░░ The process' exit code is 'exited' and its exit status is 1. 9月 20 18:28:36 VM-0-6-tencentos bash220740: {"@timestamp":"2026-09-20T18:28:36.777787861+08:00","AppId":0,"Action":"","CostTime":0,"LogLevel":"INFO","InstanceType":"","LogContent":"templatecenter received signal terminated, shutting down","Module":"","CodeLine":"","InstanceId":"","ContainerId":"","CallerIp":"","Caller":"","Version":"","Namespace":"","LocalIp":"192.168.0.6","CalleeAction":"","RequestId":"","FunctionType":"","Callee":"","RetCode":0,"CalleeEndpoint":""} 9月 20 18:28:36 VM-0-6-tencentos bash220740: {"LogLevel":"ERROR","RetCode":0,"AppId":0,"Caller":"","Callee":"","CostTime":0,"Module":"","FunctionType":"","InstanceType":"","Version":"","LocalIp":"192.168.0.6","CodeLine":"","Namespace":"","ContainerId":"","InstanceId":"","@timestamp":"2026-09-20T18:28:36.777843506+08:00","Action":"","RequestId":"","CalleeEndpoint":"","CallerIp":"","CalleeAction":"","LogContent":"templatecenter shutting down: cancelling in-flight builds"} 9月 20 18:28:36 VM-0-6-tencentos bash[220740

从日志看,CubeTemplateCenter 本身启动成功,失败点在 systemd 的健康检查脚本,不是程序启动失败

关键日志:

复制代码
templatecenter successfully booted in 0.009064s

说明服务已经起来

但是随后:

复制代码
[one-click-systemd] ERROR wait_for_http timeout:
url=http://127.0.0.1:8090/health
curl: (7) Failed to connect to 127.0.0.1 port 8090

然后:

复制代码
ExecStartPost=/usr/bin/bash ... cubetemplatecenter-postcheck.sh
status=1/FAILURE

systemd 认为启动失败,于是发送停止信号:

复制代码
templatecenter received signal terminated, shutting down

根因

你修改了:

原来:

复制代码
HttpBind: "127.0.0.1"

改成:

复制代码
HttpBind: "192.168.0.6"

或者:

复制代码
HttpBind: "0.0.0.0"

程序监听地址变了

但是健康检查脚本仍然固定检查:

复制代码
curl http://127.0.0.1:8090/health

所以:

程序:

复制代码
192.168.0.6:8090

监听

健康检查:

复制代码
127.0.0.1:8090

访问不到


验证

执行:

复制代码
ss -lntp | grep 8090

你应该看到类似:

当前配置 192.168.0.6

复制代码
LISTEN 0 4096 192.168.0.6:8090

或者:

复制代码
LISTEN 0 4096 0.0.0.0:8090

测试:

复制代码
curl http://192.168.0.6:8090/health

应该正常

但是:

复制代码
curl http://127.0.0.1:8090/health

失败


解决方案 1(推荐):修改 postcheck 健康检查地址

查看:

复制代码
cat /usr/local/services/cubetoolbox/scripts/systemd/cubetemplatecenter-postcheck.sh

应该类似:

复制代码
wait_for_http http://127.0.0.1:8090/health

改:

复制代码
wait_for_http http://192.168.0.6:8090/health

或者更推荐:

使用环境变量:

复制代码
CUBE_TEMPLATE_CENTER_NODE_IP=192.168.0.6

例如:

复制代码
wait_for_http http://${CUBE_TEMPLATE_CENTER_NODE_IP}:8090/health

然后:

复制代码
systemctl daemon-reload

systemctl restart cube-sandbox-cube-templatecenter

解决方案 2(更推荐生产):监听 0.0.0.0 + localhost健康检查

生产环境建议:

conf.yaml:

复制代码
Common:
  HttpBind: "0.0.0.0"
  HttpPort: 8090

保持:

复制代码
curl http://127.0.0.1:8090/health

不变

原因:

  • 服务监听所有网卡
  • 本机健康检查稳定
  • 避免绑定具体 IP 后 IP变化导致故障

即:

复制代码
                    +-------------+
192.168.0.6 ------> |             |
                   | TemplateCtr |
127.0.0.1 --------> |             |
                    +-------------+
                         :8090

解决方案 3:增加127.0.0.1绑定(不推荐)

如果程序支持多监听,可以:

复制代码
HttpBind: "0.0.0.0"

不要写:

复制代码
127.0.0.1,192.168.0.6

很多 Go HTTP 服务不支持


另外检查一个隐藏问题

你的配置里:

复制代码
MasterAddr":"http://192.168.0.6:8089"

之前:

复制代码
MasterAddr":"http://127.0.0.1:8089"

如果 CubeMaster 也监听:

复制代码
ss -lntp | grep 8089

确认:

复制代码
192.168.0.6:8089

或者:

复制代码
0.0.0.0:8089

否则 TemplateCenter 后续可能连接 Master 失败


建议最终生产配置

CubeTemplateCenter/conf.yaml

复制代码
Common:
  HttpBind: "0.0.0.0"
  HttpPort: 8090

  MasterAddr: "http://192.168.0.6:8089"

环境:

复制代码
export CUBE_TEMPLATE_CENTER_NODE_IP=192.168.0.6

systemd:

复制代码
ExecStartPost=/usr/bin/bash /usr/local/services/cubetoolbox/scripts/systemd/cubetemplatecenter-postcheck.sh

postcheck:

复制代码
curl -f http://127.0.0.1:8090/health

然后:

复制代码
systemctl daemon-reload
systemctl restart cube-sandbox-cube-templatecenter

systemctl status cube-sandbox-cube-templatecenter -l
相关推荐
あ-13 小时前
promtail收集日志异常排查
devops
龙智DevSecOps解决方案2 天前
JFrog供应链流量控制器架构解析:如何通过 SASE 集成阻断 AI 智能体的恶意依赖拉取?
ai编程·devops·jfrog·软件供应链安全
あ-2 天前
proxmox控制台占用内存高虚拟机占用内存低排查
devops
龙智DevSecOps解决方案2 天前
Perforce P4 2026 更新详解:P4 MCP 服务器、REST API、Spark Stream 与 P4 Signals
人工智能·devops·版本控制·p4·perforce
宋均浩2 天前
告警规则 243 砍到 27:Prometheus 降噪实战,日均打扰 47 次 → 3 次
云原生·监控·devops
智能运维指南2 天前
代码管理软件如何融入 DevOps 工具链?全链路联通的 5 个关键点
运维·devops·信创适配·嘉为蓝鲸
weixin_435247062 天前
DevOps成熟度评估与改进方案模版
devops
极小狐3 天前
CI 算力饥渴症:Runner 弹性伸缩的架构权衡与落地
ci/cd·kubernetes·devops·弹性伸缩
智能运维指南3 天前
持续集成平台怎么选?从Jenkins迁移、信创适配到质量门禁的全维评估
运维·devops·嘉为蓝鲸