cubesandbox创建模版报错
模板构建失败。
该模板构建失败,无法查看详情,但你可以将其删除

CubeSandbox 模板构建失败(FAILED),页面只显示:
模板构建失败。
该模板构建失败,无法查看详情
Job ID:
ceaa39fc-aa7b-43d4-974a-1ef430a074f7
这里需要从 CubeSandbox 后端 Job 日志定位,前端不会展示具体原因
结合你的环境(v0.7.1 + PVM + MinIO + Harbor 镜像),优先排查以下几个方向
1. 根据 Job ID 查构建日志(第一步)
你的 Job ID:
ceaa39fc-aa7b-43d4-974a-1ef430a074f7
在 CubeSandbox 节点执行:
grep -R "ceaa39fc-aa7b-43d4-974a-1ef430a074f7" /var/log 2>/dev/null
或者:
journalctl -xe | grep ceaa39fc
如果 CubeSandbox 是容器:
docker ps | grep cube
然后:
docker logs <cube-container> | grep ceaa39fc
2. 查看 CubeSandbox 服务日志
先找服务:
systemctl list-units | grep cube
常见:
cube-sandbox.service
cube-server.service
cube-agent.service
查看:
journalctl -u cube-sandbox -n 200 --no-pager
重点搜索:
journalctl -u cube-sandbox -n 500 | egrep "ERROR|failed|ceaa39fc"
3. 你的截图里有一个重要信息
模板来源:
镜像:
harbor-intranet.allintechinc.com/llm-planning/llm-planning-sandb...
说明模板创建流程大概率:
Harbor 镜像
|
v
CubeSandbox Builder
|
v
生成 CFS/MinIO 快照
|
v
Template
所以重点检查 Harbor 拉取
4. 检查 Harbor 镜像访问
在 CubeSandbox 主机:
docker login harbor-intranet.allintechinc.com
测试:
docker pull harbor-intranet.allintechinc.com/llm-planning/llm-planning-sandbox:xxx
如果失败:
常见:
认证失败
unauthorized: authentication required
需要配置 Harbor 凭证
TLS问题
例如:
x509: certificate signed by unknown authority
需要导入 CA:
/etc/docker/certs.d/harbor-intranet.allintechinc.com/ca.crt
5. 检查 MinIO/CFS
你安装时:
CUBE_S3_ENDPOINT=http://192.168.0.6:9000
bucket=cube-volumes
模板生成需要写入:
mc ls cube/cube-volumes
检查空间:
df -h
重点:
/var
/tmp
/mnt
模板构建可能需要几十 GB 临时空间
6. 检查 PVM/KVM
你的安装:
PVM host detected
kvm_pvm loaded
检查:
lsmod | grep kvm
应该:
kvm_pvm
kvm
检查:
/dev/kvm
权限:
ls -l /dev/kvm
7. 删除失败模板
页面删除失败模板即可
如果 UI 删除失败:
查模板:
kubectl get template -A
或者:
docker exec -it <cube-container> sh
查看数据库:
grep -R "tpl-464f1b5538764bd7afa240ca" /
8. 建议你现在执行这几个命令
按顺序:
① 找 Job
grep -R "ceaa39fc-aa7b-43d4-974a-1ef430a074f7" /var/log 2>/dev/null
② 看服务
systemctl list-units | grep cube
③ 看最近错误
journalctl -xe -n 200 | grep -i cube
④ 看磁盘
df -h
root@VM-0-6-tencentos \~# systemctl list-units | grep -i cube sys-devices-virtual-net-cube\x2ddev.device loaded active plugged /sys/devices/virtual/net/cube-dev sys-devices-virtual-net-cube\x2ddns0.device loaded active plugged /sys/devices/virtual/net/cube-dns0 sys-subsystem-net-devices-cube\x2ddev.device loaded active plugged /sys/subsystem/net/devices/cube-dev sys-subsystem-net-devices-cube\x2ddns0.device loaded active plugged /sys/subsystem/net/devices/cube-dns0 usr-local-services-cubetoolbox-cubeletmnt-mnt.mount loaded active mounted /usr/local/services/cubetoolbox/cubeletmnt/mnt usr-local-services-cubetoolbox-cubeletmnt.mount loaded active mounted /usr/local/services/cubetoolbox/cubeletmnt cube-sandbox-coredns.service loaded active running Cube Sandbox CoreDNS container cube-sandbox-cube-api.service loaded active running Cube Sandbox Cube API cube-sandbox-cube-egress-net.service loaded active exited Cube Sandbox cube-egress host network rules (TPROXY + ip rule) cube-sandbox-cube-egress.service loaded active running Cube Sandbox cube-egress (transparent MITM proxy) cube-sandbox-cube-lifecycle-manager.service loaded active running Cube Sandbox cube-lifecycle-manager container cube-sandbox-cube-proxy.service loaded active running Cube Sandbox cube-proxy container cube-sandbox-cube-templatecenter.service loaded active running Cube Sandbox CubeTemplateCenter cube-sandbox-cubelet.service loaded active running Cube Sandbox Cubelet cube-sandbox-cubemaster.service loaded active running Cube Sandbox CubeMaster cube-sandbox-cubeops.service loaded active running Cube Sandbox CubeOps (admin/ops API) cube-sandbox-dns.service loaded active exited Cube Sandbox host DNS routing cube-sandbox-minio.service loaded active running Cube Sandbox MinIO container cube-sandbox-mysql.service loaded active running Cube Sandbox MySQL container cube-sandbox-redis.service loaded active running Cube Sandbox Redis container cube-sandbox-webui.service loaded active running Cube Sandbox Web UI container cube-sandbox-control.target loaded active active Cube Sandbox control-plane stack root@VM-0-6-tencentos \~# journalctl -xe -n 300 | grep -Ei "cube|template|job|error|failed" 9月 20 17:47:44 VM-0-6-tencentos bash5538: 0.450ms rows:0 SELECT * FROM
t_cube_template_definitionWHERE template_id = 'tpl-464f1b5538764bd7afa240ca' ANDt_cube_template_definition.deleted_atIS NULL ORDER BYt_cube_template_definition.idLIMIT 1 9月 20 17:47:44 VM-0-6-tencentos d5c0c2c48c921915: 1.119.147.226 - - 20/Sep/2026:09:47:44 +0000 "GET /templates/tpl-464f1b5538764bd7afa240ca HTTP/1.1" 404 61 "http://152.136.174.29:12088/templates/tpl-464f1b5538764bd7afa240ca" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/152.0.0.0 Safari/537.36" 9月 20 17:47:44 VM-0-6-tencentos bash6300: cube-webui | 1.119.147.226 - - 20/Sep/2026:09:47:44 +0000 "GET /templates/tpl-464f1b5538764bd7afa240ca HTTP/1.1" 404 61 "http://152.136.174.29:12088/templates/tpl-464f1b5538764bd7afa240ca" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/152.0.0.0 Safari/537.36" 9月 20 17:47:44 VM-0-6-tencentos d5c0c2c48c921915: 1.119.147.226 - - 20/Sep/2026:09:47:44 +0000 "GET /templates/compat HTTP/1.1" 200 112 "http://152.136.174.29:12088/templates/tpl-464f1b5538764bd7afa240ca" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/152.0.0.0 Safari/537.36" 9月 20 17:47:44 VM-0-6-tencentos bash6300: cube-webui | 1.119.147.226 - - 20/Sep/2026:09:47:44 +0000 "GET /templates/compat HTTP/1.1" 200 112 "http://152.136.174.29:12088/templates/tpl-464f1b5538764bd7afa240ca" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/152.0.0.0 Safari/537.36" 9月 20 17:47:45 VM-0-6-tencentos bash6300: cube-webui | 127.0.0.1 - - 20/Sep/2026:09:47:45 +0000 "GET / HTTP/1.1" 200 1120 "-" "Wget/1.21.3" 9月 20 17:47:45 VM-0-6-tencentos bash5538: 2026/09/20 17:47:45 /workspace/CubeMaster/pkg/templatecenter/snapshot_table.go:29 record not found 9月 20 17:47:45 VM-0-6-tencentos bash5538: 0.683ms rows:0 SELECT * FROMt_cube_snapshotWHERE snapshot_id = 'tpl-464f1b5538764bd7afa240ca' ANDt_cube_snapshot.deleted_atIS NULL ORDER BYt_cube_snapshot.idLIMIT 1 9月 20 17:47:45 VM-0-6-tencentos bash5538: 2026/09/20 17:47:45 /workspace/CubeMaster/pkg/templatecenter/store.go:1118 record not found 9月 20 17:47:45 VM-0-6-tencentos bash5538: 0.397ms rows:0 SELECT * FROMt_cube_template_definitionWHERE template_id = 'tpl-464f1b5538764bd7afa240ca' ANDt_cube_template_definition.deleted_atIS NULL ORDER BYt_cube_template_definition.idLIMIT 1 9月 20 17:47:45 VM-0-6-tencentos d5c0c2c48c921915: 1.119.147.226 - - 20/Sep/2026:09:47:45 +0000 "GET /templates/tpl-464f1b5538764bd7afa240ca HTTP/1.1" 404 61 "http://152.136.174.29:12088/templates/tpl-464f1b5538764bd7afa240ca" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/152.0.0.0 Safari/537.36" 9月 20 17:47:45 VM-0-6-tencentos bash6300: cube-webui | 1.119.147.226 - - 20/Sep/2026:09:47:45 +0000 "GET /templates/tpl-464f1b5538764bd7afa240ca HTTP/1.1" 404 61 "http://152.136.174.29:12088/templates/tpl-464f1b5538764bd7afa240ca" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/152.0.0.0 Safari/537.36" 9月 20 17:47:55 VM-0-6-tencentos bash6300: cube-webui | 127.0.0.1 - - 20/Sep/2026:09:47:55 +0000 "GET / HTTP/1.1" 200 1120 "-" "Wget/1.21.3" 9月 20 17:48:05 VM-0-6-tencentos bash6300: cube-webui | 127.0.0.1 - - 20/Sep/2026:09:48:05 +0000 "GET / HTTP/1.1" 200 1120 "-" "Wget/1.21.3" 9月 20 17:48:12 VM-0-6-tencentos bash5538: 2026/09/20 17:48:12 /workspace/CubeMaster/pkg/templatecenter/store.go:1118 record not found 9月 20 17:48:12 VM-0-6-tencentos bash5538: 0.411ms rows:0 SELECT * FROMt_cube_template_definitionWHERE template_id = 'tpl-464f1b5538764bd7afa240ca' ANDt_cube_template_definition.deleted_atIS NULL ORDER BYt_cube_template_definition.idLIMIT 1 9月 20 17:48:12 VM-0-6-tencentos bash5538: 2026/09/20 17:48:12 /workspace/CubeMaster/pkg/templatecenter/snapshot_table.go:29 record not found 9月 20 17:48:12 VM-0-6-tencentos bash5538: 0.387ms rows:0 SELECT * FROMt_cube_snapshotWHERE snapshot_id = 'tpl-464f1b5538764bd7afa240ca' ANDt_cube_snapshot.deleted_atIS NULL ORDER BYt_cube_snapshot.idLIMIT 1 9月 20 17:48:15 VM-0-6-tencentos bash6300: cube-webui | 127.0.0.1 - - 20/Sep/2026:09:48:15 +0000 "GET / HTTP/1.1" 200 1120 "-" "Wget/1.21.3" 9月 20 17:48:25 VM-0-6-tencentos bash6300: cube-webui | 127.0.0.1 - - 20/Sep/2026:09:48:25 +0000 "GET / HTTP/1.1" 200 1120 "-" "Wget/1.21.3" 9月 20 17:48:35 VM-0-6-tencentos bash6300: cube-webui | 127.0.0.1 - - 20/Sep/2026:09:48:35 +0000 "GET / HTTP/1.1" 200 1120 "-" "Wget/1.21.3" 9月 20 17:48:42 VM-0-6-tencentos bash5538: 2026/09/20 17:48:42 /workspace/CubeMaster/pkg/templatecenter/store.go:1118 record not found 9月 20 17:48:42 VM-0-6-tencentos bash5538: 0.371ms rows:0 SELECT * FROMt_cube_template_definitionWHERE template_id = 'tpl-464f1b5538764bd7afa240ca' ANDt_cube_template_definition.deleted_atIS NULL ORDER BYt_cube_template_definition.idLIMIT 1 9月 20 17:48:42 VM-0-6-tencentos bash5538: 2026/09/20 17:48:42 /workspace/CubeMaster/pkg/templatecenter/snapshot_table.go:29 record not found 9月 20 17:48:42 VM-0-6-tencentos bash5538: 0.363ms rows:0 SELECT * FROMt_cube_snapshotWHERE snapshot_id = 'tpl-464f1b5538764bd7afa240ca' ANDt_cube_snapshot.deleted_atIS NULL ORDER BYt_cube_snapshot.idLIMIT 1 9月 20 17:48:45 VM-0-6-tencentos bash6300: cube-webui | 127.0.0.1 - - 20/Sep/2026:09:48:45 +0000 "GET / HTTP/1.1" 200 1120 "-" "Wget/1.21.3" 9月 20 17:48:55 VM-0-6-tencentos bash6300: cube-webui | 127.0.0.1 - - 20/Sep/2026:09:48:55 +0000 "GET / HTTP/1.1" 200 1120 "-" "Wget/1.21.3" 9月 20 17:49:05 VM-0-6-tencentos bash6300: cube-webui | 127.0.0.1 - - 20/Sep/2026:09:49:05 +0000 "GET / HTTP/1.1" 200 1120 "-" "Wget/1.21.3" 9月 20 17:49:12 VM-0-6-tencentos bash5538: 2026/09/20 17:49:12 /workspace/CubeMaster/pkg/templatecenter/store.go:1118 record not found 9月 20 17:49:12 VM-0-6-tencentos bash5538: 0.361ms rows:0 SELECT * FROMt_cube_template_definitionWHERE template_id = 'tpl-464f1b5538764bd7afa240ca' ANDt_cube_template_definition.deleted_atIS NULL ORDER BYt_cube_template_definition.idLIMIT 1 9月 20 17:49:12 VM-0-6-tencentos bash5538: 2026/09/20 17:49:12 /workspace/CubeMaster/pkg/templatecenter/snapshot_table.go:29 record not found 9月 20 17:49:12 VM-0-6-tencentos bash5538: 0.356ms rows:0 SELECT * FROMt_cube_snapshotWHERE snapshot_id = 'tpl-464f1b5538764bd7afa240ca' ANDt_cube_snapshot.deleted_atIS NULL ORDER BYt_cube_snapshot.idLIMIT 1 9月 20 17:49:15 VM-0-6-tencentos bash6300: cube-webui | 127.0.0.1 - - 20/Sep/2026:09:49:15 +0000 "GET / HTTP/1.1" 200 1120 "-" "Wget/1.21.3" 9月 20 17:49:25 VM-0-6-tencentos bash6300: cube-webui | 127.0.0.1 - - 20/Sep/2026:09:49:25 +0000 "GET / HTTP/1.1" 200 1120 "-" "Wget/1.21.3" 9月 20 17:49:35 VM-0-6-tencentos bash6300: cube-webui | 127.0.0.1 - - 20/Sep/2026:09:49:35 +0000 "GET / HTTP/1.1" 200 1120 "-" "Wget/1.21.3" 9月 20 17:49:42 VM-0-6-tencentos bash5538: 2026/09/20 17:49:42 /workspace/CubeMaster/pkg/templatecenter/store.go:1118 record not found 9月 20 17:49:42 VM-0-6-tencentos bash5538: 0.426ms rows:0 SELECT * FROMt_cube_template_definitionWHERE template_id = 'tpl-464f1b5538764bd7afa240ca' ANDt_cube_template_definition.deleted_atIS NULL ORDER BYt_cube_template_definition.idLIMIT 1 9月 20 17:49:42 VM-0-6-tencentos bash5538: 2026/09/20 17:49:42 /workspace/CubeMaster/pkg/templatecenter/snapshot_table.go:29 record not found 9月 20 17:49:42 VM-0-6-tencentos bash5538: 0.403ms rows:0 SELECT * FROMt_cube_snapshotWHERE snapshot_id = 'tpl-464f1b5538764bd7afa240ca' ANDt_cube_snapshot.deleted_atIS NULL ORDER BYt_cube_snapshot.idLIMIT 1 9月 20 17:49:45 VM-0-6-tencentos bash6300: cube-webui | 127.0.0.1 - - 20/Sep/2026:09:49:45 +0000 "GET / HTTP/1.1" 200 1120 "-" "Wget/1.21.3" 9月 20 17:49:54 VM-0-6-tencentos bash3723: cube-sandbox-redis | 1:M 20 Sep 2026 09:49:54.070 * 100 changes in 300 seconds. Saving... 9月 20 17:49:54 VM-0-6-tencentos bash3723: cube-sandbox-redis | 1:M 20 Sep 2026 09:49:54.070 * Background saving started by pid 13477 9月 20 17:49:54 VM-0-6-tencentos bash3723: cube-sandbox-redis | 13477:C 20 Sep 2026 09:49:54.073 * DB saved on disk 9月 20 17:49:54 VM-0-6-tencentos bash3723: cube-sandbox-redis | 13477:C 20 Sep 2026 09:49:54.073 * Fork CoW for RDB: current 0 MB, peak 0 MB, average 0 MB 9月 20 17:49:54 VM-0-6-tencentos bash3723: cube-sandbox-redis | 1:M 20 Sep 2026 09:49:54.171 * Background saving terminated with success 9月 20 17:49:56 VM-0-6-tencentos bash6300: cube-webui | 127.0.0.1 - - 20/Sep/2026:09:49:56 +0000 "GET / HTTP/1.1" 200 1120 "-" "Wget/1.21.3" 9月 20 17:50:06 VM-0-6-tencentos bash6300: cube-webui | 127.0.0.1 - - 20/Sep/2026:09:50:06 +0000 "GET / HTTP/1.1" 200 1120 "-" "Wget/1.21.3" 9月 20 17:50:12 VM-0-6-tencentos bash5538: 2026/09/20 17:50:12 /workspace/CubeMaster/pkg/templatecenter/store.go:1118 record not found 9月 20 17:50:12 VM-0-6-tencentos bash5538: 0.374ms rows:0 SELECT * FROMt_cube_template_definitionWHERE template_id = 'tpl-464f1b5538764bd7afa240ca' ANDt_cube_template_definition.deleted_atIS NULL ORDER BYt_cube_template_definition.idLIMIT 1 9月 20 17:50:12 VM-0-6-tencentos bash5538: 2026/09/20 17:50:12 /workspace/CubeMaster/pkg/templatecenter/snapshot_table.go:29 record not found 9月 20 17:50:12 VM-0-6-tencentos bash5538: 0.366ms rows:0 SELECT * FROMt_cube_snapshotWHERE snapshot_id = 'tpl-464f1b5538764bd7afa240ca' ANDt_cube_snapshot.deleted_atIS NULL ORDER BYt_cube_snapshot.idLIMIT 1 9月 20 17:50:16 VM-0-6-tencentos bash6300: cube-webui | 127.0.0.1 - - 20/Sep/2026:09:50:16 +0000 "GET / HTTP/1.1" 200 1120 "-" "Wget/1.21.3" 9月 20 17:50:26 VM-0-6-tencentos bash6300: cube-webui | 127.0.0.1 - - 20/Sep/2026:09:50:26 +0000 "GET / HTTP/1.1" 200 1120 "-" "Wget/1.21.3" 9月 20 17:50:36 VM-0-6-tencentos bash6300: cube-webui | 127.0.0.1 - - 20/Sep/2026:09:50:36 +0000 "GET / HTTP/1.1" 200 1120 "-" "Wget/1.21.3" 9月 20 17:50:42 VM-0-6-tencentos bash5538: 2026/09/20 17:50:42 /workspace/CubeMaster/pkg/templatecenter/store.go:1118 record not found 9月 20 17:50:42 VM-0-6-tencentos bash5538: 0.350ms rows:0 SELECT * FROMt_cube_template_definitionWHERE template_id = 'tpl-464f1b5538764bd7afa240ca' ANDt_cube_template_definition.deleted_atIS NULL ORDER BYt_cube_template_definition.idLIMIT 1 9月 20 17:50:42 VM-0-6-tencentos bash5538: 2026/09/20 17:50:42 /workspace/CubeMaster/pkg/templatecenter/snapshot_table.go:29 record not found 9月 20 17:50:42 VM-0-6-tencentos bash5538: 0.329ms rows:0 SELECT * FROMt_cube_snapshotWHERE snapshot_id = 'tpl-464f1b5538764bd7afa240ca' ANDt_cube_snapshot.deleted_atIS NULL ORDER BYt_cube_snapshot.idLIMIT 1 9月 20 17:50:46 VM-0-6-tencentos bash6300: cube-webui | 127.0.0.1 - - 20/Sep/2026:09:50:46 +0000 "GET / HTTP/1.1" 200 1120 "-" "Wget/1.21.3" 9月 20 17:50:56 VM-0-6-tencentos bash6300: cube-webui | 127.0.0.1 - - 20/Sep/2026:09:50:56 +0000 "GET / HTTP/1.1" 200 1120 "-" "Wget/1.21.3" 9月 20 17:51:06 VM-0-6-tencentos bash6300: cube-webui | 127.0.0.1 - - 20/Sep/2026:09:51:06 +0000 "GET / HTTP/1.1" 200 1120 "-" "Wget/1.21.3" 9月 20 17:51:12 VM-0-6-tencentos bash5538: 2026/09/20 17:51:12 /workspace/CubeMaster/pkg/templatecenter/store.go:1118 record not found 9月 20 17:51:12 VM-0-6-tencentos bash5538: 0.421ms rows:0 SELECT * FROMt_cube_template_definitionWHERE template_id = 'tpl-464f1b5538764bd7afa240ca' ANDt_cube_template_definition.deleted_atIS NULL ORDER BYt_cube_template_definition.idLIMIT 1 9月 20 17:51:12 VM-0-6-tencentos bash5538: 2026/09/20 17:51:12 /workspace/CubeMaster/pkg/templatecenter/snapshot_table.go:29 record not found 9月 20 17:51:12 VM-0-6-tencentos bash5538: 0.366ms rows:0 SELECT * FROMt_cube_snapshotWHERE snapshot_id = 'tpl-464f1b5538764bd7afa240ca' ANDt_cube_snapshot.deleted_atIS NULL ORDER BYt_cube_snapshot.idLIMIT 1 9月 20 17:51:16 VM-0-6-tencentos bash6300: cube-webui | 127.0.0.1 - - 20/Sep/2026:09:51:16 +0000 "GET / HTTP/1.1" 200 1120 "-" "Wget/1.21.3" 9月 20 17:51:26 VM-0-6-tencentos bash6300: cube-webui | 127.0.0.1 - - 20/Sep/2026:09:51:26 +0000 "GET / HTTP/1.1" 200 1120 "-" "Wget/1.21.3" 9月 20 17:51:36 VM-0-6-tencentos bash6300: cube-webui | 127.0.0.1 - - 20/Sep/2026:09:51:36 +0000 "GET / HTTP/1.1" 200 1120 "-" "Wget/1.21.3" 9月 20 17:51:42 VM-0-6-tencentos bash5538: 2026/09/20 17:51:42 /workspace/CubeMaster/pkg/templatecenter/store.go:1118 record not found 9月 20 17:51:42 VM-0-6-tencentos bash5538: 0.386ms rows:0 SELECT * FROMt_cube_template_definitionWHERE template_id = 'tpl-464f1b5538764bd7afa240ca' ANDt_cube_template_definition.deleted_atIS NULL ORDER BYt_cube_template_definition.idLIMIT 1 9月 20 17:51:42 VM-0-6-tencentos bash5538: 2026/09/20 17:51:42 /workspace/CubeMaster/pkg/templatecenter/snapshot_table.go:29 record not found 9月 20 17:51:42 VM-0-6-tencentos bash5538: 0.384ms rows:0 SELECT * FROMt_cube_snapshotWHERE snapshot_id = 'tpl-464f1b5538764bd7afa240ca' ANDt_cube_snapshot.deleted_atIS NULL ORDER BYt_cube_snapshot.idLIMIT 1 9月 20 17:51:46 VM-0-6-tencentos bash6300: cube-webui | 127.0.0.1 - - 20/Sep/2026:09:51:46 +0000 "GET / HTTP/1.1" 200 1120 "-" "Wget/1.21.3" 9月 20 17:51:56 VM-0-6-tencentos bash6300: cube-webui | 127.0.0.1 - - 20/Sep/2026:09:51:56 +0000 "GET / HTTP/1.1" 200 1120 "-" "Wget/1.21.3" 9月 20 17:52:06 VM-0-6-tencentos bash6300: cube-webui | 127.0.0.1 - - 20/Sep/2026:09:52:06 +0000 "GET / HTTP/1.1" 200 1120 "-" "Wget/1.21.3" 9月 20 17:52:12 VM-0-6-tencentos bash5538: 2026/09/20 17:52:12 /workspace/CubeMaster/pkg/templatecenter/store.go:1118 record not found 9月 20 17:52:12 VM-0-6-tencentos bash5538: 0.369ms rows:0 SELECT * FROMt_cube_template_definitionWHERE template_id = 'tpl-464f1b5538764bd7afa240ca' ANDt_cube_template_definition.deleted_atIS NULL ORDER BYt_cube_template_definition.idLIMIT 1 9月 20 17:52:12 VM-0-6-tencentos bash5538: 2026/09/20 17:52:12 /workspace/CubeMaster/pkg/templatecenter/snapshot_table.go:29 record not found 9月 20 17:52:12 VM-0-6-tencentos bash5538: 0.375ms rows:0 SELECT * FROMt_cube_snapshotWHERE snapshot_id = 'tpl-464f1b5538764bd7afa240ca' ANDt_cube_snapshot.deleted_atIS NULL ORDER BYt_cube_snapshot.idLIMIT 1 9月 20 17:52:16 VM-0-6-tencentos bash6300: cube-webui | 127.0.0.1 - - 20/Sep/2026:09:52:16 +0000 "GET / HTTP/1.1" 200 1120 "-" "Wget/1.21.3" 9月 20 17:52:26 VM-0-6-tencentos bash6300: cube-webui | 127.0.0.1 - - 20/Sep/2026:09:52:26 +0000 "GET / HTTP/1.1" 200 1120 "-" "Wget/1.21.3" 9月 20 17:52:36 VM-0-6-tencentos bash6300: cube-webui | 127.0.0.1 - - 20/Sep/2026:09:52:36 +0000 "GET / HTTP/1.1" 200 1120 "-" "Wget/1.21.3" 9月 20 17:52:42 VM-0-6-tencentos bash5538: 2026/09/20 17:52:42 /workspace/CubeMaster/pkg/templatecenter/store.go:1118 record not found 9月 20 17:52:42 VM-0-6-tencentos bash5538: 0.341ms rows:0 SELECT * FROMt_cube_template_definitionWHERE template_id = 'tpl-464f1b5538764bd7afa240ca' ANDt_cube_template_definition.deleted_atIS NULL ORDER BYt_cube_template_definition.idLIMIT 1 9月 20 17:52:42 VM-0-6-tencentos bash5538: 2026/09/20 17:52:42 /workspace/CubeMaster/pkg/templatecenter/snapshot_table.go:29 record not found 9月 20 17:52:42 VM-0-6-tencentos bash5538: 0.325ms rows:0 SELECT * FROMt_cube_snapshotWHERE snapshot_id = 'tpl-464f1b5538764bd7afa240ca' ANDt_cube_snapshot.deleted_atIS NULL ORDER BYt_cube_snapshot.idLIMIT 1 9月 20 17:52:46 VM-0-6-tencentos bash6300: cube-webui | 127.0.0.1 - - 20/Sep/2026:09:52:46 +0000 "GET / HTTP/1.1" 200 1120 "-" "Wget/1.21.3" 9月 20 17:52:56 VM-0-6-tencentos bash6300: cube-webui | 127.0.0.1 - - 20/Sep/2026:09:52:56 +0000 "GET / HTTP/1.1" 200 1120 "-" "Wget/1.21.3" 9月 20 17:53:06 VM-0-6-tencentos bash6300: cube-webui | 127.0.0.1 - - 20/Sep/2026:09:53:06 +0000 "GET / HTTP/1.1" 200 1120 "-" "Wget/1.21.3" 9月 20 17:53:12 VM-0-6-tencentos bash5538: 2026/09/20 17:53:12 /workspace/CubeMaster/pkg/templatecenter/store.go:1118 record not found 9月 20 17:53:12 VM-0-6-tencentos bash5538: 0.368ms rows:0 SELECT * FROMt_cube_template_definitionWHERE template_id = 'tpl-464f1b5538764bd7afa240ca' ANDt_cube_template_definition.deleted_atIS NULL ORDER BYt_cube_template_definition.idLIMIT 1 9月 20 17:53:12 VM-0-6-tencentos bash5538: 2026/09/20 17:53:12 /workspace/CubeMaster/pkg/templatecenter/snapshot_table.go:29 record not found 9月 20 17:53:12 VM-0-6-tencentos bash5538: 0.382ms rows:0 SELECT * FROMt_cube_snapshotWHERE snapshot_id = 'tpl-464f1b5538764bd7afa240ca' ANDt_cube_snapshot.deleted_atIS NULL ORDER BYt_cube_snapshot.idLIMIT 1 9月 20 17:53:16 VM-0-6-tencentos bash6300: cube-webui | 127.0.0.1 - - 20/Sep/2026:09:53:16 +0000 "GET / HTTP/1.1" 200 1120 "-" "Wget/1.21.3" 9月 20 17:53:26 VM-0-6-tencentos bash6300: cube-webui | 127.0.0.1 - - 20/Sep/2026:09:53:26 +0000 "GET / HTTP/1.1" 200 1120 "-" "Wget/1.21.3" root@VM-0-6-tencentos \~# docker ps | grep cube 5d9ef03bf1ed cube-sandbox-cn.tencentcloudcr.com/cube-sandbox/cube-egress:v0.7.1 "/usr/local/openrest..." About an hour ago Up About an hour cube-egress d5c0c2c48c92 cube-sandbox-image.tencentcloudcr.com/opensource/openresty:1.21.4.1-6-alpine-fat "/usr/local/openrest..." About an hour ago Up About an hour (healthy) 0.0.0.0:12088->80/tcp cube-webui 5041bbb48a0f cube-sandbox-cn.tencentcloudcr.com/cube-sandbox/cube-proxy:v0.7.1 "/usr/local/openrest..." About an hour ago Up About an hour cube-proxy 459a081ffa37 cube-sandbox-cn.tencentcloudcr.com/cube-sandbox/cube-lifecycle-manager:v0.7.1 "/usr/local/bin/cube..." About an hour ago Up About an hour cube-lifecycle-manager 8d12b0ccbb0b cube-sandbox-image.tencentcloudcr.com/opensource/redis:7-alpine "docker-entrypoint.s..." About an hour ago Up About an hour (healthy) 127.0.0.1:6379->6379/tcp cube-sandbox-redis 76096c049704 cube-sandbox-image.tencentcloudcr.com/opensource/mysql:8.0 "docker-entrypoint.s..." About an hour ago Up About an hour (healthy) 127.0.0.1:3306->3306/tcp, 33060/tcp cube-sandbox-mysql 861435ecd634 cube-sandbox-cn.tencentcloudcr.com/cube-sandbox/minio:RELEASE.2025-09-07T16-13-09Z "/usr/bin/docker-ent..." About an hour ago Up About an hour (healthy) 192.168.0.6:9000->9000/tcp, 127.0.0.1:9001->9001/tcp cube-sandbox-minio a0dd3c78f388 cube-sandbox-image.tencentcloudcr.com/opensource/coredns/coredns:1.14.2 "/coredns -conf /etc..." About an hour ago Up About an hour cube-proxy-coredns root@VM-0-6-tencentos \~# grep -R "ceaa39fc-aa7b-43d4-974a-1ef430a074f7" /var/log 2>/dev/null root@VM-0-6-tencentos \~#
根据贴出的日志,已经可以缩小范围:
CubeSandbox 控制面正常运行,但模板构建任务记录已经丢失,没有留下失败原因
关键日志:
SELECT * FROM `t_cube_template_definition`
WHERE template_id = 'tpl-464f1b5538764bd7afa240ca'
...
record not found
以及:
SELECT * FROM `t_cube_snapshot`
WHERE snapshot_id = 'tpl-464f1b5538764bd7afa240ca'
...
record not found
说明:
- WebUI 请求模板详情
- CubeMaster 查询模板定义
- MySQL 中已经没有该模板记录
- snapshot 记录也不存在
所以现在看到的:
模板构建失败
无法查看详情
不是当前实时错误,而是构建任务失败后清理了数据库记录
当前重点:找真正的 builder 日志
你的 systemd 列表里有:
cube-sandbox-cubelet.service
cube-sandbox-cubemaster.service
cube-sandbox-cube-templatecenter.service
cube-sandbox-cube-api.service
cube-sandbox-cubeops.service
模板构建通常不是 WebUI/CubeMaster 报错,而是在:
- cubelet
- lifecycle-manager
- templatecenter
里面
1. 查看 cubelet 日志(优先)
执行:
journalctl -u cube-sandbox-cubelet -n 500 --no-pager
过滤:
journalctl -u cube-sandbox-cubelet --no-pager | \
grep -Ei "template|snapshot|build|error|failed|harbor|image"
2. 查看 lifecycle-manager
模板创建任务通常由它执行:
journalctl -u cube-sandbox-cube-lifecycle-manager -n 500 --no-pager
过滤:
journalctl -u cube-sandbox-cube-lifecycle-manager --no-pager | \
grep -Ei "template|snapshot|error|failed"
3. 查看 TemplateCenter
journalctl -u cube-sandbox-cube-templatecenter -n 500 --no-pager
4. Docker 容器日志
你的容器:
459a081ffa37 cube-lifecycle-manager
查看:
docker logs 459a081ffa37 --tail 500
重点:
docker logs 459a081ffa37 2>&1 | \
grep -Ei "template|snapshot|error|failed"
5. 另外发现一个可能问题
你的模板来源之前截图:
harbor-intranet.allintechinc.com/llm-planning/...
但当前:
docker ps
没有看到任何 builder 容器
正常模板构建期间应该可能出现:
- sandbox builder
- qemu
- guest
- converter
现在全部没有
可能情况:
情况A
构建任务启动后立即失败
常见原因:
- Harbor 拉取失败
- 镜像权限失败
- 镜像格式不支持
情况B
PVM 创建失败
你之前安装:
CUBE_PVM_ENABLE=1
重点检查:
lsmod | grep kvm
和:
ls -l /dev/kvm
6. 检查 MinIO 是否有模板残留
因为模板 snapshot 存 MinIO:
docker exec -it cube-sandbox-minio sh
或者:
mc alias ls
mc ls <alias>/cube-volumes
看看是否有:
tpl-464f1b5538764bd7afa240ca
7. 建议下一步直接执行这几个
一次贴回来:
journalctl -u cube-sandbox-cubelet -n 200 --no-pager
journalctl -u cube-sandbox-cube-lifecycle-manager -n 200 --no-pager
docker logs 459a081ffa37 --tail 200
我预计错误会在这里出现
目前可以排除:
✅ CubeSandbox 主服务启动问题
✅ MySQL/Redis异常
✅ WebUI异常
重点怀疑:
- Harbor 镜像拉取失败
- PVM guest 创建失败
- 模板 snapshot 生成失败
- MinIO 写入失败
下一批日志基本可以直接定位。
继续排查。你当前贴的日志已经证明:
cube-sandbox-*控制面正常 ✅- MySQL/Redis/MinIO 容器正常运行 ✅
- 失败模板的数据库记录已经被清理 ❗
- 真正失败点发生在模板构建执行阶段,不在 WebUI
下一步需要抓 构建执行组件日志。
请执行下面命令:
1. Cubelet(最重要)
journalctl -u cube-sandbox-cubelet -n 500 --no-pager
重点看:
template
snapshot
pull
image
qemu
kvm
mount
error
failed
也可以过滤:
journalctl -u cube-sandbox-cubelet --no-pager | \
grep -Ei "template|snapshot|image|pull|qemu|kvm|error|failed"
2. Lifecycle Manager
模板生命周期一般由它调度:
journalctl -u cube-sandbox-cube-lifecycle-manager -n 500 --no-pager
过滤:
journalctl -u cube-sandbox-cube-lifecycle-manager --no-pager | \
grep -Ei "template|snapshot|build|error|failed"
3. TemplateCenter
journalctl -u cube-sandbox-cube-templatecenter -n 300 --no-pager
4. 查看是否有临时构建容器
模板失败后可能已经退出:
docker ps -a | grep -Ei "cube|template|builder|qemu"
5. 检查 PVM 环境
因为你安装用了:
CUBE_PVM_ENABLE=1
执行:
lsmod | grep kvm
以及:
ls -l /dev/kvm
正常应该类似:
/dev/kvm
6. 检查 Harbor 连通性
你的模板镜像来自:
harbor-intranet.allintechinc.com/llm-planning/...
执行:
curl -k https://harbor-intranet.allintechinc.com/v2/
正常:
{}
如果:
401 Unauthorized
说明需要配置 Harbor 登录
我目前最怀疑的两个点
可能1:Harbor 私有镜像无法拉取
流程:
创建模板
↓
CubeSandbox
↓
pull Harbor image
↓
失败
↓
删除 template/snapshot记录
符合你现在现象
可能2:PVM 模板转换失败
因为:
CUBE_PVM_ENABLE=1
如果:
- kernel 不匹配
- /dev/kvm 不可用
- guest kernel 创建失败
也会出现模板直接失败
journalctl -u cube-sandbox-cubelet -n 200 --no-pager journalctl -u cube-sandbox-cube-lifecycle-manager -n 200 --no-pager docker ps -a | grep -Ei "cube|template|builder|qemu" 9月 20 15:36:05 VM-0-6-tencentos systemd1: Starting cube-sandbox-cubelet.service - Cube Sandbox Cubelet... 9月 20 15:36:05 VM-0-6-tencentos bash5832: one-click-systemd updated cubelet cubeops_addr=http://127.0.0.1:3010 9月 20 15:36:05 VM-0-6-tencentos bash5922: one-click-systemd updated cubelet cubeops_addr=http://127.0.0.1:3010 9月 20 15:36:06 VM-0-6-tencentos bash5875: one-click-systemd updated cubelet cow.s3 enable=false 9月 20 15:36:06 VM-0-6-tencentos bash5875: one-click-systemd cubelet initial pid=6023 9月 20 15:36:06 VM-0-6-tencentos bash7079: 2026-09-20T15:36:06.584521396+08:00,listen file/usr/local/services/cubetoolbox/Cubelet/dynamicconf/conf.yaml 9月 20 15:36:06 VM-0-6-tencentos bash7079: cfg:{"Common":{"CommonTimeout":10000000000,"LogLevel":"","DescribeAsyncInterval":100000000,"EnablePFMode":false,"DescribeBDFInterval":10000000,"DescribeBDFTimeout":10000000000,"GetBDFByUuidCmd":"/usr/local/services/AdamPlugins-1.0/snhost_snic_sdk/cath/bm/tools/get_bdf_by_uuid","GetBDFByIfNameCmd":"/usr/local/services/AdamPlugins-1.0/snhost_snic_sdk/cath/bm/tools/get_bdf_by_ifname","CommandTimeout":1000000000,"DisableHostCgroup":true,"DisableVmCgroup":false,"EnableSandboxExecCmdBeforeExist":false,"SandboxExecCmdTimeOut":5000000000,"SandboxExecCmdBeforeExist":"sh","/usr/local/cubetools/cube_exec_cmd_before_exist.sh","SandboxExecCmdBeforeExistLogOut":false,"SandboxExecCmdOutMaxLines":2000,"SandboxExecCmdMatchLine":"","SandboxExecCmdAfterMatch":null,"SandboxExecCmdBase64AfterMatch":"","SandboxExecCmdAfterMatchLogOut":false,"CgroupDisableMemoryReparentFile":"","CgroupDisableCpusetList":"","DisableHostNetfile":true,"DefaultDNSServers":"119.29.29.29","DefaultDNSSearches":\[\],"DefaultDNSOptions":\[\],"ReconcileInterval":300000000000,"DisableCubeBoxTemplateBaseFormatPoolOfNumberVer":false},"Tenant":{"Tenants":{},"UseRespectiveNamespace":false,"DefaultNS":"default","ZiyanUinListStr":""},"MetaServerConfig":{"MetaServerEndpoint":"127.0.0.1:3010","NodeStatusMaxImages":40000,"CubeMasterHTTPAddr":"127.0.0.1:8089"},"HostConf":{"Quota":{"Cpu":0,"Mem":"","MvmLimit":0,"CreationConcurrentNum":0,"PausedResourceReleaseRatio":0},"GC":{"CodeExpirationTime":"72h","ImageExpirationTime":"24h"},"SchedulerLabel":"default-cluster"}} 9月 20 15:36:06 VM-0-6-tencentos bash7079: 2026/09/20 15:36:06 mounted state tmpfs /data/cubelet/state size=1024m 9月 20 15:36:06 VM-0-6-tencentos bash7079: 2026/09/20 15:36:06 set /data/cubelet/root as shared mount 9月 20 15:36:07 VM-0-6-tencentos bash5875: /usr/local/services/cubetoolbox/scripts/systemd/cubelet-start.sh: line 57: 6023 Killed ( export CUBE_SANDBOX_NODE_IP="{CUBE_SANDBOX_NODE_IP}"; exec "{CUBELET_BIN}" --config "{CUBELET_CONFIG}" --dynamic-conf-path "{CUBELET_DYNAMICCONF}" ) 9月 20 15:36:07 VM-0-6-tencentos bash5875: one-click-systemd cubelet stable pid=7079 9月 20 15:36:07 VM-0-6-tencentos systemd1: Started cube-sandbox-cubelet.service - Cube Sandbox Cubelet. 9月 20 16:18:54 VM-0-6-tencentos systemd1: Stopping cube-sandbox-cubelet.service - Cube Sandbox Cubelet... 9月 20 16:18:55 VM-0-6-tencentos bash87481: one-click-systemd cubelet stopped pid=7079 9月 20 16:18:55 VM-0-6-tencentos systemd1: cube-sandbox-cubelet.service: Deactivated successfully. 9月 20 16:18:55 VM-0-6-tencentos systemd1: Stopped cube-sandbox-cubelet.service - Cube Sandbox Cubelet. 9月 20 16:18:55 VM-0-6-tencentos systemd1: cube-sandbox-cubelet.service: Consumed 10.594s CPU time, 237.9M memory peak, 0B memory swap peak. -- Boot f6f2fcd1381547d990b4d2b666efa29d -- 9月 20 16:19:13 VM-0-6-tencentos systemd1: Starting cube-sandbox-cubelet.service - Cube Sandbox Cubelet... 9月 20 16:19:13 VM-0-6-tencentos bash1510: one-click-systemd updated cubelet cubeops_addr=http://127.0.0.1:3010 9月 20 16:19:13 VM-0-6-tencentos bash1587: one-click-systemd updated cubelet cubeops_addr=http://127.0.0.1:3010 9月 20 16:19:13 VM-0-6-tencentos bash1566: one-click-systemd updated cubelet cow.s3 enable=false 9月 20 16:19:13 VM-0-6-tencentos bash1566: one-click-systemd cubelet initial pid=1668 9月 20 16:19:14 VM-0-6-tencentos bash2125: 2026-09-20T16:19:14.223712774+08:00,listen file/usr/local/services/cubetoolbox/Cubelet/dynamicconf/conf.yaml 9月 20 16:19:14 VM-0-6-tencentos bash2125: cfg:{"Common":{"CommonTimeout":10000000000,"LogLevel":"","DescribeAsyncInterval":100000000,"EnablePFMode":false,"DescribeBDFInterval":10000000,"DescribeBDFTimeout":10000000000,"GetBDFByUuidCmd":"/usr/local/services/AdamPlugins-1.0/snhost_snic_sdk/cath/bm/tools/get_bdf_by_uuid","GetBDFByIfNameCmd":"/usr/local/services/AdamPlugins-1.0/snhost_snic_sdk/cath/bm/tools/get_bdf_by_ifname","CommandTimeout":1000000000,"DisableHostCgroup":true,"DisableVmCgroup":false,"EnableSandboxExecCmdBeforeExist":false,"SandboxExecCmdTimeOut":5000000000,"SandboxExecCmdBeforeExist":"sh","/usr/local/cubetools/cube_exec_cmd_before_exist.sh","SandboxExecCmdBeforeExistLogOut":false,"SandboxExecCmdOutMaxLines":2000,"SandboxExecCmdMatchLine":"","SandboxExecCmdAfterMatch":null,"SandboxExecCmdBase64AfterMatch":"","SandboxExecCmdAfterMatchLogOut":false,"CgroupDisableMemoryReparentFile":"","CgroupDisableCpusetList":"","DisableHostNetfile":true,"DefaultDNSServers":"119.29.29.29","DefaultDNSSearches":\[\],"DefaultDNSOptions":\[\],"ReconcileInterval":300000000000,"DisableCubeBoxTemplateBaseFormatPoolOfNumberVer":false},"Tenant":{"Tenants":{},"UseRespectiveNamespace":false,"DefaultNS":"default","ZiyanUinListStr":""},"MetaServerConfig":{"MetaServerEndpoint":"127.0.0.1:3010","NodeStatusMaxImages":40000,"CubeMasterHTTPAddr":"127.0.0.1:8089"},"HostConf":{"Quota":{"Cpu":0,"Mem":"","MvmLimit":0,"CreationConcurrentNum":0,"PausedResourceReleaseRatio":0},"GC":{"CodeExpirationTime":"72h","ImageExpirationTime":"24h"},"SchedulerLabel":"default-cluster"}} 9月 20 16:19:14 VM-0-6-tencentos bash2125: 2026/09/20 16:19:14 mounted state tmpfs /data/cubelet/state size=1024m 9月 20 16:19:14 VM-0-6-tencentos bash2125: 2026/09/20 16:19:14 set /data/cubelet/root as shared mount 9月 20 16:19:14 VM-0-6-tencentos bash1566: /usr/local/services/cubetoolbox/scripts/systemd/cubelet-start.sh: line 57: 1668 Killed ( export CUBE_SANDBOX_NODE_IP="{CUBE_SANDBOX_NODE_IP}"; exec "{CUBELET_BIN}" --config "{CUBELET_CONFIG}" --dynamic-conf-path "{CUBELET_DYNAMICCONF}" ) 9月 20 16:19:14 VM-0-6-tencentos bash1566: one-click-systemd cubelet stable pid=2125 9月 20 16:19:14 VM-0-6-tencentos systemd1: Started cube-sandbox-cubelet.service - Cube Sandbox Cubelet. 9月 20 16:24:13 VM-0-6-tencentos systemd1: Stopping cube-sandbox-cubelet.service - Cube Sandbox Cubelet... 9月 20 16:24:14 VM-0-6-tencentos bash18750: one-click-systemd cubelet stopped pid=2125 9月 20 16:24:14 VM-0-6-tencentos systemd1: cube-sandbox-cubelet.service: Deactivated successfully. 9月 20 16:24:14 VM-0-6-tencentos systemd1: Stopped cube-sandbox-cubelet.service - Cube Sandbox Cubelet. 9月 20 16:24:14 VM-0-6-tencentos systemd1: cube-sandbox-cubelet.service: Consumed 11.631s CPU time. -- Boot 0756528841154b8595d93af011d03619 -- 9月 20 16:24:35 VM-0-6-tencentos systemd1: Starting cube-sandbox-cubelet.service - Cube Sandbox Cubelet... 9月 20 16:24:35 VM-0-6-tencentos bash1505: one-click-systemd updated cubelet cubeops_addr=http://127.0.0.1:3010 9月 20 16:24:35 VM-0-6-tencentos bash1608: one-click-systemd updated cubelet cubeops_addr=http://127.0.0.1:3010 9月 20 16:24:35 VM-0-6-tencentos bash1580: one-click-systemd updated cubelet cow.s3 enable=false 9月 20 16:24:35 VM-0-6-tencentos bash1580: one-click-systemd cubelet initial pid=1710 9月 20 16:24:35 VM-0-6-tencentos bash1949: 2026-09-20T16:24:35.552388365+08:00,listen file/usr/local/services/cubetoolbox/Cubelet/dynamicconf/conf.yaml 9月 20 16:24:35 VM-0-6-tencentos bash1949: cfg:{"Common":{"CommonTimeout":10000000000,"LogLevel":"","DescribeAsyncInterval":100000000,"EnablePFMode":false,"DescribeBDFInterval":10000000,"DescribeBDFTimeout":10000000000,"GetBDFByUuidCmd":"/usr/local/services/AdamPlugins-1.0/snhost_snic_sdk/cath/bm/tools/get_bdf_by_uuid","GetBDFByIfNameCmd":"/usr/local/services/AdamPlugins-1.0/snhost_snic_sdk/cath/bm/tools/get_bdf_by_ifname","CommandTimeout":1000000000,"DisableHostCgroup":true,"DisableVmCgroup":false,"EnableSandboxExecCmdBeforeExist":false,"SandboxExecCmdTimeOut":5000000000,"SandboxExecCmdBeforeExist":"sh","/usr/local/cubetools/cube_exec_cmd_before_exist.sh","SandboxExecCmdBeforeExistLogOut":false,"SandboxExecCmdOutMaxLines":2000,"SandboxExecCmdMatchLine":"","SandboxExecCmdAfterMatch":null,"SandboxExecCmdBase64AfterMatch":"","SandboxExecCmdAfterMatchLogOut":false,"CgroupDisableMemoryReparentFile":"","CgroupDisableCpusetList":"","DisableHostNetfile":true,"DefaultDNSServers":"119.29.29.29","DefaultDNSSearches":\[\],"DefaultDNSOptions":\[\],"ReconcileInterval":300000000000,"DisableCubeBoxTemplateBaseFormatPoolOfNumberVer":false},"Tenant":{"Tenants":{},"UseRespectiveNamespace":false,"DefaultNS":"default","ZiyanUinListStr":""},"MetaServerConfig":{"MetaServerEndpoint":"127.0.0.1:3010","NodeStatusMaxImages":40000,"CubeMasterHTTPAddr":"127.0.0.1:8089"},"HostConf":{"Quota":{"Cpu":0,"Mem":"","MvmLimit":0,"CreationConcurrentNum":0,"PausedResourceReleaseRatio":0},"GC":{"CodeExpirationTime":"72h","ImageExpirationTime":"24h"},"SchedulerLabel":"default-cluster"}} 9月 20 16:24:35 VM-0-6-tencentos bash1949: 2026/09/20 16:24:35 mounted state tmpfs /data/cubelet/state size=1024m 9月 20 16:24:35 VM-0-6-tencentos bash1949: 2026/09/20 16:24:35 set /data/cubelet/root as shared mount 9月 20 16:24:36 VM-0-6-tencentos bash1580: /usr/local/services/cubetoolbox/scripts/systemd/cubelet-start.sh: line 57: 1710 Killed ( export CUBE_SANDBOX_NODE_IP="{CUBE_SANDBOX_NODE_IP}"; exec "{CUBELET_BIN}" --config "{CUBELET_CONFIG}" --dynamic-conf-path "{CUBELET_DYNAMICCONF}" ) 9月 20 16:24:36 VM-0-6-tencentos bash1580: one-click-systemd cubelet stable pid=1949 9月 20 16:24:36 VM-0-6-tencentos systemd1: Started cube-sandbox-cubelet.service - Cube Sandbox Cubelet. 9月 20 15:36:26 VM-0-6-tencentos systemd1: Starting cube-sandbox-cube-lifecycle-manager.service - Cube Sandbox cube-lifecycle-manager container... 9月 20 15:36:26 VM-0-6-tencentos bash9297: one-click-runtime cube-lifecycle-manager runtime files prepared under /usr/local/services/cubetoolbox/cube-lifecycle-manager 9月 20 15:36:26 VM-0-6-tencentos systemd1: Started cube-sandbox-cube-lifecycle-manager.service - Cube Sandbox cube-lifecycle-manager container. 9月 20 15:36:26 VM-0-6-tencentos bash9375: one-click-runtime pulling cube-sandbox-cn.tencentcloudcr.com/cube-sandbox/cube-lifecycle-manager:v0.7.1 9月 20 15:36:35 VM-0-6-tencentos bash11195: time="2026-09-20T15:36:35+08:00" level=warning msg="/usr/local/services/cubetoolbox/cube-lifecycle-manager/docker-compose.yaml: the attribute
versionis obsolete, it will be ignored, please remove it to avoid potential confusion" 9月 20 15:36:35 VM-0-6-tencentos bash11195: Container cube-lifecycle-manager Creating 9月 20 15:36:35 VM-0-6-tencentos bash11195: Container cube-lifecycle-manager Created 9月 20 15:36:35 VM-0-6-tencentos bash11195: Attaching to cube-lifecycle-manager 9月 20 15:36:35 VM-0-6-tencentos bash11195: Container cube-lifecycle-manager Starting 9月 20 15:36:36 VM-0-6-tencentos bash11195: cube-lifecycle-manager | {"level":"info","ts":1789889796.0163305,"caller":"cube-lifecycle-manager/main.go:59","msg":"cube-lifecycle-manager starting","redis_addr":"127.0.0.1:6379","cube_proxy_admin_urls":"http://127.0.0.1:8082","cubemaster_url":"http://192.168.0.6:8089","listen_addr":"0.0.0.0:8083","instance_identity":"VM-0-6-tencentos","leader_election_enabled":false} 9月 20 15:36:36 VM-0-6-tencentos bash11195: cube-lifecycle-manager | {"level":"info","ts":1789889796.0164294,"caller":"cube-lifecycle-manager/main.go:81","msg":"eventbus enabled"} 9月 20 15:36:36 VM-0-6-tencentos bash11195: cube-lifecycle-manager | {"level":"info","ts":1789889796.0179007,"caller":"cube-lifecycle-manager/main.go:641","msg":"bootstrap complete","entries":0} 9月 20 15:36:36 VM-0-6-tencentos bash11195: cube-lifecycle-manager | {"level":"info","ts":1789889796.0205412,"logger":"http","caller":"httpapi/server.go:86","msg":"cube-lifecycle-manager http server listening","addr":"0.0.0.0:8083"} 9月 20 15:36:36 VM-0-6-tencentos bash11195: Container cube-lifecycle-manager Started 9月 20 15:36:48 VM-0-6-tencentos bash11195: cube-lifecycle-manager | {"level":"info","ts":1789889808.0200999,"logger":"discovery","caller":"discovery/redis.go:182","msg":"discovery: proxy joined","proxy_id":"192.168.0.6:8082","admin_url":"http://192.168.0.6:8082"} 9月 20 15:36:48 VM-0-6-tencentos bash11195: cube-lifecycle-manager | {"level":"info","ts":1789889808.0201688,"logger":"replay","caller":"cube-lifecycle-manager/main.go:366","msg":"replay begin","proxy_id":"192.168.0.6:8082","admin_url":"http://192.168.0.6:8082","entries":0} 9月 20 15:36:48 VM-0-6-tencentos bash11195: cube-lifecycle-manager | {"level":"info","ts":1789889808.0202017,"logger":"replay","caller":"cube-lifecycle-manager/main.go:394","msg":"replay done","proxy_id":"192.168.0.6:8082","pushed":0,"failed":0} 9月 20 16:18:54 VM-0-6-tencentos systemd1: Stopping cube-sandbox-cube-lifecycle-manager.service - Cube Sandbox cube-lifecycle-manager container... 9月 20 16:18:54 VM-0-6-tencentos bash11195: 45B blob data 9月 20 16:18:54 VM-0-6-tencentos bash87476: one-click-runtime cube-lifecycle-manager stopped 9月 20 16:18:54 VM-0-6-tencentos systemd1: cube-sandbox-cube-lifecycle-manager.service: Deactivated successfully. 9月 20 16:18:54 VM-0-6-tencentos systemd1: Stopped cube-sandbox-cube-lifecycle-manager.service - Cube Sandbox cube-lifecycle-manager container. 9月 20 16:18:54 VM-0-6-tencentos systemd1: cube-sandbox-cube-lifecycle-manager.service: Consumed 1.073s CPU time, 29.4M memory peak, 0B memory swap peak. -- Boot f6f2fcd1381547d990b4d2b666efa29d -- 9月 20 16:19:20 VM-0-6-tencentos systemd1: Starting cube-sandbox-cube-lifecycle-manager.service - Cube Sandbox cube-lifecycle-manager container... 9月 20 16:19:20 VM-0-6-tencentos bash7085: one-click-runtime cube-lifecycle-manager runtime files prepared under /usr/local/services/cubetoolbox/cube-lifecycle-manager 9月 20 16:19:20 VM-0-6-tencentos systemd1: Started cube-sandbox-cube-lifecycle-manager.service - Cube Sandbox cube-lifecycle-manager container. 9月 20 16:19:20 VM-0-6-tencentos bash7170: one-click-runtime pulling cube-sandbox-cn.tencentcloudcr.com/cube-sandbox/cube-lifecycle-manager:v0.7.1 9月 20 16:19:21 VM-0-6-tencentos bash7978: time="2026-09-20T16:19:21+08:00" level=warning msg="/usr/local/services/cubetoolbox/cube-lifecycle-manager/docker-compose.yaml: the attributeversionis obsolete, it will be ignored, please remove it to avoid potential confusion" 9月 20 16:19:21 VM-0-6-tencentos bash7978: Container cube-lifecycle-manager Creating 9月 20 16:19:21 VM-0-6-tencentos bash7978: Container cube-lifecycle-manager Created 9月 20 16:19:21 VM-0-6-tencentos bash7978: Attaching to cube-lifecycle-manager 9月 20 16:19:21 VM-0-6-tencentos bash7978: Container cube-lifecycle-manager Starting 9月 20 16:19:21 VM-0-6-tencentos bash7978: Container cube-lifecycle-manager Started 9月 20 16:19:21 VM-0-6-tencentos bash7978: cube-lifecycle-manager | {"level":"info","ts":1789892361.5976703,"caller":"cube-lifecycle-manager/main.go:59","msg":"cube-lifecycle-manager starting","redis_addr":"127.0.0.1:6379","cube_proxy_admin_urls":"http://127.0.0.1:8082","cubemaster_url":"http://192.168.0.6:8089","listen_addr":"0.0.0.0:8083","instance_identity":"VM-0-6-tencentos","leader_election_enabled":false} 9月 20 16:19:21 VM-0-6-tencentos bash7978: cube-lifecycle-manager | {"level":"info","ts":1789892361.600487,"caller":"cube-lifecycle-manager/main.go:81","msg":"eventbus enabled"} 9月 20 16:19:21 VM-0-6-tencentos bash7978: cube-lifecycle-manager | {"level":"info","ts":1789892361.6083386,"caller":"cube-lifecycle-manager/main.go:641","msg":"bootstrap complete","entries":0} 9月 20 16:19:21 VM-0-6-tencentos bash7978: cube-lifecycle-manager | {"level":"info","ts":1789892361.6112247,"logger":"http","caller":"httpapi/server.go:86","msg":"cube-lifecycle-manager http server listening","addr":"0.0.0.0:8083"} 9月 20 16:19:24 VM-0-6-tencentos bash7978: cube-lifecycle-manager | {"level":"info","ts":1789892364.61076,"logger":"discovery","caller":"discovery/redis.go:182","msg":"discovery: proxy joined","proxy_id":"192.168.0.6:8082","admin_url":"http://192.168.0.6:8082"} 9月 20 16:19:24 VM-0-6-tencentos bash7978: cube-lifecycle-manager | {"level":"info","ts":1789892364.6108475,"logger":"replay","caller":"cube-lifecycle-manager/main.go:366","msg":"replay begin","proxy_id":"192.168.0.6:8082","admin_url":"http://192.168.0.6:8082","entries":0} 9月 20 16:19:24 VM-0-6-tencentos bash7978: cube-lifecycle-manager | {"level":"info","ts":1789892364.610934,"logger":"replay","caller":"cube-lifecycle-manager/main.go:394","msg":"replay done","proxy_id":"192.168.0.6:8082","pushed":0,"failed":0} 9月 20 16:24:13 VM-0-6-tencentos systemd1: Stopping cube-sandbox-cube-lifecycle-manager.service - Cube Sandbox cube-lifecycle-manager container... 9月 20 16:24:18 VM-0-6-tencentos bash7978: 45B blob data 9月 20 16:24:18 VM-0-6-tencentos bash18760: one-click-runtime cube-lifecycle-manager stopped 9月 20 16:24:18 VM-0-6-tencentos systemd1: cube-sandbox-cube-lifecycle-manager.service: Deactivated successfully. 9月 20 16:24:18 VM-0-6-tencentos systemd1: Stopped cube-sandbox-cube-lifecycle-manager.service - Cube Sandbox cube-lifecycle-manager container. -- Boot 0756528841154b8595d93af011d03619 -- 9月 20 16:24:41 VM-0-6-tencentos systemd1: Starting cube-sandbox-cube-lifecycle-manager.service - Cube Sandbox cube-lifecycle-manager container... 9月 20 16:24:41 VM-0-6-tencentos bash4667: one-click-runtime cube-lifecycle-manager runtime files prepared under /usr/local/services/cubetoolbox/cube-lifecycle-manager 9月 20 16:24:41 VM-0-6-tencentos systemd1: Started cube-sandbox-cube-lifecycle-manager.service - Cube Sandbox cube-lifecycle-manager container. 9月 20 16:24:41 VM-0-6-tencentos bash4762: one-click-runtime pulling cube-sandbox-cn.tencentcloudcr.com/cube-sandbox/cube-lifecycle-manager:v0.7.1 9月 20 16:24:42 VM-0-6-tencentos bash5603: time="2026-09-20T16:24:42+08:00" level=warning msg="/usr/local/services/cubetoolbox/cube-lifecycle-manager/docker-compose.yaml: the attributeversionis obsolete, it will be ignored, please remove it to avoid potential confusion" 9月 20 16:24:42 VM-0-6-tencentos bash5603: Container cube-lifecycle-manager Creating 9月 20 16:24:42 VM-0-6-tencentos bash5603: Container cube-lifecycle-manager Created 9月 20 16:24:42 VM-0-6-tencentos bash5603: Attaching to cube-lifecycle-manager 9月 20 16:24:42 VM-0-6-tencentos bash5603: Container cube-lifecycle-manager Starting 9月 20 16:24:42 VM-0-6-tencentos bash5603: Container cube-lifecycle-manager Started 9月 20 16:24:42 VM-0-6-tencentos bash5603: cube-lifecycle-manager | {"level":"info","ts":1789892682.4259038,"caller":"cube-lifecycle-manager/main.go:59","msg":"cube-lifecycle-manager starting","redis_addr":"127.0.0.1:6379","cube_proxy_admin_urls":"http://127.0.0.1:8082","cubemaster_url":"http://192.168.0.6:8089","listen_addr":"0.0.0.0:8083","instance_identity":"VM-0-6-tencentos","leader_election_enabled":false} 9月 20 16:24:42 VM-0-6-tencentos bash5603: cube-lifecycle-manager | {"level":"info","ts":1789892682.428625,"caller":"cube-lifecycle-manager/main.go:81","msg":"eventbus enabled"} 9月 20 16:24:42 VM-0-6-tencentos bash5603: cube-lifecycle-manager | {"level":"info","ts":1789892682.4323788,"caller":"cube-lifecycle-manager/main.go:641","msg":"bootstrap complete","entries":0} 9月 20 16:24:42 VM-0-6-tencentos bash5603: cube-lifecycle-manager | {"level":"info","ts":1789892682.4353082,"logger":"http","caller":"httpapi/server.go:86","msg":"cube-lifecycle-manager http server listening","addr":"0.0.0.0:8083"} 9月 20 16:24:45 VM-0-6-tencentos bash5603: cube-lifecycle-manager | {"level":"info","ts":1789892685.4343956,"logger":"discovery","caller":"discovery/redis.go:182","msg":"discovery: proxy joined","proxy_id":"192.168.0.6:8082","admin_url":"http://192.168.0.6:8082"} 9月 20 16:24:45 VM-0-6-tencentos bash5603: cube-lifecycle-manager | {"level":"info","ts":1789892685.434483,"logger":"replay","caller":"cube-lifecycle-manager/main.go:366","msg":"replay begin","proxy_id":"192.168.0.6:8082","admin_url":"http://192.168.0.6:8082","entries":0} 9月 20 16:24:45 VM-0-6-tencentos bash5603: cube-lifecycle-manager | {"level":"info","ts":1789892685.434507,"logger":"replay","caller":"cube-lifecycle-manager/main.go:394","msg":"replay done","proxy_id":"192.168.0.6:8082","pushed":0,"failed":0} 5d9ef03bf1ed cube-sandbox-cn.tencentcloudcr.com/cube-sandbox/cube-egress:v0.7.1 "/usr/local/openrest..." 2 hours ago Up 2 hours cube-egress d5c0c2c48c92 cube-sandbox-image.tencentcloudcr.com/opensource/openresty:1.21.4.1-6-alpine-fat "/usr/local/openrest..." 2 hours ago Up 2 hours (healthy) 0.0.0.0:12088->80/tcp cube-webui 5041bbb48a0f cube-sandbox-cn.tencentcloudcr.com/cube-sandbox/cube-proxy:v0.7.1 "/usr/local/openrest..." 2 hours ago Up 2 hours cube-proxy 459a081ffa37 cube-sandbox-cn.tencentcloudcr.com/cube-sandbox/cube-lifecycle-manager:v0.7.1 "/usr/local/bin/cube..." 2 hours ago Up 2 hours cube-lifecycle-manager 8d12b0ccbb0b cube-sandbox-image.tencentcloudcr.com/opensource/redis:7-alpine "docker-entrypoint.s..." 2 hours ago Up 2 hours (healthy) 127.0.0.1:6379->6379/tcp cube-sandbox-redis 76096c049704 cube-sandbox-image.tencentcloudcr.com/opensource/mysql:8.0 "docker-entrypoint.s..." 2 hours ago Up 2 hours (healthy) 127.0.0.1:3306->3306/tcp, 33060/tcp cube-sandbox-mysql 861435ecd634 cube-sandbox-cn.tencentcloudcr.com/cube-sandbox/minio:RELEASE.2025-09-07T16-13-09Z "/usr/bin/docker-ent..." 2 hours ago Up 2 hours (healthy) 192.168.0.6:9000->9000/tcp, 127.0.0.1:9001->9001/tcp cube-sandbox-minio a0dd3c78f388 cube-sandbox-image.tencentcloudcr.com/opensource/coredns/coredns:1.14.2 "/coredns -conf /etc..." 2 hours ago Up 2 hours cube-proxy-coredns
从日志看,Cube Sandbox v0.7.1 主链路基本启动正常,真正异常点在 cubelet 进程启动阶段被系统 Kill
关键异常:
cubelet initial pid=1710
/usr/local/services/cubetoolbox/scripts/systemd/cubelet-start.sh: line 57:
1710 Killed
(exec "${CUBELET_BIN}" --config ... )
但是随后:
cubelet stable pid=1949
systemd Started cube-sandbox-cubelet.service
说明:
- wrapper 脚本认为 cubelet 已稳定
- 但是第一次启动的真实 cubelet binary 被杀掉
- 后续可能启动了另一个监听进程/守护逻辑
- systemd 没有认为失败
1. 最大可能原因:OOM Killer 杀掉 cubelet
日志:
Consumed 10.594s CPU time, 237.9M memory peak
虽然 cubelet 自己峰值只有 237MB,但启动瞬间可能:
- 加载模板
- 初始化 qemu
- 扫描 image cache
- 初始化 container runtime
- 读取大量 metadata
触发瞬时内存压力
先确认:
dmesg -T | grep -Ei "killed process|out of memory|oom"
或者:
journalctl -k | grep -Ei "oom|killed"
重点看:
类似:
Out of memory: Killed process 1710 (cubelet)
Killed process 1710 (cubelet) total-vm...
如果有,就是 OOM
2. 检查机器内存
执行:
free -h
以及:
cat /proc/meminfo | head
Cube Sandbox 最低建议:
| 组件 | 建议 |
|---|---|
| cubelet | >=2G |
| cube-proxy | 512M |
| redis/mysql/minio | >=3G |
| 模板构建 | >=4G |
| 运行 sandbox | 8G+ |
如果这是腾讯云轻量机/小规格 VM,很容易发生
3. 检查 systemd 是否限制内存
查看:
systemctl cat cube-sandbox-cubelet
重点:
MemoryLimit=
MemoryMax=
如果存在:
例如:
MemoryMax=256M
会直接 kill
查看:
systemctl show cube-sandbox-cubelet | grep -i memory
4. 检查 cgroup OOM
你的配置里面:
"DisableHostCgroup":true,
"DisableVmCgroup":false
说明:
host cgroup 禁用了
但是 VM cgroup 开启
查看:
cat /sys/fs/cgroup/memory.events
如果看到:
oom 1
oom_kill 1
说明 cgroup kill
5. cubelet binary 单独启动测试
找到 binary:
grep CUBELET_BIN \
/usr/local/services/cubetoolbox/scripts/systemd/cubelet-start.sh
例如:
CUBELET_BIN=/usr/local/services/cubetoolbox/Cubelet/cubelet
直接运行:
/usr/local/services/cubetoolbox/Cubelet/cubelet \
--config /xxx/conf.yaml \
--dynamic-conf-path /xxx/conf.yaml
观察:
- panic
- segmentation fault
- permission denied
- killed
6. 检查 kernel 是否因为权限杀
执行:
journalctl -k -n 200
关注:
segfault
apparmor
selinux
denied
7. lifecycle-manager 正常
这一部分没有问题:
cube-lifecycle-manager starting
redis_addr:
127.0.0.1:6379
cubemaster_url:
http://192.168.0.6:8089
listen:
0.0.0.0:8083
状态:
bootstrap complete entries=0
http server listening
proxy joined
说明:
✅ redis OK
✅ mysql OK
✅ minio OK
✅ proxy OK
✅ lifecycle manager OK
8. 当前架构状态
现在:
cube-webui
|
cube-proxy
|
cube-lifecycle-manager
|
redis
|
cubelet ❌
|
qemu/container runtime
问题在:
cubelet -> sandbox runtime
9. 建议下一步执行
请贴下面输出:
A. 看 OOM
dmesg -T | grep -Ei "oom|killed"
B. 看机器资源
free -h
df -h
C. 看 cubelet service
systemctl cat cube-sandbox-cubelet
D. 看进程
ps aux | grep cubelet
E. 看 kernel
journalctl -k -n 100 --no-pager
目前判断:
80% 概率是内存/cgroup OOM,15% 是 cubelet 自身启动异常,5% 是权限或内核限制
v0.7.1 其它组件已经健康,先不用动 Redis、MinIO、Proxy。重点查 cubelet 被 Kill 原因
root@VM-0-6-tencentos \~# dmesg -T | grep -Ei "oom|killed" root@VM-0-6-tencentos \~# free -h df -h total used free shared buff/cache available Mem: 125Gi 9.7Gi 112Gi 9.0Mi 4.0Gi 115Gi Swap: 0B 0B 0B 文件系统 大小 已用 可用 已用% 挂载点 /dev/vda1 50G 15G 36G 29% / devtmpfs 4.0M 0 4.0M 0% /dev tmpfs 63G 24K 63G 1% /dev/shm tmpfs 26G 3.4M 26G 1% /run /dev/vdb 500G 9.7G 491G 2% /mnt/allin_data overlay 50G 15G 36G 29% /var/lib/docker/rootfs/overlayfs/a0dd3c78f388d00b5da2b45e55232b14edbe6bd76026713aee3d701c19844282 overlay 50G 15G 36G 29% /var/lib/docker/rootfs/overlayfs/861435ecd634b2db18ec9e127fd778634b7f12f55c52b76dad3185ef796c29fe overlay 50G 15G 36G 29% /var/lib/docker/rootfs/overlayfs/8d12b0ccbb0b18a4c8ae2bc343c7de840c06e3a4a00ccb67215dd9ab73537407 overlay 50G 15G 36G 29% /var/lib/docker/rootfs/overlayfs/76096c0497040f2c04de824409505bbb6a89c82c7e4ebdb5e9119c914a0ed519 overlay 50G 15G 36G 29% /var/lib/docker/rootfs/overlayfs/459a081ffa3731c574f85bdf91424a92f5e703be1dcf7c19f01cb4ba875c82d9 overlay 50G 15G 36G 29% /var/lib/docker/rootfs/overlayfs/5041bbb48a0fe1c365ccef646224fd7037a04145976ba2feeb04f48ff6742706 overlay 50G 15G 36G 29% /var/lib/docker/rootfs/overlayfs/d5c0c2c48c9251acf461358177fca6dbafbfc77e8b0f5439703b9bea9d8a8117 overlay 50G 15G 36G 29% /var/lib/docker/rootfs/overlayfs/5d9ef03bf1eda889418b5487f557960fe62caae604871929dcca5a490011e7df 192.168.0.20:/guqqwbm4/ 10G 42M 10G 1% /mnt/allin_data/allin/pod_log/data_share root@VM-0-6-tencentos \~# systemctl cat cube-sandbox-cubelet # /etc/systemd/system/cube-sandbox-cubelet.service Unit Description=Cube Sandbox Cubelet After=network-online.target Wants=network-online.target ConditionPathExists=/usr/local/services/cubetoolbox/.one-click.env RequiresMountsFor=/data/cubelet /data/cube-shim /data/snapshot_pack PartOf=cube-sandbox-control.target cube-sandbox-compute.target Service Type=forking ExecStartPre=/usr/bin/bash /usr/local/services/cubetoolbox/scripts/systemd/prepare-compute-role.sh User=root WorkingDirectory=/usr/local/services/cubetoolbox EnvironmentFile=-/usr/local/services/cubetoolbox/.one-click.env PIDFile=/run/cube-sandbox-systemd/cubelet.pid Delegate=yes KillMode=process ExecStart=/usr/bin/bash /usr/local/services/cubetoolbox/scripts/systemd/cubelet-start.sh ExecStop=/usr/bin/bash /usr/local/services/cubetoolbox/scripts/systemd/cubelet-stop.sh Restart=on-failure RestartSec=2s TimeoutStartSec=60s TimeoutStopSec=30s Install WantedBy=cube-sandbox-control.target WantedBy=cube-sandbox-compute.target root@VM-0-6-tencentos \~# ps aux | grep cubelet root 1949 0.5 0.1 4164304 221340 ? Sl 16:24 0:31 /usr/local/services/cubetoolbox/Cubelet/bin/cubelet --config /usr/local/services/cubetoolbox/Cubelet/config/config.toml --dynamic-conf-path /usr/local/services/cubetoolbox/Cubelet/dynamicconf/conf.yaml root 178554 0.0 0.0 6696 2060 pts/0 S+ 18:03 0:00 grep --color=auto cubelet root@VM-0-6-tencentos \~# journalctl -k -n 100 --no-pager 9月 20 16:24:32 VM-0-6-tencentos systemd1: Mounting dev-mqueue.mount - POSIX Message Queue File System... 9月 20 16:24:32 VM-0-6-tencentos systemd1: Mounting sys-kernel-debug.mount - Kernel Debug File System... 9月 20 16:24:32 VM-0-6-tencentos systemd1: Mounting sys-kernel-tracing.mount - Kernel Trace File System... 9月 20 16:24:32 VM-0-6-tencentos systemd1: auth-rpcgss-module.service - Kernel Module supporting RPCSEC_GSS was skipped because of an unmet condition check (ConditionPathExists=/etc/krb5.keytab). 9月 20 16:24:32 VM-0-6-tencentos systemd1: Starting kmod-static-nodes.service - Create List of Static Device Nodes... 9月 20 16:24:32 VM-0-6-tencentos systemd1: loadmodules.service - Load legacy module configuration was skipped because no trigger condition checks were met. 9月 20 16:24:32 VM-0-6-tencentos systemd1: Starting lvm2-monitor.service - Monitoring of LVM2 mirrors, snapshots etc. using dmeventd or progress polling... 9月 20 16:24:32 VM-0-6-tencentos systemd1: Starting modprobe@configfs.service - Load Kernel Module configfs... 9月 20 16:24:32 VM-0-6-tencentos systemd1: Starting modprobe@drm.service - Load Kernel Module drm... 9月 20 16:24:32 VM-0-6-tencentos systemd1: Starting modprobe@fuse.service - Load Kernel Module fuse... 9月 20 16:24:32 VM-0-6-tencentos systemd1: Starting systemd-journald.service - Journal Service... 9月 20 16:24:32 VM-0-6-tencentos systemd1: Starting systemd-modules-load.service - Load Kernel Modules... 9月 20 16:24:32 VM-0-6-tencentos systemd1: systemd-pcrmachine.service - TPM2 PCR Machine ID Measurement was skipped because of an unmet condition check (ConditionSecurity=measured-uki). 9月 20 16:24:32 VM-0-6-tencentos systemd1: Starting systemd-remount-fs.service - Remount Root and Kernel File Systems... 9月 20 16:24:32 VM-0-6-tencentos systemd1: systemd-tpm2-setup-early.service - TPM2 SRK Setup (Early) was skipped because of an unmet condition check (ConditionSecurity=measured-uki). 9月 20 16:24:32 VM-0-6-tencentos systemd1: Starting systemd-udev-trigger.service - Coldplug All udev Devices... 9月 20 16:24:32 VM-0-6-tencentos systemd1: Mounted dev-hugepages.mount - Huge Pages File System. 9月 20 16:24:32 VM-0-6-tencentos systemd1: Mounted dev-mqueue.mount - POSIX Message Queue File System. 9月 20 16:24:32 VM-0-6-tencentos systemd1: Mounted sys-kernel-debug.mount - Kernel Debug File System. 9月 20 16:24:32 VM-0-6-tencentos systemd1: Mounted sys-kernel-tracing.mount - Kernel Trace File System. 9月 20 16:24:32 VM-0-6-tencentos systemd1: Finished kmod-static-nodes.service - Create List of Static Device Nodes. 9月 20 16:24:32 VM-0-6-tencentos systemd1: modprobe@configfs.service: Deactivated successfully. 9月 20 16:24:32 VM-0-6-tencentos systemd-journald767: Collecting audit messages is disabled. 9月 20 16:24:32 VM-0-6-tencentos kernel: fuse: init (API version 7.39) 9月 20 16:24:32 VM-0-6-tencentos systemd1: Finished modprobe@configfs.service - Load Kernel Module configfs. 9月 20 16:24:32 VM-0-6-tencentos systemd1: Started systemd-journald.service - Journal Service. 9月 20 16:24:32 VM-0-6-tencentos kernel: ACPI: bus type drm_connector registered 9月 20 16:24:33 VM-0-6-tencentos systemd-journald767: Received client request to flush runtime journal. 9月 20 16:24:33 VM-0-6-tencentos systemd-journald767: /var/log/journal/2fa88b3fbc4d497eaa61f948bcf01cb9/system.journal: Journal file uses a different sequence number ID, rotating. 9月 20 16:24:33 VM-0-6-tencentos systemd-journald767: Rotating system journal. 9月 20 16:24:33 VM-0-6-tencentos kernel: input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input2 9月 20 16:24:33 VM-0-6-tencentos kernel: ACPI: button: Power Button PWRF 9月 20 16:24:33 VM-0-6-tencentos kernel: input: PC Speaker as /devices/platform/pcspkr/input/input3 9月 20 16:24:33 VM-0-6-tencentos kernel: piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 9月 20 16:24:33 VM-0-6-tencentos kernel: XFS (vdb): Mounting V5 Filesystem 2435a2e9-47a9-42c4-b1a9-3492b8ef3232 9月 20 16:24:33 VM-0-6-tencentos kernel: cirrus 0000:00:02.0: vgaarb: deactivate vga console 9月 20 16:24:33 VM-0-6-tencentos kernel: Console: switching to colour dummy device 80x25 9月 20 16:24:33 VM-0-6-tencentos kernel: drm Initialized cirrus 2.0.0 2019 for 0000:00:02.0 on minor 0 9月 20 16:24:33 VM-0-6-tencentos kernel: fbcon: cirrusdrmfb (fb0) is primary device 9月 20 16:24:33 VM-0-6-tencentos kernel: XFS (vdb): Ending clean mount 9月 20 16:24:33 VM-0-6-tencentos kernel: Console: switching to colour frame buffer device 128x48 9月 20 16:24:33 VM-0-6-tencentos kernel: cirrus 0000:00:02.0: drm fb0: cirrusdrmfb frame buffer device 9月 20 16:24:33 VM-0-6-tencentos kernel: RPC: Registered named UNIX socket transport module. 9月 20 16:24:33 VM-0-6-tencentos kernel: RPC: Registered udp transport module. 9月 20 16:24:33 VM-0-6-tencentos kernel: RPC: Registered tcp transport module. 9月 20 16:24:33 VM-0-6-tencentos kernel: RPC: Registered tcp-with-tls transport module. 9月 20 16:24:33 VM-0-6-tencentos kernel: RPC: Registered tcp NFSv4.1 backchannel transport module. 9月 20 16:24:34 VM-0-6-tencentos kernel: ISO 9660 Extensions: Microsoft Joliet Level 3 9月 20 16:24:34 VM-0-6-tencentos kernel: ISO 9660 Extensions: RRIP_1991A 9月 20 16:24:34 VM-0-6-tencentos kernel: ISO 9660 Extensions: Microsoft Joliet Level 3 9月 20 16:24:34 VM-0-6-tencentos kernel: ISO 9660 Extensions: RRIP_1991A 9月 20 16:24:34 VM-0-6-tencentos kernel: 8021q: adding VLAN 0 to HW filter on device eth0 9月 20 16:24:34 VM-0-6-tencentos kernel: IPMI message handler: version 39.2 9月 20 16:24:34 VM-0-6-tencentos kernel: ipmi device interface 9月 20 16:24:34 VM-0-6-tencentos kernel: ipmi_si: IPMI System Interface driver 9月 20 16:24:34 VM-0-6-tencentos kernel: ipmi_si: Unable to find any System Interface(s) 9月 20 16:24:35 VM-0-6-tencentos kernel: block vda: the capability attribute has been deprecated. 9月 20 16:24:35 VM-0-6-tencentos kernel: ISO 9660 Extensions: Microsoft Joliet Level 3 9月 20 16:24:35 VM-0-6-tencentos kernel: ISO 9660 Extensions: RRIP_1991A 9月 20 16:24:37 VM-0-6-tencentos kernel: br-f323d309da84: port 1(veth6c05fad) entered blocking state 9月 20 16:24:37 VM-0-6-tencentos kernel: br-f323d309da84: port 1(veth6c05fad) entered disabled state 9月 20 16:24:37 VM-0-6-tencentos kernel: veth6c05fad: entered allmulticast mode 9月 20 16:24:37 VM-0-6-tencentos kernel: veth6c05fad: entered promiscuous mode 9月 20 16:24:37 VM-0-6-tencentos kernel: br-f323d309da84: port 1(veth6c05fad) entered blocking state 9月 20 16:24:37 VM-0-6-tencentos kernel: br-f323d309da84: port 1(veth6c05fad) entered forwarding state 9月 20 16:24:37 VM-0-6-tencentos kernel: br-f323d309da84: port 1(veth6c05fad) entered disabled state 9月 20 16:24:37 VM-0-6-tencentos kernel: br-f323d309da84: port 2(vethef8fcbc) entered blocking state 9月 20 16:24:37 VM-0-6-tencentos kernel: br-f323d309da84: port 2(vethef8fcbc) entered disabled state 9月 20 16:24:37 VM-0-6-tencentos kernel: vethef8fcbc: entered allmulticast mode 9月 20 16:24:37 VM-0-6-tencentos kernel: vethef8fcbc: entered promiscuous mode 9月 20 16:24:37 VM-0-6-tencentos kernel: br-f323d309da84: port 2(vethef8fcbc) entered blocking state 9月 20 16:24:37 VM-0-6-tencentos kernel: br-f323d309da84: port 2(vethef8fcbc) entered forwarding state 9月 20 16:24:37 VM-0-6-tencentos kernel: br-f323d309da84: port 2(vethef8fcbc) entered disabled state 9月 20 16:24:37 VM-0-6-tencentos kernel: br-f323d309da84: port 3(veth82f3ebc) entered blocking state 9月 20 16:24:37 VM-0-6-tencentos kernel: br-f323d309da84: port 3(veth82f3ebc) entered disabled state 9月 20 16:24:37 VM-0-6-tencentos kernel: veth82f3ebc: entered allmulticast mode 9月 20 16:24:37 VM-0-6-tencentos kernel: veth82f3ebc: entered promiscuous mode 9月 20 16:24:37 VM-0-6-tencentos kernel: br-f323d309da84: port 3(veth82f3ebc) entered blocking state 9月 20 16:24:37 VM-0-6-tencentos kernel: br-f323d309da84: port 3(veth82f3ebc) entered forwarding state 9月 20 16:24:37 VM-0-6-tencentos kernel: eth0: renamed from vetha1cbfcc 9月 20 16:24:37 VM-0-6-tencentos kernel: br-f323d309da84: port 3(veth82f3ebc) entered disabled state 9月 20 16:24:37 VM-0-6-tencentos kernel: br-f323d309da84: port 1(veth6c05fad) entered blocking state 9月 20 16:24:37 VM-0-6-tencentos kernel: br-f323d309da84: port 1(veth6c05fad) entered forwarding state 9月 20 16:24:37 VM-0-6-tencentos kernel: eth0: renamed from veth807343f 9月 20 16:24:37 VM-0-6-tencentos kernel: eth0: renamed from veth11f3145 9月 20 16:24:37 VM-0-6-tencentos kernel: br-f323d309da84: port 2(vethef8fcbc) entered blocking state 9月 20 16:24:37 VM-0-6-tencentos kernel: br-f323d309da84: port 2(vethef8fcbc) entered forwarding state 9月 20 16:24:37 VM-0-6-tencentos kernel: br-f323d309da84: port 3(veth82f3ebc) entered blocking state 9月 20 16:24:37 VM-0-6-tencentos kernel: br-f323d309da84: port 3(veth82f3ebc) entered forwarding state 9月 20 16:24:43 VM-0-6-tencentos kernel: br-cd57826e4de1: port 1(veth61ccaa6) entered blocking state 9月 20 16:24:43 VM-0-6-tencentos kernel: br-cd57826e4de1: port 1(veth61ccaa6) entered disabled state 9月 20 16:24:43 VM-0-6-tencentos kernel: veth61ccaa6: entered allmulticast mode 9月 20 16:24:43 VM-0-6-tencentos kernel: veth61ccaa6: entered promiscuous mode 9月 20 16:24:43 VM-0-6-tencentos kernel: br-cd57826e4de1: port 1(veth61ccaa6) entered blocking state 9月 20 16:24:43 VM-0-6-tencentos kernel: br-cd57826e4de1: port 1(veth61ccaa6) entered forwarding state 9月 20 16:24:43 VM-0-6-tencentos kernel: br-cd57826e4de1: port 1(veth61ccaa6) entered disabled state 9月 20 16:24:43 VM-0-6-tencentos kernel: eth0: renamed from vetha4c7914 9月 20 16:24:43 VM-0-6-tencentos kernel: br-cd57826e4de1: port 1(veth61ccaa6) entered blocking state 9月 20 16:24:43 VM-0-6-tencentos kernel: br-cd57826e4de1: port 1(veth61ccaa6) entered forwarding state 9月 20 16:24:49 VM-0-6-tencentos kernel: FS-Cache: Loaded
根据补充的信息,可以基本排除 宿主机 OOM 和 systemd Memory 限制
当前结论调整:
cubelet 进程不是因为内存不足被杀,而是 cubelet 自己的启动流程里主动退出/被父脚本处理,或者触发了某个运行时检查
关键证据
1. 宿主机资源完全正常
你的机器:
Mem: 125Gi
used: 9.7Gi
available:115Gi
Swap:0
内存非常充足
同时:
dmesg | grep -Ei "oom|killed"
无输出
排除:
- ❌ kernel OOM killer
- ❌ docker OOM
- ❌ 内存不足
2. 当前 cubelet 实际运行正常
现在:
ps aux | grep cubelet
看到:
root 1949
/usr/local/services/cubetoolbox/Cubelet/bin/cubelet
状态:
Sl
说明:
- S = sleeping
- l = multi-thread
进程正常
3. 真正异常是这个:
启动:
cubelet initial pid=1710
马上:
1710 Killed
然后:
cubelet stable pid=1949
也就是说:
启动脚本:
cubelet-start.sh
逻辑类似:
start cubelet &
wait
if alive
use it
else
restart
fi
第一次 pid 1710 被杀,然后重新拉起 1949
所以不是服务失败
下一步重点查 cubelet-start.sh
执行:
sed -n '1,150p' \
/usr/local/services/cubetoolbox/scripts/systemd/cubelet-start.sh
重点看:
第57行附近
你的日志:
cubelet-start.sh: line 57:
1710 Killed
需要看:
nl -ba \
/usr/local/services/cubetoolbox/scripts/systemd/cubelet-start.sh \
| sed -n '45,70p'
第二个重点:为什么第一次启动被杀?
目前可能性:
可能1:cubelet 自带 watchdog/reexec
很多云沙箱组件启动:
bootstrap process
|
|
+--> real cubelet
第一次 pid:
1710
只是 bootstrap
然后:
exec/reload
换 pid:
1949
脚本误判为 killed
可能2:动态配置 reload
日志:
listen file[
/usr/local/services/cubetoolbox/Cubelet/dynamicconf/conf.yaml
]
说明 cubelet 有动态配置监听
启动流程:
cubelet
|
加载config
|
读取dynamicconf
|
重新fork
|
旧pid退出
|
新pid接管
也可能显示:
Killed
但实际是正常行为
3. 检查 cubelet 健康接口
先确认:
ss -lntp | grep cubelet
或者:
netstat -lntp | grep 1949
看监听端口
4. 查看 cubelet 日志目录
找日志:
find /usr/local/services/cubetoolbox/Cubelet \
-name "*.log" \
-o -name "*.out"
重点:
ls -lh /usr/local/services/cubetoolbox/Cubelet/log/
5. 检查是否有 sandbox runtime
你的 docker:
没有看到:
qemu
builder
template
只有:
cube-proxy
cube-egress
cube-webui
cube-lifecycle-manager
redis
mysql
minio
coredns
说明:
目前只是 control plane
还没有创建 sandbox
执行:
docker ps -a | grep -Ei "qemu|sandbox|builder|template"
如果为空:
正常
6. lifecycle-manager 状态正常
这一块不用处理:
bootstrap complete entries=0
proxy joined
replay done
说明:
cube-master
cube-proxy
lifecycle-manager
redis
通信正常
我建议现在执行这几个命令
按顺序:
① 看启动脚本第57行
nl -ba \
/usr/local/services/cubetoolbox/scripts/systemd/cubelet-start.sh \
| sed -n '45,70p'
② 看 cubelet 日志
find /usr/local/services/cubetoolbox/Cubelet \
-type f | grep -Ei "log|err"
③ 看端口
ss -lntp | grep cube
④ 看 cubelet完整启动参数
cat /proc/1949/cmdline | tr '\0' ' '
目前判断:
| 项目 | 状态 |
|---|---|
| 宿主机内存 | ✅正常 |
| 磁盘 | ✅正常 |
| Docker组件 | ✅正常 |
| Lifecycle Manager | ✅正常 |
| Cubelet最终进程 | ✅正常 |
| 首次pid killed | ⚠️启动脚本/自重启机制 |
下一步主要确认 cubelet-start.sh 第57行到底是什么逻辑
nl -ba \ /usr/local/services/cubetoolbox/scripts/systemd/cubelet-start.sh \ | sed -n '45,70p' 45 46 if \[ -n "${CUBE_SANDBOX_NODE_IP:-}" ]; then 47 ( 48 export CUBE_SANDBOX_NODE_IP="{CUBE_SANDBOX_NODE_IP}" 49 exec "{CUBELET_BIN}" --config "{CUBELET_CONFIG}" --dynamic-conf-path "{CUBELET_DYNAMICCONF}" 50 ) & 51 else 52 "{CUBELET_BIN}" --config "{CUBELET_CONFIG}" --dynamic-conf-path "{CUBELET_DYNAMICCONF}" \& 53 fi 54 initial_pid=! 55 log "cubelet initial pid={initial_pid}" 56 57 for _ in {1..20}; do 58 if ! kill -0 "{initial_pid}" >/dev/null 2>&1; then 59 break 60 fi 61 sleep 1 62 done 63 64 refresh_pidfile_from_pattern "{PID_FILE}" "{PATTERN}" 20 1 || die "failed to determine cubelet child pid" 65 log "cubelet stable pid=(\<"{PID_FILE}")" root@VM-0-6-tencentos \~# find /usr/local/services/cubetoolbox/Cubelet \ -type f | grep -Ei "log|err" root@VM-0-6-tencentos \~# ss -lntp | grep cube LISTEN 0 4096 0.0.0.0:3010 0.0.0.0:* users:(("cubeops",pid=4673,fd=10)) LISTEN 0 4096 0.0.0.0:3000 0.0.0.0:* users:(("cube-api",pid=5903,fd=9)) LISTEN 0 4096 0.0.0.0:8083 0.0.0.0:* users:(("cube-lifecycle-",pid=5708,fd=9)) LISTEN 0 4096 0.0.0.0:8089 0.0.0.0:* users:(("cubemaster",pid=5538,fd=15)) LISTEN 0 4096 0.0.0.0:9999 0.0.0.0:* users:(("cubelet",pid=1949,fd=636)) LISTEN 0 4096 0.0.0.0:9998 0.0.0.0:* users:(("cubelet",pid=1949,fd=633)) LISTEN 0 4096 0.0.0.0:9966 0.0.0.0:* users:(("cubelet",pid=1949,fd=631)) root@VM-0-6-tencentos \~# cat /proc/1949/cmdline | tr '\0' ' ' /usr/local/services/cubetoolbox/Cubelet/bin/cubelet --config /usr/local/services/cubetoolbox/Cubelet/config/config.toml --dynamic-conf-path /usr/local/services/cubetoolbox/Cubelet/dynamicconf/conf.yaml
从目前信息看,cubelet 并没有真正异常退出,之前日志里的:
cubelet-start.sh: line 57: 6023 Killed
cubelet-start.sh: line 57: 1668 Killed
cubelet-start.sh: line 57: 1710 Killed
不是 cubelet 服务最终 PID 被杀,而是 启动脚本 fork 出来的初始子进程被替换/退出。现在稳定运行:
root 1949 0.5 0.1 4164304 221340 ? Sl 16:24 0:31
/usr/local/services/cubetoolbox/Cubelet/bin/cubelet
并且监听:
0.0.0.0:9999 cubelet
0.0.0.0:9998 cubelet
0.0.0.0:9966 cubelet
说明 cubelet 已正常启动
1. 先解释为什么看到 Killed
你的脚本:
if [[ -n "${CUBE_SANDBOX_NODE_IP:-}" ]]; then
(
export CUBE_SANDBOX_NODE_IP="${CUBE_SANDBOX_NODE_IP}"
exec "${CUBELET_BIN}" ...
) &
fi
initial_pid=$!
这里:
6023
1668
1710
是 shell fork 出来的 wrapper pid
启动后:
exec cubelet
替换进程。
然后脚本继续:
refresh_pidfile_from_pattern
重新通过:
PATTERN
找到真正 cubelet PID
所以:
Killed 1710
stable pid=1949
实际过程:
cubelet-start.sh
|
+-- bash subshell PID 1710
|
exec cubelet
|
PID变成cubelet
最终:
PID 1949 cubelet
这个行为类似 Kubernetes kubelet wrapper
2. 但是有一个值得检查的问题
你的日志:
第一次:
cubelet initial pid=6023
Killed 6023
cubelet stable pid=7079
第二次:
cubelet initial pid=1668
Killed 1668
stable pid=2125
第三次:
initial pid=1710
Killed 1710
stable pid=1949
每次都是:
启动后约 1 秒:
Killed
说明 wrapper 检测机制可能有问题
建议检查:
cat /usr/local/services/cubetoolbox/scripts/systemd/cubelet-start.sh
前面定义:
PATTERN=
重点看:
refresh_pidfile_from_pattern
匹配规则
执行:
grep PATTERN -n \
/usr/local/services/cubetoolbox/scripts/systemd/cubelet-start.sh
3. 当前 cubelet 状态确认
执行:
systemctl status cube-sandbox-cubelet
应该类似:
Active: active (running)
Main PID: 1949
再看:
cat /run/cube-sandbox-systemd/cubelet.pid
应该:
1949
确认 systemd 管理的是正确 PID
4. 当前资源没有问题
你的机器:
Mem:
125Gi total
9.7Gi used
115Gi available
Swap:
0
不存在 OOM
同时:
dmesg -T | grep -Ei "oom|killed"
为空
所以排除:
- 内存不足
- kernel OOM killer
- cgroup memory kill
5. cubelet 自己日志在哪里?
你找:
find /usr/local/services/cubetoolbox/Cubelet \
-type f | grep -Ei "log|err"
没有输出
说明它大概率写 stdout
systemd 里查看:
journalctl \
-u cube-sandbox-cubelet \
-f
然后:
另开窗口:
curl http://127.0.0.1:9999
或者:
curl http://127.0.0.1:9966
观察
6. 建议检查 cubelet health
你的端口:
| 端口 | 可能用途 |
|---|---|
| 9966 | metrics |
| 9998 | RPC |
| 9999 | HTTP API |
测试:
curl -v http://127.0.0.1:9999/health
或者:
curl -v http://127.0.0.1:9999/healthz
查看:
curl -s http://127.0.0.1:9966/metrics | head
如果返回 Prometheus 指标:
说明完全正常
7. 我认为当前真正问题点
从你的日志:
cube-lifecycle-manager
bootstrap complete
entries:0
proxy joined
replay done pushed:0 failed:0
控制面正常
Docker:
cube-proxy
cube-webui
cube-egress
redis
mysql
minio
全部 healthy
cubelet:
PID 1949
ports opened
正常
目前唯一异常:
cubelet-start.sh wrapper pid killed
但不影响服务
cat /run/cube-sandbox-systemd/cubelet.pid systemctl status cube-sandbox-cubelet grep PATTERN -n \ /usr/local/services/cubetoolbox/scripts/systemd/cubelet-start.sh curl -s http://127.0.0.1:9966/metrics | head -20 curl -v http://127.0.0.1:9999/health 1949 ● cube-sandbox-cubelet.service - Cube Sandbox Cubelet Loaded: loaded (/etc/systemd/system/cube-sandbox-cubelet.service; disabled; preset: disabled) Active: active (running) since Sun 2026-09-20 16:24:36 CST; 1h 46min ago Process: 1505 ExecStartPre=/usr/bin/bash /usr/local/services/cubetoolbox/scripts/systemd/prepare-compute-role.sh (code=exited, status=0/SUCCESS) Process: 1580 ExecStart=/usr/bin/bash /usr/local/services/cubetoolbox/scripts/systemd/cubelet-start.sh (code=exited, status=0/SUCCESS) Main PID: 1949 (cubelet) Tasks: 0 (limit: 153562) Memory: 212.4M (peak: 231.0M) CPU: 10.815s CGroup: /system.slice/cube-sandbox-cubelet.service ‣ 1949 /usr/local/services/cubetoolbox/Cubelet/bin/cubelet --config /usr/local/services/cubetoolbox/Cubelet/config/config.toml --dynamic-conf-path /usr/local/services/cubetoolbox/Cubelet/dynamicconf/conf.yaml 9月 20 16:24:35 VM-0-6-tencentos bash1608: one-click-systemd updated cubelet cubeops_addr=http://127.0.0.1:3010 9月 20 16:24:35 VM-0-6-tencentos bash1580: one-click-systemd updated cubelet cow.s3 enable=false 9月 20 16:24:35 VM-0-6-tencentos bash1580: one-click-systemd cubelet initial pid=1710 9月 20 16:24:35 VM-0-6-tencentos bash1949: 2026-09-20T16:24:35.552388365+08:00,listen file/usr/local/services/cubetoolbox/Cubelet/dynamicconf/conf.yaml 9月 20 16:24:35 VM-0-6-tencentos bash1949: cfg:{"Common":{"CommonTimeout":10000000000,"LogLevel":"","DescribeAsyncInterval":100000000,"EnablePFMode":false,"DescribeBDFInterval":10000000,"DescribeBDFTimeout":10000000000,"GetBDFByUuidCmd":"/usr/local/services/AdamPlugins-1.0/snhost_snic_sdk/cath/bm/tools/get_bdf_by_uuid","GetBDFByIfNameCmd":"/usr/local/services> 9月 20 16:24:35 VM-0-6-tencentos bash1949: 2026/09/20 16:24:35 mounted state tmpfs /data/cubelet/state size=1024m 9月 20 16:24:35 VM-0-6-tencentos bash1949: 2026/09/20 16:24:35 set /data/cubelet/root as shared mount 9月 20 16:24:36 VM-0-6-tencentos bash1580: /usr/local/services/cubetoolbox/scripts/systemd/cubelet-start.sh: line 57: 1710 Killed ( export CUBE_SANDBOX_NODE_IP="{CUBE_SANDBOX_NODE_IP}"; exec "{CUBELET_BIN}" --config "{CUBELET_CONFIG}" --dynamic-conf-path "{CUBELET_DYNAMICCONF}" ) 9月 20 16:24:36 VM-0-6-tencentos bash1580: one-click-systemd cubelet stable pid=1949 9月 20 16:24:36 VM-0-6-tencentos systemd1: Started cube-sandbox-cubelet.service - Cube Sandbox Cubelet. 19:PATTERN="^{CUBELET_BIN} --config" 39: if \[\[ -n "{existing_pid}" ]] && kill -0 "{existing_pid}" \>/dev/null 2\>\&1 \&\& pid_matches_pattern "{existing_pid}" "{PATTERN}"; then 64:refresh_pidfile_from_pattern "{PID_FILE}" "{PATTERN}" 20 1 \|\| die "failed to determine cubelet child pid" 404 page not found \* Trying 127.0.0.1:9999... \* Connected to 127.0.0.1 (127.0.0.1) port 9999 \> GET /health HTTP/1.1 \> Host: 127.0.0.1:9999 \> User-Agent: curl/8.4.0 \> Accept: \*/\* \> \* Received HTTP/0.9 when not allowed \* Closing connection curl: (1) Received HTTP/0.9 when not allowed \[root@VM-0-6-tencentos \~\]# journalctl -u cube-sandbox-cubelet -f docker logs cube-proxy docker logs cube-lifecycle-manager 9月 20 16:24:35 VM-0-6-tencentos bash\[1608\]: \[one-click-systemd\] updated cubelet cubeops_addr=http://127.0.0.1:3010 9月 20 16:24:35 VM-0-6-tencentos bash\[1580\]: \[one-click-systemd\] updated cubelet cow.s3 enable=false 9月 20 16:24:35 VM-0-6-tencentos bash\[1580\]: \[one-click-systemd\] cubelet initial pid=1710 9月 20 16:24:35 VM-0-6-tencentos bash\[1949\]: 2026-09-20T16:24:35.552388365+08:00,listen file\[/usr/local/services/cubetoolbox/Cubelet/dynamicconf/conf.yaml\] 9月 20 16:24:35 VM-0-6-tencentos bash\[1949\]: cfg:{"Common":{"CommonTimeout":10000000000,"LogLevel":"","DescribeAsyncInterval":100000000,"EnablePFMode":false,"DescribeBDFInterval":10000000,"DescribeBDFTimeout":10000000000,"GetBDFByUuidCmd":"/usr/local/services/AdamPlugins-1.0/snhost_snic_sdk/cath/bm/tools/get_bdf_by_uuid","GetBDFByIfNameCmd":"/usr/local/services/AdamPlugins-1.0/snhost_snic_sdk/cath/bm/tools/get_bdf_by_ifname","CommandTimeout":1000000000,"DisableHostCgroup":true,"DisableVmCgroup":false,"EnableSandboxExecCmdBeforeExist":false,"SandboxExecCmdTimeOut":5000000000,"SandboxExecCmdBeforeExist":\["sh","/usr/local/cubetools/cube_exec_cmd_before_exist.sh"\],"SandboxExecCmdBeforeExistLogOut":false,"SandboxExecCmdOutMaxLines":2000,"SandboxExecCmdMatchLine":"","SandboxExecCmdAfterMatch":null,"SandboxExecCmdBase64AfterMatch":"","SandboxExecCmdAfterMatchLogOut":false,"CgroupDisableMemoryReparentFile":"","CgroupDisableCpusetList":"","DisableHostNetfile":true,"DefaultDNSServers":\["119.29.29.29"\],"DefaultDNSSearches":\[\],"DefaultDNSOptions":\[\],"ReconcileInterval":300000000000,"DisableCubeBoxTemplateBaseFormatPoolOfNumberVer":false},"Tenant":{"Tenants":{},"UseRespectiveNamespace":false,"DefaultNS":"default","ZiyanUinListStr":""},"MetaServerConfig":{"MetaServerEndpoint":"127.0.0.1:3010","NodeStatusMaxImages":40000,"CubeMasterHTTPAddr":"127.0.0.1:8089"},"HostConf":{"Quota":{"Cpu":0,"Mem":"","MvmLimit":0,"CreationConcurrentNum":0,"PausedResourceReleaseRatio":0},"GC":{"CodeExpirationTime":"72h","ImageExpirationTime":"24h"},"SchedulerLabel":"default-cluster"}} 9月 20 16:24:35 VM-0-6-tencentos bash\[1949\]: 2026/09/20 16:24:35 mounted state tmpfs /data/cubelet/state size=1024m 9月 20 16:24:35 VM-0-6-tencentos bash\[1949\]: 2026/09/20 16:24:35 set /data/cubelet/root as shared mount 9月 20 16:24:36 VM-0-6-tencentos bash\[1580\]: /usr/local/services/cubetoolbox/scripts/systemd/cubelet-start.sh: line 57: 1710 Killed ( export CUBE_SANDBOX_NODE_IP="{CUBE_SANDBOX_NODE_IP}"; exec "{CUBELET_BIN}" --config "{CUBELET_CONFIG}" --dynamic-conf-path "${CUBELET_DYNAMICCONF}" ) 9月 20 16:24:36 VM-0-6-tencentos bash1580: one-click-systemd cubelet stable pid=1949 9月 20 16:24:36 VM-0-6-tencentos systemd1: Started cube-sandbox-cubelet.service - Cube Sandbox Cubelet. ^C {"level":"info","ts":1789892682.4259038,"caller":"cube-lifecycle-manager/main.go:59","msg":"cube-lifecycle-manager starting","redis_addr":"127.0.0.1:6379","cube_proxy_admin_urls":"http://127.0.0.1:8082","cubemaster_url":"http://192.168.0.6:8089","listen_addr":"0.0.0.0:8083","instance_identity":"VM-0-6-tencentos","leader_election_enabled":false} {"level":"info","ts":1789892682.428625,"caller":"cube-lifecycle-manager/main.go:81","msg":"eventbus enabled"} {"level":"info","ts":1789892682.4323788,"caller":"cube-lifecycle-manager/main.go:641","msg":"bootstrap complete","entries":0} {"level":"info","ts":1789892682.4353082,"logger":"http","caller":"httpapi/server.go:86","msg":"cube-lifecycle-manager http server listening","addr":"0.0.0.0:8083"} {"level":"info","ts":1789892685.4343956,"logger":"discovery","caller":"discovery/redis.go:182","msg":"discovery: proxy joined","proxy_id":"192.168.0.6:8082","admin_url":"http://192.168.0.6:8082"} {"level":"info","ts":1789892685.434483,"logger":"replay","caller":"cube-lifecycle-manager/main.go:366","msg":"replay begin","proxy_id":"192.168.0.6:8082","admin_url":"http://192.168.0.6:8082","entries":0} {"level":"info","ts":1789892685.434507,"logger":"replay","caller":"cube-lifecycle-manager/main.go:394","msg":"replay done","proxy_id":"192.168.0.6:8082","pushed":0,"failed":0} root@VM-0-6-tencentos \~# ls -lah /data/cubelet ls -lah /data/cube-shim ls -lah /data/snapshot_pack 总计 4.0K drwxr-x--- 6 root root 127 9月20日 16:24 . drwxr-xr-x 10 root root 136 9月20日 15:36 .. drwxr-xr-x 3 root root 16 9月20日 15:36 cleanup srw-rw---- 1 root root 0 9月20日 16:24 cubelet.sock srw-rw---- 1 root root 0 9月20日 16:24 cubelet.sock.ttrpc srw-rw---- 1 root root 0 9月20日 16:24 cubetap.sock drwxr-x--- 13 root root 4.0K 9月20日 15:36 root drwx--x--x 2 root root 6 9月20日 15:36 state drwxr-xr-x 4 root root 55 9月20日 15:36 storage 总计 0 drwxr-x--- 3 root root 19 9月20日 15:36 . drwxr-xr-x 10 root root 136 9月20日 15:36 .. drwxr-x--- 2 root root 6 9月20日 15:36 disks 总计 0 drwxr-x--- 3 root root 19 9月20日 15:36 . drwxr-xr-x 10 root root 136 9月20日 15:36 .. drwxr-x--- 2 root root 6 9月20日 15:36 disks
根据补充的结果,Cube Sandbox v0.7.1 控制面和 cubelet 本身已经正常运行。之前怀疑的 OOM、磁盘、systemd PID 管理问题基本可以排除
下面是详细判断
1. cubelet PID 管理正常
现在:
cat /run/cube-sandbox-systemd/cubelet.pid
结果:
1949
systemd:
Main PID: 1949 (cubelet)
完全一致
说明:
cubelet-start.sh
|
+-- 临时启动PID 1710
|
+-- exec cubelet
|
+-- 最终PID 1949
|
+-- 写入pidfile
这个流程正常
2. Killed 不是异常
关键:
cubelet-start.sh: line 57: 1710 Killed
cubelet stable pid=1949
这里的 1710:
(
export CUBE_SANDBOX_NODE_IP
exec cubelet
) &
产生的 subshell PID
后续:
refresh_pidfile_from_pattern
找到:
/usr/local/services/cubetoolbox/Cubelet/bin/cubelet --config
真正 PID:
1949
所以不用处理
3. PATTERN 匹配也正确
你的:
19:PATTERN="^${CUBELET_BIN} --config"
当前:
cat /proc/1949/cmdline | tr '\0' ' '
输出:
/usr/local/services/cubetoolbox/Cubelet/bin/cubelet --config ...
匹配:
^cubelet --config
OK
4. cubelet API 判断
你执行:
curl -s http://127.0.0.1:9966/metrics
返回:
404 page not found
说明:
9966 有 HTTP server
只是:
/metrics
不是暴露路径
不是故障
9999:
Received HTTP/0.9 when not allowed
这个非常关键
说明:
9999 不是普通 HTTP/1.1 REST
可能是:
- gRPC
- ttrpc
- 自定义 TCP protocol
所以:
curl http://127.0.0.1:9999/health
方式不适用
5. 当前目录状态正常
你的:
/data/cubelet
里面:
cubelet.sock
cubelet.sock.ttrpc
cubetap.sock
root/
state/
storage/
说明:
cubelet runtime 初始化成功
特别:
cubelet.sock.ttrpc
说明它已经进入 sandbox runtime 阶段
现在真正需要排查的是:为什么没有 Sandbox / Template
因为:
控制面:
cubeops 3010
cube-api 3000
cubemaster 8089
lifecycle 8083
proxy 8082
全部起来
但是:
/data/snapshot_pack/disks
为空:
drwxr-x--- 2 root root 6 disks
说明:
目前还没有:
- template
- snapshot
- image
- qemu disk
下一步重点检查
1. 看 cubelet 注册状态
执行:
curl -s http://127.0.0.1:3010/api/v1/nodes | jq
如果不知道 API:
先:
ss -lntp | grep 3010
确认 cubeops
2. 查看 cubelet RPC状态
找端口:
ss -lntp | grep cubelet
已有:
9966
9998
9999
尝试:
curl -v http://127.0.0.1:9966/
和:
curl -v http://127.0.0.1:9998/
看看返回
3. 检查 qemu / shim
你的:
/data/cube-shim/disks
为空
检查:
ls -lah /data/cube-shim
以及:
find /usr/local/services/cubetoolbox \
-type f \
-name "*qemu*"
4. 看 cube-proxy 日志
你只看了 lifecycle
执行:
docker logs --tail=200 cube-proxy
重点找:
error
failed
template
sandbox
create
node
cubelet
5. 看 cubemaster
目前没有看
执行:
journalctl -u cube-sandbox-cubemaster -n 200 --no-pager
或者:
ps aux | grep cubemaster
6. 检查模板目录
执行:
find /data -maxdepth 3 \
-type d \
| grep -Ei "template|image|snapshot|box"
正常安装后通常应该有类似:
template/
snapshot/
image/
rootfs/
当前状态总结
| 组件 | 状态 |
|---|---|
| cube-api | ✅ |
| cubeops | ✅ |
| cubemaster | ✅ |
| cube-proxy | ✅ |
| cube-lifecycle-manager | ✅ |
| redis | ✅ |
| mysql | ✅ |
| minio | ✅ |
| cubelet | ✅ |
| OOM | ❌ 排除 |
| 磁盘压力 | ❌ 排除 |
| cubelet启动失败 | ❌ 排除 |
| template/image | ⚠️ 未发现 |
root@VM-0-6-tencentos \~# docker logs --tail=200 cube-proxy root@VM-0-6-tencentos \~# ps aux | grep cubemaster root 5538 1.6 0.0 3566552 42072 ? Ssl 16:24 2:01 /usr/local/services/cubetoolbox/CubeMaster/bin/cubemaster root 214146 0.0 0.0 6696 2060 pts/0 S+ 18:24 0:00 grep --color=auto cubemaster root@VM-0-6-tencentos \~# journalctl -u cube-sandbox-cubemaster -n 200 --no-pager 9月 20 17:54:12 VM-0-6-tencentos bash5538: 2026/09/20 17:54:12 /workspace/CubeMaster/pkg/templatecenter/store.go:1118 record not found 9月 20 17:54:12 VM-0-6-tencentos bash5538: 0.332ms rows:0 SELECT * FROM
t_cube_template_definitionWHERE template_id = 'tpl-464f1b5538764bd7afa240ca' ANDt_cube_template_definition.deleted_atIS NULL ORDER BYt_cube_template_definition.idLIMIT 1 9月 20 17:54:12 VM-0-6-tencentos bash5538: 9月 20 17:54:12 VM-0-6-tencentos bash5538: 2026/09/20 17:54:12 /workspace/CubeMaster/pkg/templatecenter/snapshot_table.go:29 record not found 9月 20 17:54:12 VM-0-6-tencentos bash5538: 0.502ms rows:0 SELECT * FROMt_cube_snapshotWHERE snapshot_id = 'tpl-464f1b5538764bd7afa240ca' ANDt_cube_snapshot.deleted_atIS NULL ORDER BYt_cube_snapshot.idLIMIT 1 9月 20 17:54:42 VM-0-6-tencentos bash5538: 9月 20 17:54:42 VM-0-6-tencentos bash5538: 2026/09/20 17:54:42 /workspace/CubeMaster/pkg/templatecenter/store.go:1118 record not found 9月 20 17:54:42 VM-0-6-tencentos bash5538: 0.362ms rows:0 SELECT * FROMt_cube_template_definitionWHERE template_id = 'tpl-464f1b5538764bd7afa240ca' ANDt_cube_template_definition.deleted_atIS NULL ORDER BYt_cube_template_definition.idLIMIT 1 9月 20 17:54:42 VM-0-6-tencentos bash5538: 9月 20 17:54:42 VM-0-6-tencentos bash5538: 2026/09/20 17:54:42 /workspace/CubeMaster/pkg/templatecenter/snapshot_table.go:29 record not found 9月 20 17:54:42 VM-0-6-tencentos bash5538: 0.419ms rows:0 SELECT * FROMt_cube_snapshotWHERE snapshot_id = 'tpl-464f1b5538764bd7afa240ca' ANDt_cube_snapshot.deleted_atIS NULL ORDER BYt_cube_snapshot.idLIMIT 1 9月 20 17:55:12 VM-0-6-tencentos bash5538: 9月 20 17:55:12 VM-0-6-tencentos bash5538: 2026/09/20 17:55:12 /workspace/CubeMaster/pkg/templatecenter/store.go:1118 record not found 9月 20 17:55:12 VM-0-6-tencentos bash5538: 0.421ms rows:0 SELECT * FROMt_cube_template_definitionWHERE template_id = 'tpl-464f1b5538764bd7afa240ca' ANDt_cube_template_definition.deleted_atIS NULL ORDER BYt_cube_template_definition.idLIMIT 1 9月 20 17:55:12 VM-0-6-tencentos bash5538: 9月 20 17:55:12 VM-0-6-tencentos bash5538: 2026/09/20 17:55:12 /workspace/CubeMaster/pkg/templatecenter/snapshot_table.go:29 record not found 9月 20 17:55:12 VM-0-6-tencentos bash5538: 0.374ms rows:0 SELECT * FROMt_cube_snapshotWHERE snapshot_id = 'tpl-464f1b5538764bd7afa240ca' ANDt_cube_snapshot.deleted_atIS NULL ORDER BYt_cube_snapshot.idLIMIT 1 9月 20 17:55:42 VM-0-6-tencentos bash5538: 9月 20 17:55:42 VM-0-6-tencentos bash5538: 2026/09/20 17:55:42 /workspace/CubeMaster/pkg/templatecenter/store.go:1118 record not found 9月 20 17:55:42 VM-0-6-tencentos bash5538: 0.376ms rows:0 SELECT * FROMt_cube_template_definitionWHERE template_id = 'tpl-464f1b5538764bd7afa240ca' ANDt_cube_template_definition.deleted_atIS NULL ORDER BYt_cube_template_definition.idLIMIT 1 9月 20 17:55:42 VM-0-6-tencentos bash5538: 9月 20 17:55:42 VM-0-6-tencentos bash5538: 2026/09/20 17:55:42 /workspace/CubeMaster/pkg/templatecenter/snapshot_table.go:29 record not found 9月 20 17:55:42 VM-0-6-tencentos bash5538: 0.395ms rows:0 SELECT * FROMt_cube_snapshotWHERE snapshot_id = 'tpl-464f1b5538764bd7afa240ca' ANDt_cube_snapshot.deleted_atIS NULL ORDER BYt_cube_snapshot.idLIMIT 1 9月 20 17:56:12 VM-0-6-tencentos bash5538: 9月 20 17:56:12 VM-0-6-tencentos bash5538: 2026/09/20 17:56:12 /workspace/CubeMaster/pkg/templatecenter/store.go:1118 record not found 9月 20 17:56:12 VM-0-6-tencentos bash5538: 0.358ms rows:0 SELECT * FROMt_cube_template_definitionWHERE template_id = 'tpl-464f1b5538764bd7afa240ca' ANDt_cube_template_definition.deleted_atIS NULL ORDER BYt_cube_template_definition.idLIMIT 1 9月 20 17:56:12 VM-0-6-tencentos bash5538: 9月 20 17:56:12 VM-0-6-tencentos bash5538: 2026/09/20 17:56:12 /workspace/CubeMaster/pkg/templatecenter/snapshot_table.go:29 record not found 9月 20 17:56:12 VM-0-6-tencentos bash5538: 0.401ms rows:0 SELECT * FROMt_cube_snapshotWHERE snapshot_id = 'tpl-464f1b5538764bd7afa240ca' ANDt_cube_snapshot.deleted_atIS NULL ORDER BYt_cube_snapshot.idLIMIT 1 9月 20 17:56:42 VM-0-6-tencentos bash5538: 9月 20 17:56:42 VM-0-6-tencentos bash5538: 2026/09/20 17:56:42 /workspace/CubeMaster/pkg/templatecenter/store.go:1118 record not found 9月 20 17:56:42 VM-0-6-tencentos bash5538: 0.384ms rows:0 SELECT * FROMt_cube_template_definitionWHERE template_id = 'tpl-464f1b5538764bd7afa240ca' ANDt_cube_template_definition.deleted_atIS NULL ORDER BYt_cube_template_definition.idLIMIT 1 9月 20 17:56:42 VM-0-6-tencentos bash5538: 9月 20 17:56:42 VM-0-6-tencentos bash5538: 2026/09/20 17:56:42 /workspace/CubeMaster/pkg/templatecenter/snapshot_table.go:29 record not found 9月 20 17:56:42 VM-0-6-tencentos bash5538: 0.406ms rows:0 SELECT * FROMt_cube_snapshotWHERE snapshot_id = 'tpl-464f1b5538764bd7afa240ca' ANDt_cube_snapshot.deleted_atIS NULL ORDER BYt_cube_snapshot.idLIMIT 1 9月 20 17:57:12 VM-0-6-tencentos bash5538: 9月 20 17:57:12 VM-0-6-tencentos bash5538: 2026/09/20 17:57:12 /workspace/CubeMaster/pkg/templatecenter/store.go:1118 record not found 9月 20 17:57:12 VM-0-6-tencentos bash5538: 0.418ms rows:0 SELECT * FROMt_cube_template_definitionWHERE template_id = 'tpl-464f1b5538764bd7afa240ca' ANDt_cube_template_definition.deleted_atIS NULL ORDER BYt_cube_template_definition.idLIMIT 1 9月 20 17:57:12 VM-0-6-tencentos bash5538: 9月 20 17:57:12 VM-0-6-tencentos bash5538: 2026/09/20 17:57:12 /workspace/CubeMaster/pkg/templatecenter/snapshot_table.go:29 record not found 9月 20 17:57:12 VM-0-6-tencentos bash5538: 0.376ms rows:0 SELECT * FROMt_cube_snapshotWHERE snapshot_id = 'tpl-464f1b5538764bd7afa240ca' ANDt_cube_snapshot.deleted_atIS NULL ORDER BYt_cube_snapshot.idLIMIT 1 9月 20 17:57:42 VM-0-6-tencentos bash5538: 9月 20 17:57:42 VM-0-6-tencentos bash5538: 2026/09/20 17:57:42 /workspace/CubeMaster/pkg/templatecenter/store.go:1118 record not found 9月 20 17:57:42 VM-0-6-tencentos bash5538: 0.346ms rows:0 SELECT * FROMt_cube_template_definitionWHERE template_id = 'tpl-464f1b5538764bd7afa240ca' ANDt_cube_template_definition.deleted_atIS NULL ORDER BYt_cube_template_definition.idLIMIT 1 9月 20 17:57:42 VM-0-6-tencentos bash5538: 9月 20 17:57:42 VM-0-6-tencentos bash5538: 2026/09/20 17:57:42 /workspace/CubeMaster/pkg/templatecenter/snapshot_table.go:29 record not found 9月 20 17:57:42 VM-0-6-tencentos bash5538: 0.351ms rows:0 SELECT * FROMt_cube_snapshotWHERE snapshot_id = 'tpl-464f1b5538764bd7afa240ca' ANDt_cube_snapshot.deleted_atIS NULL ORDER BYt_cube_snapshot.idLIMIT 1 9月 20 17:58:12 VM-0-6-tencentos bash5538: 9月 20 17:58:12 VM-0-6-tencentos bash5538: 2026/09/20 17:58:12 /workspace/CubeMaster/pkg/templatecenter/store.go:1118 record not found 9月 20 17:58:12 VM-0-6-tencentos bash5538: 0.360ms rows:0 SELECT * FROMt_cube_template_definitionWHERE template_id = 'tpl-464f1b5538764bd7afa240ca' ANDt_cube_template_definition.deleted_atIS NULL ORDER BYt_cube_template_definition.idLIMIT 1 9月 20 17:58:12 VM-0-6-tencentos bash5538: 9月 20 17:58:12 VM-0-6-tencentos bash5538: 2026/09/20 17:58:12 /workspace/CubeMaster/pkg/templatecenter/snapshot_table.go:29 record not found 9月 20 17:58:12 VM-0-6-tencentos bash5538: 0.352ms rows:0 SELECT * FROMt_cube_snapshotWHERE snapshot_id = 'tpl-464f1b5538764bd7afa240ca' ANDt_cube_snapshot.deleted_atIS NULL ORDER BYt_cube_snapshot.idLIMIT 1 9月 20 17:58:42 VM-0-6-tencentos bash5538: 9月 20 17:58:42 VM-0-6-tencentos bash5538: 2026/09/20 17:58:42 /workspace/CubeMaster/pkg/templatecenter/store.go:1118 record not found 9月 20 17:58:42 VM-0-6-tencentos bash5538: 0.379ms rows:0 SELECT * FROMt_cube_template_definitionWHERE template_id = 'tpl-464f1b5538764bd7afa240ca' ANDt_cube_template_definition.deleted_atIS NULL ORDER BYt_cube_template_definition.idLIMIT 1 9月 20 17:58:42 VM-0-6-tencentos bash5538: 9月 20 17:58:42 VM-0-6-tencentos bash5538: 2026/09/20 17:58:42 /workspace/CubeMaster/pkg/templatecenter/snapshot_table.go:29 record not found 9月 20 17:58:42 VM-0-6-tencentos bash5538: 0.342ms rows:0 SELECT * FROMt_cube_snapshotWHERE snapshot_id = 'tpl-464f1b5538764bd7afa240ca' ANDt_cube_snapshot.deleted_atIS NULL ORDER BYt_cube_snapshot.idLIMIT 1 9月 20 17:59:12 VM-0-6-tencentos bash5538: 9月 20 17:59:12 VM-0-6-tencentos bash5538: 2026/09/20 17:59:12 /workspace/CubeMaster/pkg/templatecenter/store.go:1118 record not found 9月 20 17:59:12 VM-0-6-tencentos bash5538: 0.332ms rows:0 SELECT * FROMt_cube_template_definitionWHERE template_id = 'tpl-464f1b5538764bd7afa240ca' ANDt_cube_template_definition.deleted_atIS NULL ORDER BYt_cube_template_definition.idLIMIT 1 9月 20 17:59:12 VM-0-6-tencentos bash5538: 9月 20 17:59:12 VM-0-6-tencentos bash5538: 2026/09/20 17:59:12 /workspace/CubeMaster/pkg/templatecenter/snapshot_table.go:29 record not found 9月 20 17:59:12 VM-0-6-tencentos bash5538: 0.341ms rows:0 SELECT * FROMt_cube_snapshotWHERE snapshot_id = 'tpl-464f1b5538764bd7afa240ca' ANDt_cube_snapshot.deleted_atIS NULL ORDER BYt_cube_snapshot.idLIMIT 1 9月 20 17:59:42 VM-0-6-tencentos bash5538: 9月 20 17:59:42 VM-0-6-tencentos bash5538: 2026/09/20 17:59:42 /workspace/CubeMaster/pkg/templatecenter/store.go:1118 record not found 9月 20 17:59:42 VM-0-6-tencentos bash5538: 0.394ms rows:0 SELECT * FROMt_cube_template_definitionWHERE template_id = 'tpl-464f1b5538764bd7afa240ca' ANDt_cube_template_definition.deleted_atIS NULL ORDER BYt_cube_template_definition.idLIMIT 1 9月 20 17:59:42 VM-0-6-tencentos bash5538: 9月 20 17:59:42 VM-0-6-tencentos bash5538: 2026/09/20 17:59:42 /workspace/CubeMaster/pkg/templatecenter/snapshot_table.go:29 record not found 9月 20 17:59:42 VM-0-6-tencentos bash5538: 0.366ms rows:0 SELECT * FROMt_cube_snapshotWHERE snapshot_id = 'tpl-464f1b5538764bd7afa240ca' ANDt_cube_snapshot.deleted_atIS NULL ORDER BYt_cube_snapshot.idLIMIT 1 9月 20 18:00:12 VM-0-6-tencentos bash5538: 9月 20 18:00:12 VM-0-6-tencentos bash5538: 2026/09/20 18:00:12 /workspace/CubeMaster/pkg/templatecenter/store.go:1118 record not found 9月 20 18:00:12 VM-0-6-tencentos bash5538: 0.400ms rows:0 SELECT * FROMt_cube_template_definitionWHERE template_id = 'tpl-464f1b5538764bd7afa240ca' ANDt_cube_template_definition.deleted_atIS NULL ORDER BYt_cube_template_definition.idLIMIT 1 9月 20 18:00:12 VM-0-6-tencentos bash5538: 9月 20 18:00:12 VM-0-6-tencentos bash5538: 2026/09/20 18:00:12 /workspace/CubeMaster/pkg/templatecenter/snapshot_table.go:29 record not found 9月 20 18:00:12 VM-0-6-tencentos bash5538: 0.385ms rows:0 SELECT * FROMt_cube_snapshotWHERE snapshot_id = 'tpl-464f1b5538764bd7afa240ca' ANDt_cube_snapshot.deleted_atIS NULL ORDER BYt_cube_snapshot.idLIMIT 1 9月 20 18:00:42 VM-0-6-tencentos bash5538: 9月 20 18:00:42 VM-0-6-tencentos bash5538: 2026/09/20 18:00:42 /workspace/CubeMaster/pkg/templatecenter/store.go:1118 record not found 9月 20 18:00:42 VM-0-6-tencentos bash5538: 0.382ms rows:0 SELECT * FROMt_cube_template_definitionWHERE template_id = 'tpl-464f1b5538764bd7afa240ca' ANDt_cube_template_definition.deleted_atIS NULL ORDER BYt_cube_template_definition.idLIMIT 1 9月 20 18:00:42 VM-0-6-tencentos bash5538: 9月 20 18:00:42 VM-0-6-tencentos bash5538: 2026/09/20 18:00:42 /workspace/CubeMaster/pkg/templatecenter/snapshot_table.go:29 record not found 9月 20 18:00:42 VM-0-6-tencentos bash5538: 0.407ms rows:0 SELECT * FROMt_cube_snapshotWHERE snapshot_id = 'tpl-464f1b5538764bd7afa240ca' ANDt_cube_snapshot.deleted_atIS NULL ORDER BYt_cube_snapshot.idLIMIT 1 9月 20 18:01:12 VM-0-6-tencentos bash5538: 9月 20 18:01:12 VM-0-6-tencentos bash5538: 2026/09/20 18:01:12 /workspace/CubeMaster/pkg/templatecenter/store.go:1118 record not found 9月 20 18:01:12 VM-0-6-tencentos bash5538: 0.360ms rows:0 SELECT * FROMt_cube_template_definitionWHERE template_id = 'tpl-464f1b5538764bd7afa240ca' ANDt_cube_template_definition.deleted_atIS NULL ORDER BYt_cube_template_definition.idLIMIT 1 9月 20 18:01:12 VM-0-6-tencentos bash5538: 9月 20 18:01:12 VM-0-6-tencentos bash5538: 2026/09/20 18:01:12 /workspace/CubeMaster/pkg/templatecenter/snapshot_table.go:29 record not found 9月 20 18:01:12 VM-0-6-tencentos bash5538: 0.417ms rows:0 SELECT * FROMt_cube_snapshotWHERE snapshot_id = 'tpl-464f1b5538764bd7afa240ca' ANDt_cube_snapshot.deleted_atIS NULL ORDER BYt_cube_snapshot.idLIMIT 1 9月 20 18:01:42 VM-0-6-tencentos bash5538: 9月 20 18:01:42 VM-0-6-tencentos bash5538: 2026/09/20 18:01:42 /workspace/CubeMaster/pkg/templatecenter/store.go:1118 record not found 9月 20 18:01:42 VM-0-6-tencentos bash5538: 0.389ms rows:0 SELECT * FROMt_cube_template_definitionWHERE template_id = 'tpl-464f1b5538764bd7afa240ca' ANDt_cube_template_definition.deleted_atIS NULL ORDER BYt_cube_template_definition.idLIMIT 1 9月 20 18:01:42 VM-0-6-tencentos bash5538: 9月 20 18:01:42 VM-0-6-tencentos bash5538: 2026/09/20 18:01:42 /workspace/CubeMaster/pkg/templatecenter/snapshot_table.go:29 record not found 9月 20 18:01:42 VM-0-6-tencentos bash5538: 0.374ms rows:0 SELECT * FROMt_cube_snapshotWHERE snapshot_id = 'tpl-464f1b5538764bd7afa240ca' ANDt_cube_snapshot.deleted_atIS NULL ORDER BYt_cube_snapshot.idLIMIT 1 9月 20 18:02:12 VM-0-6-tencentos bash5538: 9月 20 18:02:12 VM-0-6-tencentos bash5538: 2026/09/20 18:02:12 /workspace/CubeMaster/pkg/templatecenter/store.go:1118 record not found 9月 20 18:02:12 VM-0-6-tencentos bash5538: 0.389ms rows:0 SELECT * FROMt_cube_template_definitionWHERE template_id = 'tpl-464f1b5538764bd7afa240ca' ANDt_cube_template_definition.deleted_atIS NULL ORDER BYt_cube_template_definition.idLIMIT 1
从目前日志看,Cube Sandbox 主链路已经基本起来了:
- ✅
cubelet已正常运行 - ✅
cubemaster已正常运行 - ✅
cube-lifecycle-manager已正常运行 - ✅
cubeops :3010 - ✅
cube-api :3000 - ✅
cube-proxy可能无日志(需要确认) - ❌ 当前异常集中在 Template / Snapshot 数据缺失
重点不是 cubelet 启动问题,而是 模板中心状态不一致
1. cubelet 的 Killed 不是故障
这里:
cubelet-start.sh: line 57:
1710 Killed
实际发生:
initial pid=1710
...
stable pid=1949
你的脚本逻辑:
(
export CUBE_SANDBOX_NODE_IP
exec cubelet ...
) &
先启动一个探测进程:
1710
然后:
refresh_pidfile_from_pattern
找到真正 daemon:
1949
所以:
systemctl status
Main PID: 1949
正常
2. cubelet API 端口判断
你测试:
curl http://127.0.0.1:9999/health
返回:
HTTP/0.9
说明:
9999 不是 HTTP 服务
从监听:
9966
9998
9999
看:
大概率:
| 端口 | 用途 |
|---|---|
| 9966 | metrics |
| 9998 | ttrpc |
| 9999 | cubelet grpc/http2 |
所以:
curl 9999
不适用
测试:
curl http://127.0.0.1:9966/metrics | head
返回:
404 page not found
说明:
9966 也不是标准 prometheus path
建议查看:
cat /proc/1949/net/tcp
或者:
strings /usr/local/services/cubetoolbox/Cubelet/bin/cubelet | grep -Ei "metric|health|debug"
3. 当前真正问题:Template 不存在
核心日志:
SELECT * FROM t_cube_template_definition
WHERE template_id='tpl-464f1b5538764bd7afa240ca'
结果:
rows:0
还有:
SELECT * FROM t_cube_snapshot
WHERE snapshot_id='tpl-464f1b5538764bd7afa240ca'
结果:
rows:0
说明:
CubeMaster 收到请求:
template_id
tpl-464f1b5538764bd7afa240ca
但是数据库:
t_cube_template_definition
没有记录
4. 检查 TemplateCenter
你有:
/usr/local/services/cubetoolbox/CubeTemplateCenter
执行:
systemctl status cube-sandbox-cube-templatecenter
然后:
journalctl \
-u cube-sandbox-cube-templatecenter \
-n 200 \
--no-pager
重点看:
有没有:
mysql connect failed
redis failed
register template failed
5. 查数据库里的模板
需要找到 CubeMaster 数据库
先找配置:
grep -R "mysql\|database\|dsn" \
/usr/local/services/cubetoolbox \
-n
找到 DB:
例如:
mysql://user:pass@tcp(...)
进入 mysql:
mysql -h xxx -u xxx -p
查看:
show tables;
重点:
select count(*)
from t_cube_template_definition;
查看模板:
select
template_id,
name,
status
from t_cube_template_definition;
6. 检查默认镜像
你有:
/usr/local/services/cubetoolbox/cube-image/
里面:
cube-guest-image-cpu.img
version
但是:
/data/snapshot_pack
为空:
drwxr-xr-x snapshot_pack
└── disks
没有磁盘
说明:
目前只是安装了组件:
Cubelet
CubeMaster
TemplateCenter
但是:
模板镜像没有导入
7. 导入模板流程需要确认
搜索脚本:
find /usr/local/services/cubetoolbox \
-type f \
| grep -Ei "init|template|image|import|snapshot"
重点看:
cubetemplatecenter-postcheck.sh
内容:
cat \
/usr/local/services/cubetoolbox/scripts/systemd/cubetemplatecenter-postcheck.sh
8. 检查 TemplateCenter API
看看监听:
ss -lntp | grep template
或者:
docker ps
因为:
你现在:
CubeTemplateCenter/bin/templatecenter
应该是 systemd
查看:
systemctl status cube-sandbox-cube-templatecenter
9. 检查 cube-api 调用链
现在链路:
cube-api
|
cube-master :8089
|
TemplateCenter
|
DB
|
image storage
你的状态:
cube-api OK
cubemaster OK
cubelet OK
TemplateCenter 未确认
DB 有访问
Template数据 缺失
image 缺失
10. 下一步建议执行这组命令
请贴输出:
systemctl status cube-sandbox-cube-templatecenter
journalctl \
-u cube-sandbox-cube-templatecenter \
-n 100 \
--no-pager
grep -R "t_cube_template_definition" \
/usr/local/services/cubetoolbox \
-n
以及:
ls -lh \
/usr/local/services/cubetoolbox/cube-image/
目前判断:
不是 Cubelet、不是 systemd、不是网络问题
根因概率最高:
Cube Sandbox 安装完成,但默认 Guest Template / Snapshot 没有初始化导入,导致 CubeMaster 周期性查询不存在的 tpl-* 资源
root@VM-0-6-tencentos \~# systemctl status cube-sandbox-cube-templatecenter ● cube-sandbox-cube-templatecenter.service - Cube Sandbox CubeTemplateCenter Loaded: loaded (/etc/systemd/system/cube-sandbox-cube-templatecenter.service; enabled; preset: disabled) Active: active (running) since Sun 2026-09-20 16:24:44 CST; 2h 1min ago Process: 5910 ExecStartPost=/usr/bin/bash /usr/local/services/cubetoolbox/scripts/systemd/cubetemplatecenter-postcheck.sh (code=exited, status=0/SUCCESS) Main PID: 5907 (templatecenter) Tasks: 28 (limit: 153562) Memory: 45.8M (peak: 48.5M) CPU: 9.938s CGroup: /system.slice/cube-sandbox-cube-templatecenter.service └─5907 /usr/local/services/cubetoolbox/CubeTemplateCenter/bin/templatecenter 9月 20 16:24:43 VM-0-6-tencentos systemd1: Starting cube-sandbox-cube-templatecenter.service - Cube Sandbox CubeTemplateCenter... 9月 20 16:24:43 VM-0-6-tencentos bash5907: 2026-09-20T16:24:43.375476072+08:00,listen file/usr/local/services/cubetoolbox/CubeTemplateCenter/conf.yaml 9月 20 16:24:43 VM-0-6-tencentos bash5907: cfg:{"Common":{"MockUpdateAction":false,"DebugDumpHttpBody":false,"MockDebug":false,"MockNodeNum":0,"MockCreateDirect":false,"MockCreateDirectHandle":false,"MockHttpDirect":false,"MockCreateSleep":0,"MockPercents":null,"CubeDestroyCheckFilter":false,"Debug":{"Address":""},"HttpPort":8090,"HttpBind":"127.0.0.1","Write> 9月 20 16:24:43 VM-0-6-tencentos bash5907: 2026/09/20 16:24:43 migrate: 1 applied migration version(s) have no fingerprint baseline and are NOT content-checked (e.g. applied before fingerprinting existed): 0 9月 20 16:24:43 VM-0-6-tencentos bash5907: {"LocalIp":"192.168.0.6","Module":"","@timestamp":"2026-09-20T16:24:43.38896107+08:00","Action":"","InstanceId":"","CostTime":0,"Caller":"","CallerIp":"","LogLevel":"INFO","Callee":"","RetCode":0,"InstanceType":"","FunctionType":"","LogContent":"dao schema migration completed (driver= db=cube_mvp)","Version":"","Call> 9月 20 16:24:43 VM-0-6-tencentos bash5907: {"AppId":0,"Action":"","LocalIp":"192.168.0.6","InstanceType":"","CostTime":0,"Caller":"","InstanceId":"","LogLevel":"ERROR","RequestId":"","ContainerId":"","CodeLine":"","@timestamp":"2026-09-20T16:24:43.392978879+08:00","RetCode":0,"Version":"","Callee":"","CallerIp":"","LogContent":"templatecenter node identity: C> 9月 20 16:24:43 VM-0-6-tencentos bash5907: {"Callee":"","LogContent":"templatecenter listen address: CUBE_TEMPLATE_CENTER_NODE_IP=192.168.0.6 ignored for the listen address: http_bind=\"127.0.0.1\" is set explicitly","InstanceId":"","CallerIp":"","AppId":0,"InstanceType":"","Version":"","Caller":"","CodeLine":"","CalleeAction":"","FunctionType":"","RequestId"> 9月 20 16:24:43 VM-0-6-tencentos bash5907: {"Namespace":"","CostTime":0,"InstanceType":"","InstanceId":"","ContainerId":"","RetCode":0,"CallerIp":"","CalleeEndpoint":"","Version":"","Action":"","LogContent":"templatecenter successfully booted in 0.014476s","LocalIp":"192.168.0.6","Module":"","CodeLine":"","RequestId":"","Caller":"","CalleeAction":"","Function> 9月 20 16:24:44 VM-0-6-tencentos systemd1: Started cube-sandbox-cube-templatecenter.service - Cube Sandbox CubeTemplateCenter. root@VM-0-6-tencentos \~# journalctl \ -u cube-sandbox-cube-templatecenter \ -n 100 \ --no-pager 9月 20 15:36:49 VM-0-6-tencentos systemd1: Starting cube-sandbox-cube-templatecenter.service - Cube Sandbox CubeTemplateCenter... 9月 20 15:36:49 VM-0-6-tencentos bash12505: 2026-09-20T15:36:49.937762237+08:00,listen file/usr/local/services/cubetoolbox/CubeTemplateCenter/conf.yaml 9月 20 15:36:49 VM-0-6-tencentos bash12505: cfg:{"Common":{"MockUpdateAction":false,"DebugDumpHttpBody":false,"MockDebug":false,"MockNodeNum":0,"MockCreateDirect":false,"MockCreateDirectHandle":false,"MockHttpDirect":false,"MockCreateSleep":0,"MockPercents":null,"CubeDestroyCheckFilter":false,"Debug":{"Address":""},"HttpPort":8090,"HttpBind":"127.0.0.1","WriteTimeout":360,"ReadTimeout":120,"IdleTimeout":360,"GraceFullStopTimeoutInSec":120,"CubeOpsAddr":"","MasterAddr":"http://127.0.0.1:8089","CubeOpsBootRetries":5,"CubeOpsBootBackoff":1000000000,"SyncMetaDataInterval":1000000000,"SyncMetricDataInterval":1000000000,"CleanSandboxCacheInterval":7200000000000,"EnabledListRunningSandboxCache":false,"AsyncTaskQueueSize":10000,"AsyncTaskWorkerNum":32,"HeadlessServiceName":"","DefaultHeadlessServiceNodesNum":1,"ListFilterOutLables":null,"CollectMetricInterval":100000000,"ReportLocalCreateNum":false,"ReportStdevMetric":false,"GwCacheExpiredTime":15000000000,"GwCacheEnable":false,"ReportGWRedisGetMetric":false,"EnableGetStatusFromCubelet":false,"DisableHardDelete":false,"TemplateCacheTTL":21600000000000,"TemplateCenterAddrValue":"","CollectSandboxMemoryWhitelist":null,"EnableAllCollectSandboxMemory":false,"FilterErrMsgErrorCode":null,"DescribeInstancesWhiteList":null,"DescribeTaskExpireTime":86400,"EnablePrivateIpQuery":false,"DbMaxRetryCount":5,"DbRetryInterval":5000000,"EnableCheckComNetIDParam":false,"EnableDescribeInstanceFromRedis":false,"MaxNICQueue":4,"DisableCreateImageCluster":null,"EnableAGSColdStartSwitch":false},"AuthConf":{"Enable":false,"SignatureExpireTimeInsec":120,"SecretKeyMap":null},"Log":{"Region":"","Cluster":"","Module":"templatecenter","Path":"/data/log/CubeTemplateCenter","FileSize":100,"FileNum":10,"Level":"info","EnableLogMetric":false},"CubeletConf":{"Grpc":{"GrpcPort":9999,"CleanConnTaskIntervalInMin":60,"CleanConnTaskRoutinePoolSize":64,"ConnExpireTimeInSec":180},"CommonTimeoutInsec":30,"CreateImageTimeoutInSec":300,"AppSnapshotTimeoutInSec":300,"DefaultTimeoutInsec":0,"CreateTimeoutInsec":600,"AsyncFlows":null,"RetryCode":null,"LoopRetryCode":null,"ReuseRetryCode":null,"CircuitBreakCode":null,"ExcludeLoopRetryCode":null,"BackoffRetryCode":null,"MaxRetries":5,"LoopMaxRetries":100,"BufferQueueMinJob":10,"CreateConcurrentLimit":100,"DestroyConcurentLimit":50,"ExposedPortList":null,"EnableExposedPort":false,"DisableRedisProxyPort":false,"MaxDelayInSecond":1,"BackoffRetryDelay":5000000},"InstanceDBConfig":{"Driver":"","Addr":"127.0.0.1:3306","User":"cube","Pwd":"cube_pass","DBName":"cube_mvp","ConnTimeout":5,"ReadTimeout":5,"WriteTimeout":5,"MaxIdleConns":5,"MaxOpenConns":20,"MaxConnLifeTimeSeconds":300,"MigrationLockTimeoutSeconds":0},"RedisConf":{"Password":"ceuhvu123","MaxActive":32,"MaxIdle":8,"IdleTimeout":30,"DbNo":0,"Nodes":"127.0.0.1:6379","MaxRetry":2,"MasterName":"","SentinelNodes":"","SentinelPassword":"","NodeMetricTTLSec":600,"SandboxProxyTTLSec":0},"ExtraConf":{"BlkQos":"{}","BlkQosMap":null,"FsQos":"{}","FsQosMap":null,"NetQosList":"\[\]","AllowedHostMountPrefixes":null},"Scheduler":{"Overhead":{"VmMemoryOverheadBase":"42Mi","VmMemoryOverheadCoefficient":64,"HostMemoryOverheadBase":"24Mi","CubeMsgMemoryOverhead":"16Mi","VmCpuOverhead":"0","HostCpuOverhead":"0.3"},"NodeMaxMvmNum":3000,"NodeMaxMvmNumReserveNumPercent":1,"NodeMaxMemReservedInMB":10240,"NodeMaxCpuUtil":80,"PreSelectNum":-1,"PrioritySelectNum":-1,"LeastSelectName":"random","MetricUpdateTimeout":3600000000000,"LocalMetricUpdateTimeout":3600000000000,"Filter":null,"Score":null,"PostScore":null,"DisableCircuitFilter":false,"InBackoffMode":false,"AffinityConf":null,"NodeMaxMvmNumConf":null,"EnableRunInstanceHostIps":false,"MaxMvmCPU":"","MaxMvmMemory":"","DiskUsageMaxPercent":80,"LargeSizeAffinityConf":null,"NodeMaxMemReservedConf":null,"DisableBackoffFilterInstanceType":null,"ThirtpartyFilterInstanceType":null,"InstanceTypeConf":null,"NodeAffinitySelectorAllowedKeys":null,"IgnoreRedisAllocation":false,"DeprecatedOvercommitRatio":null,"DeprecatedOvercommitRatioByType":null},"ReqTemplateConf":null,"HookWhitelist":null,"CubeEgressConf":null,"CubeProxyConf":{"AdminPort":8082,"AdminToken":"","AdminURLs":null,"HeartbeatTTLMs":15000},"SoftDeletePurge":null,"VolumePlugins":null} 9月 20 15:36:49 VM-0-6-tencentos bash12505: 2026/09/20 15:36:49 migrate: 1 applied migration version(s) have no fingerprint baseline and are NOT content-checked (e.g. applied before fingerprinting existed): 0 9月 20 15:36:49 VM-0-6-tencentos bash12505: {"CallerIp":"","FunctionType":"","InstanceType":"","Action":"","CostTime":0,"AppId":0,"LogContent":"dao schema migration completed (driver= db=cube_mvp)","LocalIp":"192.168.0.6","CodeLine":"","@timestamp":"2026-09-20T15:36:49.95661439+08:00","RequestId":"","Callee":"","RetCode":0,"CalleeAction":"","CalleeEndpoint":"","Module":"","Version":"","Namespace":"","LogLevel":"INFO","ContainerId":"","Caller":"","InstanceId":""} 9月 20 15:36:49 VM-0-6-tencentos bash12505: {"LocalIp":"192.168.0.6","CallerIp":"","AppId":0,"RetCode":0,"InstanceType":"","RequestId":"","InstanceId":"","CostTime":0,"CalleeAction":"","Action":"","CalleeEndpoint":"","ContainerId":"","LogContent":"templatecenter node identity: CUBE_TEMPLATE_CENTER_NODE_IP=192.168.0.6","Module":"","@timestamp":"2026-09-20T15:36:49.963805389+08:00","LogLevel":"ERROR","Version":"","Namespace":"","CodeLine":"","FunctionType":"","Callee":"","Caller":""} 9月 20 15:36:49 VM-0-6-tencentos bash12505:
从你贴出的日志看,CubeTemplateCenter 服务本身启动正常,没有启动失败问题 。当前重点应该是排查 模板定义(t_cube_template_definition)为空 / 镜像模板未注册 / CubeTemplateCenter无法加载模板 这一类问题
我先分析现象
1. CubeTemplateCenter 状态正常
当前:
Active: active (running)
Main PID: 5907
启动流程:
dao schema migration completed (driver= db=cube_mvp)
templatecenter successfully booted in 0.014476s
说明:
✅ 二进制正常
✅ 配置文件可读取
✅ MySQL schema migration正常
✅ Redis配置没有阻塞
✅ 服务监听成功
2. 注意这个 WARN:监听地址问题
日志:
templatecenter listen address:
CUBE_TEMPLATE_CENTER_NODE_IP=192.168.0.6 ignored
for the listen address:
http_bind="127.0.0.1" is set explicitly
你的配置:
HttpPort:8090
HttpBind:"127.0.0.1"
意味着:
127.0.0.1:8090
只能本机访问
如果 CubeMaster、CubeOps、其他节点访问 TemplateCenter,需要修改:
文件:
/usr/local/services/cubetoolbox/CubeTemplateCenter/conf.yaml
找到:
http_bind: 127.0.0.1
改:
http_bind: 192.168.0.6
或者:
http_bind: 0.0.0.0
然后:
systemctl restart cube-sandbox-cube-templatecenter
验证:
ss -lntp | grep 8090
应该看到:
正常:
LISTEN 0 4096 192.168.0.6:8090
或者:
LISTEN 0 4096 0.0.0.0:8090
3. t_cube_template_definition 搜索方式不对
你执行:
grep -R "t_cube_template_definition" /usr/local/services/cubetoolbox
结果:
templatecenter binary 匹配
cubemaster binary 匹配
cubeops binary 匹配
说明:
这是 Go 二进制内嵌字符串
不是数据库表
应该查 MySQL
4. 查询 Cube 数据库
日志:
DBName:"cube_mvp"
Addr:"127.0.0.1:3306"
User:"cube"
Pwd:"cube_pass"
登录:
mysql -h127.0.0.1 \
-u cube \
-pcube_pass \
cube_mvp
然后:
查看模板相关表
执行:
show tables;
重点找:
t_cube_template_definition
或者:
show tables like '%template%';
5. 查看模板定义
如果存在:
select * from t_cube_template_definition;
重点字段:
例如:
id
name
version
image
status
created_at
正常应该有类似:
cpu
gpu
ubuntu
centos
模板
6. 你现在 cube-image 只有一个镜像
你这里:
/usr/local/services/cubetoolbox/cube-image/
cube-guest-image-cpu.img 232M
version
说明:
当前只有:
CPU Guest Image
没有 GPU image
如果你要跑:
- GPU Sandbox
- NVIDIA CUDA
- AI环境
需要额外镜像,例如:
cube-guest-image-gpu.img
否则模板创建会失败
7. 检查 CubeTemplateCenter API
本机测试:
curl http://127.0.0.1:8090/
或者:
curl http://127.0.0.1:8090/health
查看:
curl -v http://127.0.0.1:8090/
8. 检查 CubeMaster
你的配置:
MasterAddr:"http://127.0.0.1:8089"
说明:
TemplateCenter依赖:
CubeMaster
|
|
TemplateCenter
|
|
Cubelet
检查:
systemctl status cube-sandbox-cube-master
以及:
ss -lntp | grep 8089
应该:
127.0.0.1:8089
9. 检查 CubeOps
配置:
CubeOpsAddr:""
但是日志没有报错
检查:
systemctl status cube-sandbox-cube-ops
10. 当前最可能问题
根据你现在的信息:
服务层
| 组件 | 状态 |
|---|---|
| CubeTemplateCenter | ✅正常 |
| MySQL | ✅正常 |
| Redis | 未知 |
| CubeMaster | 需要确认 |
| 模板数据 | 未知 |
| 镜像 | 只有CPU |
我建议下一步执行:
① 查看数据库模板
mysql -h127.0.0.1 -ucube -pcube_pass cube_mvp
执行:
show tables like '%template%';
select count(*) from t_cube_template_definition;
② 查看 TemplateCenter 注册接口
执行:
curl -s http://127.0.0.1:8090/metrics | head
③ 查看所有 Cube 服务
systemctl list-units \
| grep cube-sandbox
你目前更像是 Cube Sandbox 初始化完成,但模板中心没有加载模板镜像定义 ,不是 TemplateCenter 启动故障。下一步重点看 t_cube_template_definition 数据和 CubeMaster 状态
http_bind改为192.168.0.6后无法启动
root@VM-0-6-tencentos \~#systemctl restart cube-sandbox-cube-templatecenter
Job for cube-sandbox-cube-templatecenter.service failed because the control process exited with error code.
See "systemctl status cube-sandbox-cube-templatecenter.service" and "journalctl -xeu cube-sandbox-cube-templatecenter.service" for details.
从前面的日志看,真正的问题不是 CUBE_TEMPLATE_CENTER_NODE_IP,而是 http_bind 修改后的监听行为
原配置:
HttpPort:8090
HttpBind:"127.0.0.1"
启动正常:
templatecenter successfully booted in 0.014476s
但是日志已经提示:
templatecenter listen address:
CUBE_TEMPLATE_CENTER_NODE_IP=192.168.0.6 ignored for the listen address:
http_bind="127.0.0.1" is set explicitly
说明:
CUBE_TEMPLATE_CENTER_NODE_IP只影响节点身份注册- 真正 HTTP 服务监听地址由
http_bind控制
你改成:
http_bind="192.168.0.6"
后启动失败,需要进一步确认原因
1. 先查看具体失败原因
执行:
systemctl status cube-sandbox-cube-templatecenter -l
以及:
journalctl -xeu cube-sandbox-cube-templatecenter --no-pager
重点看有没有:
情况1:监听失败
例如:
bind: cannot assign requested address
说明:
192.168.0.6 并不是本机地址
确认:
ip addr | grep 192.168.0.6
应该看到:
inet 192.168.0.6/xx
情况2:端口占用
例如:
listen tcp 192.168.0.6:8090: bind: address already in use
检查:
ss -lntp | grep 8090
2. 检查 conf.yaml 修改是否正确
找到:
grep -n "http_bind" \
/usr/local/services/cubetoolbox/CubeTemplateCenter/conf.yaml
应该类似:
http_bind: "192.168.0.6"
注意:
不要写:
http_bind=192.168.0.6
不要写:
http_bind:192.168.0.6
3. 推荐生产配置方式
CubeSandbox 多节点环境不建议直接绑定固定 IP
推荐:
http_bind: "0.0.0.0"
即:
Common:
HttpPort: 8090
HttpBind: "0.0.0.0"
原因:
| 配置 | 效果 |
|---|---|
| 127.0.0.1 | 只能本机访问 |
| 192.168.0.6 | 只监听这个网卡 |
| 0.0.0.0 | 监听所有网卡 |
生产环境通常:
TemplateCenter
|
|
192.168.0.6:8090
|
其它Cube节点访问
应该使用:
0.0.0.0:8090
而不是:
192.168.0.6:8090
4. 修改方式
编辑:
vi /usr/local/services/cubetoolbox/CubeTemplateCenter/conf.yaml
修改:
http_bind: "0.0.0.0"
然后:
systemctl daemon-reload
systemctl restart cube-sandbox-cube-templatecenter
检查:
ss -lntp | grep 8090
正常应该:
LISTEN 0 4096 0.0.0.0:8090
5. 同时检查 CubeMaster 配置
你的 TemplateCenter 配置:
MasterAddr:"http://127.0.0.1:8089"
说明当前是单机部署
如果 CubeSandbox 是多节点:
需要检查:
grep -R "TemplateCenter" \
/usr/local/services/cubetoolbox/CubeMaster/conf.yaml
以及:
grep -R "CubeTemplateCenter" \
/usr/local/services/cubetoolbox
应该类似:
http://192.168.0.6:8090
否则其它 Cubelet 无法访问模板中心
6. 你的当前环境判断
根据日志:
LocalIp":"192.168.0.6"
以及:
CUBE_TEMPLATE_CENTER_NODE_IP=192.168.0.6
节点IP没有问题
当前最佳配置:
CUBE_TEMPLATE_CENTER_NODE_IP=192.168.0.6
http_bind=0.0.0.0
HttpPort=8090
然后其它组件访问:
http://192.168.0.6:8090
root@VM-0-6-tencentos \~# systemctl status cube-sandbox-cube-templatecenter -l ● cube-sandbox-cube-templatecenter.service - Cube Sandbox CubeTemplateCenter Loaded: loaded (/etc/systemd/system/cube-sandbox-cube-templatecenter.service; enabled; preset: disabled) Active: activating (auto-restart) (Result: exit-code) since Sun 2026-09-20 18:38:43 CST; 1s ago Process: 240327 ExecStart=/usr/bin/bash /usr/local/services/cubetoolbox/scripts/systemd/cubetemplatecenter-start.sh (code=exited, status=0/SUCCESS) Process: 240328 ExecStartPost=/usr/bin/bash /usr/local/services/cubetoolbox/scripts/systemd/cubetemplatecenter-postcheck.sh (code=exited, status=1/FAILURE) Main PID: 240327 (code=exited, status=0/SUCCESS) CPU: 281ms root@VM-0-6-tencentos \~# journalctl -xeu cube-sandbox-cube-templatecenter --no-pager 9月 20 16:24:43 VM-0-6-tencentos systemd1: Starting cube-sandbox-cube-templatecenter.service - Cube Sandbox CubeTemplateCenter... ░░ Subject: cube-sandbox-cube-templatecenter.service 单元已开始启动 ░░ Defined-By: systemd ░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel ░░ ░░ cube-sandbox-cube-templatecenter.service 单元已开始启动。 9月 20 16:24:43 VM-0-6-tencentos bash5907: 2026-09-20T16:24:43.375476072+08:00,listen file/usr/local/services/cubetoolbox/CubeTemplateCenter/conf.yaml 9月 20 16:24:43 VM-0-6-tencentos bash5907: cfg:{"Common":{"MockUpdateAction":false,"DebugDumpHttpBody":false,"MockDebug":false,"MockNodeNum":0,"MockCreateDirect":false,"MockCreateDirectHandle":false,"MockHttpDirect":false,"MockCreateSleep":0,"MockPercents":null,"CubeDestroyCheckFilter":false,"Debug":{"Address":""},"HttpPort":8090,"HttpBind":"127.0.0.1","WriteTimeout":360,"ReadTimeout":120,"IdleTimeout":360,"GraceFullStopTimeoutInSec":120,"CubeOpsAddr":"","MasterAddr":"http://127.0.0.1:8089","CubeOpsBootRetries":5,"CubeOpsBootBackoff":1000000000,"SyncMetaDataInterval":1000000000,"SyncMetricDataInterval":1000000000,"CleanSandboxCacheInterval":7200000000000,"EnabledListRunningSandboxCache":false,"AsyncTaskQueueSize":10000,"AsyncTaskWorkerNum":32,"HeadlessServiceName":"","DefaultHeadlessServiceNodesNum":1,"ListFilterOutLables":null,"CollectMetricInterval":100000000,"ReportLocalCreateNum":false,"ReportStdevMetric":false,"GwCacheExpiredTime":15000000000,"GwCacheEnable":false,"ReportGWRedisGetMetric":false,"EnableGetStatusFromCubelet":false,"DisableHardDelete":false,"TemplateCacheTTL":21600000000000,"TemplateCenterAddrValue":"","CollectSandboxMemoryWhitelist":null,"EnableAllCollectSandboxMemory":false,"FilterErrMsgErrorCode":null,"DescribeInstancesWhiteList":null,"DescribeTaskExpireTime":86400,"EnablePrivateIpQuery":false,"DbMaxRetryCount":5,"DbRetryInterval":5000000,"EnableCheckComNetIDParam":false,"EnableDescribeInstanceFromRedis":false,"MaxNICQueue":4,"DisableCreateImageCluster":null,"EnableAGSColdStartSwitch":false},"AuthConf":{"Enable":false,"SignatureExpireTimeInsec":120,"SecretKeyMap":null},"Log":{"Region":"","Cluster":"","Module":"templatecenter","Path":"/data/log/CubeTemplateCenter","FileSize":100,"FileNum":10,"Level":"info","EnableLogMetric":false},"CubeletConf":{"Grpc":{"GrpcPort":9999,"CleanConnTaskIntervalInMin":60,"CleanConnTaskRoutinePoolSize":64,"ConnExpireTimeInSec":180},"CommonTimeoutInsec":30,"CreateImageTimeoutInSec":300,"AppSnapshotTimeoutInSec":300,"DefaultTimeoutInsec":0,"CreateTimeoutInsec":600,"AsyncFlows":null,"RetryCode":null,"LoopRetryCode":null,"ReuseRetryCode":null,"CircuitBreakCode":null,"ExcludeLoopRetryCode":null,"BackoffRetryCode":null,"MaxRetries":5,"LoopMaxRetries":100,"BufferQueueMinJob":10,"CreateConcurrentLimit":100,"DestroyConcurentLimit":50,"ExposedPortList":null,"EnableExposedPort":false,"DisableRedisProxyPort":false,"MaxDelayInSecond":1,"BackoffRetryDelay":5000000},"InstanceDBConfig":{"Driver":"","Addr":"127.0.0.1:3306","User":"cube","Pwd":"cube_pass","DBName":"cube_mvp","ConnTimeout":5,"ReadTimeout":5,"WriteTimeout":5,"MaxIdleConns":5,"MaxOpenConns":20,"MaxConnLifeTimeSeconds":300,"MigrationLockTimeoutSeconds":0},"RedisConf":{"Password":"ceuhvu123","MaxActive":32,"MaxIdle":8,"IdleTimeout":30,"DbNo":0,"Nodes":"127.0.0.1:6379","MaxRetry":2,"MasterName":"","SentinelNodes":"","SentinelPassword":"","NodeMetricTTLSec":600,"SandboxProxyTTLSec":0},"ExtraConf":{"BlkQos":"{}","BlkQosMap":null,"FsQos":"{}","FsQosMap":null,"NetQosList":"\[\]","AllowedHostMountPrefixes":null},"Scheduler":{"Overhead":{"VmMemoryOverheadBase":"42Mi","VmMemoryOverheadCoefficient":64,"HostMemoryOverheadBase":"24Mi","CubeMsgMemoryOverhead":"16Mi","VmCpuOverhead":"0","HostCpuOverhead":"0.3"},"NodeMaxMvmNum":3000,"NodeMaxMvmNumReserveNumPercent":1,"NodeMaxMemReservedInMB":10240,"NodeMaxCpuUtil":80,"PreSelectNum":-1,"PrioritySelectNum":-1,"LeastSelectName":"random","MetricUpdateTimeout":3600000000000,"LocalMetricUpdateTimeout":3600000000000,"Filter":null,"Score":null,"PostScore":null,"DisableCircuitFilter":false,"InBackoffMode":false,"AffinityConf":null,"NodeMaxMvmNumConf":null,"EnableRunInstanceHostIps":false,"MaxMvmCPU":"","MaxMvmMemory":"","DiskUsageMaxPercent":80,"LargeSizeAffinityConf":null,"NodeMaxMemReservedConf":null,"DisableBackoffFilterInstanceType":null,"ThirtpartyFilterInstanceType":null,"InstanceTypeConf":null,"NodeAffinitySelectorAllowedKeys":null,"IgnoreRedisAllocation":false,"DeprecatedOvercommitRatio":null,"DeprecatedOvercommitRatioByType":null},"ReqTemplateConf":null,"HookWhitelist":null,"CubeEgressConf":null,"CubeProxyConf":{"AdminPort":8082,"AdminToken":"","AdminURLs":null,"HeartbeatTTLMs":15000},"SoftDeletePurge":null,"VolumePlugins":null} 9月 20 16:24:43 VM-0-6-tencentos bash5907: 2026/09/20 16:24:43 migrate: 1 applied migration version(s) have no fingerprint baseline and are NOT content-checked (e.g. applied before fingerprinting existed): 0 9月 20 16:24:43 VM-0-6-tencentos bash5907: {"LocalIp":"192.168.0.6","Module":"","@timestamp":"2026-09-20T16:24:43.38896107+08:00","Action":"","InstanceId":"","CostTime":0,"Caller":"","CallerIp":"","LogLevel":"INFO","Callee":"","RetCode":0,"InstanceType":"","FunctionType":"","LogContent":"dao schema migration completed (driver= db=cube_mvp)","Version":"","CalleeEndpoint":"","RequestId":"","AppId":0,"CodeLine":"","Namespace":"","CalleeAction":"","ContainerId":""} 9月 20 16:24:43 VM-0-6-tencentos bash5907: {"AppId":0,"Action":"","LocalIp":"192.168.0.6","InstanceType":"","CostTime":0,"Caller":"","InstanceId":"","LogLevel":"ERROR","RequestId":"","ContainerId":"","CodeLine":"","@timestamp":"2026-09-20T16:24:43.392978879+08:00","RetCode":0,"Version":"","Callee":"","CallerIp":"","LogContent":"templatecenter node identity: CUBE_TEMPLATE_CENTER_NODE_IP=192.168.0.6","Module":"","CalleeAction":"","Namespace":"","FunctionType":"","CalleeEndpoint":""} 9月 20 16:24:43 VM-0-6-tencentos bash5907: {"Callee":"","LogContent":"templatecenter listen address: CUBE_TEMPLATE_CENTER_NODE_IP=192.168.0.6 ignored for the listen address: http_bind=\"127.0.0.1\" is set explicitly","InstanceId":"","CallerIp":"","AppId":0,"InstanceType":"","Version":"","Caller":"","CodeLine":"","CalleeAction":"","FunctionType":"","RequestId":"","CostTime":0,"LocalIp":"192.168.0.6","Module":"","LogLevel":"WARN","ContainerId":"","RetCode":0,"Namespace":"","CalleeEndpoint":"","@timestamp":"2026-09-20T16:24:43.393017301+08:00","Action":""} 9月 20 16:24:43 VM-0-6-tencentos bash5907: {"Namespace":"","CostTime":0,"InstanceType":"","InstanceId":"","ContainerId":"","RetCode":0,"CallerIp":"","CalleeEndpoint":"","Version":"","Action":"","LogContent":"templatecenter successfully booted in 0.014476s","LocalIp":"192.168.0.6","Module":"","CodeLine":"","RequestId":"","Caller":"","CalleeAction":"","FunctionType":"","@timestamp":"2026-09-20T16:24:43.394146817+08:00","LogLevel":"ERROR","AppId":0,"Callee":""} 9月 20 16:24:44 VM-0-6-tencentos systemd1: Started cube-sandbox-cube-templatecenter.service - Cube Sandbox CubeTemplateCenter. ░░ Subject: cube-sandbox-cube-templatecenter.service 单元已结束启动 ░░ Defined-By: systemd ░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel ░░ ░░ cube-sandbox-cube-templatecenter.service 单元已结束启动。 ░░ ░░ 启动结果为"done"。 9月 20 18:28:05 VM-0-6-tencentos bash5907: 2026-09-20T18:28:05.383136309+08:00,OnFileEvent:WRITE "/usr/local/services/cubetoolbox/CubeTemplateCenter/conf.yaml" 9月 20 18:28:05 VM-0-6-tencentos bash5907: setLogLevel:info 9月 20 18:28:05 VM-0-6-tencentos bash5907: 2026-09-20T18:28:05.384444503+08:00,OnFileEvent:CHMOD "/usr/local/services/cubetoolbox/CubeTemplateCenter/conf.yaml" 9月 20 18:28:06 VM-0-6-tencentos bash5907: {"CostTime":0,"LogLevel":"INFO","RetCode":0,"AppId":0,"Module":"","FunctionType":"","InstanceId":"","Caller":"","CalleeAction":"","ContainerId":"","Version":"","LogContent":"templatecenter received signal terminated, shutting down","LocalIp":"192.168.0.6","@timestamp":"2026-09-20T18:28:06.568537334+08:00","Namespace":"","RequestId":"","InstanceType":"","CallerIp":"","Callee":"","CodeLine":"","CalleeEndpoint":"","Action":""} 9月 20 18:28:06 VM-0-6-tencentos bash5907: {"Caller":"","LogLevel":"ERROR","Callee":"","ContainerId":"","AppId":0,"InstanceId":"","CalleeEndpoint":"","CalleeAction":"","LogContent":"templatecenter shutting down: cancelling in-flight builds","FunctionType":"","CostTime":0,"LocalIp":"192.168.0.6","Module":"","@timestamp":"2026-09-20T18:28:06.568596615+08:00","Namespace":"","RequestId":"","Version":"","RetCode":0,"InstanceType":"","CodeLine":"","Action":"","CallerIp":""} 9月 20 18:28:06 VM-0-6-tencentos bash5907: {"@timestamp":"2026-09-20T18:28:06.568607436+08:00","LogLevel":"ERROR","LogContent":"templatecenter stopped gracefully begin, pid 5907","LocalIp":"192.168.0.6","Module":"","CodeLine":""} 9月 20 18:28:06 VM-0-6-tencentos bash5907: {"CodeLine":"","@timestamp":"2026-09-20T18:28:06.56866807+08:00","LogLevel":"ERROR","LogContent":"templatecenter InternalHttp Shutdown succ","LocalIp":"192.168.0.6","Module":""} 9月 20 18:28:06 VM-0-6-tencentos systemd1: Stopping cube-sandbox-cube-templatecenter.service - Cube Sandbox CubeTemplateCenter... ░░ Subject: cube-sandbox-cube-templatecenter.service 单元已开始停止操作 ░░ Defined-By: systemd ░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel ░░ ░░ cube-sandbox-cube-templatecenter.service 单元已开始停止操作。 9月 20 18:28:06 VM-0-6-tencentos systemd1: cube-sandbox-cube-templatecenter.service: Deactivated successfully. ░░ Subject: Unit succeeded ░░ Defined-By: systemd ░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel ░░ ░░ The unit cube-sandbox-cube-templatecenter.service has successfully entered the 'dead' state. 9月 20 18:28:06 VM-0-6-tencentos systemd1: Stopped cube-sandbox-cube-templatecenter.service - Cube Sandbox CubeTemplateCenter. ░░ Subject: cube-sandbox-cube-templatecenter.service 单元已结束停止操作 ░░ Defined-By: systemd ░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel ░░ ░░ cube-sandbox-cube-templatecenter.service 单元已结束停止操作。 9月 20 18:28:06 VM-0-6-tencentos systemd1: cube-sandbox-cube-templatecenter.service: Consumed 10.129s CPU time, 48.5M memory peak, 0B memory swap peak. ░░ Subject: Resources consumed by unit runtime ░░ Defined-By: systemd ░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel ░░ ░░ The unit cube-sandbox-cube-templatecenter.service completed and consumed the indicated resources. 9月 20 18:28:06 VM-0-6-tencentos systemd1: Starting cube-sandbox-cube-templatecenter.service - Cube Sandbox CubeTemplateCenter... ░░ Subject: cube-sandbox-cube-templatecenter.service 单元已开始启动 ░░ Defined-By: systemd ░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel ░░ ░░ cube-sandbox-cube-templatecenter.service 单元已开始启动。 9月 20 18:28:06 VM-0-6-tencentos bash220740: 2026-09-20T18:28:06.627410234+08:00,listen file/usr/local/services/cubetoolbox/CubeTemplateCenter/conf.yaml 9月 20 18:28:06 VM-0-6-tencentos bash220740: cfg:{"Common":{"MockUpdateAction":false,"DebugDumpHttpBody":false,"MockDebug":false,"MockNodeNum":0,"MockCreateDirect":false,"MockCreateDirectHandle":false,"MockHttpDirect":false,"MockCreateSleep":0,"MockPercents":null,"CubeDestroyCheckFilter":false,"Debug":{"Address":""},"HttpPort":8090,"HttpBind":"0.0.0.0","WriteTimeout":360,"ReadTimeout":120,"IdleTimeout":360,"GraceFullStopTimeoutInSec":120,"CubeOpsAddr":"","MasterAddr":"http://192.168.0.6:8089","CubeOpsBootRetries":5,"CubeOpsBootBackoff":1000000000,"SyncMetaDataInterval":1000000000,"SyncMetricDataInterval":1000000000,"CleanSandboxCacheInterval":7200000000000,"EnabledListRunningSandboxCache":false,"AsyncTaskQueueSize":10000,"AsyncTaskWorkerNum":32,"HeadlessServiceName":"","DefaultHeadlessServiceNodesNum":1,"ListFilterOutLables":null,"CollectMetricInterval":100000000,"ReportLocalCreateNum":false,"ReportStdevMetric":false,"GwCacheExpiredTime":15000000000,"GwCacheEnable":false,"ReportGWRedisGetMetric":false,"EnableGetStatusFromCubelet":false,"DisableHardDelete":false,"TemplateCacheTTL":21600000000000,"TemplateCenterAddrValue":"","CollectSandboxMemoryWhitelist":null,"EnableAllCollectSandboxMemory":false,"FilterErrMsgErrorCode":null,"DescribeInstancesWhiteList":null,"DescribeTaskExpireTime":86400,"EnablePrivateIpQuery":false,"DbMaxRetryCount":5,"DbRetryInterval":5000000,"EnableCheckComNetIDParam":false,"EnableDescribeInstanceFromRedis":false,"MaxNICQueue":4,"DisableCreateImageCluster":null,"EnableAGSColdStartSwitch":false},"AuthConf":{"Enable":false,"SignatureExpireTimeInsec":120,"SecretKeyMap":null},"Log":{"Region":"","Cluster":"","Module":"templatecenter","Path":"/data/log/CubeTemplateCenter","FileSize":100,"FileNum":10,"Level":"info","EnableLogMetric":false},"CubeletConf":{"Grpc":{"GrpcPort":9999,"CleanConnTaskIntervalInMin":60,"CleanConnTaskRoutinePoolSize":64,"ConnExpireTimeInSec":180},"CommonTimeoutInsec":30,"CreateImageTimeoutInSec":300,"AppSnapshotTimeoutInSec":300,"DefaultTimeoutInsec":0,"CreateTimeoutInsec":600,"AsyncFlows":null,"RetryCode":null,"LoopRetryCode":null,"ReuseRetryCode":null,"CircuitBreakCode":null,"ExcludeLoopRetryCode":null,"BackoffRetryCode":null,"MaxRetries":5,"LoopMaxRetries":100,"BufferQueueMinJob":10,"CreateConcurrentLimit":100,"DestroyConcurentLimit":50,"ExposedPortList":null,"EnableExposedPort":false,"DisableRedisProxyPort":false,"MaxDelayInSecond":1,"BackoffRetryDelay":5000000},"InstanceDBConfig":{"Driver":"","Addr":"127.0.0.1:3306","User":"cube","Pwd":"cube_pass","DBName":"cube_mvp","ConnTimeout":5,"ReadTimeout":5,"WriteTimeout":5,"MaxIdleConns":5,"MaxOpenConns":20,"MaxConnLifeTimeSeconds":300,"MigrationLockTimeoutSeconds":0},"RedisConf":{"Password":"ceuhvu123","MaxActive":32,"MaxIdle":8,"IdleTimeout":30,"DbNo":0,"Nodes":"127.0.0.1:6379","MaxRetry":2,"MasterName":"","SentinelNodes":"","SentinelPassword":"","NodeMetricTTLSec":600,"SandboxProxyTTLSec":0},"ExtraConf":{"BlkQos":"{}","BlkQosMap":null,"FsQos":"{}","FsQosMap":null,"NetQosList":"\[\]","AllowedHostMountPrefixes":null},"Scheduler":{"Overhead":{"VmMemoryOverheadBase":"42Mi","VmMemoryOverheadCoefficient":64,"HostMemoryOverheadBase":"24Mi","CubeMsgMemoryOverhead":"16Mi","VmCpuOverhead":"0","HostCpuOverhead":"0.3"},"NodeMaxMvmNum":3000,"NodeMaxMvmNumReserveNumPercent":1,"NodeMaxMemReservedInMB":10240,"NodeMaxCpuUtil":80,"PreSelectNum":-1,"PrioritySelectNum":-1,"LeastSelectName":"random","MetricUpdateTimeout":3600000000000,"LocalMetricUpdateTimeout":3600000000000,"Filter":null,"Score":null,"PostScore":null,"DisableCircuitFilter":false,"InBackoffMode":false,"AffinityConf":null,"NodeMaxMvmNumConf":null,"EnableRunInstanceHostIps":false,"MaxMvmCPU":"","MaxMvmMemory":"","DiskUsageMaxPercent":80,"LargeSizeAffinityConf":null,"NodeMaxMemReservedConf":null,"DisableBackoffFilterInstanceType":null,"ThirtpartyFilterInstanceType":null,"InstanceTypeConf":null,"NodeAffinitySelectorAllowedKeys":null,"IgnoreRedisAllocation":false,"DeprecatedOvercommitRatio":null,"DeprecatedOvercommitRatioByType":null},"ReqTemplateConf":null,"HookWhitelist":null,"CubeEgressConf":null,"CubeProxyConf":{"AdminPort":8082,"AdminToken":"","AdminURLs":null,"HeartbeatTTLMs":15000},"SoftDeletePurge":null,"VolumePlugins":null} 9月 20 18:28:06 VM-0-6-tencentos bash220740: 2026/09/20 18:28:06 migrate: 1 applied migration version(s) have no fingerprint baseline and are NOT content-checked (e.g. applied before fingerprinting existed): 0 9月 20 18:28:06 VM-0-6-tencentos bash220740: {"Action":"","Version":"","CallerIp":"","CalleeAction":"","LogContent":"dao schema migration completed (driver= db=cube_mvp)","CodeLine":"","@timestamp":"2026-09-20T18:28:06.63496995+08:00","LogLevel":"INFO","Callee":"","FunctionType":"","AppId":0,"ContainerId":"","InstanceId":"","Module":"","InstanceType":"","RetCode":0,"RequestId":"","LocalIp":"192.168.0.6","CalleeEndpoint":"","CostTime":0,"Namespace":"","Caller":""} 9月 20 18:28:06 VM-0-6-tencentos bash220740: {"Module":"","AppId":0,"Namespace":"","ContainerId":"","Caller":"","CodeLine":"","InstanceId":"","RetCode":0,"CostTime":0,"RequestId":"","Action":"","CalleeEndpoint":"","FunctionType":"","@timestamp":"2026-09-20T18:28:06.637522696+08:00","LogLevel":"ERROR","Version":"","CalleeAction":"","Callee":"","LogContent":"templatecenter node identity: CUBE_TEMPLATE_CENTER_NODE_IP=192.168.0.6","LocalIp":"192.168.0.6","CallerIp":"","InstanceType":""} 9月 20 18:28:06 VM-0-6-tencentos bash220740: {"Callee":"","RequestId":"","Version":"","InstanceType":"","InstanceId":"","Action":"","CostTime":0,"CodeLine":"","@timestamp":"2026-09-20T18:28:06.637677728+08:00","LogLevel":"WARN","AppId":0,"Caller":"","Namespace":"","RetCode":0,"LogContent":"templatecenter listen address: listening on 192.168.0.6 from CUBE_TEMPLATE_CENTER_NODE_IP","LocalIp":"192.168.0.6","Module":"","ContainerId":"","CalleeEndpoint":"","CalleeAction":"","CallerIp":"","FunctionType":""} 9月 20 18:28:06 VM-0-6-tencentos bash220740: {"Module":"","CodeLine":"","@timestamp":"2026-09-20T18:28:06.637922399+08:00","LogLevel":"ERROR","CostTime":0,"Caller":"","FunctionType":"","AppId":0,"LocalIp":"192.168.0.6","Namespace":"","CallerIp":"","CalleeEndpoint":"","Version":"","InstanceId":"","InstanceType":"","RequestId":"","LogContent":"templatecenter successfully booted in 0.009064s","Callee":"","Action":"","RetCode":0,"CalleeAction":"","ContainerId":""} 9月 20 18:28:36 VM-0-6-tencentos bash220741: one-click-systemd ERROR wait_for_http timeout: url=http://127.0.0.1:8090/health waited=30s last_curl_error=curl: (7) Failed to connect to 127.0.0.1 port 8090 after 0 ms: Couldn't connect to server 9月 20 18:28:36 VM-0-6-tencentos bash220741: one-click-systemd ERROR: cubetemplatecenter health not ready 9月 20 18:28:36 VM-0-6-tencentos systemd1: cube-sandbox-cube-templatecenter.service: Control process exited, code=exited, status=1/FAILURE ░░ Subject: Unit process exited ░░ Defined-By: systemd ░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel ░░ ░░ An ExecStartPost= process belonging to unit cube-sandbox-cube-templatecenter.service has exited. ░░ ░░ The process' exit code is 'exited' and its exit status is 1. 9月 20 18:28:36 VM-0-6-tencentos bash220740: {"@timestamp":"2026-09-20T18:28:36.777787861+08:00","AppId":0,"Action":"","CostTime":0,"LogLevel":"INFO","InstanceType":"","LogContent":"templatecenter received signal terminated, shutting down","Module":"","CodeLine":"","InstanceId":"","ContainerId":"","CallerIp":"","Caller":"","Version":"","Namespace":"","LocalIp":"192.168.0.6","CalleeAction":"","RequestId":"","FunctionType":"","Callee":"","RetCode":0,"CalleeEndpoint":""} 9月 20 18:28:36 VM-0-6-tencentos bash220740: {"LogLevel":"ERROR","RetCode":0,"AppId":0,"Caller":"","Callee":"","CostTime":0,"Module":"","FunctionType":"","InstanceType":"","Version":"","LocalIp":"192.168.0.6","CodeLine":"","Namespace":"","ContainerId":"","InstanceId":"","@timestamp":"2026-09-20T18:28:36.777843506+08:00","Action":"","RequestId":"","CalleeEndpoint":"","CallerIp":"","CalleeAction":"","LogContent":"templatecenter shutting down: cancelling in-flight builds"} 9月 20 18:28:36 VM-0-6-tencentos bash[220740
从日志看,CubeTemplateCenter 本身启动成功,失败点在 systemd 的健康检查脚本,不是程序启动失败
关键日志:
templatecenter successfully booted in 0.009064s
说明服务已经起来
但是随后:
[one-click-systemd] ERROR wait_for_http timeout:
url=http://127.0.0.1:8090/health
curl: (7) Failed to connect to 127.0.0.1 port 8090
然后:
ExecStartPost=/usr/bin/bash ... cubetemplatecenter-postcheck.sh
status=1/FAILURE
systemd 认为启动失败,于是发送停止信号:
templatecenter received signal terminated, shutting down
根因
你修改了:
原来:
HttpBind: "127.0.0.1"
改成:
HttpBind: "192.168.0.6"
或者:
HttpBind: "0.0.0.0"
程序监听地址变了
但是健康检查脚本仍然固定检查:
curl http://127.0.0.1:8090/health
所以:
程序:
192.168.0.6:8090
监听
健康检查:
127.0.0.1:8090
访问不到
验证
执行:
ss -lntp | grep 8090
你应该看到类似:
当前配置 192.168.0.6
LISTEN 0 4096 192.168.0.6:8090
或者:
LISTEN 0 4096 0.0.0.0:8090
测试:
curl http://192.168.0.6:8090/health
应该正常
但是:
curl http://127.0.0.1:8090/health
失败
解决方案 1(推荐):修改 postcheck 健康检查地址
查看:
cat /usr/local/services/cubetoolbox/scripts/systemd/cubetemplatecenter-postcheck.sh
应该类似:
wait_for_http http://127.0.0.1:8090/health
改:
wait_for_http http://192.168.0.6:8090/health
或者更推荐:
使用环境变量:
CUBE_TEMPLATE_CENTER_NODE_IP=192.168.0.6
例如:
wait_for_http http://${CUBE_TEMPLATE_CENTER_NODE_IP}:8090/health
然后:
systemctl daemon-reload
systemctl restart cube-sandbox-cube-templatecenter
解决方案 2(更推荐生产):监听 0.0.0.0 + localhost健康检查
生产环境建议:
conf.yaml:
Common:
HttpBind: "0.0.0.0"
HttpPort: 8090
保持:
curl http://127.0.0.1:8090/health
不变
原因:
- 服务监听所有网卡
- 本机健康检查稳定
- 避免绑定具体 IP 后 IP变化导致故障
即:
+-------------+
192.168.0.6 ------> | |
| TemplateCtr |
127.0.0.1 --------> | |
+-------------+
:8090
解决方案 3:增加127.0.0.1绑定(不推荐)
如果程序支持多监听,可以:
HttpBind: "0.0.0.0"
不要写:
127.0.0.1,192.168.0.6
很多 Go HTTP 服务不支持
另外检查一个隐藏问题
你的配置里:
MasterAddr":"http://192.168.0.6:8089"
之前:
MasterAddr":"http://127.0.0.1:8089"
如果 CubeMaster 也监听:
ss -lntp | grep 8089
确认:
192.168.0.6:8089
或者:
0.0.0.0:8089
否则 TemplateCenter 后续可能连接 Master 失败
建议最终生产配置
CubeTemplateCenter/conf.yaml
Common:
HttpBind: "0.0.0.0"
HttpPort: 8090
MasterAddr: "http://192.168.0.6:8089"
环境:
export CUBE_TEMPLATE_CENTER_NODE_IP=192.168.0.6
systemd:
ExecStartPost=/usr/bin/bash /usr/local/services/cubetoolbox/scripts/systemd/cubetemplatecenter-postcheck.sh
postcheck:
curl -f http://127.0.0.1:8090/health
然后:
systemctl daemon-reload
systemctl restart cube-sandbox-cube-templatecenter
systemctl status cube-sandbox-cube-templatecenter -l