CubeSandBox配置CubeMaster CPU 超分

当前配置:

  • CubeSandbox Server:v0.7.1

  • CubeMaster:v0.7.1,构建时间为 2026-09-11

  • 在 /etc、/opt、/data/cubelet 中没有搜到 overcommit_ratio,说明当前配置可能使用了其他配置文件路径、不同的参数名称,或者默认值;不能据此认定 CPU 超分没有生效

接下来建议先定位 v0.7.1 实际加载的配置,再调整 CPU 超分,避免修改错误配置导致沙盒调度异常

1. v0.7.1 配置文件位置

CubeSandbox 官方文档给出的单机安装配置路径是:

/usr/local/services/cubetoolbox/CubeMaster/conf.yaml

CubeMaster 的 systemd 服务通常是 cube-sandbox-cubemaster.service

先执行以下命令,检查你的实际安装情况:

复制代码
# 1. 查看 CubeMaster 服务及启动参数
systemctl status cube-sandbox-cubemaster.service --no-pager
systemctl cat cube-sandbox-cubemaster.service

# 2. 检查官方常见配置路径
ls -lah /usr/local/services/cubetoolbox/CubeMaster/
grep -nE 'overcommit|quota|cpu|mem|scheduler' \
  /usr/local/services/cubetoolbox/CubeMaster/conf.yaml 2>/dev/null

# 3. 查找其他组件配置
find /usr/local/services/cubetoolbox /etc \
  -type f \( -name 'conf.yaml' -o -name '*.yaml' -o -name '*.yml' \
  -o -name 'config.toml' \) 2>/dev/null

2. 确认 CPU 超分配置

CubeSandbox 上游曾提供调度器超分参数,默认 CPU 比例为 3、内存比例为 2,并支持按实例类型单独配置

不过,这不能直接证明 v0.7.1 的单机安装包采用相同的配置结构。因此,先不要直接往 conf.yaml 添加字段,也不要贸然重启服务

另外,上游有关于并发创建时资源超额分配的公开问题:调度器使用的资源快照可能存在约 1 秒的延迟,而且 Cubelet 与 CubeMaster 的资源配额系数可能叠加。即使配置了超分比例,也不应把它当作严格的资源隔离保证

3. 下一步

请把下面两条命令的输出定位实际配置入口:

复制代码
systemctl cat cube-sandbox-cubemaster.service

sed -n '1,240p' \
  /usr/local/services/cubetoolbox/CubeMaster/conf.yaml

第二条如果提示文件不存在,也请把报错发来。配置文件可能包含数据库密码或密钥,提交前请先脱敏

我会根据你的 v0.7.1 实际配置,进一步确定如何调整 CPU 超分比例、是否还需调整 Cubelet 配额,以及修改后如何验证生效

root@VM-0-44-tencentos \~# systemctl status cube-sandbox-cubemaster.service --no-pager ● cube-sandbox-cubemaster.service - Cube Sandbox CubeMaster Loaded: loaded (/etc/systemd/system/cube-sandbox-cubemaster.service; disabled; preset: disabled) Active: active (running) since Tue 2026-10-13 16:11:54 CST; 3 days ago Main PID: 150318 (cubemaster) Tasks: 65 (limit: 461732) Memory: 209.4M (peak: 218.8M) CPU: 1h 47min 25.280s CGroup: /system.slice/cube-sandbox-cubemaster.service └─150318 /usr/local/services/cubetoolbox/CubeMaster/bin/cubemaster Oct 17 11:27:36 VM-0-44-tencentos bash150318: 0.625ms rows:0 SELECT * FROM t_cube_snapshot WHERE snapshot_id = 'tpl-bb00da010cb142298f1e16aa' AND t_cube_snapshot.deleted_at IS NULL ORDER BY t_cube_snapshot.id LIMIT 1 Oct 17 11:27:36 VM-0-44-tencentos bash150318: Oct 17 11:27:36 VM-0-44-tencentos bash150318: 2026/10/17 11:27:36 /workspace/CubeMaster/pkg/templatecenter/snapshot_table.go:29 record not found Oct 17 11:27:36 VM-0-44-tencentos bash150318: 0.272ms rows:0 SELECT * FROM t_cube_snapshot WHERE snapshot_id = 'tpl-bb00da010cb142298f1e16aa' AND t_cube_snapshot.deleted_at IS NULL ORDER BY t_cube_snapshot.id LIMIT 1 Oct 17 11:27:36 VM-0-44-tencentos bash150318: Oct 17 11:27:36 VM-0-44-tencentos bash150318: 2026/10/17 11:27:36 /workspace/CubeMaster/pkg/pausesnap/store.go:327 record not found Oct 17 11:27:36 VM-0-44-tencentos bash150318: 0.269ms rows:0 SELECT * FROM t_cube_pause_snapshot WHERE sandbox_id = '701e116bd3f547eb840a6aa81b7d6447' ORDER BY updated_at desc,t_cube_pause_snapshot.id LIMIT 1 Oct 17 11:27:36 VM-0-44-tencentos bash150318: Oct 17 11:27:36 VM-0-44-tencentos bash150318: 2026/10/17 11:27:36 /workspace/CubeMaster/pkg/templatecenter/snapshot_table.go:29 record not found Oct 17 11:27:36 VM-0-44-tencentos bash150318: 0.327ms rows:0 SELECT * FROM t_cube_snapshot WHERE snapshot_id = 'tpl-bb00da010cb142298f1e16aa' AND t_cube_snapshot.deleted_at IS NULL ORDER BY t_cube_snapshot.id LIMIT 1 root@VM-0-44-tencentos \~# systemctl cat cube-sandbox-cubemaster.service # /etc/systemd/system/cube-sandbox-cubemaster.service Unit Description=Cube Sandbox CubeMaster After=network-online.target cube-sandbox-mysql.service cube-sandbox-redis.service cube-sandbox-cubeops.service Wants=network-online.target cube-sandbox-mysql.service cube-sandbox-redis.service cube-sandbox-cubeops.service ConditionPathExists=/usr/local/services/cubetoolbox/.one-click.env RequiresMountsFor=/data/CubeMaster/storage PartOf=cube-sandbox-control.target Service Type=simple User=root WorkingDirectory=/usr/local/services/cubetoolbox EnvironmentFile=-/usr/local/services/cubetoolbox/.one-click.env Environment=CUBE_MASTER_CONFIG_PATH=/usr/local/services/cubetoolbox/CubeMaster/conf.yaml ExecStart=/usr/bin/bash /usr/local/services/cubetoolbox/scripts/systemd/cubemaster-start.sh ExecStartPost=/usr/bin/bash /usr/local/services/cubetoolbox/scripts/systemd/cubemaster-postcheck.sh Restart=on-failure RestartSec=2s # Override short distro defaults explicitly so SIGTERM has room to drain DB # connections / flush state before SIGKILL. TimeoutStartSec=60s TimeoutStopSec=30s Install WantedBy=cube-sandbox-control.target root@VM-0-44-tencentos \~# ls -lah /usr/local/services/cubetoolbox/CubeMaster/ total 32K drwxr-xr-x 4 1001 1001 103 Sep 20 18:19 . drwxr-xr-x 25 1001 1001 4.0K Sep 22 09:58 .. drwxr-xr-x 2 1001 1001 45 Sep 11 18:11 bin -rw-r--r-- 1 1001 1001 4.8K Sep 20 18:19 conf.yaml -rw-r--r-- 1 1001 1001 5.4K Sep 11 17:56 Dockerfile -rwxr-xr-x 1 1001 1001 4.8K Sep 11 18:11 docker-install-volume-deps.sh drwxr-xr-x 2 1001 1001 4.0K Sep 20 18:00 plugin root@VM-0-44-tencentos \~# root@VM-0-44-tencentos \~# grep -nE 'overcommit|quota|cpu|mem|scheduler' \ /usr/local/services/cubetoolbox/CubeMaster/conf.yaml 2>/dev/null 102:scheduler: 108: - "cpu" 109: - "mem" root@VM-0-44-tencentos \~# find /usr/local/services/cubetoolbox /etc \ -type f \( -name 'conf.yaml' -o -name '*.yaml' -o -name '*.yml' \ -o -name 'config.toml' \) 2>/dev/null /usr/local/services/cubetoolbox/CubeTemplateCenter/conf.yaml /usr/local/services/cubetoolbox/Cubelet/config/config.toml /usr/local/services/cubetoolbox/Cubelet/dynamicconf/conf.yaml /usr/local/services/cubetoolbox/webui/docker-compose.yaml /usr/local/services/cubetoolbox/support/docker-compose.yaml /usr/local/services/cubetoolbox/cubeproxy/docker-compose.yaml /usr/local/services/cubetoolbox/cube-lifecycle-manager/docker-compose.yaml /usr/local/services/cubetoolbox/CubeMaster/conf.yaml /etc/containerd/config.toml

从这次输出看,已经找到关键配置入口:

  • CubeMaster 配置文件:/usr/local/services/cubetoolbox/CubeMaster/conf.yaml

  • Cubelet 静态配置:/usr/local/services/cubetoolbox/Cubelet/config/config.toml

  • Cubelet 动态配置:/usr/local/services/cubetoolbox/Cubelet/dynamicconf/conf.yaml

  • CubeMaster 通过 CUBE_MASTER_CONFIG_PATH 环境变量加载 conf.yaml,服务当前正常运行。

在 CubeMaster 配置中已经定位到 scheduler:,但 grep 只显示了 cpu、mem 两个字段,还不足以判断调度器是否支持超分参数。CPU 超分可能涉及 CubeMaster 调度器分配比例和 Cubelet 实例资源配额两层配置,需要分别确认,避免重复放大资源比例

1. 先查看调度器的完整配置

执行:

复制代码
sed -n '85,145p' \
  /usr/local/services/cubetoolbox/CubeMaster/conf.yaml

再检查 Cubelet 的资源配置:

复制代码
grep -nEi -C 4 \
  'cpu|mem|quota|overcommit|resource|limit|factor|ratio' \
  /usr/local/services/cubetoolbox/Cubelet/config/config.toml \
  /usr/local/services/cubetoolbox/Cubelet/dynamicconf/conf.yaml

这两组命令只读取配置,不会修改或重启服务

2. 结合 v0.7.1 官方信息,特别注意这一点

CubeSandbox v0.7.1 发布于 2026-09-11。上游公开的超分配置说明确认了 overcommit_ratio 和 overcommit_ratio_conf 这两个参数,但公开的并发创建问题也指出,CubeMaster 与 Cubelet 的资源配额可能叠加,导致实际超分高于预期

建议在修改配置前,再执行下面这条命令,确认 Cubelet 是否配置了主机 CPU、内存配额:

复制代码
grep -n -A 30 -B 5 '\[host.quota\]' \
  /usr/local/services/cubetoolbox/Cubelet/config/config.toml

如果没有匹配到,再查看文件开头及相关配置:

复制代码
sed -n '1,220p' \
  /usr/local/services/cubetoolbox/Cubelet/config/config.toml

查看:

  1. CubeMaster/conf.yaml 的第 85--145 行

  2. Cubelet/config/config.toml 中 [host.quota] 附近的内容

拿到这两段后,就可以进一步判断应该只调 CubeMaster 调度比例,还是同时调整 Cubelet 的主机配额。暂时不要直接设置成 CPU 6 倍或更高;应先明确当前实际生效的配额,避免节点内存耗尽或沙盒运行不稳定

sed -n '85,145p' \ /usr/local/services/cubetoolbox/CubeMaster/conf.yaml redis: nodes: "127.0.0.1:6379" password: "ceuhvu123" db_no: 0 max_idle: 8 max_active: 32 idle_timeout: 30 max_retry: 2 # Safety TTL (seconds) for node-metric keys; refreshed on every heartbeat so # live nodes never expire, offline nodes auto-clean. <=0 disables. node_metric_ttl_sec: 600 # Sandbox proxy route TTL. Keep 0 (disabled): the route key has no refresh # path, so any TTL below the max sandbox lifetime would break live routing. # Teardown removes it via DEL. Only set > max sandbox lifetime if needed. sandbox_proxy_ttl_sec: 0 scheduler: priority_select_num: 1 metric_update_timeout: 300s local_metric_update_timeout: 300s filter: enable_filters: - "cpu" - "mem" - "template_locality" - "realtime_create_num" # Volume plugin (S3 / COS binary reference). Credentials: # CubeMaster/plugin/volume-s3.conf (written by one-click from MinIO or CUBE_S3_*) # CubeMaster/plugin/volume-cos.conf # List order sets the default driver when Volume.create() omits driver; # S3 is first so the default install (bundled MinIO) works out of the box. volume_plugins: - name: s3 type: binary binary_path: /usr/local/services/cubetoolbox/CubeMaster/plugin/cube-volume-s3 - name: cos type: binary binary_path: /usr/local/services/cubetoolbox/CubeMaster/plugin/cube-volume-cos extra_conf: allowed_host_mount_prefixes: - "/mnt/allin_data/allin/pod_log/data_share/llm-planning/" root@VM-0-44-tencentos \~# root@VM-0-44-tencentos \~# grep -nEi -C 4 \ 'cpu|mem|quota|overcommit|resource|limit|factor|ratio' \ /usr/local/services/cubetoolbox/Cubelet/config/config.toml \ /usr/local/services/cubetoolbox/Cubelet/dynamicconf/conf.yaml /usr/local/services/cubetoolbox/Cubelet/config/config.toml-26-cubetap /usr/local/services/cubetoolbox/Cubelet/config/config.toml-27- address="/data/cubelet/cubetap.sock" /usr/local/services/cubetoolbox/Cubelet/config/config.toml-28- gid = 0 /usr/local/services/cubetoolbox/Cubelet/config/config.toml-29- uid = 0 /usr/local/services/cubetoolbox/Cubelet/config/config.toml:30:operation_server /usr/local/services/cubetoolbox/Cubelet/config/config.toml:31: address="/data/cubelet/cubelet-operation.sock" /usr/local/services/cubetoolbox/Cubelet/config/config.toml-32- gid = 0 /usr/local/services/cubetoolbox/Cubelet/config/config.toml-33- uid = 0 /usr/local/services/cubetoolbox/Cubelet/config/config.toml-34- disable = true /usr/local/services/cubetoolbox/Cubelet/config/config.toml-35-debug -- /usr/local/services/cubetoolbox/Cubelet/config/config.toml-39- file_num = 10 # retained files (current + numbered backups) /usr/local/services/cubetoolbox/Cubelet/config/config.toml-40- file_size = "500m" # per-file roll size: 500m, 1g, 2g (binary MiB/GiB); unitless integer is MiB /usr/local/services/cubetoolbox/Cubelet/config/config.toml-41-plugins /usr/local/services/cubetoolbox/Cubelet/config/config.toml-42- plugins."io.cubelet.controller.config.v1.cubelet" /usr/local/services/cubetoolbox/Cubelet/config/config.toml:43: # Static-only cadence for node status and resource reporting. /usr/local/services/cubetoolbox/Cubelet/config/config.toml-44- # Do not configure this in dynamicconf/conf.yaml. /usr/local/services/cubetoolbox/Cubelet/config/config.toml-45- node_status_update_frequency = "1s" /usr/local/services/cubetoolbox/Cubelet/config/config.toml-46- cubeops_addr = "http://127.0.0.1:3010" /usr/local/services/cubetoolbox/Cubelet/config/config.toml-47- cubeops_timeout = "10m" -- /usr/local/services/cubetoolbox/Cubelet/config/config.toml-58- plugins."io.cubelet.internal.v1.cgroup" /usr/local/services/cubetoolbox/Cubelet/config/config.toml-59- pool_size = 3000 /usr/local/services/cubetoolbox/Cubelet/config/config.toml-60- pool_workers = 1 /usr/local/services/cubetoolbox/Cubelet/config/config.toml-61- pool_trigger_interval_in_ms = 1000 /usr/local/services/cubetoolbox/Cubelet/config/config.toml:62: vm_cpu_overhead = "0" /usr/local/services/cubetoolbox/Cubelet/config/config.toml:63: vm_memory_overhead_base = "42Mi" /usr/local/services/cubetoolbox/Cubelet/config/config.toml:64: vm_memory_overhead_coefficient = 64 /usr/local/services/cubetoolbox/Cubelet/config/config.toml:65: host_cpu_overhead = "0.3" /usr/local/services/cubetoolbox/Cubelet/config/config.toml:66: host_memory_overhead_base = "20Mi" /usr/local/services/cubetoolbox/Cubelet/config/config.toml-67- vm_snapshot_specs_config = "/usr/local/services/cubetoolbox/cube-snapshot/spec.json" /usr/local/services/cubetoolbox/Cubelet/config/config.toml-68- snapshot_disk_dir = "/data/snapshot_pack/disks" /usr/local/services/cubetoolbox/Cubelet/config/config.toml-69- /usr/local/services/cubetoolbox/Cubelet/config/config.toml:70: plugins."io.cubelet.internal.v1.resource-metrics" /usr/local/services/cubetoolbox/Cubelet/config/config.toml-71- enabled = true /usr/local/services/cubetoolbox/Cubelet/config/config.toml-72- collection_interval = "5s" /usr/local/services/cubetoolbox/Cubelet/config/config.toml-73- request_timeout = "2s" /usr/local/services/cubetoolbox/Cubelet/config/config.toml-74- max_concurrent_requests = 8 -- /usr/local/services/cubetoolbox/Cubelet/dynamicconf/conf.yaml-5- sandbox_exec_cmd_before_exist_log_out: false /usr/local/services/cubetoolbox/Cubelet/dynamicconf/conf.yaml-6- sandbox_exec_cmd_before_exist: /usr/local/services/cubetoolbox/Cubelet/dynamicconf/conf.yaml-7- - sh /usr/local/services/cubetoolbox/Cubelet/dynamicconf/conf.yaml-8- - "/usr/local/cubetools/cube_exec_cmd_before_exist.sh" /usr/local/services/cubetoolbox/Cubelet/dynamicconf/conf.yaml:9: cgroup_set_memory_reparent_file: "" /usr/local/services/cubetoolbox/Cubelet/dynamicconf/conf.yaml-10- disable_host_cgroup: true /usr/local/services/cubetoolbox/Cubelet/dynamicconf/conf.yaml-11- # Disable the legacy netfile. /usr/local/services/cubetoolbox/Cubelet/dynamicconf/conf.yaml-12- disable_host_netfile: true /usr/local/services/cubetoolbox/Cubelet/dynamicconf/conf.yaml-13- # Default DNS servers for guest containers. -- /usr/local/services/cubetoolbox/Cubelet/dynamicconf/conf.yaml-22- default_dns_options: \[\] /usr/local/services/cubetoolbox/Cubelet/dynamicconf/conf.yaml-23- /usr/local/services/cubetoolbox/Cubelet/dynamicconf/conf.yaml-24-host: /usr/local/services/cubetoolbox/Cubelet/dynamicconf/conf.yaml-25- scheduler_label: "default-cluster" /usr/local/services/cubetoolbox/Cubelet/dynamicconf/conf.yaml:26: quota: /usr/local/services/cubetoolbox/Cubelet/dynamicconf/conf.yaml:27: mcpu_limit: 0 /usr/local/services/cubetoolbox/Cubelet/dynamicconf/conf.yaml:28: mem_limit: "" /usr/local/services/cubetoolbox/Cubelet/dynamicconf/conf.yaml:29: mvm_limit: 0 /usr/local/services/cubetoolbox/Cubelet/dynamicconf/conf.yaml-30- creation_concurrent_num: 0 /usr/local/services/cubetoolbox/Cubelet/dynamicconf/conf.yaml:31: # Fraction of a paused sandbox's CPU/memory quota to release back to the /usr/local/services/cubetoolbox/Cubelet/dynamicconf/conf.yaml:32: # scheduler, in 0,1, applied symmetrically to CPU and memory. A paused /usr/local/services/cubetoolbox/Cubelet/dynamicconf/conf.yaml:33: # sandbox is snapshotted to disk with its MicroVM shut down, so the quota is /usr/local/services/cubetoolbox/Cubelet/dynamicconf/conf.yaml:34: # physically free; this ratio trades density against resume headroom: /usr/local/services/cubetoolbox/Cubelet/dynamicconf/conf.yaml:35: # 0.0 (default) release nothing -- paused keeps full quota, resume /usr/local/services/cubetoolbox/Cubelet/dynamicconf/conf.yaml-36- # guaranteed (legacy behaviour, safe zero-value default). /usr/local/services/cubetoolbox/Cubelet/dynamicconf/conf.yaml-37- # 1.0 release everything -- max density, resume best-effort and /usr/local/services/cubetoolbox/Cubelet/dynamicconf/conf.yaml-38- # rejected when the node can no longer fit the sandbox. /usr/local/services/cubetoolbox/Cubelet/dynamicconf/conf.yaml:39: # 0 < r < 1 release r, reserve (1-r) as headroom; the reserved quota /usr/local/services/cubetoolbox/Cubelet/dynamicconf/conf.yaml:40: # still shows up in QuotaCpuUsage/QuotaMemUsage, so /usr/local/services/cubetoolbox/Cubelet/dynamicconf/conf.yaml:41: # pause-heavy nodes are deprioritised by the cpu/mem scoring /usr/local/services/cubetoolbox/Cubelet/dynamicconf/conf.yaml:42: # factors. /usr/local/services/cubetoolbox/Cubelet/dynamicconf/conf.yaml:43: paused_resource_release_ratio: 0.0 /usr/local/services/cubetoolbox/Cubelet/dynamicconf/conf.yaml-44- gc: /usr/local/services/cubetoolbox/Cubelet/dynamicconf/conf.yaml:45: code_expiration_time: "72h" /usr/local/services/cubetoolbox/Cubelet/dynamicconf/conf.yaml:46: image_expiration_time: "24h" /usr/local/services/cubetoolbox/Cubelet/dynamicconf/conf.yaml-47- /usr/local/services/cubetoolbox/Cubelet/dynamicconf/conf.yaml-48-meta_server_config: /usr/local/services/cubetoolbox/Cubelet/dynamicconf/conf.yaml-49- # Endpoints are injected by the installer (one-click / prepare-compute-role) /usr/local/services/cubetoolbox/Cubelet/dynamicconf/conf.yaml-50- # or helm at deploy time. Defaults live in pkg/config/config.go (single-node).

根据上述实际配置,已经可以把 CubeSandbox v0.7.1 的几个资源参数区分开了

目前的配置判断

配置项 当前值 含义与注意事项
CubeMaster scheduler.filter cpu、mem 等 调度器启用 CPU、内存等资源筛选,不代表已经配置了超分比例
Cubelet host.quota.mcpu_limit 0 主机 CPU 配额参数;需结合 v0.7.1 源码确认 0 的具体语义
Cubelet host.quota.mem_limit 空字符串 主机内存配额参数;需确认空值的默认处理方式
host_cpu_overhead 0.3 主机 CPU 开销估算参数,不应直接理解为 CPU 超分比例
paused_resource_release_ratio 0.0 暂停沙盒时不释放其资源配额;这会影响暂停实例较多时的可调度容量

关键结论:现在不能仅靠修改 mcpu_limit 来实现 CPU 超分。 需要确认调度器的超分比例到底由哪个配置项控制,以及 Cubelet 的配额如何参与资源统计

2. 配置 CubeMaster CPU 超分比例

上游调度器支持 overcommit_ratio,其中 CPU 和内存比例分别使用 cpu_ratio、mem_ratio。默认值为 CPU 3、内存 2

可以先备份配置,再在 scheduler: 下增加该参数。下面是建议的配置示例,修改前应确认 v0.7.1 的配置解析器接受此结构

复制代码
scheduler:
  priority_select_num: 1
  metric_update_timeout: 300s
  local_metric_update_timeout: 300s

  overcommit_ratio:
    cpu_ratio: 2
    mem_ratio: 2

  filter:
    enable_filters:
      - "cpu"
      - "mem"
      - "template_locality"
      - "realtime_create_num"

这里 CPU 设为 2,表示调度器配置的 CPU 超分比例为 2 倍,不保证最终实际超分恰好是 2 倍

3. 修改前还需要确认 Cubelet 的配额行为

目前的配置是:

复制代码
host:
  quota:
    mcpu_limit: 0
    mem_limit: ""
    mvm_limit: 0
    creation_concurrent_num: 0

公开问题 #1565 提到,Cubelet 在未显式指定 CPU、内存配额时可能采用额外的默认超分系数,再与 CubeMaster 比例叠加;还存在高并发创建时调度器读取过期资源快照的问题。该问题描述的是上游 master 分支,不能直接认定你的 v0.7.1 安装一定存在完全相同的行为

因此,我建议先不要直接改动 mcpu_limit,也不要立即重启正在运行的服务

请再执行:

复制代码
# 查看 Cubelet 的服务启动参数
systemctl list-units --type=service | grep -i cubelet

# 检查配置解析器及相关默认值
grep -RniE \
  'overcommit_ratio|cpu_ratio|mem_ratio|mcpu_limit|resolveHostQuotaCPU|resolveHostQuotaMem' \
  /usr/local/services/cubetoolbox/CubeMaster \
  /usr/local/services/cubetoolbox/Cubelet \
  2>/dev/null | head -150

如果源码或配置中没有包含可读的实现(例如只有二进制文件),输出结果即可

最终目标是明确两件事: 一是如何把 CubeMaster 的 CPU 比例设置为你想要的值;二是如何避免 Cubelet 默认系数叠加,造成实际超分过高。配置完成后,还应通过小批量沙盒创建测试验证实际可调度容量,并监控主机 CPU、内存及 OOM 事件

root@VM-0-44-tencentos \~# systemctl list-units --type=service | grep -i cubelet cube-sandbox-cubelet.service loaded active running Cube Sandbox Cubelet root@VM-0-44-tencentos \~# grep -RniE \ 'overcommit_ratio|cpu_ratio|mem_ratio|mcpu_limit|resolveHostQuotaCPU|resolveHostQuotaMem' \ /usr/local/services/cubetoolbox/CubeMaster \ /usr/local/services/cubetoolbox/Cubelet \ 2>/dev/null | head -150 /usr/local/services/cubetoolbox/Cubelet/dynamicconf/conf.yaml:27: mcpu_limit: 0

这次结果说明,当前安装目录下的 CubeMaster、Cubelet 配置文件没有暴露 overcommit_ratio 相关字段,而且 mcpu_limit: 0 是目前能确认的唯一主机 CPU 配额配置

这不代表 v0.7.1 不支持 CPU 超分:相关参数可能由程序内置默认值处理,也可能使用其他配置入口。由于当前安装包中没有找到相关源码,暂时不建议直接添加未经验证的 YAML 字段

下一步:确认 CubeMaster 的真实配置格式

先查看 CubeMaster 启动脚本、配置文件结构和版本帮助:

复制代码
# 1. 查看启动脚本
sed -n '1,220p' \
  /usr/local/services/cubetoolbox/scripts/systemd/cubemaster-start.sh

# 2. 查看完整配置文件的一级结构
grep -nE '^[[:alnum:]_-]+:' \
  /usr/local/services/cubetoolbox/CubeMaster/conf.yaml

# 3. 查看 Cubelet 启动参数
systemctl cat cube-sandbox-cubelet.service

# 4. 查看命令行帮助
/usr/local/services/cubetoolbox/CubeMaster/bin/cubemaster --help 2>&1 | head -100

这些操作均为只读,不会影响正在运行的沙盒

3. 已核实 v0.7.1 的真实实现

进一步检查了官方 v0.7.1 标签的源码,发现一个重要区别:从 v0.7.1 开始,CubeMaster 的 scheduler.overcommit_ratio 和 overcommit_ratio_conf 已被弃用,配置会被解析,但运行时忽略。 因此,前面给出的 CubeMaster overcommit_ratio 配置示例不适用于当前版本,请不要按该示例修改

官方 Cubelet 源码还明确说明:

  • host.quota.mcpu_limit 大于 0 时,使用配置的 CPU 配额

  • mcpu_limit: 0 时,默认 CPU 配额按主机逻辑 CPU 数量的 2 倍计算

  • mem_limit 为空时,默认内存配额按主机总内存的 1.25 倍计算

所以,当前配置的 CPU 默认超分倍数是 2 倍,而不是通过 CubeMaster 的 overcommit_ratio 设置的

4. 如何设置指定的 CPU 超分比例

v0.7.1 应通过 Cubelet 的 host.quota.mcpu_limit 设置绝对 CPU 配额,单位是 millicpu:1000 等于 1 个 CPU

例如,假设主机有 32 个逻辑 CPU:

目标超分比例 mcpu_limit
1 倍 32000
1.5 倍 48000
2 倍 64000
3 倍 96000

先执行以下命令,确认主机逻辑 CPU 数量:

复制代码
nproc
lscpu | grep -E '^(CPU\(s\)|Core\(s\) per socket|Socket\(s\)|Thread\(s\) per core):'

假设 nproc 返回 32,如果目标是 1.5 倍超分,Cubelet 配置可设为:

复制代码
host:
  quota:
    mcpu_limit: 48000

注意:这是配置示例,暂时不要直接覆盖现有 host 配置块,因为还需要保留其中的 scheduler_label、mem_limit、mvm_limit 等其他设置

另外,设置更高的 CPU 配额只是允许调度器分配更多虚拟 CPU,不代表主机拥有更多物理算力;实际负载过高仍会产生 CPU 争用

请先把 nproc 的输出发给我,并告诉我希望设置为 1.5 倍、2 倍还是 3 倍。我可以据此给出精确的 mcpu_limit 数值和保留原配置的修改命令

root@VM-0-44-tencentos \~# sed -n '1,220p' \ /usr/local/services/cubetoolbox/scripts/systemd/cubemaster-start.sh #!/usr/bin/env bash # SPDX-License-Identifier: Apache-2.0 # Copyright (C) 2026 Tencent. All rights reserved. set -euo pipefail SCRIPT_DIR="(cd "(dirname "{BASH_SOURCE\[0\]}")" \&\& pwd)" # shellcheck source=./common.sh source "{SCRIPT_DIR}/common.sh" require_root ensure_systemd_runtime_dirs CUBEMASTER_BIN="{TOOLBOX_ROOT}/CubeMaster/bin/cubemaster" CUBEMASTER_CFG="{TOOLBOX_ROOT}/CubeMaster/conf.yaml" CUBEMASTER_ROOTFS_ARTIFACT_STORE_DIR_DEFAULT="/data/CubeMaster/storage" CUBEMASTER_ROOTFS_ARTIFACT_STORE_DIR_CONFIGURED="{CUBEMASTER_ROOTFS_ARTIFACT_STORE_DIR:-}" CUBEMASTER_ROOTFS_ARTIFACT_STORE_DIR="{CUBEMASTER_ROOTFS_ARTIFACT_STORE_DIR_CONFIGURED:-{CUBEMASTER_ROOTFS_ARTIFACT_STORE_DIR_DEFAULT}}" ensure_executable "{CUBEMASTER_BIN}" ensure_file "{CUBEMASTER_CFG}" export CUBE_MASTER_CONFIG_PATH="{CUBEMASTER_CFG}" if mkdir -p "{CUBEMASTER_ROOTFS_ARTIFACT_STORE_DIR}" \>/dev/null 2\>\&1; then export CUBEMASTER_ROOTFS_ARTIFACT_STORE_DIR="{CUBEMASTER_ROOTFS_ARTIFACT_STORE_DIR}" else log "cubemaster artifact store {CUBEMASTER_ROOTFS_ARTIFACT_STORE_DIR} unavailable, fallback handled by cubemaster" fi # CubeTemplateCenter's address. Can be set here (ad-hoc override) or in # conf.yaml as template_center_addr (persistent, per-deployment). Env wins # over yaml. The default points at the TC unit the same bundle installs on # this host, so split deployments need only edit this export or the conf.yaml # field. Export unconditionally: the binary reads it on every request, and # having it set keeps the later enable to a single edit. export CUBE_TEMPLATE_CENTER_ADDR="{CUBE_TEMPLATE_CENTER_ADDR:-http://127.0.0.1:8090}" exec "${CUBEMASTER_BIN}" root@VM-0-44-tencentos \~# grep -nE '^\[:alnum:_-]+:' \ /usr/local/services/cubetoolbox/CubeMaster/conf.yaml 1:common: 33:log: 40:cubelet_conf: 58:auth: 61:req_template_conf: 72:instance_db_config: 86:redis: 102:scheduler: 118:volume_plugins: 125:extra_conf: root@VM-0-44-tencentos \~# systemctl cat cube-sandbox-cubelet.service # /etc/systemd/system/cube-sandbox-cubelet.service Unit Description=Cube Sandbox Cubelet After=network-online.target Wants=network-online.target ConditionPathExists=/usr/local/services/cubetoolbox/.one-click.env RequiresMountsFor=/data/cubelet /data/cube-shim /data/snapshot_pack PartOf=cube-sandbox-control.target cube-sandbox-compute.target Service Type=forking ExecStartPre=/usr/bin/bash /usr/local/services/cubetoolbox/scripts/systemd/prepare-compute-role.sh User=root WorkingDirectory=/usr/local/services/cubetoolbox EnvironmentFile=-/usr/local/services/cubetoolbox/.one-click.env PIDFile=/run/cube-sandbox-systemd/cubelet.pid Delegate=yes KillMode=process ExecStart=/usr/bin/bash /usr/local/services/cubetoolbox/scripts/systemd/cubelet-start.sh ExecStop=/usr/bin/bash /usr/local/services/cubetoolbox/scripts/systemd/cubelet-stop.sh Restart=on-failure RestartSec=2s TimeoutStartSec=60s TimeoutStopSec=30s Install WantedBy=cube-sandbox-control.target WantedBy=cube-sandbox-compute.target root@VM-0-44-tencentos \~# /usr/local/services/cubetoolbox/CubeMaster/bin/cubemaster --help 2>&1 | head -100 Usage of /usr/local/services/cubetoolbox/CubeMaster/bin/cubemaster: -v show version -version show version

这些输出进一步确认:CubeMaster 没有提供可见的命令行超分参数,启动脚本默认加载 CubeMaster/conf.yaml。接下来应聚焦 Cubelet 的配额配置,不需要再修改 CubeMaster 的 scheduler

1. 确认 Cubelet 实际加载的配置

执行:

复制代码
sed -n '1,240p' \
  /usr/local/services/cubetoolbox/scripts/systemd/cubelet-start.sh

sed -n '1,240p' \
  /usr/local/services/cubetoolbox/scripts/systemd/prepare-compute-role.sh

grep -nE 'CUBELET|CONFIG|dynamicconf|config.toml' \
  /usr/local/services/cubetoolbox/.one-click.env

这一步是为了确认修改 dynamicconf/conf.yaml 后,Cubelet 是否会读取新值,以及需要怎样重启才能生效

2. 获取 CPU 数量并确定超分比例

复制代码
nproc

假设主机有 32 个逻辑 CPU,按前面核实的 mcpu_limit 配置方式计算:

目标 CPU 配额 mcpu_limit
1 倍 32000
1.5 倍 48000
2 倍 64000
3 倍 96000

计算公式:

\ \\text{mcpu\\_limit}=\\text{逻辑CPU数}\\times1000\\times\\text{目标倍数} \\

注意:这里是调度配额,不是物理 CPU 算力;如果使用容器、虚拟机或 CPU affinity 限制了 Cubelet 可用的 CPU,还要考虑实际可用资源

现在已经确认了 Cubelet 的启动方式:

复制代码
/usr/local/services/cubetoolbox/Cubelet/bin/cubelet \
  --config /usr/local/services/cubetoolbox/Cubelet/config/config.toml \
  --dynamic-conf-path /usr/local/services/cubetoolbox/Cubelet/dynamicconf/conf.yaml

也就是说,Cubelet/dynamicconf/conf.yaml 是实际传给 Cubelet 的动态配置文件,其中的 host.quota.mcpu_limit 是接下来需要重点确认的参数

另外,prepare-compute-role.sh 会在启动时处理部分配置,但从你提供的脚本看,它没有主动重写 host.quota.mcpu_limit。因此,修改前可以先备份动态配置,避免丢失现有的节点地址、配额和其他参数

现在可以确定配置修改位置

你的启动脚本明确传入了:

复制代码
--dynamic-conf-path \
  /usr/local/services/cubetoolbox/Cubelet/dynamicconf/conf.yaml

而且启动前的 prepare-compute-role.sh 不会主动覆盖 host.quota.mcpu_limit。因此,修改这个文件是合理的配置方向

需要注意:CubeSandbox 上游公开的超分问题指出,Cubelet 默认 CPU 配额系数为 2 倍;但并发创建还可能因资源快照延迟而超额调度

请最后确认两个值:

复制代码
nproc
grep -n -A 8 '^host:' \
  /usr/local/services/cubetoolbox/Cubelet/dynamicconf/conf.yaml

然后告诉我你希望设置的比例,例如 2 倍、3 倍或 4 倍

将按实际逻辑 CPU 数计算 mcpu_limit,并给出保留原配置的修改命令。修改后需要重启 Cubelet 才能验证新配额是否生效;建议安排维护窗口,先确认现有沙盒的运行状态,不要直接在生产高峰期重启

节点 nproc 返回 96,当前 mcpu_limit: 0

按前面讨论的配额计算方式,如果这 96 个 CPU 都是 Cubelet 实际可用的逻辑 CPU,那么目标配额如下:

目标 CPU 超分比例 mcpu_limit
1 倍 96000
1.5 倍 144000
2 倍 192000
3 倍 288000
4 倍 384000

例如,设置为 3 倍时,对应 mcpu_limit: 288000。但要注意,nproc 是当前进程可见的 CPU 数量,不一定等于主机全部逻辑 CPU;还需要确认是否存在 CPU affinity 或容器限制

你的节点有 96 个逻辑 CPU。按当前 Cubelet 的默认配额逻辑,mcpu_limit: 0 对应的默认 CPU 配额是 2 倍,即 192000 millicpu

推荐配置

如果这是用于 AI Agent 沙盒的生产节点,建议先从 2 倍开始;如果沙盒大多空闲、CPU 使用率较低,再考虑 3 倍

目标 mcpu_limit 说明
1 倍 96000 不超分
2 倍 192000 当前默认水平
3 倍 288000 更高并发,CPU 争用风险增加
4 倍 384000 需要更严格的负载监控

例如,若确定要设置 3 倍,先备份并修改配置:

复制代码
CONF=/usr/local/services/cubetoolbox/Cubelet/dynamicconf/conf.yaml

# 备份
cp -a "$CONF" "${CONF}.bak.$(date +%Y%m%d%H%M%S)"

# 仅修改 mcpu_limit: 0 这一项
sed -i '/^host:/,/^[^ ]/s/^\([[:space:]]*mcpu_limit:[[:space:]]*\)0$/\1288000/' "$CONF"

# 检查修改结果
grep -n -A 8 '^host:' "$CONF"

注意:上面的修改命令只适用于已经决定设置 3 倍的情况,暂时不要执行。 还需要确认 Cubelet 是否热加载该字段,以及当前运行中的沙盒是否会受到配额变更影响。不要直接重启生产节点上的 Cubelet,以免影响正在运行的沙盒

以4倍为例

目标是将 CubeSandbox v0.7.1 节点 CPU 配额设置为 4 倍

按 nproc = 96 计算,目标值为:

\ 96 \\times 1000 \\times 4 = 384000 \\

因此,目标配置是:

复制代码
host:
  quota:
    mcpu_limit: 384000

修改文件:

/usr/local/services/cubetoolbox/Cubelet/dynamicconf/conf.yaml

注意:这会设置主机 CPU 配额上限,并不代表主机获得了额外的物理 CPU 算力。上游还报告过并发创建时调度资源快照滞后、导致超额分配的风险,因此设置为 4 倍后仍需监控 CPU 争用和 OOM

1. 备份并修改配置

下面的命令会先备份原文件,再仅修改 host.quota 下的 mcpu_limit,保留其他配置

复制代码
CONF=/usr/local/services/cubetoolbox/Cubelet/dynamicconf/conf.yaml

# 备份
cp -a "$CONF" "${CONF}.bak.$(date +%Y%m%d%H%M%S)"

# 修改指定配置项
python3 - "$CONF" <<'PY'
import re
import sys
from pathlib import Path

path = Path(sys.argv[1])
text = path.read_text()

pattern = r'(?m)^host:\s*\n(?:(?!^[^\s#]).*\n)*'
match = re.search(pattern, text)
if not match:
    raise SystemExit("ERROR: 未找到 host 配置块,未修改文件")

block = match.group(0)
quota_pattern = r'(?m)^([ \t]+mcpu_limit:[ \t]*)\d+[ \t]*$'
updated, count = re.subn(quota_pattern, r'\g<1>384000', block)

if count != 1:
    raise SystemExit(
        f"ERROR: host 配置块内匹配到 {count} 个 mcpu_limit,未修改文件"
    )

path.write_text(text[:match.start()] + updated + text[match.end():])
print(f"已设置 {path}: host.quota.mcpu_limit=384000")
PY

# 检查
sed -n '24,45p' "$CONF"

预期能看到:

复制代码
host:
  scheduler_label: "default-cluster"
  quota:
    mcpu_limit: 384000

2. 让配置生效

先不要立即重启 Cubelet。 当前节点可能运行着沙盒,重启可能影响正在运行的实例

先确认 Cubelet 是否支持动态热加载:

复制代码
journalctl -u cube-sandbox-cubelet.service \
  --since "30 minutes ago" --no-pager | tail -100

如果确认必须重启,建议在维护窗口操作:

复制代码
systemctl restart cube-sandbox-cubelet.service
systemctl status cube-sandbox-cubelet.service --no-pager

重启后再检查日志,确认配置加载成功、节点重新注册且沙盒创建正常。不要仅凭服务显示 active 就认定新配额已经生效

3. 验证 CPU 超分是否合理

配置完成后建议持续观察:

复制代码
# 主机 CPU 使用率
top

# CPU 数量与可见范围
nproc
grep Cpus_allowed_list /proc/$(pgrep -xo cubelet)/status

# OOM 记录
journalctl -k --since "1 hour ago" --no-pager |
  grep -Ei 'oom|out of memory|killed process'

建议先进行小批量沙盒创建测试,再逐步提高并发。4 倍是配额目标,不是安全并发数保证;如果沙盒 CPU 使用率普遍较高,实际负载仍可能超过主机承载能力

相关推荐
且从容.2 小时前
Ubuntu安装flume
linux·ubuntu·flume
承渊政道2 小时前
Linux网络学习【Reactor反应堆模式】
linux·网络·学习·ubuntu·reactor
讯捷蓝达4 小时前
戴尔R720xd服务器卡死在 “Configuring Memory...Done“,换了内存还是卡?问题根本不在内存
运维·服务器
且从容.5 小时前
ubuntu安装flink
linux·ubuntu·flink
数字新视界7 小时前
动环监控信创适配在智能机房管理中的应用价值与未来展望
大数据·服务器·数据库·动环监控系统·动力与环境监控系统
爱和冰阔落7 小时前
【Linux网络】TCP 服务器如何并发处理多个客户端?从 fork、僵尸进程到双重 fork
linux·运维·服务器·网络
程序员沉梦听雨7 小时前
WSL2的安装与使用
linux·wsl
微小的鱼7 小时前
Linux的渲染与显示系统链路
android·java·linux
lisanmengmeng8 小时前
alertmanger 标签 rules 打上分组标签
linux·运维·服务器
Android系统攻城狮8 小时前
Linux FFmpeg深度解析之av_write_frame调用流程与实战(三十三)
linux·运维·服务器·ffmpeg·音视频进阶·ffmpeg音视频