Part 2: 容器运行时与CRI------超深度逐行分析
一、模块定位
1.1 CRI(Container Runtime Interface)业务职责
CRI是Kubernetes容器运行时接口,是Kubelet与底层容器运行时之间的标准通信协议。其核心业务职责包括:
- 容器生命周期管理:容器的创建、启动、停止、删除、查询
- Pod Sandbox管理:Pod沙箱的创建、停止、删除、状态查询
- 镜像管理:镜像的拉取、查询、列表、删除、文件系统信息
- 流式操作:Exec、Attach、PortForward等交互式操作
- 运行时状态查询:版本信息、运行时状态、容器统计
- 容器日志管理:日志读取、日志轮转、日志清理
- 容器GC:死亡容器回收、沙箱回收、日志目录回收
- 安全上下文:Seccomp、AppArmor、SELinux、Capabilities等安全策略应用
- RuntimeClass动态选择:根据Pod Spec的runtimeClassName选择不同运行时
1.2 在Kubelet中的位置
CRI模块位于Kubelet架构的核心层,是Kubelet与容器运行时之间的桥梁:
┌─────────────────────────────────────────────┐
│ Kubelet │
│ ┌─────────┐ ┌──────────┐ ┌────────────┐ │
│ │ PLEG │ │SyncLoop │ │PodWorker │ │
│ └────┬────┘ └─────┬────┘ └──────┬─────┘ │
│ │ │ │ │
│ └─────────────┼──────────────┘ │
│ ▼ │
│ ┌──────────────────────────────────────┐ │
│ │ kubeGenericRuntimeManager │ │
│ │ (kuberuntime包 - CRI适配层) │ │
│ └──────────────┬───────────────────────┘ │
│ │ │
│ ┌──────────────▼───────────────────────┐ │
│ │ instrumentedRuntimeService │ │
│ │ instrumentedImageManagerService │ │
│ └──────────────┬───────────────────────┘ │
│ │ │
│ ┌──────────────▼───────────────────────┐ │
│ │ remoteRuntimeService (gRPC) │ │
│ │ 或 dockershim (内置适配) │ │
│ └──────────────┬───────────────────────┘ │
│ │ │
└─────────────────┼────────────────────────────┘
▼
┌──────────────────────────────┐
│ 容器运行时 (containerd/CRI-O) │
└──────────────────────────────┘
二、模块整体结构
2.1 类结构与接口定义
#mermaid-svg-p8nZdPplqS34ORRB{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-p8nZdPplqS34ORRB .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-p8nZdPplqS34ORRB .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-p8nZdPplqS34ORRB .error-icon{fill:#552222;}#mermaid-svg-p8nZdPplqS34ORRB .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-p8nZdPplqS34ORRB .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-p8nZdPplqS34ORRB .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-p8nZdPplqS34ORRB .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-p8nZdPplqS34ORRB .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-p8nZdPplqS34ORRB .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-p8nZdPplqS34ORRB .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-p8nZdPplqS34ORRB .marker{fill:#333333;stroke:#333333;}#mermaid-svg-p8nZdPplqS34ORRB .marker.cross{stroke:#333333;}#mermaid-svg-p8nZdPplqS34ORRB svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-p8nZdPplqS34ORRB p{margin:0;}#mermaid-svg-p8nZdPplqS34ORRB g.classGroup text{fill:#9370DB;stroke:none;font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:10px;}#mermaid-svg-p8nZdPplqS34ORRB g.classGroup text .title{font-weight:bolder;}#mermaid-svg-p8nZdPplqS34ORRB .cluster-label text{fill:#333;}#mermaid-svg-p8nZdPplqS34ORRB .cluster-label span{color:#333;}#mermaid-svg-p8nZdPplqS34ORRB .cluster-label span p{background-color:transparent;}#mermaid-svg-p8nZdPplqS34ORRB .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-p8nZdPplqS34ORRB .cluster text{fill:#333;}#mermaid-svg-p8nZdPplqS34ORRB .cluster span{color:#333;}#mermaid-svg-p8nZdPplqS34ORRB .nodeLabel,#mermaid-svg-p8nZdPplqS34ORRB .edgeLabel{color:#131300;}#mermaid-svg-p8nZdPplqS34ORRB .edgeLabel .label rect{fill:#ECECFF;}#mermaid-svg-p8nZdPplqS34ORRB .label text{fill:#131300;}#mermaid-svg-p8nZdPplqS34ORRB .labelBkg{background:#ECECFF;}#mermaid-svg-p8nZdPplqS34ORRB .edgeLabel .label span{background:#ECECFF;}#mermaid-svg-p8nZdPplqS34ORRB .classTitle{font-weight:bolder;}#mermaid-svg-p8nZdPplqS34ORRB .node rect,#mermaid-svg-p8nZdPplqS34ORRB .node circle,#mermaid-svg-p8nZdPplqS34ORRB .node ellipse,#mermaid-svg-p8nZdPplqS34ORRB .node polygon,#mermaid-svg-p8nZdPplqS34ORRB .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-p8nZdPplqS34ORRB .divider{stroke:#9370DB;stroke-width:1;}#mermaid-svg-p8nZdPplqS34ORRB g.clickable{cursor:pointer;}#mermaid-svg-p8nZdPplqS34ORRB g.classGroup rect{fill:#ECECFF;stroke:#9370DB;}#mermaid-svg-p8nZdPplqS34ORRB g.classGroup line{stroke:#9370DB;stroke-width:1;}#mermaid-svg-p8nZdPplqS34ORRB .classLabel .box{stroke:none;stroke-width:0;fill:#ECECFF;opacity:0.5;}#mermaid-svg-p8nZdPplqS34ORRB .classLabel .label{fill:#9370DB;font-size:10px;}#mermaid-svg-p8nZdPplqS34ORRB .relation{stroke:#333333;stroke-width:1;fill:none;}#mermaid-svg-p8nZdPplqS34ORRB .dashed-line{stroke-dasharray:3;}#mermaid-svg-p8nZdPplqS34ORRB .dotted-line{stroke-dasharray:1 2;}#mermaid-svg-p8nZdPplqS34ORRB #compositionStart,#mermaid-svg-p8nZdPplqS34ORRB .composition{fill:#333333!important;stroke:#333333!important;stroke-width:1;}#mermaid-svg-p8nZdPplqS34ORRB #compositionEnd,#mermaid-svg-p8nZdPplqS34ORRB .composition{fill:#333333!important;stroke:#333333!important;stroke-width:1;}#mermaid-svg-p8nZdPplqS34ORRB #dependencyStart,#mermaid-svg-p8nZdPplqS34ORRB .dependency{fill:#333333!important;stroke:#333333!important;stroke-width:1;}#mermaid-svg-p8nZdPplqS34ORRB #dependencyStart,#mermaid-svg-p8nZdPplqS34ORRB .dependency{fill:#333333!important;stroke:#333333!important;stroke-width:1;}#mermaid-svg-p8nZdPplqS34ORRB #extensionStart,#mermaid-svg-p8nZdPplqS34ORRB .extension{fill:transparent!important;stroke:#333333!important;stroke-width:1;}#mermaid-svg-p8nZdPplqS34ORRB #extensionEnd,#mermaid-svg-p8nZdPplqS34ORRB .extension{fill:transparent!important;stroke:#333333!important;stroke-width:1;}#mermaid-svg-p8nZdPplqS34ORRB #aggregationStart,#mermaid-svg-p8nZdPplqS34ORRB .aggregation{fill:transparent!important;stroke:#333333!important;stroke-width:1;}#mermaid-svg-p8nZdPplqS34ORRB #aggregationEnd,#mermaid-svg-p8nZdPplqS34ORRB .aggregation{fill:transparent!important;stroke:#333333!important;stroke-width:1;}#mermaid-svg-p8nZdPplqS34ORRB #lollipopStart,#mermaid-svg-p8nZdPplqS34ORRB .lollipop{fill:#ECECFF!important;stroke:#333333!important;stroke-width:1;}#mermaid-svg-p8nZdPplqS34ORRB #lollipopEnd,#mermaid-svg-p8nZdPplqS34ORRB .lollipop{fill:#ECECFF!important;stroke:#333333!important;stroke-width:1;}#mermaid-svg-p8nZdPplqS34ORRB .edgeTerminals{font-size:11px;line-height:initial;}#mermaid-svg-p8nZdPplqS34ORRB .classTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-p8nZdPplqS34ORRB .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-p8nZdPplqS34ORRB .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-p8nZdPplqS34ORRB :root{--mermaid-font-family:"trebuchet ms",verdana,arial,sans-serif;} <<interface>>
KubeGenericRuntime
+Runtime
+StreamingRuntime
+CommandRunner
kubeGenericRuntimeManager
-runtimeName: string
-recorder: EventRecorder
-osInterface: OSInterface
-machineInfo: MachineInfo
-containerGC: containerGC
-keyring: DockerKeyring
-runner: HandlerRunner
-runtimeHelper: RuntimeHelper
-livenessManager: Manager
-readinessManager: Manager
-startupManager: Manager
-cpuCFSQuota: bool
-cpuCFSQuotaPeriod: Duration
-imagePuller: ImageManager
-runtimeService: RuntimeService
-imageService: ImageManagerService
-versionCache: ObjectCache
-seccompProfileRoot: string
-internalLifecycle: InternalContainerLifecycle
-legacyLogProvider: LegacyLogProvider
-logManager: ContainerLogManager
-runtimeClassManager: Manager
-logReduction: LogReduction
-podStateProvider: podStateProvider
+Type() : string
+Version() : Version
+APIVersion() : Version
+Status() : RuntimeStatus
+GetPods(all) : Pod\[\]
+SyncPod(pod, podStatus, pullSecrets, backOff) : PodSyncResult
+KillPod(pod, runningPod, gracePeriodOverride) : error
+GetPodStatus(uid, name, namespace) : PodStatus
+GetContainerLogs(ctx, pod, containerID, logOptions, stdout, stderr) : error
+DeleteContainer(containerID) : error
+GarbageCollect(gcPolicy, allSourcesReady, evictNonDeletedPods) : error
+UpdatePodCIDR(podCIDR) : error
+GetExec(id, cmd, stdin, stdout, stderr, tty) : URL
+GetAttach(id, stdin, stdout, stderr, tty) : URL
+GetPortForward(podName, podNamespace, podUID, ports) : URL
+RunInContainer(id, cmd, timeout) : byte\[\]
+PullImage(image, pullSecrets, podSandboxConfig) : string
+GetImageRef(image) : string
+ListImages() : Image\[\]
+RemoveImage(image) : error
+ImageStats() : ImageStats
+SupportsSingleFileMapping() : bool
<<interface>>
RuntimeService
+Version(apiVersion) : VersionResponse
+CreateContainer(podSandboxID, config, sandboxConfig) : string
+StartContainer(containerID) : error
+StopContainer(containerID, timeout) : error
+RemoveContainer(containerID) : error
+ListContainers(filter) : Container\[\]
+ContainerStatus(containerID) : ContainerStatus
+UpdateContainerResources(containerID, resources) : error
+ReopenContainerLog(containerID) : error
+ExecSync(containerID, cmd, timeout) : byte\[\]
+Exec(req) : ExecResponse
+Attach(req) : AttachResponse
+RunPodSandbox(config, runtimeHandler) : string
+StopPodSandbox(podSandboxID) : error
+RemovePodSandbox(podSandboxID) : error
+PodSandboxStatus(podSandboxID) : PodSandboxStatus
+ListPodSandbox(filter) : PodSandbox\[\]
+ContainerStats(containerID) : ContainerStats
+ListContainerStats(filter) : ContainerStats\[\]
+PortForward(req) : PortForwardResponse
+UpdateRuntimeConfig(runtimeConfig) : error
+Status() : RuntimeStatus
<<interface>>
ImageManagerService
+ListImages(filter) : Image\[\]
+ImageStatus(image) : Image
+PullImage(image, auth, podSandboxConfig) : string
+RemoveImage(image) : error
+ImageFsInfo() : FilesystemUsage\[\]
instrumentedRuntimeService
-service: RuntimeService
instrumentedImageManagerService
-service: ImageManagerService
containerGC
-client: RuntimeService
-manager: kubeGenericRuntimeManager
-podStateProvider: podStateProvider
+GarbageCollect(gcPolicy, allSourcesReady, evictTerminatedPods) : error
remoteRuntimeService
-timeout: Duration
-runtimeClient: RuntimeServiceClient
-logReduction: LogReduction
2.2 CRI接口层次图
#mermaid-svg-ZWAe6mjRJS7Dtwuh{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-ZWAe6mjRJS7Dtwuh .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-ZWAe6mjRJS7Dtwuh .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-ZWAe6mjRJS7Dtwuh .error-icon{fill:#552222;}#mermaid-svg-ZWAe6mjRJS7Dtwuh .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-ZWAe6mjRJS7Dtwuh .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-ZWAe6mjRJS7Dtwuh .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-ZWAe6mjRJS7Dtwuh .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-ZWAe6mjRJS7Dtwuh .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-ZWAe6mjRJS7Dtwuh .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-ZWAe6mjRJS7Dtwuh .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-ZWAe6mjRJS7Dtwuh .marker{fill:#333333;stroke:#333333;}#mermaid-svg-ZWAe6mjRJS7Dtwuh .marker.cross{stroke:#333333;}#mermaid-svg-ZWAe6mjRJS7Dtwuh svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-ZWAe6mjRJS7Dtwuh p{margin:0;}#mermaid-svg-ZWAe6mjRJS7Dtwuh .label{font-family:"trebuchet ms",verdana,arial,sans-serif;color:#333;}#mermaid-svg-ZWAe6mjRJS7Dtwuh .cluster-label text{fill:#333;}#mermaid-svg-ZWAe6mjRJS7Dtwuh .cluster-label span{color:#333;}#mermaid-svg-ZWAe6mjRJS7Dtwuh .cluster-label span p{background-color:transparent;}#mermaid-svg-ZWAe6mjRJS7Dtwuh .label text,#mermaid-svg-ZWAe6mjRJS7Dtwuh span{fill:#333;color:#333;}#mermaid-svg-ZWAe6mjRJS7Dtwuh .node rect,#mermaid-svg-ZWAe6mjRJS7Dtwuh .node circle,#mermaid-svg-ZWAe6mjRJS7Dtwuh .node ellipse,#mermaid-svg-ZWAe6mjRJS7Dtwuh .node polygon,#mermaid-svg-ZWAe6mjRJS7Dtwuh .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-ZWAe6mjRJS7Dtwuh .rough-node .label text,#mermaid-svg-ZWAe6mjRJS7Dtwuh .node .label text,#mermaid-svg-ZWAe6mjRJS7Dtwuh .image-shape .label,#mermaid-svg-ZWAe6mjRJS7Dtwuh .icon-shape .label{text-anchor:middle;}#mermaid-svg-ZWAe6mjRJS7Dtwuh .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-ZWAe6mjRJS7Dtwuh .rough-node .label,#mermaid-svg-ZWAe6mjRJS7Dtwuh .node .label,#mermaid-svg-ZWAe6mjRJS7Dtwuh .image-shape .label,#mermaid-svg-ZWAe6mjRJS7Dtwuh .icon-shape .label{text-align:center;}#mermaid-svg-ZWAe6mjRJS7Dtwuh .node.clickable{cursor:pointer;}#mermaid-svg-ZWAe6mjRJS7Dtwuh .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-ZWAe6mjRJS7Dtwuh .arrowheadPath{fill:#333333;}#mermaid-svg-ZWAe6mjRJS7Dtwuh .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-ZWAe6mjRJS7Dtwuh .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-ZWAe6mjRJS7Dtwuh .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-ZWAe6mjRJS7Dtwuh .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-ZWAe6mjRJS7Dtwuh .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-ZWAe6mjRJS7Dtwuh .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-ZWAe6mjRJS7Dtwuh .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-ZWAe6mjRJS7Dtwuh .cluster text{fill:#333;}#mermaid-svg-ZWAe6mjRJS7Dtwuh .cluster span{color:#333;}#mermaid-svg-ZWAe6mjRJS7Dtwuh div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-ZWAe6mjRJS7Dtwuh .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-ZWAe6mjRJS7Dtwuh rect.text{fill:none;stroke-width:0;}#mermaid-svg-ZWAe6mjRJS7Dtwuh .icon-shape,#mermaid-svg-ZWAe6mjRJS7Dtwuh .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-ZWAe6mjRJS7Dtwuh .icon-shape p,#mermaid-svg-ZWAe6mjRJS7Dtwuh .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-ZWAe6mjRJS7Dtwuh .icon-shape .label rect,#mermaid-svg-ZWAe6mjRJS7Dtwuh .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-ZWAe6mjRJS7Dtwuh .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-ZWAe6mjRJS7Dtwuh .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-ZWAe6mjRJS7Dtwuh :root{--mermaid-font-family:"trebuchet ms",verdana,arial,sans-serif;} 辅助模块
传输实现层
度量封装层
CRI抽象接口(internalapi)
CRI核心层
Kubelet内部接口层
Kubelet上层调用者
PodWorker/SyncLoop
PLEG
Kubelet Server
kubecontainer.Runtime
kubecontainer.StreamingRuntime
kubecontainer.ImageService
kubecontainer.CommandRunner
kubeGenericRuntimeManager
podActions/computePodActions
startContainer/killContainer
createPodSandbox/generatePodSandboxConfig
RuntimeService
ImageManagerService
RuntimeVersioner
ContainerManager
PodSandboxManager
ContainerStatsManager
instrumentedRuntimeService
instrumentedImageManagerService
remoteRuntimeService - gRPC
remoteImageService - gRPC
dockershim - Docker适配
images.ImageManager
logs.ContainerLogManager
runtimeclass.Manager
containerGC
2.3 核心方法清单
| 所属结构 | 方法 | 功能 |
|---|---|---|
kubeGenericRuntimeManager |
SyncPod |
Pod同步核心方法,7步完成Pod的创建/更新 |
kubeGenericRuntimeManager |
computePodActions |
计算Pod变更动作(kill/start/sandbox) |
kubeGenericRuntimeManager |
startContainer |
容器启动4步:拉镜像→创建→启动→postStart |
kubeGenericRuntimeManager |
killContainer |
容器终止:preStop hook→StopContainer |
kubeGenericRuntimeManager |
killPodWithSyncResult |
并发kill所有容器+stop沙箱 |
kubeGenericRuntimeManager |
createPodSandbox |
创建Pod沙箱 |
kubeGenericRuntimeManager |
generatePodSandboxConfig |
生成Pod沙箱配置 |
kubeGenericRuntimeManager |
generateContainerConfig |
生成容器配置 |
kubeGenericRuntimeManager |
PullImage |
镜像拉取(含凭证链) |
kubeGenericRuntimeManager |
GetPodStatus |
获取Pod状态 |
kubeGenericRuntimeManager |
doBackOff |
容器退避检查 |
kubeGenericRuntimeManager |
podSandboxChanged |
沙箱变更检测 |
containerGC |
GarbageCollect |
垃圾回收三步:容器→沙箱→日志 |
imageManager |
EnsureImageExists |
确保镜像存在(拉取策略) |
runtimeclass.Manager |
LookupRuntimeHandler |
RuntimeClass查找 |
containerLogManager |
Start/Clean |
日志轮转与清理 |
2.4 内部调用关系
渲染错误: Mermaid 渲染失败: Parse error on line 7: ...rToRun] CPA -> SR[shouldRestartO ----------------------^ Expecting 'SEMI', 'NEWLINE', 'EOF', 'AMP', 'START_LINK', 'LINK', 'LINK_ID', got 'MINUS'
2.5 数据流入流出方式
数据流入:
SyncPod接收:v1.Pod、kubecontainer.PodStatus、[]v1.Secret(pullSecrets)、flowcontrol.Backoff- CRI gRPC响应:
runtimeapi.VersionResponse、runtimeapi.ContainerStatus、runtimeapi.PodSandboxStatus等 - 探测结果:
proberesults.Manager提供liveness/readiness/startup探测结果
数据流出:
SyncPod输出:kubecontainer.PodSyncResult- CRI gRPC请求:
runtimeapi.RunPodSandboxRequest、runtimeapi.CreateContainerRequest等 - 事件记录:
recorder.Event输出到Kubernetes事件系统 - 日志输出:通过
io.Writer输出到kubectl log请求
三、核心业务逻辑深度解析
3.1 CRI完整调用链路
logManager runtimeclass.Manager imageManager 容器运行时(containerd/CRI-O) remoteRuntimeService(gRPC) instrumentedRuntimeService computePodActions kubeGenericRuntimeManager PodWorker logManager runtimeclass.Manager imageManager 容器运行时(containerd/CRI-O) remoteRuntimeService(gRPC) instrumentedRuntimeService computePodActions kubeGenericRuntimeManager PodWorker #mermaid-svg-6HO1E8LKdmHERPaK{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-6HO1E8LKdmHERPaK .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-6HO1E8LKdmHERPaK .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-6HO1E8LKdmHERPaK .error-icon{fill:#552222;}#mermaid-svg-6HO1E8LKdmHERPaK .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-6HO1E8LKdmHERPaK .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-6HO1E8LKdmHERPaK .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-6HO1E8LKdmHERPaK .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-6HO1E8LKdmHERPaK .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-6HO1E8LKdmHERPaK .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-6HO1E8LKdmHERPaK .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-6HO1E8LKdmHERPaK .marker{fill:#333333;stroke:#333333;}#mermaid-svg-6HO1E8LKdmHERPaK .marker.cross{stroke:#333333;}#mermaid-svg-6HO1E8LKdmHERPaK svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-6HO1E8LKdmHERPaK p{margin:0;}#mermaid-svg-6HO1E8LKdmHERPaK .actor{stroke:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);fill:#ECECFF;}#mermaid-svg-6HO1E8LKdmHERPaK text.actor>tspan{fill:black;stroke:none;}#mermaid-svg-6HO1E8LKdmHERPaK .actor-line{stroke:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);}#mermaid-svg-6HO1E8LKdmHERPaK .innerArc{stroke-width:1.5;stroke-dasharray:none;}#mermaid-svg-6HO1E8LKdmHERPaK .messageLine0{stroke-width:1.5;stroke-dasharray:none;stroke:#333;}#mermaid-svg-6HO1E8LKdmHERPaK .messageLine1{stroke-width:1.5;stroke-dasharray:2,2;stroke:#333;}#mermaid-svg-6HO1E8LKdmHERPaK #arrowhead path{fill:#333;stroke:#333;}#mermaid-svg-6HO1E8LKdmHERPaK .sequenceNumber{fill:white;}#mermaid-svg-6HO1E8LKdmHERPaK #sequencenumber{fill:#333;}#mermaid-svg-6HO1E8LKdmHERPaK #crosshead path{fill:#333;stroke:#333;}#mermaid-svg-6HO1E8LKdmHERPaK .messageText{fill:#333;stroke:none;}#mermaid-svg-6HO1E8LKdmHERPaK .labelBox{stroke:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);fill:#ECECFF;}#mermaid-svg-6HO1E8LKdmHERPaK .labelText,#mermaid-svg-6HO1E8LKdmHERPaK .labelText>tspan{fill:black;stroke:none;}#mermaid-svg-6HO1E8LKdmHERPaK .loopText,#mermaid-svg-6HO1E8LKdmHERPaK .loopText>tspan{fill:black;stroke:none;}#mermaid-svg-6HO1E8LKdmHERPaK .loopLine{stroke-width:2px;stroke-dasharray:2,2;stroke:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);fill:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);}#mermaid-svg-6HO1E8LKdmHERPaK .note{stroke:#aaaa33;fill:#fff5ad;}#mermaid-svg-6HO1E8LKdmHERPaK .noteText,#mermaid-svg-6HO1E8LKdmHERPaK .noteText>tspan{fill:black;stroke:none;}#mermaid-svg-6HO1E8LKdmHERPaK .activation0{fill:#f4f4f4;stroke:#666;}#mermaid-svg-6HO1E8LKdmHERPaK .activation1{fill:#f4f4f4;stroke:#666;}#mermaid-svg-6HO1E8LKdmHERPaK .activation2{fill:#f4f4f4;stroke:#666;}#mermaid-svg-6HO1E8LKdmHERPaK .actorPopupMenu{position:absolute;}#mermaid-svg-6HO1E8LKdmHERPaK .actorPopupMenuPanel{position:absolute;fill:#ECECFF;box-shadow:0px 8px 16px 0px rgba(0,0,0,0.2);filter:drop-shadow(3px 5px 2px rgb(0 0 0 / 0.4));}#mermaid-svg-6HO1E8LKdmHERPaK .actor-man line{stroke:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);fill:#ECECFF;}#mermaid-svg-6HO1E8LKdmHERPaK .actor-man circle,#mermaid-svg-6HO1E8LKdmHERPaK line{stroke:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);fill:#ECECFF;stroke-width:2px;}#mermaid-svg-6HO1E8LKdmHERPaK :root{--mermaid-font-family:"trebuchet ms",verdana,arial,sans-serif;} Step1: 计算Pod变更 Step2: 杀掉Pod Step3: Kill指定容器 altKillPod=true个别容器需要kill Step4: 创建Sandbox altCreateSandbox=true Step5-7: 启动容器 loop每个待启动容器 SyncPod(pod, podStatus, pullSecrets, backOff)computePodActions(pod, podStatus)podActions{KillPod,CreateSandbox,ContainersToStart,ContainersToKill}killPodWithSyncResult(pod, runningPod)killContainersWithSyncResult(并发kill所有容器)StopPodSandbox(sandboxID)gRPC StopPodSandboxStopPodSandboxkillContainer(pod, containerID, name, message, reason)LookupRuntimeHandler(pod.Spec.RuntimeClassName)runtimeHandlerRunPodSandbox(config, runtimeHandler)gRPC RunPodSandboxRunPodSandbox(创建网络命名空间+CNI)podSandboxIDpodSandboxIDpodSandboxIDPodSandboxStatus(podSandboxID)gRPC PodSandboxStatusdeterminePodSandboxIPs(获取Pod IP)EnsureImageExists(pod, container, pullSecrets)ImageStatus / PullImageimageRefgenerateContainerConfig(构建CRI ContainerConfig)CreateContainer(podSandboxID, containerConfig, sandboxConfig)gRPC CreateContainerCreateContainercontainerIDStartContainer(containerID)gRPC StartContainerStartContainernil(成功)记录日志symlinkPodSyncResult
3.2 容器创建/启动逐行解析------startContainer方法
startContainer是容器启动的核心4步流程,位于kuberuntime_container.go:
go
func (m *kubeGenericRuntimeManager) startContainer(
podSandboxID string, // Pod沙箱ID
podSandboxConfig *runtimeapi.PodSandboxConfig, // 沙箱配置
spec *startSpec, // 启动规格(普通/init/ephemeral容器)
pod *v1.Pod, // Pod对象
podStatus *kubecontainer.PodStatus, // Pod当前状态
pullSecrets []v1.Secret, // 镜像拉取凭证
podIP string, // Pod IP(主IP)
podIPs []string, // Pod所有IP(双栈支持)
) (string, error) {
Step 1: 拉取镜像
go
imageRef, msg, err := m.imagePuller.EnsureImageExists(pod, container, pullSecrets, podSandboxConfig)
- 调用
imageManager.EnsureImageExists,根据ImagePullPolicy决定是否拉取 - 拉取失败则记录事件并返回
Step 2: 创建容器
go
restartCount := 0
containerStatus := podStatus.FindContainerStatusByName(container.Name)
if containerStatus != nil {
restartCount = containerStatus.RestartCount + 1 // 递增重启计数
}
target, err := spec.getTargetID(podStatus) // ephemeral container的namespace target
containerConfig, cleanupAction, err := m.generateContainerConfig(...) // 构建CRI配置
err = m.internalLifecycle.PreCreateContainer(pod, container, containerConfig) // 内部生命周期钩子
containerID, err := m.runtimeService.CreateContainer(podSandboxID, containerConfig, podSandboxConfig) // CRI调用
generateContainerConfig构建完整的CRIContainerConfig,包含metadata、mounts、envs、security context等PreCreateContainer是device plugin等内部钩子
Step 3: 启动容器
go
err = m.internalLifecycle.PreStartContainer(pod, container, containerID) // 内部钩子
err = m.runtimeService.StartContainer(containerID) // CRI调用启动
// 创建legacy日志symlink(向后兼容)
Step 4: PostStart Hook
go
if container.Lifecycle != nil && container.Lifecycle.PostStart != nil {
msg, handlerErr := m.runner.Run(kubeContainerID, pod, container, container.Lifecycle.PostStart)
if handlerErr != nil {
// PostStart失败 → kill容器
m.killContainer(pod, kubeContainerID, container.Name, "FailedPostStartHook", ...)
return msg, fmt.Errorf("%s: %v", ErrPostStartHook, handlerErr)
}
}
3.3 Pod创建CRI调用时序图
ImageManagerService(CRI) RuntimeService(CRI) kubeGenericRuntimeManager ImageManagerService(CRI) RuntimeService(CRI) kubeGenericRuntimeManager #mermaid-svg-Wg6MjzjDRcBhhlen{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-Wg6MjzjDRcBhhlen .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-Wg6MjzjDRcBhhlen .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-Wg6MjzjDRcBhhlen .error-icon{fill:#552222;}#mermaid-svg-Wg6MjzjDRcBhhlen .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-Wg6MjzjDRcBhhlen .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-Wg6MjzjDRcBhhlen .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-Wg6MjzjDRcBhhlen .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-Wg6MjzjDRcBhhlen .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-Wg6MjzjDRcBhhlen .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-Wg6MjzjDRcBhhlen .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-Wg6MjzjDRcBhhlen .marker{fill:#333333;stroke:#333333;}#mermaid-svg-Wg6MjzjDRcBhhlen .marker.cross{stroke:#333333;}#mermaid-svg-Wg6MjzjDRcBhhlen svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-Wg6MjzjDRcBhhlen p{margin:0;}#mermaid-svg-Wg6MjzjDRcBhhlen .actor{stroke:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);fill:#ECECFF;}#mermaid-svg-Wg6MjzjDRcBhhlen text.actor>tspan{fill:black;stroke:none;}#mermaid-svg-Wg6MjzjDRcBhhlen .actor-line{stroke:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);}#mermaid-svg-Wg6MjzjDRcBhhlen .innerArc{stroke-width:1.5;stroke-dasharray:none;}#mermaid-svg-Wg6MjzjDRcBhhlen .messageLine0{stroke-width:1.5;stroke-dasharray:none;stroke:#333;}#mermaid-svg-Wg6MjzjDRcBhhlen .messageLine1{stroke-width:1.5;stroke-dasharray:2,2;stroke:#333;}#mermaid-svg-Wg6MjzjDRcBhhlen #arrowhead path{fill:#333;stroke:#333;}#mermaid-svg-Wg6MjzjDRcBhhlen .sequenceNumber{fill:white;}#mermaid-svg-Wg6MjzjDRcBhhlen #sequencenumber{fill:#333;}#mermaid-svg-Wg6MjzjDRcBhhlen #crosshead path{fill:#333;stroke:#333;}#mermaid-svg-Wg6MjzjDRcBhhlen .messageText{fill:#333;stroke:none;}#mermaid-svg-Wg6MjzjDRcBhhlen .labelBox{stroke:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);fill:#ECECFF;}#mermaid-svg-Wg6MjzjDRcBhhlen .labelText,#mermaid-svg-Wg6MjzjDRcBhhlen .labelText>tspan{fill:black;stroke:none;}#mermaid-svg-Wg6MjzjDRcBhhlen .loopText,#mermaid-svg-Wg6MjzjDRcBhhlen .loopText>tspan{fill:black;stroke:none;}#mermaid-svg-Wg6MjzjDRcBhhlen .loopLine{stroke-width:2px;stroke-dasharray:2,2;stroke:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);fill:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);}#mermaid-svg-Wg6MjzjDRcBhhlen .note{stroke:#aaaa33;fill:#fff5ad;}#mermaid-svg-Wg6MjzjDRcBhhlen .noteText,#mermaid-svg-Wg6MjzjDRcBhhlen .noteText>tspan{fill:black;stroke:none;}#mermaid-svg-Wg6MjzjDRcBhhlen .activation0{fill:#f4f4f4;stroke:#666;}#mermaid-svg-Wg6MjzjDRcBhhlen .activation1{fill:#f4f4f4;stroke:#666;}#mermaid-svg-Wg6MjzjDRcBhhlen .activation2{fill:#f4f4f4;stroke:#666;}#mermaid-svg-Wg6MjzjDRcBhhlen .actorPopupMenu{position:absolute;}#mermaid-svg-Wg6MjzjDRcBhhlen .actorPopupMenuPanel{position:absolute;fill:#ECECFF;box-shadow:0px 8px 16px 0px rgba(0,0,0,0.2);filter:drop-shadow(3px 5px 2px rgb(0 0 0 / 0.4));}#mermaid-svg-Wg6MjzjDRcBhhlen .actor-man line{stroke:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);fill:#ECECFF;}#mermaid-svg-Wg6MjzjDRcBhhlen .actor-man circle,#mermaid-svg-Wg6MjzjDRcBhhlen line{stroke:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);fill:#ECECFF;stroke-width:2px;}#mermaid-svg-Wg6MjzjDRcBhhlen :root{--mermaid-font-family:"trebuchet ms",verdana,arial,sans-serif;} Phase 1: Sandbox创建 创建网络命名空间执行CNI插件设置resolv.conf设置hostname Phase 2: 镜像准备 alt镜像不存在或PullPolicy=Always Phase 3: 容器创建与启动(循环每个容器) 创建容器rootfs设置cgroup应用安全策略 启动容器进程设置namespace执行entrypoint RunPodSandbox(config, runtimeHandler)podSandboxIDPodSandboxStatus(podSandboxID)sandboxStatus(IP/Network信息)ImageStatus(imageSpec)imageStatus / nilPullImage(imageSpec, auth, sandboxConfig)imageRefCreateContainer(sandboxID, containerConfig, sandboxConfig)containerIDStartContainer(containerID)nil(成功)
3.4 容器生命周期状态机
#mermaid-svg-wuW9La0hDF001WNC{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-wuW9La0hDF001WNC .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-wuW9La0hDF001WNC .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-wuW9La0hDF001WNC .error-icon{fill:#552222;}#mermaid-svg-wuW9La0hDF001WNC .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-wuW9La0hDF001WNC .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-wuW9La0hDF001WNC .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-wuW9La0hDF001WNC .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-wuW9La0hDF001WNC .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-wuW9La0hDF001WNC .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-wuW9La0hDF001WNC .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-wuW9La0hDF001WNC .marker{fill:#333333;stroke:#333333;}#mermaid-svg-wuW9La0hDF001WNC .marker.cross{stroke:#333333;}#mermaid-svg-wuW9La0hDF001WNC svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-wuW9La0hDF001WNC p{margin:0;}#mermaid-svg-wuW9La0hDF001WNC defs #statediagram-barbEnd{fill:#333333;stroke:#333333;}#mermaid-svg-wuW9La0hDF001WNC g.stateGroup text{fill:#9370DB;stroke:none;font-size:10px;}#mermaid-svg-wuW9La0hDF001WNC g.stateGroup text{fill:#333;stroke:none;font-size:10px;}#mermaid-svg-wuW9La0hDF001WNC g.stateGroup .state-title{font-weight:bolder;fill:#131300;}#mermaid-svg-wuW9La0hDF001WNC g.stateGroup rect{fill:#ECECFF;stroke:#9370DB;}#mermaid-svg-wuW9La0hDF001WNC g.stateGroup line{stroke:#333333;stroke-width:1;}#mermaid-svg-wuW9La0hDF001WNC .transition{stroke:#333333;stroke-width:1;fill:none;}#mermaid-svg-wuW9La0hDF001WNC .stateGroup .composit{fill:white;border-bottom:1px;}#mermaid-svg-wuW9La0hDF001WNC .stateGroup .alt-composit{fill:#e0e0e0;border-bottom:1px;}#mermaid-svg-wuW9La0hDF001WNC .state-note{stroke:#aaaa33;fill:#fff5ad;}#mermaid-svg-wuW9La0hDF001WNC .state-note text{fill:black;stroke:none;font-size:10px;}#mermaid-svg-wuW9La0hDF001WNC .stateLabel .box{stroke:none;stroke-width:0;fill:#ECECFF;opacity:0.5;}#mermaid-svg-wuW9La0hDF001WNC .edgeLabel .label rect{fill:#ECECFF;opacity:0.5;}#mermaid-svg-wuW9La0hDF001WNC .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-wuW9La0hDF001WNC .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-wuW9La0hDF001WNC .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-wuW9La0hDF001WNC .edgeLabel .label text{fill:#333;}#mermaid-svg-wuW9La0hDF001WNC .label div .edgeLabel{color:#333;}#mermaid-svg-wuW9La0hDF001WNC .stateLabel text{fill:#131300;font-size:10px;font-weight:bold;}#mermaid-svg-wuW9La0hDF001WNC .node circle.state-start{fill:#333333;stroke:#333333;}#mermaid-svg-wuW9La0hDF001WNC .node .fork-join{fill:#333333;stroke:#333333;}#mermaid-svg-wuW9La0hDF001WNC .node circle.state-end{fill:#9370DB;stroke:white;stroke-width:1.5;}#mermaid-svg-wuW9La0hDF001WNC .end-state-inner{fill:white;stroke-width:1.5;}#mermaid-svg-wuW9La0hDF001WNC .node rect{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-wuW9La0hDF001WNC .node polygon{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-wuW9La0hDF001WNC #statediagram-barbEnd{fill:#333333;}#mermaid-svg-wuW9La0hDF001WNC .statediagram-cluster rect{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-wuW9La0hDF001WNC .cluster-label,#mermaid-svg-wuW9La0hDF001WNC .nodeLabel{color:#131300;}#mermaid-svg-wuW9La0hDF001WNC .statediagram-cluster rect.outer{rx:5px;ry:5px;}#mermaid-svg-wuW9La0hDF001WNC .statediagram-state .divider{stroke:#9370DB;}#mermaid-svg-wuW9La0hDF001WNC .statediagram-state .title-state{rx:5px;ry:5px;}#mermaid-svg-wuW9La0hDF001WNC .statediagram-cluster.statediagram-cluster .inner{fill:white;}#mermaid-svg-wuW9La0hDF001WNC .statediagram-cluster.statediagram-cluster-alt .inner{fill:#f0f0f0;}#mermaid-svg-wuW9La0hDF001WNC .statediagram-cluster .inner{rx:0;ry:0;}#mermaid-svg-wuW9La0hDF001WNC .statediagram-state rect.basic{rx:5px;ry:5px;}#mermaid-svg-wuW9La0hDF001WNC .statediagram-state rect.divider{stroke-dasharray:10,10;fill:#f0f0f0;}#mermaid-svg-wuW9La0hDF001WNC .note-edge{stroke-dasharray:5;}#mermaid-svg-wuW9La0hDF001WNC .statediagram-note rect{fill:#fff5ad;stroke:#aaaa33;stroke-width:1px;rx:0;ry:0;}#mermaid-svg-wuW9La0hDF001WNC .statediagram-note rect{fill:#fff5ad;stroke:#aaaa33;stroke-width:1px;rx:0;ry:0;}#mermaid-svg-wuW9La0hDF001WNC .statediagram-note text{fill:black;}#mermaid-svg-wuW9La0hDF001WNC .statediagram-note .nodeLabel{color:black;}#mermaid-svg-wuW9La0hDF001WNC .statediagram .edgeLabel{color:red;}#mermaid-svg-wuW9La0hDF001WNC #dependencyStart,#mermaid-svg-wuW9La0hDF001WNC #dependencyEnd{fill:#333333;stroke:#333333;stroke-width:1;}#mermaid-svg-wuW9La0hDF001WNC .statediagramTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-wuW9La0hDF001WNC :root{--mermaid-font-family:"trebuchet ms",verdana,arial,sans-serif;} Running
Killing
CreateContainer(CRI)
容器启动失败
确认已退出
执行PostStart Hook
Hook成功
执行探测
探测通过
执行PreStop Hook
Hook完成
无Hook
StopContainer(CRI)
超过gracePeriod
优雅退出
GC回收
StartContainer(CRI)
正常退出/错误退出
运行时通信失败
重启(restartPolicy=Always/OnFailure)
恢复通信(仍运行)
Hook失败
Liveness/Startup失败
Created
Exited
Unknown
PostStartPending
PostStartRunning
PostStartDone
ProbeRunning
PreStopPending
PreStopRunning
GracefulStop
SIGTERM
SIGKILL
3.5 Sandbox管理
3.5.1 Sandbox管理流程
#mermaid-svg-aUOqZboMhKZEfV7e{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-aUOqZboMhKZEfV7e .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-aUOqZboMhKZEfV7e .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-aUOqZboMhKZEfV7e .error-icon{fill:#552222;}#mermaid-svg-aUOqZboMhKZEfV7e .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-aUOqZboMhKZEfV7e .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-aUOqZboMhKZEfV7e .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-aUOqZboMhKZEfV7e .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-aUOqZboMhKZEfV7e .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-aUOqZboMhKZEfV7e .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-aUOqZboMhKZEfV7e .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-aUOqZboMhKZEfV7e .marker{fill:#333333;stroke:#333333;}#mermaid-svg-aUOqZboMhKZEfV7e .marker.cross{stroke:#333333;}#mermaid-svg-aUOqZboMhKZEfV7e svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-aUOqZboMhKZEfV7e p{margin:0;}#mermaid-svg-aUOqZboMhKZEfV7e .label{font-family:"trebuchet ms",verdana,arial,sans-serif;color:#333;}#mermaid-svg-aUOqZboMhKZEfV7e .cluster-label text{fill:#333;}#mermaid-svg-aUOqZboMhKZEfV7e .cluster-label span{color:#333;}#mermaid-svg-aUOqZboMhKZEfV7e .cluster-label span p{background-color:transparent;}#mermaid-svg-aUOqZboMhKZEfV7e .label text,#mermaid-svg-aUOqZboMhKZEfV7e span{fill:#333;color:#333;}#mermaid-svg-aUOqZboMhKZEfV7e .node rect,#mermaid-svg-aUOqZboMhKZEfV7e .node circle,#mermaid-svg-aUOqZboMhKZEfV7e .node ellipse,#mermaid-svg-aUOqZboMhKZEfV7e .node polygon,#mermaid-svg-aUOqZboMhKZEfV7e .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-aUOqZboMhKZEfV7e .rough-node .label text,#mermaid-svg-aUOqZboMhKZEfV7e .node .label text,#mermaid-svg-aUOqZboMhKZEfV7e .image-shape .label,#mermaid-svg-aUOqZboMhKZEfV7e .icon-shape .label{text-anchor:middle;}#mermaid-svg-aUOqZboMhKZEfV7e .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-aUOqZboMhKZEfV7e .rough-node .label,#mermaid-svg-aUOqZboMhKZEfV7e .node .label,#mermaid-svg-aUOqZboMhKZEfV7e .image-shape .label,#mermaid-svg-aUOqZboMhKZEfV7e .icon-shape .label{text-align:center;}#mermaid-svg-aUOqZboMhKZEfV7e .node.clickable{cursor:pointer;}#mermaid-svg-aUOqZboMhKZEfV7e .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-aUOqZboMhKZEfV7e .arrowheadPath{fill:#333333;}#mermaid-svg-aUOqZboMhKZEfV7e .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-aUOqZboMhKZEfV7e .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-aUOqZboMhKZEfV7e .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-aUOqZboMhKZEfV7e .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-aUOqZboMhKZEfV7e .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-aUOqZboMhKZEfV7e .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-aUOqZboMhKZEfV7e .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-aUOqZboMhKZEfV7e .cluster text{fill:#333;}#mermaid-svg-aUOqZboMhKZEfV7e .cluster span{color:#333;}#mermaid-svg-aUOqZboMhKZEfV7e div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-aUOqZboMhKZEfV7e .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-aUOqZboMhKZEfV7e rect.text{fill:none;stroke-width:0;}#mermaid-svg-aUOqZboMhKZEfV7e .icon-shape,#mermaid-svg-aUOqZboMhKZEfV7e .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-aUOqZboMhKZEfV7e .icon-shape p,#mermaid-svg-aUOqZboMhKZEfV7e .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-aUOqZboMhKZEfV7e .icon-shape .label rect,#mermaid-svg-aUOqZboMhKZEfV7e .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-aUOqZboMhKZEfV7e .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-aUOqZboMhKZEfV7e .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-aUOqZboMhKZEfV7e :root{--mermaid-font-family:"trebuchet ms",verdana,arial,sans-serif;} Sandbox清理
READY或有容器关联
NOT READY且无容器
是
否
evictSandboxes
获取所有containers和sandboxes
sandbox是否active?
标记active=true
可驱逐
Pod已删除/终止?
删除所有sandbox
保留最新1个,删除其余
Sandbox变更检测
无
有
是
否
否
是
是
否
是
否
podSandboxChanged
有sandbox?
changed=true, attempt=0
readySandboxCount>1?
changed=true, 需要reconcile
sandbox READY?
changed=true, attempt+1
网络命名空间变更?
changed=true, attempt+1
非HostNetwork但无IP?
changed=true, attempt+1
changed=false, 保留sandbox
Sandbox创建流程
非空
空
createPodSandbox
generatePodSandboxConfig
设置Metadata: Name/Namespace/UID/Attempt
设置DNS配置
设置Hostname: 非HostNetwork时
设置PortMappings
generatePodSandboxLinuxConfig
设置CgroupParent
设置SecurityContext: Privileged/Seccomp/SELinux/RunAsUser
设置Sysctls
设置NamespaceOptions: IPC/Network/PID
设置LogDirectory: /var/log/pods/NAMESPACE_NAME_UID
runtimeClassManager.LookupRuntimeHandler
runtimeHandler?
RunPodSandbox with handler
RunPodSandbox default
podSandboxID
3.5.2 generatePodSandboxConfig深度解析
generatePodSandboxConfig方法将v1.Pod转换为CRI的PodSandboxConfig:
关键配置项:
- Metadata :
Name/Namespace/Uid/Attempt(attempt用于跟踪沙箱重建次数) - DNS :通过
runtimeHelper.GetPodDNS获取,包含nameservers/searches/options - Hostname :非HostNetwork时,通过
GeneratePodHostNameAndDomain+GetNodenameForKernel生成 - PortMappings:遍历Pod所有容器的端口映射
- Linux配置 :
CgroupParent:从runtimeHelper.GetPodCgroupParent获取SecurityContext.Privileged:只要Pod中有任一特权容器则为trueSecurityContext.Seccomp:默认RuntimeDefaultSecurityContext.NamespaceOptions:根据Pod的hostNetwork/hostIPC/hostPID决定Sysctls:从pod.Spec.SecurityContext.Sysctls读取RunAsUser/RunAsGroup/FSGroup/SupplementalGroups/SELinuxOptions
3.6 镜像拉取策略与逻辑
3.6.1 镜像拉取流程图
渲染错误: Mermaid 渲染失败: Parse error on line 40: ...|是| Ukeyring.Lookup(repoToPull) U -----------------------^ Expecting 'SQE', 'DOUBLECIRCLEEND', 'PE', '-)', 'STADIUMEND', 'SUBROUTINEEND', 'PIPE', 'CYLINDEREND', 'DIAMOND_STOP', 'TAGEND', 'TRAPEND', 'INVTRAPEND', 'UNICODE_TEXT', 'TEXT', 'TAGSTART', got 'PS'
3.6.2 PullImage凭证链解析
kubeGenericRuntimeManager.PullImage实现了多凭证链尝试机制:
go
func (m *kubeGenericRuntimeManager) PullImage(image kubecontainer.ImageSpec, pullSecrets []v1.Secret, ...) (string, error) {
// 1. 解析镜像名获取仓库名
repoToPull, _, _, err := parsers.ParseImageName(img)
// 2. 合并pullSecrets与默认keyring创建凭证环
keyring, err := credentialprovidersecrets.MakeDockerKeyring(pullSecrets, m.keyring)
// 3. 查找凭证
creds, withCredentials := keyring.Lookup(repoToPull)
if !withCredentials {
// 无凭证:直接拉取
imageRef, err := m.imageService.PullImage(imgSpec, nil, podSandboxConfig)
return imageRef, err
}
// 4. 有凭证:遍历所有credential尝试
var pullErrs []error
for _, currentCreds := range creds {
auth := &runtimeapi.AuthConfig{
Username: currentCreds.Username,
Password: currentCreds.Password,
Auth: currentCreds.Auth,
...
}
imageRef, err := m.imageService.PullImage(imgSpec, auth, podSandboxConfig)
if err == nil {
return imageRef, nil // 任一凭证成功即返回
}
pullErrs = append(pullErrs, err)
}
// 5. 所有凭证失败:聚合错误
return "", utilerrors.NewAggregate(pullErrs)
}
3.6.3 镜像拉取QPS控制
在NewImageManager中,通过throttleImagePulling包装器实现QPS限流:
go
imageService = throttleImagePulling(imageService, qps, burst)
序列化/并行拉取策略:
- serialImagePuller :通过channel队列+单goroutine顺序处理,最大队列深度
maxImagePullRequests=10 - parallelImagePuller:每个拉取请求起独立goroutine,无并发限制
3.7 容器日志管理
3.7.1 容器日志管理架构图
渲染错误: Mermaid 渲染失败: Parse error on line 5: ...日志文件] Note: CRI日志格式: timestamp s ----------------------^ Expecting 'SEMI', 'NEWLINE', 'EOF', 'AMP', 'START_LINK', 'LINK', 'LINK_ID', got 'NODE_STRING'
3.7.2 CRI日志格式详解
CRI定义了标准日志格式,每行结构为:
<timestamp> <stream> <tag> <log content>
- timestamp :RFC3339NanoFixed格式,如
2016-10-06T00:17:09.669794202Z - stream :
stdout或stderr - tag :日志标签,
P表示partial line(未结束行),F表示full line(完整行) - log content:实际日志内容
parseCRILog解析逻辑:
- 以空格分割,第一段为timestamp
- 第二段为stream type(stdout/stderr)
- 第三段为tag,使用
LogTagDelimiter分割 - 如果是partial line(tag0==P),去除尾部换行符
- 剩余部分为log content
同时兼容Docker JSON日志格式parseDockerJSONLog:
json
{"log":"content","stream":"stdout","time":"2016-10-20T18:39:20.57606443Z"}
3.7.3 日志轮转机制
containerLogManager的轮转策略:
- 触发条件 :每10秒检查一次,当日志文件大小超过
MaxSize时触发 - 轮转步骤 :
- 清理上次轮转失败留下的临时文件(
.tmp) - 删除超出
MaxFiles限制的旧日志 - 将未压缩的旧日志用gzip压缩(
.gz后缀) - 将当前日志重命名为
<log>.<timestamp> - 调用
ReopenContainerLog让运行时重新打开日志文件 - 如果Reopen失败,将重命名的日志改回原名(避免日志丢失)
- 清理上次轮转失败留下的临时文件(
3.8 关键判断、分支、循环
3.8.1 computePodActions决策树
这是SyncPod最关键的决策逻辑,决定整个Pod的变更动作:
#mermaid-svg-2ThJKqtkQQtIMqP4{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-2ThJKqtkQQtIMqP4 .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-2ThJKqtkQQtIMqP4 .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-2ThJKqtkQQtIMqP4 .error-icon{fill:#552222;}#mermaid-svg-2ThJKqtkQQtIMqP4 .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-2ThJKqtkQQtIMqP4 .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-2ThJKqtkQQtIMqP4 .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-2ThJKqtkQQtIMqP4 .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-2ThJKqtkQQtIMqP4 .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-2ThJKqtkQQtIMqP4 .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-2ThJKqtkQQtIMqP4 .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-2ThJKqtkQQtIMqP4 .marker{fill:#333333;stroke:#333333;}#mermaid-svg-2ThJKqtkQQtIMqP4 .marker.cross{stroke:#333333;}#mermaid-svg-2ThJKqtkQQtIMqP4 svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-2ThJKqtkQQtIMqP4 p{margin:0;}#mermaid-svg-2ThJKqtkQQtIMqP4 .label{font-family:"trebuchet ms",verdana,arial,sans-serif;color:#333;}#mermaid-svg-2ThJKqtkQQtIMqP4 .cluster-label text{fill:#333;}#mermaid-svg-2ThJKqtkQQtIMqP4 .cluster-label span{color:#333;}#mermaid-svg-2ThJKqtkQQtIMqP4 .cluster-label span p{background-color:transparent;}#mermaid-svg-2ThJKqtkQQtIMqP4 .label text,#mermaid-svg-2ThJKqtkQQtIMqP4 span{fill:#333;color:#333;}#mermaid-svg-2ThJKqtkQQtIMqP4 .node rect,#mermaid-svg-2ThJKqtkQQtIMqP4 .node circle,#mermaid-svg-2ThJKqtkQQtIMqP4 .node ellipse,#mermaid-svg-2ThJKqtkQQtIMqP4 .node polygon,#mermaid-svg-2ThJKqtkQQtIMqP4 .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-2ThJKqtkQQtIMqP4 .rough-node .label text,#mermaid-svg-2ThJKqtkQQtIMqP4 .node .label text,#mermaid-svg-2ThJKqtkQQtIMqP4 .image-shape .label,#mermaid-svg-2ThJKqtkQQtIMqP4 .icon-shape .label{text-anchor:middle;}#mermaid-svg-2ThJKqtkQQtIMqP4 .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-2ThJKqtkQQtIMqP4 .rough-node .label,#mermaid-svg-2ThJKqtkQQtIMqP4 .node .label,#mermaid-svg-2ThJKqtkQQtIMqP4 .image-shape .label,#mermaid-svg-2ThJKqtkQQtIMqP4 .icon-shape .label{text-align:center;}#mermaid-svg-2ThJKqtkQQtIMqP4 .node.clickable{cursor:pointer;}#mermaid-svg-2ThJKqtkQQtIMqP4 .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-2ThJKqtkQQtIMqP4 .arrowheadPath{fill:#333333;}#mermaid-svg-2ThJKqtkQQtIMqP4 .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-2ThJKqtkQQtIMqP4 .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-2ThJKqtkQQtIMqP4 .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-2ThJKqtkQQtIMqP4 .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-2ThJKqtkQQtIMqP4 .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-2ThJKqtkQQtIMqP4 .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-2ThJKqtkQQtIMqP4 .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-2ThJKqtkQQtIMqP4 .cluster text{fill:#333;}#mermaid-svg-2ThJKqtkQQtIMqP4 .cluster span{color:#333;}#mermaid-svg-2ThJKqtkQQtIMqP4 div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-2ThJKqtkQQtIMqP4 .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-2ThJKqtkQQtIMqP4 rect.text{fill:none;stroke-width:0;}#mermaid-svg-2ThJKqtkQQtIMqP4 .icon-shape,#mermaid-svg-2ThJKqtkQQtIMqP4 .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-2ThJKqtkQQtIMqP4 .icon-shape p,#mermaid-svg-2ThJKqtkQQtIMqP4 .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-2ThJKqtkQQtIMqP4 .icon-shape .label rect,#mermaid-svg-2ThJKqtkQQtIMqP4 .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-2ThJKqtkQQtIMqP4 .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-2ThJKqtkQQtIMqP4 .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-2ThJKqtkQQtIMqP4 :root{--mermaid-font-family:"trebuchet ms",verdana,arial,sans-serif;} 是: changed=true
是
是
否
否
是
否
否
是
否
是
否: sandbox未变更
否
是且restartPolicy=Never
是且restartPolicy!=Never
否: init仍在运行
是
不存在/非Running
是
Unknown状态
否
Running
容器spec变更
LivenessProbe失败
StartupProbe失败
正常
是
computePodActions
podSandboxChanged
sandbox变更?
restartPolicy!=Never AND attempt!=0 AND 有容器状态?
不应重启?
CreateSandbox=false, 直接返回
需要创建新sandbox
有Init容器?
NextInitContainerToStart=第一个init容器
过滤RestartPolicy=OnFailure中已成功的容器
还有容器要启动?
init容器是否全部完成?
CreateSandbox=false, 返回
ContainersToStart=待启动容器列表
检查Ephemeral容器
findNextInitContainerToRun
init完成?
init容器失败?
KillPod=true
NextInitContainerToStart=失败的init容器
等待,返回
遍历所有普通容器
容器状态检查
ShouldContainerBeRestarted?
加入ContainersToStart
加入ContainersToKill先杀再启
跳过
变更检查
加入ContainersToKill+ToStart
加入ContainersToKill+ToStart, reason=LivenessProbe
加入ContainersToKill+ToStart, reason=StartupProbe
keepCount++
keepCount==0 AND ToStart==0?
KillPod=true
3.8.2 容器终止与优雅关闭流程
渲染错误: Mermaid 渲染失败: Parse error on line 4: ...erLabels] Note: 从容器label/annotation恢 ----------------------^ Expecting 'SEMI', 'NEWLINE', 'EOF', 'AMP', 'START_LINK', 'LINK', 'LINK_ID', got 'UNICODE_TEXT'
3.8.3 killContainer中的gracePeriod计算优先级
优先级从高到低:
1. gracePeriodOverride(仅硬驱逐等路径使用)
2. pod.DeletionGracePeriodSeconds
3. pod.Spec.TerminationGracePeriodSeconds
- 若ProbeTerminationGracePeriod特性启用:
a. StartupProbe失败 → containerSpec.StartupProbe.TerminationGracePeriodSeconds
b. LivenessProbe失败 → containerSpec.LivenessProbe.TerminationGracePeriodSeconds
4. 最小值minimumGracePeriodInSeconds = 2秒
3.8.4 PreStop Hook执行机制
executePreStopHook在独立goroutine中执行,受gracePeriod限制:
go
func (m *kubeGenericRuntimeManager) executePreStopHook(pod, containerID, containerSpec, gracePeriod) int64 {
start := metav1.Now()
done := make(chan struct{})
go func() {
defer close(done)
msg, err := m.runner.Run(containerID, pod, containerSpec, containerSpec.Lifecycle.PreStop)
}()
select {
case <-time.After(time.Duration(gracePeriod) * time.Second):
// PreStop超时,继续执行StopContainer
case <-done:
// PreStop完成
}
return int64(metav1.Now().Sub(start.Time).Seconds()) // 返回实际耗时
}
3.8.5 Init容器状态判断逻辑
findNextInitContainerToRun从后向前遍历init容器:
- 如果任何主容器在Running状态 → 所有init已完成,返回done=true
- 从后向前查找第一个失败的init容器 → 返回该容器需要重启
- 无失败容器时,从后向前查找已完成的init容器 → 返回下一个待执行的init
- 找不到任何init容器状态 → 返回第一个init容器需要启动
3.8.6 容器重启策略处理流程
渲染错误: Mermaid 渲染失败: Parse error on line 18: ...art] Note: computePodActions中特殊 ---------------------^ Expecting 'SEMI', 'NEWLINE', 'EOF', 'AMP', 'START_LINK', 'LINK', 'LINK_ID', got 'NODE_STRING'
3.9 dockershim适配层架构
#mermaid-svg-Yem9yxkV5ksygVYg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-Yem9yxkV5ksygVYg .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-Yem9yxkV5ksygVYg .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-Yem9yxkV5ksygVYg .error-icon{fill:#552222;}#mermaid-svg-Yem9yxkV5ksygVYg .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-Yem9yxkV5ksygVYg .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-Yem9yxkV5ksygVYg .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-Yem9yxkV5ksygVYg .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-Yem9yxkV5ksygVYg .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-Yem9yxkV5ksygVYg .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-Yem9yxkV5ksygVYg .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-Yem9yxkV5ksygVYg .marker{fill:#333333;stroke:#333333;}#mermaid-svg-Yem9yxkV5ksygVYg .marker.cross{stroke:#333333;}#mermaid-svg-Yem9yxkV5ksygVYg svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-Yem9yxkV5ksygVYg p{margin:0;}#mermaid-svg-Yem9yxkV5ksygVYg .label{font-family:"trebuchet ms",verdana,arial,sans-serif;color:#333;}#mermaid-svg-Yem9yxkV5ksygVYg .cluster-label text{fill:#333;}#mermaid-svg-Yem9yxkV5ksygVYg .cluster-label span{color:#333;}#mermaid-svg-Yem9yxkV5ksygVYg .cluster-label span p{background-color:transparent;}#mermaid-svg-Yem9yxkV5ksygVYg .label text,#mermaid-svg-Yem9yxkV5ksygVYg span{fill:#333;color:#333;}#mermaid-svg-Yem9yxkV5ksygVYg .node rect,#mermaid-svg-Yem9yxkV5ksygVYg .node circle,#mermaid-svg-Yem9yxkV5ksygVYg .node ellipse,#mermaid-svg-Yem9yxkV5ksygVYg .node polygon,#mermaid-svg-Yem9yxkV5ksygVYg .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-Yem9yxkV5ksygVYg .rough-node .label text,#mermaid-svg-Yem9yxkV5ksygVYg .node .label text,#mermaid-svg-Yem9yxkV5ksygVYg .image-shape .label,#mermaid-svg-Yem9yxkV5ksygVYg .icon-shape .label{text-anchor:middle;}#mermaid-svg-Yem9yxkV5ksygVYg .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-Yem9yxkV5ksygVYg .rough-node .label,#mermaid-svg-Yem9yxkV5ksygVYg .node .label,#mermaid-svg-Yem9yxkV5ksygVYg .image-shape .label,#mermaid-svg-Yem9yxkV5ksygVYg .icon-shape .label{text-align:center;}#mermaid-svg-Yem9yxkV5ksygVYg .node.clickable{cursor:pointer;}#mermaid-svg-Yem9yxkV5ksygVYg .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-Yem9yxkV5ksygVYg .arrowheadPath{fill:#333333;}#mermaid-svg-Yem9yxkV5ksygVYg .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-Yem9yxkV5ksygVYg .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-Yem9yxkV5ksygVYg .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-Yem9yxkV5ksygVYg .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-Yem9yxkV5ksygVYg .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-Yem9yxkV5ksygVYg .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-Yem9yxkV5ksygVYg .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-Yem9yxkV5ksygVYg .cluster text{fill:#333;}#mermaid-svg-Yem9yxkV5ksygVYg .cluster span{color:#333;}#mermaid-svg-Yem9yxkV5ksygVYg div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-Yem9yxkV5ksygVYg .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-Yem9yxkV5ksygVYg rect.text{fill:none;stroke-width:0;}#mermaid-svg-Yem9yxkV5ksygVYg .icon-shape,#mermaid-svg-Yem9yxkV5ksygVYg .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-Yem9yxkV5ksygVYg .icon-shape p,#mermaid-svg-Yem9yxkV5ksygVYg .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-Yem9yxkV5ksygVYg .icon-shape .label rect,#mermaid-svg-Yem9yxkV5ksygVYg .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-Yem9yxkV5ksygVYg .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-Yem9yxkV5ksygVYg .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-Yem9yxkV5ksygVYg :root{--mermaid-font-family:"trebuchet ms",verdana,arial,sans-serif;} dockershim网络流程
RunPodSandbox
创建pause容器
获取pause容器PID
获取netns路径 /proc/PID/ns/net
plugin.Setup - 执行CNI
设置HostPort
返回sandboxID
dockershim架构
Legacy兼容
docker_legacy_service.go
GetContainerLogs - Docker日志驱动
legacyLogProvider - journald等
DockerService接口
RuntimeServiceServer
ImageServiceServer
DockerService实现
DockerClient - libdocker
NetworkPlugin管理
Streaming Server
CheckpointManager
ContainerManager - cm
CNI插件
kubenet插件
HostPort管理
关键适配逻辑
docker_sandbox.go
RunPodSandbox: 创建pause容器+设置网络
StopPodSandbox: 停止pause容器
RemovePodSandbox: 删除pause容器
docker_container.go
CreateContainer: docker create
StartContainer: docker start
StopContainer: docker stop -t gracePeriod
docker_image.go
PullImage: docker pull
ListImages: docker images
RemoveImage: docker rmi
convert.go
CRI类型 ↔ Docker类型转换
dockershim关键设计:
- 使用
pause容器作为Pod Sandbox,pause容器只持有Linux命名空间 - 通过Docker API创建/管理容器,将CRI语义映射到Docker操作
- 网络插件通过netns路径调用CNI/kubenet
- 容器标签使用
io.kubernetes.docker.type区分sandbox和container - 支持legacy Docker日志驱动(journald等),通过
legacyLogProvider接口
3.10 RuntimeClass动态选择流程
渲染错误: Mermaid 渲染失败: Parse error on line 14: ...择runtime] Note: containerd: 多shim进程< ----------------------^ Expecting 'SEMI', 'NEWLINE', 'EOF', 'AMP', 'START_LINK', 'LINK', 'LINK_ID', got 'NODE_STRING'
RuntimeClass Manager实现:
- 使用
SharedInformerFactory监听RuntimeClass资源变更 resyncPeriod=0,不定期全量同步LookupRuntimeHandler直接从缓存读取,无API调用开销- 默认RuntimeClass(nil/空名)映射到空handler字符串
3.11 容器GC策略
#mermaid-svg-PVPwscSWiLB6oQCI{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-PVPwscSWiLB6oQCI .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-PVPwscSWiLB6oQCI .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-PVPwscSWiLB6oQCI .error-icon{fill:#552222;}#mermaid-svg-PVPwscSWiLB6oQCI .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-PVPwscSWiLB6oQCI .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-PVPwscSWiLB6oQCI .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-PVPwscSWiLB6oQCI .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-PVPwscSWiLB6oQCI .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-PVPwscSWiLB6oQCI .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-PVPwscSWiLB6oQCI .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-PVPwscSWiLB6oQCI .marker{fill:#333333;stroke:#333333;}#mermaid-svg-PVPwscSWiLB6oQCI .marker.cross{stroke:#333333;}#mermaid-svg-PVPwscSWiLB6oQCI svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-PVPwscSWiLB6oQCI p{margin:0;}#mermaid-svg-PVPwscSWiLB6oQCI .label{font-family:"trebuchet ms",verdana,arial,sans-serif;color:#333;}#mermaid-svg-PVPwscSWiLB6oQCI .cluster-label text{fill:#333;}#mermaid-svg-PVPwscSWiLB6oQCI .cluster-label span{color:#333;}#mermaid-svg-PVPwscSWiLB6oQCI .cluster-label span p{background-color:transparent;}#mermaid-svg-PVPwscSWiLB6oQCI .label text,#mermaid-svg-PVPwscSWiLB6oQCI span{fill:#333;color:#333;}#mermaid-svg-PVPwscSWiLB6oQCI .node rect,#mermaid-svg-PVPwscSWiLB6oQCI .node circle,#mermaid-svg-PVPwscSWiLB6oQCI .node ellipse,#mermaid-svg-PVPwscSWiLB6oQCI .node polygon,#mermaid-svg-PVPwscSWiLB6oQCI .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-PVPwscSWiLB6oQCI .rough-node .label text,#mermaid-svg-PVPwscSWiLB6oQCI .node .label text,#mermaid-svg-PVPwscSWiLB6oQCI .image-shape .label,#mermaid-svg-PVPwscSWiLB6oQCI .icon-shape .label{text-anchor:middle;}#mermaid-svg-PVPwscSWiLB6oQCI .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-PVPwscSWiLB6oQCI .rough-node .label,#mermaid-svg-PVPwscSWiLB6oQCI .node .label,#mermaid-svg-PVPwscSWiLB6oQCI .image-shape .label,#mermaid-svg-PVPwscSWiLB6oQCI .icon-shape .label{text-align:center;}#mermaid-svg-PVPwscSWiLB6oQCI .node.clickable{cursor:pointer;}#mermaid-svg-PVPwscSWiLB6oQCI .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-PVPwscSWiLB6oQCI .arrowheadPath{fill:#333333;}#mermaid-svg-PVPwscSWiLB6oQCI .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-PVPwscSWiLB6oQCI .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-PVPwscSWiLB6oQCI .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-PVPwscSWiLB6oQCI .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-PVPwscSWiLB6oQCI .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-PVPwscSWiLB6oQCI .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-PVPwscSWiLB6oQCI .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-PVPwscSWiLB6oQCI .cluster text{fill:#333;}#mermaid-svg-PVPwscSWiLB6oQCI .cluster span{color:#333;}#mermaid-svg-PVPwscSWiLB6oQCI div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-PVPwscSWiLB6oQCI .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-PVPwscSWiLB6oQCI rect.text{fill:none;stroke-width:0;}#mermaid-svg-PVPwscSWiLB6oQCI .icon-shape,#mermaid-svg-PVPwscSWiLB6oQCI .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-PVPwscSWiLB6oQCI .icon-shape p,#mermaid-svg-PVPwscSWiLB6oQCI .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-PVPwscSWiLB6oQCI .icon-shape .label rect,#mermaid-svg-PVPwscSWiLB6oQCI .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-PVPwscSWiLB6oQCI .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-PVPwscSWiLB6oQCI .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-PVPwscSWiLB6oQCI :root{--mermaid-font-family:"trebuchet ms",verdana,arial,sans-serif;} 日志目录驱逐
沙箱驱逐策略
容器驱逐策略
容器GC三步流程
是
否
是
否
是
否
是
否
是
是
否
GarbageCollect
Step1: evictContainers
Step2: evictSandboxes
Step3: evictPodLogsDirectories
evictableContainers: 非Running且创建超过MinAge
allSourcesReady?
删除已删除/终止Pod的容器
保守策略
enforceMaxContainersPerEvictUnit
NumContainers > MaxContainers?
均分配额后删除最旧容器
保持
获取所有containers和sandboxes
标记active: READY或有容器关联
非active的sandbox可驱逐
Pod已删除/终止?
删除所有sandbox
保留最新1个
removeOldestNSandboxes
先StopPodSandbox再RemovePodSandbox
遍历/var/log/pods/下所有目录
Pod已删除?
RemoveAll整个日志目录
保留
清理dead container的legacy symlink
symlink目标不存在?
ContainerStatus检查
容器已EXITED?
删除symlink
可能是日志轮转间隙,保留
3.12 instrumentedServices度量封装
instrumentedRuntimeService和instrumentedImageManagerService是装饰器模式,为所有CRI调用添加度量指标:
记录的指标:
metrics.RuntimeOperations:操作计数(按operation label)metrics.RuntimeOperationsDuration:操作耗时直方图metrics.RuntimeOperationsErrors:错误计数metrics.RunPodSandboxDuration:沙箱创建耗时(按runtimeHandler label)metrics.RunPodSandboxErrors:沙箱创建错误计数
每个方法的标准包装模式:
go
func (in instrumentedRuntimeService) XXX(...) (...) {
const operation = "xxx"
defer recordOperation(operation, time.Now()) // 记录操作次数和耗时
out, err := in.service.XXX(...) // 调用底层服务
recordError(operation, err) // 错误时记录错误指标
return out, err
}
3.13 remoteRuntimeService gRPC客户端
remoteRuntimeService是CRI的gRPC客户端实现,连接远程容器运行时:
关键设计:
- 使用
grpc.DialContext连接运行时endpoint(Unix socket或TCP) maxMsgSize=16*1024*1024(16MB),设置gRPC最大消息大小- 默认超时
connectionTimeout,RunPodSandbox使用2倍超时 - 每次调用创建带超时的context:
getContextWithTimeout(r.timeout)
连接建立:
go
func NewRemoteRuntimeService(endpoint string, connectionTimeout time.Duration) (internalapi.RuntimeService, error) {
addr, dialer, err := util.GetAddressAndDialer(endpoint) // 解析endpoint
conn, err := grpc.DialContext(ctx, addr,
grpc.WithInsecure(),
grpc.WithContextDialer(dialer),
grpc.WithDefaultCallOptions(grpc.MaxCallRecvMsgSize(maxMsgSize)))
return &remoteRuntimeService{
timeout: connectionTimeout,
runtimeClient: runtimeapi.NewRuntimeServiceClient(conn),
}, nil
}
3.14 安全上下文处理
3.14.1 determineEffectiveSecurityContext
安全上下文合并优先级:
container.SecurityContext字段优先- 回退到
pod.Spec.SecurityContext - 最终从镜像获取默认用户(uid/username)
- 默认以root运行(
uid=new(int64), 值为0)
设置的安全属性:
RunAsUser/RunAsGroup/RunAsUsernamePrivilegedCapabilities(Add/Drop)SELinuxOptionsSeccompProfilePath/Seccomp(SecurityProfile)ApparmorProfileNamespaceOptions(IPC/Network/PID mode + TARGET ID)SupplementalGroups/FSGroupNoNewPrivsReadonlyRootfsMaskedPaths/ReadonlyPaths(ProcMount类型)
3.14.2 Seccomp优先级链
1. container.SecurityContext.SeccompProfile ← 最高优先
2. container annotation: container.seccomp.security.alpha.kubernetes.io/<container_name> ← 已废弃
3. pod.Spec.SecurityContext.SeccompProfile
4. pod annotation: seccomp.security.alpha.kubernetes.io/pod ← 已废弃
5. 默认: Unconfined(容器级别)/ RuntimeDefault(沙箱级别)
3.15 标签与注解系统
3.15.1 容器标签(Labels)
用于标识和过滤容器,存储在CRI容器的Labels中:
| Label Key | 含义 |
|---|---|
io.kubernetes.pod.name |
Pod名称 |
io.kubernetes.pod.namespace |
Pod命名空间 |
io.kubernetes.pod.uid |
Pod UID |
io.kubernetes.container.name |
容器名称 |
3.15.2 容器注解(Annotations)
用于存储恢复容器所需的信息,存储在CRI容器的Annotations中:
| Annotation Key | 含义 |
|---|---|
io.kubernetes.container.hash |
容器spec的hash值,用于变更检测 |
io.kubernetes.container.restartCount |
重启次数 |
io.kubernetes.container.terminationMessagePath |
终止消息路径 |
io.kubernetes.container.terminationMessagePolicy |
终止消息策略 |
io.kubernetes.container.preStopHandler |
PreStop Hook(JSON序列化) |
io.kubernetes.container.ports |
容器端口(JSON序列化) |
io.kubernetes.pod.deletionGracePeriod |
Pod删除优雅期 |
io.kubernetes.pod.terminationGracePeriod |
Pod终止优雅期 |
设计意图: 当kubelet重启时,可能无法获取Pod spec(Pod已被删除),此时通过labels/annotations恢复必要信息来执行容器终止操作。
3.16 Termination Message处理
容器终止时,终止消息的获取逻辑:
- 从文件读取 :如果容器设置了
terminationMessagePath,从挂载的文件读取(最大4KB) - 从日志读取 :如果
terminationMessagePolicy=FallbackToLogsOnError且退出码非0:- 优先使用
legacyLogProvider(Docker journald等) - 否则使用
readLastStringFromContainerLogs(读取CRI日志最后80行,最大2KB)
- 优先使用
- 消息合并:将终止消息追加到ContainerStatus.Message中
3.17 Ephemeral容器处理
Ephemeral容器是临时调试容器,特殊逻辑:
- 在Step 5中先于init容器启动
- 不受init容器状态影响(即使init失败也能启动)
- 永远不会重启(
FindContainerStatusByName为nil时才启动) - 支持namespace targeting:
TargetContainerName指定目标容器的PID命名空间 - 通过
getTargetID解析目标容器ID,设置NamespaceOptions.Pid=TARGET
四、总结
4.1 架构特点
- 接口分层清晰 :
kubecontainer.Runtime→kubeGenericRuntimeManager→instrumentedRuntimeService→remoteRuntimeService→gRPC→运行时,每层职责单一 - 装饰器模式:instrumented services为CRI调用透明添加度量
- 标签恢复机制:通过容器labels/annotations存储关键信息,解决kubelet重启后的Pod spec丢失问题
- 多凭证链:镜像拉取支持多凭证自动尝试
- BackOff机制:容器重启和镜像拉取都有退避策略,避免频繁重试
- 优雅关闭:PreStop Hook→SIGTERM→gracePeriod→SIGKILL,支持Probe级别的gracePeriod覆盖
4.2 关键路径
- Pod创建 :
SyncPod→computePodActions→createPodSandbox→startContainer(×N) - Pod更新 :
SyncPod→computePodActions→killContainer(变更的)→startContainer(新的) - Pod删除 :
KillPod→killContainersWithSyncResult(并发)→StopPodSandbox - 镜像拉取 :
EnsureImageExists→shouldPullImage→PullImage(带凭证链) - 日志读取 :
ReadLogs→fsnotify→parseCRILog/parseDockerJSONLog→logWriter - GC回收 :
GarbageCollect→evictContainers→evictSandboxes→evictPodLogsDirectories