独立Kubelet静态Pod实践使用ECR Credential Provider拉取镜像的实践

ecr-credential-provider的工作原理

Kubernetes 从 v1.20 开始引入 kubelet image credential provider 插件机制（KEP-2133），用于替代之前内置在 kubelet 中的云厂商镜像凭证逻辑。AWS 的实现就是 ecr-credential-provider。

EKS 节点上默认已经部署了这个插件：

二进制：/etc/eks/image-credential-provider/ecr-credential-provider
配置：/etc/eks/image-credential-provider/config.json

ecr-credential-provider的运行时序图
AWS ECR API ecr-credential-provider kubelet AWS ECR API ecr-credential-provider kubelet 需要拉取镜像检查 config.json matchImages 规则匹配 stdin: CredentialProviderRequest (image地址) 解析镜像地址，提取 region 和 account ID GetAuthorizationToken (使用节点 IAM 角色) 返回临时 token (有效期12h) stdout: CredentialProviderResponse (username=AWS, password=token) 使用凭证拉取镜像，按 cacheDuration 缓存

配置文件 (config.json)示例

name：对应 bin 目录下的二进制文件名
matchImages：镜像地址匹配规则，支持通配符
defaultCacheDuration：凭证缓存时间（实际以 provider 返回的 cacheDuration 为准）

json 复制代码

{
  "kind": "CredentialProviderConfig",
  "apiVersion": "kubelet.config.k8s.io/v1",
  "providers": [
    {
      "name": "ecr-credential-provider",
      "matchImages": [
        "*.dkr.ecr.*.amazonaws.com",
        "*.dkr.ecr.*.amazonaws.com.cn",
        "*.dkr.ecr-fips.*.amazonaws.com",
        "public.ecr.aws"
      ],
      "defaultCacheDuration": "12h0m0s",
      "apiVersion": "credentialprovider.kubelet.k8s.io/v1"
    }
  ]
}

ecr-credential-provider请求的协议格式

请求 (stdin → provider)

json 复制代码

{
  "kind": "CredentialProviderRequest",
  "apiVersion": "credentialprovider.kubelet.k8s.io/v1",
  "image": "112233445566.dkr.ecr.cn-northwest-1.amazonaws.com.cn/my-app:latest"
}

响应 (provider → stdout)

json 复制代码

{
  "kind": "CredentialProviderResponse",
  "apiVersion": "credentialprovider.kubelet.k8s.io/v1",
  "cacheKeyType": "Registry",
  "cacheDuration": "6h0m0s",
  "auth": {
    "112233445566.dkr.ecr.cn-northwest-1.amazonaws.com.cn": {
      "username": "AWS",
      "password": "<base64-encoded-token>"
    }
  }
}

可以直接通过 stdin/stdout 测试该二进制：

bash 复制代码

echo '{"kind":"CredentialProviderRequest","apiVersion":"credentialprovider.kubelet.k8s.io/v1","image":"112233445566.dkr.ecr.cn-northwest-1.amazonaws.com.cn/my-app:latest"}' | /etc/eks/image-credential-provider/ecr-credential-provider

为kubelet 添加启动参数

bash 复制代码

kubelet \
  --image-credential-provider-bin-dir=/etc/eks/image-credential-provider/ \
  --image-credential-provider-config=/etc/eks/image-credential-provider/config.json

节点的 IAM 角色需要以下权限：

json 复制代码

{
  "Effect": "Allow",
  "Action": [
    "ecr:GetAuthorizationToken",
    "ecr:BatchCheckLayerAvailability",
    "ecr:GetDownloadUrlForLayer",
    "ecr:BatchGetImage"
  ],
  "Resource": "*"
}

ecr-credential-provider 二进制获取可以从 EKS 公开 S3 桶下载：

bash 复制代码

aws s3 cp s3://amazon-eks/1.30.9/2025-02-11/bin/linux/amd64/ecr-credential-provider .
chmod +x ecr-credential-provider

独立kubelet启动静态pod

在一台普通 EC2 实例上，不加入任何 K8s 集群，仅通过独立 kubelet + ecr-credential-provider 拉取 ECR 私有镜像并运行静态 Pod。

安装 containerd

bash 复制代码

sudo dnf install -y containerd
sudo systemctl enable --now containerd

注：AL2023 的 containerd 包会自动安装 runc 作为依赖，无需单独安装。可通过 runc --version 验证。

配置 containerd（指定可达的 pause 镜像）

中国区无法直接拉取 registry.k8s.io/pause，需要配置替代源：

bash 复制代码

sudo mkdir -p /etc/containerd
sudo tee /etc/containerd/config.toml << 'EOF'
version = 2

[plugins."io.containerd.grpc.v1.cri"]
  sandbox_image = "registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.9"

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
  runtime_type = "io.containerd.runc.v2"

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
  SystemdCgroup = true
EOF

sudo systemctl restart containerd

安装 kubelet

bash 复制代码

KUBE_VERSION=v1.31.4
curl -LO "https://dl.k8s.io/release/${KUBE_VERSION}/bin/linux/amd64/kubelet"
chmod +x kubelet
sudo mv kubelet /usr/local/bin/

安装 CNI 插件和 iptables

bash 复制代码

# CNI 插件
sudo mkdir -p /opt/cni/bin
curl -LO https://github.com/containernetworking/plugins/releases/download/v1.6.2/cni-plugins-linux-amd64-v1.6.2.tgz
sudo tar -xzf cni-plugins-linux-amd64-v1.6.2.tgz -C /opt/cni/bin

# CNI 网络配置
sudo mkdir -p /etc/cni/net.d
cat << 'EOF' | sudo tee /etc/cni/net.d/10-bridge.conflist
{
  "cniVersion": "1.0.0",
  "name": "bridge",
  "plugins": [
    {
      "type": "bridge",
      "bridge": "cni0",
      "isGateway": true,
      "ipMasq": true,
      "ipam": {
        "type": "host-local",
        "ranges": [[{"subnet": "10.244.0.0/24"}]]
      }
    },
    {
      "type": "portmap",
      "capabilities": {"portMappings": true}
    }
  ]
}
EOF

# iptables（AL2023 默认未安装）
sudo dnf install -y iptables-nft

配置 ECR Credential Provider

bash 复制代码

sudo mkdir -p /etc/eks/image-credential-provider

# 下载二进制
aws s3 cp s3://amazon-eks/1.30.9/2025-02-11/bin/linux/amd64/ecr-credential-provider .
chmod +x ecr-credential-provider
sudo mv ecr-credential-provider /etc/eks/image-credential-provider/

# 写入配置
sudo tee /etc/eks/image-credential-provider/config.json << 'EOF'
{
  "kind": "CredentialProviderConfig",
  "apiVersion": "kubelet.config.k8s.io/v1",
  "providers": [
    {
      "name": "ecr-credential-provider",
      "matchImages": [
        "*.dkr.ecr.*.amazonaws.com",
        "*.dkr.ecr.*.amazonaws.com.cn",
        "*.dkr.ecr-fips.*.amazonaws.com",
        "public.ecr.aws"
      ],
      "defaultCacheDuration": "12h0m0s",
      "apiVersion": "credentialprovider.kubelet.k8s.io/v1"
    }
  ]
}
EOF

配置 kubelet systemd service

bash 复制代码

sudo tee /etc/systemd/system/kubelet.service << 'EOF'
[Unit]
Description=Kubelet (Standalone)
After=containerd.service
Requires=containerd.service

[Service]
ExecStart=/usr/local/bin/kubelet \
  --container-runtime-endpoint=unix:///run/containerd/containerd.sock \
  --pod-manifest-path=/etc/kubernetes/manifests \
  --image-credential-provider-bin-dir=/etc/eks/image-credential-provider/ \
  --image-credential-provider-config=/etc/eks/image-credential-provider/config.json \
  --root-dir=/var/lib/kubelet \
  --fail-swap-on=false \
  --v=2
Restart=always
RestartSec=5

[Install]
WantedBy=multi-user.target
EOF

创建静态 Pod（使用 ECR 私有镜像）

bash 复制代码

sudo mkdir -p /etc/kubernetes/manifests
sudo tee /etc/kubernetes/manifests/nginx-static.yaml << 'EOF'
apiVersion: v1
kind: Pod
metadata:
  name: nginx-static
  namespace: default
spec:
  containers:
  - name: nginx
    image: 037047667284.dkr.ecr.cn-north-1.amazonaws.com.cn/nginx:latest
    ports:
    - containerPort: 80
      hostPort: 8080
    resources:
      requests:
        memory: "64Mi"
        cpu: "100m"
      limits:
        memory: "128Mi"
        cpu: "250m"
EOF

启动 kubelet

bash 复制代码

sudo systemctl daemon-reload
sudo systemctl enable --now kubelet

验证

bash 复制代码

# 检查容器是否运行
sudo ctr -n k8s.io containers list

CONTAINER                IMAGE                                                          RUNTIME
25d2cc3e...    registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.9    io.containerd.runc.v2
2bce332f...    037047667284.dkr.ecr.cn-north-1.amazonaws.com.cn/nginx:latest   io.containerd.runc.v2

# 测试 nginx 响应
curl http://localhost:8080
HTTP_CODE:200