shell-operator 配置 K8s Pod 保护策略

保障 Pod 高可用分为以下几个方面

资源池划分：打污点、独占、隔离
Qos 分级 & 优先级：按应用等级保障稳定性
自动弹性伸缩 HPA
Pod 干扰预算 PDB

本篇主要介绍第四点：巧用 Shell-operator 配置 K8s Pod 保护策略

什么是 Shell-operator?

Shell-operator 是一个在 Kubernetes 集群中运行事件驱动脚本的工具。

该操作器不是针对特定软件产品的操作器，例如 prometheus-operator 或 kafka-operator。Shell-operator 通过将脚本视为由事件触发的钩子，提供了 Kubernetes 集群事件和 Shell 脚本之间的集成层。您可以将其视为 operator-sdk，但用于脚本。

Shell-operator 用作更高级的 addon-operator 的基础，该 operator 支持 Helm charts 和值存储。

Shell-operator 提供了以下功能：

Kubernetes 集群的简易管理：使用运维人员熟悉的工具。可以是 bash、python、kubectl 等，真是运维人的实用工具。
Kubernetes 对象事件：钩子可以由添加、更新或删除事件触发。
对象选择器和属性过滤器：shell-operator 可以监视特定的对象集，并检测其属性的更改。
简单的配置：钩子绑定定义是脚本的标准输出中的 JSON 或 YAML 文档。
验证 Webhook 机制：钩子可以处理 Kubernetes 资源的验证。
转换 Webhook 机制：钩子可以处理 Kubernetes 资源的版本转换。

借助 Shell-operator 配置 PDB

配置 PDB 需要用到便签选择器 selector，示例中选择 AppID 作为标签匹配 Shell-operator 需要监控一个对象的事件，来更新 PDB 的状态，示例中选择 rollouts.argoproj.io 对象 namespace 选择 prod

1.创建基础模版 pdb-init.sh

bash 复制代码

#!/usr/bin/env bash

NAMESPACE="prod"

pdb_templates() {
  cat <<EOF
apiVersion: policy/v1beta1
kind: PodDisruptionBudget
metadata:
  name: clay-test
  namespace: ${NAMESPACE}
spec:
  minAvailable: 1
  selector:
    matchLabels:
      appid: clay-test
EOF
}

replace_or_create() {
  object=$(cat)

  if ! kubectl get -f - <<< "$object" >/dev/null 2>/dev/null; then
    kubectl create -f - <<< "$object" >/dev/null
  else
    kubectl replace --force -f - <<< "$object" >/dev/null
  fi
}

init() {
  echo "Init PDB templates"
  pdb_templates | replace_or_create
  for resourceName in $(kubectl -n ${NAMESPACE} get rollouts.argoproj.io | grep default | awk '{print $1}'); do
    appid=${resourceName%-default} 
    echo "Init Add PDB '${appid}'"
    kubectl -n ${NAMESPACE} get pdb clay-test -o json | \
      jq -r ".metadata.name="${appid}" | .spec.selector.matchLabels["appid"]="${appid}" |
              .metadata |= with_entries(select([.key] | inside(["name", "namespace", "labels"])))" \
      | replace_or_create
  done
}

init "$@"

2.编写钩子脚本pdb-hooks.sh

bash 复制代码

#!/usr/bin/env bash  
  
NAMESPACE="prod"  
ARRAY_COUNT=$(jq -r '. | length-1' $BINDING_CONTEXT_PATH)  
  
run_hook() {  
  if [[ $1 == "--config" ]] ; then  
    config  
  else  
    trigger  
  fi  
}  
  
config() {  
  cat <<EOF  
configVersion: v1  
kubernetes:  
- apiVersion: argoproj.io/v1alpha1  
  kind: Rollout  
  executeHookOnEvent:  
  - Added  
  - Deleted  
  namespace:  
    nameSelector:  
      matchNames:  
      - ${NAMESPACE}  
EOF  
}  
  
trigger() {  
  for IND in `seq 0 $ARRAY_COUNT`; do  
    resourceEvent=$(jq -r ".[$IND].watchEvent" $BINDING_CONTEXT_PATH)  
    resourceName=$(jq -r ".[$IND].object.metadata.name" $BINDING_CONTEXT_PATH)  
    appid=${resourceName%-default}  
    if [[ $resourceEvent == "Added" ]] ; then  
      echo "Add PDB '${appid}'"  
      kubectl -n ${NAMESPACE} get pdb clay-test -o json | \  
        jq -r ".metadata.name=\"${appid}\" | .spec.selector.matchLabels[\"appid\"]=\"${appid}\" |  
                .metadata |= with_entries(select([.key] | inside([\"name\", \"namespace\", \"labels\"])))" \  
        | replace_or_create  
    elif [[ $resourceEvent == "Deleted" ]]; then  
      echo "Delete PDB '${appid}'"  
      kubectl -n ${NAMESPACE} delete pdb ${appid}  
    else  
      echo "Do nothing"  
    fi  
  done  
}  
  
replace_or_create() {  
  object=$(cat)  
  
  if ! kubectl get -f - <<< "$object" >/dev/null 2>/dev/null; then  
    kubectl create -f - <<< "$object" >/dev/null  
  else  
    kubectl replace --force -f - <<< "$object" >/dev/null  
  fi  
}  
  
run_hook "$@"

3.打包镜像，Dockerfile 如下

Dockerfile 复制代码

FROM ghcr.io/flant/shell-operator:latest

ADD pdb-hooks.sh /hooks

4.编写 RBAC、Deployment等shell-operator-pdb.yaml，部署到 K8s 集群中

yaml 复制代码

apiVersion: v1  
kind: ServiceAccount  
metadata:  
  name: shell-operator-pdb  
  namespace: monitor  
  labels:  
    app: shell-operator-pdb  
    appid: shell-operator-pdb  
  
  
---  
apiVersion: rbac.authorization.k8s.io/v1  
kind: ClusterRole  
metadata:  
  labels:  
    app: shell-operator-pdb  
    appid: shell-operator-pdb  
  name: shell-operator-pdb  
rules:  
- apiGroups:  
  - ""  
  resources:  
  - pods  
  verbs:  
  - get  
  - watch  
  - list  
- apiGroups:  
  - argoproj.io  
  resources:  
  - '*'  
  verbs:  
  - get  
  - list  
  - watch  
- apiGroups:  
  - policy  
  resources:  
  - poddisruptionbudgets  
  verbs:  
  - get  
  - create  
  - patch  
  - update  
  - delete  
  
  
---  
apiVersion: rbac.authorization.k8s.io/v1  
kind: ClusterRoleBinding  
metadata:  
  name: shell-operator-pdb  
  labels:  
    app: shell-operator-pdb  
    appid: shell-operator-pdb  
roleRef:  
  apiGroup: rbac.authorization.k8s.io  
  kind: ClusterRole  
  name: shell-operator-pdb  
subjects:  
- kind: ServiceAccount  
  name: shell-operator-pdb  
  namespace: monitor  
  
  
---  
apiVersion: apps/v1  
kind: Deployment  
metadata:  
  name: shell-operator-pdb  
  labels:  
    app: shell-operator-pdb  
    appid: shell-operator-pdb  
  namespace: monitor  
spec:  
  replicas: 1  
  selector:  
    matchLabels:  
      app: shell-operator-pdb  
  strategy:  
    rollingUpdate:  
      maxSurge: 25%  
      maxUnavailable: 1  
    type: RollingUpdate  
  template:  
    metadata:  
      labels:  
        app: shell-operator-pdb  
        appid: shell-operator-pdb  
    spec:  
      containers:  
      - image: wangzhichidocker/shell-operator-pdb:v1.0  
        imagePullPolicy: IfNotPresent  
        name: shell-operator-pdb  
        resources:  
          limits:  
            cpu: 500m  
            memory: 1000Mi  
          requests:  
            cpu: 100m  
            memory: 200Mi  
      dnsPolicy: ClusterFirst  
      nodeSelector:  
        kubernetes.io/os: linux  
      serviceAccountName: shell-operator-pdb

bash 复制代码

# 部署
kubectl apply -f deploy/shell-operator-pdb.yaml

快速体验

bash 复制代码

git clone https://github.com/clay-wangzhi/shell-operator-pdb  
bash pdb-init.sh  
kubectl apply -f deploy/shell-operator-pdb.yaml

详见：github.com/clay-wangzh... 参考链接：

shell-operator：github.com/flant/shell...