Jenkins Pipeline脚本优化：为Kubernetes应用部署增加状态检测

saynaihe2023-12-24 22:32

引言

在软件部署的世界中，Jenkins已经成为自动化流程的代名词。不断变化的技术环境要求我们持续改进部署流程以满足现代应用部署的需要。在本篇博客中，作为一位资深运维工程师，我将分享如何将Jenkins Pipeline进化至不仅能支持部署应用直至Running状态检测，同时也能兼顾Deployment和StatefulSet资源的轮询更新，并详细介绍滚动更新策略的配置方法。

初始Jenkins Pipeline分析

参照前文：Jenkins Pipeline 脚本优化实践：从繁琐到简洁，初始化pipeline如下：

bash 复制代码 
pipeline {
    agent none // Use none at the top level, each stage will define its own agent.

    environment {
        REGISTRY = "swr.cn-north-4.myhuaweicloud.com/master-metaspace"
        KUBE_CONFIG = "--namespace=master-metaspace --context=master"
        KUBE_YAML_PATH = "/home/jenkins/workspace/yaml/master-metaspace"
        // Assume that 'data' is defined elsewhere or injected as a parameter.
        BASE_WORKSPACE = "xxxxxxx" // 定义一个基础工作空间路径
    }

    stages {
        stage("GetCode") {
            agent { label "build01" }
            steps {
                script {
                    checkout scm: [
                        $class: 'GitSCM',
                        branches: [[name: env.branchName]],
                        extensions: [[$class: 'CloneOption', depth: 1, noTags: false, shallow: true]],
                        userRemoteConfigs: [[credentialsId: 'xxxx', url: env.gitHttpURL]]
                    ]
                }
            }
        }
        
        stage("Docker Builds") {
            parallel {
                stage('Build dataloader-game-ucenter') {
                    agent { label "build01" }
                    when { environment name: 'dataloader', value: 'true' }
                    steps {
                        buildAndPushDockerImage("dataloader-game-ucenter", env.data, env.BASE_WORKSPACE)
                    }
                }
                stage('Build datawriter-game-ucenter') {
                    agent { label "build01" }
                    when { environment name: 'datawriter', value: 'true' }
                    steps {
                        buildAndPushDockerImage("datawriter-game-ucenter", env.data, env.BASE_WORKSPACE)
                    }
                }
                stage('Build game-ucenter') {
                    agent { label "build01" }
                    when { environment name: 'game-ucenter', value: 'true' }
                    steps {
                        buildAndPushDockerImage("game-ucenter", env.data, env.BASE_WORKSPACE)
                    }
                }
            }
        }
        
        stage('Development Deployment') {
            parallel {
                stage("Deploy datawriter-game-ucenter") {
                    when { environment name: 'datawriter-game-ucenter', value: 'true' }
                    agent { label  "huaweiyun-xx" }
                    steps {
                        deployToKubernetes("datawriter-game-ucenter")
                    }
                }
                stage("Deploy dataloader-game-ucenter") {
                    when { environment name: 'dataloader', value: 'true' }
                    agent { label  "huaweiyun-xx" }
                    steps {
                        deployToKubernetes("dataloader-game-ucenter")
                    }
                }
                stage("Deploy game-ucenter") {
                    when { environment name: 'game-ucenter', value: 'true' }
                    agent { label  "huaweiyun-xx" }
                    steps {
                        deployToKubernetes("game-ucenter")
                    }
                }
            }
        }
    }
}

// Define methods outside pipeline to avoid repetition

def buildAndPushDockerImage(String imageName, String tag, String workspacePath) {
    sh "cd ${workspacePath} && echo 'Current directory: \$(pwd)'" // 使用基础工作空间变量
    sh "cd ${workspacePath}/${imageName}&& docker build --build-arg NODE_ENV=$imageName -t $REGISTRY/$imageName:$tag ."
    withCredentials([usernamePassword(credentialsId: 'hw-registry', passwordVariable: 'dockerPassword', usernameVariable: 'dockerUser')]) {
        sh "docker login -u $dockerUser -p $dockerPassword $REGISTRY"
        sh "docker push $REGISTRY/$imageName:$tag"
    }
}

def deployToKubernetes(String kubernetesComponent) {
    String templateFile = "${KUBE_YAML_PATH}/${kubernetesComponent}.tpl"
    String outputFile = "${KUBE_YAML_PATH}/${kubernetesComponent}.yaml"
    sh "sed -e 's/{data}/$data/g' $templateFile > $outputFile"
    sh "sudo kubectl apply -f $outputFile $KUBE_CONFIG"
}

初始的Jenkins Pipeline定义了一个基本的CI/CD流程，涵盖了代码拉取、Docker镜像构建、推送及在Kubernetes环境中的部署。然而，流程中缺少了对部署状态的检查，这是在确保部署稳定性方面至关重要的一个环节。

进化 I：探针引入Deployment部署

现代应用部署不仅仅需要一个"部署到Kubernetes"的指令，更需要在部署后进行健康检查。对于Deployment类型的应用来说，我们需要在所有Pods运行并处于READY状态后才认为部署成功。

状态检测方法介绍

为此，我们引入了checkKubernetesResourceStatus方法来检查资源的状态。该方法通过kubectl的get命令和jsonpath查询输出来轮询检查ready副本数。如果指定时间内资源不达状态，则流程失败。

Jenkinsfile变更详解：

引入checkKubernetesResourceStatus方法来检测deployment各个阶段部署的状态。

bash 复制代码 
def checkKubernetesResourceStatus(String deploymentName, String namespace) {
    int attempts = 30 // Set the number of retry attempts
    int sleepTime = 10 // Set the sleep time between attempts in seconds
    String readyReplicasJsonPath = ".status.readyReplicas"
    for (int i = 1; i <= attempts; i++) {
        // Check the deployment status
        String statusCheck = sh (
            script: "kubectl get deployment ${deploymentName} --namespace=${namespace} -o jsonpath=\"{${readyReplicasJsonPath}}\"",
            returnStdout: true
        ).trim()

        // If the number of ready replicas is not empty and greater than 0
        if (statusCheck && statusCheck.isInteger() && statusCheck.toInteger() > 0) {
            echo "Deployment ${deploymentName} is ready."
            return
        } else {
            echo "Waiting for Deployment ${deploymentName} to be ready. Attempt ${i}/${attempts}"
            sleep sleepTime
        }
    }
    error "Deployment ${deploymentName} did not become ready after ${attempts} attempts"
}

以Deploy game-ucenter stage为例：

bash 复制代码 
                stage("Deploy game-ucenter") {
                    when { environment name: 'game-ucenter', value: 'true' }
                    agent { label  "xxxx" }
                    steps {
                        deployToKubernetes("game-ucenter")
                        checkKubernetesResourceStatus("game-ucenter", "master-metaspace")
                    }
                }

game-ucenter模板文件如下：

bash 复制代码 
apiVersion: apps/v1
kind: Deployment
metadata:
  name: game-ucenter
spec:
  replicas: 1
  selector:
    matchLabels:
      app: game-ucenter
  template:
    metadata:
      labels:
        app: game-ucenter
    spec:
      containers:
        - name: game-ucenter
          image: xxxxxx/xxxx/game-ucenter:{data}
          envFrom:
          - configMapRef:
              name: deploy
          ports:
            - containerPort: 80
          resources:
            requests:
              memory: "4096M"
              cpu: "2000m"
            limits:
              memory: "4096M"
              cpu: "2000m" 
          livenessProbe:
            httpGet:
              scheme: HTTP
              path: /test.html
              port: 80
            initialDelaySeconds: 20
            periodSeconds: 120
            successThreshold: 1
            failureThreshold: 3
          readinessProbe:
            httpGet:
              scheme: HTTP
              path: /test.html
              port: 80
            initialDelaySeconds: 20
            periodSeconds: 120
      imagePullSecrets:                                              
        - name: xxxx
---

apiVersion: v1
kind: Service
metadata:
  name: game-ucenter
  labels:
    app: game-ucenter
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    app: game-ucenter

尝试修改以下**replicas: 3 **重新运行以下pipeline：

也没有问题，pipeline 脚本有效！

进化 II：兼容StatefulSet的健康检查

考虑到某些应用可能采用StatefulSet作为工作负载类型，我们必须确保Jenkins Pipeline能够针对不同的工作负载类型执行健康检查。

状态检测兼容性改进

为了适配StatefulSet，我们对checkKubernetesResourceStatus方法做了略微修改，使其可以接受一个resourceType参数来区分资源类型，进而查询对应的状态字段，代码片段如下：

bash 复制代码 
def checkKubernetesResourceStatus(String resourceName, String namespace, String resourceType) {
    int attempts = 30 // Set the number of retry attempts
    int sleepTime = 10 // Set the sleep time between attempts in seconds
    String readyReplicasJsonPath = resourceType == "deployment" ? ".status.readyReplicas" : ".status.readyReplicas"
    for (int i = 1; i <= attempts; i++) {
        // Check the resource status
        String statusCheck = sh (
            script: "kubectl get ${resourceType} ${resourceName} --namespace=${namespace} -o jsonpath=\"{${readyReplicasJsonPath}}\"",
            returnStdout: true
        ).trim()

        // If the number of ready replicas is not empty and equal to the desired number
        if (statusCheck && statusCheck.isInteger() && statusCheck.toInteger() > 0) {
            echo "${resourceType} ${resourceName} is ready."
            return
        } else {
            echo "Waiting for ${resourceType} ${resourceName} to be ready. Attempt ${i}/${attempts}"
            sleep(sleepTime)
        }
    }
    error "${resourceType} ${resourceName} did not become ready after ${attempts} attempts"
}

修改game-ucenter stage:

bash 复制代码 
                stage("Deploy game-ucenter") {
                    when { environment name: 'game-ucenter', value: 'true' }
                    agent { label  "k8s-node-06" }
                    steps {
                        deployToKubernetes("game-ucenter")
                        checkKubernetesResourceStatus("game-ucenter", "master-metaspace", "deployment")
                    }
                }

创建一个statefulset datawriter-game-ucenter stage:

bash 复制代码 
                stage("Deploy datawriter-game-ucenter") {
                    when { environment name: 'datawriter-game-ucenter', value: 'true' }
                    agent { label  "xxxxx" }
                    steps {
                        deployToKubernetes("datawriter-game-ucenter")
                        checkKubernetesResourceStatus("datawriter-game-ucenter", "master-metaspace", "statefulset")
                    }
                }


注意 :我这里截图还是用了game-ucenter 做的测试，其实我想用我的datawriter-game-ucenter ，but这个服务是一个node 应用没有没有livenessProbe readinessProbe ，所以截图我还是使用了game-ucenter!

进化 III：引入滚动更新策略配置和检测

当我们更新Deployment资源时，通常会采用滚动更新策略，以逐步替换旧Pods，最小化部署时的中断。

更新策略检测逻辑

bash 复制代码 
def checkDeploymentUpdateStatus(String deploymentName, String namespace) {
    int attempts = 30 // Set the number of retry attempts
    int sleepTime = 10 // Set the sleep time between attempts in seconds
    echo "Checking the update status of Deployment: ${deploymentName}"

    for (int i = 1; i <= attempts; i++) {
        String updateStatus = sh (
            script: "kubectl rollout status deployment/${deploymentName} --namespace=${namespace}",
            returnStdout: true
        ).trim()

        if (updateStatus.contains("successfully rolled out")) {
            echo "Update status: ${updateStatus}"
            return
        } else {
            echo "Waiting for Deployment ${deploymentName} to successfully roll out. Attempt ${i}/${attempts}"
            sleep(sleepTime)
        }
    }

    error "Deployment ${deploymentName} did not successfully roll out after ${attempts} attempts"
}
  1. 我们增加了checkDeploymentUpdateStatus方法，该方法通过kubectl命令rollout status监控Deployment的更新状态。
  2. 当检测到successfully rolled out时，表示滚动更新成功。
  3. 如果在给定时间内更新没有成功，则流程将失败。

继续考虑一下如果statefulset多实例呢？不想写两个了整合成一个方法如下：

bash 复制代码 
def checkRolloutStatus(String resourceName, String namespace, String resourceType) {
    int attempts = 30 // Set the number of retry attempts
    int sleepTime = 10 // Set the sleep time between attempts in seconds

    if (!(resourceType in ["deployment", "statefulset"])) {
        error "Unknown resource type: ${resourceType}. Only 'deployment' and 'statefulset' are supported."
    }

    echo "Checking the update status of ${resourceType} '${resourceName}' in namespace '${namespace}'"

    for (int i = 1; i <= attempts; i++) {
        String rolloutCommand = "kubectl rollout status ${resourceType}/${resourceName} --namespace=${namespace}"
        String updateStatus = sh (
            script: rolloutCommand,
            returnStdout: true
        ).trim()

        if (updateStatus.contains("successfully rolled out") || updateStatus.contains("partitioned roll out complete"))  {
            echo "Update status: ${updateStatus}"
            return
        } else {
            echo "Waiting for ${resourceType} '${resourceName}' to successfully roll out. Attempt ${i}/${attempts}."
            sleep(sleepTime)
        }
    }

    error "${resourceType} '${resourceName}' did not successfully roll out after ${attempts} attempts in namespace '${namespace}'"
}

Jenkinsfile更新实现

经过上述进化，Jenkinsfile中现在包含了完整的部署状态检查逻辑，以应对不同类型资源的部署监控需求。

groovy 复制代码 
pipeline {
    agent none // Use none at the top level, each stage will define its own agent.

    environment {
        REGISTRY = "ccr.ccs.tencentyun.com/xxxxx"
        KUBE_CONFIG = "--namespace=master-metaspace"
        KUBE_YAML_PATH = "/home/jenkins/workspace/yaml/master-metaspace"
        // Assume that 'data' is defined elsewhere or injected as a parameter.
        BASE_WORKSPACE = "xxxxxx" // 定义一个基础工作空间路径
    }

    stages {
        stage("GetCode") {
            agent { label "build01" }
            steps {
                script {
                    checkout scm: [
                        $class: 'GitSCM',
                        branches: [[name: env.branchName]],
                        extensions: [[$class: 'CloneOption', depth: 1, noTags: false, shallow: true]],
                        userRemoteConfigs: [[credentialsId: 'xxxxx', url: env.gitHttpURL]]
                    ]
                }
            }
        }
        
        stage("Docker Builds") {
            parallel {
                stage('Build dataloader-game-ucenter') {
                    agent { label "build01" }
                    when { environment name: 'dataloader-game-ucenter', value: 'true' }
                    steps {
                        buildAndPushDockerImage("dataloader-game-ucenter", env.data, env.BASE_WORKSPACE)
                    }
                }
                stage('Build datawriter-game-ucenter') {
                    agent { label "build01" }
                    when { environment name: 'datawriter-game-ucenter', value: 'true' }
                    steps {
                        buildAndPushDockerImage("datawriter-game-ucenter", env.data, env.BASE_WORKSPACE)
                    }
                }
                stage('Build game-ucenter') {
                    agent { label "build01" }
                    when { environment name: 'game-ucenter', value: 'true' }
                    steps {
                        buildAndPushDockerImage("game-ucenter", env.data, env.BASE_WORKSPACE)
                    }
                }
            }
        }
        
        stage('Development Deployment') {
            parallel {
                stage("Deploy datawriter-game-ucenter") {
                    when { environment name: 'datawriter-game-ucenter', value: 'true' }
                    agent { label  "xxxx" }
                    steps {
                        deployToKubernetes("datawriter-game-ucenter")
                        checkKubernetesResourceStatus("datawriter-game-ucenter", "master-metaspace", "statefulset")
                        checkRolloutStatus("datawriter-game-ucenter", "master-metaspace", "statefulset")
                    }
                }
                stage("Deploy dataloader-game-ucenter") {
                    when { environment name: 'dataloader-game-ucenter', value: 'true' }
                    agent { label  "xxxx" }
                    steps {
                        deployToKubernetes("dataloader-game-ucenter")
                        checkKubernetesResourceStatus("dataloader-game-ucenter", "master-metaspace", "statefulset")
                    }
                }
                stage("Deploy game-ucenter") {
                    when { environment name: 'game-ucenter', value: 'true' }
                    agent { label  "xxxx" }
                    steps {
                        deployToKubernetes("game-ucenter")
                        checkRolloutStatus("game-ucenter", "master-metaspace", "deployment")
                        checkKubernetesResourceStatus("game-ucenter", "master-metaspace", "deployment")
                    }
                }
            }
        }
    }
}

// Define methods outside pipeline to avoid repetition

def buildAndPushDockerImage(String imageName, String tag, String workspacePath) {
    sh "cd ${workspacePath} && echo 'Current directory: \$(pwd)'" // 使用基础工作空间变量
    sh "cd ${workspacePath}/${imageName}&& docker build --build-arg NODE_ENV=game-ucenter -t $REGISTRY/$imageName:$tag ."
    withCredentials([usernamePassword(credentialsId: 'xxxxx', passwordVariable: 'dockerPassword', usernameVariable: 'dockerUser')]) {
        sh "docker login -u $dockerUser -p $dockerPassword $REGISTRY"
        sh "docker push $REGISTRY/$imageName:$tag"
    }
}

def deployToKubernetes(String kubernetesComponent) {
    String templateFile = "${KUBE_YAML_PATH}/${kubernetesComponent}.tpl"
    String outputFile = "${KUBE_YAML_PATH}/${kubernetesComponent}.yaml"
    sh "sed -e 's/{data}/$data/g' $templateFile > $outputFile"
    sh "sudo kubectl apply -f $outputFile $KUBE_CONFIG"
}

def checkRolloutStatus(String resourceName, String namespace, String resourceType) {
    int attempts = 30 // 设置重试次数
    int sleepTime = 10 // 设置重试间隔时间（秒）

    if (!(resourceType in ["deployment", "statefulset"])) {
        error "未知资源类型：${resourceType}。只支持 'deployment' 和 'statefulset' 。"
    }

    echo "正在检查${resourceType} '${resourceName}' 在命名空间 '${namespace}' 的更新状态"

    for (int i = 1; i <= attempts; i++) {
        String rolloutCommand = "kubectl rollout status ${resourceType}/${resourceName} --namespace=${namespace}"
        try {
            String updateStatus = sh (
                script: rolloutCommand,
                returnStdout: true
            ).trim()

            // 添加对 "partitioned roll out complete" 状态的检查
            if (updateStatus.contains("successfully rolled out") || updateStatus.contains("partitioned roll out complete")) {
                echo "更新状态：${updateStatus}"
                return
            } else {
                echo "等待 ${resourceType} '${resourceName}' 成功发布。尝试次数：${i}/${attempts}。"
                sleep(sleepTime)
            }
        } catch (Exception e) {
            echo "获取更新状态时发生错误：${e.getMessage()}。尝试次数：${i}/${attempts}。"
            sleep(sleepTime)
        }
    }

    error "${resourceType} '${resourceName}' 在命名空间 '${namespace}' 内未能在 ${attempts} 次尝试之后成功发布"
}
def checkKubernetesResourceStatus(String resourceName, String namespace, String resourceType) {
    int attempts = 30 // Set the number of retry attempts
    int sleepTime = 10 // Set the sleep time between attempts in seconds
    String readyReplicasJsonPath = resourceType == "deployment" ? ".status.readyReplicas" : ".status.readyReplicas"
    for (int i = 1; i <= attempts; i++) {
        // Check the resource status
        String statusCheck = sh (
            script: "kubectl get ${resourceType} ${resourceName} --namespace=${namespace} -o jsonpath=\"{${readyReplicasJsonPath}}\"",
            returnStdout: true
        ).trim()

        // If the number of ready replicas is not empty and equal to the desired number
        if (statusCheck && statusCheck.isInteger() && statusCheck.toInteger() > 0) {
            echo "${resourceType} ${resourceName} is ready."
            return
        } else {
            echo "Waiting for ${resourceType} ${resourceName} to be ready. Attempt ${i}/${attempts}"
            sleep(sleepTime)
        }
    }
    error "${resourceType} ${resourceName} did not become ready after ${attempts} attempts"
}// 更新后的Jenkins Pipeline代码详细定义参照本文开头给出的代码

总结

本篇博客通过对Jenkins Pipeline的进化过程展开讲解，展现了如何从简单的部署任务转变为一个健壮且兼顾各类工作负载状态监测的CI/CD流程。我们强化了状态检测的逻辑，引入了更新策略的检测，并保持了对不同Kubernetes资源类型的兼容性。这些改进确保了自动化流程能够与现代部署实践保持同步，给运维团队带来极大便利，并最大化地保障了部署的可靠性。

后记

由于篇幅限制，本篇文章未作其他更详细演示。然而，在实际应用中，运维团队可以根据自己的具体需求和环境进一步丰富和细化每个步骤的实现，确保Pipeline的健壮性和高可用性，以适应不断变化的技术挑战。

相关推荐
老汉忒cpp5 分钟前
手动部署并测试内网穿透（ssh 和 nginx）
运维·服务器
hardStudy_h21 分钟前
Linux——常用系统设置和快捷键操作指令
linux·运维·服务器
空指针异常Null_Point_Ex39 分钟前
大模型LLM之SpringAI：Web+AI（一）
人工智能·chatgpt·nlp
我叫啥都行1 小时前
计算机基础知识复习9.7
运维·服务器·网络·笔记·后端
骅青1 小时前
kubernetes调度2
容器·kubernetes
AIPaPerPass写论文1 小时前
写论文去哪个网站？2024最佳五款AI毕业论文学术网站
人工智能·深度学习·chatgpt·powerpoint·ai写作
qq 1778036222 小时前
智能新时代，游戏盾守护顺畅体验
运维·服务器·网络·游戏·云计算·ddos·ip
Alone80462 小时前
K8s中HPA自动扩缩容及hml
云原生·容器·kubernetes
神秘的土鸡3 小时前
Linux中使用Docker容器构建Tomcat容器完整教程
linux·运维·服务器·docker·容器·tomcat
TravisBytes4 小时前
linux 系统是如何收发数据包
linux·运维·服务器
热门推荐
01RAG 实践- Ollama+RagFlow 部署本地知识库02组基轨迹建模 GBTM的介绍与实现（Stata 或 R）032024年高教社杯数学建模国赛C题超详细解题思路分析04苍穹外卖面试总结05yolov8实战第五天——yolov8+ffmpg实时视频流检测并进行实时推流——（推流，保姆教学）06【2024数模国赛赛题思路公开】国赛B题思路丨附可运行代码丨无偿自提07CCF-CSP认证考试 202406-3 文本分词 100分题解08Coze扣子平台完整体验和实践（附国内和国际版对比）09【2024高教社杯全国大学生数学建模竞赛】B题 生产过程中的决策问题——解题思路 代码 论文10软件工程从理论到实践客观题汇总(头歌第一章至第八章)