持续集成交付CICD:GitLabCI 封装Python类 并结合 ArgoCD 完成前端项目应用发布

目录

一、实验

[1. 环境](#1. 环境)

[2. Python代码实现获取文件](#2. Python代码实现获取文件)

3.Python代码实现创建文件

4.Python代码实现更新文件

5.GitLab更新库文件与运行流水线

[6.ArgoCD 完成前端项目应用发布](#6.ArgoCD 完成前端项目应用发布)

二、问题

1.Python获取GitLab指定仓库文件报错

[2. K8S master节点运行Python代码报错](#2. K8S master节点运行Python代码报错)


一、实验

1. 环境

(1)主机

表1 主机

|---------|--------------|----------|-----------------------|------------------------------------------------------------------------------------------------|
| 主机 | 架构 | 版本 | IP | 备注 |
| master1 | K8S master节点 | 1.20.6 | 192.168.204.180 | jenkins slave (从节点) |
| | argocd | 2.9.3 | 192.168.204.180:31767 | |
| node1 | K8S node节点 | 1.20.6 | 192.168.204.181 | |
| node2 | K8S node节点 | 1.20.6 | 192.168.204.182 | |
| jenkins | jenkins主节点 | 2.414.2 | 192.168.204.15:8080 | gitlab runner (从节点) |
| | harbor私有仓库 | 1.2.2 | 192.168.204.15 | |
| | python | 2.7.5 | | 系统自带 |
| gitlab | gitlab 主节点 | 12.10.14 | 192.168.204.8:82 | jenkins slave (从节点) |
| | sonarqube | 9.6 | 192.168.204.8:9000 | |

2. Python代码实现获取文件

(1) GitLab官网查询通过API操作获取raw文件

Repository files API | GitLab

bash 复制代码
curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/13083/repository/files/app%2Fmodels%2Fkey%2Erb/raw?ref=main"

(2)GitLab 创建TOKEN

(3)GitLab查看项目编号

(4)Postman测试完成,转换成Python代码

python 复制代码
import requests

url = "http://192.168.204.8:82/api/v4/projects/22/repository/files/deployment.yaml/raw?ref=master"

payload = {}
headers = {
  'PRIVATE-TOKEN': 'Z6RKxDgK7ort7i9K6f6p'
}

response = requests.request("GET", url, headers=headers, data=payload)

print(response.text)

(3)python实现获取GitLab指定仓库deplyment.yaml文件

python 复制代码
import requests
import json

class GitlabUtil():
    def __init__(self):
        self.gitlab_url = "http://192.168.204.8:82/api/v4"
        self.headers = {'PRIVATE-TOKEN': 'Z6RKxDgK7ort7i9K6f6p'}

    def http_req(self,method,apiUrl,data={}):
        url = "{0}/{1}".format(self.gitlab_url,apiUrl)
        response = requests.request(method,url,headers=self.headers,data=data)
        return response.text

    def write_file(self,content,filePath):
        with open(filePath,'w') as f:
            f.write(content)

    def get_repo_file(self,projectId,filePath,branch,targetFile):
        apiurl = "projects/{0}/repository/files/{1}/raw?ref={2}".format(projectId,filePath,branch)

        response = self.http_req("GET",apiurl)
        # print(response.txt)
        self.write_file(response,targetFile)

if __name__ == '__main__':
    runner = GitlabUtil()
    runner.get_repo_file("22","deployment.yaml","master","deployment.yaml")

(4)运行Python代码(Windows11 Python环境为3.8)

(5)本地生成deployment.yaml

(6)K8S master节点同样运行Python代码进行测试(Python环境为2.7)

bash 复制代码
1)创建目录及python文件
# mkdir pygitlabtest
# cd pygitlabtest/
# vim test.py
# ls

2)查看版本
# python --version

3)运行代码
# python test.py 

3.Python代码实现创建文件

(1)GitLab官网查询通过API操作在指定仓库创建文件

bash 复制代码
curl --request POST --header 'PRIVATE-TOKEN: <your_access_token>' \
     --header "Content-Type: application/json" \
     --data '{"branch": "main", "author_email": "author@example.com", "author_name": "Firstname Lastname",
               "content": "some content", "commit_message": "create a new file"}' \
     "https://gitlab.example.com/api/v4/projects/13083/repository/files/app%2Fproject%2Erb"

(2)Postman测试完成,转换成Python代码

bash 复制代码
import requests
import json

url = "http://192.168.204.8:82/api/v4/projects/22/repository/files/demo.yaml"

payload = json.dumps({
  "branch": "master",
  "content": "Hello World",
  "commit_message": "commmit by autorobot"
})
headers = {
  'PRIVATE-TOKEN': 'Z6RKxDgK7ort7i9K6f6p',
  'Content-Type': 'application/json'
}

response = requests.request("POST", url, headers=headers, data=payload)

print(response.text)

(3)GitLab查看项目生成了demo.yaml文件

(4)python实现在GitLab指定项目创建demo02.yaml文件

python 复制代码
# -*- coding: utf-8 -*-
import requests
import json
import base64

class GitlabUtil():
    def __init__(self,projectId):
        self.gitlab_url = "http://192.168.204.8:82/api/v4"
        self.gitlab_token = 'Z6RKxDgK7ort7i9K6f6p'
        self.projectId = projectId
        self.encoding = "base64"

    def http_req(self,method,apiUrl,headers,data):
        url = "{0}/{1}".format(self.gitlab_url,apiUrl)
        response = requests.request(method,url,headers=headers,data=data)
        return response.text

    def write_file(self,content,filePath):
        with open(filePath,'w') as f:
            f.write(content)

    def get_repo_file(self,filePath,branch,targetFile):
        apiurl = "projects/{0}/repository/files/{1}/raw?ref={2}".format(self.projectId,filePath,branch)
        headers = {
            'PRIVATE-TOKEN': self.gitlab_token,
            'Content-Type': 'application/json'
        }
        response = self.http_req("GET",apiurl,headers, {})
        # print(response.txt)
        self.write_file(response,targetFile)

    def create_repo_file(self,filePath,branch,content,commit_message):
        apiurl = "/projects/{0}/repository/files/{1}".format(self.projectId,filePath)

        data = json.dumps({
            "branch": branch,
            "content": content,
            "commit_message": commit_message
            # "encoding": self.encoding
        })

        headers = {
            'PRIVATE-TOKEN': self.gitlab_token,
            'Content-Type': 'application/json'
        }

        self.http_req("POST", apiurl,headers=headers,data=data)


if __name__ == '__main__':
    runner = GitlabUtil("22")
    # runner.get_repo_file("deployment.yaml","master","deployment.yaml")

    f = open("deployment.yaml",'r',encoding='utf-8')
    content = f.read()
    f.close()
    # content = base64.b64encode(bytes(content,"utf-8"))

    runner.create_repo_file("demo02.yaml","master",content,"Hello World 2")

(5)运行Python代码(Windows11 Python环境为3.8)

(6)GitLab查看项目生成了demo2.yaml文件

4.Python代码实现更新文件

(1)GitLab官网查询通过API操作在指定仓库更新文件

bash 复制代码
curl --request PUT --header 'PRIVATE-TOKEN: <your_access_token>' \
     --header "Content-Type: application/json" \
     --data '{"branch": "main", "author_email": "author@example.com", "author_name": "Firstname Lastname",
       "content": "some content", "commit_message": "update file"}' \
     "https://gitlab.example.com/api/v4/projects/13083/repository/files/app%2Fproject%2Erb"

(2)python实现在GitLab指定项目更新demo02.yaml文件

python 复制代码
# -*- coding: utf-8 -*-
import requests
import json
import base64

class GitlabUtil():
    def __init__(self,projectId):
        self.gitlab_url = "http://192.168.204.8:82/api/v4"
        self.gitlab_token = 'Z6RKxDgK7ort7i9K6f6p'
        self.projectId = projectId
        self.encoding = "base64"

    def http_req(self,method,apiUrl,headers,data):
        url = "{0}/{1}".format(self.gitlab_url,apiUrl)
        response = requests.request(method,url,headers=headers,data=data)
        return response.text

    def write_file(self,content,filePath):
        with open(filePath,'w') as f:
            f.write(content)

    # 下载文件
    def get_repo_file(self,filePath,branch,targetFile):
        apiurl = "projects/{0}/repository/files/{1}/raw?ref={2}".format(self.projectId,filePath,branch)
        headers = {
            'PRIVATE-TOKEN': self.gitlab_token,
            'Content-Type': 'application/json'
        }
        response = self.http_req("GET",apiurl,headers, {})
        # print(response.txt)
        self.write_file(response,targetFile)

    # 创建文件
    def create_repo_file(self,filePath,branch,content,commit_message):
        apiurl = "/projects/{0}/repository/files/{1}".format(self.projectId,filePath)

        data = json.dumps({
            "branch": branch,
            "content": content,
            "commit_message": commit_message
            # "encoding": self.encoding
        })

        headers = {
            'PRIVATE-TOKEN': self.gitlab_token,
            'Content-Type': 'application/json'
        }

        self.http_req("POST", apiurl,headers=headers,data=data)

    # 更新文件
    def update_repo_file(self,filePath,branch,content,commit_message):
        apiurl = "/projects/{0}/repository/files/{1}".format(self.projectId,filePath)

        data = json.dumps({
            "branch": branch,
            "content": content,
            "commit_message": commit_message
            # "encoding": self.encoding
        })

        headers = {
            'PRIVATE-TOKEN': self.gitlab_token,
            'Content-Type': 'application/json'
        }

        self.http_req("PUT", apiurl,headers=headers,data=data)


if __name__ == '__main__':
    runner = GitlabUtil("22")
    # runner.get_repo_file("deployment.yaml","master","deployment.yaml")

    f = open("deployment.yaml",'r',encoding='utf-8')
    content = f.read()
    f.close()
    # content = base64.b64encode(bytes(content,"utf-8"))

    # runner.create_repo_file("demo02.yaml","master",content,"Hello World 2")
    runner.update_repo_file("demo02.yaml", "master", content, "Hello World 3")

(3)运行Python代码(Windows11 Python环境为3.8)

(4)GitLab查看项目更新了提交信息

5.GitLab更新库文件与运行流水线

(1)查看GitLab共享库

(2)新建流水线文件ui.gitlabutil.yaml

(3)复制raw格式

(4)在GitLab devops03-devops-env 环境库项目添加CI配置文件路径

(5)查看前端项目devops03-devops-ui 修改Dockerfile,注释本地CI流水线文件,避免后续运行错误

(6)查看前端项目目录下的index.html文件

(7) GitLab共享库新建目录util及GitLabUtil.py文件,用来封装python类

(8)GitLabUtil.py文件代码

bash 复制代码
import requests
import json
import base64
import sys

class GitlabUtil():
    def __init__(self,projectId):
        self.gitlab_url = "http://192.168.204.8:82/api/v4"
        self.gitlab_token = 'Z6RKxDgK7ort7i9K6f6p'
        self.projectId = projectId
        self.encoding = "base64"

    def http_req(self,method,apiUrl,headers,data):
        url = "{0}/{1}".format(self.gitlab_url,apiUrl)
        response = requests.request(method,url,headers=headers,data=data)
        return response.text

    def write_file(self,content,filePath):
        with open(filePath,'w') as f:
            f.write(content)

 
    def get_repo_file(self,filePath,branch,targetFile):
        apiurl = "projects/{0}/repository/files/{1}/raw?ref={2}".format(self.projectId,filePath,branch)
        headers = {
            'PRIVATE-TOKEN': self.gitlab_token,
            'Content-Type': 'application/json'
        }
        response = self.http_req("GET",apiurl,headers, {})
        # print(response.txt)
        self.write_file(response,targetFile)


    def create_repo_file(self,filePath,branch,content,commit_message):
        apiurl = "/projects/{0}/repository/files/{1}".format(self.projectId,filePath)

        data = json.dumps({
            "branch": branch,
            "content": content,
            "commit_message": commit_message
            # "encoding": self.encoding
        })

        headers = {
            'PRIVATE-TOKEN': self.gitlab_token,
            'Content-Type': 'application/json'
        }

        mes = self.http_req("POST", apiurl,headers=headers,data=data)
        if json.loads(mes)["message"] == "A file with this name already exists":
            raise Exception("A file with this name already exists")

  
    def update_repo_file(self,filePath,branch,content,commit_message):
        apiurl = "/projects/{0}/repository/files/{1}".format(self.projectId,filePath)

        data = json.dumps({
            "branch": branch,
            "content": content,
            "commit_message": commit_message

        })

        headers = {
            'PRIVATE-TOKEN': self.gitlab_token,
            'Content-Type': 'application/json'
        }

        self.http_req("PUT", apiurl,headers=headers,data=data)


if __name__ == '__main__':
    if sys.argv[1] == "getfile":
        projectId,filename,branch,targetFile = sys.argv[2:]
        GitlabUtil(projectId).get_repo_file(filename,branch,targetFile)

    if sys.argv[1] == "updatefile":
        projectId, filename, branch, targetFile = sys.argv[2:]
        f = open(filename, 'r')
        content = f.read()
        f.close()
        try:
            GitlabUtil(projectId).create_repo_file(targetFile, branch, content, "Auto K8S Deployment")
        except Exception as e:
            print(e)
            GitlabUtil(projectId).update_repo_file(targetFile, branch, content, "Auto K8S Deployment")

(9)修改流水线文件ui.gitlabutil.yaml

bash 复制代码
include:
 - project: 'devops03/devops03-gitlabci-lib'
   ref: master
   file: "/jobs/CI.yaml"


workflow:
  rules:
    #新建分支永远不执行
    - if: $CI_PIPELINE_SOURCE == "web"      #允许手动触发
      when: always
    - if: $CI_COMMIT_BEFORE_SHA == "0000000000000000000000000000000000000000"
      when: never
    # 其他情况永远执行
    - when: always

#取消每个阶段自动下载代码,即全局关闭作业代码下载
variables:
  GIT_CHECKOUT: "false"  ## 全局关闭作业代码下载
  PROJECT_TYPE: "npm"   ## 定义项目类型
                        
  BUILD_SHELL: "npm run build"   ## 构建命令
  TEST_SHELL: "echo test"                         ## 测试命令
  ARTIFACT_PATH: "dist/**"                                  ## 制品路径
 # TEST_REPORTS: "target/surefire-reports/TEST-*.xml"             ## 测试报告

stages:
  - build
  - sonarscan
  - dockerbuild
  - releasefile


pipelineInit:
  extends: 
    - .pipelineInit


cibuild:
  before_script:
   - "npm install"
  extends:
   - .cibuild

releasefile:
  tags:
    - build
  stage: releasefile
  script:
    - curl  "http://192.168.204.8:82/devops03/devops03-gitlabci-lib/-/raw/master/utils/GitLabUtil.py" -o GitLabUtil.py -s
    - python GitLabUtil.py getfile "22" "deployment.yaml" "master" "deployment.yaml"
    - ls -l
    - imageName=192.168.204.15/${CI_PROJECT_NAMESPACE}/${CI_PROJECT_NAME}:${CI_COMMIT_SHA}
    - sed -i 's#__PORT__#80#g' deployment.yaml
    - sed -i "s#__APPNAME__#${CI_PROJECT_NAME}#g" deployment.yaml
    - sed -i "s#__NAMESPACE__#${CI_PROJECT_NAMESPACE}#g" deployment.yaml
    - sed -i "s#__IMAGENAME__#${imageName}#g" deployment.yaml
    - python GitLabUtil.py updatefile "22" "deployment.yaml" "master" "${CI_PROJECT_NAME}%2f${CI_COMMIT_BRANCH}.yaml"



#sonarscan:
# extends:
# - .sonarscan

#pushartifact:
# extends:
# - .pushartifact

dockerbuild:
  extends:
   - .dockerbuild

(10)共享库完整目录

(11)GitLab 前端项目运行流水线

(12)完成

(13)查看各阶段日志

(14)环境库项目显示更新master.yaml文件

(15) 查看master.yaml文件(已同步更新端口号、命名空间、项目名称及镜像名称)

6.ArgoCD 完成前端项目应用发布

(1)K8S查看集群状态

bash 复制代码
# kubectl get node

(2)K8S master节点另开一个终端用watch命令观察pod变化

# watch -n 1 "kubectl get pod -n devops03"

(3)外部测试访问

# curl http://devops03-devops-ui.devops.com:31291

(4)K8S 删除命名空间devops03

bash 复制代码
# kubectl delete ns devops03

(5) 观察pod变化

(5)ArgoCD 查看已有项目

(6)ArgoCD 删除已有项目

(7)GitLab环境库注释其他的yaml文件

(8)ArgoCD 创建application (手动策略、自动创建命名空间、指定仓库与集群)

(9)填写配置

(10)完成创建

(11)点击 SYNCHRONIZE (同步)

(12)观察pod变化

(13)ArgoCD 观察pod变化

(14)查看Harbor仓库镜像文件

(15)K8S node节点连接Harbor拉取镜像

bash 复制代码
# docker login -u admin -p Harbor12345 192.168.204.15
 
# docker pull 192.168.204.15/devops03/devops03-devops-ui:RELEASE-1.1.7
 
# docker logout 192.168.204.15

node1 节点

node2节点

(16)观察pod变化

(17)ArgoCD 再次观察pod变化

(18) 外部测试访问

# curl http://devops03-devops-ui.devops.com:31291

二、问题

1.Python获取GitLab指定仓库文件报错

(1)报错

(2)原因分析

函数名错误

(3)解决方法

修改函数名称。

修改前:

修改后:

2. K8S master节点运行Python代码报错

(1)报错

(2)原因分析

encoding不是有效的关键词。

(3)解决方法

去掉encoding。

修改前:

修改后:

成功:

3. GitLabCI 运行流水线报错

(1)报错

(2)原因分析

行尾缺少双引号

(3)解决方法

添加双引号。

成功:

相关推荐
A ?Charis1 小时前
Gitlab-runner running on Kubernetes - hostAliases
容器·kubernetes·gitlab
wclass-zhengge1 小时前
Docker篇(Docker Compose)
运维·docker·容器
李启柱1 小时前
项目开发流程规范文档
运维·软件构建·个人开发·设计规范
北漂IT民工_程序员_ZG2 小时前
k8s集群安装(minikube)
云原生·容器·kubernetes
2401_857622663 小时前
SpringBoot健身房管理:敏捷与自动化
spring boot·后端·自动化
力姆泰克3 小时前
看电动缸是如何提高农机的自动化水平
大数据·运维·服务器·数据库·人工智能·自动化·1024程序员节
BPM_宏天低代码3 小时前
低代码 BPA:简化业务流程自动化的新趋势
运维·低代码·自动化
sun0077003 小时前
拷贝 cp -rdp 和 cp -a
linux·运维·服务器
wowocpp3 小时前
ubuntu 22.04 server 安装 anaconda3
linux·运维·ubuntu
tingting01194 小时前
Linux 普通用户禁用sudo su - 命令
linux·运维·服务器