目录
[1. 环境](#1. 环境)
[2. Python代码实现获取文件](#2. Python代码实现获取文件)
[6.ArgoCD 完成前端项目应用发布](#6.ArgoCD 完成前端项目应用发布)
[2. K8S master节点运行Python代码报错](#2. K8S master节点运行Python代码报错)
一、实验
1. 环境
(1)主机
表1 主机
|---------|--------------|----------|-----------------------|------------------------------------------------------------------------------------------------|
| 主机 | 架构 | 版本 | IP | 备注 |
| master1 | K8S master节点 | 1.20.6 | 192.168.204.180 | jenkins slave (从节点) |
| | argocd | 2.9.3 | 192.168.204.180:31767 | |
| node1 | K8S node节点 | 1.20.6 | 192.168.204.181 | |
| node2 | K8S node节点 | 1.20.6 | 192.168.204.182 | |
| jenkins | jenkins主节点 | 2.414.2 | 192.168.204.15:8080 | gitlab runner (从节点) |
| | harbor私有仓库 | 1.2.2 | 192.168.204.15 | |
| | python | 2.7.5 | | 系统自带 |
| gitlab | gitlab 主节点 | 12.10.14 | 192.168.204.8:82 | jenkins slave (从节点) |
| | sonarqube | 9.6 | 192.168.204.8:9000 | |
2. Python代码实现获取文件
(1) GitLab官网查询通过API操作获取raw文件
bash
curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/13083/repository/files/app%2Fmodels%2Fkey%2Erb/raw?ref=main"
(2)GitLab 创建TOKEN
(3)GitLab查看项目编号
(4)Postman测试完成,转换成Python代码
python
import requests
url = "http://192.168.204.8:82/api/v4/projects/22/repository/files/deployment.yaml/raw?ref=master"
payload = {}
headers = {
'PRIVATE-TOKEN': 'Z6RKxDgK7ort7i9K6f6p'
}
response = requests.request("GET", url, headers=headers, data=payload)
print(response.text)
(3)python实现获取GitLab指定仓库deplyment.yaml文件
python
import requests
import json
class GitlabUtil():
def __init__(self):
self.gitlab_url = "http://192.168.204.8:82/api/v4"
self.headers = {'PRIVATE-TOKEN': 'Z6RKxDgK7ort7i9K6f6p'}
def http_req(self,method,apiUrl,data={}):
url = "{0}/{1}".format(self.gitlab_url,apiUrl)
response = requests.request(method,url,headers=self.headers,data=data)
return response.text
def write_file(self,content,filePath):
with open(filePath,'w') as f:
f.write(content)
def get_repo_file(self,projectId,filePath,branch,targetFile):
apiurl = "projects/{0}/repository/files/{1}/raw?ref={2}".format(projectId,filePath,branch)
response = self.http_req("GET",apiurl)
# print(response.txt)
self.write_file(response,targetFile)
if __name__ == '__main__':
runner = GitlabUtil()
runner.get_repo_file("22","deployment.yaml","master","deployment.yaml")
(4)运行Python代码(Windows11 Python环境为3.8)
(5)本地生成deployment.yaml
(6)K8S master节点同样运行Python代码进行测试(Python环境为2.7)
bash
1)创建目录及python文件
# mkdir pygitlabtest
# cd pygitlabtest/
# vim test.py
# ls
2)查看版本
# python --version
3)运行代码
# python test.py
3.Python代码实现创建文件
(1)GitLab官网查询通过API操作在指定仓库创建文件
bash
curl --request POST --header 'PRIVATE-TOKEN: <your_access_token>' \
--header "Content-Type: application/json" \
--data '{"branch": "main", "author_email": "author@example.com", "author_name": "Firstname Lastname",
"content": "some content", "commit_message": "create a new file"}' \
"https://gitlab.example.com/api/v4/projects/13083/repository/files/app%2Fproject%2Erb"
(2)Postman测试完成,转换成Python代码
bash
import requests
import json
url = "http://192.168.204.8:82/api/v4/projects/22/repository/files/demo.yaml"
payload = json.dumps({
"branch": "master",
"content": "Hello World",
"commit_message": "commmit by autorobot"
})
headers = {
'PRIVATE-TOKEN': 'Z6RKxDgK7ort7i9K6f6p',
'Content-Type': 'application/json'
}
response = requests.request("POST", url, headers=headers, data=payload)
print(response.text)
(3)GitLab查看项目生成了demo.yaml文件
(4)python实现在GitLab指定项目创建demo02.yaml文件
python
# -*- coding: utf-8 -*-
import requests
import json
import base64
class GitlabUtil():
def __init__(self,projectId):
self.gitlab_url = "http://192.168.204.8:82/api/v4"
self.gitlab_token = 'Z6RKxDgK7ort7i9K6f6p'
self.projectId = projectId
self.encoding = "base64"
def http_req(self,method,apiUrl,headers,data):
url = "{0}/{1}".format(self.gitlab_url,apiUrl)
response = requests.request(method,url,headers=headers,data=data)
return response.text
def write_file(self,content,filePath):
with open(filePath,'w') as f:
f.write(content)
def get_repo_file(self,filePath,branch,targetFile):
apiurl = "projects/{0}/repository/files/{1}/raw?ref={2}".format(self.projectId,filePath,branch)
headers = {
'PRIVATE-TOKEN': self.gitlab_token,
'Content-Type': 'application/json'
}
response = self.http_req("GET",apiurl,headers, {})
# print(response.txt)
self.write_file(response,targetFile)
def create_repo_file(self,filePath,branch,content,commit_message):
apiurl = "/projects/{0}/repository/files/{1}".format(self.projectId,filePath)
data = json.dumps({
"branch": branch,
"content": content,
"commit_message": commit_message
# "encoding": self.encoding
})
headers = {
'PRIVATE-TOKEN': self.gitlab_token,
'Content-Type': 'application/json'
}
self.http_req("POST", apiurl,headers=headers,data=data)
if __name__ == '__main__':
runner = GitlabUtil("22")
# runner.get_repo_file("deployment.yaml","master","deployment.yaml")
f = open("deployment.yaml",'r',encoding='utf-8')
content = f.read()
f.close()
# content = base64.b64encode(bytes(content,"utf-8"))
runner.create_repo_file("demo02.yaml","master",content,"Hello World 2")
(5)运行Python代码(Windows11 Python环境为3.8)
(6)GitLab查看项目生成了demo2.yaml文件
4.Python代码实现更新文件
(1)GitLab官网查询通过API操作在指定仓库更新文件
bash
curl --request PUT --header 'PRIVATE-TOKEN: <your_access_token>' \
--header "Content-Type: application/json" \
--data '{"branch": "main", "author_email": "author@example.com", "author_name": "Firstname Lastname",
"content": "some content", "commit_message": "update file"}' \
"https://gitlab.example.com/api/v4/projects/13083/repository/files/app%2Fproject%2Erb"
(2)python实现在GitLab指定项目更新demo02.yaml文件
python
# -*- coding: utf-8 -*-
import requests
import json
import base64
class GitlabUtil():
def __init__(self,projectId):
self.gitlab_url = "http://192.168.204.8:82/api/v4"
self.gitlab_token = 'Z6RKxDgK7ort7i9K6f6p'
self.projectId = projectId
self.encoding = "base64"
def http_req(self,method,apiUrl,headers,data):
url = "{0}/{1}".format(self.gitlab_url,apiUrl)
response = requests.request(method,url,headers=headers,data=data)
return response.text
def write_file(self,content,filePath):
with open(filePath,'w') as f:
f.write(content)
# 下载文件
def get_repo_file(self,filePath,branch,targetFile):
apiurl = "projects/{0}/repository/files/{1}/raw?ref={2}".format(self.projectId,filePath,branch)
headers = {
'PRIVATE-TOKEN': self.gitlab_token,
'Content-Type': 'application/json'
}
response = self.http_req("GET",apiurl,headers, {})
# print(response.txt)
self.write_file(response,targetFile)
# 创建文件
def create_repo_file(self,filePath,branch,content,commit_message):
apiurl = "/projects/{0}/repository/files/{1}".format(self.projectId,filePath)
data = json.dumps({
"branch": branch,
"content": content,
"commit_message": commit_message
# "encoding": self.encoding
})
headers = {
'PRIVATE-TOKEN': self.gitlab_token,
'Content-Type': 'application/json'
}
self.http_req("POST", apiurl,headers=headers,data=data)
# 更新文件
def update_repo_file(self,filePath,branch,content,commit_message):
apiurl = "/projects/{0}/repository/files/{1}".format(self.projectId,filePath)
data = json.dumps({
"branch": branch,
"content": content,
"commit_message": commit_message
# "encoding": self.encoding
})
headers = {
'PRIVATE-TOKEN': self.gitlab_token,
'Content-Type': 'application/json'
}
self.http_req("PUT", apiurl,headers=headers,data=data)
if __name__ == '__main__':
runner = GitlabUtil("22")
# runner.get_repo_file("deployment.yaml","master","deployment.yaml")
f = open("deployment.yaml",'r',encoding='utf-8')
content = f.read()
f.close()
# content = base64.b64encode(bytes(content,"utf-8"))
# runner.create_repo_file("demo02.yaml","master",content,"Hello World 2")
runner.update_repo_file("demo02.yaml", "master", content, "Hello World 3")
(3)运行Python代码(Windows11 Python环境为3.8)
(4)GitLab查看项目更新了提交信息
5.GitLab更新库文件与运行流水线
(1)查看GitLab共享库
(2)新建流水线文件ui.gitlabutil.yaml
(3)复制raw格式
(4)在GitLab devops03-devops-env 环境库项目添加CI配置文件路径
(5)查看前端项目devops03-devops-ui 修改Dockerfile,注释本地CI流水线文件,避免后续运行错误
(6)查看前端项目目录下的index.html文件
(7) GitLab共享库新建目录util及GitLabUtil.py文件,用来封装python类
(8)GitLabUtil.py文件代码
bash
import requests
import json
import base64
import sys
class GitlabUtil():
def __init__(self,projectId):
self.gitlab_url = "http://192.168.204.8:82/api/v4"
self.gitlab_token = 'Z6RKxDgK7ort7i9K6f6p'
self.projectId = projectId
self.encoding = "base64"
def http_req(self,method,apiUrl,headers,data):
url = "{0}/{1}".format(self.gitlab_url,apiUrl)
response = requests.request(method,url,headers=headers,data=data)
return response.text
def write_file(self,content,filePath):
with open(filePath,'w') as f:
f.write(content)
def get_repo_file(self,filePath,branch,targetFile):
apiurl = "projects/{0}/repository/files/{1}/raw?ref={2}".format(self.projectId,filePath,branch)
headers = {
'PRIVATE-TOKEN': self.gitlab_token,
'Content-Type': 'application/json'
}
response = self.http_req("GET",apiurl,headers, {})
# print(response.txt)
self.write_file(response,targetFile)
def create_repo_file(self,filePath,branch,content,commit_message):
apiurl = "/projects/{0}/repository/files/{1}".format(self.projectId,filePath)
data = json.dumps({
"branch": branch,
"content": content,
"commit_message": commit_message
# "encoding": self.encoding
})
headers = {
'PRIVATE-TOKEN': self.gitlab_token,
'Content-Type': 'application/json'
}
mes = self.http_req("POST", apiurl,headers=headers,data=data)
if json.loads(mes)["message"] == "A file with this name already exists":
raise Exception("A file with this name already exists")
def update_repo_file(self,filePath,branch,content,commit_message):
apiurl = "/projects/{0}/repository/files/{1}".format(self.projectId,filePath)
data = json.dumps({
"branch": branch,
"content": content,
"commit_message": commit_message
})
headers = {
'PRIVATE-TOKEN': self.gitlab_token,
'Content-Type': 'application/json'
}
self.http_req("PUT", apiurl,headers=headers,data=data)
if __name__ == '__main__':
if sys.argv[1] == "getfile":
projectId,filename,branch,targetFile = sys.argv[2:]
GitlabUtil(projectId).get_repo_file(filename,branch,targetFile)
if sys.argv[1] == "updatefile":
projectId, filename, branch, targetFile = sys.argv[2:]
f = open(filename, 'r')
content = f.read()
f.close()
try:
GitlabUtil(projectId).create_repo_file(targetFile, branch, content, "Auto K8S Deployment")
except Exception as e:
print(e)
GitlabUtil(projectId).update_repo_file(targetFile, branch, content, "Auto K8S Deployment")
(9)修改流水线文件ui.gitlabutil.yaml
bash
include:
- project: 'devops03/devops03-gitlabci-lib'
ref: master
file: "/jobs/CI.yaml"
workflow:
rules:
#新建分支永远不执行
- if: $CI_PIPELINE_SOURCE == "web" #允许手动触发
when: always
- if: $CI_COMMIT_BEFORE_SHA == "0000000000000000000000000000000000000000"
when: never
# 其他情况永远执行
- when: always
#取消每个阶段自动下载代码,即全局关闭作业代码下载
variables:
GIT_CHECKOUT: "false" ## 全局关闭作业代码下载
PROJECT_TYPE: "npm" ## 定义项目类型
BUILD_SHELL: "npm run build" ## 构建命令
TEST_SHELL: "echo test" ## 测试命令
ARTIFACT_PATH: "dist/**" ## 制品路径
# TEST_REPORTS: "target/surefire-reports/TEST-*.xml" ## 测试报告
stages:
- build
- sonarscan
- dockerbuild
- releasefile
pipelineInit:
extends:
- .pipelineInit
cibuild:
before_script:
- "npm install"
extends:
- .cibuild
releasefile:
tags:
- build
stage: releasefile
script:
- curl "http://192.168.204.8:82/devops03/devops03-gitlabci-lib/-/raw/master/utils/GitLabUtil.py" -o GitLabUtil.py -s
- python GitLabUtil.py getfile "22" "deployment.yaml" "master" "deployment.yaml"
- ls -l
- imageName=192.168.204.15/${CI_PROJECT_NAMESPACE}/${CI_PROJECT_NAME}:${CI_COMMIT_SHA}
- sed -i 's#__PORT__#80#g' deployment.yaml
- sed -i "s#__APPNAME__#${CI_PROJECT_NAME}#g" deployment.yaml
- sed -i "s#__NAMESPACE__#${CI_PROJECT_NAMESPACE}#g" deployment.yaml
- sed -i "s#__IMAGENAME__#${imageName}#g" deployment.yaml
- python GitLabUtil.py updatefile "22" "deployment.yaml" "master" "${CI_PROJECT_NAME}%2f${CI_COMMIT_BRANCH}.yaml"
#sonarscan:
# extends:
# - .sonarscan
#pushartifact:
# extends:
# - .pushartifact
dockerbuild:
extends:
- .dockerbuild
(10)共享库完整目录
(11)GitLab 前端项目运行流水线
(12)完成
(13)查看各阶段日志
(14)环境库项目显示更新master.yaml文件
(15) 查看master.yaml文件(已同步更新端口号、命名空间、项目名称及镜像名称)
6.ArgoCD 完成前端项目应用发布
(1)K8S查看集群状态
bash
# kubectl get node
(2)K8S master节点另开一个终端用watch命令观察pod变化
# watch -n 1 "kubectl get pod -n devops03"
(3)外部测试访问
# curl http://devops03-devops-ui.devops.com:31291
(4)K8S 删除命名空间devops03
bash
# kubectl delete ns devops03
(5) 观察pod变化
(5)ArgoCD 查看已有项目
(6)ArgoCD 删除已有项目
(7)GitLab环境库注释其他的yaml文件
(8)ArgoCD 创建application (手动策略、自动创建命名空间、指定仓库与集群)
(9)填写配置
(10)完成创建
(11)点击 SYNCHRONIZE (同步)
(12)观察pod变化
(13)ArgoCD 观察pod变化
(14)查看Harbor仓库镜像文件
(15)K8S node节点连接Harbor拉取镜像
bash
# docker login -u admin -p Harbor12345 192.168.204.15
# docker pull 192.168.204.15/devops03/devops03-devops-ui:RELEASE-1.1.7
# docker logout 192.168.204.15
node1 节点
node2节点
(16)观察pod变化
(17)ArgoCD 再次观察pod变化
(18) 外部测试访问
# curl http://devops03-devops-ui.devops.com:31291
二、问题
1.Python获取GitLab指定仓库文件报错
(1)报错
(2)原因分析
函数名错误
(3)解决方法
修改函数名称。
修改前:
修改后:
2. K8S master节点运行Python代码报错
(1)报错
(2)原因分析
encoding不是有效的关键词。
(3)解决方法
去掉encoding。
修改前:
修改后:
成功:
3. GitLabCI 运行流水线报错
(1)报错
(2)原因分析
行尾缺少双引号
(3)解决方法
添加双引号。
成功: