01-Pipeline核心语法


stages 阶段控制
- .pre阶段的作业总是在流水线开始时执行;
- .post阶段的作业总是在流水线结束时执行;
CI代码:
bash
stages:
- build
- test
- deploy
job0:
tags:
- go
stage: .pre
script:
- echo " init"
job1:
tags:
- go
stage: build
script:
- echo "build"
job2-1:
tags:
- mvn
stage: test
script:
- echo "test"
- sleep 10
job2-2:
tags:
- mvn
stage: test
script:
- echo "test"
job3:
tags:
- go
stage: deploy
script:
- echo "deploy"
job10:
tags:
- go
stage: .post
script:
- echo "end"

如果两个或者多个作业,指向同一个阶段名称,则该阶段下的所有作业都并行运行;如果不能并行运行,需要检查runner的配置文件中的concurrent值, 要大于1。
variables 环境变量
变量可以分为全局变量和局部变量;全局变量是整个流水线可以用的,局部变量是仅在作业中生效的;
bash
stages:
- build
variables:
BUILD_TOOLS:
value: "mvn"
description: "choice build tools" // 描述信息
RUNNER_TAG: "go"
job1:
tags:
- "${RUNNER_TAG}"
stage: build
variables:
BUILD_TOOLS: "gradle"
script:
- echo "${BUILD_TOOLS}"
outputlog:
bash
Running with gitlab-runner 15.3.0 (bbcb5aba)
on build_project AJSs2JiV
Preparing the "shell" executor
00:00
Using Shell executor...
Preparing environment
00:00
Running on zeyang-nuc-service...
Getting source from Git repository
00:00
Fetching changes with git depth set to 20...
Reinitialized existing Git repository in /home/gitlab-runner/builds/AJSs2JiV/0/devops/devops05-app-service/.git/
Checking out b563129a as main...
Skipping Git submodules setup
Executing "step_script" stage of the job script
00:00
$ echo "${BUILD_TOOLS}"
gradle
Job succeeded
job 作业默认配置
定义一个作业的时候,一般定义哪些关键字呢? 作业在哪个runner运行? 作业属于流水线中的哪个阶段? 这个作业要做什么?
bash
stages:
- build
variables:
RUNNER_TAG: "go"
before_script:
- echo "pipeline before script"
after_script:
- echo "pipeline after script"
job1:
tags:
- ${RUNNER_TAG}
stage: build
before_script:
- echo "before script...."
script:
- echo "mvn package"
after_script:
- echo "after script..."
job2:
tags:
- ${RUNNER_TAG}
stage: build
script:
- echo "build"
参数解析:
|---------------|------------------------|
| 语法关键字 | 作用 |
| variables | 定义作业中的环境变量; |
| tags | 根据标签选择运行作业的构建节点; |
| stage | 指定当前作业所属的阶段名称; |
| before_script | 作业在运行前执行的Shell命令行; |
| script | 作业在运行中执行的Shell命令行; |
| after_script | 作业在运行后执行的Shell命令行; |
job 作业运行控制
|---------------|-------------------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| 语法关键字 | 作用 | 备注 |
| allow_failure | 控制作业状态,是否允许作业失败,默认值为false 。启用后,如果作业运行失败,该作业将在用户界面中显示橙色警告。
| * 管道将认为作业成功/通过,不会被阻塞。 * 假设所有其他作业均成功,则该作业的阶段及其管道将显示相同的橙色警告。但是,关联的提交将被标记为"通过",而不会发出警告。 |
| when | 根据状态控制作业运行, 当前面作业成功或者失败时运行。 | * on_success 前面阶段成功时执行(默认值); * on_failure 前面阶段失败时执行; * always 总是执行; * manual 手动执行; * delayed 延迟执行; * * start_in * * * '5' * 5 seconds * 30 minutes * 1 day * 1 week * never 永不执行; |
| retry | 作业重新运行,遇到错误重新运行的次数。 | * 值为整数 * 等于或大于0,但小于或等于2 异常分类 |
| timeout | 作业运行超时时间; | |
| rules | 根据特定的变量或文件变更来控制作业运行; | * if * changes * exists |
| needs | 作业依赖控制; | needs: ["作业名称"] |
| parllel | 生成多个作业,并行运行 | parallel:5 * 值 2-50之间 |
parallel 并行运行
bash
stages:
- build
variables:
RUNNER_TAG: "go"
job1:
tags:
- ${RUNNER_TAG}
stage: build
parallel: 5
script:
- echo "mvn package"

needs 作业关联运行
- needs 指定要依赖的作业名称
bash
stages:
- build
- test
variables:
RUNNER_TAG: "go"
job1:
tags:
- ${RUNNER_TAG}
stage: build
script:
- echo "mvn package"
- sleep 10
job2:
tags:
- ${RUNNER_TAG}
stage: build
script:
- echo "build"
job3:
tags:
- ${RUNNER_TAG}
stage: test
script:
- echo "mvn package"
job4:
tags:
- ${RUNNER_TAG}
stage: test
needs: ["job2"]
script:
- echo "mvn package"

rules根据变量/文件控制
citest1 config key may not be used with `rules`: when.
根据条件(变量)判断: IF
|----|--------------------------------------------------|
| if | 定义变量条件; 运算符: * = * != * =~ 条件链接符: * && * || |

根据文件判断: changes exists


ci代码文件:
bash
stages:
- build
- test
variables:
RUNNER_TAG: "go"
SKIP_BUILD: "true"
job1:
tags:
- ${RUNNER_TAG}
stage: build
rules:
- if: '$SKIP_BUILD == "true"' # 此时job1 不会被运行
when: never
- when: always
before_script:
- echo "before script...."
script:
- echo "mvn package"
after_script:
- echo "after script..."
job2:
tags:
- ${RUNNER_TAG}
stage: test
rules:
- changes:
- README.md # 提交时仅该文件存在变更时才会触发
when: always
- when: never
script:
- echo "test"
job3:
tags:
- ${RUNNER_TAG}
stage: test
rules:
- exists:
- Dockerfile #只要项目中存在该文件就会运行,提交存在也算。
when: manual
- when: never
script:
- echo "test"
job4:
tags:
- ${RUNNER_TAG}
stage: test
script:
- echo "test"

when 状态控制和运行方式
- 根据上游作业的状态决定
-
- 当前作业是否运行?
- 运行的方式?(手动/自动/定时)
bash
stages:
- build
- test
- deploy
variables:
RUNNER_TAG: "go"
job1:
tags:
- ${RUNNER_TAG}
stage: build
before_script:
- echo "before script...."
script:
- echo "mvn package"
after_script:
- echo "after script..."
job2:
tags:
- ${RUNNER_TAG}
stage: test
when: on_success
script:
- echo "build"
job3:
tags:
- ${RUNNER_TAG}
stage: test
when: on_failure
script:
- echo "build"
job4:
tags:
- ${RUNNER_TAG}
stage: deploy
when: manual
script:
- echo "build"

timeout作业运行超时时间
bash
build:
script: build.sh
timeout: 3 hours 30 minutes
test:
script: rspec
timeout: 3h 30m
retry 作业失败后重试次数

bash
stages:
- build
- test
variables:
RUNNER_TAG: "go"
job1:
tags:
- ${RUNNER_TAG}
stage: build
retry: 2
before_script:
- echo "before script...."
script:
- echo "mvn package"
- mvs
after_script:
- echo "after script..."
job2:
tags:
- ${RUNNER_TAG}
stage: test
script:
- echo "build"
根据特定的错误匹配:
bash
always :在发生任何故障时重试(默认)。
unknown_failure :当失败原因未知时。
script_failure :脚本失败时重试。
api_failure :API失败重试。
stuck_or_timeout_failure :作业卡住或超时时。
runner_system_failure :构建节点的系统发生故障。
missing_dependency_failure: 依赖丢失。
runner_unsupported :Runner不受支持。
stale_schedule :无法执行延迟的作业。
job_execution_timeout:作业运行超时。
archived_failure :作业已存档且无法运行。
unmet_prerequisites :作业未能完成先决条件任务。
scheduler_failure :调度失败。
data_integrity_failure :结构完整性问题。
####
max :最大重试次数 when :重试失败的错误类型
stages:
- build
- test
variables:
RUNNER_TAG: "go"
job1:
tags:
- ${RUNNER_TAG}
stage: build
retry: 2 # 不管任何错误,都重试2次
before_script:
- echo "before script...."
script:
- echo "mvn package"
- mvs # 命令错误
after_script:
- echo "after script..."
job2:
tags:
- ${RUNNER_TAG}
stage: test
when: on_failure # 为了让这个作业运行所以添加的,不然前面作业失败这个作业就不运行了。
script:
- echo "build"
- aaa # 命令错误
retry:
max: 2
#when: api_failure
when: script_failure # 定义脚本错误: 正常会retry两次

allow_failure 允许作业失败
bash
stages:
- build
- test
variables:
RUNNER_TAG: "go"
job1:
tags:
- ${RUNNER_TAG}
stage: build
before_script:
- echo "before script...."
script:
- echo "mvn package"
- mvs
after_script:
- echo "after script..."
job2:
tags:
- ${RUNNER_TAG}
stage: test
script:
- echo "build"

02-Pipeline运行控制

workflow 控制流水线
控制管道是否创建和运行。根据条件(变量)判断: IF
- if 定义变量条件;
- variables 重新定义变量的值;
- when
-
- always
- never
|----------------------------------------------------|-------------|
| if: 'CI_PIPELINE_SOURCE == "merge_request_event"' | 合并请求时运行流水线; |
| if: 'CI_PIPELINE_SOURCE == "push"' | 提交代码运行流水线; |
bash
workflow:
rules:
- if: $CI_PIPELINE_SOURCE == "push"
when: never
ci代码
bash
variables:
SKIP_RUN: "true"
RUNNER_TAG: "go"
workflow:
rules:
- if: '$CI_COMMIT_BRANCH == "main"'
when: always
- if: '$SKIP_RUN == "true"'
when: never
- when: always
stages:
- build
job2:
tags:
- ${RUNNER_TAG}
stage: build
script:
- echo "build"
Git 选项跳过流水线
- 提交信息中添加关键字
[ci skip]或者[skip ci]; - Git 2.10 更高版本,可以通过以下配置设置CI/CD;
bash
## 跳过
git push -o ci.skip
## 传递变量
git push -o ci.variable="MAX_RETRIES=10" -o ci.variable="MAX_TIME=600"
trigger 触发下游管道
- 触发项目管道
- 触发子管道
strategy: 默认情况下,一旦创建了下游管道,trigger作业就会以success状态完成。要强制等待下游管道完成,使用 strategy: depend。
触发项目管道:
bash
triggers:
stage: deploy
trigger:
project: devops/devops-maven-service
branch: main
strategy: depend ## 状态同步
触发子管道:
bash
stages:
- deploy
trigger-cd-pipeline:
stage: deploy
trigger:
include: ci/stages.yml ## 触发当前项目的子流水线
trigger-project-pipeline:
stage: deploy
trigger:
include:
- project: "devops/devops05-app-service" ## 项目名称
ref: "main"
file: "ci/stages.yml"

API触发Pipeline
创建一个认证token

使用API触发
使用curl命令进行测试
bash
curl -X POST \
-F token=b2ba95b229f5fd0eb834454bc9aad8 \
-F ref=main \
http://192.168.1.200/api/v4/projects/31/trigger/pipeline
使用postman测试

gitlabci作业中触发
bash
script:
- "curl -X POST -F token=TOKEN -F ref=REF_NAME http://192.168.1.200/api/v4/projects/31/trigger/pipeline"
优化: 将token以变量的方式存储到项目中。
bash
stages:
- build
build-job:
tags:
- build
stage: build
script:
- "curl -X POST -F token=${CITOKEN} -F ref=main http://192.168.1.200/api/v4/projects/29/trigger/pipeline"
- echo "Compile complete."
- sleep 1
触发并传递参数
bash
curl -X POST \
-F token=TOKEN \
-F "ref=main" \
-F "variables[BUILD_TOOL]=mavenandmaven" \
http://192.168.1.200/api/v4/projects/31/trigger/pipeline
bash
ciTriggerTest:
tags:
- build
stage: build
script:
- echo ${BUILD_TOOL}

03-PipelineTemplate实践


https://gitlab.com/gitlab-org/gitlab/-/tree/master/lib/gitlab/ci/templates
extends
extends可以指定多个模板, 相同的参数最后模板会覆盖前面的。例如: 下面两个模板中,继承关系是:
- 继承 .test1 (此时rspec的变量
NAME的值为gitlab) - 继承 .test2 (此时rspec的变量
NAME的值为gitlabCI , 覆盖了.test1中的值)
bash
.test1:
variables:
NAME: "gitlab"
tags:
- build
stage: test
rules:
- if: $CI_COMMIT_BRANCH == "main"
script: echo "mvn test"
.test2:
variables:
NAME: "gitlabCI"
tags:
- build01
stage: test
rspec:
extends:
- .test1
- .test2
script: echo " DevOps"
###### 结果
rspec:
variables:
NAME: "gitlabCI"
tags:
- build01
stage: test
rules:
- if: $CI_COMMIT_BRANCH == "main"
script: echo " DevOps"
include
include用于在CI/CD 配置中引入外部 YAML 文件。可以将一个长.gitlab-ci.yml文件分解为多个文件以提高可读性,或减少同一配置在多个地方的重复。
- local 导入当前仓库中的文件;
- file 导入当前项目或其他项目库中的文件;
- remote 导入一个远程的文件(例如:http://xxx.yaml)
- template 导入GitLab官方提供的模板文件;
bash
## 本地仓库文件
include:
- local: '/templates/.gitlab-ci-java.yml'
## 其他仓库文件
include:
- project: 'devops/my-project'
ref: main
file:
- '/templates/.gitlab-ci-java.yml'
- '/templates/.tests.yml'
## 远程文件
include:
- remote: 'https://192.168.1.200//-/raw/main/.gitlab-ci.yml'
综合实例:
将模板文件,放到单独的一个仓库中;
bash
.build:
tags:
- build
stage: build
variables:
BUILD_TOOL: "maven3"
BUILD_SHELL: "mvn clean package"
script:
- "${BUILD_SHELL}"
.test:
tags:
- "linux"
stage: test
variables:
TEST_SHELL: "mvn test"
script:
- "${TEST_SHELL}"
.gitlab-ci.yml
bash
workflow:
rules:
- if: $CI_COMMIT_BEFORE_SHA == "0000000000000000000000000000000000000000" # 40个零 创建分支或者tag
when: never
include:
- project: 'devops/devops-ci-lib' ## 引入ci.yml模板
ref: main
file: '/CI/ci.yml'
variables:
GIT_CHECKOUT: "false"
RUNNER_TAG: "mvn"
BUILD_SHELL: "mvn clean package -DskipTests"
TEST_SHELL: "mvn test"
stages:
- build
- test
# 下载代码
checkout:
tags:
- ${RUNNER_TAG}
stage: .pre
variables:
GIT_CHECKOUT: "true"
script:
- echo "GetCode..."
# maven构建
build:
tags:
- ${RUNNER_TAG}
extends: .build ##重用build作业
# maven test
test:
tags:
- ${RUNNER_TAG}
extends: .test
04-提交阶段流水线

因为GitLabCI与GitLab版本控制系统深度集成,所以不需要配置触发器。
- 默认已经支持了提交代码、合并代码触发流水线的运行;
- 默认流水线运行后会自动的下载本项目代码;
05-流水线优化
过滤新建分支和tag的触发
bash
workflow:
rules:
- if: $CI_COMMIT_BEFORE_SHA == "0000000000000000000000000000000000000000"
when: never
解决job运行重新下载代码
通过下面的CI作业,观察第二个作业是否可以拿到第一个作业的文件;
bash
variables:
ENV_TYPE: "dev"
cibuild:
tags:
- build
stage: build
script:
- ls -l
- echo 123 >test.yaml
- ls -l
citest1:
tags:
- build
stage: test
script:
- ls -l
分析: citest1作业并没有获取上个阶段产出文件, 因为默认每个作业运行时都会重新下载代码;重新下载代码会把其他文件删除掉;

如何解决?
- 禁止后面的作业重复的下载代码;
- 使用artifact关键字, 收集制品传递给其他作业;
GIT_CHECKOUT
GIT_CHECKOUT变量,默认值为true,即作业每次运行都下载代码。按照我们目前的需求,需要禁止下载代码:
我们将此变量的值在全局配置为false,然后在第一个作业(.pre)中配置为true。也就实现了只在第一个作业下载代码而不会出现其他作业下载代码了。
bash
GIT_CHECKOUT: "false"
优化一下pipeline, 仅在pipelineInit作业中运行代码下载,其他作业跳过;
bash
variables:
ENV_TYPE: "dev"
GIT_CHECKOUT: "false"
pipelineInit:
tags:
- build
stage: .pre
variables:
GIT_CHECKOUT: "true"
script:
- ls -l
cibuild:
tags:
- build
stage: build
script:
- ls -l
- echo 123 >test.yaml
- ls -l
citest1:
tags:
- build
stage: test
script:
- ls -l
查看citest1 作业中的输出, 发现工作目录中已经存在之前作业的产出文件;

Artifacts
在作业结束时,收集作业中的文件或目录并以附件的形式保存在关联的作业中。作业运行完成后,制品将被存储到GitLab,并且可以在GitLab UI中下载。
bash
artifacts:
name: "$CI_JOB_NAME-$CI_COMMIT_REF_NAME"
when: on_success
expire_in: '1 week'
paths:
- target/*.jar
- name : 定义所创建的制品存档的名称;
- when :定义何时进行制品收集;
- expire_in : 制品的过期时间,过期自动清理;
- paths: 定义要收集的制品文件或者目录信息;
bash
dependencies:
- build ## 作业名称
dependencies:
要获取哪些作业制品, 作业列表;只能是当前阶段之前的作业。如果空数组则跳过下载任何工件;不考虑先前作业的状态,因此,如果它失败或是未运行的手动作业,则不会发生错误。
基于artifacts和dependencies调整pipeline:
bash
variables:
ENV_TYPE: "dev"
#GIT_CHECKOUT: "false"
pipelineInit:
tags:
- build
stage: .pre
#variables:
#GIT_CHECKOUT: "true"
script:
- ls -l
cibuild:
tags:
- build
stage: build
script:
- ls -l
- echo 123 >test.yaml
- ls -l
artifacts:
name: "$CI_JOB_NAME-$CI_COMMIT_REF_NAME"
when: on_success
expire_in: '1 week'
paths:
- test.yaml
citest1:
tags:
- build
stage: test
dependencies:
- cibuild
script:
- ls -l
查看结果:citest1作业会下载cibuild作业生成的制品;

邮件通知反馈
编辑/etc/gitlab/gitlab.rb文件开启gitlab email。这里以QQ邮箱为例
bash
### GitLab email server settings
###! Docs: https://docs.gitlab.com/omnibus/settings/smtp.html
###! **Use smtp instead of sendmail/postfix.**
gitlab_rails['smtp_enable'] = true
gitlab_rails['smtp_address'] = "smtp.qq.com"
gitlab_rails['smtp_port'] = 465
gitlab_rails['smtp_user_name'] = "2560350642@qq.com"
gitlab_rails['smtp_password'] = "exnyva"
gitlab_rails['smtp_domain'] = "smtp.qq.com"
gitlab_rails['smtp_authentication'] = "login"
gitlab_rails['smtp_enable_starttls_auto'] = true
gitlab_rails['smtp_tls'] = true
### Email Settings
gitlab_rails['gitlab_email_enabled'] = true
gitlab_rails['gitlab_email_from'] = '2560350642@qq.com'
gitlab_rails['gitlab_email_display_name'] = 'GitLab Admin'
重新配置
bash
gitlab-ctl stop ;
gitlab-ctl reconfigure ;
gitlab-ctl start gitlab-ctl status
登录gitlab-rails控制台,发送测试邮件。
bash
su - git
gitlab-rails console
irb(main):002:0> Notify.test_email('2560350642@qq.com', 'test email', 'gitlab email test').deliver_now
Notify#test_email: processed outbound mail in 0.5ms
Delivered mail 5eba1b04de4e5_12903fe2ca0c79b0519ec@gitlab-995f97976-2nmb4.mail (1055.9ms)
Date: Tue, 12 May 2020 03:41:56 +0000
From: GitLab Admin <2560350642@qq.com>
Reply-To: GitLab Admin <noreply@192.168.1.200>
To: 2560350642@qq.com
Message-ID: <5eba1b04de4e5_12903fe2ca0c79b0519ec@gitlab-995f97976-2nmb4.mail>
Subject: Message Subject
Mime-Version: 1.0
Content-Type: text/html;
charset=UTF-8
Content-Transfer-Encoding: 7bit
Auto-Submitted: auto-generated
X-Auto-Response-Suppress: All
<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN" "http://www.w3.org/TR/REC-html40/loose.dtd">
<html><body><p>Message Body And Linuxea.com</p></body></html>
=> #<Mail::Message:70243016426420, Multipart: false, Headers: <Date: Tue, 12 May 2020 03:41:56 +0000>, <From: GitLab Admin <2560350642@qq.com>>, <Reply-To: GitLab Admin <noreply@192.168.1.200>>, <To: 2560350642@qq.com>, <Message-ID: <5eba1b04de4e5_12903fe2ca0c79b0519ec@gitlab-995f97976-2nmb4.mail>>, <Subject: Message Subject>, <Mime-Version: 1.0>, <Content-Type: text/html; charset=UTF-8>, <Content-Transfer-Encoding: 7bit>, <Auto-Submitted: auto-generated>, <X-Auto-Response-Suppress: All>>
测试邮件:

配置流水线状态通知



