GitLabCI/CD语法

01-Pipeline核心语法

stages 阶段控制

  • .pre阶段的作业总是在流水线开始时执行;
  • .post阶段的作业总是在流水线结束时执行;

CI代码:

bash 复制代码
stages:
  - build
  - test
  - deploy 

job0:
  tags:
    - go
  stage: .pre
  script:
    - echo " init"

job1:
  tags:
    - go
  stage: build
  script:
    - echo "build"

job2-1:
  tags:
    - mvn
  stage: test
  script:
    - echo "test"
    - sleep 10

job2-2:
  tags:
    - mvn
  stage: test
  script:
    - echo "test"

job3:
  tags:
    - go
  stage: deploy
  script:
    - echo "deploy"

job10:
  tags:
    - go
  stage: .post
  script:
    - echo "end"

如果两个或者多个作业,指向同一个阶段名称,则该阶段下的所有作业都并行运行;如果不能并行运行,需要检查runner的配置文件中的concurrent值, 要大于1。

variables 环境变量

变量可以分为全局变量和局部变量;全局变量是整个流水线可以用的,局部变量是仅在作业中生效的;

bash 复制代码
stages:
  - build

variables:
  BUILD_TOOLS: 
    value: "mvn"
    description: "choice build tools"   // 描述信息
  RUNNER_TAG: "go"


job1:
  tags:
    - "${RUNNER_TAG}"
  stage: build
  variables:
    BUILD_TOOLS: "gradle"
  script:
    - echo "${BUILD_TOOLS}"

outputlog:

bash 复制代码
Running with gitlab-runner 15.3.0 (bbcb5aba)
on build_project AJSs2JiV
Preparing the "shell" executor
00:00
Using Shell executor...
Preparing environment
00:00
Running on zeyang-nuc-service...
Getting source from Git repository
00:00
Fetching changes with git depth set to 20...
Reinitialized existing Git repository in /home/gitlab-runner/builds/AJSs2JiV/0/devops/devops05-app-service/.git/
Checking out b563129a as main...
Skipping Git submodules setup
Executing "step_script" stage of the job script
00:00
$ echo "${BUILD_TOOLS}"
gradle
Job succeeded

job 作业默认配置

定义一个作业的时候,一般定义哪些关键字呢? 作业在哪个runner运行? 作业属于流水线中的哪个阶段? 这个作业要做什么?

bash 复制代码
stages:
  - build

variables:
  RUNNER_TAG: "go"

before_script:
  - echo "pipeline before script"
after_script:
  - echo "pipeline after script"

job1:
  tags:
    - ${RUNNER_TAG}
  stage: build
  before_script:
    - echo "before script...."
  script:
    - echo "mvn package"
  after_script:
    - echo "after script..."

job2:
  tags:
    - ${RUNNER_TAG}
  stage: build
  script: 
    - echo "build"

参数解析:

|---------------|------------------------|
| 语法关键字 | 作用 |
| variables | 定义作业中的环境变量; |
| tags | 根据标签选择运行作业的构建节点; |
| stage | 指定当前作业所属的阶段名称; |
| before_script | 作业在运行前执行的Shell命令行; |
| script | 作业在运行中执行的Shell命令行; |
| after_script | 作业在运行后执行的Shell命令行; |

job 作业运行控制

|---------------|-------------------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| 语法关键字 | 作用 | 备注 |
| allow_failure | 控制作业状态,是否允许作业失败,默认值为false 。启用后,如果作业运行失败,该作业将在用户界面中显示橙色警告。 | * 管道将认为作业成功/通过,不会被阻塞。 * 假设所有其他作业均成功,则该作业的阶段及其管道将显示相同的橙色警告。但是,关联的提交将被标记为"通过",而不会发出警告。 |
| when | 根据状态控制作业运行, 当前面作业成功或者失败时运行。 | * on_success 前面阶段成功时执行(默认值); * on_failure 前面阶段失败时执行; * always 总是执行; * manual 手动执行; * delayed 延迟执行; * * start_in * * * '5' * 5 seconds * 30 minutes * 1 day * 1 week * never 永不执行; |
| retry | 作业重新运行,遇到错误重新运行的次数。 | * 值为整数 * 等于或大于0,但小于或等于2 异常分类 |
| timeout | 作业运行超时时间; | |
| rules | 根据特定的变量或文件变更来控制作业运行; | * if * changes * exists |
| needs | 作业依赖控制; | needs: ["作业名称"] |
| parllel | 生成多个作业,并行运行 | parallel:5 * 值 2-50之间 |

parallel 并行运行

bash 复制代码
stages:
  - build

variables:
  RUNNER_TAG: "go"

job1:
  tags:
    - ${RUNNER_TAG}
  stage: build
  parallel: 5
  script:
    - echo "mvn package"

needs 作业关联运行

  • needs 指定要依赖的作业名称
bash 复制代码
stages:
  - build
  - test

variables:
  RUNNER_TAG: "go"

job1:
  tags:
    - ${RUNNER_TAG}
  stage: build
  script:
    - echo "mvn package"
    - sleep 10

job2:
  tags:
    - ${RUNNER_TAG}
  stage: build
  script: 
    - echo "build"


job3:
  tags:
    - ${RUNNER_TAG}
  stage: test
  script:
    - echo "mvn package"

job4:
  tags:
    - ${RUNNER_TAG}
  stage: test
  needs: ["job2"]
  script:
    - echo "mvn package"

rules根据变量/文件控制

citest1 config key may not be used with `rules`: when.

根据条件(变量)判断: IF

|----|--------------------------------------------------|
| if | 定义变量条件; 运算符: * = * != * =~ 条件链接符: * && * || |

根据文件判断: changes exists

ci代码文件:

bash 复制代码
stages:
  - build
  - test

variables:
  RUNNER_TAG: "go"
  SKIP_BUILD: "true"


job1:
  tags:
    - ${RUNNER_TAG}
  stage: build
  rules:
    - if: '$SKIP_BUILD == "true"'   # 此时job1 不会被运行
      when: never
    - when: always
  before_script:
    - echo "before script...."
  script:
    - echo "mvn package"
  after_script:
    - echo "after script..."


job2:
  tags:
    - ${RUNNER_TAG}
  stage: test
  rules:
    - changes:
      - README.md     # 提交时仅该文件存在变更时才会触发
      when: always
    - when: never
  script:
    - echo "test"

job3:
  tags:
    - ${RUNNER_TAG}
  stage: test
  rules:
    - exists:
        - Dockerfile  #只要项目中存在该文件就会运行,提交存在也算。
      when: manual
    - when: never
  script:
    - echo "test"

job4:
  tags:
    - ${RUNNER_TAG}
  stage: test
  script:
    - echo "test"

when 状态控制和运行方式

  • 根据上游作业的状态决定
    • 当前作业是否运行?
    • 运行的方式?(手动/自动/定时)
bash 复制代码
stages:
  - build
  - test
  - deploy

variables:
  RUNNER_TAG: "go"

job1:
  tags:
    - ${RUNNER_TAG}
  stage: build
  before_script:
    - echo "before script...."
  script:
    - echo "mvn package"
  after_script:
    - echo "after script..."



job2:
  tags:
    - ${RUNNER_TAG}
  stage: test
  when: on_success
  script: 
    - echo "build"


job3:
  tags:
    - ${RUNNER_TAG}
  stage: test
  when: on_failure
  script: 
    - echo "build"

job4:
  tags:
    - ${RUNNER_TAG}
  stage: deploy
  when: manual
  script: 
    - echo "build"

timeout作业运行超时时间

bash 复制代码
build:
  script: build.sh
  timeout: 3 hours 30 minutes

test:
  script: rspec
  timeout: 3h 30m

retry 作业失败后重试次数

bash 复制代码
stages:
  - build
  - test

variables:
  RUNNER_TAG: "go"

job1:
  tags:
    - ${RUNNER_TAG}
  stage: build
  retry: 2
  before_script:
    - echo "before script...."
  script:
    - echo "mvn package"
    - mvs
  after_script:
    - echo "after script..."

job2:
  tags:
    - ${RUNNER_TAG}
  stage: test
  script: 
    - echo "build"

根据特定的错误匹配:

bash 复制代码
always :在发生任何故障时重试(默认)。
unknown_failure :当失败原因未知时。
script_failure :脚本失败时重试。
api_failure :API失败重试。
stuck_or_timeout_failure :作业卡住或超时时。
runner_system_failure :构建节点的系统发生故障。
missing_dependency_failure: 依赖丢失。
runner_unsupported :Runner不受支持。
stale_schedule :无法执行延迟的作业。
job_execution_timeout:作业运行超时。
archived_failure :作业已存档且无法运行。
unmet_prerequisites :作业未能完成先决条件任务。
scheduler_failure :调度失败。
data_integrity_failure :结构完整性问题。


####
max :最大重试次数  when :重试失败的错误类型

stages:
  - build
  - test

variables:
  RUNNER_TAG: "go"

job1:
  tags:
    - ${RUNNER_TAG}
  stage: build
  retry: 2  # 不管任何错误,都重试2次
  before_script:
    - echo "before script...."
  script:
    - echo "mvn package"
    - mvs  # 命令错误
  after_script:
    - echo "after script..."

job2:
  tags:
    - ${RUNNER_TAG}
  stage: test
  when: on_failure  # 为了让这个作业运行所以添加的,不然前面作业失败这个作业就不运行了。
  script: 
    - echo "build"
    - aaa   # 命令错误
  retry:
    max: 2
    #when: api_failure
    when: script_failure   # 定义脚本错误:  正常会retry两次

allow_failure 允许作业失败

bash 复制代码
stages:
  - build
  - test

variables:
  RUNNER_TAG: "go"

job1:
  tags:
    - ${RUNNER_TAG}
  stage: build
  before_script:
    - echo "before script...."
  script:
    - echo "mvn package"
    - mvs
  after_script:
    - echo "after script..."


job2:
  tags:
    - ${RUNNER_TAG}
  stage: test
  script: 
    - echo "build"

02-Pipeline运行控制

workflow 控制流水线

控制管道是否创建和运行。根据条件(变量)判断: IF

  • if 定义变量条件;
  • variables 重新定义变量的值;
  • when
    • always
    • never

|----------------------------------------------------|-------------|
| if: 'CI_PIPELINE_SOURCE == "merge_request_event"' | 合并请求时运行流水线; | | if: 'CI_PIPELINE_SOURCE == "push"' | 提交代码运行流水线; |

bash 复制代码
workflow:
  rules:
    - if: $CI_PIPELINE_SOURCE == "push"
      when: never

ci代码

bash 复制代码
variables:
  SKIP_RUN: "true"
  RUNNER_TAG: "go"

workflow:
  rules:
    - if: '$CI_COMMIT_BRANCH == "main"'
      when: always
    - if: '$SKIP_RUN == "true"'
      when: never
    - when: always

stages:
  - build

job2:
  tags:
    - ${RUNNER_TAG}
  stage: build
  script: 
    - echo "build"

Git 选项跳过流水线

  • 提交信息中添加关键字 [ci skip] 或者 [skip ci]
  • Git 2.10 更高版本,可以通过以下配置设置CI/CD;
bash 复制代码
## 跳过
git push -o ci.skip

## 传递变量
git push -o ci.variable="MAX_RETRIES=10" -o ci.variable="MAX_TIME=600"

trigger 触发下游管道

  • 触发项目管道
  • 触发子管道

strategy: 默认情况下,一旦创建了下游管道,trigger作业就会以success状态完成。要强制等待下游管道完成,使用 strategy: depend

触发项目管道:

bash 复制代码
triggers:
  stage: deploy
  trigger:
    project: devops/devops-maven-service
    branch: main
    strategy: depend    ## 状态同步

触发子管道:

bash 复制代码
stages:
  - deploy

trigger-cd-pipeline:   
  stage: deploy
  trigger:
    include: ci/stages.yml  ## 触发当前项目的子流水线


trigger-project-pipeline:
  stage: deploy
  trigger:
    include:
    - project: "devops/devops05-app-service"   ## 项目名称
      ref: "main"
      file: "ci/stages.yml"

API触发Pipeline

创建一个认证token

使用API触发

使用curl命令进行测试

bash 复制代码
curl -X POST \
     -F token=b2ba95b229f5fd0eb834454bc9aad8 \
     -F ref=main \
     http://192.168.1.200/api/v4/projects/31/trigger/pipeline

使用postman测试

gitlabci作业中触发

bash 复制代码
script:
  - "curl -X POST -F token=TOKEN -F ref=REF_NAME http://192.168.1.200/api/v4/projects/31/trigger/pipeline"

优化: 将token以变量的方式存储到项目中。

bash 复制代码
stages:         
  - build

build-job: 
  tags:
    - build     
  stage: build
  script:
    - "curl -X POST -F token=${CITOKEN} -F ref=main  http://192.168.1.200/api/v4/projects/29/trigger/pipeline"
    - echo "Compile complete."
    - sleep 1

触发并传递参数

bash 复制代码
curl -X POST \
     -F token=TOKEN \
     -F "ref=main" \
     -F "variables[BUILD_TOOL]=mavenandmaven" \
     http://192.168.1.200/api/v4/projects/31/trigger/pipeline
bash 复制代码
ciTriggerTest:
  tags:
    - build
  stage: build
  script:
    - echo ${BUILD_TOOL}

03-PipelineTemplate实践

​​​​​​https://gitlab.com/gitlab-org/gitlab/-/tree/master/lib/gitlab/ci/templates

extends

extends可以指定多个模板, 相同的参数最后模板会覆盖前面的。例如: 下面两个模板中,继承关系是:

  • 继承 .test1 (此时rspec的变量NAME的值为gitlab)
  • 继承 .test2 (此时rspec的变量NAME的值为gitlabCI , 覆盖了.test1中的值)
bash 复制代码
.test1:
  variables:
    NAME: "gitlab"
  tags:
    - build
  stage: test
  rules:
    - if: $CI_COMMIT_BRANCH == "main"
  script: echo "mvn test"

.test2:
  variables:
    NAME: "gitlabCI"
  tags:
    - build01
  stage: test
  
rspec:
  extends: 
    - .test1
    - .test2
  script: echo " DevOps"

      
      
      
###### 结果

rspec:
  variables:
    NAME: "gitlabCI"
  tags:
    - build01
  stage: test
  rules:
    - if: $CI_COMMIT_BRANCH == "main"
  script: echo " DevOps"

include

include用于在CI/CD 配置中引入外部 YAML 文件。可以将一个长.gitlab-ci.yml文件分解为多个文件以提高可读性,或减少同一配置在多个地方的重复。

  • local 导入当前仓库中的文件;
  • file 导入当前项目或其他项目库中的文件;
  • remote 导入一个远程的文件(例如:http://xxx.yaml
  • template 导入GitLab官方提供的模板文件;
bash 复制代码
## 本地仓库文件
include:
  - local: '/templates/.gitlab-ci-java.yml'
 
## 其他仓库文件
include:
  - project: 'devops/my-project'
    ref: main
    file: 
      - '/templates/.gitlab-ci-java.yml'
      - '/templates/.tests.yml'
    
## 远程文件
include:
  - remote: 'https://192.168.1.200//-/raw/main/.gitlab-ci.yml'

综合实例:

将模板文件,放到单独的一个仓库中;

bash 复制代码
.build:
  tags:
    - build
  stage: build
  variables:
    BUILD_TOOL: "maven3"
    BUILD_SHELL: "mvn clean package"
  script:
    - "${BUILD_SHELL}"

.test:
  tags:
    - "linux"
  stage: test
  variables:
    TEST_SHELL: "mvn test"
  script:
    - "${TEST_SHELL}"

.gitlab-ci.yml

bash 复制代码
workflow:
  rules:
    - if: $CI_COMMIT_BEFORE_SHA == "0000000000000000000000000000000000000000"  # 40个零 创建分支或者tag
      when: never

include:
  - project: 'devops/devops-ci-lib'   ## 引入ci.yml模板
    ref: main
    file: '/CI/ci.yml'

variables:
  GIT_CHECKOUT: "false"
  RUNNER_TAG: "mvn"
  BUILD_SHELL: "mvn clean package -DskipTests"
  TEST_SHELL: "mvn test"

stages:
  - build 
  - test

# 下载代码
checkout:
  tags:
    - ${RUNNER_TAG}
  stage: .pre 
  variables:
    GIT_CHECKOUT: "true"
  script:
    - echo "GetCode..."

# maven构建
build:
  tags:
    - ${RUNNER_TAG}
  extends: .build  ##重用build作业

# maven test
test:
  tags:
    - ${RUNNER_TAG}
  extends: .test

04-提交阶段流水线

因为GitLabCI与GitLab版本控制系统深度集成,所以不需要配置触发器。

  • 默认已经支持了提交代码、合并代码触发流水线的运行;
  • 默认流水线运行后会自动的下载本项目代码;

05-流水线优化

过滤新建分支和tag的触发

bash 复制代码
workflow:
  rules:
    - if: $CI_COMMIT_BEFORE_SHA == "0000000000000000000000000000000000000000"
      when: never

解决job运行重新下载代码

通过下面的CI作业,观察第二个作业是否可以拿到第一个作业的文件;

bash 复制代码
variables:
  ENV_TYPE: "dev"

cibuild:
  tags:
    - build
  stage: build
  script:
    - ls -l 
    - echo 123 >test.yaml 
    - ls -l 

citest1:
  tags:
    - build
  stage: test
  script:
    - ls -l 

分析: citest1作业并没有获取上个阶段产出文件, 因为默认每个作业运行时都会重新下载代码;重新下载代码会把其他文件删除掉;

如何解决?

  • 禁止后面的作业重复的下载代码;
  • 使用artifact关键字, 收集制品传递给其他作业;

GIT_CHECKOUT

GIT_CHECKOUT变量,默认值为true,即作业每次运行都下载代码。按照我们目前的需求,需要禁止下载代码:

我们将此变量的值在全局配置为false,然后在第一个作业(.pre)中配置为true。也就实现了只在第一个作业下载代码而不会出现其他作业下载代码了。

bash 复制代码
GIT_CHECKOUT: "false" 

优化一下pipeline, 仅在pipelineInit作业中运行代码下载,其他作业跳过;

bash 复制代码
variables:
  ENV_TYPE: "dev"
  GIT_CHECKOUT: "false" 

pipelineInit:
  tags:
    - build
  stage: .pre
  variables:
    GIT_CHECKOUT: "true" 
  script:
    - ls -l 

cibuild:
  tags:
    - build
  stage: build
  script:
    - ls -l 
    - echo 123 >test.yaml 
    - ls -l 

citest1:
  tags:
    - build
  stage: test
  script:
    - ls -l 

查看citest1 作业中的输出, 发现工作目录中已经存在之前作业的产出文件;

Artifacts

在作业结束时,收集作业中的文件或目录并以附件的形式保存在关联的作业中。作业运行完成后,制品将被存储到GitLab,并且可以在GitLab UI中下载。

bash 复制代码
  artifacts:
    name: "$CI_JOB_NAME-$CI_COMMIT_REF_NAME"
    when: on_success
    expire_in: '1 week'
    paths:
      - target/*.jar
  • name : 定义所创建的制品存档的名称;
  • when :定义何时进行制品收集;
  • expire_in : 制品的过期时间,过期自动清理;
  • paths: 定义要收集的制品文件或者目录信息;
bash 复制代码
  dependencies:
    - build   ## 作业名称

dependencies:

要获取哪些作业制品, 作业列表;只能是当前阶段之前的作业。如果空数组则跳过下载任何工件;不考虑先前作业的状态,因此,如果它失败或是未运行的手动作业,则不会发生错误。

基于artifacts和dependencies调整pipeline:

bash 复制代码
variables:
  ENV_TYPE: "dev"
  #GIT_CHECKOUT: "false" 

pipelineInit:
  tags:
    - build
  stage: .pre
  #variables:
    #GIT_CHECKOUT: "true" 
  script:
    - ls -l 

cibuild:
  tags:
    - build
  stage: build
  script:
    - ls -l 
    - echo 123 >test.yaml 
    - ls -l 
  artifacts:
    name: "$CI_JOB_NAME-$CI_COMMIT_REF_NAME"
    when: on_success
    expire_in: '1 week'
    paths:
      - test.yaml

citest1:
  tags:
    - build
  stage: test
  dependencies:
    - cibuild
  script:
    - ls -l 

查看结果:citest1作业会下载cibuild作业生成的制品;

邮件通知反馈

编辑/etc/gitlab/gitlab.rb文件开启gitlab email。这里以QQ邮箱为例

bash 复制代码
### GitLab email server settings
###! Docs: https://docs.gitlab.com/omnibus/settings/smtp.html
###! **Use smtp instead of sendmail/postfix.**

gitlab_rails['smtp_enable'] = true
gitlab_rails['smtp_address'] = "smtp.qq.com"
gitlab_rails['smtp_port'] = 465
gitlab_rails['smtp_user_name'] = "2560350642@qq.com"
gitlab_rails['smtp_password'] = "exnyva"
gitlab_rails['smtp_domain'] = "smtp.qq.com"
gitlab_rails['smtp_authentication'] = "login"
gitlab_rails['smtp_enable_starttls_auto'] = true
gitlab_rails['smtp_tls'] = true

### Email Settings
gitlab_rails['gitlab_email_enabled'] = true
gitlab_rails['gitlab_email_from'] = '2560350642@qq.com'
gitlab_rails['gitlab_email_display_name'] = 'GitLab Admin'

重新配置

bash 复制代码
gitlab-ctl stop ;
gitlab-ctl reconfigure ;
gitlab-ctl start gitlab-ctl status 

登录gitlab-rails控制台,发送测试邮件。

bash 复制代码
su - git
gitlab-rails console

irb(main):002:0> Notify.test_email('2560350642@qq.com', 'test email', 'gitlab email test').deliver_now


Notify#test_email: processed outbound mail in 0.5ms
Delivered mail 5eba1b04de4e5_12903fe2ca0c79b0519ec@gitlab-995f97976-2nmb4.mail (1055.9ms)
Date: Tue, 12 May 2020 03:41:56 +0000
From: GitLab Admin <2560350642@qq.com>
Reply-To: GitLab Admin <noreply@192.168.1.200>
To: 2560350642@qq.com
Message-ID: <5eba1b04de4e5_12903fe2ca0c79b0519ec@gitlab-995f97976-2nmb4.mail>
Subject: Message Subject
Mime-Version: 1.0
Content-Type: text/html;
 charset=UTF-8
Content-Transfer-Encoding: 7bit
Auto-Submitted: auto-generated
X-Auto-Response-Suppress: All

<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN" "http://www.w3.org/TR/REC-html40/loose.dtd">
<html><body><p>Message Body And Linuxea.com</p></body></html>

=> #<Mail::Message:70243016426420, Multipart: false, Headers: <Date: Tue, 12 May 2020 03:41:56 +0000>, <From: GitLab Admin <2560350642@qq.com>>, <Reply-To: GitLab Admin <noreply@192.168.1.200>>, <To: 2560350642@qq.com>, <Message-ID: <5eba1b04de4e5_12903fe2ca0c79b0519ec@gitlab-995f97976-2nmb4.mail>>, <Subject: Message Subject>, <Mime-Version: 1.0>, <Content-Type: text/html; charset=UTF-8>, <Content-Transfer-Encoding: 7bit>, <Auto-Submitted: auto-generated>, <X-Auto-Response-Suppress: All>>

测试邮件:

配置流水线状态通知

相关推荐
若风的雨3 小时前
WC (Write-Combining) 内存类型优化原理
linux
YMWM_3 小时前
不同局域网下登录ubuntu主机
linux·运维·ubuntu
黑屋里的马3 小时前
GitExtension下载、安装
git·gitextension
zmjjdank1ng3 小时前
restart与reload的区别
linux·运维
哼?~3 小时前
进程替换与自主Shell
linux
FIT2CLOUD飞致云3 小时前
赛道第一!1Panel成功入选Gitee 2025年度开源项目
服务器·ai·开源·1panel
Geoking.3 小时前
Git 中的 Rebase 与 Merge:原理、区别与最佳实践
git
C澒3 小时前
面单打印服务的监控检查事项
前端·后端·安全·运维开发·交通物流
Bruk.Liu3 小时前
Gitea Actions 的概念及基础使用
运维·ci/cd·持续集成
yanlou2333 小时前
[C++/Linux HTTP项目] HTTP服务器基于muduo高性能服务器搭载【深入详解】
运维·服务器·http·muduo库·http高性能服务器