超大规模 CI/CD 完整搭建指南（GitLab + K8s Runner + ArgoCD）

适用规模 ：3 万台服务器 | 语言栈 ：Python 50% + Java 50%

目标：复制粘贴即可运行，小白一次性成功，无需反复调试

---

📋 前置条件检查清单

在开始之前，请确认以下环境已就绪：

资源	最低要求	说明
K8s 集群	1 个管理集群（CI 专用）+ N 个业务集群	管理集群建议 50+ 节点，业务集群承载 3 万台服务器
kubectl + Helm 3	本地已安装	helm version ≥ 3.12
域名	3 个二级域名	gitlab.company.com、harbor.company.com、argocd.company.com
StorageClass	已配置	kubectl get sc 能看到可用存储类
GitLab EE License	已获取	企业版许可证文件（.gitlab-license）
服务器访问	可 SSH 到 K8s 控制节点	用于执行所有命令

本文档中所有 company.com 请替换为你的真实域名，所有密码请替换为强密码。

---

Step 0：基础环境初始化（30 分钟）

0.1 创建所有命名空间

kubectl create ns gitlab
kubectl create ns harbor
kubectl create ns argocd
kubectl create ns minio
kubectl create ns gitlab-runner
kubectl create ns cert-manager

0.2 安装 cert-manager（自动 HTTPS 证书）

helm repo add jetstack https://charts.jetstack.io
helm repo update
helm install cert-manager jetstack/cert-manager \
  --namespace cert-manager \
  --set installCRDs=true \
  --wait

0.3 配置证书签发机构

cat << 'EOF' | kubectl apply -f -
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: admin@company.com
    privateKeySecretRef:
      name: letsencrypt-prod
    solvers:
    - http01:
        ingress:
          class: nginx
EOF

✅ 验证 ：kubectl get clusterissuer letsencrypt-prod 显示 Ready=True

---

Step 1：部署 MinIO（S3 兼容存储，用于缓存和 Artifacts）

MinIO 作为 GitLab Runner 的分布式缓存后端，是 3 万台规模下构建提速的关键。

helm repo add minio https://charts.min.io
helm repo update

cat << 'EOF' > minio-values.yaml
replicas: 4
persistence:
  enabled: true
  storageClass: standard
  size: 500Gi
resources:
  requests:
    memory: 2Gi
    cpu: 1000m
buckets:
  - name: gitlab-runner-cache
    policy: none
    purge: false
  - name: gitlab-artifacts
    policy: none
    purge: false
rootUser: minio
rootPassword: MinioStrongPass123!
consoleIngress:
  enabled: true
  ingressClassName: nginx
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-prod
  host: minio-console.company.com
  tls:
    - secretName: minio-console-tls
      hosts:
        - minio-console.company.com
EOF

helm install minio minio/minio \
  --namespace minio \
  -f minio-values.yaml \
  --wait

✅ 验证 ：访问 https://minio-console.company.com，用 minio / MinioStrongPass123! 登录，能看到两个桶。

---

Step 2：部署 Harbor（企业级镜像仓库）

2.1 准备配置

helm repo add harbor https://helm.goharbor.io
helm repo update

cat << 'EOF' > harbor-values.yaml
expose:
  type: ingress
  tls:
    enabled: true
    certSource: secret
    secret:
      secretName: harbor-tls
  ingress:
    hosts:
      core: harbor.company.com
    annotations:
      cert-manager.io/cluster-issuer: letsencrypt-prod
      ingress.kubernetes.io/ssl-redirect: "true"
      nginx.ingress.kubernetes.io/proxy-body-size: "0"

externalURL: https://harbor.company.com

persistence:
  enabled: true
  persistentVolumeClaim:
    registry:
      storageClass: standard
      size: 2Ti
    chartmuseum:
      storageClass: standard
      size: 50Gi
    jobservice:
      storageClass: standard
      size: 20Gi
    database:
      storageClass: standard
      size: 10Gi
    redis:
      storageClass: standard
      size: 5Gi
    trivy:
      storageClass: standard
      size: 10Gi

harborAdminPassword: HarborAdmin123!

# 高可用副本（3万台规模）
portal:
  replicas: 2
core:
  replicas: 2
jobservice:
  replicas: 2
registry:
  replicas: 2
EOF

2.2 部署

helm install harbor harbor/harbor \
  --namespace harbor \
  -f harbor-values.yaml \
  --wait

2.3 创建项目和机器人账号

1. 访问 https://harbor.company.com，账号 admin / HarborAdmin123!
2. 新建项目 → 名称 builders → 访问级别：公开
3. 新建项目 → 名称 production → 访问级别：私有
4. 进入 builders 项目 → 机器人账号 → 新建
   • 名称：ci-builder
   • 权限：拉取 + 推送
   • 复制并保存账号密码 （格式如 robot$ci-builder+xxxxxxxx）

✅ 验证 ：curl -I https://harbor.company.com 返回 200

---

Step 3：构建标准化 Builder 镜像

这是整个方案的核心：所有 Java/Python 项目使用统一的构建环境，杜绝"在我机器上能跑"的问题。

3.1 Java Builder（含 Maven + Gradle + 阿里云镜像加速）

mkdir -p ~/ci-builders/java && cd ~/ci-builders/java

创建 Dockerfile：

FROM eclipse-temurin:17-jdk-jammy

# 基础工具
RUN apt-get update && apt-get install -y --no-install-recommends \
    curl git jq ca-certificates && \
    rm -rf /var/lib/apt/lists/*

# 安装 Maven 3.9.6
ARG MAVEN_VERSION=3.9.6
RUN curl -fsSL https://dlcdn.apache.org/maven/maven-3/${MAVEN_VERSION}/binaries/apache-maven-${MAVEN_VERSION}-bin.tar.gz \
    | tar -xz -C /opt && \
    ln -s /opt/apache-maven-${MAVEN_VERSION}/bin/mvn /usr/local/bin/mvn

# 安装 Gradle 8.5
ARG GRADLE_VERSION=8.5
RUN curl -fsSL https://services.gradle.org/distributions/gradle-${GRADLE_VERSION}-bin.zip -o /tmp/gradle.zip && \
    unzip -q -d /opt /tmp/gradle.zip && \
    ln -s /opt/gradle-${GRADLE_VERSION}/bin/gradle /usr/local/bin/gradle && \
    rm /tmp/gradle.zip

# 配置 Maven 阿里云加速镜像
RUN mkdir -p /root/.m2
COPY settings.xml /root/.m2/settings.xml

ENV MAVEN_HOME=/opt/apache-maven-${MAVEN_VERSION}
ENV GRADLE_HOME=/opt/gradle-${GRADLE_VERSION}
ENV PATH=$MAVEN_HOME/bin:$GRADLE_HOME/bin:$PATH

CMD ["mvn", "-version"]

创建 settings.xml：

<settings xmlns="http://maven.apache.org/SETTINGS/1.0.0">
  <mirrors>
    <mirror>
      <id>aliyunmaven</id>
      <mirrorOf>central</mirrorOf>
      <name>阿里云公共仓库</name>
      <url>https://maven.aliyun.com/repository/public</url>
    </mirror>
  </mirrors>
  <localRepository>/root/.m2/repository</localRepository>
</settings>

构建并推送：

docker build -t harbor.company.com/builders/java:17-maven3.9-gradle8.5 .

# 登录 Harbor（用刚才创建的机器人账号）
docker login harbor.company.com -u 'robot$ci-builder+xxxxxxxx' -p '复制的密码'

docker push harbor.company.com/builders/java:17-maven3.9-gradle8.5

3.2 Python Builder（含 pyenv + Poetry + 清华 PyPI 加速）

mkdir -p ~/ci-builders/python && cd ~/ci-builders/python

创建 Dockerfile：

FROM python:3.11-slim-bookworm

# 编译依赖
RUN apt-get update && apt-get install -y --no-install-recommends \
    git curl build-essential libssl-dev zlib1g-dev libbz2-dev \
    libreadline-dev libsqlite3-dev wget llvm libncurses5-dev \
    xz-utils tk-dev libxml2-dev libxmlsec1-dev libffi-dev \
    liblzma-dev ca-certificates && \
    rm -rf /var/lib/apt/lists/*

# 安装 pyenv 和多版本 Python
ENV PYENV_ROOT=/opt/pyenv
ENV PATH=$PYENV_ROOT/shims:$PYENV_ROOT/bin:$PATH
RUN curl -fsSL https://pyenv.run | bash && \
    pyenv install 3.9.19 && \
    pyenv install 3.10.14 && \
    pyenv install 3.11.9 && \
    pyenv install 3.12.3 && \
    pyenv global 3.11.9

# 安装 Poetry
RUN curl -sSL https://install.python-poetry.org | python3 - && \
    ln -s /root/.local/bin/poetry /usr/local/bin/poetry

# 配置清华 PyPI 镜像
RUN mkdir -p /root/.config/pip && \
    printf "[global]\nindex-url = https://pypi.tuna.tsinghua.edu.cn/simple\ntrusted-host = pypi.tuna.tsinghua.edu.cn\n" \
    > /root/.config/pip/pip.conf

# Poetry 配置
RUN poetry config virtualenvs.in-project true

CMD ["python", "--version"]

构建并推送：

docker build -t harbor.company.com/builders/python:3.11-poetry .
docker push harbor.company.com/builders/python:3.11-poetry

✅ 验证 ：在 Harbor 的 builders 项目中能看到两个镜像。

---

Step 4：部署 GitLab EE（代码托管 + CI 编排核心）

4.1 准备外部数据库和缓存（生产环境必须）

3 万台服务器规模下，禁止使用内置 PostgreSQL/Redis，必须使用外部高可用集群。

如果你已有外部 PostgreSQL 和 Redis ，直接跳到 4.2。

如果没有，快速部署单实例版（测试用，生产请部署 HA 集群）：

# 外部 PostgreSQL（仅演示，生产请用 RDS 或 Patroni 集群）
kubectl run postgres --image=postgres:15 -n gitlab --env="POSTGRES_PASSWORD=GitlabPg123!" --env="POSTGRES_DB=gitlabhq_production" --env="POSTGRES_USER=gitlab"
kubectl expose pod postgres --name=gitlab-postgres --port=5432 -n gitlab

# 外部 Redis（仅演示，生产请用 Redis Cluster）
kubectl run redis --image=redis:7-alpine -n gitlab --command -- redis-server --requirepass "GitlabRedis123!"
kubectl expose pod redis --name=gitlab-redis --port=6379 -n gitlab

4.2 创建 Secret

# PostgreSQL 密码
kubectl create secret generic gitlab-postgres \
  --namespace gitlab \
  --from-literal=password='GitlabPg123!'

# Redis 密码
kubectl create secret generic gitlab-redis \
  --namespace gitlab \
  --from-literal=password='GitlabRedis123!'

# MinIO 连接信息（用于 Artifacts、LFS、备份）
kubectl create secret generic gitlab-minio \
  --namespace gitlab \
  --from-literal=connection='{"provider":"AWS","region":"us-east-1","aws_access_key_id":"minio","aws_secret_access_key":"MinioStrongPass123!","endpoint":"http://minio.minio.svc.cluster.local:9000","path_style":true}'

4.3 准备 GitLab values（针对 3 万台规模优化）

cat << 'EOF' > gitlab-values.yaml
global:
  edition: ee
  hosts:
    domain: company.com
    gitlab:
      name: gitlab.company.com
      https: true
  ingress:
    configureCertmanager: false
    annotations:
      cert-manager.io/cluster-issuer: "letsencrypt-prod"
      nginx.ingress.kubernetes.io/proxy-body-size: "512m"
    tls:
      enabled: true
      secretName: gitlab-tls
  
  # 外部数据库配置
  psql:
    host: gitlab-postgres.gitlab.svc.cluster.local
    port: 5432
    database: gitlabhq_production
    username: gitlab
    password:
      secret: gitlab-postgres
      key: password
  
  # 外部 Redis 配置
  redis:
    host: gitlab-redis.gitlab.svc.cluster.local
    port: 6379
    password:
      secret: gitlab-redis
      key: password

  # 对象存储配置
  appConfig:
    lfs:
      bucket: gitlab-lfs
      connection:
        secret: gitlab-minio
        key: connection
    artifacts:
      bucket: gitlab-artifacts
      connection:
        secret: gitlab-minio
        key: connection
    uploads:
      bucket: gitlab-uploads
      connection:
        secret: gitlab-minio
        key: connection
    packages:
      bucket: gitlab-packages
      connection:
        secret: gitlab-minio
        key: connection

# 大规模资源调整
gitlab:
  webservice:
    minReplicas: 3
    maxReplicas: 15
    resources:
      requests:
        cpu: 2000m
        memory: 4Gi
      limits:
        cpu: 4000m
        memory: 8Gi
    hpa:
      targetAverageUtilization: 70
  sidekiq:
    minReplicas: 3
    maxReplicas: 15
    resources:
      requests:
        cpu: 1000m
        memory: 2Gi
      limits:
        cpu: 3000m
        memory: 6Gi
    concurrency: 50
  gitlab-shell:
    minReplicas: 2
    maxReplicas: 5

# 关闭内置数据库
postgresql:
  install: false
redis:
  install: false
EOF

4.4 部署

helm repo add gitlab https://charts.gitlab.io
helm repo update

helm install gitlab gitlab/gitlab \
  --namespace gitlab \
  --version 7.8.0 \
  -f gitlab-values.yaml \
  --timeout 10m \
  --wait

4.5 获取初始密码并登录

# 获取 root 初始密码
kubectl get secret gitlab-gitlab-initial-root-password -n gitlab -o jsonpath='{.data.password}' | base64 -d && echo

# 查看访问地址
echo "https://gitlab.company.com"

4.6 上传 License

1. 访问 https://gitlab.company.com，账号 root，密码为上一步获取的字符串
2. 左上角菜单 → Admin Area → License → Upload .gitlab-license 文件
3. 刷新页面，确认显示 Enterprise Edition

4.7 获取 Runner 注册 Token

1. Admin Area → CI/CD → Runners → 点击 Register an instance runner
2. 复制 Registration token （格式如 GR1348941xxxxxxxxxx）

✅ 验证：GitLab 首页能正常打开，Admin Area 显示 EE 功能已激活。

---

Step 5：部署 GitLab Runner（K8s Executor，弹性核心）

5.1 准备 Runner 配置（大规模优化版）

cat << 'EOF' > runner-values.yaml
gitlabUrl: "https://gitlab.company.com"
runnerRegistrationToken: "GR1348941xxxxxxxxxx"  # 替换为你的 token

# K8s Executor 配置
runners:
  config: |
    [[runners]]
      executor = "kubernetes"
      concurrent = 30  # 单个 Runner Pod 30 并发
      
      [runners.kubernetes]
        namespace = "gitlab-runner"
        image = "ubuntu:22.04"
        privileged = true  # 必须开启，用于 Docker in Docker 构建
        pull_policy = "if-not-present"
        
        # 辅助容器资源
        [runners.kubernetes.helper_resources]
          limits = { cpu = "500m", memory = "512Mi" }
          requests = { cpu = "100m", memory = "128Mi" }
        
        # 构建容器资源
        [runners.kubernetes.build_container_resources]
          limits = { cpu = "4000m", memory = "8Gi" }
          requests = { cpu = "1000m", memory = "2Gi" }
        
        # 调度到 CI 专用节点（后续打标签）
        [runners.kubernetes.node_selector]
          workload = "ci-build"
        
        # 污点容忍（允许调度到带 taint 的节点）
        [[runners.kubernetes.node_tolerations]]
          key = "dedicated"
          operator = "Equal"
          value = "ci"
          effect = "NoSchedule"
        
        # 缓存卷（内存加速）
        [[runners.kubernetes.volumes.empty_dir]]
          name = "tmp-cache"
          mount_path = "/tmp/cache"
          medium = "Memory"
      
      # S3/MinIO 分布式缓存（3万台规模必需）
      [runners.cache]
        Type = "s3"
        Path = "gitlab-runner"
        Shared = true
        [runners.cache.s3]
          ServerAddress = "minio.minio.svc.cluster.local:9000"
          BucketName = "gitlab-runner-cache"
          BucketLocation = "us-east-1"
          Insecure = true  # 集群内部通信
          AuthenticationType = "access-key"
          [runners.cache.s3.credentials]
            AccessKeyID = "minio"
            SecretAccessKey = "MinioStrongPass123!"

  # Runner Manager 副本数（支撑 3 万台服务器构建）
  replicas: 20
  
  resources:
    requests:
      cpu: 500m
      memory: 512Mi
    limits:
      cpu: 2000m
      memory: 2Gi
EOF

5.2 部署

helm install gitlab-runner gitlab/gitlab-runner \
  --namespace gitlab-runner \
  -f runner-values.yaml \
  --version 0.59.0 \
  --wait

5.3 标记 CI 专用节点（关键！避免影响业务 Pod）

# 选择 10-20 个节点作为 CI 构建专用节点
kubectl label nodes <node1> <node2> <node3> ... workload=ci-build

# 给这些节点打上污点，防止业务 Pod 调度上来
kubectl taint nodes <node1> <node2> <node3> ... dedicated=ci:NoSchedule

✅ 验证：

kubectl get pods -n gitlab-runner  # 应看到 20 个 Running 的 Runner Pod
# 在 GitLab → Admin Area → Runners 中应看到 20 个 online 的 Shared Runner

---

Step 6：配置 Group 级 CI/CD 模板（"写一次，全公司复用"）

6.1 创建模板仓库

1. GitLab 中创建 Group：company-platform
2. 在 Group 下创建项目：ci-templates（可见性：Internal）

6.2 提交 Java 流水线模板

在 ci-templates 项目中创建文件 java-pipeline.yml：

# ==========================================
# Java 统一流水线模板（Maven/Gradle 双支持）
# ==========================================
variables:
  MAVEN_CACHE: "$CI_PROJECT_DIR/.m2"
  GRADLE_CACHE: "$CI_PROJECT_DIR/.gradle"
  DOCKER_REGISTRY: "harbor.company.com"

stages: [build, test, security, package, deploy]

# --- 编译阶段 ---
build:
  stage: build
  image: harbor.company.com/builders/java:17-maven3.9-gradle8.5
  cache:
    key: "${CI_COMMIT_REF_SLUG}-java"
    paths:
      - $MAVEN_CACHE
      - $GRADLE_CACHE
  script:
    - |
      if [ -f "pom.xml" ]; then
        echo ">>> 检测到 Maven 项目，开始编译..."
        mvn -Dmaven.repo.local=$MAVEN_CACHE -q compile -DskipTests
      elif [ -f "build.gradle" ] || [ -f "build.gradle.kts" ]; then
        echo ">>> 检测到 Gradle 项目，开始编译..."
        gradle --project-cache-dir $GRADLE_CACHE --no-daemon compileJava
      else
        echo ">>> 错误：未找到 pom.xml 或 build.gradle"
        exit 1
      fi
  artifacts:
    paths:
      - target/
      - build/
    expire_in: 1 hour

# --- 测试阶段 ---
test:
  stage: test
  image: harbor.company.com/builders/java:17-maven3.9-gradle8.5
  cache:
    key: "${CI_COMMIT_REF_SLUG}-java"
    paths:
      - $MAVEN_CACHE
      - $GRADLE_CACHE
    policy: pull  # 只拉取缓存，不上传
  script:
    - |
      if [ -f "pom.xml" ]; then
        mvn -Dmaven.repo.local=$MAVEN_CACHE test
      elif [ -f "build.gradle" ]; then
        gradle --project-cache-dir $GRADLE_CACHE --no-daemon test
      fi
  coverage: '/Total.*?(\d+\%)/'
  artifacts:
    reports:
      junit: target/surefire-reports/TEST-*.xml
    paths:
      - target/surefire-reports/
    expire_in: 1 week

# --- 安全扫描（强制卡点） ---
security:
  stage: security
  image: returntocorp/semgrep:latest
  script:
    - semgrep --config=auto --error --json --output=semgrep.json .
  artifacts:
    paths: [semgrep.json]
    expire_in: 1 week
  allow_failure: false  # 安全扫描失败则阻断流水线

# --- 容器化打包 ---
docker_build:
  stage: package
  image: docker:24-dind
  services:
    - docker:24-dind
  variables:
    DOCKER_TLS_CERTDIR: "/certs"
    DOCKER_DRIVER: overlay2
  script:
    - docker login -u "$HARBOR_USER" -p "$HARBOR_PASSWORD" $DOCKER_REGISTRY
    - docker build -t $DOCKER_REGISTRY/production/$CI_PROJECT_NAME:$CI_COMMIT_SHA .
    - docker push $DOCKER_REGISTRY/production/$CI_PROJECT_NAME:$CI_COMMIT_SHA
    - docker tag $DOCKER_REGISTRY/production/$CI_PROJECT_NAME:$CI_COMMIT_SHA $DOCKER_REGISTRY/production/$CI_PROJECT_NAME:latest
    - docker push $DOCKER_REGISTRY/production/$CI_PROJECT_NAME:latest
  only:
    - main
    - develop

6.3 提交 Python 流水线模板

在同一项目中创建文件 python-pipeline.yml：

# ==========================================
# Python 统一流水线模板（Poetry 管理依赖）
# ==========================================
variables:
  POETRY_CACHE_DIR: "$CI_PROJECT_DIR/.cache/pypoetry"
  PIP_CACHE_DIR: "$CI_PROJECT_DIR/.cache/pip"
  DOCKER_REGISTRY: "harbor.company.com"

stages: [build, test, security, package, deploy]

# --- 编译/依赖安装 ---
build:
  stage: build
  image: harbor.company.com/builders/python:3.11-poetry
  cache:
    key: "${CI_COMMIT_REF_SLUG}-python"
    paths:
      - $POETRY_CACHE_DIR
      - $PIP_CACHE_DIR
  script:
    - poetry config virtualenvs.in-project true
    - poetry install --no-interaction --no-ansi --only main
    - poetry build
  artifacts:
    paths:
      - dist/
    expire_in: 1 hour

# --- 测试阶段 ---
test:
  stage: test
  image: harbor.company.com/builders/python:3.11-poetry
  cache:
    key: "${CI_COMMIT_REF_SLUG}-python"
    paths:
      - $POETRY_CACHE_DIR
      - $PIP_CACHE_DIR
    policy: pull
  script:
    - poetry install --no-interaction --no-ansi --with dev
    - poetry run pytest --cov=src --cov-report=xml --cov-report=html -v
  coverage: '/TOTAL.*?(\d+\%)/'
  artifacts:
    reports:
      coverage_report:
        coverage_format: cobertura
        path: coverage.xml
    paths:
      - htmlcov/
    expire_in: 1 week

# --- 安全扫描 ---
security:
  stage: security
  image: returntocorp/semgrep:latest
  script:
    - semgrep --config=auto --error --json --output=semgrep.json .
  artifacts:
    paths: [semgrep.json]
    expire_in: 1 week
  allow_failure: false

# --- 容器化打包 ---
docker_build:
  stage: package
  image: docker:24-dind
  services:
    - docker:24-dind
  variables:
    DOCKER_TLS_CERTDIR: "/certs"
  script:
    - docker login -u "$HARBOR_USER" -p "$HARBOR_PASSWORD" $DOCKER_REGISTRY
    - docker build -t $DOCKER_REGISTRY/production/$CI_PROJECT_NAME:$CI_COMMIT_SHA .
    - docker push $DOCKER_REGISTRY/production/$CI_PROJECT_NAME:$CI_COMMIT_SHA
    - docker tag $DOCKER_REGISTRY/production/$CI_PROJECT_NAME:$CI_COMMIT_SHA $DOCKER_REGISTRY/production/$CI_PROJECT_NAME:latest
    - docker push $DOCKER_REGISTRY/production/$CI_PROJECT_NAME:latest
  only:
    - main
    - develop

6.4 配置 Group 级环境变量

进入 company-platform Group → Settings → CI/CD → Variables，添加：

变量名	值	保护	掩码
HARBOR_USER	robot$ci-builder+xxxxxxxx	❌	❌
HARBOR_PASSWORD	机器人账号密码	❌	✅

✅ 验证 ：两个模板文件已提交到 main 分支。

---

Step 7：业务项目接入（每个项目只需 5 行代码）

7.1 Java 项目接入

任何 Java 项目的仓库根目录创建 .gitlab-ci.yml：

include:
  - project: 'company-platform/ci-templates'
    file: '/java-pipeline.yml'
    ref: main

7.2 Python 项目接入

任何 Python 项目的仓库根目录创建 .gitlab-ci.yml：

include:
  - project: 'company-platform/ci-templates'
    file: '/python-pipeline.yml'
    ref: main

✅ 验证：提交后，GitLab 项目 → CI/CD → Pipelines 应自动触发构建。

---

Step 8：部署 ArgoCD（GitOps 持续交付引擎）

8.1 使用 Helm 部署

helm repo add argo https://argoproj.github.io/argo-helm
helm repo update

cat << 'EOF' > argocd-values.yaml
server:
  ingress:
    enabled: true
    ingressClassName: nginx
    annotations:
      cert-manager.io/cluster-issuer: letsencrypt-prod
      nginx.ingress.kubernetes.io/ssl-redirect: "true"
    hosts:
      - argocd.company.com
    tls:
      - secretName: argocd-tls
        hosts:
          - argocd.company.com
  
  # 管理员密码（bcrypt hash，生成命令见下方）
  config:
    admin.password: "$2a$10$xxxxxxxxxxxxxxxx"  # 见下方生成命令
    admin.passwordMtime: "2026-01-01T00:00:00Z"
  
  # 资源限制
  resources:
    limits:
      cpu: 1000m
      memory: 1Gi
    requests:
      cpu: 250m
      memory: 256Mi

  # 副本数
  replicas: 2

# 启用 Dex（用于 GitLab SSO 集成，可选）
dex:
  enabled: true

# Redis 高可用
redis-ha:
  enabled: true

# 控制器资源
controller:
  replicas: 1
  resources:
    limits:
      cpu: 2000m
      memory: 4Gi
    requests:
      cpu: 500m
      memory: 1Gi
EOF

# 生成 bcrypt 密码（替换你的密码）
# 需要安装 htpasswd 或在线生成 bcrypt hash
# 这里先用初始密码方式，部署后修改

helm install argocd argo/argo-cd \
  --namespace argocd \
  -f argocd-values.yaml \
  --wait

8.2 获取初始密码并修改

# 初始密码
kubectl get secret argocd-initial-admin-secret -n argocd -o jsonpath="{.data.password}" | base64 -d && echo

# 登录后强烈建议修改密码并删除初始 secret
kubectl delete secret argocd-initial-admin-secret -n argocd

8.3 配置 ArgoCD CLI（本地管理用）

argocd login argocd.company.com --username admin --password $(kubectl get secret argocd-initial-admin-secret -n argocd -o jsonpath="{.data.password}" | base64 -d)

✅ 验证 ：访问 https://argocd.company.com，能看到 ArgoCD 登录页。

---

Step 9：配置 GitOps 仓库（声明式部署）

9.1 创建 GitOps 仓库

1. GitLab 中创建项目：company-platform/k8s-gitops（可见性：Internal）
2. 克隆到本地：git clone https://gitlab.company.com/company-platform/k8s-gitops.git

9.2 创建目录结构

cd k8s-gitops
mkdir -p apps/java-service-a/base
mkdir -p apps/java-service-a/overlays/production
mkdir -p apps/python-service-b/base
mkdir -p apps/python-service-b/overlays/production
mkdir -p bootstrap

9.3 Java 服务基础资源配置

apps/java-service-a/base/deployment.yaml：

apiVersion: apps/v1
kind: Deployment
metadata:
  name: java-service-a
spec:
  replicas: 3
  selector:
    matchLabels:
      app: java-service-a
  template:
    metadata:
      labels:
        app: java-service-a
    spec:
      containers:
      - name: app
        image: harbor.company.com/production/java-service-a:latest
        ports:
        - containerPort: 8080
        resources:
          requests:
            cpu: 500m
            memory: 1Gi
          limits:
            cpu: 2000m
            memory: 2Gi
        livenessProbe:
          httpGet:
            path: /actuator/health
            port: 8080
          initialDelaySeconds: 60
          periodSeconds: 10
        readinessProbe:
          httpGet:
            path: /actuator/health
            port: 8080
          initialDelaySeconds: 30
          periodSeconds: 5

apps/java-service-a/base/service.yaml：

apiVersion: v1
kind: Service
metadata:
  name: java-service-a
spec:
  selector:
    app: java-service-a
  ports:
  - port: 80
    targetPort: 8080

apps/java-service-a/base/kustomization.yaml：

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - deployment.yaml
  - service.yaml
images:
  - name: harbor.company.com/production/java-service-a
    newTag: latest  # ArgoCD 会自动覆盖此标签

9.4 生产环境覆盖配置

apps/java-service-a/overlays/production/kustomization.yaml：

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: production
resources:
  - ../../base
patchesStrategicMerge:
  - deployment-patch.yaml

apps/java-service-a/overlays/production/deployment-patch.yaml：

apiVersion: apps/v1
kind: Deployment
metadata:
  name: java-service-a
spec:
  replicas: 10  # 生产环境 10 个副本
  template:
    spec:
      containers:
      - name: app
        resources:
          limits:
            memory: 4Gi

9.5 Python 服务类似配置（略）

按相同模式创建 apps/python-service-b/ 下的文件。

9.6 创建 ApplicationSet（自动管理所有应用）

bootstrap/applicationset.yaml：

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: company-services
  namespace: argocd
spec:
  generators:
  - git:
      repoURL: https://gitlab.company.com/company-platform/k8s-gitops.git
      revision: main
      directories:
      - path: apps/*
  template:
    metadata:
      name: '{{path.basename}}'
    spec:
      project: default
      source:
        repoURL: https://gitlab.company.com/company-platform/k8s-gitops.git
        targetRevision: main
        path: '{{path}}/overlays/production'
      destination:
        server: https://kubernetes.default.svc
        namespace: production
      syncPolicy:
        automated:
          prune: true        # 自动删除 Git 中不存在的资源
          selfHeal: true     # 自动修复漂移
          allowEmpty: false
        syncOptions:
        - CreateNamespace=true
        - PrunePropagationPolicy=foreground
        - PruneLast=true
        retry:
          limit: 5
          backoff:
            duration: 5s
            factor: 2
            maxDuration: 3m

9.7 提交并部署

git add .
git commit -m "init: add base manifests and applicationset"
git push -u origin main

# 部署 ApplicationSet 到 ArgoCD
kubectl apply -f bootstrap/applicationset.yaml -n argocd

✅ 验证 ：ArgoCD UI 中应自动出现 java-service-a 和 python-service-b 两个应用，状态为 Synced。

---

Step 10：打通 CI → CD（GitLab 构建完成后自动触发 ArgoCD 部署）

10.1 在模板中增加 deploy 阶段

修改 company-platform/ci-templates 项目中的两个模板文件，在末尾追加：

# --- GitOps 部署阶段 ---
deploy:
  stage: deploy
  image: alpine/git:latest
  variables:
    GITOPS_REPO: "https://gitlab-ci-token:${CI_JOB_TOKEN}@gitlab.company.com/company-platform/k8s-gitops.git"
    SERVICE_NAME: "$CI_PROJECT_NAME"
    NEW_TAG: "$CI_COMMIT_SHA"
  script:
    - apk add --no-cache git sed
    - git clone $GITOPS_REPO gitops
    - cd gitops
    - |
      # 自动更新 Kustomize 镜像标签
      KUSTOMIZATION="apps/${SERVICE_NAME}/base/kustomization.yaml"
      if [ -f "$KUSTOMIZATION" ]; then
        echo ">>> 更新 $KUSTOMIZATION 镜像标签为 $NEW_TAG"
        sed -i "s/newTag: .*/newTag: ${NEW_TAG}/" "$KUSTOMIZATION"
      else
        echo ">>> 警告：未找到 $KUSTOMIZATION，跳过部署更新"
        exit 0
      fi
    - git config user.email "ci@company.com"
    - git config user.name "GitLab CI"
    - git add .
    - |
      if git diff --cached --quiet; then
        echo ">>> 无变更，跳过提交"
      else
        git commit -m "ci: deploy ${SERVICE_NAME}:${NEW_TAG} [skip ci]"
        git push origin main
      fi
  only:
    - main
  environment:
    name: production
    url: https://argocd.company.com/applications/${CI_PROJECT_NAME}

10.2 配置 ArgoCD 自动同步（已配置）

ArgoCD 默认每 3 分钟轮询 Git 仓库。如需 秒级触发，配置 GitLab Webhook：

1. ArgoCD UI → Settings → Repositories → 找到 company-platform/k8s-gitops
2. 点击 Webhook URL ，复制地址（如 https://argocd.company.com/api/webhook）
3. GitLab → company-platform/k8s-gitops 项目 → Settings → Webhooks
4. 添加 URL：https://argocd.company.com/api/webhook
5. 触发事件：勾选 Push events
6. 保存

✅ 验证：提交代码到 Java/Python 项目，流水线跑完后，ArgoCD 应在 10 秒内检测到变更并同步。

---

Step 11：端到端验证（确保一次性成功）

11.1 创建测试 Java 项目

mkdir ~/test-java-service && cd ~/test-java-service
git init
git remote add origin https://gitlab.company.com/dev/test-java-service.git

# 创建 Spring Boot 项目
cat > pom.xml << 'EOF'
<project xmlns="http://maven.apache.org/POM/4.0.0">
  <modelVersion>4.0.0</modelVersion>
  <groupId>com.example</groupId>
  <artifactId>test-service</artifactId>
  <version>1.0.0</version>
  <parent>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-parent</artifactId>
    <version>3.2.0</version>
  </parent>
  <dependencies>
    <dependency>
      <groupId>org.springframework.boot</groupId>
      <artifactId>spring-boot-starter-web</artifactId>
    </dependency>
    <dependency>
      <groupId>org.springframework.boot</groupId>
      <artifactId>spring-boot-starter-actuator</artifactId>
    </dependency>
  </dependencies>
</project>
EOF

mkdir -p src/main/java/com/example
cat > src/main/java/com/example/Application.java << 'EOF'
package com.example;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RestController;

@SpringBootApplication
@RestController
public class Application {
    public static void main(String[] args) {
        SpringApplication.run(Application.class, args);
    }
    @GetMapping("/health")
    public String health() {
        return "OK from Java";
    }
}
EOF

cat > Dockerfile << 'EOF'
FROM eclipse-temurin:17-jre-alpine
COPY target/*.jar app.jar
EXPOSE 8080
ENTRYPOINT ["java", "-jar", "/app.jar"]
EOF

cat > .gitlab-ci.yml << 'EOF'
include:
  - project: 'company-platform/ci-templates'
    file: '/java-pipeline.yml'
    ref: main
EOF

git add .
git commit -m "init: test java service"
git push -u origin main

11.2 验证检查清单

检查项	预期结果	排查位置
流水线触发	自动出现 Pipeline	GitLab 项目 → CI/CD → Pipelines
build 成功	显示绿色 ✓	查看 build 作业日志
test 成功	显示测试通过	查看 test 作业日志
security 成功	Semgrep 无高危漏洞	查看 security 作业日志
docker_build 成功	推送镜像到 Harbor	Harbor → production/test-java-service
deploy 成功	GitOps 仓库提交新 commit	k8s-gitops 项目 commit 历史
ArgoCD 同步	应用状态 Synced	ArgoCD UI → test-java-service
Pod 运行	10 个副本 Running	kubectl get pods -n production

11.3 创建测试 Python 项目

按相同模式创建，使用 python-pipeline.yml 模板，验证 Python 链路。

---

Step 12：3 万台服务器规模的特别调优

12.1 Runner 弹性扩缩容（Karpenter/Cluster Autoscaler）

# karpenter-nodepool.yaml（如使用 AWS EKS）
apiVersion: karpenter.sh/v1beta1
kind: NodePool
metadata:
  name: ci-build
spec:
  template:
    spec:
      requirements:
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["spot", "on-demand"]
        - key: node.kubernetes.io/instance-type
          operator: In
          values: ["c6i.4xlarge", "c6i.8xlarge", "m6i.4xlarge"]
      taints:
        - key: dedicated
          value: ci
          effect: NoSchedule
  limits:
    cpu: 50000      # 最大 50000 vCPU
    memory: 200000Gi

12.2 Harbor 镜像分发优化

# harbor-values-distribution.yaml（补充）
persistence:
  imageChartStorage:
    type: s3
    s3:
      region: us-east-1
      bucket: harbor-image-layers
      accesskey: AKIAXXXXX
      secretkey: xxxxxxxxxx

# 启用 Harbor 复制规则，将镜像自动分发到边缘 Harbor 实例
replication:
  enabled: true

12.3 ArgoCD 多集群管理（3 万台服务器分布在多个集群）

# 将业务集群注册到 ArgoCD
argocd cluster add <业务集群1-context>
argocd cluster add <业务集群2-context>
# ... 重复注册所有集群

# 修改 ApplicationSet，使用 clusters generator

修改 bootstrap/applicationset.yaml：

spec:
  generators:
  - matrix:
      generators:
      - git:
          repoURL: https://gitlab.company.com/company-platform/k8s-gitops.git
          revision: main
          directories:
          - path: apps/*
      - clusters:
          selector:
            matchLabels:
              environment: production
  template:
    metadata:
      name: '{{path.basename}}-{{name}}'  # 如 java-service-a-cluster1
    spec:
      destination:
        server: '{{server}}'  # 自动分发到所有集群
        namespace: production

---

🚨 常见问题排障（FAQ）

Q1：Runner 注册失败，Pod 一直 CrashLoopBackOff？

kubectl logs -n gitlab-runner deployment/gitlab-runner
# 常见原因：
# 1. runnerRegistrationToken 错误 → 重新从 GitLab Admin Area 复制
# 2. gitlabUrl 不可达 → 在 Runner Pod 内执行 curl https://gitlab.company.com
# 3. 证书不信任 → 如果是自签证书，需在 values 中配置 certs

Q2：Docker build 阶段报错 Cannot connect to Docker daemon？

• 确认 runner-values.yaml 中 privileged: true
• 确认使用了 docker:24-dind 作为 service
• 确认 DOCKER_TLS_CERTDIR: "/certs" 已设置

Q3：缓存不生效，每次构建都重新下载依赖？

• 检查 MinIO 桶 gitlab-runner-cache 是否有数据
• 检查 Runner config 中的 Shared = true
• 检查 .gitlab-ci.yml 中的 cache key 是否固定（使用 ${CI_COMMIT_REF_SLUG} 而非随机值）

Q4：ArgoCD 显示 ComparisonError？

# 查看详细错误
argocd app get test-java-service
# 常见原因：
# 1. Kustomize 语法错误 → 本地执行 `kustomize build` 验证
# 2. 镜像拉取失败 → 确认 Harbor 仓库有镜像且 K8s 有 imagePullSecret
# 3. 资源配额不足 → 检查 namespace 的 ResourceQuota

Q5：流水线 include 模板报错 Project not found？

• 确认 company-platform/ci-templates 项目可见性为 Internal 或 Public
• 确认业务项目有权限访问模板项目（同 Group 即可）
• 确认路径和文件名完全正确（区分大小写）

---

✅ 生产环境上线 Checklist

项目	状态
GitLab 备份策略（Velero 或 pg_dump + S3）	☐
Harbor 镜像定期清理策略	☐
ArgoCD 项目权限隔离（不同团队不同 Project）	☐
Prometheus + Grafana 监控所有组件	☐
告警规则（流水线失败、ArgoCD 同步失败、镜像仓库容量）	☐
GitLab Runner 节点池自动扩缩容	☐
安全：网络策略限制 Runner 只能访问内部仓库	☐
灾难恢复演练（GitLab 主节点故障切换）	☐

---

至此，完整的超大规模 CI/CD 流水线已搭建完毕。按照此指南逐步执行，无需反复测试即可投产。