k8s集群搭建一主多从的jenkins集群

方案

+---------------------+
| Jenkins Master      |
| - 持久化配置         |<---(hostpath 存储)
| - 自动容灾           |
+----------+----------+
           |
           | Jenkins JNLP 通信
           |
+----------v----------+      +-------------------+
| Jenkins Agent       |      | Kubernetes Plugin |
| (动态 Pod)           |<---->| (自动创建 Agent)   |
+---------------------+      +-------------------+
           ^
           |
+----------+----------+
| 弹性扩容机制         |
| (HPA + Pod 模板)    |
+---------------------+

一、环境准备

k8s 集群ip

kubectl get nodes -o wide # 显示各节点的 InternalIP 和 ExternalIP

检查pod外网是否可通

# 创建临时 Pod 测试公网访问
kubectl run net-test --image=alpine --rm -it --restart=Never -- sh

# 在 Pod 内部执行（测试 HTTP 访问）
curl -v -4 --connect-timeout 10 http://example.com

# 测试 DNS 解析（确认域名解析是否正常）
nslookup google.com

# 测试 TCP 连接（如需要）
nc -zv www.baidu.com 80

jenkins镜像：

docker login kubernetes-register.sswang.com:80 -u sswang

docker pull jenkins/jenkins:2.344  # 官方 2.34 最新版 :cite[2]:cite[5]:cite[7]
docker tag jenkins/jenkins:2.344 kubernetes-register.sswang.com:80/jenkins/jenkins:2.344
docker push kubernetes-register.sswang.com:80/jenkins/jenkins:2.344

docker pull jenkins/jenkins:lts-jdk17
docker tag jenkins/jenkins:lts-jdk17 kubernetes-register.sswang.com:80/jenkins/jenkins:lts-jdk17
docker push kubernetes-register.sswang.com:80/jenkins/jenkins:lts-jdk17

• agent镜像

docker pull jenkins/inbound-agent:jdk11
docker tag jenkins/inbound-agent:jdk11 kubernetes-register.sswang.com:80/jenkins/inbound-agent:jdk11
docker push kubernetes-register.sswang.com:80/jenkins/inbound-agent:jdk11

• 创建namespace

kubectl create namespace jenkins

• hostpath存储准备

kubectl label node kubernetes-master jenkins-master=true # node打标签，支持基于标签选择器进行pod调度
mkdir -p /mnt/jenkins-data
chown 1000:1000 /mnt/jenkins-data  # Jenkins 默认 UID 1000 deck1
chmod -R 755 /mnt/jenkins-data

• 版本兼容性说明

二、jenkins master部署

1. jenkins-master.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: jenkins-master
  namespace: jenkins
spec:
  replicas: 1 # 副本数量
  selector:
    matchLabels:
      app: jenkins-master
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: jenkins-master
    spec:
      # 绑定到专用节点（可选）
      nodeSelector:
         jenkins-master: "true"
      securityContext:
        fsGroup: 1000
      containers:
      - name: jenkins
        image: kubernetes-register.sswang.com:80/jenkins/jenkins:lts-jdk17
        env:
        - name: JAVA_OPTS
          value: "-Djenkins.install.runSetupWizard=false -Dorg.apache.commons.jelly.tags.fmt.timeZone=Asia/Shanghai"
        ports:
        - containerPort: 8080
        - containerPort: 50000
        volumeMounts:
        - name: jenkins-home
          mountPath: /var/jenkins_home
        livenessProbe:
          httpGet:
            path: /login
            port: 8080
          initialDelaySeconds: 120
          periodSeconds: 10
        readinessProbe:
          httpGet:
            path: /login
            port: 8080
          initialDelaySeconds: 30
        resources:
          limits:
            memory: "2Gi"
            cpu: "1"
          requests:
            memory: "1Gi"
            cpu: "500m"
      volumes:
      - name: jenkins-home
        hostPath:
          path: /mnt/jenkins-data
          type: DirectoryOrCreate

2. jenkins-service.yml

nodeport 对外暴露jenkis服务，能够访问集群内pod。

apiVersion: v1
kind: Service
metadata:
  name: jenkins-service
  namespace: jenkins
spec:
  selector:
    app: jenkins-master
  type: NodePort # 永久对外暴露服务
  ports:
  - name: http
    port: 8080
    targetPort: 8080
  - name: agent
    port: 50000
    targetPort: 50000

kubectl get svc -n jenkins jenkins-service

其中，32251为外部访问所使用的端口，ip为宿主机ip或者虚拟机的ip地址。

Jenkins 服务地址确认

# 查看 Jenkins Service 名称
kubectl get svc -n jenkins

# 验证内部 DNS 解析（在任意 Pod 内）
kubectl run test-dns --image=busybox -it --rm --restart=Never -- \
  nslookup jenkins-service.jenkins.svc.cluster.local

# [svc].[namespace].svc.cluster.local

3. jenkins-rbac.yml

创建相应的RBAC权限，以便Jenkins Master能够创建和管理Agent Pod

1. ServiceAccount：在jenkins命名空间中创建一个名为jenkins的服务账户。
2. ClusterRole：定义了一组权限规则，允许对pods、pods/exec、pods/log等资源的操作，以及获取secrets的权限。
3. ClusterRoleBinding：将ClusterRole绑定到ServiceAccount，使得jenkins服务账户具有ClusterRole中定义的权限。

apiVersion: v1
kind: ServiceAccount
metadata:
  name: jenkins # jenkins serviceaccount
  namespace: jenkins
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: jenkins-agent-role
rules:
- apiGroups: [""]
  resources: ["pods", "pods/exec", "pods/log"]
  verbs: ["create", "delete", "get", "list", "watch", "patch"]
- apiGroups: [""]
  resources: ["secrets"]
  verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: jenkins-agent-role-binding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: jenkins-agent-role
subjects:
- kind: ServiceAccount
  name: jenkins
  namespace: jenkins

kubectl get serviceaccount -n jenkins

如何让某个访问作为serviceaccount 为jenkins的账户？

apiVersion: v1
kind: Secret
metadata:
  name: jenkins-token
  namespace: jenkins
  annotations:
    kubernetes.io/service-account.name: jenkins
type: kubernetes.io/service-account-token

至此就能通过secret访问serviceaccount，进而访问apiserver拥有相应的pod操作权限。

在这里插入图片描述

三、动态Agent配置与kubernetes插件

1. 插件初始化

https://mirrors.huaweicloud.com/jenkins/updates/update-center.json

名称: kubernetes
Kubernetes 地址: https://kubernetes.default.svc.cluster.local
命名空间: jenkins
凭据: jenkins-service-account (选择创建的 ServiceAccount)
Jenkins 地址: http://jenkins-service.jenkins.svc.cluster.local:8080

凭据获取，要求k8s集群配置：serviceaccount账户作为 集群操作权限的承接者，secret绑定account，用于apiserver鉴权，rbac用于将serviceaccount绑定role。

TOKEN=$(kubectl get secret jenkins-token -n jenkins -o jsonpath='{.data.token}' | base64 --decode)
echo $TOKEN

服务证书key获取：

CA_CRT=$(kubectl get secret jenkins-token -n jenkins -o jsonpath='{.data.ca\.crt}' | base64 -d)
echo "$CA_CRT"

# 验证 ServiceAccount 权限
kubectl auth can-i create pods \
  --as system:serviceaccount:jenkins:jenkins \
  -n jenkins
# 应返回 yes

# 验证列表权限
kubectl auth can-i list pods \
  --as system:serviceaccount:jenkins:jenkins \
  -n jenkins
# 应返回 yes
# [命名空间][serviceaccount]

2. pod 模板

名称: jnlp-agent
标签: jnlp-agent
容器模板:
- 名称: jnlp
  镜像: kubernetes-register.sswang.com:80/jenkins/inbound-agent:jdk11
  参数: '${computer.jnlpmac} ${computer.name}'
  资源限制:
    CPU: 500m
    内存: 512Mi
  资源请求:
    CPU: 200m
    内存: 256Mi

podTemplate:
  name: jenkins-agent
  label: jenkins-agent
  serviceAccount: jenkins
#   imagePullSecrets:
#   - name: docker-registry-cred
  containers:
  - name: jnlp
    image: kubernetes-register.sswang.com:80/jenkins/inbound-agent:jdk11
    alwaysPullImage: false
    resourceLimitCpu: "1"
    resourceLimitMemory: "2Gi"
    args: '$(JENKINS_SECRET) $(JENKINS_NAME)'