CI/CD(十) Jenkins共享库与k8s集成

一、创建k8skey（v1.28.2版本）

1、查看k8s集群名称

root@k8s-master:~# kubectl config get-contexts
CURRENT   NAME                          CLUSTER      AUTHINFO           NAMESPACE
*         kubernetes-admin@kubernetes   kubernetes   kubernetes-admin 

2、查看集群详细信息

运行以下命令查看集群 kubernetes 的完整配置：

kubectl config view --raw -o jsonpath='{.clusters[?(@.name=="kubernetes")]}'

输出示例：

{
  "name": "kubernetes",
  "cluster": {
    "certificate-authority-data": "LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0t...",  # Base64 编码的 CA 证书
    "server": "https://<API-SERVER-IP>:6443"  # Kubernetes API Server 地址
  }
}

• 关键字段：

  • certificate-authority-data：集群的 CA 证书（Base64 编码）。

  • server：API Server 的访问地址（如 https://10.51.3.3:6443）。

3、生成新的kubeconfig文件

假设你要为 Jenkins 创建一个专用的 kubeconfig 文件（例如 k8s-config-key-prod-10-60-0-20），步骤如下：

（1）定义变量

# 从现有集群配置中提取信息
CLUSTER_NAME="kubernetes"
API_SERVER=$(kubectl config view --raw -o jsonpath='{.clusters[?(@.name=="kubernetes")].cluster.server}')
CA_CERT=$(kubectl config view --raw -o jsonpath='{.clusters[?(@.name=="kubernetes")].cluster.certificate-authority-data}')

# 新 kubeconfig 的路径和名称（根据你的 k8sKey 规则）
K8S_KEY="k8s-config-key-prod-10-60-0-20"
KUBECONFIG_PATH="/opt/k8s-deploy-config/${K8S_KEY}"

（2）创建 ServiceAccount 并获取 Token（推荐用于自动化）

注意：高版本k8s（v1.24）创建serviceAccount时不再自动创建secret，需要手动显示创建

# 创建 ServiceAccount 和权限绑定
kubectl create serviceaccount jenkins-sa -n default
kubectl create clusterrolebinding jenkins-sa-admin \
  --clusterrole=cluster-admin \
  --serviceaccount=default:jenkins-sa

（3）手动创建 Secret 并绑定到 ServiceAccount

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Secret
type: kubernetes.io/service-account-token
metadata:
  name: jenkins-sa-token  # Secret 名称
  namespace: default
  annotations:
    kubernetes.io/service-account.name: "jenkins-sa"  # 绑定到指定的 ServiceAccount
EOF

（4） 验证 Secret 和 Token

# 查看 ServiceAccount 是否关联了 Secret
kubectl describe serviceaccount jenkins-sa -n default

# 获取新创建的 Secret 中的 Token
TOKEN=$(kubectl get secret jenkins-sa-token -n default -o jsonpath='{.data.token}' | base64 -d)

（5） 生成 kubeconfig 文件

cat <<EOF | sudo tee ${KUBECONFIG_PATH}
apiVersion: v1
kind: Config
clusters:
- name: ${CLUSTER_NAME}
  cluster:
    certificate-authority-data: ${CA_CERT}
    server: ${API_SERVER}
users:
- name: jenkins-sa
  user:
    token: ${TOKEN}
contexts:
- name: jenkins-context
  context:
    cluster: ${CLUSTER_NAME}
    user: jenkins-sa
    namespace: default
current-context: jenkins-context
EOF

（6）验证新 kubeconfig 文件

kubectl --kubeconfig /opt/k8s-deploy-config/k8s-config-key-prod-10-60-0-20 get pods

结果如下：

二、build kubectl:v1.28.2镜像

1、windows离线下载

https://dl.k8s.io/release/v1.28.2/bin/linux/amd64/kubectl 此链接直接提供Linux amd64架构的kubectl二进制文件

2、Dockerfile

FROM alpine:3.18

# 创建并设置工作目录
WORKDIR /opt/k8sdeploy

# 安装 kubectl
COPY kubectl /usr/local/bin/
RUN chmod +x /usr/local/bin/kubectl

# 验证安装（使用新版命令）
RUN kubectl version --client

# 设置默认命令（仍保留原功能）
CMD ["kubectl", "version"]

推送到harbor

docker build -t 10.60.0.20:18080/customer-software/kubectl:v1.28.2 . 
docker push 10.60.0.20:18080/customer-software/kubectl:v1.28.2

推送成功修改此处配置：

三、创建基本环境

1、创建namespace（dev | test）

kubectl create namespace wict-dev

2、创建storgeClass（略）

CI/CD(三) 安装nfs并指定k8s默认storageClass-CSDN博客文章浏览阅读587次，点赞4次，收藏9次。通过以上步骤，可为 Kubernetes 集群提供基于 NFS 的持久化存储支持，解决 "没有存储类" 的问题。https://blog.csdn.net/qq_41369135/article/details/146444419?spm=1001.2014.3001.5502

3、创建日志pvc

logs-pvc-dev.yml

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: logs-pvc-dev
  namespace: wict-dev
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 5Gi
  storageClassName: nfs-storage  # 需与集群中可用 StorageClass 匹配

kubectl apply -f /k8s/project/logs/logs-pvc-dev.yml

4、创建harbor-secret

config.json

{"auths":{"10.60.0.20:18080":{"username":"admin","password":"Harbor12345"}}}

base64命令

cat config.json | base64 -w0  

harbor-secret.yml

apiVersion: v1
kind: Secret
metadata:
  name: harbor-secret
  namespace: wict-dev
type: kubernetes.io/dockerconfigjson
data:
  .dockerconfigjson: eyJhdXRocyI6eyIxMC42MC4wLjIwOjE4MDgwIjp7InVzZXJuYW1lIjoiYWRtaW4iLCJwYXNzd29yZCI6IkhhcmJvcjEyMzQ1In19fQ==

kubectl apply -f /k8s/namespace/wict-dev/secret/harbor-secret.yml

所有节点添加私有配置（http）

/etc/docker/daemon.json新增私有仓库，必须新增，不然宿主机登录不了Harbor，或者开启https，才能登录

"insecure-registries": ["10.60.0.20:18080"]

热加载（不用重启docker）

# 发送SIGHUP信号热加载
sudo kill -SIGHUP $(pidof dockerd)

四、执行（后端）

五、执行（前端）

1、修改部署类型为k8s

2、执行

六、访问配置

1、安装ingress-nginx（略）

手动安装：部署ingress-nginx(国内安装)_ingress-nginx 镜像-CSDN博客

helm安装：CI/CD(六) helm部署ingress-nginx（阿里云）_helm 阿里云 nginx-ingress-CSDN博客

2、修改yml模板(自动创建相关组件)

后端模版新增service

apiVersion: apps/v1
kind: Deployment
metadata:
  name: {{obj.param.metadataName}}-{{obj.param.env}}
  namespace: {{obj.param.metadataNameSpace}}-{{obj.param.env}}
spec:
  selector:
    matchLabels:
      app: {{obj.param.metadataName}}
      project: {{obj.param.project}}
      env: {{obj.param.env}}
  replicas: 1
  template:
    metadata:
      labels:
        app: {{obj.param.metadataName}}
        env: {{obj.param.env}}
        project: {{obj.param.project}}
        ci: {{obj.param.ci}}
    spec:
      terminationGracePeriodSeconds: 60
      imagePullSecrets:
        - name: harbor-secret
#      initContainers:
#        - image: harbor.cqxyy.net/wict/skywalking-agent-wiat:v8.16.2
#          name: sw-agent-sidecar
#          imagePullPolicy: IfNotPresent
#          command: ['sh']
#          args: ['-c','mkdir -p /skywalking/agent && cp -r /usr/skywalking/agent/* /skywalking/agent']
#          volumeMounts:
#            - mountPath: /skywalking/agent
#              name: sw-agent
      containers:
      - name: {{obj.param.metadataName}}
        image: {{obj.param.containersImage}}
        imagePullPolicy: Always
        ports:
        - containerPort: 80
          name: web
          protocol: TCP
        resources:
          limits:
            memory: {{obj.param.resourcesMaxMemory}}
          requests:
            memory: 128Mi
        livenessProbe:
          httpGet:
            path: {{obj.param.actuator}}
            port: 80
          initialDelaySeconds: 60
          periodSeconds: 10
          timeoutSeconds: 5
          failureThreshold: 12
        readinessProbe:
          httpGet:
            path: {{obj.param.actuator}}
            port: 80
          initialDelaySeconds: 60
          periodSeconds: 60
          timeoutSeconds: 5
          failureThreshold: 12
        volumeMounts:
        - name: logs-pvc
          subPath: logs
          mountPath: /opt/jar/logs
        - name: sw-agent
          mountPath: /skywalking/agent
        - name: date-config
          mountPath: /etc/localtime
        {% for obj in obj.param.mountDirList -%}
        - name: {{obj.name}}
          mountPath: {{obj.mountPath}}
        {% endfor %}
        env:
        - name: TZ
          value: Asia/Shanghai
        - name: LIMITS_MEMORY
          valueFrom:
            resourceFieldRef:
              resource: limits.memory
              divisor: 1Mi
        - name: JAVA_OPTS
          value: -XX:+UseContainerSupport -XX:MaxRAMPercentage=60.0 -XX:NativeMemoryTracking=summary {{obj.param.javaOpts}}
        lifecycle:
          preStop:
            exec:
              command: ["/bin/sh", "-c", "rm -f /opt/jar/logs/dump/{{obj.param.metadataName}}.hprof && jmap -dump:live,format=b,file=/opt/jar/logs/dump/{{obj.param.metadataName}}.hprof $(pgrep java)"]
      securityContext:
        fsGroup: 1000
      volumes:
      - name: logs-pvc
        persistentVolumeClaim:
          claimName: logs-pvc-{{obj.param.env}}
      - name: sw-agent
        emptyDir: {}
      - name: date-config
        hostPath:
          path: /etc/localtime
      {% for obj in obj.param.mountDirList -%}
      - name: {{obj.name}}
        hostPath:
          path: {{obj.hostPath}}
      {% endfor %}


---
apiVersion: v1
kind: Service
metadata:
  name: {{obj.param.metadataName}}-{{obj.param.env}}-service
  namespace: {{obj.param.metadataNameSpace}}-{{obj.param.env}}
spec:
  selector:
    app: {{obj.param.metadataName}}
    project: {{obj.param.project}}
    env: {{obj.param.env}}
  ports:
    - protocol: TCP
      port: 80       # Service 端口
      targetPort: 80 # Pod 容器端口
  type: ClusterIP    # 仅集群内访问（由 Ingress 处理外部流量）

前端模板新增service和ingress

apiVersion: apps/v1
kind: Deployment
metadata:
  name: {{obj.param.metadataName}}-{{obj.param.env}}
  namespace: {{obj.param.metadataNameSpace}}-{{obj.param.env}}
spec:
  selector:
    matchLabels:
      app: {{obj.param.metadataName}}
      project: {{obj.param.project}}
      env: {{obj.param.env}}
  replicas: 1
  template:
    metadata:
      labels:
        app: {{obj.param.metadataName}}
        project: {{obj.param.project}}
        env: {{obj.param.env}}
        ci: {{obj.param.ci}}
    spec:
      terminationGracePeriodSeconds: 4
      imagePullSecrets:
        - name: harbor-secret
      containers:
      - name: {{obj.param.metadataName}}
        image: {{obj.param.containersImage}}
        imagePullPolicy: Always
        ports:
        - containerPort: {{obj.param.containerPort}}
          name: web
          protocol: TCP

---
apiVersion: v1
kind: Service
metadata:
  name: {{obj.param.metadataName}}-{{obj.param.env}}-service
  namespace: {{obj.param.metadataNameSpace}}-{{obj.param.env}}
spec:
  selector:
    app: {{obj.param.metadataName}}
    project: {{obj.param.project}}
    env: {{obj.param.env}}
  ports:
    - protocol: TCP
      port: 80       # Service 端口
      targetPort: {{obj.param.containerPort}} # Pod 容器端口
  type: ClusterIP    # 仅集群内访问（由 Ingress 处理外部流量）

---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: {{obj.param.metadataName}}-{{obj.param.env}}-ingress
  namespace: {{obj.param.metadataNameSpace}}-{{obj.param.env}}
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /$1
    nginx.ingress.kubernetes.io/use-regex: "true"
spec:
  ingressClassName: nginx  # 关联 IngressClass
  rules:
    - host: {{obj.param.metadataName}}-{{obj.param.env}}.10.60.0.20.nip.io  # 自定义域名
      http:
        paths:
          # 前端路由规则（处理所有非 /prod-api 的请求）
          - path: /(.*)   # 正则：排除以 /prod-api 开头的路径
            pathType: Prefix
            backend:
              service:
                name: {{obj.param.metadataName}}-{{obj.param.env}}-service  # 指向 Service 名称
                port:
                  number: 80
          - path: /prod-api/(.*)  # 正则：匹配 /prod-api 前缀并分组后续路径
            pathType: Prefix  # 必须设置为 ImplementationSpecific
            backend:
              service:
                name: rc-vue-first-{{obj.param.env}}-service  # 指向 Service 名称
                port:
                  number: 80

3、访问

验证码没出来，有时间在解决