K8S自定义CRD - 技术栈

文章目录

1_引入
[2_CRD 的概念](#2_CRD 的概念)
[3_CRD 安装](#3_CRD 安装)

1_引入

随着 Kubernetes 生态系统的持续发展，越来越多高层次的对象将会不断涌现。比起目前使用的对象，新对象将更加专业化。

有了它们，开发者将不再需要逐一进行 Deployment、Service、configMap 等步骤，而是创建并管理一些用于表达整个应用程序或者软件服务的对象。

我们能使用自定义控制器观察高阶对象，并在这些高阶对象的基础上创建底层对象。

例如，你想在 Kubernetes 集群中运行一个 messaging 代理，只需要创建一个队列资源实例，而自定义队列控制器将自动完成所需的 Secret、Deployment 和 Service。目前，Kubernetes 已经提供了类似的自定义资源添加方式。

2_CRD 的概念

CustomResourceDefinitions（CRD）允许开发者向 Kubernetes API 服务提交 CRD 对象，即可以定义新的资源类型。在成功提交后，开发者可以通过 API 服务提交 JSON 清单或 YAML 清单来创建自定义资源，以及其他 Kubernetes 资源实例。

注意：在 Kubernetes 1.7 之前的版本中，需要通过 ThirdPartyResource 对象的方式来定义自定义资源，ThirdPartyResource 于 Kubernetes 1.8 中被 CRD 替代。

开发者可以通过创建 CRD 来创建新的对象类型。不过，如果创建的对象无法在集群中解决实际问题，那么它就是一个无效特性。通常，CRD 与所有 Kubernetes 核心资源都有一个基于自定义对象有效实现目标的控制器。

CRD 的创建流程

3_CRD 安装

CRD 添加

yaml 复制代码

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata: # 资源名复数.组名
  name: websites.extensions.example.com
spec: # API 组名
  group: extensions.example.com
  versions:
    - name: v1
      served: true
      storage: true
      schema:
        openAPIV3Schema: # 资源的结构模式，用于校验资源字段
          type: object # 顶层资源是一个对象
          properties: # 定义了资源的 spec 部分，类型为 object
            spec:
              type: object
              properties:
                gitRepo:
                  type: string
          required: # 指定 spec 是必须字段
            - spec
  scope: Namespaced
  names:
    plural: websites # 定义资源的复数形式，供 API 使用
    singular: website # 定义资源的单数形式
    kind: Website # 定义资源的 Kubernetes 对象类型（kind 字段值）

代理当前接口，主要为了跳过一些认证

shell 复制代码

kubectl proxy

发起对当前接口的监听

shell 复制代码

curl http://localhost:8001/apis/extensions.example.com/v1/websites?watch=true

测试 CRD 是否有效，创建 Website 的资源清单文件

yaml 复制代码

apiVersion: extensions.example.com/v1
kind: Website
metadata:
  name: website
  namespace: default
spec:
  gitRepo: https://gitee.com/efewagtehqwedqw/website.git

虽然资源出来了，但是没有人为其做实际的动作

查看结果

Json数据模型创建：

json 复制代码

{
  "type": "ADDED",
  "object": {
    "apiVersion": "extensions.example.com/v1",
    "kind": "Website",
    "metadata": {
      "creationTimestamp": "2025-01-14T17:56:32Z",
      "generation": 1,
      "managedFields": [
        {
          "apiVersion": "extensions.example.com/v1",
          "fieldsType": "FieldsV1",
          "fieldsV1": {
            "f:spec": {
              ".": {},
              "f:gitRepo": {}
            }
          },
          "manager": "kubectl-create",
          "operation": "Update",
          "time": "2025-01-14T17:56:32Z"
        }
      ],
      "name": "website",
      "namespace": "default",
      "resourceVersion": "448560",
      "uid": "b0ef73a5-cd1c-4f18-b8d2-08e2bb612bcb"
    },
    "spec": {
      "gitRepo": "https://gitee.com/efewagtehqwedqw/website.git"
    }
  }
}

数据模型------删除

json 复制代码

{
  "type": "DELETED",
  "object": {
    "apiVersion": "extensions.example.com/v1",
    "kind": "Website",
    "metadata": {
      "creationTimestamp": "2025-01-14T18:54:00Z",
      "generation": 1,
      "managedFields": [
        {
          "apiVersion": "extensions.example.com/v1",
          "fieldsType": "FieldsV1",
          "fieldsV1": {
            "f:spec": {
              ".": {},
              "f:gitRepo": {}
            }
          },
          "manager": "kubectl-create",
          "operation": "Update",
          "time": "2025-01-14T18:54:00Z"
        }
      ],
      "name": "website",
      "namespace": "default",
      "resourceVersion": "456357",
      "uid": "e6467fdd-1583-4b1a-84c6-d391f0fb691b"
    },
    "spec": {
      "gitRepo": "https://gitee.com/efewagtehqwedqw/website.git"
    }
  }
}

我们只需要根据监听到的消息做出动作即可，自定义案例逻辑：只需要提供 Website 类型资源清单文件，我们就直接自动创建好对应的 deployment、Service，还能根据给出的 gitRepo 准备好 index.html

可以发现两个容器和保证两个容器数据一致性的 emptyDir

创建 website-controller 代码，主要逻辑就是接收到 JSON 格式消息后根据自定义逻辑请求 ApiServer

java 复制代码

package org.example.controller;

import com.fasterxml.jackson.databind.ObjectMapper;
import lombok.extern.slf4j.Slf4j;
import org.example.WebsiteControllerApplication;
import org.example.models.WebsiteWatchEvent;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.stereotype.Service;
import org.springframework.web.reactive.function.client.WebClient;
import reactor.core.publisher.Flux;

import java.io.IOException;
import java.io.InputStream;
import java.net.URI;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;
import java.nio.charset.StandardCharsets;

@Slf4j
@Service
public class WebsiteController {


    private final WebClient webClient;

    @Value("${k8s.api.url}") // 用于 Kubernetes API 地址配置
    private String k8sApiUrl;

    public WebsiteController() {
        this.webClient = WebClient.create(); // 使用 WebClient 创建基础客户端
    }

    private final ObjectMapper objectMapper = new ObjectMapper();

    public void startWatching() {
        log.info("website-controller started.");
        // 开启一个新的线程处理任务
        Runnable runnable = this::watchWebsites;
        Thread thread = new Thread(runnable);
        thread.start();
    }

    public void watchWebsites() {
        // 使用响应式流的方式处理连续数据流
        Flux<String> flux = webClient.get()
                .uri(k8sApiUrl + "/apis/extensions.example.com/v1/websites?watch=true")
                .retrieve()
                .bodyToFlux(String.class);  // 以 String 流的形式处理返回的每一部分数据

        flux.subscribe(eventJson -> {
            try {
                // 这里是处理每个事件的逻辑
                WebsiteWatchEvent eventObj = objectMapper.readValue(eventJson, WebsiteWatchEvent.class);
                log.info("Received event: {}: {}: {}: {}", eventObj.type, eventObj.object.getApiVersion(), eventObj.object.metadata.name, eventObj.object.spec.gitRepo);

                if ("ADDED".equals(eventObj.type)) {
                    createWebsite(eventObj.object);
                } else if ("DELETED".equals(eventObj.type)) {
                    deleteWebsite(eventObj.object);
                } else {
                    log.warn("Unexpected event: {}", eventObj.type);
                }
            } catch (IOException e) {
                log.error("Error occurred while watching websites IO", e);
            }
        });
    }


    private void createWebsite(WebsiteWatchEvent.Website website) {
        createResource(website, "api/v1", "services", "service-template.json");
        createResource(website, "apis/apps/v1", "deployments", "deployment-template.json");
    }

    private void deleteWebsite(WebsiteWatchEvent.Website website) {
        deleteResource(website, "api/v1", "services");
        deleteResource(website, "apis/apps/v1", "deployments");
    }

    private void createResource(WebsiteWatchEvent.Website website, String apiGroup, String kind, String filename) {
        try {
            log.info("Creating {} with name {} in namespace {}", kind, website.metadata.name, website.metadata.namespace);
            // 读取模板文件
            InputStream inputStream = WebsiteControllerApplication.class.getClassLoader().getResourceAsStream(filename);
            if (inputStream == null) {
                log.error("Resource file not found: {}", filename);
                return;
            }
            String template = new String(inputStream.readAllBytes(), StandardCharsets.UTF_8);
            template = template.replace("[NAME]", website.metadata.name);
            template = template.replace("[GIT-REPO]", website.spec.gitRepo);

            // 发送 POST 请求
            String url = String.format("%s/%s/namespaces/%s/%s", k8sApiUrl, apiGroup, website.metadata.namespace, kind);
            HttpRequest request = HttpRequest.newBuilder()
                    .uri(URI.create(url))
                    .POST(HttpRequest.BodyPublishers.ofString(template))
                    .header("Content-Type", "application/json")
                    .build();
            HttpClient.newHttpClient().send(request, HttpResponse.BodyHandlers.ofString());

            log.info("Resource {} created successfully.", kind);
        } catch (IOException | InterruptedException e) {
            log.error("Error creating resource", e);
        }
    }

    private void deleteResource(WebsiteWatchEvent.Website website, String apiGroup, String kind) {
        try {
            log.info("Deleting {} with name {} in namespace {}", kind, website.metadata.name, website.metadata.namespace);

            String url = String.format("%s/%s/namespaces/%s/%s/%s", k8sApiUrl, apiGroup, website.metadata.namespace, kind, website.metadata.name);
            HttpRequest request = HttpRequest.newBuilder()
                    .uri(URI.create(url))
                    .DELETE()
                    .build();
            HttpClient.newHttpClient().send(request, HttpResponse.BodyHandlers.ofString());

            log.info("Resource {} deleted successfully.", kind);
        } catch (IOException | InterruptedException e) {
            log.error("Error deleting resource", e);
        }
    }

}

准备好的资源清单模版，deployment

json 复制代码

{
  "apiVersion": "apps/v1",
  "kind": "Deployment",
  "metadata": {
    "name": "[NAME]",
    "labels": {
      "webserver": "[NAME]"
    }
  },
  "spec": {
    "replicas": 1,
    "selector": {
      "matchLabels": {
        "webserver": "[NAME]"
      }
    },
    "template": {
      "metadata": {
        "name": "[NAME]",
        "labels": {
          "webserver": "[NAME]"
        }
      },
      "spec": {
        "containers": [
          {
            "image": "nginx:1.27.3-alpine",
            "name": "main",
            "volumeMounts": [
              {
                "name": "html",
                "mountPath": "/usr/share/nginx/html",
                "readOnly": true
              }
            ],
            "ports": [
              {
                "containerPort": 80,
                "protocol": "TCP"
              }
            ]
          },
          {
            "image": "assigned/website:gitsync",
            "name": "git-sync",
            "env": [
              {
                "name": "GIT_SYNC_REPO",
                "value": "[GIT-REPO]"
              },
              {
                "name": "GIT_SYNC_DEST",
                "value": "/gitrepo"
              },
              {
                "name": "GIT_SYNC_BRANCH",
                "value": "master"
              },
              {
                "name": "GIT_SYNC_REV",
                "value": "FETCH_HEAD"
              },
              {
                "name": "GIT_SYNC_WAIT",
                "value": "10"
              }
            ],
            "volumeMounts": [
              {
                "name": "html",
                "mountPath": "/gitrepo"
              }
            ]
          }
        ],
        "volumes": [
          {
            "name": "html",
            "emptyDir": {}
          }
        ]
      }
    }
  }
}

Service

json 复制代码

{
  "apiVersion": "v1",
  "kind": "Service",
  "metadata": {
    "labels": {
      "webserver": "[NAME]"
    },
    "name": "[NAME]"
  },
  "spec": {
    "type": "NodePort",
    "ports": [
      {
        "port": 80,
        "protocol": "TCP",
        "targetPort": 80
      }
    ],
    "selector": {
      "webserver": "[NAME]"
    }
  }
}

打包后封装容器镜像

dockerfile 复制代码

FROM openjdk:11-jdk
LABEL maintainer="sy<[email protected]>"
COPY website-controller.jar /usr/local
WORKDIR /usr/local
ENTRYPOINT [ "java","-jar","website-controller.jar" ]

使用docker build命令构建镜像并分发到其他各个节点上，封装镜像已上传至 hub.docker.com，镜像名为assigned/website:controller

shell 复制代码

docker build -t assigned/website:controller .

配置 kubectl proxy 容器权限

yaml 复制代码

apiVersion: v1
kind: Namespace
metadata:
  name: website
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: website-controller
  namespace: website
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: website-controller
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
  - kind: ServiceAccount
    name: website-controller
    namespace: website

部署 website-controller

yaml 复制代码

apiVersion: apps/v1
kind: Deployment
metadata:
  name: website-controller
  namespace: website
spec:
  replicas: 1
  selector:
    matchLabels:
      app: website-controller
  template:
    metadata:
      labels:
        app: website-controller
    spec:
      containers:
        - image: assigned/website:controller
          imagePullPolicy: IfNotPresent
          name: main
        - image: assigned/website:kubectl-proxy
          name: proxy
      serviceAccount: website-controller
      serviceAccountName: website-controller

再次使用 Website 的资源清单文件进行创建，过一段时间后查看结果

powershell 复制代码

[root@k8s-master 3]# kubectl get pod,svc,deploy -o wide
NAME                           READY   STATUS    RESTARTS      AGE     IP              NODE         NOMINATED NODE   READINESS GATES
pod/website-86bc8fb756-pzh2q   2/2     Running   0             4m36s   10.244.85.241   k8s-node01   <none>           <none>

NAME                 TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)        AGE     SELECTOR
service/kubernetes   ClusterIP   10.0.0.1      <none>        443/TCP        21d     <none>
service/website      NodePort    10.2.104.81   <none>        80:30418/TCP   4m36s   webserver=website

NAME                      READY   UP-TO-DATE   AVAILABLE   AGE     CONTAINERS      IMAGES                                         SELECTOR
deployment.apps/website   1/1     1            1           4m36s   main,git-sync   nginx:1.27.3-alpine,assigned/website:gitsync   webserver=website

在浏览器中访问 masterIP:30418 即可