2. 浅尝一口 Kubernetes Operator

本文是胡涛大佬所出版的《Kubernetes Operator 进阶开发》，强烈推荐各位阅读原书，本文仅仅留作个人心得，如有侵权立马删除。

---

在个人阅读本书的理解中，认为 operator 就是首先通过 CRD 自定义需要控制的资源，然后通过 controller 来自定义该资源的控制器。

1 Operator 开发环境准备

首先安装一个 kubebuilder 工具，可以前往 kubebuilder 下载：

mv kubebuilder_darwin_amd64 /<path-to-cmd>/kubebuilder

./kubebuilder version
Version: main.version{KubeBuilderVersion:"3.14.1", KubernetesVendor:"1.27.1", GitCommit:"cc338d729c2a578ae491860e3eb71e63864b1390", BuildDate:"2024-03-30T09:26:49Z", GoOs:"darwin", GoArch:"amd64"}

如果在 macOS 提示无法使用，就进入 find 中然后右键打开就行了。

2 开始 demo 开发

然后作者举了一个例子用来学习 operator：开始 -> 获取 application -> 根据 Application.Sepc.Replicas 来构造 Pod 的数目 -> 创建 Pod

输入命令：

 kubebuilder init --domain=sunstrider.cn --repo=gitee.com/langzijiangnan/xxx-operator --owner sunstrider 

然后在不进行深究细节的情况，接着进行添加 API ：

base ❯ kubebuilder create api --group apps --version v1 --kind Application                                    
INFO Create Resource [y/n]                        
y
INFO Create Controller [y/n]                      
y
INFO Writing kustomize manifests for you to edit... 
INFO Writing scaffold for you to edit...          
INFO api/v1/application_types.go                  
INFO api/v1/groupversion_info.go                  
INFO internal/controller/suite_test.go            
INFO internal/controller/application_controller.go 
INFO internal/controller/application_controller_test.go 
INFO Update dependencies:
$ go mod tidy           
INFO Running make:
$ make generate                
mkdir -p /Users/xxxxxx/files/project/operator/chapter-02/a01-demo-operator/bin
Downloading sigs.k8s.io/controller-tools/cmd/controller-gen@v0.14.0
go: downloading sigs.k8s.io/controller-tools v0.14.0
go: downloading github.com/spf13/cobra v1.8.0
go: downloading github.com/gobuffalo/flect v1.0.2
go: downloading github.com/fatih/color v1.16.0
go: downloading github.com/go-logr/logr v1.3.0
go: downloading golang.org/x/sys v0.15.0
/Users/xxxxxx/files/project/operator/chapter-02/a01-demo-operator/bin/controller-gen-v0.14.0 object:headerFile="hack/boilerplate.go.txt" paths="./..."
Next: implement your new API and generate the manifests (e.g. CRDs,CRs) with:
$ make manifests

然后可以发现其中多处来的文件：

base ❯ git status                  
On branch main
Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
        modified:   PROJECT
        modified:   cmd/main.go
        modified:   config/default/kustomization.yaml

Untracked files:
  (use "git add <file>..." to include in what will be committed)
        api/
        config/crd/
        config/rbac/application_editor_role.yaml
        config/rbac/application_viewer_role.yaml
        config/samples/
        internal/

修改其中的 api/v1/application_types.go，其中最主要的修改部分：

package v1

import (
	corev1 "k8s.io/api/core/v1"
	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)

// ApplicationSpec defines the desired state of Application
type ApplicationSpec struct {
	Replicas int32                  `json:"replicas,omitempty"`
	Template corev1.PodTemplateSpec `json:"template,omitempty"`
}

然后通过下面的命令来安装CRD：

make manifests
make install 

但是在运行的时候遇见了这么个奇怪的错误：

 make install  
/Users/xxxxxx/files/project/operator/chapter-02/a01-demo-operator/bin/controller-gen-v0.14.0 rbac:roleName=manager-role crd webhook paths="./..." output:crd:artifacts:config=config/crd/bases
Downloading sigs.k8s.io/kustomize/kustomize/v5@v5.3.0
go: downloading sigs.k8s.io/kustomize/kustomize/v5 v5.3.0
go: downloading sigs.k8s.io/kustomize/kyaml v0.16.0
go: downloading sigs.k8s.io/kustomize/api v0.16.0
go: downloading sigs.k8s.io/kustomize/cmd/config v0.13.0
go: downloading golang.org/x/exp v0.0.0-20231006140011-7918f672742d
go: downloading github.com/go-errors/errors v1.4.2
go: downloading k8s.io/kube-openapi v0.0.0-20230601164746-7562a1006961
go: downloading github.com/xlab/treeprint v1.2.0
go: downloading github.com/imdario/mergo v0.3.13
go: downloading gopkg.in/evanphx/json-patch.v5 v5.6.0
/Users/xxxxxx/files/project/operator/chapter-02/a01-demo-operator/bin/kustomize-v5.3.0 build config/crd | kubectl apply -f -
The CustomResourceDefinition "applications.apps.sunstrider.cn" is invalid: metadata.annotations: Too long: must have at most 262144 bytes
make: *** [install] Error 1

然后经过一番查找，修改 Makefile 里面：

.PHONY: install  
install: manifests kustomize ## Install CRDs into the K8s cluster specified in ~/.kube/config.  
    $(KUSTOMIZE) build config/crd | $(KUBECTL) apply -f -

换成：

.PHONY: install
install: manifests kustomize ## Install CRDs into the K8s cluster specified in ~/.kube/config.
	$(KUSTOMIZE) build config/crd | $(KUBECTL) create -f -

然后就可以成功了：

base ❯ kubectl get crd                                                    
NAME                              CREATED AT
applications.apps.sunstrider.cn   2024-04-04T01:25:13Z

然后进一步修改 config/samples 下的 apps_v1_application.yaml 文件的内容：

apiVersion: apps.sunstrider.cn/v1
kind: Application
metadata:
  labels:
    app.kubernetes.io/name: a01-demo-operator
    app.kubernetes.io/managed-by: kustomize
    app: a01-demo-operator
  name: application-sample
  namespace: default
spec:
  # TODO(user): Add fields here
  replicas: 2
  template:
    spec:
      containers:
      - image: nginx:1.14.2
        name: nginx
        ports:
        - containerPort: 80

然后就可以和作者一样得到很和谐的输出：

~/files/project/operator/chapter-02/a01-demo-operator main*
base ❯ kubectl apply -f config/samples/apps_v1_application.yaml 
application.apps.sunstrider.cn/application-sample created

~/files/project/operator/chapter-02/a01-demo-operator main*
base ❯ kubectl get application                                 
NAME                 AGE
application-sample   7s

然后在 internal/controllers/application_controller.go 中的 Reconcile() 修改代码：

package controller

import (
	"context"
	"fmt"
	corev1 "k8s.io/api/core/v1"
	"k8s.io/apimachinery/pkg/api/errors"
	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
	"time"

	"k8s.io/apimachinery/pkg/runtime"
	ctrl "sigs.k8s.io/controller-runtime"
	"sigs.k8s.io/controller-runtime/pkg/client"
	"sigs.k8s.io/controller-runtime/pkg/log"

	sunstriderappsv1 "gitee.com/langzijiangnan/xxx-operator/api/v1"
)

// ApplicationReconciler reconciles a Application object
type ApplicationReconciler struct {
	client.Client
	Scheme *runtime.Scheme
}


func (r *ApplicationReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
	l := log.FromContext(ctx)

	// 获取 application的信息
	app := &sunstriderappsv1.Application{}
	if err := r.Get(ctx, req.NamespacedName, app); err != nil {
		if errors.IsNotFound(err) {
			l.Info("application is not found")
			return ctrl.Result{}, nil
		}
		l.Error(err, "get application error")
		return ctrl.Result{RequeueAfter: 1 * time.Minute}, err
	}

	for i := 0; i < int(app.Spec.Replicas); i++ {
		pod := &corev1.Pod{
			ObjectMeta: metav1.ObjectMeta{
				Name:      fmt.Sprintf("this-is-sunstrider-%s-%d", app.Name, i),
				Namespace: app.Namespace,
				Labels:    app.Labels,
			},
			Spec: app.Spec.Template.Spec,
		}
		if err := r.Create(ctx, pod); err != nil {
			l.Error(err, "create pod error")
			return ctrl.Result{RequeueAfter: 1 * time.Minute}, err
		}
	}
	l.Info("create pod success")

	return ctrl.Result{}, nil
}

// SetupWithManager sets up the controller with the Manager.
func (r *ApplicationReconciler) SetupWithManager(mgr ctrl.Manager) error {
	return ctrl.NewControllerManagedBy(mgr).
		For(&sunstriderappsv1.Application{}).
		Complete(r)
}

然后就通过下面的命令查看运行情况：

base ❯ make run
...

go run ./cmd/main.go
2024-04-04T09:43:36+08:00       INFO    setup   starting manager
2024-04-04T09:43:36+08:00       INFO    controller-runtime.metrics      Starting metrics server
2024-04-04T09:43:36+08:00       INFO    starting server {"kind": "health probe", "addr": "[::]:8081"}
2024-04-04T09:43:36+08:00       INFO    Starting EventSource    {"controller": "application", "controllerGroup": "apps.sunstrider.cn", "controllerKind": "Application", "source": "kind source: *v1.Application"}
2024-04-04T09:43:36+08:00       INFO    Starting Controller     {"controller": "application", "controllerGroup": "apps.sunstrider.cn", "controllerKind": "Application"}
2024-04-04T09:43:36+08:00       INFO    controller-runtime.metrics      Serving metrics server  {"bindAddress": ":8080", "secure": false}
2024-04-04T09:43:36+08:00       INFO    Starting workers        {"controller": "application", "controllerGroup": "apps.sunstrider.cn", "controllerKind": "Application", "worker count": 1}
2024-04-04T09:43:36+08:00       INFO    create pod success      {"controller": "application", "controllerGroup": "apps.sunstrider.cn", "controllerKind": "Application", "Application": {"name":"application-sample","namespace":"default"}, "namespace": "default", "name": "application-sample", "reconcileID": "ac2d46e8-0079-4cc9-87c2-189db15a63e6"}

同时在另一个终端可以看到下面的内容：

~/files/project/operator/chapter-02/a01-demo-operator main*
base ❯ kubectl get pod               
NAME                                      READY   STATUS              RESTARTS   AGE
this-is-sunstrider-application-sample-0   0/1     ContainerCreating   0          15s
this-is-sunstrider-application-sample-1   0/1     ContainerCreating   0          15s

~/files/project/operator/chapter-02/a01-demo-operator main*
base ❯ kubectl get pod 
NAME                                      READY   STATUS    RESTARTS   AGE
this-is-sunstrider-application-sample-0   1/1     Running   0          101s
this-is-sunstrider-application-sample-1   1/1     Running   0          101s

然后就是采用 image 的方式进行部署，不过这里我遇见了一些 bug 还没有运行成功，但是先记录一下：

make docker-build IMG=application-operator:v0.0.1

kind load docker-image  IMG=application-operator:v0.0.1 --name <kind-cluster> 

make deploy IMG=application-operator:v0.0.1

同样还是出现了那个 annotation too long 的问题，采用同样的方式进行修改：

.PHONY: deploy
deploy: manifests kustomize ## Deploy controller to the K8s cluster specified in ~/.kube/config.
	cd config/manager && $(KUSTOMIZE) edit set image controller=${IMG}
	$(KUSTOMIZE) build config/default | $(KUBECTL) create -f -