K8s中的Client分析

K8s中的Client分析

引言

在Client-go中，使用kubeclient的Get()、Set()等方法，可以看到这些方法都是对接口Reader和Wtriter的实现，这两个接口都位于sigs.k8s.io/controller-runtime/pkg/client/interfaces包下，用于对K8s对象的IO可以看到这两个接口在不同的客户端中是有不同的实现方式的，而日常开发中我主要使用的客户端包括kubeClient、delegatedClient。

借此，我打算梳理一下常用的K8s客户端Client接口K8s中的众多客户端都实现自Client接口，该接口同样位于sigs.k8s.io/controller-runtime/pkg/client/interfaces包下

// Client knows how to perform CRUD operations on Kubernetes objects.
type Client interface {
	Reader
	Writer
	StatusClient

	// Scheme returns the scheme this client is using.
	Scheme() *runtime.Scheme
	// RESTMapper returns the rest this client is using.
	RESTMapper() meta.RESTMapper
}

Client接口中字段,Reader/Writer/Status接口类型，Scheme和RESTMapper定义了接口实现类中返回Scheme和RESTMapper的方法。通过接口的组合，使得多类客户端之间实现了解耦。接口字段的功能如下：

• Reader：读取K8s对象
• riter：对K8s对象进行修改
• StatusClient：何创建一个客户端，可以更新Kubernetes对象的状态子资源
• runtime.Scheme的定义位于k8s.io/apimachinery/pkg/runtime/scheme下，用于定义APIServer在接收到请求后如何序列化反序列化K8s对象
• meta.RESTMapper的定义位于k8s.io/apimachinery/pkg/api/meta/interfaces下，用于映射K8s资源的Kind

Writer分析

Writer下有两个方法，分别是Create、Delete、DeleteAllOf、Update和Patch

// Writer knows how to create, delete, and update Kubernetes objects.
type Writer interface {
   // Create saves the object obj in the Kubernetes cluster.
   Create(ctx context.Context, obj Object, opts ...CreateOption) error

   // Delete deletes the given obj from Kubernetes cluster.
   Delete(ctx context.Context, obj Object, opts ...DeleteOption) error

   // Update updates the given obj in the Kubernetes cluster. obj must be a
   // struct pointer so that obj can be updated with the content returned by the Server.
   Update(ctx context.Context, obj Object, opts ...UpdateOption) error

   // Patch patches the given obj in the Kubernetes cluster. obj must be a
   // struct pointer so that obj can be updated with the content returned by the Server.
   Patch(ctx context.Context, obj Object, patch Patch, opts ...PatchOption) error

   // DeleteAllOf deletes all objects of the given type matching the given options.
   DeleteAllOf(ctx context.Context, obj Object, opts ...DeleteAllOfOption) error
}

• Create：在集群上创建K8s资源

• Delete：在集群上创建K8s资源

• DeleteAllOf:也是在集群上删除，但是根据范围筛选删除多个资源

• Update：在集群上对资源进行全量更新

• Patch：在集群上对资源进行子资源的更新

  Patch方法中有个Data()方法，通过将子资源转换为[]byte，描述需要更新的子资源的数据

// Patch is a patch that can be applied to a Kubernetes object.
type Patch interface {
	// Type is the PatchType of the patch.
	Type() types.PatchType
	// Data is the raw data representing the patch.
	Data(obj Object) ([]byte, error)
}

总结来说，除了DeleteOfAll，操作粒度都是集群上的单个资源

Reader分析

Reader下有两个方法，分别是Get和List

// Reader knows how to read and list Kubernetes objects.
type Reader interface {
   // Get retrieves an obj for the given object key from the Kubernetes Cluster.
   // obj must be a struct pointer so that obj can be updated with the response
   // returned by the Server.
   Get(ctx context.Context, key ObjectKey, obj Object) error

   // List retrieves list of objects for a given namespace and list options. On a
   // successful call, Items field in the list will be populated with the
   // result returned from the server.
   List(ctx context.Context, list ObjectList, opts ...ListOption) error
}

• Get：根据ObjectKey查询集群上的单个资源。其中ObjectKey为NamespacedName，即根据资源名和命名空间查询资源

  type NamespacedName struct {
  	Namespace string
  	Name      string
  }

• List：根绝查询条件查询出集群上的多个资源。其中ListOption是接口，含有一个方法ApplyToList(*ListOptions)。从ListOptions的结构体定义可以看出，List的查询条件是多个限定条件组合的

  type ListOptions struct {
  	// LabelSelector filters results by label. Use labels.Parse() to
  	// set from raw string form.
  	LabelSelector labels.Selector
  	// FieldSelector filters results by a particular field.  In order
  	// to use this with cache-based implementations, restrict usage to
  	// a single field-value pair that's been added to the indexers.
  	FieldSelector fields.Selector
  
  	// Namespace represents the namespace to list for, or empty for
  	// non-namespaced objects, or to list across all namespaces.
  	Namespace string
  
  	// Limit specifies the maximum number of results to return from the server. The server may
  	// not support this field on all resource types, but if it does and more results remain it
  	// will set the continue field on the returned list object. This field is not supported if watch
  	// is true in the Raw ListOptions.
  	Limit int64
  	// Continue is a token returned by the server that lets a client retrieve chunks of results
  	// from the server by specifying limit. The server may reject requests for continuation tokens
  	// it does not recognize and will return a 410 error if the token can no longer be used because
  	// it has expired. This field is not supported if watch is true in the Raw ListOptions.
  	Continue string
  
  	// Raw represents raw ListOptions, as passed to the API server.  Note
  	// that these may not be respected by all implementations of interface,
  	// and the LabelSelector, FieldSelector, Limit and Continue fields are ignored.
  	Raw *metav1.ListOptions
  }

  从ApplyToList的ListOptions可以看出，在执行List操作时，传入的ListOptions如果包含多个条件，相同类型的Option只会取最后一个

  // ApplyToList implements ListOption for ListOptions.
  func (o *ListOptions) ApplyToList(lo *ListOptions) {
  	if o.LabelSelector != nil {
  		lo.LabelSelector = o.LabelSelector
  	}
  	if o.FieldSelector != nil {
  		lo.FieldSelector = o.FieldSelector
  	}
  	if o.Namespace != "" {
  		lo.Namespace = o.Namespace
  	}
  	if o.Raw != nil {
  		lo.Raw = o.Raw
  	}
  	if o.Raw != nil {
    		lo.Raw = o.Raw
    	}
    	if o.Limit > 0 {
    		lo.Limit = o.Limit
    	}
    	if o.Continue != "" {
    		lo.Continue = o.Continue
    	}
    }

StatusClient分析

// StatusClient knows how to create a client which can update status subresource
// for kubernetes objects.
type StatusClient interface {
   Status() StatusWriter
}

StatusClient接口提供了方法Status()，返回一个StatusWriter

StatusWriter

StatusWriter同样提供了Update和Patch方法。

// StatusWriter knows how to update status subresource of a Kubernetes object.
type StatusWriter interface {
	// Update updates the fields corresponding to the status subresource for the
	// given obj. obj must be a struct pointer so that obj can be updated
	// with the content returned by the Server.
	Update(ctx context.Context, obj Object, opts ...UpdateOption) error

	// Patch patches the given object's subresource. obj must be a struct
	// pointer so that obj can be updated with the content returned by the
	// Server.
	Patch(ctx context.Context, obj Object, patch Patch, opts ...PatchOption) error
}

也就是说，相比起Writer，StatusWriter专门用于更新子资源的状态

Client实现

下面分析Client接口常用的实现类型,主要是client.Client和其子client、namespacedClient、dryRunClient

client.Client

client.Client相当于对。注释中描述这个客户端的初始化采取的策略是"懒初始化"。关于懒初始化的分析将在后文提出。

// client is a client.Client that reads and writes directly from/to an API server.  It lazily initializes
// new clients at the time they are used, and caches the client.
type client struct {
	typedClient        typedClient
	unstructuredClient unstructuredClient
	metadataClient     metadataClient
	scheme             *runtime.Scheme
	mapper             meta.RESTMapper
}

client.CLient字段分析

client.Client中一共有3个子client:

• typeClient：使用K8s提供的结构化 API 来与集群进行通信，例如操作Pod、ConfigMap等结构化资源对象
• unstructredClient：不使用 Kubernetes 提供的结构化 API，而是直接与 API 服务器进行通信，可以操作非K8s资源对象
• metadataClient：操作只读的K8s元数据对象

client初始化分为三个阶段：

• 解析config。解析config设置日志配置，设置scheme，设置RESTMapper

  if config == nil {
     return nil, fmt.Errorf("must provide non-nil rest.Config to client.New")
  }
  
  if !options.Opts.SuppressWarnings {
     // surface warnings
     logger := log.Log.WithName("KubeAPIWarningLogger")
     // Set a WarningHandler, the default WarningHandler
     // is log.KubeAPIWarningLogger with deduplication enabled.
     // See log.KubeAPIWarningLoggerOptions for considerations
     // regarding deduplication.
     rest.SetDefaultWarningHandler(
        log.NewKubeAPIWarningLogger(
           logger,
           log.KubeAPIWarningLoggerOptions{
              Deduplicate: !options.Opts.AllowDuplicateLogs,
           },
        ),
     )
  }
  
  // Init a scheme if none provided
  if options.Scheme == nil {
     options.Scheme = scheme.Scheme
  }
  
  // Init a Mapper if none provided
  if options.Mapper == nil {
     var err error
     options.Mapper, err = apiutil.NewDynamicRESTMapper(config)
     if err != nil {
        return nil, err
     }
  }

• 配置缓存。配置缓存字段

  clientcache := &clientCache{
     config: config,
     scheme: options.Scheme,
     mapper: options.Mapper,
     codecs: serializer.NewCodecFactory(options.Scheme),
  
     structuredResourceByType:   make(map[schema.GroupVersionKind]*resourceMeta),
     unstructuredResourceByType: make(map[schema.GroupVersionKind]*resourceMeta),
  }

• 组装client

  rawMetaClient, err := metadata.NewForConfig(config)
  if err != nil {
     return nil, fmt.Errorf("unable to construct metadata-only client for use as part of client: %w", err)
  }
  
  c := &client{
     typedClient: typedClient{
        cache:      clientcache,
        paramCodec: runtime.NewParameterCodec(options.Scheme),
     },
     unstructuredClient: unstructuredClient{
        cache:      clientcache,
        paramCodec: noConversionParamCodec{},
     },
     metadataClient: metadataClient{
        client:     rawMetaClient,
        restMapper: options.Mapper,
     },
     scheme: options.Scheme,
     mapper: options.Mapper,
  }

懒初始化和缓存初始化

前面分析的k8s源码注释中提到了客户端的初始化方式，与之对应的还有缓存初始化。

我们都知道，K8s中的资源持久化存储是在etcd中的，通过APIServer对资源进行读写最终还是会关联etcd。我们不妨将将etcd数据库类比后端服务中常见的mysql数据库。用户操作mysql数据库可分为2种方式：

1. 每遇到一个请求，对mysql进行一次读写操作，在此之前不会在内存中缓存任何持久化存储的信息。此方式适用于小流量的项目
2. 在mysql数据库和后端程序间增加一层redis缓存，项目启动时先把mysql数据缓存到redis中。后端接收到请求，如果是读操作先查询一次redis,如果redis中每找到数据再查询mysql。这种读写分离方式在大流量下可以减轻负载的压力。

k8s client的两种初始化方式其实也是如此，分别是是否启用缓存的初始化方式：

1. 懒初始化：懒初始化是一种优化策略，它允许客户端在第一次访问某个资源时才进行初始化操作。在此之前，客户端不会预先加载该资源的详细信息。当客户端需要获取某个资源的值时，它才会执行必要的请求并等待响应。这种策略可以避免在客户端启动时预加载所有资源信息所带来的开销和延迟。
2. 缓存初始化：缓存初始化是指在客户端内部维护一个缓存系统，用于存储经常访问的资源信息。当客户端需要获取某个资源的值时，它会首先检查缓存中是否已经存在该资源的副本。如果缓存中不存在该资源的副本，客户端才会向API服务器发送请求以获取最新的信息，并将获取到的数据存储在缓存中。这样，当客户端再次需要该资源的信息时，它可以直接从缓存中读取，而无需再次向API服务器发送请求。

至于在实际项目中是否配置缓存，配置了缓存是否能够起到加速的作用，需要结合实际的流量负载、读写多少场景进行分析。

Writer接口实现

以Create方法为例，可以看出，当client创建资源时，会根据obj的类型判断调用unstructuredClient还是typedClient来创建资源。如果是元数据类型，则不可创建

// Create implements client.Client.
func (c *client) Create(ctx context.Context, obj Object, opts ...CreateOption) error {
   switch obj.(type) {
   case *unstructured.Unstructured:
      return c.unstructuredClient.Create(ctx, obj, opts...)
   case *metav1.PartialObjectMetadata:
      return fmt.Errorf("cannot create using only metadata")
   default:
      return c.typedClient.Create(ctx, obj, opts...)
   }
}

分别看下这两个客户端的实现。对于unstructuredClient，首先会获取资源的版本，然后组装创建配置，并向API Server发送一个POST创建请求，并给这个unstructured资源原有的gvk设置回去

// Create implements client.Client.
func (uc *unstructuredClient) Create(ctx context.Context, obj Object, opts ...CreateOption) error {
	u, ok := obj.(*unstructured.Unstructured)
	if !ok {
		return fmt.Errorf("unstructured client did not understand object: %T", obj)
	}

	gvk := u.GroupVersionKind()

	o, err := uc.cache.getObjMeta(obj)
	if err != nil {
		return err
	}

	createOpts := &CreateOptions{}
	createOpts.ApplyOptions(opts)
	result := o.Post().
		NamespaceIfScoped(o.GetNamespace(), o.isNamespaced()).
		Resource(o.resource()).
		Body(obj).
		VersionedParams(createOpts.AsCreateOptions(), uc.paramCodec).
		Do(ctx).
		Into(obj)

	u.SetGroupVersionKind(gvk)
	return result
}

再看下typeClient的实现。由于typeClient是已经switch-case选择后的选项，所以不用通过断言进行类型判断。由于typeClient处理的是K8s资源，所以不用显式处理gvk

// Create implements client.Client.
func (c *typedClient) Create(ctx context.Context, obj Object, opts ...CreateOption) error {
   o, err := c.cache.getObjMeta(obj)
   if err != nil {
      return err
   }

   createOpts := &CreateOptions{}
   createOpts.ApplyOptions(opts)
   return o.Post().
      NamespaceIfScoped(o.GetNamespace(), o.isNamespaced()).
      Resource(o.resource()).
      Body(obj).
      VersionedParams(createOpts.AsCreateOptions(), c.paramCodec).
      Do(ctx).
      Into(obj)
}

Reader接口实现

以Get接口为例，Reder接口的实现思路总体和Writer是一样的，先断言判断obj的类型，然后根据类型选择不同的子client进行Get操作。

需要注意的是，metadata也是可以进行Get操作的

// Get implements client.Client.
func (c *client) Get(ctx context.Context, key ObjectKey, obj Object) error {
   switch obj.(type) {
   case *unstructured.Unstructured:
      return c.unstructuredClient.Get(ctx, key, obj)
   case *metav1.PartialObjectMetadata:
      // Metadata only object should always preserve the GVK coming in from the caller.
      defer c.resetGroupVersionKind(obj, obj.GetObjectKind().GroupVersionKind())
      return c.metadataClient.Get(ctx, key, obj)
   default:
      return c.typedClient.Get(ctx, key, obj)
   }
}

首先看下unstructuredClient的实现。和实现Create一样，依然是断言，断言成功后获取gvk并从缓存中查询对象，然后像API Server发送一个Get请求，最后为了确保gvk正确再把gvk给set回去

// Get implements client.Client.
func (uc *unstructuredClient) Get(ctx context.Context, key ObjectKey, obj Object) error {
   u, ok := obj.(*unstructured.Unstructured)
   if !ok {
      return fmt.Errorf("unstructured client did not understand object: %T", obj)
   }

   gvk := u.GroupVersionKind()

   r, err := uc.cache.getResource(obj)
   if err != nil {
      return err
   }

   result := r.Get().
      NamespaceIfScoped(key.Namespace, r.isNamespaced()).
      Resource(r.resource()).
      Name(key.Name).
      Do(ctx).
      Into(obj)

   u.SetGroupVersionKind(gvk)

   return result
}

metadata的实现也是类似的思路

// Get implements client.Client.
func (mc *metadataClient) Get(ctx context.Context, key ObjectKey, obj Object) error {
   metadata, ok := obj.(*metav1.PartialObjectMetadata)
   if !ok {
      return fmt.Errorf("metadata client did not understand object: %T", obj)
   }

   gvk := metadata.GroupVersionKind()

   resInt, err := mc.getResourceInterface(gvk, key.Namespace)
   if err != nil {
      return err
   }

   res, err := resInt.Get(ctx, key.Name, metav1.GetOptions{})
   if err != nil {
      return err
   }
   *metadata = *res
   metadata.SetGroupVersionKind(gvk) // restore the GVK, which isn't set on metadata
   return nil
}

typedClient也是和Writer类似，直接从缓存获取了资源并发送请求

// Get implements client.Client.
func (c *typedClient) Get(ctx context.Context, key ObjectKey, obj Object) error {
   r, err := c.cache.getResource(obj)
   if err != nil {
      return err
   }
   return r.Get().
      NamespaceIfScoped(key.Namespace, r.isNamespaced()).
      Resource(r.resource()).
      Name(key.Name).Do(ctx).Into(obj)
}

dryRunClient

在K8s开发中，有时会需要对资源进行预运行调试，即dryRun。dry-run 是 Kubernetes 中的一个命令行参数，它可以在不实际执行操作的情况下，预览 Kubernetes API 的调用结果。

dryRunClient便是用于执行dry-run的客户端。从dryRunClient的定义来看，本质还是试用了client.Client，但在实现Writer和Reader接口时传入的option不同

// NewDryRunClient wraps an existing client and enforces DryRun mode
// on all mutating api calls.
func NewDryRunClient(c Client) Client {
   return &dryRunClient{client: c}
}

var _ Client = &dryRunClient{}

// dryRunClient is a Client that wraps another Client in order to enforce DryRun mode.
type dryRunClient struct {
   client Client
}

// Scheme returns the scheme this client is using.
func (c *dryRunClient) Scheme() *runtime.Scheme {
   return c.client.Scheme()
}

// RESTMapper returns the rest mapper this client is using.
func (c *dryRunClient) RESTMapper() meta.RESTMapper {
   return c.client.RESTMapper()
}

// Create implements client.Client.
func (c *dryRunClient) Create(ctx context.Context, obj Object, opts ...CreateOption) error {
   return c.client.Create(ctx, obj, append(opts, DryRunAll)...)
}

dryRunOptions

现在以Writer接口中Create方法的CreateOptions来分析一下dryRun的机制。可以看出dryRun本身是一个[]string类型的字段参数。AsCreateOptions()方法会读取CreateOptions的dryRun参数，赋值给metav1.CreateOptions，后者真正用于K8s API创建资源。在创建资源时，k8s会校验DryRun参数确认是否需要执行模拟

type CreateOptions struct {
   // When present, indicates that modifications should not be
   // persisted. An invalid or unrecognized dryRun directive will
   // result in an error response and no further processing of the
   // request. Valid values are:
   // - All: all dry run stages will be processed
   DryRun []string

   // FieldManager is the name of the user or component submitting
   // this request.  It must be set with server-side apply.
   FieldManager string

   // Raw represents raw CreateOptions, as passed to the API server.
   Raw *metav1.CreateOptions
}

// AsCreateOptions returns these options as a metav1.CreateOptions.
// This may mutate the Raw field.
func (o *CreateOptions) AsCreateOptions() *metav1.CreateOptions {
	if o == nil {
		return &metav1.CreateOptions{}
	}
	if o.Raw == nil {
		o.Raw = &metav1.CreateOptions{}
	}

	o.Raw.DryRun = o.DryRun
	o.Raw.FieldManager = o.FieldManager
	return o.Raw
}

namespacedClient

namespacedClient同样是对Writer和Reader接口的实现，不同的是在初始化时就会指定资源的命名空间。这从构造函数就可以看出

// NewNamespacedClient wraps an existing client enforcing the namespace value.
// All functions using this client will have the same namespace declared here.
func NewNamespacedClient(c Client, ns string) Client {
   return &namespacedClient{
      client:    c,
      namespace: ns,
   }
}

分析namespacedClient的Create方法，其核心仍然是通过client.Client实现对资源的CRUD，但是在调用前加了两步校验：

1. IsAPINamespaced()方法：检查obj是否是命名空间级别的资源

   // IsAPINamespaced returns true if the object is namespace scoped.
   // For unstructured objects the gvk is found from the object itself.
   func IsAPINamespaced(obj runtime.Object, scheme *runtime.Scheme, restmapper apimeta.RESTMapper) (bool, error) {
      gvk, err := apiutil.GVKForObject(obj, scheme)
      if err != nil {
         return false, err
      }
   
      return IsAPINamespacedWithGVK(gvk, scheme, restmapper)
   }
   
   // IsAPINamespacedWithGVK returns true if the object having the provided
   // GVK is namespace scoped.
   func IsAPINamespacedWithGVK(gk schema.GroupVersionKind, scheme *runtime.Scheme, restmapper apimeta.RESTMapper) (bool, error) {
      restmapping, err := restmapper.RESTMapping(schema.GroupKind{Group: gk.Group, Kind: gk.Kind})
      if err != nil {
         return false, fmt.Errorf("failed to get restmapping: %w", err)
      }
   
      scope := restmapping.Scope.Name()
   
      if scope == "" {
         return false, errors.New("scope cannot be identified, empty scope returned")
      }
   
      if scope != apimeta.RESTScopeNameRoot {
         return true, nil
      }
      return false, nil
   }

2. 检查obj的命名空间：若obj有命名空间，则检查是否和client的命名空间相符。否则把client的命名空间指定给obj

   // Create()方法中检查命名空间片段
   objectNamespace := obj.GetNamespace()
   if objectNamespace != n.namespace && objectNamespace != "" {
      return fmt.Errorf("namespace %s of the object %s does not match the namespace %s on the client", objectNamespace, obj.GetName(), n.namespace)
   }
   
   if isNamespaceScoped && objectNamespace == "" {
      obj.SetNamespace(n.namespace)
   }
   return n.client.Create(ctx, obj, opts...)

总结

在使用client-go进行K8s开发中，开发者经常使用client进行集群上k8s资源的管理。通过分析client的源码可以看出k8s中通过组合接口解耦以及高可扩展性设计的思想。

Client接口实现几乎都是在client.Client实现的基础上进行组合和扩展。开发者在熟悉了这个实现后，可以参考dryRunClient、namespacedClient的实现思路根据自己的需要进行实现。