k8s 1.28.2 集群部署 docker registry 接入 MinIO 存储

### 文章目录

[@[toc]](#文章目录 @[toc] docker registry 部署生成 htpasswd 文件生成 secret 文件生成 registry 配置文件创建 service 创建 statefulset 创建 ingress 验证 docker registry docker registry 监控 docker registry ui)
[docker registry 部署](#文章目录 @[toc] docker registry 部署生成 htpasswd 文件生成 secret 文件生成 registry 配置文件创建 service 创建 statefulset 创建 ingress 验证 docker registry docker registry 监控 docker registry ui)
[生成 htpasswd 文件](#文章目录 @[toc] docker registry 部署生成 htpasswd 文件生成 secret 文件生成 registry 配置文件创建 service 创建 statefulset 创建 ingress 验证 docker registry docker registry 监控 docker registry ui)
[生成 secret 文件](#文章目录 @[toc] docker registry 部署生成 htpasswd 文件生成 secret 文件生成 registry 配置文件创建 service 创建 statefulset 创建 ingress 验证 docker registry docker registry 监控 docker registry ui)
[生成 registry 配置文件](#文章目录 @[toc] docker registry 部署生成 htpasswd 文件生成 secret 文件生成 registry 配置文件创建 service 创建 statefulset 创建 ingress 验证 docker registry docker registry 监控 docker registry ui)
[创建 service](#文章目录 @[toc] docker registry 部署生成 htpasswd 文件生成 secret 文件生成 registry 配置文件创建 service 创建 statefulset 创建 ingress 验证 docker registry docker registry 监控 docker registry ui)
[创建 statefulset](#文章目录 @[toc] docker registry 部署生成 htpasswd 文件生成 secret 文件生成 registry 配置文件创建 service 创建 statefulset 创建 ingress 验证 docker registry docker registry 监控 docker registry ui)
[创建 ingress](#文章目录 @[toc] docker registry 部署生成 htpasswd 文件生成 secret 文件生成 registry 配置文件创建 service 创建 statefulset 创建 ingress 验证 docker registry docker registry 监控 docker registry ui)
[验证 docker registry](#文章目录 @[toc] docker registry 部署生成 htpasswd 文件生成 secret 文件生成 registry 配置文件创建 service 创建 statefulset 创建 ingress 验证 docker registry docker registry 监控 docker registry ui)
[docker registry 监控](#文章目录 @[toc] docker registry 部署生成 htpasswd 文件生成 secret 文件生成 registry 配置文件创建 service 创建 statefulset 创建 ingress 验证 docker registry docker registry 监控 docker registry ui)
[docker registry ui](#文章目录 @[toc] docker registry 部署生成 htpasswd 文件生成 secret 文件生成 registry 配置文件创建 service 创建 statefulset 创建 ingress 验证 docker registry docker registry 监控 docker registry ui)

docker registry dockerfile

docker registry 配置文件

S3 storage driver

registry:2.8.3 Image hierarchy

docker registry 部署

生成 htpasswd 文件

<username> <password> 改成自己想配置的，如果密码有特殊字符，要用单引号包起来

docker run --rm \
  docker.m.daocloud.io/httpd:latest \
  htpasswd -Bbn <username> <password> > htpasswd

生成 secret 文件

shell 复制代码

kubectl create secret generic docker-registry-auth \
  -n registry \
  --from-file=htpasswd

生成 registry 配置文件

因为涉及到 MinIO 的 accesskey 和 secretkey，这里采用 secret 的方式来生成配置文件

yaml 复制代码

---
apiVersion: v1
kind: Secret
metadata:
  name: docker-registry-cm
  namespace: registry
stringData:
  config.yml: |-
    version: 0.1
    log:
      level: info
      fields:
        service: registry
    storage:
      delete:
        enabled: true
      cache:
          blobdescriptor: inmemory
      s3:
        accesskey: wJpkHB8rznvZBRLfKmBz
        secretkey: ZHIyklv5tktYvGR0iFqBiL9NKh7JKbhyDR9SNAYp
        region: default
        regionendpoint: http://minio.api.devops.icu
        forcepathstyle: true
        accelerate: false
        bucket: docker-registry
        encrypt: false
        secure: false
        v4auth: true
        chunksize: 5242880
        multipartcopymaxconcurrency: 10
    http:
      addr: :5000
      debug:
        addr: :5001
        prometheus:
            enabled: true
            path: /metrics
      headers:
        X-Content-Type-Options: [nosniff]
    health:
      storagedriver:
        enabled: true
        interval: 10s
        threshold: 3
    auth:
      htpasswd:
        realm: basic-realm
        path: /auth/htpasswd
type: Opaque

创建 service

yaml 复制代码

---
apiVersion: v1
kind: Service
metadata:
  labels:
    app.kubernetes.io/name: docker-registry
  name: docker-registry-svc
  namespace: registry
spec:
  ports:
  - name: http
    port: 5000
    targetPort: http
  - name: http-metrics
    port: 5001
    targetPort: http-metrics
  selector:
    app.kubernetes.io/name: docker-registry
  type: ClusterIP

创建 statefulset

yaml 复制代码

---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  labels:
    app.kubernetes.io/name: docker-registry
  name: docker-registry
  namespace: registry
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: docker-registry
  serviceName: docker-registry-svc
  template:
    metadata:
      labels:
        app.kubernetes.io/name: docker-registry
    spec:
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - podAffinityTerm:
              labelSelector:
                matchLabels:
                  app.kubernetes.io/name: docker-registry
              topologyKey: kubernetes.io/hostname
            weight: 1
      containers:
      - image: docker.m.daocloud.io/registry:2.8.3
        livenessProbe:
          failureThreshold: 60
          initialDelaySeconds: 5
          periodSeconds: 10
          successThreshold: 1
          tcpSocket:
            port: http
          timeoutSeconds: 1
        name: docker-registry
        ports:
        - containerPort: 5000
          name: http
        - containerPort: 5001
          name: http-metrics
        readinessProbe:
          failureThreshold: 60
          initialDelaySeconds: 5
          periodSeconds: 10
          successThreshold: 1
          tcpSocket:
            port: http
          timeoutSeconds: 1
        resources:
          limits:
            cpu: 2000m
            memory: 2.5Gi
          requests:
            cpu: 100m
            memory: 100Mi
        startupProbe:
          failureThreshold: 60
          initialDelaySeconds: 5
          periodSeconds: 10
          successThreshold: 1
          tcpSocket:
            port: http
          timeoutSeconds: 1
        volumeMounts:
        - mountPath: /etc/docker/registry
          name: config
        - mountPath: /auth
          name: auth
      terminationGracePeriodSeconds: 30
      volumes:
      - name: config
        secret:
          secretName: docker-registry-cm
      - name: auth
        secret:
          secretName: docker-registry-auth

创建 ingress

没有 ingress 可以开 nodeport 来实现

yaml 复制代码

---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/proxy-body-size: 5G
  name: docker-registry
  namespace: registry
spec:
  ingressClassName: nginx
  rules:
  - host: registry.devops.icu
    http:
      paths:
      - backend:
          service:
            name: docker-registry-svc
            port:
              number: 5000
        path: /
        pathType: Prefix

验证 docker registry

/etc/docker/daemon.json 增加 registry 地址

"insecure-registries": ["ip:端口"]
# 或者
"insecure-registries": ["域名"]

登录 docker registry

shell 复制代码

docker login http://registry.devops.icu

修改 tag

shell 复制代码

docker tag docker.m.daocloud.io/registry:2.8.3 registry.devops.icu/registry:2.8.3

上传镜像

shell 复制代码

docker push registry.devops.icu/registry:2.8.3

docker registry 监控

grafana id：9621

prometheus 配置文件参考

yaml 复制代码

    - job_name: docker-registry
      kubernetes_sd_configs:
      - role: endpoints
      relabel_configs:
      - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name]
        regex: registry;docker-registry-svc
        action: keep
      - source_labels: [__meta_kubernetes_pod_ip]
        regex: (.+)
        target_label: __address__
        replacement: ${1}:5001
      - source_labels: [__meta_kubernetes_endpoints_name]
        action: replace
        target_label: endpoint
      - source_labels: [__meta_kubernetes_pod_name]
        action: replace
        target_label: pod
      - source_labels: [__meta_kubernetes_service_name]
        action: replace
        target_label: service
      - source_labels: [__meta_kubernetes_namespace]
        action: replace
        target_label: namespace

docker registry ui

Github 项目地址：Joxit/docker-registry-ui-2.5.7

相关的变量和参数详见：available-options

yaml 复制代码

---
apiVersion: v1
kind: Service
metadata:
  labels:
    app.kubernetes.io/name: docker-registry-ui
  name: docker-registry-ui-svc
  namespace: registry
spec:
  ports:
  - name: http
    port: 8080
    protocol: TCP
    targetPort: 8080
  selector:
    app.kubernetes.io/name: docker-registry-ui
  type: ClusterIP
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app.kubernetes.io/name: docker-registry-ui
  name: docker-registry-ui
  namespace: registry
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: docker-registry-ui
  template:
    metadata:
      labels:
        app.kubernetes.io/name: docker-registry-ui
    spec:
      containers:
      - env:
        - name: SINGLE_REGISTRY
          value: "true"
        - name: SHOW_CATALOG_NB_TAGS
          value: "true"
        - name: REGISTRY_SECURED
          value: "true"
        - name: NGINX_PROXY_PASS_URL
          value: http://docker-registry-svc.registry.svc.cluster.local:5000
        - name: NGINX_PROXY_HEADER_Authorization
          value: $http_authorization
        image: joxit/docker-registry-ui:2.5.7
        imagePullPolicy: IfNotPresent
        name: docker-registry-ui
      securityContext:
        fsGroup: 101
        runAsUser: 101
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: docker-registry-ui
  namespace: registry
spec:
  ingressClassName: nginx
  rules:
  - host: registry.ui.devops.icu
    http:
      paths:
      - backend:
          service:
            name: docker-registry-ui-svc
            port:
              number: 8080
        path: /
        pathType: Prefix