Velero 备份k8s - 技术栈

1. 什么是 Velero?

Velero 是一个 Vmware 大厂用 Go 语言开发的开源工具，用于备份和恢复 Kubernetes 集群中的资源和持久化数据。

它支持在公有云平台或本地运行，提供备份集群数据、集群资源迁移到其他集群、以及将生产集群复制到开发和测试集群的能力。

2. Velero 与 ETCD 快照备份的区别

备份粒度：ETCD 快照是全局备份，而 Velero 可以对 Kubernetes 集群内对象级别进行备份，支持按 Type、Namespace、Label 等对象进行分类备份或恢复。
恢复影响：ETCD 快照恢复会影响其他 namespace 中的 pod 运行服务，而 Velero 可以针对性地恢复，不影响其他服务。
存储位置：Velero 支持多种对象存储，如 MiniO、Ceph、OSS、S3等存储，而 ETCD 快照是一个本地文件。
备份方式：Velero 支持增量备份，只备份变化的数据，而 ETCD 快照通常是全量备份。

3. Velero 整体架构

Velero 由客户端和服务端两部分组成。客户端是本地运行的命令行工具，服务端运行在 Kubernetes 集群中，负责执行具体的备份和恢复操作。

4. Velero 备份工作流程

本地 Velero 客户端发送备份指令。
Kubernetes 集群内创建一个 Backup 对象。
BackupController 监测 Backup 对象并开始备份过程。
BackupController 向 API Server 查询相关数据。
BackupController 将查询到的数据备份到远端的对象存储（MiniO）。

5. 对象存储MiniO

当前 K8S 集群中，已经有完整 MiniO 服务，部署步骤略；

bash 复制代码

root@k8s-master:/# kubectl get pod -n minio-system
NAME      READY   STATUS    RESTARTS   AGE
minio-0   1/1     Running   0          41h
minio-1   1/1     Running   0          41h

在 MiniO 集群中创建 S3桶

bash 复制代码

mc mb minio/velero-k8s

6. 部署Velero

安装 Velero 客户端（最新版 1.15）

bash 复制代码

wget https://github.com/vmware-tanzu/velero/releases/download/v1.14.0/velero-v1.14.0-linux-amd64.tar.gz
tar -zxvf  velero-v1.14.0-linux-amd64.tar.gz
cp velero-v1.14.0-linux-amd64/velero /usr/local/bin/
velero version

创建 Velero 使用的 MiniO 密钥

[root@master1 /root]# cat >>credentials-velero <EOF
[default]
aws_access_key_id = admin
aws_secret_access_key = XXXXXXX
EOF

创建 Velero namespace

bash 复制代码

kubectl create ns velero

在K8S中安装 Velero 服务端

velero install \
    # 指定存储提供商，minio用的是s3，所以这里是aws
    --provider aws \
    # 指定aws插件
    --plugins velero/velero-plugin-for-aws:v1.10.1 \
    # 指定velero镜像
    --image velero/velero:v1.14.1 \
    # 限制agent CPU和内存资源
    --node-agent-pod-cpu-limit 2 \
    --node-agent-pod-mem-limit 2048Mi \
    # 指定命名空间
    --namespace velero \
    # 指定bucket名称
    --bucket velero-k8s \
    # 默认以文件形式备份持久化卷
    --default-volumes-to-fs-backup \
    # 启用node agent 进行备份和恢复操作，--use-restic已启用，从v1.10开始默认使用kopia备份
    --use-node-agent \
    # 指定S3存储桶凭证
    --secret-file ./credentials-velero \
    # 禁用卷快照功能，只使用文件系统级别的备份
    --use-volume-snapshots=false \
    # 配置备份存储位置的详细信息，其中<minio-api-url>填写minio的api地址，如192.168.10.25:9000
    --backup-location-config region=minio,s3ForcePathStyle="true",s3Url=http://<minio-api-url>

7. 无状态备份

#【1】在demo/demoapp 创建app
[root@master1 /root]# kubectl create ns demo
namespace/demo created
[root@master1 /root]# kubectl create deployment demoapp --image=ikubernetes/demoapp:v1.0 --replicas=3 -n demo
deployment.apps/demoapp created
[root@master1 /root]# kubectl get pods -n demo
NAME                      READY   STATUS    RESTARTS   AGE
demoapp-7c58cd6bb-cs2fc   1/1     Running   0          49s
demoapp-7c58cd6bb-r4rcb   1/1     Running   0          49s
demoapp-7c58cd6bb-w7ckj   1/1     Running   0          49s

#【2】创建一个备份 - 只备份 demo名称空间
[root@master1 /root]# velero backup create demo --include-namespaces demo
Backup request "demo" submitted successfully.
Run `velero backup describe demo` or `velero backup logs demo` for more details.

[root@master1 /root]# velero backup get
NAME   STATUS      ERRORS   WARNINGS   CREATED                         EXPIRES   STORAGE LOCATION   SELECTOR
demo   Completed   0        0          2024-11-21 06:46:30 +0000 UTC   29d       default            <none>

#【3】模拟误删除demo名称空间 进行恢复操作
[root@master1 /root]# kubectl delete ns demo
[root@master1 /root]# kubectl get ns #确定无demo名称空间
NAME              STATUS   AGE
cilium-secrets    Active   20d
default           Active   21d
kube-node-lease   Active   21d
kube-public       Active   21d
kube-system       Active   21d
metallb-system    Active   20d
velero            Active   40m

#进行恢复
[root@master1 /root]# velero restore create --from-backup demo
Restore request "demo-20240321065255" submitted successfully.
Run `velero restore describe demo-20240321065255` or `velero restore logs demo-20240321065255` for more details.

[root@master1 /root]# velero restore get 
NAME                  BACKUP   STATUS      STARTED                         COMPLETED                       ERRORS   WARNINGS   CREATED                         SELECTOR
demo-20240321065255   demo     Completed   2024-11-21 06:52:55 +0000 UTC   2024-11-21 06:52:55 +0000 UTC   0        2          2024-11-21 06:52:55 +0000 UTC   <none>
[root@master1 /root]# kubectl get ns #已恢复demo名称空间
NAME              STATUS   AGE
cilium-secrets    Active   20d
default           Active   21d
demo              Active   23s
kube-node-lease   Active   21d
kube-public       Active   21d
kube-system       Active   21d
metallb-system    Active   20d
velero            Active   42m

[root@master1 /root]# kubectl get pods -n demo #pod重新拉起
NAME                      READY   STATUS    RESTARTS   AGE
demoapp-7c58cd6bb-cs2fc   1/1     Running   0          36s
demoapp-7c58cd6bb-r4rcb   1/1     Running   0          36s
demoapp-7c58cd6bb-w7ckj   1/1     Running   0          36s

8. 有状态备份

**背景：**创建一个有状态redis应用，底层数据持久化使用NFS，验证数据的备份恢复

#【1】验证集群storageclass
[root@master1 /root]# kg sc
NAME                    PROVISIONER                    RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
nfs-csi (default)       fuseim.pri/ifs                 Delete          Immediate              false                  466d

#【2】创建demo/redis 使用nfs-pvc持久化
[root@master1 /root]# cat redis-with-nfs-pvc.yaml
apiVersion: v1
kind: Pod
metadata:
  name: redis-with-nfs-pvc
spec:
  containers:
  - name: redis
    image: redis:7-alpine
    ports:
    - containerPort: 6379
      name: redis
    volumeMounts:
    - mountPath: /data
      name: data-storage
  volumes:
  - name: data-storage
    persistentVolumeClaim:
      claimName: nfs-csi

[root@master1 /root]# kubectl apply -f redis-with-nfs-pvc.yaml -n demo
pod/redis-with-nfs-pvc created

[root@master1 /root]# kubectl get pods -n demo
NAME                      READY   STATUS    RESTARTS   AGE
redis-with-nfs-pvc        1/1     Running   0          51s

#【3】测试写数据
[root@master1 /root]# kubectl exec -it redis-with-nfs-pvc -n demo -- /bin/sh
/data # redis-cli
127.0.0.1:6379> set mykey "BIRKHOFF 2024-11-21"
OK
127.0.0.1:6379> BGSAVE
Background saving started
127.0.0.1:6379> exit
/data # ls
dump.rdb

#【4】备份数据 - redis-backup
[root@master1 /root]# velero backup create redis-backup --include-namespaces demo --default-volumes-to-fs-backup
Backup request "redis-backup" submitted successfully.
Run `velero backup describe redis-backup` or `velero backup logs redis-backup` for more details.
[root@master1 /root]# velero backup get
NAME           STATUS      ERRORS   WARNINGS   CREATED                         EXPIRES   STORAGE LOCATION   SELECTOR
demo           Completed   0        0          2024-11-21 06:46:30 +0000 UTC   29d       default            <none>
redis-backup   Completed   0        0          2024-11-21 08:01:48 +0000 UTC   29d       default            <none>
[root@master1 /root]# velero backup describe redis-backup --details
Name:         redis-backup
Namespace:    velero
Labels:       velero.io/storage-location=default
Annotations:  velero.io/resource-timeout=10m0s
              velero.io/source-cluster-k8s-gitversion=v1.29.2
              velero.io/source-cluster-k8s-major-version=1
              velero.io/source-cluster-k8s-minor-version=29

Phase:  Completed


Namespaces:
  Included:  demo
  Excluded:  <none>

Resources:
  Included:        *
  Excluded:        <none>
  Cluster-scoped:  auto

Label selector:  <none>

Or label selector:  <none>

Storage Location:  default

Velero-Native Snapshot PVs:  auto
Snapshot Move Data:          false
Data Mover:                  velero

TTL:  720h0m0s

CSISnapshotTimeout:    10m0s
ItemOperationTimeout:  4h0m0s

Hooks:  <none>

Backup Format Version:  1.1.0

Started:    2024-11-21 08:01:48 +0000 UTC
Completed:  2024-11-21 08:01:52 +0000 UTC

Expiration:  2024-12-20 08:01:48 +0000 UTC

Total items to be backed up:  23
Items backed up:              23

Resource List:
  apiextensions.k8s.io/v1/CustomResourceDefinition:
    - ciliumendpoints.cilium.io
  apps/v1/Deployment:
    - demo/demoapp
  apps/v1/ReplicaSet:
    - demo/demoapp-7c58cd6bb
  cilium.io/v2/CiliumEndpoint:
    - demo/demoapp-7c58cd6bb-cs2fc
    - demo/demoapp-7c58cd6bb-r4rcb
    - demo/demoapp-7c58cd6bb-w7ckj
    - demo/redis-with-nfs-pvc
  v1/ConfigMap:
    - demo/kube-root-ca.crt
  v1/Event:
    - demo/nfs-pvc.17beb8890487bc71
    - demo/nfs-pvc.17beb88905411edd
    - demo/nfs-pvc.17beb8890aa62b1b
    - demo/redis-with-nfs-pvc.17beb88c587fdc81
    - demo/redis-with-nfs-pvc.17beb88c78733a92
    - demo/redis-with-nfs-pvc.17beb88c78e45daf
    - demo/redis-with-nfs-pvc.17beb88c7abb19b1
  v1/Namespace:
    - demo
  v1/PersistentVolume:
    - pvc-6a17223d-6628-4eb4-877c-954e9b0eef09
  v1/PersistentVolumeClaim:
    - demo/nfs-pvc
  v1/Pod:
    - demo/demoapp-7c58cd6bb-cs2fc
    - demo/demoapp-7c58cd6bb-r4rcb
    - demo/demoapp-7c58cd6bb-w7ckj
    - demo/redis-with-nfs-pvc
  v1/ServiceAccount:
    - demo/default

Backup Volumes:
  Velero-Native Snapshots: <none included>

  CSI Snapshots: <none included>

  Pod Volume Backups - kopia:
    Completed:
      demo/redis-with-nfs-pvc: data-storage

HooksAttempted:  0
HooksFailed:     0

#【5】模拟故障 删除ns ：demo
[root@master1 /root]# kubectl delete ns demo
namespace "demo" deleted
[root@master1 /root]# kubectl get pv
No resources found
[root@master1 /root]# velero backup get
NAME           STATUS      ERRORS   WARNINGS   CREATED                         EXPIRES   STORAGE LOCATION   SELECTOR
demo           Completed   0        0          2024-11-21 06:46:30 +0000 UTC   29d       default            <none>
redis-backup   Completed   0        0          2024-11-21 08:01:48 +0000 UTC   29d       default            <none>
[root@master1 /root]# velero restore create --from-backup redis-backup
Restore request "redis-backup-20240321080651" submitted successfully.
Run `velero restore describe redis-backup-20240321080651` or `velero restore logs redis-backup-20240321080651` for more details.
[root@master1 /root]# kubectl get pvc
No resources found in default namespace.
root@ubuntu-k8s-master01:~# kubectl get pvc -n demo
NAME      STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   VOLUMEATTRIBUTESCLASS   AGE
nfs-pvc   Bound    pvc-5628dd5b-5360-4e33-93b6-8ce42e2da07e   3Gi        RWX            nfs-csi        <unset>                 9s
[root@master1 /root]# kubectl get pv
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM          STORAGECLASS   VOLUMEATTRIBUTESCLASS   REASON   AGE
pvc-5628dd5b-5360-4e33-93b6-8ce42e2da07e   3Gi        RWX            Delete           Bound    demo/nfs-pvc   nfs-csi        <unset>                          24s
[root@master1 /root]# kubectl get pods -n demo
NAME                      READY   STATUS    RESTARTS   AGE
redis-with-nfs-pvc        1/1     Running   0          2m13s

#【6】-验证数据恢复情况
#NFS 服务确认
[root@k8s-harbor ~]# cd /data/nfs/
[root@k8s-harbor nfs]# ll
total 0
drwxr-xr-x 3 polkitd root 74 Mar 21 16:08 pvc-5628dd5b-5360-4e33-93b6-8ce42e2da07e

#服务中数据确认
[root@master1 /root]# kubectl exec -it redis-with-nfs-pvc -n demo -- /bin/sh
Defaulted container "redis" out of: redis, restore-wait (init)
/data # ls
BGSAVE    dump.rdb  exit      set
/data # redis-cli
127.0.0.1:6379> get mykey
"BIRKHOFF 2024-11-21"

10. 总结

Velero 是一个强大的 Kubernetes 集群备份和恢复工具，它提供了细粒度的备份选项，支持多种存储解决方案，并允许跨集群迁移。与 ETCD 快照备份相比，Velero 提供了更多的灵活性和选择性，也让他成为 Kubernetes 集群数据保护的理想选择。