ETCD 备份与还原

楠奕2024-06-05 9:57

安装etcdctl

准备看下etcd如何命令行操作，才发现，主机上，只用kubeadm拉起了etcd，但没有etcdctl命令。

bash 复制代码 
# sudo docker ps -a | awk '/etcd-master/{print $1}'
c4e3a57f05d7
26a11608b270
836dabc8e254

找到正在运行的etcd，将pod中的etcdctl命令复制到主机上，使得在主机上就能直接使用etcdctl命令。

bash 复制代码 
# sudo docker cp c4e3a57f05d7:/usr/local/bin/etcdctl /usr/local/bin/etcdctl

在此执行etcdctl 命令，已成功执行

bash 复制代码 
# etcdctl
NAME:
        etcdctl - A simple command line client for etcd3.

USAGE:
        etcdctl [flags]

VERSION:
        3.5.1

API VERSION:
        3.5

查看etcd节点成员

bash 复制代码 
# etcdctl --endpoints=https://[127.0.0.1]:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/healthcheck-client.crt --key=/etc/kubernetes/pki/etcd/healthcheck-client.key member list -w table
+-----------------+---------+-------------------+---------------------------+---------------------------+------------+
|       ID        | STATUS  |       NAME        |        PEER ADDRS         |       CLIENT ADDRS        | IS LEARNER |
+-----------------+---------+-------------------+---------------------------+---------------------------+------------+
| dd7a929be676b37 | started |  | https://192.168.1.120:2380 | https://192.168.1.120:2379 |      false |
+-----------------+---------+-------------------+---------------------------+---------------------------+------------+

etcd的证书列表

bash 复制代码 
# ll /etc/kubernetes/pki/etcd/
total 32
-rw-r----- 1 root root 1086 Mar 26 16:52 ca.crt
-rw------- 1 root root 1675 Mar 26 16:52 ca.key
-rw-r----- 1 root root 1159 Mar 26 16:52 healthcheck-client.crt
-rw------- 1 root root 1675 Mar 26 16:52 healthcheck-client.key
-rw-r----- 1 root root 1220 Mar 26 16:52 peer.crt
-rw------- 1 root root 1675 Mar 26 16:52 peer.key
-rw-r----- 1 root root 1220 Mar 26 16:52 server.crt
-rw------- 1 root root 1675 Mar 26 16:52 server.key

etcdctl设置别名

bash 复制代码 
# alias etcdctl='etcdctl --endpoints=https://[127.0.0.1]:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/healthcheck-client.crt --key=/etc/kubernetes/pki/etcd/healthcheck-client.key'
[root@192.168.1.120 ~]# 
[root@192.168.1.120 ~]# etcdctl member list -w table
+-----------------+---------+-------------------+---------------------------+---------------------------+------------+
|       ID        | STATUS  |       NAME        |        PEER ADDRS         |       CLIENT ADDRS        | IS LEARNER |
+-----------------+---------+-------------------+---------------------------+---------------------------+------------+
| dd7a929be676b37 | started | 192.168.1.120 | https://192.168.1.120:2380 | https://192.168.1.120:2379 |      false |
+-----------------+---------+-------------------+---------------------------+---------------------------+------------+

查看etcd的详情

IS LEADER：当前是主节点

RAFT TERM: 做了多少轮的选举

bash 复制代码 
# etcdctl endpoint status -w table 
+--------------------------+-----------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
|         ENDPOINT         |       ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+--------------------------+-----------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| https://[127.0.0.1]:2379 | dd7a929be676b37 |   3.5.1 |   18 MB |      true |      false |        22 |    7579742 |            7579742 |        |
+--------------------------+-----------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+

查看etcd是否健康

bash 复制代码 
# etcdctl endpoint health
https://[127.0.0.1]:2379 is healthy: successfully committed proposal: took = 27.76824ms

操作表小试牛刀

etcd中的数据，存放的目录从 / 开始

bash 复制代码 
# etcdctl put /skywell/byd bus
OK
# etcdctl get /skywell/ --prefix=true
/skywell/byd
bus

查看表中数据

--keys-only： 只查看key，不看value

--limit: 表中有很多，限制只查询几条

bash 复制代码 
# etcdctl get / --prefix=true --keys-only --limit 10
/registry/apiregistration.k8s.io/apiservices/v1.

/registry/apiregistration.k8s.io/apiservices/v1.admissionregistration.k8s.io

/registry/apiregistration.k8s.io/apiservices/v1.apiextensions.k8s.io

/registry/apiregistration.k8s.io/apiservices/v1.apps

/registry/apiregistration.k8s.io/apiservices/v1.authentication.k8s.io

/registry/apiregistration.k8s.io/apiservices/v1.authorization.k8s.io

/registry/apiregistration.k8s.io/apiservices/v1.autoscaling

/registry/apiregistration.k8s.io/apiservices/v1.batch

/registry/apiregistration.k8s.io/apiservices/v1.certificates.k8s.io

/registry/apiregistration.k8s.io/apiservices/v1.coordination.k8s.io

# etcdctl get /skywell --prefix=true --keys-only --limit 10
/skywell/byd

etcd数据备份

bash 复制代码 
# etcdctl snapshot save etcdbackup.db
{"level":"info","ts":1716971751.0047052,"caller":"snapshot/v3_snapshot.go:68","msg":"created temporary db file","path":"etcdbackup.db.part"}
{"level":"info","ts":1716971751.028518,"logger":"client","caller":"v3/maintenance.go:211","msg":"opened snapshot stream; downloading"}
{"level":"info","ts":1716971751.0286477,"caller":"snapshot/v3_snapshot.go:76","msg":"fetching snapshot","endpoint":"https://[127.0.0.1]:2379"}
{"level":"info","ts":1716971751.3699682,"logger":"client","caller":"v3/maintenance.go:219","msg":"completed snapshot read; closing"}
{"level":"info","ts":1716971751.8124714,"caller":"snapshot/v3_snapshot.go:91","msg":"fetched snapshot","endpoint":"https://[127.0.0.1]:2379","size":"18 MB","took":"now"}
{"level":"info","ts":1716971751.8127532,"caller":"snapshot/v3_snapshot.go:100","msg":"saved","path":"etcdbackup.db"}
Snapshot saved at etcdbackup.db

验证备份数据

bash 复制代码 
# etcdctl --write-out=table snapshot status etcdbackup.db 
Deprecated: Use `etcdutl snapshot status` instead.

+----------+----------+------------+------------+
|   HASH   | REVISION | TOTAL KEYS | TOTAL SIZE |
+----------+----------+------------+------------+
| b91b2b0e |  6454813 |        947 |      18 MB |
+----------+----------+------------+------------+

此时，我们删除测试的nginx的deployment

bash 复制代码 
# kubectl get deployment -A
NAMESPACE              NAME                        READY   UP-TO-DATE   AVAILABLE   AGE
default                nginx                       3/3     3            3           151m
ingress-nginx          nginx-deployment            1/1     1            1           15d
ingress-nginx          nginx-ingress-controller    1/1     1            1           15d
kube-system            coredns                     2/2     2            2           63d
kube-system            metrics-server              1/1     1            1           56d
kubernetes-dashboard   dashboard-metrics-scraper   1/1     1            1           56d
kubernetes-dashboard   kubernetes-dashboard        1/1     1            1           56d

# kubectl delete deployment -n default nginx
deployment.apps "nginx" deleted

# kubectl get deployment -A
NAMESPACE              NAME                        READY   UP-TO-DATE   AVAILABLE   AGE
ingress-nginx          nginx-deployment            1/1     1            1           15d
ingress-nginx          nginx-ingress-controller    1/1     1            1           15d
kube-system            coredns                     2/2     2            2           63d
kube-system            metrics-server              1/1     1            1           56d
kubernetes-dashboard   dashboard-metrics-scraper   1/1     1            1           56d
kubernetes-dashboard   kubernetes-dashboard        1/1     1            1           56d

将备份恢复到集群

将备份的数据还原到 --data-dir 指定的目录

bash 复制代码 
# etcdctl  snapshot restore etcdbackup.db  --data-dir=/data/foot/etcdtest/restore
Deprecated: Use `etcdutl snapshot restore` instead.

2024-05-29T16:44:33+08:00       info    snapshot/v3_snapshot.go:251     restoring snapshot      {"path": "etcdbackup.db", "wal-dir": "/data/foot/etcdtest/restore/member/wal", "data-dir": "/data/foot/etcdtest/restore", "snap-dir": "/data/foot/etcdtest/restore/member/snap", "stack": "go.etcd.io/etcd/etcdutl/v3/snapshot.(*v3Manager).Restore\n\t/tmp/etcd-release-3.5.1/etcd/release/etcd/etcdutl/snapshot/v3_snapshot.go:257\ngo.etcd.io/etcd/etcdutl/v3/etcdutl.SnapshotRestoreCommandFunc\n\t/tmp/etcd-release-3.5.1/etcd/release/etcd/etcdutl/etcdutl/snapshot_command.go:147\ngo.etcd.io/etcd/etcdctl/v3/ctlv3/command.snapshotRestoreCommandFunc\n\t/tmp/etcd-release-3.5.1/etcd/release/etcd/etcdctl/ctlv3/command/snapshot_command.go:128\ngithub.com/spf13/cobra.(*Command).execute\n\t/home/remote/sbatsche/.gvm/pkgsets/go1.16.3/global/pkg/mod/github.com/spf13/cobra@v1.1.3/command.go:856\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\t/home/remote/sbatsche/.gvm/pkgsets/go1.16.3/global/pkg/mod/github.com/spf13/cobra@v1.1.3/command.go:960\ngithub.com/spf13/cobra.(*Command).Execute\n\t/home/remote/sbatsche/.gvm/pkgsets/go1.16.3/global/pkg/mod/github.com/spf13/cobra@v1.1.3/command.go:897\ngo.etcd.io/etcd/etcdctl/v3/ctlv3.Start\n\t/tmp/etcd-release-3.5.1/etcd/release/etcd/etcdctl/ctlv3/ctl.go:107\ngo.etcd.io/etcd/etcdctl/v3/ctlv3.MustStart\n\t/tmp/etcd-release-3.5.1/etcd/release/etcd/etcdctl/ctlv3/ctl.go:111\nmain.main\n\t/tmp/etcd-release-3.5.1/etcd/release/etcd/etcdctl/main.go:59\nruntime.main\n\t/home/remote/sbatsche/.gvm/gos/go1.16.3/src/runtime/proc.go:225"}
2024-05-29T16:44:33+08:00       info    membership/store.go:141 Trimming membership information from the backend...
2024-05-29T16:44:34+08:00       info    membership/cluster.go:421       added member    {"cluster-id": "cdf818194e3a8c32", "local-member-id": "0", "added-peer-id": "8e9e05c52164694d", "added-peer-peer-urls": ["http://localhost:2380"]}
2024-05-29T16:44:34+08:00       info    snapshot/v3_snapshot.go:272     restored snapshot       {"path": "etcdbackup.db", "wal-dir": "/data/foot/etcdtest/restore/member/wal", "data-dir": "/data/foot/etcdtest/restore", "snap-dir": "/data/foot/etcdtest/restore/member/snap"}

在指定的位置，重新生成了新的etcd数据

bash 复制代码 
# ll restore/member/
total 8
drwx------ 2 root root 4096 May 29 16:44 snap
drwx------ 2 root root 4096 May 29 16:44 wal

# ll /var/lib/etcd/member/
total 0
drwx------ 2 root root 246 May 29 15:08 snap
drwx------ 2 root root 244 May 29 09:14 wal

现在，需要停止所有的 kubernetes 组件以更新 etcd 数据。

将/etc/kubernetes/manifests/kubernetes中的组件清单文件， 将此文件移除。

bash 复制代码 
# ll /etc/kubernetes/manifests/
total 16
-rw------- 1 root root 2260 Mar 26 16:52 etcd.yaml
-rw------- 1 root root 3367 Mar 26 16:52 kube-apiserver.yaml
-rw------- 1 root root 2878 Mar 26 16:52 kube-controller-manager.yaml
-rw------- 1 root root 1464 Mar 26 16:52 kube-scheduler.yaml

# mv /etc/kubernetes/manifetes/* /tmp

kubelet会自动删除pod.【说会自动删除，我试了下，修改了etcd-data的目录，node节点没显示，kubectl get po -A 也没有显示，最后将移除去的yaml再放回来，k8s环境正常】

bash 复制代码 
# kubectl get po -A
NAMESPACE              NAME                                         READY   STATUS    RESTARTS          AGE
ingress-nginx          nginx-deployment-64d5f7665c-56cpz            1/1     Running   0                 15d
ingress-nginx          nginx-ingress-controller-7cfc988f46-cszsd    1/1     Running   0                 15d
kube-flannel           kube-flannel-ds-lpm9c                        1/1     Running   0                 64d
kube-system            coredns-6d8c4cb4d-sml87                      1/1     Running   0                 64d
kube-system            coredns-6d8c4cb4d-w4hgz                      1/1     Running   0                 64d
kube-system            etcd-master                                  1/1     Running   181 (18d ago)     64d
kube-system            kube-apiserver-master                        1/1     Running   159               64d
kube-system            kube-controller-manager-master               1/1     Running   241 (3d7h ago)    64d
kube-system            kube-proxy-6ct9f                             1/1     Running   0                 64d
kube-system            kube-scheduler-master                        1/1     Running   3256 (3d7h ago)   64d
kube-system            metrics-server-5d6946c85b-5585p              1/1     Running   0                 56d
kubernetes-dashboard   dashboard-metrics-scraper-6f669b9c9b-hmw4b   1/1     Running   0                 19d
kubernetes-dashboard   kubernetes-dashboard-57dd8bd998-ghrhd        1/1     Running   26 (18d ago)      19d

当组件都已删除后，修改/etc/kubernetes/manifests/etcd.yaml中的etcd-data中hostPath路径参数

bash 复制代码 
    volumeMounts:
    - mountPath: /var/lib/etcd  #将这里的路径换成新的etcd的路径，刚才restore所在的目录
      name: etcd-data
    - mountPath: /etc/kubernetes/pki/etcd
      name: etcd-certs
  hostNetwork: true
  priorityClassName: system-node-critical
  securityContext:
    seccompProfile:
      type: RuntimeDefault
  volumes:
  - hostPath:
      path: /etc/kubernetes/pki/etcd
      type: DirectoryOrCreate
    name: etcd-certs
  - hostPath:
      path: /var/lib/etcd
      type: DirectoryOrCreate
    name: etcd-data
status: {}
~

查看k8s集群状态

bash 复制代码 
# kubectl get cs
Warning: v1 ComponentStatus is deprecated in v1.19+
NAME                 STATUS    MESSAGE                         ERROR
controller-manager   Healthy   ok                              
scheduler            Healthy   ok                              
etcd-0               Healthy   {"health":"true","reason":""}
相关推荐
小松学前端1 小时前
第六章 7.0 LinkList
java·开发语言·网络
城南vision1 小时前
计算机网络——TCP篇
网络·tcp/ip·计算机网络
Ciderw2 小时前
块存储、文件存储和对象存储详细介绍
网络·数据库·nvme·对象存储·存储·块存储·文件存储
Tony聊跨境3 小时前
独立站SEO类型及优化：来检查这些方面你有没有落下
网络·人工智能·tcp/ip·ip
2403_875736873 小时前
道品科技智慧农业中的自动气象检测站
网络·人工智能·智慧城市
Tassel_YUE5 小时前
网络自动化04：python实现ACL匹配信息（主机与主机信息）
网络·python·自动化
Diamond技术流5 小时前
从0开始学习Linux——网络配置
linux·运维·网络·学习·安全·centos
Spring_java_gg5 小时前
如何抵御 Linux 服务器黑客威胁和攻击
linux·服务器·网络·安全·web安全
方方怪7 小时前
与IP网络规划相关的知识点
服务器·网络·tcp/ip
weixin_442643428 小时前
推荐FileLink数据跨网摆渡系统 — 安全、高效的数据传输解决方案
服务器·网络·安全·filelink数据摆渡系统
热门推荐
01【HarmonyOS】HUAWEI DevEco Studio 下载地址汇总02(欧拉)openEuler系统添加网卡文件配置流程、(欧拉)openEuler系统手动配置ipv6地址流程、(欧拉)openEuler系统网络管理说明03组基轨迹建模 GBTM的介绍与实现（Stata 或 R）04【AIGC】重塑未来的科技巨轮05全面解析：构建基于深度学习的安全帽检测系统（UI界面+YOLO代码+数据集）06【经验分享】Ubuntu22.04安装微信(linux官方版)07基于YOLOv10深度学习的CT扫描图像肾结石智能检测系统【python源码+Pyqt5界面+数据集+训练代码】深度学习实战、目标检测08Ubuntu 20.04使用Livox mid 360 测试 FAST_LIO09RAG 实践- Ollama+RagFlow 部署本地知识库10【TC3xx芯片】TC3xx芯片电源管理系统PMS详解