ETCD集群部署具体步骤文档
⚠️ 核心提示:本文档仅需替换文档中对应节点的主机名和IP地址,所有命令可直接复制粘贴执行,无需修改其他参数。
一、部署前提
1.1 集群规划
本次部署为3节点ETCD集群,节点角色、IP及主机名如下,需确保节点间网络互通。
| 节点角色 | 主机名 | IP地址 | 核心职责 |
|---|---|---|---|
| 主节点(优先部署) | etcd01 | 192.168.133.141 | 生成证书、配置模板,同步至其他节点 |
| 从节点 | etcd02 | 192.168.133.142 | 同步主节点配置,修改专属参数后启动服务 |
| 从节点 | etcd03 | 192.168.133.143 | 同步主节点配置,修改专属参数后启动服务 |
1.1.1 核心配置文件和证书文件
以下为etcd运行不可或缺的核心文件,除/etc/etcd.env需按节点修改IP/主机名外,其余文件全节点一致。
| 文件类型 | 文件名称 | 存放路径 | 核心作用 | 生成/同步方式 |
|---|---|---|---|---|
| CA证书文件 | ca.pem、ca-key.pem | /etc/etcd/pki/etcd/ | 证书签名与身份验证 | etcd01生成后同步 |
| Peer证书文件 | etcd-peer.pem、etcd-peer-key.pem | /etc/etcd/pki/etcd/ | 节点间通信加密与身份认证 | etcd01生成后同步 |
| 环境变量文件 | etcd.env | /etc/ | 定义etcd运行参数 | 各节点单独创建 |
| 系统服务文件 | etcd.service | /etc/systemd/system/ | 通过systemd管理etcd服务 | etcd01生成后同步 |
| 数据存储目录 | etcd(目录) | /var/lib/etcd/ | 存储etcd集群数据、日志 | 全节点统一创建 |
1.2.1 关闭防火墙与禁用SELinux
bash
systemctl stop firewalld && systemctl disable firewalld
setenforce 0 && sed -i 's/^SELINUX=enforcing/SELINUX=disabled/' /etc/selinux/config
1.2.2 时间同步(chrony)
bash
yum install -y chrony && systemctl enable chronyd --now && chronyc -a makestep
1.2.3 常用工具下载(telnet、wget)
bash
yum install -y telnet wget
1.2.4 下载并安装ETCD软件(适配CentOS 7.9)
bash
cd /tmp
wget https://mirrors.huaweicloud.com/etcd/v3.5.13/etcd-v3.5.13-linux-amd64.tar.gz -O etcd.tar.gz
tar -zxf etcd.tar.gz
mv etcd-v3.5.13-linux-amd64/etcd etcd-v3.5.13-linux-amd64/etcdctl /usr/local/bin/
chmod +x /usr/local/bin/etcd /usr/local/bin/etcdctl
etcd --version
etcdctl version
rm -f /tmp/etcd.tar.gz /tmp/etcd-v3.5.13-linux-amd64
1.3 主节点专属依赖安装(仅etcd01执行)
bash
wget https://github.com/cloudflare/cfssl/releases/download/v1.6.4/cfssl_1.6.4_linux_amd64 -O /usr/local/bin/cfssl
wget https://github.com/cloudflare/cfssl/releases/download/v1.6.4/cfssljson_1.6.4_linux_amd64 -O /usr/local/bin/cfssljson
chmod +x /usr/local/bin/cfssl /usr/local/bin/cfssljson
cfssl version
-
所有命令均以root用户执行。
-
etcd01完成证书和配置生成后,需同步至etcd02、etcd03。
-
证书生成后不可随意修改,生成后建议执行校验。
-
3个节点建议尽量同时启动etcd服务,确保集群正常组建。
-
仅/etc/etcd.env配置文件存在节点间差异,其他文件完全一致。
二、主节点(etcd01)配置部署
步骤1:创建统一目录结构
bash
mkdir -p /opt/cert
cd /opt/cert
mkdir -p /etc/etcd/pki/etcd /var/lib/etcd
步骤2:生成CA证书配置文件
bash
cat > ca-config.json << EOF
{
"signing": {
"default": {
"expiry": "876000h"
},
"profiles": {
"etcd": {
"usages": ["signing", "key encipherment", "server auth", "client auth"],
"expiry": "876000h"
}
}
}
}
EOF
步骤3:生成CA证书请求文件
bash
cat > ca-csr.json << EOF
{
"CN": "etcd-ca",
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "Beijing",
"L": "Beijing",
"O": "etcd",
"OU": "CA"
}
],
"ca": {
"expiry": "876000h"
}
}
EOF
步骤4:生成并校验CA证书(采用ca自签配置,确保100年有效期)
bash
cfssl gencert -initca -config=ca-config.json -profile=etcd ca-csr.json | cfssljson -bare ca
mv ca*.pem /etc/etcd/pki/etcd/
chmod 600 /etc/etcd/pki/etcd/*.pem
openssl x509 -in /etc/etcd/pki/etcd/ca.pem -noout -enddate
步骤5:生成etcd-peer证书请求文件
bash
cat > etcd-peer-csr.json << EOF
{
"CN": "etcd-peer",
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "Beijing",
"L": "Beijing",
"O": "etcd",
"OU": "etcd"
}
],
"hosts": [
"127.0.0.1",
"192.168.133.141",
"192.168.133.142",
"192.168.133.143"
]
}
EOF
步骤6:生成并校验etcd-peer证书
bash
cfssl gencert -ca=/etc/etcd/pki/etcd/ca.pem \
-ca-key=/etc/etcd/pki/etcd/ca-key.pem \
-config=ca-config.json \
-profile=etcd etcd-peer-csr.json | cfssljson -bare etcd-peer
mv etcd-peer*.pem /etc/etcd/pki/etcd/
chmod 600 /etc/etcd/pki/etcd/*.pem
openssl x509 -in /etc/etcd/pki/etcd/etcd-peer.pem -noout -text
openssl verify -CAfile /etc/etcd/pki/etcd/ca.pem /etc/etcd/pki/etcd/etcd-peer.pem
openssl x509 -in /etc/etcd/pki/etcd/etcd-peer.pem -noout -enddate
步骤7:创建etcd环境变量文件
bash
cat > /etc/etcd.env << EOF
# etcd01
ETCD_DATA_DIR=/var/lib/etcd
ETCD_ADVERTISE_CLIENT_URLS=https://192.168.133.141:2379
ETCD_INITIAL_ADVERTISE_PEER_URLS=https://192.168.133.141:2380
ETCD_INITIAL_CLUSTER_STATE=new
ETCD_METRICS=basic
ETCD_LISTEN_CLIENT_URLS=https://192.168.133.141:2379,https://127.0.0.1:2379
ETCD_INITIAL_CLUSTER_TOKEN=k8s_etcd
ETCD_LISTEN_PEER_URLS=https://192.168.133.141:2380
ETCD_NAME=etcd01
ETCD_PROXY=off
ETCD_ENABLE_V2=true
ETCD_INITIAL_CLUSTER=etcd01=https://192.168.133.141:2380,etcd02=https://192.168.133.142:2380,etcd03=https://192.168.133.143:2380
ETCD_ELECTION_TIMEOUT=5000
ETCD_HEARTBEAT_INTERVAL=250
ETCD_AUTO_COMPACTION_RETENTION=8
ETCD_SNAPSHOT_COUNT=10000
ETCD_TRUSTED_CA_FILE=/etc/etcd/pki/etcd/ca.pem
ETCD_CERT_FILE=/etc/etcd/pki/etcd/etcd-peer.pem
ETCD_KEY_FILE=/etc/etcd/pki/etcd/etcd-peer-key.pem
ETCD_CLIENT_CERT_AUTH=true
ETCD_PEER_TRUSTED_CA_FILE=/etc/etcd/pki/etcd/ca.pem
ETCD_PEER_CERT_FILE=/etc/etcd/pki/etcd/etcd-peer.pem
ETCD_PEER_KEY_FILE=/etc/etcd/pki/etcd/etcd-peer-key.pem
ETCD_PEER_CLIENT_CERT_AUTH=true
ETCDCTL_ENDPOINTS=https://127.0.0.1:2379
ETCDCTL_CACERT=/etc/etcd/pki/etcd/ca.pem
ETCDCTL_KEY=/etc/etcd/pki/etcd/etcd-peer-key.pem
ETCDCTL_CERT=/etc/etcd/pki/etcd/etcd-peer.pem
EOF
步骤8:创建etcd系统服务文件
bash
cat > /etc/systemd/system/etcd.service << EOF
[Unit]
Description=etcd
After=network.target
[Service]
User=root
Type=notify
Nice=-20
OOMScoreAdjust=-1000
EnvironmentFile=-/etc/etcd.env
ExecStart=/usr/local/bin/etcd
NotifyAccess=all
RestartSec=10s
LimitNOFILE=40000
Restart=always
[Install]
WantedBy=multi-user.target
EOF
步骤9:主节点启动准备
bash
systemctl daemon-reload
systemctl enable etcd
步骤10:同步配置文件至从节点
10.1 同步至etcd02节点
bash
# 登录etcd02 创建 /etc/etcd/pki/etcd 目录
ssh root@192.168.133.142 "mkdir -p /etc/etcd/pki/etcd"
# 同步证书 至 etcd02
scp -r /etc/etcd/pki/etcd root@192.168.133.142:/etc/etcd/pki/
# 同步 etc 启动文件 至 etcd02
scp /etc/systemd/system/etcd.service root@192.168.133.142:/etc/systemd/system/
bash
# 同步参考案列
[root@etcd01 cert]#
[root@etcd01 cert]# ssh root@192.168.133.142 "mkdir -p /etc/etcd/pki/etcd"
The authenticity of host '192.168.133.142 (192.168.133.142)' can't be established.
ECDSA key fingerprint is SHA256:Rf1ffiQ5+zAc/yCiMWvhOM2Vz0w2SRgqvGDuoySycg0.
ECDSA key fingerprint is MD5:8f:cc:aa:f7:33:c3:03:5f:e7:88:cb:f5:71:bb:cf:a9.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '192.168.133.142' (ECDSA) to the list of known hosts.
root@192.168.133.142's password:
[root@etcd01 cert]#
[root@etcd01 cert]# scp -r /etc/etcd/pki/etcd root@192.168.133.142:/etc/etcd/pki/
root@192.168.133.142's password:
ca-key.pem 100% 1675 1.1MB/s 00:00
ca.pem 100% 1298 914.4KB/s 00:00
etcd-peer-key.pem 100% 1675 1.2MB/s 00:00
etcd-peer.pem 100% 1432 1.1MB/s 00:00
[root@etcd01 cert]#
[root@etcd01 cert]# scp /etc/systemd/system/etcd.service root@192.168.133.142:/etc/systemd/system/
root@192.168.133.142's password:
etcd.service 100% 272 88.7KB/s 00:00
[root@etcd01 cert]#
[root@etcd01 cert]#
10.2 同步至etcd03节点
bash
# 登录etcd02 创建 /etc/etcd/pki/etcd 目录
ssh root@192.168.133.143 "mkdir -p /etc/etcd/pki/etcd"
# 同步证书 至 etcd03
scp -r /etc/etcd/pki/etcd root@192.168.133.143:/etc/etcd/pki/
# 同步 etc 启动文件 至 etcd03
scp /etc/systemd/system/etcd.service root@192.168.133.143:/etc/systemd/system/
三、从节点(etcd02)配置部署
步骤1:创建etcd环境变量文件
bash
cat > /etc/etcd.env << EOF
# etcd02
ETCD_DATA_DIR=/var/lib/etcd
ETCD_ADVERTISE_CLIENT_URLS=https://192.168.133.142:2379
ETCD_INITIAL_ADVERTISE_PEER_URLS=https://192.168.133.142:2380
ETCD_INITIAL_CLUSTER_STATE=new
ETCD_METRICS=basic
ETCD_LISTEN_CLIENT_URLS=https://192.168.133.142:2379,https://127.0.0.1:2379
ETCD_INITIAL_CLUSTER_TOKEN=k8s_etcd
ETCD_LISTEN_PEER_URLS=https://192.168.133.142:2380
ETCD_NAME=etcd02
ETCD_PROXY=off
ETCD_ENABLE_V2=true
ETCD_INITIAL_CLUSTER=etcd01=https://192.168.133.141:2380,etcd02=https://192.168.133.142:2380,etcd03=https://192.168.133.143:2380
ETCD_ELECTION_TIMEOUT=5000
ETCD_HEARTBEAT_INTERVAL=250
ETCD_AUTO_COMPACTION_RETENTION=8
ETCD_SNAPSHOT_COUNT=10000
ETCD_TRUSTED_CA_FILE=/etc/etcd/pki/etcd/ca.pem
ETCD_CERT_FILE=/etc/etcd/pki/etcd/etcd-peer.pem
ETCD_KEY_FILE=/etc/etcd/pki/etcd/etcd-peer-key.pem
ETCD_CLIENT_CERT_AUTH=true
ETCD_PEER_TRUSTED_CA_FILE=/etc/etcd/pki/etcd/ca.pem
ETCD_PEER_CERT_FILE=/etc/etcd/pki/etcd/etcd-peer.pem
ETCD_PEER_KEY_FILE=/etc/etcd/pki/etcd/etcd-peer-key.pem
ETCD_PEER_CLIENT_CERT_AUTH=true
ETCDCTL_ENDPOINTS=https://127.0.0.1:2379
ETCDCTL_CACERT=/etc/etcd/pki/etcd/ca.pem
ETCDCTL_KEY=/etc/etcd/pki/etcd/etcd-peer-key.pem
ETCDCTL_CERT=/etc/etcd/pki/etcd/etcd-peer.pem
EOF
步骤2:从节点启动准备
bash
systemctl daemon-reload
systemctl enable etcd
四、从节点(etcd03)配置部署
步骤1:创建etcd环境变量文件
bash
cat > /etc/etcd.env << EOF
# etcd03
ETCD_DATA_DIR=/var/lib/etcd
ETCD_ADVERTISE_CLIENT_URLS=https://192.168.133.143:2379
ETCD_INITIAL_ADVERTISE_PEER_URLS=https://192.168.133.143:2380
ETCD_INITIAL_CLUSTER_STATE=new
ETCD_METRICS=basic
ETCD_LISTEN_CLIENT_URLS=https://192.168.133.143:2379,https://127.0.0.1:2379
ETCD_INITIAL_CLUSTER_TOKEN=k8s_etcd
ETCD_LISTEN_PEER_URLS=https://192.168.133.143:2380
ETCD_NAME=etcd03
ETCD_PROXY=off
ETCD_ENABLE_V2=true
ETCD_INITIAL_CLUSTER=etcd01=https://192.168.133.141:2380,etcd02=https://192.168.133.142:2380,etcd03=https://192.168.133.143:2380
ETCD_ELECTION_TIMEOUT=5000
ETCD_HEARTBEAT_INTERVAL=250
ETCD_AUTO_COMPACTION_RETENTION=8
ETCD_SNAPSHOT_COUNT=10000
ETCD_TRUSTED_CA_FILE=/etc/etcd/pki/etcd/ca.pem
ETCD_CERT_FILE=/etc/etcd/pki/etcd/etcd-peer.pem
ETCD_KEY_FILE=/etc/etcd/pki/etcd/etcd-peer-key.pem
ETCD_CLIENT_CERT_AUTH=true
ETCD_PEER_TRUSTED_CA_FILE=/etc/etcd/pki/etcd/ca.pem
ETCD_PEER_CERT_FILE=/etc/etcd/pki/etcd/etcd-peer.pem
ETCD_PEER_KEY_FILE=/etc/etcd/pki/etcd/etcd-peer-key.pem
ETCD_PEER_CLIENT_CERT_AUTH=true
ETCDCTL_ENDPOINTS=https://127.0.0.1:2379
ETCDCTL_CACERT=/etc/etcd/pki/etcd/ca.pem
ETCDCTL_KEY=/etc/etcd/pki/etcd/etcd-peer-key.pem
ETCDCTL_CERT=/etc/etcd/pki/etcd/etcd-peer.pem
EOF
步骤2:从节点启动准备
bash
systemctl daemon-reload
systemctl enable etcd
五、集群启动与验证
步骤1:集群同步启动(所有节点尽量同时执行)
bash
systemctl start etcd
步骤2:单节点启动状态验证(所有节点执行)
bash
systemctl status etcd
journalctl -u etcd
步骤3:集群整体状态验证(任意一个节点执行即可)
bash
etcdctl --cacert=/etc/etcd/pki/etcd/ca.pem --cert=/etc/etcd/pki/etcd/etcd-peer.pem --key=/etc/etcd/pki/etcd/etcd-peer-key.pem --endpoints=127.0.0.1:2379 endpoint health -w table
etcdctl --cacert=/etc/etcd/pki/etcd/ca.pem \
--cert=/etc/etcd/pki/etcd/etcd-peer.pem \
--key=/etc/etcd/pki/etcd/etcd-peer-key.pem \
--endpoints=127.0.0.1:2379 member list -w table
etcdctl --cacert=/etc/etcd/pki/etcd/ca.pem \
--cert=/etc/etcd/pki/etcd/etcd-peer.pem \
--key=/etc/etcd/pki/etcd/etcd-peer-key.pem \
--endpoints=https://192.168.133.141:2379,https://192.168.133.142:2379,https://192.168.133.143:2379 \
endpoint status -w table
步骤4:集群验证失败则在所有节点重复执行以下命令直到集群正常
bash
systemctl stop etcd && rm -rvf /var/lib/etcd/
# 所有节点尽量保证同时执行启动命令,然后重复 步骤 3 验证。
systemctl start etcd # 所有节点尽量同时执行
# 如果启动不同时,很容易失败,需要重复 步骤 4 直到集群正常。
附、集群正常运行状态(即: 第五章 步骤3 的结果截图)
bash
[root@etcd01 ~]#
[root@etcd01 ~]# etcdctl --cacert=/etc/etcd/pki/etcd/ca.pem --cert=/etc/etcd/pki/etcd/etcd-peer.pem --key=/etc/etcd/pki/etcd/etcd-peer-key.pem --endpoints=127.0.0.1:2379 endpoint health -w table
+----------------+--------+------------+-------+
| ENDPOINT | HEALTH | TOOK | ERROR |
+----------------+--------+------------+-------+
| 127.0.0.1:2379 | true | 9.126683ms | |
+----------------+--------+------------+-------+
[root@etcd01 ~]#
[root@etcd01 ~]#
[root@etcd01 ~]# etcdctl --cacert=/etc/etcd/pki/etcd/ca.pem \
> --cert=/etc/etcd/pki/etcd/etcd-peer.pem \
> --key=/etc/etcd/pki/etcd/etcd-peer-key.pem \
> --endpoints=127.0.0.1:2379 member list -w table
+------------------+---------+--------+------------------------------+------------------------------+------------+
| ID | STATUS | NAME | PEER ADDRS | CLIENT ADDRS | IS LEARNER |
+------------------+---------+--------+------------------------------+------------------------------+------------+
| 7a35cc86cb8d3d14 | started | etcd01 | https://192.168.133.141:2380 | https://192.168.133.141:2379 | false |
| 8d2e290c7f8dc0c5 | started | etcd03 | https://192.168.133.143:2380 | https://192.168.133.143:2379 | false |
| fa389007ecd0613e | started | etcd02 | https://192.168.133.142:2380 | https://192.168.133.142:2379 | false |
+------------------+---------+--------+------------------------------+------------------------------+------------+
[root@etcd01 ~]#
[root@etcd01 ~]#
[root@etcd01 ~]# etcdctl --cacert=/etc/etcd/pki/etcd/ca.pem \
> --cert=/etc/etcd/pki/etcd/etcd-peer.pem \
> --key=/etc/etcd/pki/etcd/etcd-peer-key.pem \
> --endpoints=https://192.168.133.141:2379,https://192.168.133.142:2379,https://192.168.133.143:2379 \
> endpoint status -w table
+------------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| ENDPOINT | ID | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+------------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| https://192.168.133.141:2379 | 7a35cc86cb8d3d14 | 3.5.10 | 25 kB | true | false | 2 | 13 | 13 | |
| https://192.168.133.142:2379 | fa389007ecd0613e | 3.5.10 | 25 kB | false | false | 2 | 13 | 13 | |
| https://192.168.133.143:2379 | 8d2e290c7f8dc0c5 | 3.5.10 | 25 kB | false | false | 2 | 13 | 13 | |
+------------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
[root@etcd01 ~]#
[root@etcd01 ~]#
