目录
更多精彩博文详见：
《CentOS7二进制安装包方式部署K8S集群之CA根证书生成》
《CentOS7二进制安装包方式部署K8S集群之ETCD集群部署》
本文主要介绍了在CentOS 7系统下，使用二进制安装包方式部署K8S集群中etcd集群部署的相关内容。本次Kubernetes集群使用分布式键值存储系统etcd进行数据存储，其中etcd采取三节点集群方式部署。为了节约服务器资源，etcd集群节点复用Kubernetes集群节点。
etcd集群节点规划如下：
|----------------|-----------------|----------|
| 主机名称 | IP | 部署组件 |
| K8s-controller | 192.168.110.150 | etcd-01 |
| K8s-node01 | 192.168.110.151 | etcd-02 |
| K8s-node02 | 192.168.110.152 | etcd-03 |
1、集群各节点创建etcd相关目录
此处集群各节点分别创建如下目录：
- /K8s/etcd/bin：该目录下为etcd服务相关的可执行文件。
- /K8s/etcd/data：该目录为etcd服务的数据文件目录。
- /K8s/etcd/ssl：该目录下为etcd服务相关的证书文件。
(1)、K8s-controller节点
root@K8s-controller \~\]# mkdir -p /K8s/etcd/{bin,data,ssl} \[root@K8s-controller \~\]# ll /K8s/etcd/ 
(2)、K8s-node01节点
root@K8s-node01 \~\]# mkdir -p /K8s/etcd/{bin,data,ssl} \[root@K8s-node01 \~\]# ll /K8s/etcd/ 
(3)、K8s-node02节点
root@K8s-node02 \~\]# mkdir -p /K8s/etcd/{bin,data,ssl}
\[root@K8s-node02 \~\]# ll /K8s/etcd/
#### 2、创建etcd集群服务证书和私钥
##### (1)、创建etcd集群服务证书签名请求文件
> * **创建etcd集群服务证书文件存放目录**
>
> \[root@K8s-controller \~\]# mkdir -pv /K8s/cfssl/cert_file/etcd
> 
>
> * **创建etcd集群服务证书签名请求文件etcd-csr.json**
>
> \[root@K8s-controller \~\]# cat \> /K8s/cfssl/cert_file/etcd/etcd-server-csr.json \<\
}
EOF
hosts字段是etcd集群节点IP地址。
(2)、签发etcd-server证书文件和私钥文件
使用etcd集群CA根证书、CA根证书私钥、CA根证书配置文件、etcd-server证书签名请求json文件共同签发etcd-server证书文件和私钥文件。此处会生成三个文件，分别是证书etcd-server.pem、证书私钥etcd-server-key.pem、证书签名请求etcd-server.csr（用于交叉签名或重新签名）。
root@K8s-controller \~\]# cd /K8s/cfssl/cert_file/etcd/ \[root@K8s-controller etcd\]# cfssl gencert -ca=/K8s/cfssl/cert_file/etcd-ca.pem -ca-key=/K8s/cfssl/cert_file/etcd-ca-key.pem -config=/K8s/cfssl/cert_file/etcd-ca-config.json -profile=etcd /K8s/cfssl/cert_file/etcd/etcd-server-csr.json \| cfssljson -bare etcd-server 
(4)、分发etcd证书文件至etcd集群各节点
此处分发etcd集群CA根证书文件etcd-ca.pem、etcd集群服务证书文件etcd-server.pem、etcd私钥文件etcd-server-key.pem。
1）分发到K8s-controller节点
root@K8s-controller \~\]# cp /K8s/cfssl/cert_file/etcd/etcd-server.pem /K8s/cfssl/cert_file/etcd/etcd-server-key.pem /K8s/cfssl/cert_file/etcd-ca.pem /K8s/etcd/ssl/ \[root@K8s-controller \~\]# ll /K8s/etcd/ssl/ 
2）分发到K8s-node01节点
- K8s-controller节点分发
root@K8s-controller \~\]# scp -P 22 /K8s/cfssl/cert_file/etcd/etcd-server.pem /K8s/cfssl/cert_file/etcd/etcd-server-key.pem /K8s/cfssl/cert_file/etcd-ca.pem root@192.168.110.151:/K8s/etcd/ssl/  * **K8s-node01节点查看** \[root@K8s-node01 \~\]# ll /K8s/etcd/ssl/ 
3）分发到K8s-node02节点
- K8s-controller节点分发
root@K8s-controller \~\]# scp -P 22 /K8s/cfssl/cert_file/etcd/etcd-server.pem /K8s/cfssl/cert_file/etcd/etcd-server-key.pem /K8s/cfssl/cert_file/etcd-ca.pem root@192.168.110.152:/K8s/etcd/ssl/  * **K8s-node02节点查看** \[root@K8s-node02 \~\]# ll /K8s/etcd/ssl/ 
3、下载etcd安装文件
(1)、下载etcd安装文件
此处部署的是v3.4.18-linux-amd64版本，可根据实际环境选择相应的版本。
root@K8s-controller \~\]# cd /K8s/etcd/ \[root@K8s-controller etcd\]# wget https://github.com/etcd-io/etcd/releases/download/v3.4.18/etcd-v3.4.18-linux-amd64.tar.gz  
(2)、分发etcd安装文件
1）分发到K8s-node01节点
root@K8s-controller \~\]# scp -P 22 /K8s/etcd/etcd-v3.4.18-linux-amd64.tar.gz root@192.168.110.151:/K8s/etcd/  \[root@K8s-node01 \~\]# ll /K8s/etcd/ 
2）分发到K8s-node02节点
root@K8s-controller \~\]# scp -P 22 /K8s/etcd/etcd-v3.4.18-linux-amd64.tar.gz root@192.168.110.152:/K8s/etcd/  \[root@K8s-node02 \~\]# ll /K8s/etcd/ 
4、部署集群各节点etcd服务
(1)、K8s-controller节点
1）解压etcd安装文件
- 解压etcd安装文件
root@K8s-controller \~\]# cd /K8s/etcd/ \[root@K8s-controller etcd\]# tar -xzvf etcd-v3.4.18-linux-amd64.tar.gz  * 复制解压后的可执行文件到etcd的bin目录 \[root@K8s-controller etcd\]# cp /K8s/etcd/etcd-v3.4.18-linux-amd64/{etcd,etcdctl} /K8s/etcd/bin/ \[root@K8s-controller etcd\]# ll /K8s/etcd/bin/  * 创建etcdctl可执行文件软链接 \[root@K8s-controller etcd\]# ln -s /K8s/etcd/bin/etcdctl /usr/bin/etcdctl \[root@K8s-controller etcd\]# ll /usr/bin/etcdctl 
2）创建etcd服务配置文件
此处创建etcd服务配置文件/K8s/etcd/etcd.conf
root@K8s-controller \~\]# cat \> /K8s/etcd/etcd.conf \<\< EOF #\[member
ETCD_NAME：节点名称，集群中唯一，此处指集群节点01
ETCD_NAME="etcd-01"
ETCD_DATA_DIR：当前集群节点的数据目录
ETCD_DATA_DIR="/K8s/etcd/data"
ETCD_LISTEN_PEER_URLS：集群通信的监听地址
ETCD_LISTEN_PEER_URLS="https://192.168.110.150:2380"
ETCD_LISTEN_CLIENT_URLS：客户端访问的监听地址
ETCD_LISTEN_CLIENT_URLS="https://192.168.110.150:2379,http://127.0.0.1:2379"
#[cluster]
ETCD_INITIAL_ADVERTISE_PEER_URLS：集群的通告地址
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.110.150:2380"
ETCD_ADVERTISE_CLIENT_URLS：客户端的通告地址
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.110.150:2379"
ETCD_INITIAL_CLUSTER：集群的节点地址
ETCD_INITIAL_CLUSTER="etcd-01=https://192.168.110.150:2380,etcd-02=https://192.168.110.151:2380,etcd-03=https://192.168.110.152:2380"
ETCD_INITIAL_CLUSTER_TOKEN：集群Token
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE：即将加入的集群的当前状态，new是新建的集群，existing表示加入已存在的集群。
ETCD_INITIAL_CLUSTER_STATE="new"
EOF
3）创建etcd.service文件
创建etcd.service文件，通过systemd来管理etcd服务。
root@K8s-controller \~\]# cat \> /usr/lib/systemd/system/etcd.service \<\< EOF \[Unit
Description=etcd server
After=network.target
After=network-online.target
Wants=network-online.target
Service
Type=notify
EnvironmentFile=/K8s/etcd/etcd.confExecStart=/K8s/etcd/bin/etcd --cert-file=/K8s/etcd/ssl/etcd-server.pem --key-file=/K8s/etcd/ssl/etcd-server-key.pem --peer-cert-file=/K8s/etcd/ssl/etcd-server.pem --peer-key-file=/K8s/etcd/ssl/etcd-server-key.pem --trusted-ca-file=/K8s/etcd/ssl/etcd-ca.pem --peer-trusted-ca-file=/K8s/etcd/ssl/etcd-ca.pem --logger=zap
Restart=on-failure
LimitNOFILE=65536
Install
WantedBy=multi-user.target
EOF
(2)、K8s-node01节点
1）解压etcd安装文件
- 解压etcd安装文件
root@K8s-node01 \~\]# cd /K8s/etcd/ \[root@K8s-node01 etcd\]# tar -xzvf etcd-v3.4.18-linux-amd64.tar.gz  * 复制解压后的可执行文件到etcd的bin目录 \[root@K8s-node01 etcd\]# cp /K8s/etcd/etcd-v3.4.18-linux-amd64/{etcd,etcdctl} /K8s/etcd/bin/ \[root@K8s-node01 etcd\]# ll /K8s/etcd/bin/  * 创建etcdctl可执行文件软链接 \[root@K8s-node01 etcd\]# ln -s /K8s/etcd/bin/etcdctl /usr/bin/etcdctl \[root@K8s-node01 etcd\]# ll /usr/bin/etcdctl 
2）创建etcd服务配置文件
此处创建etcd服务配置文件/K8s/etcd/etcd.conf
root@K8s-node01 \~\]# cat \> /K8s/etcd/etcd.conf \<\< EOF #\[member
ETCD_NAME：节点名称，集群中唯一，此处指集群节点01
ETCD_NAME="etcd-02"
ETCD_DATA_DIR：当前集群节点的数据目录
ETCD_DATA_DIR="/K8s/etcd/data"
ETCD_LISTEN_PEER_URLS：集群通信的监听地址
ETCD_LISTEN_PEER_URLS="https://192.168.110.151:2380"
ETCD_LISTEN_CLIENT_URLS：客户端访问的监听地址
ETCD_LISTEN_CLIENT_URLS="https://192.168.110.151:2379,http://127.0.0.1:2379"
#[cluster]
ETCD_INITIAL_ADVERTISE_PEER_URLS：集群的通告地址
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.110.151:2380"
ETCD_ADVERTISE_CLIENT_URLS：客户端的通告地址
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.110.151:2379"
ETCD_INITIAL_CLUSTER：集群的节点地址
ETCD_INITIAL_CLUSTER="etcd-01=https://192.168.110.150:2380,etcd-02=https://192.168.110.151:2380,etcd-03=https://192.168.110.152:2380"
ETCD_INITIAL_CLUSTER_TOKEN：集群Token
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE：即将加入的集群的当前状态，new是新建的集群，existing表示加入已存在的集群。
ETCD_INITIAL_CLUSTER_STATE="new"
EOF
3）创建etcd.service文件
创建etcd.service文件，通过systemd来管理etcd服务。
root@K8s-node01 \~\]# cat \> /usr/lib/systemd/system/etcd.service \<\< EOF \[Unit
Description=etcd server
After=network.target
After=network-online.target
Wants=network-online.target
Service
Type=notify
EnvironmentFile=/K8s/etcd/etcd.conf
ExecStart=/K8s/etcd/bin/etcd --cert-file=/K8s/etcd/ssl/etcd-server.pem --key-file=/K8s/etcd/ssl/etcd-server-key.pem --peer-cert-file=/K8s/etcd/ssl/etcd-server.pem --peer-key-file=/K8s/etcd/ssl/etcd-server-key.pem --trusted-ca-file=/K8s/etcd/ssl/etcd-ca.pem --peer-trusted-ca-file=/K8s/etcd/ssl/etcd-ca.pem --logger=zap
Restart=on-failure
LimitNOFILE=65536
Install
WantedBy=multi-user.target
EOF
(3)、K8s-node02节点
1）解压etcd安装文件
- 解压etcd安装文件
root@K8s-node02 \~\]# cd /K8s/etcd/ \[root@K8s-node02 etcd\]# tar -xzvf etcd-v3.4.18-linux-amd64.tar.gz  * 复制解压后的可执行文件到etcd的bin目录 \[root@K8s-node02 etcd\]# cp /K8s/etcd/etcd-v3.4.18-linux-amd64/{etcd,etcdctl} /K8s/etcd/bin/ \[root@K8s-node02 etcd\]# ll /K8s/etcd/bin/  * 创建etcdctl可执行文件软链接 \[root@K8s-node02 etcd\]# ln -s /K8s/etcd/bin/etcdctl /usr/bin/etcdctl \[root@K8s-node02 etcd\]# ll /usr/bin/etcdctl 
2）创建etcd服务配置文件
此处创建etcd服务配置文件/K8s/etcd/etcd.conf
root@K8s-node02 \~\]# cat \> /K8s/etcd/etcd.conf \<\< EOF #\[member
ETCD_NAME：节点名称，集群中唯一，此处指集群节点01
ETCD_NAME="etcd-03"
ETCD_DATA_DIR：当前集群节点的数据目录
ETCD_DATA_DIR="/K8s/etcd/data"
ETCD_LISTEN_PEER_URLS：集群通信的监听地址
ETCD_LISTEN_PEER_URLS="https://192.168.110.152:2380"
ETCD_LISTEN_CLIENT_URLS：客户端访问的监听地址
ETCD_LISTEN_CLIENT_URLS="https://192.168.110.152:2379,http://127.0.0.1:2379"
#[cluster]
ETCD_INITIAL_ADVERTISE_PEER_URLS：集群的通告地址
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.110.152:2380"
ETCD_ADVERTISE_CLIENT_URLS：客户端的通告地址
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.110.152:2379"
ETCD_INITIAL_CLUSTER：集群的节点地址
ETCD_INITIAL_CLUSTER="etcd-01=https://192.168.110.150:2380,etcd-02=https://192.168.110.151:2380,etcd-03=https://192.168.110.152:2380"
ETCD_INITIAL_CLUSTER_TOKEN：集群Token
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE：即将加入的集群的当前状态，new是新建的集群，existing表示加入已存在的集群。
ETCD_INITIAL_CLUSTER_STATE="new"
EOF
3）创建etcd.service文件
创建etcd.service文件，通过systemd来管理etcd服务。
root@K8s-node02 \~\]# cat \> /usr/lib/systemd/system/etcd.service \<\< EOF \[Unit
Description=etcd server
After=network.target
After=network-online.target
Wants=network-online.target
Service
Type=notify
EnvironmentFile=/K8s/etcd/etcd.conf
ExecStart=/K8s/etcd/bin/etcd --cert-file=/K8s/etcd/ssl/etcd-server.pem --key-file=/K8s/etcd/ssl/etcd-server-key.pem --peer-cert-file=/K8s/etcd/ssl/etcd-server.pem --peer-key-file=/K8s/etcd/ssl/etcd-server-key.pem --trusted-ca-file=/K8s/etcd/ssl/etcd-ca.pem --peer-trusted-ca-file=/K8s/etcd/ssl/etcd-ca.pem --logger=zap
Restart=on-failure
LimitNOFILE=65536
Install
WantedBy=multi-user.target
EOF
5、启动集群各节点etcd服务
此处需要注意的是，至少同时启动两个集群节点，否则会有集群节点连接失败的信息，导致集群启动失败。
当集群无法正常启动时，可以使用命令"journalctl -xeu etcd.service"或者命令"journalctl -u etcd.service"查看启动失败的原因。
(1)、启动K8s-controller节点etcd服务
1）设置开机启动etcd服务
root@K8s-controller \~\]# systemctl enable etcd 
2）启动etcd服务
root@K8s-controller \~\]# systemctl start etcd.service \[root@K8s-controller \~\]# systemctl status etcd.service  \[root@K8s-controller \~\]# netstat -tnlp 
(2)、启动K8s-node01节点etcd服务
1）设置开机启动etcd服务
root@K8s-node01 \~\]# systemctl enable etcd 
2）启动etcd服务
root@K8s-node01 \~\]# systemctl start etcd.service \[root@K8s-node01 \~\]# systemctl status etcd.service  \[root@K8s-node01 \~\]# netstat -tnlp 
(3)、启动K8s-node02节点etcd服务
1）设置开机启动etcd服务
root@K8s-node02 \~\]# systemctl enable etcd 
2）启动etcd服务
root@K8s-node02 \~\]# systemctl start etcd.service \[root@K8s-node02 \~\]# systemctl status etcd.service  \[root@K8s-node02 data\]# netstat -tnlp 
6、验证etcd集群状态
(1)、查看etcd集群健康状态
root@K8s-controller \~\]# ETCDCTL_API=3 etcdctl --cacert=/K8s/etcd/ssl/etcd-ca.pem --cert=/K8s/etcd/ssl/etcd-server.pem --key=/K8s/etcd/ssl/etcd-server-key.pem --endpoints="https://192.168.110.150:2379,https://192.168.110.151:2379,https://192.168.110.152:2379" endpoint health 
由上可见，etcd三个集群节点状态均为健康。
(2)、查看etcd集群LEADER节点
root@K8s-controller \~\]# ETCDCTL_API=3 etcdctl -w table --cacert=/K8s/etcd/ssl/etcd-ca.pem --cert=/K8s/etcd/ssl/etcd-server.pem --key=/K8s/etcd/ssl/etcd-server-key.pem --endpoints="https://192.168.110.150:2379,https://192.168.110.151:2379,https://192.168.110.152:2379" endpoint status  由上可见，192.168.110.151节点(K8s-controller)为LEADER节点