CentOS7二进制安装包方式部署K8S集群之ETCD集群部署

|----------------|-----------------|----------|
| 主机名称 | IP | 部署组件 |
| K8s-controller | 192.168.110.150 | etcd-01 |
| K8s-node01 | 192.168.110.151 | etcd-02 |
| K8s-node02 | 192.168.110.152 | etcd-03 |

1、集群各节点创建etcd相关目录

此处集群各节点分别创建如下目录：

/K8s/etcd/bin：该目录下为etcd服务相关的可执行文件。

/K8s/etcd/data：该目录为etcd服务的数据文件目录。

/K8s/etcd/ssl：该目录下为etcd服务相关的证书文件。

(1)、K8s-controller节点

$root@K8s-controller \~$ # mkdir -p /K8s/etcd/{bin,data,ssl}

$root@K8s-controller \~$ # ll /K8s/etcd/

(2)、K8s-node01节点

$root@K8s-node01 \~$ # mkdir -p /K8s/etcd/{bin,data,ssl}

$root@K8s-node01 \~$ # ll /K8s/etcd/

(3)、K8s-node02节点

$root@K8s-node02 \~$ # mkdir -p /K8s/etcd/{bin,data,ssl}

$root@K8s-node02 \~$ # ll /K8s/etcd/

2、创建etcd集群服务证书和私钥

(1)、创建etcd集群服务证书签名请求文件

创建etcd集群服务证书文件存放目录

$root@K8s-controller \~$ # mkdir -pv /K8s/cfssl/cert_file/etcd

创建etcd集群服务证书签名请求文件etcd-csr.json

$root@K8s-controller \~$ # cat > /K8s/cfssl/cert_file/etcd/etcd-server-csr.json <<EOF

{

"CN": "etcd-server",

"hosts": [

"192.168.110.150",

"192.168.110.151",

"192.168.110.152"

],

"key": {

"algo": "rsa",

"size": 2048

},

"names": [

{

"C": "CN",

"L": "Nanning",

"ST": "Guangxi",

"O": "k8s",

"OU": "lbj"

}

]

}

EOF

hosts字段是etcd集群节点IP地址。

(2)、签发etcd-server证书文件和私钥文件

使用etcd集群CA根证书、CA根证书私钥、CA根证书配置文件、etcd-server证书签名请求json文件共同签发etcd-server证书文件和私钥文件。此处会生成三个文件，分别是证书etcd-server.pem、证书私钥etcd-server-key.pem、证书签名请求etcd-server.csr（用于交叉签名或重新签名）。

$root@K8s-controller \~$ # cd /K8s/cfssl/cert_file/etcd/

$root@K8s-controller etcd$ # cfssl gencert -ca=/K8s/cfssl/cert_file/etcd-ca.pem -ca-key=/K8s/cfssl/cert_file/etcd-ca-key.pem -config=/K8s/cfssl/cert_file/etcd-ca-config.json -profile=etcd /K8s/cfssl/cert_file/etcd/etcd-server-csr.json | cfssljson -bare etcd-server

(4)、分发etcd证书文件至etcd集群各节点

此处分发etcd集群CA根证书文件etcd-ca.pem、etcd集群服务证书文件etcd-server.pem、etcd私钥文件etcd-server-key.pem。

1）分发到K8s-controller节点

$root@K8s-controller \~$ # cp /K8s/cfssl/cert_file/etcd/etcd-server.pem /K8s/cfssl/cert_file/etcd/etcd-server-key.pem /K8s/cfssl/cert_file/etcd-ca.pem /K8s/etcd/ssl/

$root@K8s-controller \~$ # ll /K8s/etcd/ssl/

2）分发到K8s-node01节点

K8s-controller节点分发

$root@K8s-controller \~$ # scp -P 22 /K8s/cfssl/cert_file/etcd/etcd-server.pem /K8s/cfssl/cert_file/etcd/etcd-server-key.pem /K8s/cfssl/cert_file/etcd-ca.pem root@192.168.110.151:/K8s/etcd/ssl/

K8s-node01节点查看

$root@K8s-node01 \~$ # ll /K8s/etcd/ssl/

3）分发到K8s-node02节点

K8s-controller节点分发

$root@K8s-controller \~$ # scp -P 22 /K8s/cfssl/cert_file/etcd/etcd-server.pem /K8s/cfssl/cert_file/etcd/etcd-server-key.pem /K8s/cfssl/cert_file/etcd-ca.pem root@192.168.110.152:/K8s/etcd/ssl/

K8s-node02节点查看

$root@K8s-node02 \~$ # ll /K8s/etcd/ssl/

3、下载etcd安装文件

(1)、下载etcd安装文件

此处部署的是v3.4.18-linux-amd64版本，可根据实际环境选择相应的版本。

$root@K8s-controller \~$ # cd /K8s/etcd/

$root@K8s-controller etcd$ # wget https://github.com/etcd-io/etcd/releases/download/v3.4.18/etcd-v3.4.18-linux-amd64.tar.gz

(2)、分发etcd安装文件

1）分发到K8s-node01节点

$root@K8s-controller \~$ # scp -P 22 /K8s/etcd/etcd-v3.4.18-linux-amd64.tar.gz root@192.168.110.151:/K8s/etcd/

$root@K8s-node01 \~$ # ll /K8s/etcd/

2）分发到K8s-node02节点

$root@K8s-controller \~$ # scp -P 22 /K8s/etcd/etcd-v3.4.18-linux-amd64.tar.gz root@192.168.110.152:/K8s/etcd/

$root@K8s-node02 \~$ # ll /K8s/etcd/

4、部署集群各节点etcd服务

(1)、K8s-controller节点

1）解压etcd安装文件

解压etcd安装文件

$root@K8s-controller \~$ # cd /K8s/etcd/

$root@K8s-controller etcd$ # tar -xzvf etcd-v3.4.18-linux-amd64.tar.gz

复制解压后的可执行文件到etcd的bin目录

$root@K8s-controller etcd$ # cp /K8s/etcd/etcd-v3.4.18-linux-amd64/{etcd,etcdctl} /K8s/etcd/bin/

$root@K8s-controller etcd$ # ll /K8s/etcd/bin/

创建etcdctl可执行文件软链接

$root@K8s-controller etcd$ # ln -s /K8s/etcd/bin/etcdctl /usr/bin/etcdctl

$root@K8s-controller etcd$ # ll /usr/bin/etcdctl

2）创建etcd服务配置文件

此处创建etcd服务配置文件/K8s/etcd/etcd.conf

$root@K8s-controller \~$ # cat > /K8s/etcd/etcd.conf << EOF

# $member$

ETCD_NAME：节点名称，集群中唯一，此处指集群节点01

ETCD_NAME="etcd-01"

ETCD_DATA_DIR：当前集群节点的数据目录

ETCD_DATA_DIR="/K8s/etcd/data"

ETCD_LISTEN_PEER_URLS：集群通信的监听地址

ETCD_LISTEN_PEER_URLS="https://192.168.110.150:2380"

ETCD_LISTEN_CLIENT_URLS：客户端访问的监听地址

ETCD_LISTEN_CLIENT_URLS="https://192.168.110.150:2379,http://127.0.0.1:2379"

# $cluster$

ETCD_INITIAL_ADVERTISE_PEER_URLS：集群的通告地址

ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.110.150:2380"

ETCD_ADVERTISE_CLIENT_URLS：客户端的通告地址

ETCD_ADVERTISE_CLIENT_URLS="https://192.168.110.150:2379"

ETCD_INITIAL_CLUSTER：集群的节点地址

ETCD_INITIAL_CLUSTER="etcd-01=https://192.168.110.150:2380,etcd-02=https://192.168.110.151:2380,etcd-03=https://192.168.110.152:2380"

ETCD_INITIAL_CLUSTER_TOKEN：集群Token

ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"

ETCD_INITIAL_CLUSTER_STATE：即将加入的集群的当前状态，new是新建的集群，existing表示加入已存在的集群。

ETCD_INITIAL_CLUSTER_STATE="new"

EOF

3）创建etcd.service文件

创建etcd.service文件，通过systemd来管理etcd服务。

$root@K8s-controller \~$ # cat > /usr/lib/systemd/system/etcd.service << EOF
$Unit$
Description=etcd server

After=network.target

After=network-online.target

Wants=network-online.target
$Service$
Type=notify

EnvironmentFile=/K8s/etcd/etcd.confExecStart=/K8s/etcd/bin/etcd --cert-file=/K8s/etcd/ssl/etcd-server.pem --key-file=/K8s/etcd/ssl/etcd-server-key.pem --peer-cert-file=/K8s/etcd/ssl/etcd-server.pem --peer-key-file=/K8s/etcd/ssl/etcd-server-key.pem --trusted-ca-file=/K8s/etcd/ssl/etcd-ca.pem --peer-trusted-ca-file=/K8s/etcd/ssl/etcd-ca.pem --logger=zap

Restart=on-failure

LimitNOFILE=65536
$Install$
WantedBy=multi-user.target

EOF

(2)、K8s-node01节点

1）解压etcd安装文件

解压etcd安装文件

$root@K8s-node01 \~$ # cd /K8s/etcd/

$root@K8s-node01 etcd$ # tar -xzvf etcd-v3.4.18-linux-amd64.tar.gz

复制解压后的可执行文件到etcd的bin目录

$root@K8s-node01 etcd$ # cp /K8s/etcd/etcd-v3.4.18-linux-amd64/{etcd,etcdctl} /K8s/etcd/bin/

$root@K8s-node01 etcd$ # ll /K8s/etcd/bin/

创建etcdctl可执行文件软链接

$root@K8s-node01 etcd$ # ln -s /K8s/etcd/bin/etcdctl /usr/bin/etcdctl

$root@K8s-node01 etcd$ # ll /usr/bin/etcdctl

2）创建etcd服务配置文件

此处创建etcd服务配置文件/K8s/etcd/etcd.conf

$root@K8s-node01 \~$ # cat > /K8s/etcd/etcd.conf << EOF

# $member$

ETCD_NAME：节点名称，集群中唯一，此处指集群节点01

ETCD_NAME="etcd-02"

ETCD_DATA_DIR：当前集群节点的数据目录

ETCD_DATA_DIR="/K8s/etcd/data"

ETCD_LISTEN_PEER_URLS：集群通信的监听地址

ETCD_LISTEN_PEER_URLS="https://192.168.110.151:2380"

ETCD_LISTEN_CLIENT_URLS：客户端访问的监听地址

ETCD_LISTEN_CLIENT_URLS="https://192.168.110.151:2379,http://127.0.0.1:2379"

# $cluster$

ETCD_INITIAL_ADVERTISE_PEER_URLS：集群的通告地址

ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.110.151:2380"

ETCD_ADVERTISE_CLIENT_URLS：客户端的通告地址

ETCD_ADVERTISE_CLIENT_URLS="https://192.168.110.151:2379"

ETCD_INITIAL_CLUSTER：集群的节点地址

ETCD_INITIAL_CLUSTER="etcd-01=https://192.168.110.150:2380,etcd-02=https://192.168.110.151:2380,etcd-03=https://192.168.110.152:2380"

ETCD_INITIAL_CLUSTER_TOKEN：集群Token

ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"

ETCD_INITIAL_CLUSTER_STATE：即将加入的集群的当前状态，new是新建的集群，existing表示加入已存在的集群。

ETCD_INITIAL_CLUSTER_STATE="new"

EOF

3）创建etcd.service文件

创建etcd.service文件，通过systemd来管理etcd服务。

$root@K8s-node01 \~$ # cat > /usr/lib/systemd/system/etcd.service << EOF
$Unit$
Description=etcd server

After=network.target

After=network-online.target

Wants=network-online.target
$Service$
Type=notify

EnvironmentFile=/K8s/etcd/etcd.conf

ExecStart=/K8s/etcd/bin/etcd --cert-file=/K8s/etcd/ssl/etcd-server.pem --key-file=/K8s/etcd/ssl/etcd-server-key.pem --peer-cert-file=/K8s/etcd/ssl/etcd-server.pem --peer-key-file=/K8s/etcd/ssl/etcd-server-key.pem --trusted-ca-file=/K8s/etcd/ssl/etcd-ca.pem --peer-trusted-ca-file=/K8s/etcd/ssl/etcd-ca.pem --logger=zap

Restart=on-failure

LimitNOFILE=65536
$Install$
WantedBy=multi-user.target

EOF

(3)、K8s-node02节点

1）解压etcd安装文件

解压etcd安装文件

$root@K8s-node02 \~$ # cd /K8s/etcd/

$root@K8s-node02 etcd$ # tar -xzvf etcd-v3.4.18-linux-amd64.tar.gz

复制解压后的可执行文件到etcd的bin目录

$root@K8s-node02 etcd$ # cp /K8s/etcd/etcd-v3.4.18-linux-amd64/{etcd,etcdctl} /K8s/etcd/bin/

$root@K8s-node02 etcd$ # ll /K8s/etcd/bin/

创建etcdctl可执行文件软链接

$root@K8s-node02 etcd$ # ln -s /K8s/etcd/bin/etcdctl /usr/bin/etcdctl

$root@K8s-node02 etcd$ # ll /usr/bin/etcdctl

2）创建etcd服务配置文件

此处创建etcd服务配置文件/K8s/etcd/etcd.conf

$root@K8s-node02 \~$ # cat > /K8s/etcd/etcd.conf << EOF

# $member$

ETCD_NAME：节点名称，集群中唯一，此处指集群节点01

ETCD_NAME="etcd-03"

ETCD_DATA_DIR：当前集群节点的数据目录

ETCD_DATA_DIR="/K8s/etcd/data"

ETCD_LISTEN_PEER_URLS：集群通信的监听地址

ETCD_LISTEN_PEER_URLS="https://192.168.110.152:2380"

ETCD_LISTEN_CLIENT_URLS：客户端访问的监听地址

ETCD_LISTEN_CLIENT_URLS="https://192.168.110.152:2379,http://127.0.0.1:2379"

# $cluster$

ETCD_INITIAL_ADVERTISE_PEER_URLS：集群的通告地址

ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.110.152:2380"

ETCD_ADVERTISE_CLIENT_URLS：客户端的通告地址

ETCD_ADVERTISE_CLIENT_URLS="https://192.168.110.152:2379"

ETCD_INITIAL_CLUSTER：集群的节点地址

ETCD_INITIAL_CLUSTER="etcd-01=https://192.168.110.150:2380,etcd-02=https://192.168.110.151:2380,etcd-03=https://192.168.110.152:2380"

ETCD_INITIAL_CLUSTER_TOKEN：集群Token

ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"

ETCD_INITIAL_CLUSTER_STATE：即将加入的集群的当前状态，new是新建的集群，existing表示加入已存在的集群。

ETCD_INITIAL_CLUSTER_STATE="new"

EOF

3）创建etcd.service文件

创建etcd.service文件，通过systemd来管理etcd服务。

$root@K8s-node02 \~$ # cat > /usr/lib/systemd/system/etcd.service << EOF
$Unit$
Description=etcd server

After=network.target

After=network-online.target

Wants=network-online.target
$Service$
Type=notify

EnvironmentFile=/K8s/etcd/etcd.conf

ExecStart=/K8s/etcd/bin/etcd --cert-file=/K8s/etcd/ssl/etcd-server.pem --key-file=/K8s/etcd/ssl/etcd-server-key.pem --peer-cert-file=/K8s/etcd/ssl/etcd-server.pem --peer-key-file=/K8s/etcd/ssl/etcd-server-key.pem --trusted-ca-file=/K8s/etcd/ssl/etcd-ca.pem --peer-trusted-ca-file=/K8s/etcd/ssl/etcd-ca.pem --logger=zap

Restart=on-failure

LimitNOFILE=65536
$Install$
WantedBy=multi-user.target

EOF

5、启动集群各节点etcd服务

此处需要注意的是，至少同时启动两个集群节点，否则会有集群节点连接失败的信息，导致集群启动失败。

当集群无法正常启动时，可以使用命令"journalctl -xeu etcd.service"或者命令"journalctl -u etcd.service"查看启动失败的原因。

(1)、启动K8s-controller节点etcd服务

1）设置开机启动etcd服务

$root@K8s-controller \~$ # systemctl enable etcd

2）启动etcd服务

$root@K8s-controller \~$ # systemctl start etcd.service

$root@K8s-controller \~$ # systemctl status etcd.service

$root@K8s-controller \~$ # netstat -tnlp

(2)、启动K8s-node01节点etcd服务

1）设置开机启动etcd服务

$root@K8s-node01 \~$ # systemctl enable etcd

2）启动etcd服务

$root@K8s-node01 \~$ # systemctl start etcd.service

$root@K8s-node01 \~$ # systemctl status etcd.service

$root@K8s-node01 \~$ # netstat -tnlp

(3)、启动K8s-node02节点etcd服务

1）设置开机启动etcd服务

$root@K8s-node02 \~$ # systemctl enable etcd

2）启动etcd服务

$root@K8s-node02 \~$ # systemctl start etcd.service

$root@K8s-node02 \~$ # systemctl status etcd.service

$root@K8s-node02 data$ # netstat -tnlp

6、验证etcd集群状态

(1)、查看etcd集群健康状态

$root@K8s-controller \~$ # ETCDCTL_API=3 etcdctl --cacert=/K8s/etcd/ssl/etcd-ca.pem --cert=/K8s/etcd/ssl/etcd-server.pem --key=/K8s/etcd/ssl/etcd-server-key.pem --endpoints="https://192.168.110.150:2379,https://192.168.110.151:2379,https://192.168.110.152:2379" endpoint health

由上可见，etcd三个集群节点状态均为健康。

(2)、查看etcd集群LEADER节点

$root@K8s-controller \~$ # ETCDCTL_API=3 etcdctl -w table --cacert=/K8s/etcd/ssl/etcd-ca.pem --cert=/K8s/etcd/ssl/etcd-server.pem --key=/K8s/etcd/ssl/etcd-server-key.pem --endpoints="https://192.168.110.150:2379,https://192.168.110.151:2379,https://192.168.110.152:2379" endpoint status

由上可见，192.168.110.151节点(K8s-controller)为LEADER节点