Ubuntu 安装带证书的 etcd 集群

1.概念

etcd 是由GO语言编写的分布式的、可靠的键值存储系统，主要用于分布式系统中关键数据的存储和服务发现。

2.核心概念

节点（Node）

每个运行 etcd 的实例被称为一个节点。一个或多个节点可以组成一个集群。

集群（Cluster）

由多个节点组成的集合，这些节点共同工作以提供一致的数据存储服务。通过 Raft 共识算法确保集群中各节点间数据的一致性。

键值对（Key-Value Pair）

etcd 存储的基本单位是键值对，其中键和值都是字节数组。键用于唯一标识存储的数据项，而值则包含实际的数据内容。

3.etcd集群准备

节点	IP地址	操作系统版本	etcd版本
etcd-node1	192.168.100.5	Ubuntu 24.04.2 LTS	v3.6.4
etcd-node2	192.168.100.6	Ubuntu 24.04.2 LTS	v3.6.4
etcd-node3	192.168.100.7	Ubuntu 24.04.2 LTS	v3.6.4

3.1 配置IP地址

配置etcd-node1节点IP

sudo cat /etc/netplan/ens32-cloud-init.yaml 
network:
  version: 2
  ethernets:
    ens32:
      dhcp4: false
      addresses:
        - "192.168.100.5/24"
      routes:
        - to: default
          via: 192.168.100.254
      nameservers:
        addresses:
          - 114.114.114.114
sudo netplan apply

配置etcd-node2节点IP

sudo cat /etc/netplan/ens32-cloud-init.yaml 
network:
  version: 2
  ethernets:
    ens32:
      dhcp4: false
      addresses:
        - "192.168.100.6/24"
      routes:
        - to: default
          via: 192.168.100.254
      nameservers:
        addresses:
          - 114.114.114.114
sudo netplan apply

配置etcd-node3节点IP

sudo cat /etc/netplan/ens32-cloud-init.yaml 
network:
  version: 2
  ethernets:
    ens32:
      dhcp4: false
      addresses:
        - "192.168.100.7/24"
      routes:
        - to: default
          via: 192.168.100.254
      nameservers:
        addresses:
          - 114.114.114.114
sudo netplan apply

3.2 配置主机名

配置etcd-node1节点主机名

sudo hostnamectl set-hostname etcd-node1

配置etcd-node2节点主机名

sudo hostnamectl set-hostname etcd-node2

配置etcd-node3节点主机名

sudo hostnamectl set-hostname etcd-node3

3.3 配置主机名与IP解析

3个节点均需要执行

sudo cat >> /etc/hosts <<EOF
192.168.100.5 etcd-node1
192.168.100.6 etcd-node2
192.168.100.7 etcd-node3
EOF

3.4 关闭防火墙

3个节点均需要执行

sudo ufw stop
sudo ufw status

3.5 时钟同步

3个节点均需要执行

sudo apt install chrony
sudo sed -i '/pool.*ubuntu\.pool\.ntp\.org/s/^/# /' /etc/chrony/chrony.conf
sudo sed -i 's/^pool ntp\.ubuntu\.com.*$/server ntp.aliyun.com iburst/' /etc/chrony/chrony.conf
sudo systemctl restart chrony
sudo chronyc sources

3.6 配置节点互信

3个节点均需要执行

sudo ssh-keygen
sudo cp /root/.ssh/id_rsa.pub /root/.ssh/authorized_keys

在etcd-node1节点上执行

sudo for i in 5 6 7
> do
> scp -r /root/.ssh 192.168.100.$i:/root/
> done

4.etcd集群部署

4.1 下载etcd

3个节点均需要执行

sudo wget https://github.com/etcd-io/etcd/releases/download/v3.6.4/etcd-v3.6.4-linux-amd64.tar.gz

4.2解压etcd

3个节点均需要执行

sudo tar xzvf etcd-v3.6.4-linux-amd64.tar.gz -C /usr/local
sudo ln -s /usr/local/etcd-v3.6.4-linux-amd64/ /usr/local/etcd

4.3复制解压文件至系统标准可执行文件路径中

3个节点均需要执行

sudo cp /usr/local/etcd/etcd* /usr/local/bin/

4.4创建etcd用户

3个节点均需要执行

sudo useradd --system --shell /bin/false --home-dir /var/lib/etcd etcd

4.5创建数据目录

3个节点均需要执行

sudo mkdir -p /var/lib/etcd 
sudo mkdir /var/lib/etcd/default.etcd
sudo chown -R etcd:etcd /var/lib/etcd /usr/local/etcd

4.6 创建etcd配置文件

配置etcd-node1节点配置文件

sudo cat > /usr/local/etcd/etcd.conf <<EOF
ETCD_NAME="etcd-node1"
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="https://192.168.100.5:2380"
ETCD_LISTEN_CLIENT_URLS="https://192.168.100.5:2379,http://127.0.0.1:2379"

#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.100.5:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.100.5:2379"
ETCD_INITIAL_CLUSTER="etcd-node1=https://192.168.100.5:2380,etcd-node2=https://192.168.100.6:2380,etcd-node3=https://192.168.100.7:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"
EOF

配置etcd-node2节点配置文件

sudo cat > /usr/local/etcd/etcd.conf <<EOF
ETCD_NAME="etcd-node2"
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="https://192.168.100.6:2380"
ETCD_LISTEN_CLIENT_URLS="https://192.168.100.6:2379,http://127.0.0.1:2379"

#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.100.6:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.100.6:2379"
ETCD_INITIAL_CLUSTER="etcd-node1=https://192.168.100.5:2380,etcd-node2=https://192.168.100.6:2380,etcd-node3=https://192.168.100.7:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"
EOF

配置etcd-node3节点配置文件

sudo cat > /usr/local/etcd/etcd.conf <<EOF
ETCD_NAME="etcd-node3"
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="https://192.168.100.7:2380"
ETCD_LISTEN_CLIENT_URLS="https://192.168.100.7:2379,http://127.0.0.1:2379"

#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.100.7:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.100.7:2379"
ETCD_INITIAL_CLUSTER="etcd-node1=https://192.168.100.5:2380,etcd-node2=https://192.168.100.6:2380,etcd-node3=https://192.168.100.7:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"
EOF

相关参数说明

参数	说明
ETCD_NAME	当前etcd节点名称
ETCD_DATA_DIR	数据存储目录
ETCD_LISTEN_CLIENT_URLS	当前节点通过该地址监听客户端发送的信息
ETCD_LISTEN_PEER_URLS	当前节点通过该地址监听集群其他节点发送的信息
ETCD_INITIAL_ADVERTISE_PEER_URLS	集群的其他节点通过该地址与当前节点通信
ETCD_ADVERTISE_CLIENT_URLS	客户端通过该地址与当前节点通信
ETCD_INITIAL_CLUSTER	当前集群的所有节点信息，当前节点根据此信息与其他节点取得联系
ETCD_INITIAL_CLUSTER_TOKEN	用于区分不同的集群，同一集群的所有节点配置相同的值
ETCD_INITIAL_CLUSTER_STATE	本次是否为新建集群，取值为 new 或者 existing

4.7下载cfssl 证书生成工具

在etcd-node1节点上下载cfssl

wget https://github.com/cloudflare/cfssl/releases/download/v1.6.5/cfssl_1.6.5_linux_amd64

在etcd-node1节点上下载cfssljson

wget https://github.com/cloudflare/cfssl/releases/download/v1.6.5/cfssljson_1.6.5_linux_amd64

4.8授权并移至系统标准可执行文件路径中

在etcd-node1节点上执行

 chmod +x cfssl_1.6.5_linux_amd64
 chmod +x cfssljson_1.6.5_linux_amd64
 mv cfssl_1.6.5_linux_amd64 /usr/local/bin/cfssljson
 mv cfssljson_1.6.5_linux_amd64 /usr/local/bin/cfssljson

4.9创建CA证书

在etcd-node1节点上配置CA证书策略

mkdir /usr/local/etcd/ssl
cat > /usr/local/etcd/ssl/ca-config.json <<EOF
{
    "signing": {
        "default": {
            "expiry": "87600h"
        },
        "profiles": {
            "etcd-server": {
                "usages": [
                    "signing",
                    "key encipherment",
                    "client auth",
                    "server auth"
                ],
                "expiry": "87600h"
            }
        }
    }
}
EOF

在etcd-node1节点上配置CA证书请求文件

cat > /usr/local/etcd/ssl/ca-csr.json <<EOF 
{
    "CN": "My etcd CA",
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "L": "LANZHOU",
            "O": "LZ",
            "ST": "LANZHOU",
            "OU": "CN"
        }
    ],
    "ca": {
            "expiry": "87600h"
    }
}
EOF

在etcd-node1节点上生成CA证书

cfssl gencert -initca ca-csr.json | cfssljson -bare ca -

4.10创建etcd证书

在etcd-node1节点上配置etcd请求文件

cat > /usr/local/etcd/ssl/etcd-server.json  <<EOF
{
    "CN": "etcd",
    "hosts": [
        "127.0.0.1",
        "192.168.100.5",
        "192.168.100.6",
        "192.168.100.7"
    ],
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "L": "LANZHOU",
            "ST": "LANZHOU",
	    "OU": "CN"
        }
    ]
}
EOF

在etcd-node1节点上生成 Etcd 证书

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=etcd-server etcd-server.json | cfssljson -bare etcd-server

说明：

参数	说明
-ca-key	指定CA证书机构的私钥
-config	指定CA证书策略
-profile	指定使用CA证书策略
etcd-server.pem	证书/公钥
etcd-server-key.pem	私钥

4.7创建 systemd 服务

3个节点均需要执行

sudo cat > /usr/lib/systemd/system/etcd.service <<EOF
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target

[Service]
Type=notify
EnvironmentFile=-/usr/local/etcd/etcd.conf
WorkingDirectory=/var/lib/etcd/
ExecStart=/usr/local/bin/etcd \
  --cert-file=/usr/local/etcd/ssl/etcd-server.pem \
  --key-file=/usr/local/etcd/ssl/etcd-server-key.pem \
  --trusted-ca-file=/usr/local/etcd/ssl/ca.pem \
  --peer-cert-file=/usr/local/etcd/ssl/etcd-server.pem \
  --peer-key-file=/usr/local/etcd/ssl/etcd-server-key.pem \
  --peer-trusted-ca-file=/usr/local/etcd/ssl/ca.pem \
  --peer-client-cert-auth \
  --client-cert-auth
Restart=on-failure
RestartSec=5
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
EOF

4.8 启动etcd

3个节点均需要执行

sudo systemctl daemon-reload
sudo systemctl enable etcd
sudo systemctl start etcd
sudo systemctl status etcd

4.9查看集群成员

任意节点执行

sudo etcdctl member list

4.10查看集群节点健康状态

ETCDCTL_API=3 /usr/local/bin/etcdctl --write-out=table --cacert=/usr/local/etcd/ssl/ca.pem --cert=/usr/local/etcd/ssl/etcd-server.pem --key=/usr/local/etcd/ssl/etcd-server-key.pem --endpoints=https://192.168.100.5:2379,https://192.168.100.6:2379,https://192.168.100.7:2379 endpoint health