使用 keepalived 和 haproxy 实现Kubernetes Control Plane的高可用 (HA)

文章目录

• 小结
• 问题
• 解决
• • keepalived设置
  • haproxy设置
  • [部署Kubernetes control plane集群](#部署Kubernetes control plane集群)
• 参考

小结

本文记录了使用 keepalived 和 haproxy 实现Kubernetes Control Plane的高可用 (HA)。

VRRP(Virtual Router Redundancy Protocol) - 虚拟路由冗余协议

VIP - Virtual IP, 虚拟IP（浮动IP）

问题

需要实现Kubernetes Control Plane的高可用 (HA)， 也就是Kubernetes Control Plane部署成一个集群，集群中的单个Kubernetes Control Plane损坏，Kubernetes Control Plane仍然工作。

解决

Kubernetes的Worker Nodes通过负载均衡连接Kubernetes Control Plane的集群，负载均衡使用keepalived 和 haproxy 来实现。

keepalived设置

在三台主机上安装并启动keepalived （Master, Node1 和 Node2），这三个节点也同时运行haproxy 和 Kubernetes Control Plane的集群。

yum install keepalived
systemctl enable keepalived

设置keepalived

vim /etc/keepalived/keepalived.conf

配置如下，

vrrp_instance VI_1 {
    state MASTER
    interface ens160
    virtual_router_id 51
    priority 100  # 优先级
    advert_int 1
    mcast_src_ip 192.168.238.130 #本机IP
    unicast_src_ip 192.168.238.130 #本机IP
    unicast_peer {
        192.168.238.131, #另外两个节点
        192.168.238.132
    }
    nopreempt
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        192.168.238.100/24  #VIP
    }
    track_script {
        check_apiserver
    }
}

创建心跳线探测脚本：

vim /etc/keepalived/check_apiserver.sh
chmod +x /etc/keepalived/check_apiserver.sh

脚本如下：

#!/bin/sh

errorExit() {
    echo "*** $*" 1>&2
    exit 1
}

curl --silent --max-time 2 --insecure https://localhost:4300/ -o /dev/null || errorExit "Error GET https://localhost:4300/"
if ip addr | grep -q 192.168.238.100; then
    curl --silent --max-time 2 --insecure https://192.168.238.100:4300/ -o /dev/null || errorExit "Error GET https://192.168.238.100:4300/"
fi

启动keepalived服务

systemctl start keepalived

haproxy设置

安装并启动haproxy

yum install haproxy
systemctl enable haproxy

配置haproxy

vim /etc/haproxy/haproxy.cfg 

配置的内容如下：

...
#---------------------------------------------------------------------
# apiserver frontend which proxys to the control plane nodes
#---------------------------------------------------------------------
frontend apiserver
    bind *:4300
    mode tcp
    option tcplog
    default_backend apiserver
...
#---------------------------------------------------------------------
# round robin balancing for apiserver
#---------------------------------------------------------------------
backend apiserver
    option httpchk GET /healthz
    http-check expect status 200
    mode tcp
    option ssl-hello-chk
    balance     roundrobin
        #server ${HOST1_ID} ${HOST1_ADDRESS}:${APISERVER_SRC_PORT} check
        # [...]
        server Master 192.168.238.130:6443 check
        server Node1 192.168.238.131:6443 check
        server Node2 192.168.238.132:6443 check
...

系统设置：

[root@Master ~]# echo net.ipv4.ip_nonlocal_bind=1 >> /etc/sysctl.d/haproxy-keepalived.conf
[root@Master ~]# echo net.ipv4.ip_forward=1 >> /etc/sysctl.d/haproxy-keepalived.conf
[root@Master ~]# cat /etc/sysctl.d/haproxy-keepalived.conf
net.ipv4.ip_nonlocal_bind=1
net.ipv4.ip_forward=1
[root@Master ~]#
[root@Master ~]# sysctl -p /etc/sysctl.d/haproxy-keepalived.conf
net.ipv4.ip_nonlocal_bind = 1
net.ipv4.ip_forward = 1
[root@Master ~]# 

启动haproxy服务

systemctl start haproxy

查看虚拟IP绑定：

[root@Master ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: ens160: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 00:0c:29:9e:d5:84 brd ff:ff:ff:ff:ff:ff
    inet 192.168.238.130/24 brd 192.168.238.255 scope global dynamic noprefixroute ens160
       valid_lft 1790sec preferred_lft 1790sec
    inet 192.168.238.100/24 scope global secondary ens160
       valid_lft forever preferred_lft forever
    inet6 fe80::b52:173:210c:48d9/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever
[root@Master ~]# ip a | grep 192.168.238.100
    inet 192.168.238.100/24 scope global secondary ens160
[root@Master ~]# 

验证VIP和haproxy在工作：

[root@Master ~]# curl --insecure https://localhost:4300
{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {},
  "status": "Failure",
  "message": "forbidden: User \"system:anonymous\" cannot get path \"/\"",
  "reason": "Forbidden",
  "details": {},
  "code": 403
}
[root@Master ~]# curl --insecure https://192.168.238.100:4300
{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {},
  "status": "Failure",
  "message": "forbidden: User \"system:anonymous\" cannot get path \"/\"",
  "reason": "Forbidden",
  "details": {},
  "code": 403
}[root@Master ~]# 

在浏览器中输入https://192.168.238.100:4300/可以得到以上同样的内容。如果输入http://192.168.238.100:4300/，则返回Client sent an HTTP request to an HTTPS server.。

部署Kubernetes control plane集群

根据以上配置，使用以下指令

#以下指令在三个节点中的一个节点执行，这里在Master上
kubeadm init --pod-network-cidr=10.244.0.0/16 --control-plane-endpoint 192.168.238.100

按提示在三个节点上运行以下指令：

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

拷贝证书文件到剩下的节点，如下：

scp /etc/kubernetes/pki/{ca.*,sa.*,front-proxy-ca.*} root@Node1:/etc/kubernetes/pki/
mkdir -p /etc/kubernetes/pki/etcd
scp /etc/kubernetes/pki/etcd/ca.* root@Node1:/etc/kubernetes/pki/etcd

在加入其它的两个Kubernetes Control Plane之前，需要部署flannel网络模块：

kubectl apply -f kube-flannel.yml 

以下指令根据提示在剩下下两个节点执行

kubeadm join 192.168.238.100:4300 --token si5oek.mbrw418p8mr357qt --discovery-token-ca-cert-hash sha256:0e23eb637e09afc4c6dbb1f891409b314d5731e46fe33d84793ba2d58da006d6 --control-plane 

再安装剩下的模块即可成功，结果如下：

[root@Master ~]#kubectl get nodes -o wide 
NAME     STATUS   ROLES           AGE     VERSION   INTERNAL-IP       EXTERNAL-IP   OS-IMAGE                               KERNEL-VERSION          CONTAINER-RUNTIME
master   Ready    control-plane   3h57m   v1.27.3   192.168.238.130   <none>        Red Hat Enterprise Linux 8.3 (Ootpa)   4.18.0-240.el8.x86_64   containerd://1.6.21
node1    Ready    control-plane   3h53m   v1.27.3   192.168.238.131   <none>        Red Hat Enterprise Linux 8.3 (Ootpa)   4.18.0-240.el8.x86_64   containerd://1.6.21
node2    Ready    control-plane   3h51m   v1.27.3   192.168.238.132   <none>        Red Hat Enterprise Linux 8.3 (Ootpa)   4.18.0-240.el8.x86_64   containerd://1.6.21
[root@Master ~]# 

这里的核心组件如下：

[root@Master ~]# kubectl get pods -o wide -A
NAMESPACE      NAME                             READY   STATUS    RESTARTS         AGE     IP                NODE     NOMINATED NODE   READINESS GATES
kube-flannel   kube-flannel-ds-6n8rh            1/1     Running   0                3m11s   192.168.238.131   node1    <none>           <none>
kube-flannel   kube-flannel-ds-c65rz            1/1     Running   0                80s     192.168.238.132   node2    <none>           <none>
kube-flannel   kube-flannel-ds-ss8pw            1/1     Running   0                10m     192.168.238.130   master   <none>           <none>
kube-system    coredns-5d78c9869d-cw5cz         1/1     Running   0                17m     10.244.1.96       node1    <none>           <none>
kube-system    coredns-5d78c9869d-z99r8         1/1     Running   0                17m     10.244.1.95       node1    <none>           <none>
kube-system    etcd-master                      1/1     Running   13               17m     192.168.238.130   master   <none>           <none>
kube-system    etcd-node1                       1/1     Running   0                3m10s   192.168.238.131   node1    <none>           <none>
kube-system    etcd-node2                       1/1     Running   0                79s     192.168.238.132   node2    <none>           <none>
kube-system    kube-apiserver-master            1/1     Running   11               17m     192.168.238.130   master   <none>           <none>
kube-system    kube-apiserver-node1             1/1     Running   1                3m11s   192.168.238.131   node1    <none>           <none>
kube-system    kube-apiserver-node2             1/1     Running   2                79s     192.168.238.132   node2    <none>           <none>
kube-system    kube-controller-manager-master   1/1     Running   20 (2m58s ago)   17m     192.168.238.130   master   <none>           <none>
kube-system    kube-controller-manager-node1    1/1     Running   1                3m11s   192.168.238.131   node1    <none>           <none>
kube-system    kube-controller-manager-node2    1/1     Running   2                79s     192.168.238.132   node2    <none>           <none>
kube-system    kube-proxy-87gfk                 1/1     Running   0                17m     192.168.238.130   master   <none>           <none>
kube-system    kube-proxy-crjc4                 1/1     Running   0                80s     192.168.238.132   node2    <none>           <none>
kube-system    kube-proxy-mnl2d                 1/1     Running   0                3m11s   192.168.238.131   node1    <none>           <none>
kube-system    kube-scheduler-master            1/1     Running   20 (2m55s ago)   17m     192.168.238.130   master   <none>           <none>
kube-system    kube-scheduler-node1             1/1     Running   1                3m11s   192.168.238.131   node1    <none>           <none>
kube-system    kube-scheduler-node2             1/1     Running   2                79s     192.168.238.132   node2    <none>           <none>
[root@Master ~]# 

参考

Github: High Availability Considerations
Kubernetes: Options for Highly Available Topology
刘达的博客: haproxy+keepalived配置示例
阿里云：搭建高可用负载均衡器: haproxy+keepalived
HAproxy and keepAlived for Multiple Kubernetes Master Nodes
Create a Highly Available Kubernetes Cluster Using Keepalived and HAproxy
Install and Configure a Multi-Master HA Kubernetes Cluster with kubeadm, HAProxy and Keepalived on CentOS 7
CSDN: haproxy+keepalived搭建高可用k8s集群
CSDN: kubernetes学习---安装haproxy并配置keepalived高可用
使用 Keepalived 和 HAproxy 创建高可用 Kubernetes 集群
CSDN: HAProxy + Keepalived 配置架构解析
cnblog: haproxy+keepalived原理特点
Keepalived与HaProxy的协调合作原理分析
High availability Kubernetes cluster on bare metal - part 2