keepalived理论--实验

一 . 高可用集群
1.1 集群类型
LB ： Load Balance 负载均衡
LVS/HAProxy/nginx （ http/upstream, stream/upstream ）
HA ： High Availability 高可用集群
数据库、 Redis
SPoF: Single Point of Failure ，解决单点故障
HPC ： High Performance Computing 高性能集群
1.2 系统可用性
SLA ： Service-Level Agreement 服务等级协议（提供服务的企业与客户之间就服务的品质、水准、性能
等方面所达成的双方共同认可的协议或契约）
A = MTBF / (MTBF+MTTR ）
指标 ： 99.9%, 99.99%, 99.999%,99.9999%
1.3 系统故障
硬件故障：设计缺陷、 wear out （损耗）、非人为不可抗拒因素
软件故障：设计缺陷 bug
1.4 实现高可用
提升系统高用性的解决方案：降低 MTTR- Mean Time To Repair( 平均故障时间 )
解决方案：建立冗余机制
active/passive 主 / 备
active/active 双主
active --> HEARTBEAT --> passive
active <--> HEARTBEAT <--> active
1.5.VRRP ： Virtual Router Redundancy Protocol
虚拟路由冗余协议 , 解决静态网关单点风险
物理层 : 路由器、三层交换机
软件层 :keepalived
1.5.1 VRRP 相关术语
虚拟路由器： Virtual Router
虚拟路由器标识： VRID(0-255) ，唯一标识虚拟路由器
VIP ： Virtual IP
VMAC ： Virutal MAC (00-00-5e-00-01-VRID)
物理路由器：
master ：主设备
backup ：备用设备
priority ：优先级
1.5.2 VRRP 相关技术
通告：心跳，优先级等；周期性
工作方式：抢占式，非抢占式
安全认证：
无认证
简单字符认证：预共享密钥
MD5
工作模式：
主 / 备：单虚拟路由器
主 / 主：主 / 备（虚拟路由器 1 ），备 / 主（虚拟路由器 2 ）
keepalived部署

keepalived 简介

vrrp 协议的软件实现，原生设计目的为了高可用 ipvs服务

功能：

基于vrrp协议完成地址流动

为vip地址所在的节点生成ipvs规则(在配置文件中预先定义)

为ipvs集群的各RS做健康状态检测

基于脚本调用接口完成脚本中定义的功能，进而影响集群事务，以此支持nginx、haproxy等服务

Keepalived 环境准备

Keepalived 相关文件

软件包名：keepalived

主程序文件：/usr/sbin/keepalived

主配置文件：/etc/keepalived/keepalived.conf

配置文件示例：/usr/share/doc/keepalived/

Unit File：/lib/systemd/system/keepalived.service

Unit File的环境配置文件：/etc/sysconfig/keepalived

Keepalived环境配置

ka1的IP配置

[root@localhost network-scripts]# vmset.sh eth0 172.25.254.10 ka1.timinglee.org

ka2的IP配置

[root@localhost ~]# vmset.sh eth0 172.25.254.20 ka2.timinglee.org

server1的IP配置

[root@localhost ~]# vmset.sh eth0 172.25.254.110 server1.timinglee.org

server2的IP配置

[root@localhost ~]# vmset.sh eth0 172.25.254.120 server2.timinglee.org

server1和server2都安装httpd服务

[root@server1 ~]# echo 172.25.254.110 > /var/www/html/index.html
[root@server1 ~]# systemctl enable --now httpd
Created symlink from /etc/systemd/system/multi-user.target.wants/httpd.service to /usr/lib/systemd/system/httpd.service.
[root@server2 ~]# echo 172.25.254.120 > /var/www/html/index.html 
[root@server2 ~]# systemctl enable --now httpd
Created symlink from /etc/systemd/system/multi-user.target.wants/httpd.service to /usr/lib/systemd/system/httpd.service.

ka1上测试

keepalived虚拟路由配置

ka1和ka2都安装keepalived

root@ka1 \~\]# yum install keepalived -y  \[root@ka1 \~\]# man 5 keepalived.conf---查看帮助 \[root@ka1 \~\]# vim /etc/keepalived/keepalived.conf --配置文件 全局配置 ! Configuration File for keepalived global_defs { notification_email { [email protected] #keepalived 发生故障切换时邮件发送的目标邮箱，可以按行区分写多个 [email protected] } notification_email_from [email protected] #发邮件的地址 smtp_server 127.0.0.1 #邮件服务器地址 smtp_connect_timeout 30 #邮件服务器连接timeout router_id KA1.timinglee.org #每个keepalived主机唯一标识建议使用当前主机名，但多节点重名不影响 vrrp_skip_check_adv_addr #对所有通告报文都检查，会比较消耗性能，启用此配置后，如果收到的通告报文和上一个报文是同 一个路由器，则跳过检查，默认值为全检查 vrrp_strict #严格遵循vrrp协议 #启用此项后以下状况将无法启动服务: #1.无VIP地址 #2.配置了单播邻居 #3.在VRRP版本2中有IPv6地址 #建议不加此项配置 vrrp_garp_interval 0 #报文发送延迟，0表示不延迟 vrrp_gna_interval 0 #消息发送延迟 vrrp_mcast_group4 224.0.0.18 #指定组播IP地址范围： }  **配置虚拟路由器** vrrp_instance VI_1 { state MASTER interface eth0 #绑定为当前虚拟路由器使用的物理接口，如：eth0,可以和VIP不在一个网卡 virtual_router_id 51 #每个虚拟路由器惟一标识,范围：0-255，每个虚拟路由器此值必须唯一 #否则服务无法启动 #同属一个虚拟路由器的多个keepalived节点必须相同 #务必要确认在同一网络中此值必须唯一 priority 100 #当前物理节点在此虚拟路由器的优先级，范围：1-254 #值越大优先级越高,每个keepalived主机节点此值不同 advert_int 1 #vrrp通告的时间间隔，默认1s authentication { #认证机制 auth_type AH|PASS #AH为IPSEC认证(不推荐),PASS为简单密码(建议使用) uth_pass 1111 #预共享密钥，仅前8位有效 #同一个虚拟路由器的多个keepalived节点必须一样 } virtual_ipaddress { #虚拟IP,生产环境可能指定上百个IP地址 / brd dev scope label 172.25.254.100 #指定VIP，不指定网卡，默认为eth0,注意：不指定/prefix,默认32 172.25.254.101/24 dev eth1 172.25.254.102/24 dev eth2 label eth2:1 } } **实验实例----虚拟路由器** ka1 \[root@ka1 \~\]# vim /etc/keepalived/keepalived.conf  ka2 \[root@ka2 \~\]# vim /etc/keepalived/keepalived.conf  也可复制去配置 [root@ka2 ~]# scp /etc/keepalived/keepalived.conf [email protected]:/etc/keepalived/keepalived.conf The authenticity of host '172.25.254.20 (172.25.254.20)' can't be established. ECDSA key fingerprint is SHA256:mT1nI9LQcSBAxtMowpMdk/qQ/rIPxWexv72aZL+B0o4. ECDSA key fingerprint is MD5:1e:d3:71:ec:af:6d:76:a2:d5:bf:36:cb:e4:07:00:fd. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added '172.25.254.20' (ECDSA) to the list of known hosts. [email protected]'s password: keepalived.conf 100% 3598 1.8MB/s 00:00 [root@ka2 ~]# vim /etc/keepalived/keepalived.conf [root@ka2 ~]# systemctl enable --now keepalived.service 测试  **开启通信功能及独立日志** \[root@ka1 \~\]# vim /etc/sysconfig/keepalived  \[root@ka1 \~\]#vim /etc/rsyslog.conf  做完这两步后重启 \[root@ka1 \~\]# systemctl restart keepalived.service \[root@ka1 \~\]# systemctl restart rsyslog.service \[root@ka1 \~\]# systemctl restart keepalived.service 测试  **独立子配置文件**    **非抢占和延迟抢占** 非抢占  延迟抢占  **组播变单播** ka1和ka2都要配置 \[root@ka1 \~\]# vim /etc/keepalived/keepalived.conf  \[root@ka2 \~\]# vim /etc/keepalived/keepalived.conf  ka1上测试  ka2上测试  **通知脚本类型** 当前节点成为主节点时触发的脚本 notify_master | 当前节点转为备节点时触发的脚本 notify_backup | 当前节点转为"失败"状态时触发的脚本 notify_fault | 通用格式的通知触发机制，一个脚本可完成以上三种状态的转换时的通知 notify | 当停止VRRP时触发的脚本 notify_stop | **脚本的调用方法** 在vim /etc/keepalived/keepalived.conf 中 vrrp_instance VI_1 语句块的末尾加下面行 notify_master "/etc/keepalived/notify.sh master" notify_backup "/etc/keepalived/notify.sh backup" notify_fault "/etc/keepalived/notify.sh fault" **邮件通知** ka1和ka2都安装邮箱发送工具 [root@ka1 ~]# yum install mailx -y [root@ka2 ~]# yum install mailx -y 邮箱配置ka1和ka2都需要配置 \[root@ka1 \~\]# vim /etc/mail.rc  发送测试邮件 echo hello world \| mail -s test \*\*\*\*\*\*\*\*\*\*@qq.com  **实现** **Keepalived** **状态切换的通知脚本** 创建通知脚本---ka1和ka2 都配置 \[root@ka1 \~\]# vim /etc/keepalived/mail.sh #!/bin/bash mail_dst="[email protected]" send_message() { mail_sub="$HOSTNAME to be $1 vip move" mail_meg="`date +%F\ %T`: vrrp move $HOSTNAME chage $1" echo $mail_meg | mail -s "$mail_sub" $mail_dst } case $1 in master) send_message master ;; backup) send_message backup ;; fault) send_message fault ;; *) ;; esac 给执行权限 \[root@ka1 \~\]# chmod +x /etc/keepalived/mail.sh \[root@ka2 \~\]# chmod +x /etc/keepalived/mail.sh ka1和ka2都重启服务 \[root@ka1 \~\]# systemctl restart keepalived.service \[root@ka2 \~\]# systemctl restart keepalived.service 测试 \[root@ka1 \~\]# /etc/keepalived/mail.sh master  \[root@ka2 \~\]# /etc/keepalived/mail.sh master  ##### **实现** **master/master** **的** **Keepalived** **双主架构** #ka1 vrrp_instance VI_1 { state MASTER interface eth0 virtual_router_id 100 priority 100 advert_int 1 authentication { auth_type PASS auth_pass 1111 } virtual_ipaddress { 172.25.254.100/24 dev eth0 label eth0:1 } unicast_src_ip 172.25.254.10 unicast_peer { 172.25.254.20 } track_script { check_haproxy } } vrrp_instance VI_2 { state BACKUP #备 interface eth0 virtual_router_id 200 priority 80 advert_int 1 authentication { auth_type PASS auth_pass 1111 } virtual_ipaddress { 172.25.254.200/24 dev eth0 label eth0:2 } unicast_src_ip 172.25.254.10 unicast_peer { 172.25.254.20 } } ​ # ka2 vrrp_instance VI_1 { state BACKUP interface eth0 virtual_router_id 100 priority 80 advert_int 1 authentication { auth_type PASS auth_pass 1111 } virtual_ipaddress { 172.25.254.100/24 dev eth0 label eth0:1 } unicast_src_ip 172.25.254.20 unicast_peer { 172.25.254.10 } } vrrp_instance VI_2 { state MASTER #主 interface eth0 virtual_router_id 200 priority 100 advert_int 1 authentication { auth_type PASS auth_pass 1111 } virtual_ipaddress { 172.25.254.200/24 dev eth0 label eth0:2 } unicast_src_ip 172.25.254.20 unicast_peer { 172.25.254.10 } } ##### 实现IPVS的高可用性 # 在server1\2 [root@server2 ~]# ip a a 172.25.254.100/32 dev lo [root@server2 ~]# vim /etc/sysctl.d/arp.conf net.ipv4.conf.all.arp_ignore=1 net.ipv4.conf.all.arp_announce=2 net.ipv4.conf.lo.arp_ignore=1 net.ipv4.conf.lo.arp_announce=2 ​ [root@webserver2 ~]# sysctl --system ​ # ka1\2 [root@ka1 ~]# yum install ipvsadm -y [root@ka1 ~]# vim /etc/keepalived/keepalived.conf virtual_server 172.25.254.100 80 { delay_loop 6 lb_algo wrr lb_kind DR #persistence_timeout 50 protocol TCP ​ real_server 172.25.254.110 80 { weight 1 HTTP_GET { url { path / status_code 200 } connect_timeout 3 nb_get_retry 2 delay_before_retry 2 } } real_server 172.25.254.120 80 { weight 1 HTTP_GET { url { path / status_code 200 } connect_timeout 3 nb_get_retry 2 delay_before_retry 2 } } } ​ [root@ka1 ~]# systemctl restart keepalived.service [root@ka1 ~]# ipvsadm -Ln IP Virtual Server version 1.2.1 (size=4096) Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConn ​ ​ [root@ka2 ~]# ipvsadm -A -t 172.255.254.100:80 -s wrr [root@ka2 ~]# ipvsadm -Ln (有) [root@ka1 ~]# ipvsadm -Ln (有) ​ # 测试 [root@ka2 ~]# for i in {1..6}; do curl 172.25.254.100; done server2 - 172.25.254.120 server1 - 172.25.254.110 server2 - 172.25.254.120 server1 - 172.25.254.110 server2 - 172.25.254.120 server1 - 172.25.254.110 **实战案例：实现HAProxy高可用** #ka1、ka2 [root@ka1 ~]# yum install haproxy -y [root@ka1 ~]# vim /etc/sysctl.conf net.ipv4.ip_nonlocal_bind = 1 [root@ka1 ~]# sysctl -p [root@ka1 ~]# vim /etc/haproxy/haproxy.cfg listen webcluster bind 172.25.254.100:80 mode http balance roundrobin server web1 172.25.254.110:80 check inter 3 fall 2 rise 5 server web2 172.25.254.120:80 check inter 3 fall 2 rise 5 [root@ka1 ~]# sysetemctl enable ---now haproxy [root@ka1 ~]# netstat -antlupe | grep haproxy [root@ka1 ~]# curl 172.25.254.100 (不能，和lvs冲突) [root@ka1 ~]# vim /etc/keepalived/keepalived.conf 注释掉 #virtual_server 172.25.254.100 80 { [root@ka1 ~]# systemctl restart keepalived.service #server1、2 [root@server2 ~]# systemctl restart network (删掉lo) [root@server2 ~]# vim /etc/sysctl.d/arp.conf net.ipv4.conf.all.arp_ignore=0 net.ipv4.conf.all.arp_announce=0 net.ipv4.conf.lo.arp_ignore=0 net.ipv4.conf.lo.arp_announce=0 [root@server2 ~]# sysctl --system ​ [root@ka1 ~]# curl 172.25.254.100 [root@ka1 ~]# curl 172.25.254.110 [root@ka1 ~]# curl 172.25.254.120 ​ ​#现在 在ka1关掉haproxy，则在外面访问不了172.25.254.100，解决这个问题 # 在ka1 [root@ka1 ~]# vim /etc/keepalived/test.sh #!/bin/bash killall -0 haproxy [root@ka1 ~]# sh /etc/keepalived/test.sh [root@ka1 ~]# echo $? [root@ka1 ~]# vim /etc/keepalived/keepalived.conf vrrp_script check_haproxy { script "/etc/keepalived/test.sh" interval 1 weight -30 fall 2 rise 2 timeout 2 } vrrp_instance VI_1 { state MASTER interface eth0 virtual_router_id 100 priority 100 advert_int 1 authentication { auth_type PASS auth_pass 1111 } virtual_ipaddress { 172.25.254.100/24 dev eth0 label eth0:1 } unicast_src_ip 172.25.254.10 unicast_peer { 172.25.254.20 } track_script { check_haproxy } } \[root@ka1 \~\]# systemctl restart keepalived.service # 测试 关掉ka1的haproxy，也能一直访问 [root@ka1 ~]# systemctl stop haproxy.service [root@ka1 ~]# systemctl start haproxy.service [root@ka2 ~]# while true; do curl 172.25.254.100; done