基于keepalived(主从+双主) + LVS(DR模型) + DNS实现http高可用集群
keepalived高可用主机IP:172.21.5.22和172.21.5.21
http服务高可用主机IP:172.21.5.16和172.21.5.18
VIP采用172.16.32.5
各虚拟机及主机名和IP对应关系如下所示:
虚拟机 | 主机名 | IP地址 |
---|---|---|
HA1 | node1 | 172.21.5.22 |
HA2 | node2 | 172.21.5.21 |
HA3 | node3 | 172.21.5.16 |
HA4 | node4 | 172.21.5.18 |
接下来我们先去配置好http高可用服务两台主机,然后再回来配置keepalived。既然是LVS的DR模型,那我们先来配置另外两台主机,来实现http高可用集群。
首先,打开另外两外两台提供http服务的虚拟机,这里是HA3和HA4。为了方便你也可以改为RS1和RS2。
在DR模型中,只添加个http服务还不行,还有许多需要修改,貌似不是很简单,好吧,为了节约时间,这里我们通过一个脚本实现修改各个数据。
bash
#!/bin/bash
#
# Script to start LVS DR real server.
# chkconfig: - 90 10
# description: LVS DR real server
#
. /etc/rc.d/init.d/functions
VIP=172.16.32.5 # 定义VIP
host=$(/bin/hostname)
case "$1" in
start)
# Start LVS-DR real server on this machine.
/sbin/ifconfig lo down
/sbin/ifconfig lo up
echo 1 > /proc/sys/net/ipv4/conf/lo/arp_ignore
echo 2 > /proc/sys/net/ipv4/conf/lo/arp_announce
echo 1 > /proc/sys/net/ipv4/conf/all/arp_ignore
echo 2 > /proc/sys/net/ipv4/conf/all/arp_announce
/sbin/ifconfig lo:0 $VIP broadcast $VIP netmask 255.255.255.255 up
/sbin/route add -host $VIP dev lo:0
;;
stop)
# Stop LVS-DR real server loopback device(s).
/sbin/ifconfig lo:0 down
echo 0 > /proc/sys/net/ipv4/conf/lo/arp_ignore
echo 0 > /proc/sys/net/ipv4/conf/lo/arp_announce
echo 0 > /proc/sys/net/ipv4/conf/all/arp_ignore
echo 0 > /proc/sys/net/ipv4/conf/all/arp_announce
;;
status)
# Status of LVS-DR real server.
islothere=$(/sbin/ifconfig lo:0 | grep $VIP)
isrothere=$(netstat -rn | grep "lo:0" | grep $VIP)
if [ ! "$islothere" -o ! "$isrothere" ]; then
# Either the route or the lo:0 device not found.
echo "LVS-DR real server Stopped."
else
echo "LVS-DR real server Running."
fi
;;
*)
# Invalid entry.
echo "$0: Usage: $0 {start|status|stop}"
exit 1
;;
esac
记得该脚本需要在另一个主机上也要执行一下。
两台主机上都执行过上边的脚本后,验证下各参数是否已经修改:下面几个命令均在node3主机上执行,可在node3上使用ssh node4 'COMMAND'来验证下node4上各参数是否已修改
bash
[root@wh ~]# ifconfig #在node3上执行该命令,查看是否有VIP,使用ssh node4 'ifconfig'命令查看node4上是否也有VIP
eth0 Link encap:Ethernet HWaddr 00:0C:29:7F:8F:44
inet addr:172.21.5.18Bcast:172.16.255.255Mask:255.255.0.0
UP BROADCAST RUNNING MULTICAST MTU:1500Metric:1
RX packets:162748errors:0dropped:0overruns:0frame:0
TX packets:2368errors:0dropped:0overruns:0carrier:0
collisions:0txqueuelen:1000
RX bytes:26502458(25.2MiB) TX bytes:200681(195.9KiB)
Interrupt:59Base address:0x2000
lo Link encap:Local Loopback
inet addr:127.0.0.1Mask:255.0.0.0
UP LOOPBACK RUNNING MTU:16436Metric:1
RX packets:10errors:0dropped:0overruns:0frame:0
TX packets:10errors:0dropped:0overruns:0carrier:0
collisions:0txqueuelen:0
RX bytes:666(666.0b) TX bytes:666(666.0b)
lo:0Link encap:Local Loopback
inet addr:172.16.32.5Mask:255.255.255.255
UP LOOPBACK RUNNING MTU:16436Metric:1
[root@wh ~]# route -n #查看是否有配置的VIP特定路由
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
172.16.32.50.0.0.0255.255.255.255UH 000lo
169.254.0.00.0.0.0255.255.0.0U 000eth0
172.16.0.00.0.0.0255.255.0.0U 000eth0
0.0.0.0172.16.0.10.0.0.0UG 000eth0
[root@wh ~]# cat /proc/sys/net/ipv4/conf/all/arp_ignore
1
[root@wh ~]# cat /proc/sys/net/ipv4/conf/all/arp_announce
2
到此,我们的两台RealServer都以配置完毕,而且其http服务也已正常工作。
现在去编辑我们的keepalived的配置文件,并修改成我们所需要的。
bash
#先在node1主机上进行修改:
[root@wh ~]# vim /etc/keepalived/keepalived.conf
global_defs {
}
vrrp_script chk_httpd {
script "killall -0 httpd"
interval 2
weight -2
fall 2
rise 1
}
vrrp_script chk_schedown {
script "[[ -f /etc/keepalived/down ]] && exit 1 || exit 0"
interval 2
weight -2
}
vrrp_instance VI_1 {
state MASTER
interface eth0
virtual_router_id 132
priority 101
advert_int 1
authentication {
auth_type PASS
auth_pass langdu
}
virtual_ipaddress {
172.16.32.5/16 dev eth0 label eth0:0
}
track_script {
chk_httpd
chk_schedown
}
notify_master "/etc/keepalived/notify.sh master"
notify_backup "/etc/keepalived/notify.sh backup"
notify_fault "/etc/keepalived/notify.sh fault"
}
virtual_server 172.16.32.5 80 {
delay_loop 6
lb_algo rr
lb_kind DR
nat_mask 255.255.0.0
protocol TCP
real_server 172.21.5.16 80 {
weight 1
HTTP_GET {
url {
path /
status_code 200
}
connect_timeout 2
nb_get_retry 3
delay_before_retry 2
}
}
real_server 172.21.5.18 80 {
weight 2
HTTP_GET {
url {
path /
status_code 200
}
connect_timeout 2
nb_get_retry 3
delay_before_retry 3
}
}
}
下面这个脚本时实现健康检查用的。即上边用到的notify.sh脚本
bash
#!/bin/bash
# Author: onlyyou
# description: 一个通知脚本示例
ifalias=${2:-eth0:0}
interface=$(echo $ifalias | awk -F: '{print $1}')
vip=$(ip addr show $interface | grep $ifalias | awk '{print $2}')
contact='root@localhost'
workspace=$(dirname $0)
notify() {
subject="$vip 地址切换为 $1"
body="$vip 地址切换为 $1 $(date '+%F %H:%M:%S')"
echo $body | mail -s "$1 过渡" $contact # 实现发送邮件
}
case "$1" in
master)
notify master
exit 0
;;
backup)
notify backup
/etc/rc.d/init.d/httpd restart
exit 0
;;
fault)
notify fault
exit 0
;;
*)
echo "用法: $(basename $0) {master|backup|fault}"
exit 1
;;
esac
bash
scp /etc/keepalived/keepalived.conf node2:/etc/keepalived/
# 将配置文件发给另一个keepalived主机,这里发给node2,假如你的主机不是node2,请做相应修改
# 在node2主机上,修改刚传过来的keepalived的配置文件。这里只需修改两项即可:
state BACKUP #设置该node2主机为backup路由
priority 100 #设定node2主机的优先级为100,低于node1
# 在node2主机上,我们只需修改上述两项即可。
修改完成后保存退出
现在我们去安装ipvsadm软件包,然后再启动keepalived服务
bash
ssh node2 'yum -y install ipvsadm' #通过node1主机在node2上安装ipvsadm软件包
yum -y install ipvsadm #在node1上安装软件包
安装完毕后启动我们的keepalived服务。
service keepalived start #启动node1上的keepalived服务
ssh node2 'service keepalived start' #在node1上启动node2的keepalived服务
查看下ipvs规则
bash
ipvsadm -l -n #使用该命令,查看是否有ipvs规则,显示如下:
IP Virtual Server version 1.2.1(size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 172.16.32.5:80rr
-> 172.21.5.18:80Route 100
-> 172.21.5.16:80Route 100
bash
ifconfig #查看下node1上的IP配置情况
eth0 Link encap:Ethernet HWaddr 00:0C:29:9F:2F:AF
inet addr:172.21.5.22Bcast:172.16.255.255Mask:255.255.0.0
UP BROADCAST RUNNING MULTICAST MTU:1500Metric:1
RX packets:290653errors:1dropped:0overruns:0frame:0
TX packets:13874errors:0dropped:0overruns:0carrier:0
collisions:0txqueuelen:1000
RX bytes:46393162(44.2MiB) TX bytes:2014631(1.9MiB)
Interrupt:59Base address:0x2000
eth0:0Link encap:Ethernet HWaddr 00:0C:29:9F:2F:AF
inet addr:172.16.32.5Bcast:0.0.0.0Mask:255.255.0.0
UP BROADCAST RUNNING MULTICAST MTU:1500Metric:1
Interrupt:59Base address:0x2000
lo Link encap:Local Loopback
inet addr:127.0.0.1Mask:255.0.0.0
UP LOOPBACK RUNNING MTU:16436Metric:1
RX packets:10errors:0dropped:0overruns:0frame:0
TX packets:10errors:0dropped:0overruns:0carrier:0
collisions:0txqueuelen:0
RX bytes:666(666.0b) TX bytes:666(666.0b)
配置的VIP也已生效,我们的ipvs规则已经实现。现在在我们的物理机上访问下172.16.32.5,看一下,是否可以访问,显示什么信息吧。
到目前为止,貌似我们还没有实现IP地址漂移。我们只需在master路由主机上,在相应目录下创建一个down文件即可实现手动漂移IP地址。
在node1上
bash
cd /etc/keepalived/#进入该目录
touch down #创建该文件,用来实现手动漂移IP地址
bash
tail /var/log/messages #停几秒钟后,查看日志
May 1619:15:32node1 Keepalived_vrrp[2816]: VRRP_Instance(VI_1) Entering MASTER STATE
May 1619:15:32node1 Keepalived_vrrp[2816]: VRRP_Instance(VI_1) setting protocol VIPs.
May 1619:15:32node1 Keepalived_vrrp[2816]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for172.16.32.5
May 1619:15:32node1 Keepalived_healthcheckers[2815]: Netlink reflector reports IP 172.16.32.5added
May 1619:15:32node1 Keepalived_vrrp[2816]: Netlink reflector reports IP 172.16.32.5added
May 1619:15:32node1 Keepalived_vrrp[2816]: VRRP_Instance(VI_1) Received higher prio advert #收到更高优先级的通告信息
May 1619:15:32node1 Keepalived_vrrp[2816]: VRRP_Instance(VI_1) Entering BACKUP STATE #进入backup状态
May 1619:15:32node1 Keepalived_vrrp[2816]: VRRP_Instance(VI_1) removing protocol VIPs. #转移VIP地址
May 1619:15:32node1 Keepalived_healthcheckers[2815]: Netlink reflector reports IP 172.16.32.5removed
May 1619:15:32node1 Keepalived_vrrp[2816]: Netlink reflector reports IP 172.16.32.5removed
bash
ifconfig #使用该命令,查看下node1主机的VIP是否存在,可看到已转移到其他主机
eth0 Link encap:Ethernet HWaddr 00:0C:29:9F:2F:AF
inet addr:172.21.5.22Bcast:172.16.255.255Mask:255.255.0.0
UP BROADCAST RUNNING MULTICAST MTU:1500Metric:1
RX packets:347881errors:1dropped:0overruns:0frame:0
TX packets:21333errors:0dropped:0overruns:0carrier:0
collisions:0txqueuelen:1000
RX bytes:50760780(48.4MiB) TX bytes:2653767(2.5MiB)
Interrupt:59Base address:0x2000
lo Link encap:Local Loopback
inet addr:127.0.0.1Mask:255.0.0.0
UP LOOPBACK RUNNING MTU:16436Metric:1
RX packets:10errors:0dropped:0overruns:0frame:0
TX packets:10errors:0dropped:0overruns:0carrier:0
collisions:0txqueuelen:0
RX bytes:666(666.0b) TX bytes:666(666.0b)
> 在node2主机上查看其日志。
```bash
tail /var/log/messages #查看node2的日志信息
May 1619:15:32node1 Keepalived_vrrp[2464]: VRRP_Instance(VI_1) forcing a new MASTER election
May 1619:15:32node1 Keepalived_vrrp[2464]: VRRP_Instance(VI_1) forcing a new MASTER election
May 1619:15:33node1 Keepalived_vrrp[2464]: VRRP_Instance(VI_1) Transition to MASTER STATE
May 1619:15:34node1 Keepalived_vrrp[2464]: VRRP_Instance(VI_1) Entering MASTER STATE #进入master状态
May 1619:15:34node1 Keepalived_vrrp[2464]: VRRP_Instance(VI_1) setting protocol VIPs. #设定VIP
May 1619:15:34node1 Keepalived_healthcheckers[2463]: Netlink reflector reports IP 172.16.32.5added
May 1619:15:34node1 avahi-daemon[3375]: Registering new address record for172.16.32.5on eth0.
May 1619:15:34node1 Keepalived_vrrp[2464]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for172.16.32.5
May 1619:15:34node1 Keepalived_vrrp[2464]: Netlink reflector reports IP 172.16.32.5added
May 1619:15:39node1 Keepalived_vrrp[2464]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for172.16.32.5
在在我们的物理机访问下,看能否访问。依然在浏览器地址栏输入172.16.32.5,可以看到,访问正常。
现在,我们删掉/etc/keepalived/down这个文件,看能否实现IP漂移回来。
bash
rm -f /etc/keepalived/down #删除node1主机上该文件
bash
tail /var/log/messages #查看日志信息
May 1619:15:32node1 Keepalived_vrrp[2816]: Netlink reflector reports IP 172.16.32.5removed
May 1619:27:54node1 Keepalived_vrrp[2816]: VRRP_Script(chk_schedown) succeeded
May 1619:27:55node1 Keepalived_vrrp[2816]: VRRP_Instance(VI_1) forcing a new MASTER election
May 1619:27:55node1 Keepalived_vrrp[2816]: VRRP_Instance(VI_1) forcing a new MASTER election
May 1619:27:56node1 Keepalived_vrrp[2816]: VRRP_Instance(VI_1) Transition to MASTER STATE
May 1619:27:57node1 Keepalived_vrrp[2816]: VRRP_Instance(VI_1) Entering MASTER STATE #进入master状态
May 1619:27:57node1 Keepalived_vrrp[2816]: VRRP_Instance(VI_1) setting protocol VIPs.
May 1619:27:57node1 Keepalived_vrrp[2816]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for172.16.32.5
May 1619:27:57node1 Keepalived_healthcheckers[2815]: Netlink reflector reports IP 172.16.32.5added
May 1619:27:57node1 Keepalived_vrrp[2816]: Netlink reflector reports IP 172.16.32.5added
至此,我们已成功实现了keepalived的相关功能。以上演示的仅仅是主从模式下地址漂移。那我们能否实现在双主模式下实现地址漂移呢?答案是肯定的。