lvs dr+keepalived - 技术栈

基于keepalived(主从+双主) + LVS(DR模型) + DNS实现http高可用集群

keepalived高可用主机IP：172.21.5.22和172.21.5.21

http服务高可用主机IP：172.21.5.16和172.21.5.18

VIP采用172.16.32.5

各虚拟机及主机名和IP对应关系如下所示：

虚拟机	主机名	IP地址
HA1	node1	172.21.5.22
HA2	node2	172.21.5.21
HA3	node3	172.21.5.16
HA4	node4	172.21.5.18

接下来我们先去配置好http高可用服务两台主机，然后再回来配置keepalived。既然是LVS的DR模型，那我们先来配置另外两台主机，来实现http高可用集群。

首先，打开另外两外两台提供http服务的虚拟机，这里是HA3和HA4。为了方便你也可以改为RS1和RS2。
在DR模型中，只添加个http服务还不行，还有许多需要修改，貌似不是很简单，好吧，为了节约时间，这里我们通过一个脚本实现修改各个数据。

bash 复制代码

#!/bin/bash
#
# Script to start LVS DR real server.
# chkconfig: - 90 10
# description: LVS DR real server
#

. /etc/rc.d/init.d/functions

VIP=172.16.32.5 # 定义VIP
host=$(/bin/hostname)

case "$1" in
start)
    # Start LVS-DR real server on this machine.
    /sbin/ifconfig lo down
    /sbin/ifconfig lo up
    echo 1 > /proc/sys/net/ipv4/conf/lo/arp_ignore
    echo 2 > /proc/sys/net/ipv4/conf/lo/arp_announce
    echo 1 > /proc/sys/net/ipv4/conf/all/arp_ignore
    echo 2 > /proc/sys/net/ipv4/conf/all/arp_announce
    /sbin/ifconfig lo:0 $VIP broadcast $VIP netmask 255.255.255.255 up
    /sbin/route add -host $VIP dev lo:0
    ;;
stop)
    # Stop LVS-DR real server loopback device(s).
    /sbin/ifconfig lo:0 down
    echo 0 > /proc/sys/net/ipv4/conf/lo/arp_ignore
    echo 0 > /proc/sys/net/ipv4/conf/lo/arp_announce
    echo 0 > /proc/sys/net/ipv4/conf/all/arp_ignore
    echo 0 > /proc/sys/net/ipv4/conf/all/arp_announce
    ;;
status)
    # Status of LVS-DR real server.
    islothere=$(/sbin/ifconfig lo:0 | grep $VIP)
    isrothere=$(netstat -rn | grep "lo:0" | grep $VIP)
    if [ ! "$islothere" -o ! "$isrothere" ]; then
        # Either the route or the lo:0 device not found.
        echo "LVS-DR real server Stopped."
    else
        echo "LVS-DR real server Running."
    fi
    ;;
*)
    # Invalid entry.
    echo "$0: Usage: $0 {start|status|stop}"
    exit 1
    ;;
esac

记得该脚本需要在另一个主机上也要执行一下。

两台主机上都执行过上边的脚本后，验证下各参数是否已经修改：下面几个命令均在node3主机上执行，可在node3上使用ssh node4 'COMMAND'来验证下node4上各参数是否已修改

bash 复制代码

[root@wh ~]# ifconfig    #在node3上执行该命令，查看是否有VIP，使用ssh node4 'ifconfig'命令查看node4上是否也有VIP
eth0 Link encap:Ethernet  HWaddr 00:0C:29:7F:8F:44
	inet addr:172.21.5.18Bcast:172.16.255.255Mask:255.255.0.0
	UP BROADCAST RUNNING MULTICAST  MTU:1500Metric:1
	RX packets:162748errors:0dropped:0overruns:0frame:0
	TX packets:2368errors:0dropped:0overruns:0carrier:0
	collisions:0txqueuelen:1000
	RX bytes:26502458(25.2MiB)  TX bytes:200681(195.9KiB)
	Interrupt:59Base address:0x2000
lo  Link encap:Local Loopback
	inet addr:127.0.0.1Mask:255.0.0.0
	UP LOOPBACK RUNNING  MTU:16436Metric:1
	RX packets:10errors:0dropped:0overruns:0frame:0
	TX packets:10errors:0dropped:0overruns:0carrier:0
	collisions:0txqueuelen:0
	RX bytes:666(666.0b)  TX bytes:666(666.0b)
	lo:0Link encap:Local Loopback
	inet addr:172.16.32.5Mask:255.255.255.255
	UP LOOPBACK RUNNING  MTU:16436Metric:1
[root@wh ~]# route -n   #查看是否有配置的VIP特定路由
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
172.16.32.50.0.0.0255.255.255.255UH    000lo
169.254.0.00.0.0.0255.255.0.0U     000eth0
172.16.0.00.0.0.0255.255.0.0U     000eth0
0.0.0.0172.16.0.10.0.0.0UG    000eth0
[root@wh ~]# cat /proc/sys/net/ipv4/conf/all/arp_ignore
1
[root@wh ~]# cat /proc/sys/net/ipv4/conf/all/arp_announce
2

到此，我们的两台RealServer都以配置完毕，而且其http服务也已正常工作。
现在去编辑我们的keepalived的配置文件，并修改成我们所需要的。

bash 复制代码

#先在node1主机上进行修改：
[root@wh ~]# vim /etc/keepalived/keepalived.conf
global_defs {

}

vrrp_script chk_httpd {
    script "killall -0 httpd"
    interval 2
    weight -2
    fall 2
    rise 1
}

vrrp_script chk_schedown {
    script "[[ -f /etc/keepalived/down ]] && exit 1 || exit 0"
    interval 2
    weight -2
}

vrrp_instance VI_1 {
    state MASTER
    interface eth0
    virtual_router_id 132
    priority 101
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass langdu
    }
    virtual_ipaddress {
        172.16.32.5/16 dev eth0 label eth0:0
    }
    track_script {
        chk_httpd
        chk_schedown
    }
    notify_master "/etc/keepalived/notify.sh master"
    notify_backup "/etc/keepalived/notify.sh backup"
    notify_fault "/etc/keepalived/notify.sh fault"
}

virtual_server 172.16.32.5 80 {
    delay_loop 6
    lb_algo rr
    lb_kind DR
    nat_mask 255.255.0.0
    protocol TCP

    real_server 172.21.5.16 80 {
        weight 1
        HTTP_GET {
            url {
                path /
                status_code 200
            }
            connect_timeout 2
            nb_get_retry 3
            delay_before_retry 2
        }
    }

    real_server 172.21.5.18 80 {
        weight 2
        HTTP_GET {
            url {
                path /
                status_code 200
            }
            connect_timeout 2
            nb_get_retry 3
            delay_before_retry 3
        }
    }
}

下面这个脚本时实现健康检查用的。即上边用到的notify.sh脚本

bash 复制代码

#!/bin/bash
# Author: onlyyou
# description: 一个通知脚本示例

ifalias=${2:-eth0:0}
interface=$(echo $ifalias | awk -F: '{print $1}')
vip=$(ip addr show $interface | grep $ifalias | awk '{print $2}')
contact='root@localhost'
workspace=$(dirname $0)

notify() {
    subject="$vip 地址切换为 $1"
    body="$vip 地址切换为 $1 $(date '+%F %H:%M:%S')"
    echo $body | mail -s "$1 过渡" $contact # 实现发送邮件
}

case "$1" in
master)
    notify master
    exit 0
    ;;
backup)
    notify backup
    /etc/rc.d/init.d/httpd restart
    exit 0
    ;;
fault)
    notify fault
    exit 0
    ;;
*)
    echo "用法: $(basename $0) {master|backup|fault}"
    exit 1
    ;;
esac

bash 复制代码

scp /etc/keepalived/keepalived.conf node2:/etc/keepalived/
# 将配置文件发给另一个keepalived主机，这里发给node2，假如你的主机不是node2，请做相应修改
# 在node2主机上，修改刚传过来的keepalived的配置文件。这里只需修改两项即可：
state BACKUP          #设置该node2主机为backup路由

priority 100 #设定node2主机的优先级为100，低于node1
# 在node2主机上，我们只需修改上述两项即可。

修改完成后保存退出
现在我们去安装ipvsadm软件包，然后再启动keepalived服务

bash 复制代码

ssh node2 'yum -y install ipvsadm'   #通过node1主机在node2上安装ipvsadm软件包
yum -y install ipvsadm   #在node1上安装软件包
安装完毕后启动我们的keepalived服务。
service keepalived start    #启动node1上的keepalived服务
ssh node2 'service keepalived start'   #在node1上启动node2的keepalived服务

查看下ipvs规则

bash 复制代码

ipvsadm -l -n   #使用该命令，查看是否有ipvs规则，显示如下：
IP Virtual Server version 1.2.1(size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  172.16.32.5:80rr
-> 172.21.5.18:80Route   100
-> 172.21.5.16:80Route   100

bash 复制代码

ifconfig    #查看下node1上的IP配置情况
eth0      Link encap:Ethernet  HWaddr 00:0C:29:9F:2F:AF
	inet addr:172.21.5.22Bcast:172.16.255.255Mask:255.255.0.0
	UP BROADCAST RUNNING MULTICAST  MTU:1500Metric:1
	RX packets:290653errors:1dropped:0overruns:0frame:0
	TX packets:13874errors:0dropped:0overruns:0carrier:0
	collisions:0txqueuelen:1000
	RX bytes:46393162(44.2MiB)  TX bytes:2014631(1.9MiB)
	Interrupt:59Base address:0x2000
eth0:0Link encap:Ethernet  HWaddr 00:0C:29:9F:2F:AF
	inet addr:172.16.32.5Bcast:0.0.0.0Mask:255.255.0.0
	UP BROADCAST RUNNING MULTICAST  MTU:1500Metric:1
	Interrupt:59Base address:0x2000
lo        Link encap:Local Loopback
	inet addr:127.0.0.1Mask:255.0.0.0
	UP LOOPBACK RUNNING  MTU:16436Metric:1
	RX packets:10errors:0dropped:0overruns:0frame:0
	TX packets:10errors:0dropped:0overruns:0carrier:0
	collisions:0txqueuelen:0
	RX bytes:666(666.0b)  TX bytes:666(666.0b)

配置的VIP也已生效，我们的ipvs规则已经实现。现在在我们的物理机上访问下172.16.32.5，看一下，是否可以访问，显示什么信息吧。
到目前为止，貌似我们还没有实现IP地址漂移。我们只需在master路由主机上，在相应目录下创建一个down文件即可实现手动漂移IP地址。
在node1上

bash 复制代码

cd /etc/keepalived/#进入该目录
touch down    #创建该文件，用来实现手动漂移IP地址

bash 复制代码

tail /var/log/messages    #停几秒钟后，查看日志
May 1619:15:32node1 Keepalived_vrrp[2816]: VRRP_Instance(VI_1) Entering MASTER STATE
May 1619:15:32node1 Keepalived_vrrp[2816]: VRRP_Instance(VI_1) setting protocol VIPs.
May 1619:15:32node1 Keepalived_vrrp[2816]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for172.16.32.5
May 1619:15:32node1 Keepalived_healthcheckers[2815]: Netlink reflector reports IP 172.16.32.5added
May 1619:15:32node1 Keepalived_vrrp[2816]: Netlink reflector reports IP 172.16.32.5added
May 1619:15:32node1 Keepalived_vrrp[2816]: VRRP_Instance(VI_1) Received higher prio advert    #收到更高优先级的通告信息
May 1619:15:32node1 Keepalived_vrrp[2816]: VRRP_Instance(VI_1) Entering BACKUP STATE     #进入backup状态
May 1619:15:32node1 Keepalived_vrrp[2816]: VRRP_Instance(VI_1) removing protocol VIPs.     #转移VIP地址
May 1619:15:32node1 Keepalived_healthcheckers[2815]: Netlink reflector reports IP 172.16.32.5removed
May 1619:15:32node1 Keepalived_vrrp[2816]: Netlink reflector reports IP 172.16.32.5removed

bash 复制代码

ifconfig   #使用该命令，查看下node1主机的VIP是否存在，可看到已转移到其他主机
eth0      Link encap:Ethernet  HWaddr 00:0C:29:9F:2F:AF
	inet addr:172.21.5.22Bcast:172.16.255.255Mask:255.255.0.0
	UP BROADCAST RUNNING MULTICAST  MTU:1500Metric:1
	RX packets:347881errors:1dropped:0overruns:0frame:0
	TX packets:21333errors:0dropped:0overruns:0carrier:0
	collisions:0txqueuelen:1000
	RX bytes:50760780(48.4MiB)  TX bytes:2653767(2.5MiB)
	Interrupt:59Base address:0x2000
lo        Link encap:Local Loopback
	inet addr:127.0.0.1Mask:255.0.0.0
	UP LOOPBACK RUNNING  MTU:16436Metric:1
	RX packets:10errors:0dropped:0overruns:0frame:0
	TX packets:10errors:0dropped:0overruns:0carrier:0
	collisions:0txqueuelen:0
	RX bytes:666(666.0b)  TX bytes:666(666.0b)

> 在node2主机上查看其日志。
```bash
tail /var/log/messages   #查看node2的日志信息
May 1619:15:32node1 Keepalived_vrrp[2464]: VRRP_Instance(VI_1) forcing a new MASTER election
May 1619:15:32node1 Keepalived_vrrp[2464]: VRRP_Instance(VI_1) forcing a new MASTER election
May 1619:15:33node1 Keepalived_vrrp[2464]: VRRP_Instance(VI_1) Transition to MASTER STATE
May 1619:15:34node1 Keepalived_vrrp[2464]: VRRP_Instance(VI_1) Entering MASTER STATE    #进入master状态
May 1619:15:34node1 Keepalived_vrrp[2464]: VRRP_Instance(VI_1) setting protocol VIPs.    #设定VIP
May 1619:15:34node1 Keepalived_healthcheckers[2463]: Netlink reflector reports IP 172.16.32.5added
May 1619:15:34node1 avahi-daemon[3375]: Registering new address record for172.16.32.5on eth0.
May 1619:15:34node1 Keepalived_vrrp[2464]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for172.16.32.5
May 1619:15:34node1 Keepalived_vrrp[2464]: Netlink reflector reports IP 172.16.32.5added
May 1619:15:39node1 Keepalived_vrrp[2464]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for172.16.32.5

在在我们的物理机访问下，看能否访问。依然在浏览器地址栏输入172.16.32.5，可以看到，访问正常。

现在，我们删掉/etc/keepalived/down这个文件，看能否实现IP漂移回来。

bash 复制代码

rm -f /etc/keepalived/down      #删除node1主机上该文件

bash 复制代码

tail /var/log/messages    #查看日志信息
May 1619:15:32node1 Keepalived_vrrp[2816]: Netlink reflector reports IP 172.16.32.5removed
May 1619:27:54node1 Keepalived_vrrp[2816]: VRRP_Script(chk_schedown) succeeded
May 1619:27:55node1 Keepalived_vrrp[2816]: VRRP_Instance(VI_1) forcing a new MASTER election
May 1619:27:55node1 Keepalived_vrrp[2816]: VRRP_Instance(VI_1) forcing a new MASTER election
May 1619:27:56node1 Keepalived_vrrp[2816]: VRRP_Instance(VI_1) Transition to MASTER STATE
May 1619:27:57node1 Keepalived_vrrp[2816]: VRRP_Instance(VI_1) Entering MASTER STATE    #进入master状态
May 1619:27:57node1 Keepalived_vrrp[2816]: VRRP_Instance(VI_1) setting protocol VIPs.
May 1619:27:57node1 Keepalived_vrrp[2816]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for172.16.32.5
May 1619:27:57node1 Keepalived_healthcheckers[2815]: Netlink reflector reports IP 172.16.32.5added
May 1619:27:57node1 Keepalived_vrrp[2816]: Netlink reflector reports IP 172.16.32.5added

至此，我们已成功实现了keepalived的相关功能。以上演示的仅仅是主从模式下地址漂移。那我们能否实现在双主模式下实现地址漂移呢？答案是肯定的。