lvs的nat和dr模式混合用

马昌伟2024-07-20 17:26

机器部署信息

lvs ：

10.0.0.200 vip

10.0.0.19 外网IP ，

172.168.1.19 内网IP

dr rs:

10.0.0.200 vip

10.0.0.18 rip

nat rs:

172.168.1.17 rip

客户端：

10.0.0.14 cip

lvs机器：

ip addr add 10.0.0.200/24 dev ens33:0

IP：

复制代码 
[root@mcw09 ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:f0:dd:56 brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.19/24 brd 10.0.0.255 scope global ens33
       valid_lft forever preferred_lft forever
    inet 10.0.0.200/24 scope global secondary ens33
       valid_lft forever preferred_lft forever
    inet6 fe80::495b:ff7:d185:f95d/64 scope link 
       valid_lft forever preferred_lft forever
    inet6 fe80::9335:fbc:5cf6:ad83/64 scope link tentative dadfailed 
       valid_lft forever preferred_lft forever
3: ens34: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:f0:dd:60 brd ff:ff:ff:ff:ff:ff
    inet 172.168.1.19/24 brd 172.168.1.255 scope global ens34
       valid_lft forever preferred_lft forever
    inet6 fe80::64e9:3463:3319:8689/64 scope link 
       valid_lft forever preferred_lft forever
    inet6 fe80::428e:4a2b:802a:fccc/64 scope link tentative dadfailed 
       valid_lft forever preferred_lft forever
    inet6 fe80::c7c4:97e9:a77b:a70b/64 scope link tentative dadfailed 
       valid_lft forever preferred_lft forever
[root@mcw09 ~]#

路由没有啥变动，

复制代码 
[root@mcw09 ~]# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         172.168.1.254   0.0.0.0         UG    100    0        0 ens34
0.0.0.0         10.0.0.254      0.0.0.0         UG    101    0        0 ens33
10.0.0.0        0.0.0.0         255.255.255.0   U     100    0        0 ens33
172.168.1.0     0.0.0.0         255.255.255.0   U     100    0        0 ens34
[root@mcw09 ~]#

dr的rs和lvs用同一个网段，nat的转发和lvs可以是两个网段

复制代码 
[root@mcw09 ~]# ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  10.0.0.200:80 rr
  -> 172.168.1.17:80              Masq    1      0          0         
  -> 10.0.0.18:80                 Route   1      0          0         
[root@mcw09 ~]#

dr机器：

route add -host 10.0.0.200 dev lo

复制代码 
[root@mcw08 ~]# cat /etc/sysctl.conf
net.ipv4.conf.all.arp_ignore = 1
net.ipv4.conf.all.arp_announce = 2
net.ipv4.conf.lo.arp_ignore = 1
net.ipv4.conf.lo.arp_announce = 2

lo添加了VIP

复制代码 
[root@mcw08 ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet 10.0.0.200/32 scope global lo:0
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:26:33:3f brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.18/24 brd 10.0.0.255 scope global ens33
       valid_lft forever preferred_lft forever
    inet6 fe80::f32c:166d:40de:8f2e/64 scope link tentative dadfailed 
       valid_lft forever preferred_lft forever
    inet6 fe80::495b:ff7:d185:f95d/64 scope link tentative dadfailed 
       valid_lft forever preferred_lft forever
    inet6 fe80::9335:fbc:5cf6:ad83/64 scope link tentative dadfailed 
       valid_lft forever preferred_lft forever
3: ens34: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:26:33:49 brd ff:ff:ff:ff:ff:ff
    inet 172.168.1.18/24 brd 172.168.1.255 scope global ens34
       valid_lft forever preferred_lft forever
    inet6 fe80::64e9:3463:3319:8689/64 scope link tentative dadfailed 
       valid_lft forever preferred_lft forever
    inet6 fe80::428e:4a2b:802a:fccc/64 scope link tentative dadfailed 
       valid_lft forever preferred_lft forever
    inet6 fe80::c7c4:97e9:a77b:a70b/64 scope link tentative dadfailed 
       valid_lft forever preferred_lft forever
15: tun0: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN qlen 100
    link/none 
    inet 10.8.0.1/24 brd 10.8.0.255 scope global tun0
       valid_lft forever preferred_lft forever
    inet6 fe80::923d:6caf:c22:c8a5/64 scope link flags 800 
       valid_lft forever preferred_lft forever
[root@mcw08 ~]#

nat机器：

复制代码 
[root@mcw07 ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:5d:df:62 brd ff:ff:ff:ff:ff:ff
3: ens34: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:5d:df:6c brd ff:ff:ff:ff:ff:ff
    inet 172.168.1.17/24 brd 172.168.1.255 scope global ens34
       valid_lft forever preferred_lft forever
    inet6 fe80::64e9:3463:3319:8689/64 scope link tentative dadfailed 
       valid_lft forever preferred_lft forever
    inet6 fe80::428e:4a2b:802a:fccc/64 scope link tentative dadfailed 
       valid_lft forever preferred_lft forever
    inet6 fe80::c7c4:97e9:a77b:a70b/64 scope link tentative dadfailed 
       valid_lft forever preferred_lft forever
[root@mcw07 ~]#

只开启了一个内网IP，并且默认网关指向dip。这里的dip和rip是同一个网段的。不在同一个网段的不清楚咋弄

复制代码 
[root@mcw07 ~]# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         172.168.1.19    0.0.0.0         UG    0      0        0 ens34
172.168.1.0     0.0.0.0         255.255.255.0   U     100    0        0 ens34
[root@mcw07 ~]#

测试：

mcw04上测试，访问lvs上VIP 10.0.0.200,从而访问到两个rs,一个rs是nat，一个是dr

添加其他网络测试

当把nat的另外一个网卡起起来之后

复制代码 
[root@mcw07 ~]# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         172.168.1.19    0.0.0.0         UG    0      0        0 ens34
172.168.1.0     0.0.0.0         255.255.255.0   U     100    0        0 ens34
[root@mcw07 ~]# ifup ens33
Connection successfully activated (D-Bus active path: /org/freedesktop/NetworkManager/ActiveConnection/45)
[root@mcw07 ~]# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         172.168.1.19    0.0.0.0         UG    0      0        0 ens34
0.0.0.0         10.0.0.254      0.0.0.0         UG    100    0        0 ens33
10.0.0.0        0.0.0.0         255.255.255.0   U     100    0        0 ens33
172.168.1.0     0.0.0.0         255.255.255.0   U     100    0        0 ens34
[root@mcw07 ~]#

nat的这个rs机器就不能访问到了

复制代码 
[root@mcw04 ~]# curl 10.0.0.200:80
curl: (7) Failed connect to 10.0.0.200:80; Connection timed out
[root@mcw04 ~]# curl 10.0.0.200:80
rs1 mcw08 ^_^ 10.0.0.18
[root@mcw04 ~]# curl 10.0.0.200:80
curl: (7) Failed connect to 10.0.0.200:80; Connection timed out
[root@mcw04 ~]# curl 10.0.0.200:80
rs1 mcw08 ^_^ 10.0.0.18
[root@mcw04 ~]#

删掉一条nat的rs的默认路由之后还是无法访问到nat的

复制代码 
[root@mcw07 ~]# 
[root@mcw07 ~]# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         172.168.1.19    0.0.0.0         UG    0      0        0 ens34
0.0.0.0         10.0.0.254      0.0.0.0         UG    100    0        0 ens33
10.0.0.0        0.0.0.0         255.255.255.0   U     100    0        0 ens33
172.168.1.0     0.0.0.0         255.255.255.0   U     100    0        0 ens34
[root@mcw07 ~]# ip route del default via  10.0.0.254 dev ens33
[root@mcw07 ~]# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         172.168.1.19    0.0.0.0         UG    0      0        0 ens34
10.0.0.0        0.0.0.0         255.255.255.0   U     100    0        0 ens33
172.168.1.0     0.0.0.0         255.255.255.0   U     100    0        0 ens34
[root@mcw07 ~]#
复制代码 
[root@mcw04 ~]# curl 10.0.0.200:80
rs1 mcw08 ^_^ 10.0.0.18
[root@mcw04 ~]# curl 10.0.0.200:80
curl: (7) Failed connect to 10.0.0.200:80; Connection timed out
[root@mcw04 ~]#

因为内网dip做路由

内网网卡卡做默认路由时，如何去通外网。需要添加外网网段指向外网网卡(lvs nat rs中有多网卡处理案例)

dip和rip是同一个内网，因为将nat模型的lvs的dip作为默认网关后，这样ens33的能通223.5.5.5的网卡，现在不通了

复制代码 
[root@mcw07 ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:5d:df:62 brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.17/24 brd 10.0.0.255 scope global ens33
       valid_lft forever preferred_lft forever
    inet6 fe80::f32c:166d:40de:8f2e/64 scope link tentative dadfailed 
       valid_lft forever preferred_lft forever
    inet6 fe80::495b:ff7:d185:f95d/64 scope link tentative dadfailed 
       valid_lft forever preferred_lft forever
    inet6 fe80::9335:fbc:5cf6:ad83/64 scope link tentative dadfailed 
       valid_lft forever preferred_lft forever
3: ens34: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:5d:df:6c brd ff:ff:ff:ff:ff:ff
    inet 172.168.1.17/24 brd 172.168.1.255 scope global ens34
       valid_lft forever preferred_lft forever
    inet6 fe80::64e9:3463:3319:8689/64 scope link tentative dadfailed 
       valid_lft forever preferred_lft forever
    inet6 fe80::428e:4a2b:802a:fccc/64 scope link tentative dadfailed 
       valid_lft forever preferred_lft forever
    inet6 fe80::c7c4:97e9:a77b:a70b/64 scope link tentative dadfailed 
       valid_lft forever preferred_lft forever
[root@mcw07 ~]# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         172.168.1.19    0.0.0.0         UG    100    0        0 ens34
10.0.0.0        0.0.0.0         255.255.255.0   U     100    0        0 ens33
172.168.1.0     0.0.0.0         255.255.255.0   U     100    0        0 ens34
[root@mcw07 ~]# 
[root@mcw07 ~]# 
[root@mcw07 ~]# 
[root@mcw07 ~]# ping 223.5.5.5
PING 223.5.5.5 (223.5.5.5) 56(84) bytes of data.
From 172.168.1.19 icmp_seq=1 Redirect Host(New nexthop: 172.168.1.254)
From 172.168.1.19: icmp_seq=1 Redirect Host(New nexthop: 172.168.1.254)
^C
--- 223.5.5.5 ping statistics ---
3 packets transmitted, 0 received, +1 errors, 100% packet loss, time 2003ms

[root@mcw07 ~]#

添加走223.5.5.5的，还是用ens33网卡，这个网卡之前默认网关是10.0.0.254.现在直接指定走这个网段的，都指定网关和网卡。这样就能重新通223.5.5.5这个网络了。

ip route add 223.0.0.0/8 via 10.0.0.254 dev ens33

复制代码 
[root@mcw07 ~]# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         172.168.1.19    0.0.0.0         UG    100    0        0 ens34
10.0.0.0        0.0.0.0         255.255.255.0   U     100    0        0 ens33
172.168.1.0     0.0.0.0         255.255.255.0   U     100    0        0 ens34
[root@mcw07 ~]# ip route add 223.0.0.0/8 via 10.0.0.254 dev ens33
[root@mcw07 ~]# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         172.168.1.19    0.0.0.0         UG    100    0        0 ens34
10.0.0.0        0.0.0.0         255.255.255.0   U     100    0        0 ens33
172.168.1.0     0.0.0.0         255.255.255.0   U     100    0        0 ens34
223.0.0.0       10.0.0.254      255.0.0.0       UG    0      0        0 ens33
[root@mcw07 ~]# ping 223.5.5.5
PING 223.5.5.5 (223.5.5.5) 56(84) bytes of data.
64 bytes from 223.5.5.5: icmp_seq=1 ttl=128 time=9.00 ms
64 bytes from 223.5.5.5: icmp_seq=2 ttl=128 time=7.96 ms
^C
--- 223.5.5.5 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 7.969/8.485/9.002/0.524 ms
[root@mcw07 ~]#

将下面跟ens33网卡的路由删除。最后只保留ens34的两条路由。这样情况下，两个网卡内的网段，发现网络都是互通的，也就是路由上没有这个网卡的配置，这个网卡相关的网段好像也是可以直接通的。再添加一个走223.0.0.0网段的路由，走ens33网卡接口和它对应的网关，这样就能通223.5.5.5了。

10.0.0.0 0.0.0.0 255.255.255.0 U 100 0 0 ens33

复制代码 
[root@mcw07 ~]# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         172.168.1.19    0.0.0.0         UG    100    0        0 ens34
10.0.0.0        0.0.0.0         255.255.255.0   U     100    0        0 ens33
172.168.1.0     0.0.0.0         255.255.255.0   U     100    0        0 ens34
[root@mcw07 ~]# ip route add 223.0.0.0/8 via 10.0.0.254 dev ens33
[root@mcw07 ~]# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         172.168.1.19    0.0.0.0         UG    100    0        0 ens34
10.0.0.0        0.0.0.0         255.255.255.0   U     100    0        0 ens33
172.168.1.0     0.0.0.0         255.255.255.0   U     100    0        0 ens34
223.0.0.0       10.0.0.254      255.0.0.0       UG    0      0        0 ens33
[root@mcw07 ~]# ip route del 10.0.0.0/24 dev ens33
[root@mcw07 ~]# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         172.168.1.19    0.0.0.0         UG    100    0        0 ens34
172.168.1.0     0.0.0.0         255.255.255.0   U     100    0        0 ens34
223.0.0.0       10.0.0.254      255.0.0.0       UG    0      0        0 ens33
[root@mcw07 ~]# ping 223.5.5.5
PING 223.5.5.5 (223.5.5.5) 56(84) bytes of data.
64 bytes from 223.5.5.5: icmp_seq=1 ttl=128 time=25.8 ms
64 bytes from 223.5.5.5: icmp_seq=2 ttl=128 time=8.13 ms
^C
--- 223.5.5.5 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1002ms
rtt min/avg/max/mdev = 8.133/16.975/25.818/8.843 ms
[root@mcw07 ~]# ping 10.0.0.18
PING 10.0.0.18 (10.0.0.18) 56(84) bytes of data.
64 bytes from 10.0.0.18: icmp_seq=1 ttl=64 time=6.44 ms
64 bytes from 10.0.0.18: icmp_seq=2 ttl=64 time=0.690 ms
^C
--- 10.0.0.18 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 0.690/3.569/6.449/2.880 ms
[root@mcw07 ~]# ping 10.0.0.19
PING 10.0.0.19 (10.0.0.19) 56(84) bytes of data.
64 bytes from 10.0.0.19: icmp_seq=1 ttl=64 time=0.919 ms
^C
--- 10.0.0.19 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.919/0.919/0.919/0.000 ms
[root@mcw07 ~]#

现在下面ens33的跟10.0.0.0/24 10.0.0.254 相关的路由都已经删除，只保留了ens34的172.168.1.0/24 172.168.1.254这个两条路由，然后新增一个走向223.5.5.5这个外网IP时，走ens33网卡接口的路由，指定ens33的网关，这样223.5.5.5在mcw07上由不通变为通。并且不会影响lvs nat 下rs的功能。也就是lvs那里正常访问到mcw07.。如果加上ens33的路由，比如只加上10.0.0.0/24 ，网关是0.0.0.0时就无法让lvs nat正常响应数据了。

复制代码 
[root@mcw07 ~]# 
[root@mcw07 ~]# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         172.168.1.19    0.0.0.0         UG    100    0        0 ens34
172.168.1.0     0.0.0.0         255.255.255.0   U     100    0        0 ens34
223.0.0.0       10.0.0.254      255.0.0.0       UG    0      0        0 ens33
[root@mcw07 ~]#

客户端正常响应数据，返回nat模型的数据，也就是mcw07的数据。不知道是否在某种情况下，可以添加ens33网卡原本有的路由条目，不过即使没加，但是10.0.0.0/24网段的IP也是通的，但是其它网段想通，需要指定走ens33，也就是我们这里这个案例的这种配置，不然可能是走的内网网卡ens34,这个本来就是默认不通外网的。也就是你想要通的网段，如果现在的默认网关ens34不通，但是用ens33通的话，需要添加该网段路由，指向ens33，这样它就知道怎么走，可以通网了，不然就是走的默认的ens34，这个不通的了

复制代码 
[root@mcw04 ~]# curl 10.0.0.200:80
rs1 mcw08 ^_^ 10.0.0.18
[root@mcw04 ~]# curl 10.0.0.200:80
rs2 mcw07 ^_^ 10.0.0.17
[root@mcw04 ~]#

下面是lvs规则

复制代码 
[root@mcw09 ~]# ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  10.0.0.200:80 rr
  -> 172.168.1.17:80              Masq    1      0          0         
  -> 10.0.0.18:80                 Route   1      0          0         
[root@mcw09 ~]#
热门推荐
01【HarmonyOS】HUAWEI DevEco Studio 下载地址汇总02(欧拉)openEuler系统添加网卡文件配置流程、(欧拉)openEuler系统手动配置ipv6地址流程、(欧拉)openEuler系统网络管理说明03组基轨迹建模 GBTM的介绍与实现（Stata 或 R）04【AIGC】重塑未来的科技巨轮05全面解析：构建基于深度学习的安全帽检测系统（UI界面+YOLO代码+数据集）06【经验分享】Ubuntu22.04安装微信(linux官方版)07基于YOLOv10深度学习的CT扫描图像肾结石智能检测系统【python源码+Pyqt5界面+数据集+训练代码】深度学习实战、目标检测08Ubuntu 20.04使用Livox mid 360 测试 FAST_LIO09RAG 实践- Ollama+RagFlow 部署本地知识库10【TC3xx芯片】TC3xx芯片电源管理系统PMS详解