openvpn 和 keepalived 能一起用 ？

最近在尝试 openvpn 和 keepalived 能一起用，尝试了一下，发现 openvpn 启动的时候即使监听在全 0.0.0.0， 但是回包的时候依然会用默认路由的网卡的主 ip 回包，即使我是基于 vip 访问的。 所以这个现象我感觉还是 特别的离谱。 虽然 openvpn 可以指定 监听 vip，但是 keepalived 维护的 vip 如果不在当前节点，服务都起不来。

小结： 总感觉这个现象应该是个 bug。这个bug 导致，openvpn 即使要用 vip, 也只能等 keepalived 漂移过来的时候才能用。

也就是说：在部分等待时间和服务启动的时间，都是在冷备。

以下是实际测试步骤，基于 kube-ovn pod 测试的。 vip 绑定到 fip。

ssl vpn gw 测试

之前是用 pod fip 直接测的， 现在用的是 vip fip, （用fip 直接复测下？） 确实如此， vip 的 fip 根本就无法建立连接

➜  ~ nc -uvz 192.168.7.4 1194
Connection to 192.168.7.4 port 1194 [udp/openvpn] succeeded!  可以连上
➜  ~
➜  ~ nc -uvz 192.168.7.3 1194
Connection to 192.168.7.3 port 1194 [udp/openvpn] succeeded! fip-vip 连不上

➜  ~ nc -uvz 192.168.7.5 1194
Connection to 192.168.7.5 port 1194 [udp/openvpn] succeeded!




2023-12-06 12:08:09 192.168.7.100:60141 peer info: IV_GUI_VER="net.tunnelblick.tunnelblick_5860_4.0.0beta06__build_5860)"
2023-12-06 12:08:09 192.168.7.100:60141 WARNING: 'link-mtu' is used inconsistently, local='link-mtu 1300', remote='link-mtu 1541'
2023-12-06 12:08:09 192.168.7.100:60141 WARNING: 'tun-mtu' is used inconsistently, local='tun-mtu 1179', remote='tun-mtu 1500'
2023-12-06 12:08:09 192.168.7.100:60141 WARNING: 'auth' is used inconsistently, local='auth [null-digest]', remote='auth SHA1'
2023-12-06 12:08:09 192.168.7.100:60141 WARNING: 'keysize' is used inconsistently, local='keysize 256', remote='keysize 128'

处理掉这些告警
--tun-mtu or --link-mtu may be defined (note that --ifconfig implies --link-mtu 1500) 两者只能存其一

可能是掩码的问题，因为配置中写了掩码, （实际上根本不是， keepalived 最佳实践掩码就是 /32）

root@keepalived01-0:/# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: tun0: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1279 qdisc fq_codel state UNKNOWN group default qlen 500
    link/none
    inet 10.240.0.1 peer 10.240.0.2/32 scope global tun0
       valid_lft forever preferred_lft forever
293: eth0@if294: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue state UP group default
    link/ether 00:00:00:c5:58:24 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 10.1.0.35/24 brd 10.1.0.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet 10.1.0.2/32 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::200:ff:fec5:5824/64 scope link
       valid_lft forever preferred_lft forever

这种 keepalived （跑了 vpn gw）维护的 ip ping不通

1. 手动配在其他 pod 里面试试

root@empty:~# k exec -it -n ns1            vpc1-nginx -c netshoot -- bash
vpc1-nginx:~#
vpc1-nginx:~#
vpc1-nginx:~#
vpc1-nginx:~#
vpc1-nginx:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
202: eth0@if203: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue state UP group default
    link/ether 00:00:00:69:a8:1a brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 10.1.0.6/24 brd 10.1.0.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::200:ff:fe69:a81a/64 scope link
       valid_lft forever preferred_lft forever
vpc1-nginx:~# ip addr add 10.1.0.2/32 dev eth0
vpc1-nginx:~#
vpc1-nginx:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
202: eth0@if203: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue state UP group default
    link/ether 00:00:00:69:a8:1a brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 10.1.0.6/24 brd 10.1.0.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet 10.1.0.2/32 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::200:ff:fe69:a81a/64 scope link
       valid_lft forever preferred_lft forever

root@empty:~/kubecombo/docs/docs/guide/ssl-vpn/03-ssl-vpn-gw# ping 192.168.7.3
PING 192.168.7.3 (192.168.7.3) 56(84) bytes of data.

vpc1-nginx:~#  tcpdump -i eth0 icmp -nn
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), snapshot length 262144 bytes



^C
0 packets captured
0 packets received by filter
0 packets dropped by kernel
vpc1-nginx:~#  tcpdump -i eth0 arp -nn
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), snapshot length 262144 bytes

实际测试 32 位掩码 确实不通

换个 vip 试试

root@empty:~/kubecombo/docs/docs/guide/ssl-vpn/03-ssl-vpn-gw# k get ofip
NAME             VPC    V4EIP           V4IP        READY   IPTYPE   IPNAME
keepalived-fip   vpc1   192.168.7.3     10.1.0.2    true    vip      keepalived-vip
keepalived01-0   vpc1   192.168.7.4     10.1.0.35   true             keepalived01-0.ns1
keepalived01-1   vpc1   192.168.7.5     10.1.0.36   true             keepalived01-1.ns1
nginx            vpc1   192.168.7.110   10.1.0.6    true             vpc1-nginx.ns1
test-vip         vpc1   192.168.7.9     10.1.0.29   true    vip      test-vip
root@empty:~/kubecombo/docs/docs/guide/ssl-vpn/03-ssl-vpn-gw# k get vip
NAME             V4IP        V6IP   MAC                 PMAC   SUBNET         READY   TYPE
keepalived-vip   10.1.0.2           00:00:00:9C:54:D7          vpc1-subnet1   true
test-vip         10.1.0.29          00:00:00:05:90:F7          vpc1-subnet1   true


root@empty:~/kubecombo/docs/docs/guide/ssl-vpn/03-ssl-vpn-gw# ping 192.168.7.9
PING 192.168.7.9 (192.168.7.9) 56(84) bytes of data.
64 bytes from 192.168.7.9: icmp_seq=1 ttl=63 time=2.43 ms
64 bytes from 192.168.7.9: icmp_seq=2 ttl=63 time=0.823 ms
^C
--- 192.168.7.9 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1002ms
rtt min/avg/max/mdev = 0.823/1.627/2.431/0.804 ms
root@empty:~/kubecombo/docs/docs/guide/ssl-vpn/03-ssl-vpn-gw#


vpc1-nginx:~# ip addr add 10.1.0.29/32 dev eth0
vpc1-nginx:~#
vpc1-nginx:~#
vpc1-nginx:~#  tcpdump -i eth0 icmp -nn
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
08:43:46.665606 IP 192.168.7.200 > 10.1.0.29: ICMP echo request, id 56035, seq 1, length 64
08:43:46.665653 IP 10.1.0.29 > 192.168.7.200: ICMP echo reply, id 56035, seq 1, length 64
08:43:47.666542 IP 192.168.7.200 > 10.1.0.29: ICMP echo request, id 56035, seq 2, length 64
08:43:47.666592 IP 10.1.0.29 > 192.168.7.200: ICMP echo reply, id 56035, seq 2, length 64
^C
4 packets captured
4 packets received by filter
0 packets dropped by kernel
vpc1-nginx:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
202: eth0@if203: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue state UP group default
    link/ether 00:00:00:69:a8:1a brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 10.1.0.6/24 brd 10.1.0.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet 10.1.0.2/32 scope global eth0
       valid_lft forever preferred_lft forever
    inet 10.1.0.29/32 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::200:ff:fe69:a81a/64 scope link
       valid_lft forever preferred_lft forever

这两个 ip 没有任何区别， 说明 inet 10.1.0.2/32 这个 vip 经过使用后，确实出现了一些"隐晦"的问题

目前测试的pod fip 和 vip 都不要固定，感觉固定之后的一段时间里， pod 的 fip 和 vip 的 fip 都会出现一定的网络问题

目前问题

root@keepalived01-0:/# tcpdump -i eth0 host 192.168.7.200 -nn
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
09:14:22.884804 IP 192.168.7.200.31801 > 10.1.0.40.1194: UDP, length 14
09:14:22.885071 IP 10.1.0.41.1194 > 192.168.7.200.31801: UDP, length 26

从 vip 40 进来，但是从 41 回包


root@keepalived01-0:/# ip route list
default via 10.1.0.1 dev eth0
10.1.0.0/24 dev eth0 proto kernel scope link src 10.1.0.41 # 跟这个路由有关系
10.240.0.0/16 via 10.240.0.2 dev tun0
10.240.0.2 dev tun0 proto kernel scope link src 10.240.0.1




ip route replace 10.1.0.0/24 dev eth0 src 10.1.0.40



root@keepalived01-0:/# ip route replace 10.1.0.0/24 dev eth0 src 10.1.0.40
root@keepalived01-0:/#
root@keepalived01-0:/#
root@keepalived01-0:/# ip route list
default via 10.1.0.1 dev eth0
10.1.0.0/24 dev eth0 scope link src 10.1.0.40
10.240.0.0/16 via 10.240.0.2 dev tun0
10.240.0.2 dev tun0 proto kernel scope link src 10.240.0.1



但是没有效果

root@keepalived01-0:/#
root@keepalived01-0:/# tcpdump -i eth0 host 192.168.7.200 -nn
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
09:23:52.715172 IP 192.168.7.200.14333 > 10.1.0.40.1194: UDP, length 14
09:23:52.715400 IP 10.1.0.41.1194 > 192.168.7.200.14333: UDP, length 26


感觉是 openvpn 的业务使用了 优先级最高的那个 ip


把主 ip 删掉，看是否会变

root@keepalived01-0:/# ip addr del 10.1.0.41/24 dev eth0
root@keepalived01-0:/#
root@keepalived01-0:/# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
10.240.0.0      10.240.0.2      255.255.0.0     UG    0      0        0 tun0
10.240.0.2      0.0.0.0         255.255.255.255 UH    0      0        0 tun0

但是这样的话，因为vip 的掩码是 32 位的，所以默认路由没了



所以如果采取把 vip 当做主ip的方式，起码  掩码应该和主ip保持一致

换了个掩码测试

root@empty:~# k exec -it -n ns1            keepalived01-0 -- bash
Defaulted container "ssl" out of: ssl, keepalived
root@keepalived01-0:/#
root@keepalived01-0:/#
root@keepalived01-0:/# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: tun0: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1279 qdisc fq_codel state UNKNOWN group default qlen 500
    link/none
    inet 10.240.0.1 peer 10.240.0.2/32 scope global tun0
       valid_lft forever preferred_lft forever
309: eth0@if310: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue state UP group default
    link/ether 00:00:00:07:47:55 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 10.1.0.44/24 brd 10.1.0.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet 10.1.0.43/24 scope global secondary eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::200:ff:fe07:4755/64 scope link
       valid_lft forever preferred_lft forever
root@keepalived01-0:/#
root@keepalived01-0:/#
root@keepalived01-0:/# ip route list
default via 10.1.0.1 dev eth0
10.1.0.0/24 dev eth0 proto kernel scope link src 10.1.0.44
10.240.0.0/16 via 10.240.0.2 dev tun0
10.240.0.2 dev tun0 proto kernel scope link src 10.240.0.1
root@keepalived01-0:/# ip addr del 10.1.0.44/2 dev eth0
RTNETLINK answers: Cannot assign requested address
root@keepalived01-0:/# ip addr del 10.1.0.44/24 dev eth0
root@keepalived01-0:/#
root@keepalived01-0:/# ip route list
10.240.0.0/16 via 10.240.0.2 dev tun0
10.240.0.2 dev tun0 proto kernel scope link src 10.240.0.1
root@keepalived01-0:/# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
10.240.0.0      10.240.0.2      255.255.0.0     UG    0      0        0 tun0
10.240.0.2      0.0.0.0         255.255.255.255 UH    0      0        0 tun0

即使这样，掩码一致，但是路由还是丢了，路由丢了， keepalived 就把 vip切走了。

这里是一个 keepalived 和 openvpn 结合使用的示例

github.com/mrbobbytabl...

# 主
#!/bin/bash
docker run -d --net=host --cap-add NET_ADMIN \
-e ENVIRONMENT=production \
-e PARENT_HOST=$(hostname) \ # 这个就是vip， 这个不是openvpn的配置
-e OVPN_LOCAL=172.16.1.20 \ # 这个在文档中时可选的
-e OVPN_PUSH_1="route 10.10.0.0 255.255.255.0" \
-e OVPN_PUSH_2="dhcp-option DNS 10.10.0.111" \
-e OVPN_PUSH_3="dhcp-option DNS 10.10.0.112" \
-e OVPN_NET_1="eth0:10.10.0.0/24" \ # 这个虽然特殊，但是我找不到具体的配置
-e KEEPALIVED_STATE=MASTER \
-e KEEPALIVED_INTERFACE=eth0 \
-e KEEPALIVED_VIRTUAL_ROUTER_ID=2 \
-e KEEPALIVED_VRRP_UNICAST_BIND=10.10.0.21 \
-e KEEPALIVED_VRRP_UNICAST_PEER=10.10.0.22 \
-e KEEPALIVED_TRACK_INTERFACE_1=eth0 \
-e KEEPALVED_TRACK_INTERFACE_2=eth1 \
-e KEEPALIVED_VIRTUAL_IPADDRESS_1="10.10.0.3 dev eth0" \
-e KEEPALIVED_VIRTUAL_IPADDRESS_EXCLUDED_1="172.16.1.20 dev eth1" \
openvpn-ldap

# 备
#!/bin/bash
docker run -d --net=host --cap-add NET_ADMIN \
-e ENVIRONMENT=production \
-e PARENT_HOST=$(hostname) \
-e OVPN_LOCAL=172.16.1.20 \ # 这个就是vip
-e OVPN_PUSH_1="route 10.10.0.0 255.255.255.0" \
-e OVPN_PUSH_2="dhcp-option DNS 10.10.0.111" \
-e OVPN_PUSH_3="dhcp-option DNS 10.10.0.112" \
-e OVPN_NET_1="eth0:10.10.0.0/24" \
-e KEEPALIVED_STATE=BACKUP \
-e KEEPALIVED_INTERFACE=eth0 \
-e KEEPALIVED_VIRTUAL_ROUTER_ID=2 \
-e KEEPALIVED_VRRP_UNICAST_BIND=10.10.0.22 \
-e KEEPALIVED_VRRP_UNICAST_PEER=10.10.0.21 \
-e KEEPALIVED_TRACK_INTERFACE_1=eth0 \
-e KEEPALVED_TRACK_INTERFACE_2=eth1 \
-e KEEPALIVED_VIRTUAL_IPADDRESS_1="10.10.0.3 dev eth0" \
-e KEEPALIVED_VIRTUAL_IPADDRESS_EXCLUDED_1="172.16.1.20 dev eth1" \
openvpn-ldap

github.com/OpenVPN/ope...

2023-12-07 10:10:07 TCP/UDP: Socket bind failed on local address [AF_INET]10.1.0.49:1194: Cannot assign requested address (errno=99)
2023-12-07 10:10:07 Exiting due to fatal error
2023-12-07 10:10:07 net_route_v4_del: 10.240.0.0/16 via 10.240.0.2 dev [NULL] table 0 metric -1
2023-12-07 10:10:07 Closing TUN/TAP interface
2023-12-07 10:10:07 net_addr_ptp_v4_del: 10.240.0.1 dev tun0

可以看到 如果 keepalived vip 不存在 ， openvpn 就起不来。 所以 没有 vip 的 openvpn要有个逻辑一直等 vip。

最后成功基于 vip 的 配置：

root@keepalived01-0:/# cat /etc/openvpn/
cat: /etc/openvpn/: Is a directory
root@keepalived01-0:/# cat /etc/openvpn/openvpn.conf
ca /etc/openvpn/certs/pki/ca.crt
cert /etc/openvpn/certs/pki/issued/server.crt
key /etc/openvpn/certs/pki/private/server.key
dh /etc/openvpn/certs/pki/dh.pem

dev tun0
persist-key
persist-tun
duplicate-cn
link-mtu 1400
keysize 256
user nobody
group nogroup
key-direction 0
keepalive 10 120
status /openvpn-status.log
verb 3

local 10.1.0.49
server 10.240.0.0 255.255.0.0
cipher AES-256-GCM
auth SHA1
proto udp
port  1194
push "route 10.1.0.0 255.255.255.0"
push "dhcp-option DOMAIN-SEARCH ns1.svc.cluster.local"
push "dhcp-option DOMAIN-SEARCH svc.cluster.local"
push "dhcp-option DOMAIN-SEARCH cluster.local"

root@keepalived01-0:/# ss -tunlp
Netid                     State                      Recv-Q                     Send-Q                                          Local Address:Port                                           Peer Address:Port                     Process
udp                       UNCONN                     0                          0                                                   10.1.0.49:1194                                                0.0.0.0:*                         users:(("openvpn",pid=55,fd=5))
root@keepalived01-0:/#

总结：

目前确认该现象就是一个 bug， 除非我再加个 haproxy 基于 vip 再转发一下。 但是我是不愿意再加个 haproxy。 还不如去社区提个 issue，或者研究下怎么提 PR。

如果短时间内 fip eip 不变，但是内网 ip 或者 vip 一直变，pod 的 fip 首包会丢十个以上， vip 的 fip 甚至都不通。

这个只是不开启安全组情况下使用 vip fip 的情况，流表中没有维护。 全靠 arp 广播和缓存。可能 arp 确实会国企没清理导致乱掉。