学习笔记三十五：Ingress-controller高可用

Ingress-controller高可用

Ingress-controller高可用
- 特别注意：
通过keepalived+nginx实现ingress-nginx-controller高可用
[测试Ingress HTTP代理k8s内部pod](#测试Ingress HTTP代理k8s内部pod)
[测试Ingress HTTPS代理k8s内部pod](#测试Ingress HTTPS代理k8s内部pod)
- 构建TLS站点
- 创建Ingress
同一个k8s搭建多套Ingress-controller
- 原理以及注意事项
- 创建Ingress规则：

Ingress-controller高可用

Ingress Controller是集群流量的接入层，对它做高可用非常重要，可以基于keepalive实现nginx-ingress-controller高可用，具体实现如下：

Ingress-controller根据Deployment+ nodeSeletor+pod反亲和性方式部署在k8s指定的两个work节点，nginx-ingress-controller这个pod共享宿主机ip，然后通过keepalive+nginx实现nginx-ingress-controller高可用

参考：https://github.com/kubernetes/ingress-nginx

https://github.com/kubernetes/ingress-nginx/tree/main/deploy/static/provider/baremetal

特别注意：

因为我们现在安装的k8s版本是1.25，那就需要按照文档步骤ctr -n=k8s.io images import导出镜像，如果k8s版本是1.24之前的，可以用docker load -i解压，现在安装更新到了k8s1.25，所以导出镜像需要用ctr -n=k8s.io images import

将ingress-nginx-controllerv1.1.0.tar.gz 上传至节点

链接：https://pan.baidu.com/s/1BP1Vr0IUxkMYqrP-MDs8vQ?pwd=0n68

提取码：0n68

python 复制代码

ctr -n=k8s.io images import ingress-nginx-controllerv1.1.0.tar.gz
docker load -i ingress-nginx-controllerv1.1.0.tar.gz
ctr -n=k8s.io images import kube-webhook-certgen-v1.1.0.tar.gz
docker load -i kube-webhook-certgen-v1.1.0.tar.gz

python 复制代码

##如果k8s版本是1.24之前的用docker load -i解压
#docker load -i ingress-nginx-controllerv1.1.0.tar.gz

ingress-deploy.yaml

链接：https://pan.baidu.com/s/1nNx9ZDHCu1stqtV7qo6bJA?pwd=x9pg

提取码：x9pg

python 复制代码

kubectl apply -f ingress-deploy.yaml

通过keepalived+nginx实现ingress-nginx-controller高可用

安装nginx主备：

在k8snode1和k8snode2上做nginx主备安装

python 复制代码

yum install  epel-release  nginx keepalived nginx-mod-stream  -y

修改nginx配置文件。主备一样

python 复制代码

cat /etc/nginx/nginx.conf

python 复制代码

user nginx;
worker_processes auto;
error_log /var/log/nginx/error.log;
pid /run/nginx.pid;

include /usr/share/nginx/modules/*.conf;

events {
    worker_connections 1024;
}
# 四层负载均衡，为两台Master apiserver组件提供负载均衡
stream {

    log_format  main  '$remote_addr $upstream_addr - [$time_local] $status $upstream_bytes_sent';

    access_log  /var/log/nginx/k8s-access.log  main;

    upstream k8s-ingress-controller {
       server 192.168.40.12:80 weight=5 max_fails=3 fail_timeout=30s;   # k8snode1 IP:PORT
       server 192.168.40.13:80 weight=5 max_fails=3 fail_timeout=30s;  # k8snode2 IP:PORT
    }
    
    server {
       listen 30080; 
       proxy_pass k8s-ingress-controller;
    }
}

http {
    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;

    sendfile            on;
    tcp_nopush          on;
    tcp_nodelay         on;
    keepalive_timeout   65;
    types_hash_max_size 2048;

    include             /etc/nginx/mime.types;
    default_type        application/octet-stream;

}

#备注：
#server 反向服务地址和端口
#weight 权重
#max_fails 失败多少次，认为主机已挂掉，则踢出
#fail_timeout 踢出后重新探测时间

注意：nginx监听端口变成大于30000的端口，比方说30080,这样访问域名:30080就可以了，必须是满足大于30000以上，才能代理ingress-controller

keepalive配置

主keepalived

python 复制代码

cat /etc/keepalived/keepalived.conf

python 复制代码

global_defs { 
   notification_email { 
     acassen@firewall.loc 
     failover@firewall.loc 
     sysadmin@firewall.loc 
   } 
   notification_email_from Alexandre.Cassen@firewall.loc  
   smtp_server 127.0.0.1 
   smtp_connect_timeout 30 
   router_id NGINX_MASTER
} 

vrrp_script check_nginx {
    script "/etc/keepalived/check_nginx.sh"
}

vrrp_instance VI_1 { 
    state MASTER 
    interface ens33  # 修改为实际网卡名
    virtual_router_id 51 # VRRP 路由 ID实例，每个实例是唯一的 
    priority 100    # 优先级，备服务器设置 90 
    advert_int 1    # 指定VRRP 心跳包通告间隔时间，默认1秒 
    authentication { 
        auth_type PASS      
        auth_pass 1111 
    }  
    # 虚拟IP
    virtual_ipaddress { 
        192.168.40.199/24
    } 
    track_script {
        check_nginx
    } 
}

#vrrp_script：指定检查nginx工作状态脚本（根据nginx状态判断是否故障转移）
#virtual_ipaddress：虚拟IP（VIP）

python 复制代码

cat /etc/keepalived/check_nginx.sh

python 复制代码

#!/bin/bash
#1、判断Nginx是否存活
counter=$(ps -ef |grep nginx | grep sbin | egrep -cv "grep|$$" )
if [ $counter -eq 0 ]; then
    #2、如果不存活则尝试启动Nginx
    service nginx start
    sleep 2
    #3、等待2秒后再次获取一次Nginx状态
    counter=$(ps -ef |grep nginx | grep sbin | egrep -cv "grep|$$" )
    #4、再次进行判断，如Nginx还不存活则停止Keepalived，让地址进行漂移
    if [ $counter -eq 0 ]; then
        service  keepalived stop
    fi
fi

python 复制代码

chmod +x  /etc/keepalived/check_nginx.sh

备keepalive

python 复制代码

cat /etc/keepalived/keepalived.conf

python 复制代码

global_defs { 
   notification_email { 
     acassen@firewall.loc 
     failover@firewall.loc 
     sysadmin@firewall.loc 
   } 
   notification_email_from Alexandre.Cassen@firewall.loc  
   smtp_server 127.0.0.1 
   smtp_connect_timeout 30 
   router_id NGINX_BACKUP
} 

vrrp_script check_nginx {
    script "/etc/keepalived/check_nginx.sh"
}

vrrp_instance VI_1 { 
    state BACKUP 
    interface ens33
    virtual_router_id 51 # VRRP 路由 ID实例，每个实例是唯一的 
    priority 90
    advert_int 1
    authentication { 
        auth_type PASS      
        auth_pass 1111 
    }  
    virtual_ipaddress { 
        192.168.1.199/24
    } 
    track_script {
        check_nginx
    } 
}

python 复制代码

cat /etc/keepalived/check_nginx.sh

python 复制代码

#!/bin/bash
#1、判断Nginx是否存活
counter=$(ps -ef |grep nginx | grep sbin | egrep -cv "grep|$$" )
if [ $counter -eq 0 ]; then
    #2、如果不存活则尝试启动Nginx
    service nginx start
    sleep 2
    #3、等待2秒后再次获取一次Nginx状态
    counter=$(ps -ef |grep nginx | grep sbin | egrep -cv "grep|$$" )
    #4、再次进行判断，如Nginx还不存活则停止Keepalived，让地址进行漂移
    if [ $counter -eq 0 ]; then
        service  keepalived stop
    fi
fi

python 复制代码

chmod +x /etc/keepalived/check_nginx.sh

注：keepalived根据脚本返回状态码（0为工作正常，非0不正常）判断是否故障转移。

k8snode1和k8snode2启动服务

python 复制代码

systemctl daemon-reload
systemctl enable nginx keepalived
systemctl start nginx
systemctl start keepalived

测试vip是否绑定成功

在k8snode1执行

python 复制代码

ip addr

测试keepalived：

停掉k8snode1上的keepalived。Vip会漂移到k8snode2

python 复制代码

service  keepalived stop

在k8snode1启动keepalived ，Vip又会漂移到k8snode1

python 复制代码

systemctl start keepalived

测试Ingress HTTP代理k8s内部pod

部署后端tomcat服务

tomcat:8.5.34-jre8-alpine

链接：https://pan.baidu.com/s/18AU47-mvEMeqFS9IGUQURw?pwd=abyo

提取码：abyo

python 复制代码

cat ingress-demo.yaml

python 复制代码

apiVersion: v1
kind: Service
metadata:
  name: tomcat
  namespace: default
spec:
  selector:
    app: tomcat
    release: canary
  ports:
  - name: http
    targetPort: 8080
    port: 8080
  - name: ajp
    targetPort: 8009
    port: 8009
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: tomcat-deploy
  namespace: default
spec:
  replicas: 2
  selector:
    matchLabels:
      app: tomcat
      release: canary
  template:
    metadata:
      labels:
        app: tomcat
        release: canary
    spec:
      containers:
      - name: tomcat
        image: tomcat:8.5.34-jre8-alpine 
        imagePullPolicy: IfNotPresent  
        ports:
        - name: http
          containerPort: 8080
          name: ajp
          containerPort: 8009

python 复制代码

kubectl apply -f ingress-demo.yaml

python 复制代码

kubectl get pods -l app=tomcat

编写ingress规则

ingress-myapp.yaml

链接：https://pan.baidu.com/s/1UE81sqHEELqPQhB6lx5xEQ?pwd=hxmc

提取码：hxmc

python 复制代码

kubectl apply -f ingress-myapp.yaml

编写ingress的配置清单

python 复制代码

vim ingress-myapp.yaml

python 复制代码

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress-myapp
  namespace: default
spec:
  ingressClassName: nginx
  rules:
  - host: tomcat.lucky.com
    http:
      paths:
      - backend:
          service:
            name: tomcat
            port:
              number: 8080
        path: /
        pathType: Prefix

python 复制代码

kubectl apply -f ingress-myapp.yaml

如果有报错

报错如下：

解决办法：

python 复制代码

kubectl delete -A ValidatingWebhookConfiguration ingress-nginx-admission
kubectl apply -f ingress-myapp.yaml
kubectl get pods -n ingress-nginx -o wide

修改电脑本地的host文件，增加如下一行，IP为VIP

本地hosts文件位置

python 复制代码

C:\Windows\System32\drivers\etc

宿主机浏览器访问：tomcat.lucky.com

代理流程：

tomcat.lucky.com:30080->192.168.40.199:30080->192.168.40.181:80,192.168.40.182:80->svc: tomcat:8080->tomcat-deploy-

测试Ingress HTTPS代理k8s内部pod

构建TLS站点

准备证书，在k8smaster1节点操作

python 复制代码

cd /root/
openssl genrsa -out tls.key 2048
openssl req -new -x509 -key tls.key -out tls.crt -subj /C=CN/ST=Beijing/L=Beijing/O=DevOps/CN=tomcat.lucky.com

生成secret，在k8smaster1节点操作

python 复制代码

kubectl create secret tls tomcat-ingress-secret --cert=tls.crt --key=tls.key

查看secret

python 复制代码

kubectl get secret

查看tomcat-ingress-secret详细信息

python 复制代码

kubectl describe secret tomcat-ingress-secret

创建Ingress

Ingress规则可以参考官方：

https://kubernetes.io/zh/docs/concepts/services-networking/ingress/

python 复制代码

cat ingress-tomcat-tls.yaml

python 复制代码

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress-tomcat-tls
  namespace: default
spec:
  ingressClassName: nginx
  tls:
  - hosts:
    -  tomcat.lucky.com
    secretName: tomcat-ingress-secret
  rules:
  - host: tomcat.lucky.com
    http:
      paths:
      - path: /
        pathType:  Prefix
        backend:
         service:
           name: tomcat
           port:
            number: 8080

更新yaml文件

python 复制代码

kubectl delete -f ingress-myapp.yaml
kubectl apply -f ingress-tomcat-tls.yaml

再次访问 tomcat.lucky.com

同一个k8s搭建多套Ingress-controller

原理以及注意事项

ingress可以简单理解为service的service，他通过独立的ingress对象来制定请求转发的规则，把请求路由到一个或多个service中。这样就把服务与请求规则解耦了，可以从业务维度统一考虑业务的暴露，而不用为每个service单独考虑。
在同一个k8s集群里，部署两个ingress nginx。一个deploy部署给A的API网关项目用。另一个daemonset部署给其它项目作域名访问用。这两个项目的更新频率和用法不一致，暂时不用合成一个。
为了满足多租户场景，需要在k8s集群部署多个ingress-controller，给不同用户不同环境使用。

主要参数设置：

python 复制代码

containers:
        - name: nginx-ingress-controller
          image: registry.cn-hangzhou.aliyuncs.com/google_containers/nginx-ingress-controller:v1.1.0
          args:
            - /nginx-ingress-controller
            - --ingress-class=ngx-ds

注意：--ingress-class设置该Ingress Controller可监听的目标Ingress Class标识；

注意：同一个集群中不同套Ingress Controller监听的Ingress Class标识必须唯一，且不能设置为nginx关键字（其是集群默认Ingress Controller的监听标识）；

创建Ingress规则：

python 复制代码

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress-myapp
  namespace: default
spec:
  ingressClassName: nginx
  rules:
  - host: tomcat.lucky.com
    http:
      paths:
      - path: /
        pathType:  Prefix
        backend:
         service:
           name: tomcat
           port:
            number: 8080

注意：这里要设置为你前面配置的controller.ingressClass唯一标识

python 复制代码

ingressClassName: nginx-ds