k8s service

二.service的ipvs模式

1.在所有节点中安装ipvsadm工具

root@k8s-master \~# for name in 100 10 20 254

> do

> ssh -l root 172.25.254.$name "dnf install ipvsadm -y"

> done

2.配置集群中的kube-proxy

root@k8s-master \~# kubectl -n kube-system edit cm kube-proxy

59 mode: "ipvs" root@k8s-master \~# kubectl -n kube-system get pods NAME READY STATUS RESTARTS AGE

coredns-697886855d-c9lh5 1/1 Running 1 (3d19h ago) 4d

coredns-697886855d-hgfxf 1/1 Running 1 (3d19h ago) 4d

etcd-k8s-master 1/1 Running 1 (3d19h ago) 4d

kube-apiserver-k8s-master 1/1 Running 1 (3d19h ago) 4d

kube-controller-manager-k8s-master 1/1 Running 1 (3d19h ago) 4d

kube-proxy-6kp78 1/1 Running 1 (3d19h ago) 4d

kube-proxy-hvk7k 1/1 Running 0 93m

kube-proxy-p7fdf 1/1 Running 1 (3d19h ago) 4d

kube-proxy-tb29j 1/1 Running 1 (3d19h ago) 3d19h

kube-scheduler-k8s-master 1/1 Running 1 (3d19h ago) 4d

root@k8s-master \~# kubectl -n kube-system delete pods kube-proxy-6kp78 kube-proxy-hvk7k kube-proxy-p7fdf kube-proxy-tb29j

pod "kube-proxy-6kp78" deleted from kube-system namespace

pod "kube-proxy-hvk7k" deleted from kube-system namespace

pod "kube-proxy-p7fdf" deleted from kube-system namespace

pod "kube-proxy-tb29j" deleted from kube-system namespace

tips:kube‑proxy 是 DaemonSet 管理的 Pod
我修改的是 ConfigMap(kube‑proxy)，改的是集群里面的配置对象
DaemonSet 不会自动热更新 ConfigMap 的内容到正在运行的 Pod 里面
Pod 一旦启动完成，会把 ConfigMap 的内容加载进内存，ConfigMap 变了，已经运行的 Pod 不会自动重新读取新配置，所以删除旧 Pod，DaemonSet 控制器会自动重建出新的 kube‑proxy Pod

root@k8s-master \~# ipvsadm -Ln

三.构建metalLB

tips：我本地有压缩包就省略解压那一步了

1.上传镜像

root@k8s-master \~# cd metaLLB/

root@k8s-master metaLLB# docker load -i metallb-v0.16.1.tar

Loaded image: quay.io/metallb/controller:v0.16.1

Loaded image: quay.io/metallb/speaker:v0.16.1

root@k8s-master metaLLB# docker push reg.LLJ.org/metallb/controller:v0.16.1 root@k8s-master metaLLB# docker push reg.LLJ.org/metallb/speaker:v0.16.1

2.更改kube-proxy

root@k8s-master mateLLB# kubectl edit configmap -n kube-system kube-proxy

configmap/kube-proxy edited

59 mode: "ipvs"

60 ipvs:

61 strictARP: true

3.更改metallb的配置文件

root@k8s-master metaLLB# vim 1-metallb-native.yaml

2136: image: metallb/controller:v0.16.1

2233: image: metallb/speaker:v0.16.1

root@k8s-master metaLLB# vim 2-metallb-confmap.yml

apiVersion: metallb.io/v1beta1

kind: IPAddressPool #自定义资源：IP地址池，告诉MetalLB我可以分配哪些IP给LoadBalancer Service，当 Service 是type:LoadBalancer时，metallb 就从这里拿一个 IP 分配给 Service 的EXTERNAL‑IP

metadata:

name: first-pool

namespace: metallb-system #必须和metallb组件同一个namespace

spec:

addresses:

• 172.25.254.50-172.25.254.99 #可供LoadBalancer使用的IP地址段，一共50个IP，给Service分配External‑IP

--- #必须加这个

apiVersion: metallb.io/v1beta1

kind: L2Advertisement #二层广播模式配置（L2模式，最常用，不需要BGP路由器）

metadata:

name: example

namespace: metallb-system

spec:

ipAddressPools:

• first-pool #绑定上面定义的first‑pool地址池

root@k8s-master metaLLB# kubectl apply -f 1-metallb-native.yaml #先部署本体，安装CRD、controller、speaker root@k8s-master metaLLB# kubectl apply -f 2-metallb-confmap.yml #等待CRD注册完成，可以执行查看，再部署IP池配置

tip:切记不可同时kubectl apply -f 2-metallb-confmap.yml -f 1-metallb-native.yaml因为你同时 apply 两个文件：1-metallb-native.yaml（安装 metallb 本体 + CRD） 和 2-metallb-confmap.yml（IPAddressPool、L2Advertisement 配置）K8s 并行执行 apply：还没把 CRD 资源创建完成，就尝试去创建IPAddressPool / L2Advertisement自定义资源CRD 还不存在，API 识别不到这两个 kind，直接报错。虽然输出看到一堆 customresourcedefinition ... created，只是打印日志，实际 CRD 对象还没在集群内部就绪，立刻创建 CRD 实例就报这个错

4.验证

root@k8s-master metaLLB# kubectl get namespaces root@k8s-master metaLLB# kubectl -n metallb-system get pods root@k8s-master metaLLB# kubectl -n metallb-system get cm

四.ingress七层代理

1.ingress部署

(1)下载deploy文件

wget https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.15.1/deploy/static/provider/baremetal/deploy.yaml

(2)准备ingress所需镜像

root@k8s-master \~# cd ingress-v1.5.1/

root@k8s-master ingress-v1.5.1# ls

deploy.yaml ingress-v1.15.1.tar

root@k8s-master ingress-v1.5.1# docker load -i ingress-v1.15.1.tar

Loaded image: registry.k8s.io/ingress-nginx/controller:v1.15.1

Loaded image: registry.k8s.io/ingress-nginx/kube-webhook-certgen:v1.6.9

root@k8s-master ingress-v1.5.1# docker tag registry.k8s.io/ingress-nginx/co ntroller:v1.15.1 reg.LLJ.org/ingress-nginx/controller:v1.15.1 push前先docker login reg.LLJ.org，还必须在habor网页上提前创建项目 root@k8s-master ingress-v1.5.1# docker push reg.LLJ.org/ingress-nginx/controller:v1.15.1 root@k8s-master ingress-v1.5.1# docker tag registry.k8s.io/ingress-nginx/kube-webhook -certgen:v1.6.9 reg.LLJ.org/ingress_nginx/kube-webhook-certgen:v1.6.9

root@k8s-master ingress-v1.5.1# docker push reg.LLJ.org/ingress_nginx/kube-webhook-c ertgen:v1.6.9

(3)更改deploy.yaml

root@k8s-master ingress-v1.5.1# vim deploy.yaml

444: image: ingress-nginx/controller:v1.15.1

547: image: ingress-nginx/kube-webhook-certgen:v1.6.9

603: image: ingress-nginx/kube-webhook-certgen:v1.6.9

365 type: LoadBalancer

(4)通过deplay.yaml部署ingress

root@k8s-master ingress-v1.5.1# kubectl apply -f deploy.yaml

root@k8s-master ingress-v1.5.1# kubectl -n ingress-nginx get svc

2.ingress功能的基本实现

(1)建立ingress代理的微服务

#业务1

root@k8s-master metaLLB# kubectl create deployment myappv1 --image myapp:v1 --replicas 2 --dry-run=client -o yaml > myappv1.yaml

root@k8s-master metaLLB# kubectl create service clusterip servicev1 --tcp 80:80 --dry-run=client -o yaml >> myappv1.yaml

root@k8s-master metaLLB# vim myappv1.yaml

apiVersion: apps/v1

kind: Deployment

metadata:

labels:

app: myappv1

name: myappv1

spec:

replicas: 2

selector:

matchLabels:

app: myappv1

template:

metadata:

labels:

app: myappv1

spec:

containers:

• image: myapp:v1

name: myapp

---

apiVersion: v1

kind: Service

metadata:

labels:

app: servicev1

name: servicev1

spec:

ports:

• name: 80-80

port: 80

protocol: TCP

targetPort: 80

selector:

app: myappv1

type: ClusterIP

---

#业务2

root@k8s-master metaLLB# vim myappv2.yaml

apiVersion: apps/v1

kind: Deployment

metadata:

labels:

app: myappv2

name: myappv2

spec:

replicas: 2

selector:

matchLabels:

app: myappv2

template:

metadata:

labels:

app: myappv2

spec:

containers:

• image: myapp:v2

name: myapp

---

apiVersion: v1

kind: Service

metadata:

labels:

app: servicev2

name: servicev2

spec:

ports:

• name: 80-80

port: 80

protocol: TCP

targetPort: 80

selector:

app: myappv2

type: ClusterIP

root@k8s-master metaLLB# kubectl apply -f myappv2.yaml

deployment.apps/myappv2 created

service/servicev2 created

[root@k8s-master metaLLB# kubectl get svc

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE

kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 6d1h

servicev1 ClusterIP 10.97.35.20 <none> 80/TCP 4m4s

servicev2 ClusterIP 10.100.116.247 <none> 80/TCP 40s

root@k8s-master metaLLB# curl 10.97.35.20

Hello MyApp | Version: v1 | <a href="hostname.html">Pod Name</a>

root@k8s-master services# curl 10.100.116.247

Hello MyApp | Version: v2 | <a href="hostname.html">Pod Name</a>

root@k8s-master metaLLB# kubectl get pods

NAME READY STATUS RESTARTS AGE

myappv1-8676894bf8-2d4jh 1/1 Running 0 4m34s

myappv1-8676894bf8-rqrh8 1/1 Running 0 4m34s

myappv2-688fc4d776-5t7kt 1/1 Running 0 70s

myappv2-688fc4d776-d2xgd 1/1 Running 0 70s

(2)建立ingress控制器(基于路径 path路由)

tips:Ingress‑Nginx 两种情况： 情况 1：同一个域名，不同路径 /v1 /v2 → 需要 rewrite-target，同一域名，靠 URL 路径区分业务 情况 2：两套不同域名，分别指向不同服务 → 不需要 rewrite‑target 原理： 1.不加重写： 用户请求到达 ingress‑nginx（网关）
Ingress 匹配到api.test.com、路径/v1
默认直接把收到的完整 URL 原样转发给后端 Pod
发给后端的请求是：GET /v1/xxx HTTP/1.1 #GET：请求方式 # /v1/xxx：请求路径(URL 路径部分) #HTTP/1.1：协议版本

后端收到 /v1/xxx，它里面没有 /v1 这个目录，程序找不到页面 → 返回 404 Not Found 2.加了重写 变的只是网关转发给后端的 URLIngress 网关在转发之前，做路径改写：把 /v1/xxx → 改成 /xxx，再发给后端 Pod。
发给后端 Pod 的请求变成：GET /xxx HTTP/1.1
后端识别/xxx，成功返回页面

1.配置yml文件

root@k8s-master metaLLB# kubectl create ingress ingress1 --class nginx --rule "*/=servicev1:80" --dry-run=client -o yaml > 1-ingress.yml #匹配到上面的域名 + 路径之后，把请求转发给名叫 servicev1 的 Service 的 80 端口

apiVersion: networking.k8s.io/v1

kind: Ingress

metadata:

name: ingress1

annotations:
nginx.ingress.kubernetes.io/rewrite-target: / #前缀告诉集群这条注解交给 ingress‑nginx 控制器去解析，/rewrite‑target是路径重写
/：重写之后，替换成的路径开头 tips：1.nginx.ingress.kubernetes.io是 ingress‑nginx 控制器专属注解前缀，是nginx‑ingress 的私有的扩展参数，用来微调 Nginx 底层配置（重写、限流、https 跳转、跨域、认证等）只能写在 Ingress 的 yaml 里面 2.rewrite‑target 是Ingress‑Nginx 网关的行为：网关收到用户 http 请求，再修改 url 转发给后端 Service，网关干什么事，配置就写在网关资源 (Ingress) 上，不要写到业务资源

spec:

ingressClassName: nginx #集群里存在同名的 IngressClass 资源，否则这条 Ingress 等于没人处理，完全不生效，安装 ingress‑nginx 的时候，安装脚本自动帮你创建好名字叫 nginx 的 IngressClass，所以平时直接写 ingressClassName: nginx 就可以用，查看：kubectl get ingressclass nginx -o yaml

rules:

• host: 'www.llj.org'

http:

paths:

• backend:

service:

name: servicev1

port:

number: 80

path: /v1

pathType: Prefix #prefix只能匹配前缀

• backend:

service:

name: servicev2

port:

number: 80

path: /v2

pathType: Prefix

root@k8s-master metaLLB# kubectl apply -f 1-ingress.yml

root@k8s-master metaLLB# kubectl describe ingress ingress1

Name: ingress1

Labels: <none>

Namespace: default

Address:

Ingress Class: nginx

Default backend: <default>

Rules:

Host Path Backends

---

www.llj.org

/v1 servicev1:80 (10.244.2.26:80,10.244.5.109:80)

/v2 servicev2:80 (10.244.1.66:80,10.244.2.27:80)

Annotations: nginx.ingress.kubernetes.io/rewrite-target: /

Events:

Type Reason Age From Message

---

Normal Sync 9s nginx-ingress-controller Scheduled for sync

2.测试：

先查找ingress 控制器真正的 LBIP，以 ingress‑nginx-controller 这个 Service 的 EXTERNAL‑IP 为准 root@k8s-master metaLLB# kubectl -n ingress-nginx get svc

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE

ingress-nginx-controller LoadBalancer 10.107.77.23 172.25.254.51 80:30596/TCP,443:32423/TCP 22h

ingress-nginx-controller-admission LoadBalancer 10.110.28.86 172.25.254.52 443:30967/TCP 22h

在其他验证主机配置解析 vim /etc/hosts

172.25.254.51 www.llj.org

验证 root@harbor \~# curl www.llj.org/v1 匹配前面rule下的paths.path.v1

Hello MyApp | Version: v1 | <a href="hostname.html">Pod Name</a>

root@harbor \~# curl www.llj.org/v2

Hello MyApp | Version: v2 | <a href="hostname.html">Pod Name</a>

(3)建立ingress控制器(基于域名host路由)

tips：不同域名，path 统一/，不需要重写注解，不再用同一个域名下/v1、/v2路径分流，写两组host，两个不同域名，path 全部为/，依靠 HTTP 请求里的Host头部来判断转发到哪个 Service，hosts 文件把两个域名都解析到 ingress‑nginx 入口 IP，curl 访问不同域名拿到不同版本

root@k8s-master metaLLB# vim 2-ingress.yaml

apiVersion: networking.k8s.io/v1

kind: Ingress

metadata:

name: ingress1

annotations:

nginx.ingress.kubernetes.io/rewrite-target: / #不用写注释掉

spec:

ingressClassName: nginx

rules:

• host: myapp1.llj.org

http:

paths:

• backend:

service:

name: servicev1

port:

number: 80

path: / #改动成/

pathType: Prefix

• host: myapp2.llj.org #改动通过写两组host来分流

http:

paths:

• backend:

service:

name: servicev2

port:

number: 80

path: /

pathType: Prefix

root@k8s-master metaLLB# kubectl apply -f 2-ingress.yaml

ingress.networking.k8s.io/ingress1 created

root@k8s-master metaLLB# kubectl describe ingress ingress1

Name: ingress1

Labels: <none>

Namespace: default

Address: 172.25.254.20

Ingress Class: nginx

Default backend: <default>

Rules:

Host Path Backends

---

myapp1.llj.org

/ servicev1:80 (10.244.2.26:80,10.244.5.109:80)

myapp2.llj.org

/ servicev2:80 (10.244.1.66:80,10.244.2.27:80)

Annotations: <none>

Events:

Type Reason Age From Message

---

Normal Sync 8s (x2 over 14s) nginx-ingress-controller Scheduled for sync

验证：

vim /etc/hosts

172.25.254.51 www.llj.org myapp1.llj.org myapp2.llj.org

curl myapp1.llj.org

Hello MyApp | Version: v1 | <a href="hostname.html">Pod Name</a>

curl myapp2.llj.org

Hello MyApp | Version: v2 | <a href="hostname.html">Pod Name</a>