我们的k8s集群是二进制部署,版本是1.20.4 同时选择的kube-prometheus版本是kube-prometheus-0.8.0
一、prometheus添加自定义监控与告警（etcd）
1、步骤及注意事项（前提，部署参考部署篇）
1.1 一般etcd集群会开启HTTPS认证，因此访问etcd需要对应的证书
1.2 使用证书创建etcd的secret
1.3 将etcd的secret挂在到prometheus
1.4创建etcd的servicemonitor对象（匹配kube-system空间下具有k8s-app=etcd标签的service）
1.5 创建service关联被监控对象
2、操作部署
2.1 创建etcd的secret
ETC的自建证书路径：/opt/etcd/ssl，cd /opt/etcd/ssl
kubectl create secret generic etcd-certs --from-file=server.pem --from-file=server-key.pem --from-file=ca.pem -n monitoring
可以用下面的命令验证下是否有内容产出，由内存说明是没有问题的
curl --cert /opt/etcd/ssl/server.pem --key /opt/etcd/ssl/server-key.pem https://192.168.7.108:2379/metrics -k |more
可以进入容器查看，查看证书挂载进去了
root@master manifests# kubectl exec -it -n monitoring prometheus-k8s-0 /bin/sh
/prometheus $ ls /etc/prometheus/secrets/etcd-certs/
ca.pem server-key.pem server.pem
2.2 添加secret到名为k8s的prometheus对象上（kubectl edit prometheus k8s -n monitoring或者修改yaml文件并更新资源）
apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
labels:
prometheus: k8s
name: k8s
namespace: monitoring
spec:
alerting:
alertmanagers:
- name: alertmanager-main
namespace: monitoring
port: web
baseImage: quay.io/prometheus/prometheus
nodeSelector:
kubernetes.io/os: linux
podMonitorNamespaceSelector: {}
podMonitorSelector: {}
replicas: 2
secrets:
- etcd-certs
resources:
requests:
memory: 400Mi
ruleSelector:
matchLabels:
prometheus: k8s
role: alert-rules
securityContext:
fsGroup: 2000
runAsNonRoot: true
runAsUser: 1000
serviceAccountName: prometheus-k8s
serviceMonitorNamespaceSelector: {}
serviceMonitorSelector: {}
version: v2.11.0或者可以直接找到配置文件更新vim prometheus-prometheus.yaml
kubectl replace -f prometheus-prometheus.yaml
3、创建servicemonitoring对象
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
labels:
k8s-app: etcd1 #这个serviceMonitor的标签
name: etcd
namespace: monitoring
spec:
endpoints:
- interval: 30s
port: etcd #port名字就是service里面的spec.ports.name
scheme: https #访问的方式
tlsConfig:
caFile: /etc/prometheus/secrets/etcd-certs/ca.pem #证书位置/etc/prometheus/secrets,这个路径是默认的挂载路径
certFile: /etc/prometheus/secrets/etcd-certs/server.pem
keyFile: /etc/prometheus/secrets/etcd-certs/server-key.pem
selector:
matchLabels:
k8s-app: etcd1
namespaceSelector:
matchNames:
- monitoring #匹配的命名空间4、创建service并自定义endpoint
---
apiVersion: v1
kind: Endpoints
metadata:
labels:
k8s-app: etcd1
name: etcd
namespace: monitoring
subsets:
- addresses:
- ip: 192.168.7.108
- ip: 192.168.7.109
- ip: 192.168.7.106
ports:
- name: etcd #name
port: 2379 #port
protocol: TCP
---
apiVersion: v1
kind: Service
metadata:
labels:
k8s-app: etcd1
name: etcd
namespace: monitoring
spec:
ports:
- name: etcd
port: 2379
protocol: TCP
targetPort: 2379
sessionAffinity: None
type: ClusterIPkubectl replace -f prometheus-prometheus.yaml
kubectl apply -f servicemonitor.yaml
kubectl apply -f service.yam
到这里就可以在prometheus中查看到etcd的监控信息了
添加告警
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
labels:
prometheus: k8s
role: alert-rules
name: etcd-rules
namespace: monitoring
spec:
groups:
- name: etcd-exporter.rules
rules:
- alert: EtcdClusterUnavailable
annotations:
summary: etcd cluster small
description: If one more etcd peer goes down the cluster will be unavailable
expr: |
count(up{job="etcd"} == 0) > (count(up{job="etcd"}) / 2-1)
for: 3m
labels:
severity: critical二、prometheus添加自定义监控与告警（kube-controller-manager）
Kube-prometheus默认是配置了kube-controller-manager的servicemonitor的，但是因为我们是二进制部署的，所以无法找到对应的kube-contorller-manager的service和endpoints,所以这里我们需要自己去手动创建service和endpoints
kubectl get servicemonitor -n monitoring
通过查看servicemonitor去查看需要匹配的service的labels
vim kubernetes-serviceMonitorKubeControllerManager.yaml
好了，下面我们就开始创建service和endpoints了
创建service
首先创建一个endpoint，指向宿主机ip+10252，然后在创建一个同名的service，和上面查出来的标签
---
apiVersion: v1
kind: Endpoints
metadata:
annotations:
app.kubernetes.io/name: kube-controller-manager
name: kube-controller-manager-monitoring
namespace: kube-system
subsets:
- addresses:
- ip: 192.168.7.100
ports:
- name: https-metrics
port: 10252
protocol: TCP
---
apiVersion: v1
kind: Service
metadata:
labels:
app.kubernetes.io/name: kube-controller-manager
name: kube-controller-manager-monitoring
namespace: kube-system
spec:
ports:
- name: https-metrics
port: 10252
protocol: TCP
targetPort: 10252
sessionAffinity: None
type: ClusterIP#注：kube-prometheus使用的是https、而暴露使用的是http，将https改成http
kubectl edit servicemonitor -n monitoring kube-controller-manager
60 scheme: http
配置完成后就可以再prometheus界面查看到监控信息了，这样就是成功了
三、prometheus添加自定义监控与告警（kube-scheduler）
Kube-scheduler的配置和kube-controller-manager的配置类似
kubectl edit servicemonitor -n monitoring kube-scheduler
#scheme: https 改为scheme: http
好了，下面我们就开始创建service和endpoints了
apiVersion: v1
kind: Service
metadata:
labels:
app.kubernetes.io/name: kube-scheduler
name: scheduler
namespace: kube-system
spec:
ports:
- name: https-metrics
port: 10251
protocol: TCP
targetPort: 10251
sessionAffinity: None
type: ClusterIP
---
apiVersion: v1
kind: Endpoints
metadata:
labels:
app.kubernetes.io/name: kube-scheduler
name: scheduler
namespace: kube-system
subsets:
- addresses:
- ip: 192.168.7.100
ports:
- name: https-metrics
port: 10251
protocol: TCPKubectl apply -f svc-kube-scheduler.yaml
配置完成后就可以再prometheus界面查看到监控信息了，这样就是成功了
至此 controller-manager、scheduler 已经起来
备注：看到不少的资料都是说需要修改kube-controller-manager和kube-scheduler的监听地址，从127.0.0.1修改成0.0.0.0
但是因为我的配置中kube-controller-manager的配置文件一开是就是0.0.0.0所以没有修改，kube-scheduler的配置文件监听地址是127.0.0.1也没有进行修改但是依然成功了，所以这里还有待验证。