Prometheus监控——云原生与非云原生监控

一.云原生应用监控

[1.1云原生组件 etcd 集群监控](#1.1云原生组件 etcd 集群监控)

1.1.1查看etcd证书

[1.1.2创建etcd service](#1.1.2创建etcd service)

1.1.3创建etcd证书

1.1.4创建ServiceMonitor

1.1.5查看监控数据

1.1.6grafana监控界面展示

[1.2云原生 kube-proxy 监控](#1.2云原生 kube-proxy 监控)

1.2.1查看端口

1.2.2修改监听地址

[1.2.3创建kube-proxy Service](#1.2.3创建kube-proxy Service)

[1.2.4创建 kube-proxy servicemonitor 资源](#1.2.4创建 kube-proxy servicemonitor 资源)

1.2.5查看监控数据

1.2.6grafana监控界面展示

[1.3云原生 calico 网络插件监控](#1.3云原生 calico 网络插件监控)

[1.3.1开启 Metrics 端口](#1.3.1开启 Metrics 端口)

1.3.2创建podMonitor对象

1.3.3查看监控数据

1.3.4grafana监控界面展示

1.4云原生nginx-ingress监控

[1.4.1开启 Metrics 端口](#1.4.1开启 Metrics 端口)

[1.4.2修改service monitor 资源](#1.4.2修改service monitor 资源)

1.4.3查看监控数据

1.4.4grafana监控界面展示

二.非云原生应用监控

[2.1 Linux 主机监控](#2.1 Linux 主机监控)

2.1.1检查暴露服务

2.1.2查看监控数据

2.1.3grafana监控界面展示

[2.2非云原生应用 Redis 集群监控](#2.2非云原生应用 Redis 集群监控)

[2.3非云原生应用 NGINX 监控](#2.3非云原生应用 NGINX 监控)

2.3.1编辑configmap文件

2.2.2上传nginx-exporter镜像

2.3.3部署Nginx资源

2.3.4创建service

2.3.5创建servicemonitor

2.3.6查看监控数据

2.3.7grafana监控界面展示

一.云原生应用监控

1.1云原生组件 etcd 集群监控

1.1.1查看etcd证书

说明：在控制面板查看证书

bash 复制代码

grep -E "key-file|cert-file"   /etc/kubernetes/manifests/etcd.yaml

查看端口：

bash 复制代码

ss -tunlp | grep etcd

测试etcd能否通过metric接口暴露出去

bash 复制代码

curl -s \
  --cert /etc/kubernetes/pki/etcd/server.crt \
  --key /etc/kubernetes/pki/etcd/server.key \
  https://192.168.7.163:2379/metrics -k | tail -1

1.1.2创建etcd service

1.编写yaml文件

bash 复制代码

vim etcd-service.yml

添加以下内容：

bash 复制代码

---
apiVersion: v1
kind: Endpoints
metadata:
  labels:
    app: etcd-prom
  name: etcd-prom
  namespace: kube-system
subsets:
  - addresses:
    - ip: 192.168.7.163
    ports:
    - name: https-metrics
      port: 2379
      protocol: TCP
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: etcd-prom
  name: etcd-prom
  namespace: kube-system
spec:
  ports:
  - name: https-metrics
    port: 2379
    protocol: TCP
    targetPort: 2379
  type: ClusterIP

2.运行yaml文件

bash 复制代码

kubectl apply -f etcd-service.yml
kubectl get svc etcd-prom -o wide -n kube-system

3.测试通过svc抓取监控

bash 复制代码

curl -s --cert /etc/kubernetes/pki/etcd/server.crt  --key /etc/kubernetes/pki/etcd/server.key https://10.101.115.69:2379/metrics -k | tail -1

1.1.3创建etcd证书

说明：确保k8s集群安全

bash 复制代码

kubectl -n monitoring create secret generic etcd-ssl  \
--from-file=/etc/kubernetes/pki/etcd/ca.crt  \
--from-file=/etc/kubernetes/pki/etcd/server.crt  \
--from-file=/etc/kubernetes/pki/etcd/server.key

查看证书：

bash 复制代码

kubectl get secret etcd-ssl -n monitoring

绑定证书

bash 复制代码

vim prometheus-prometheus.yaml

添加以下内容：

bash 复制代码

secrets:
  - etcd-ssl

运行yaml文件：

bash 复制代码

kubectl apply -f prometheus-prometheus.yaml
kubectl exec prometheus-k8s-0 -c prometheus -n monitoring -- ls /etc/prometheus/secrets/etcd-ssl

1.1.4创建ServiceMonitor

说明：ServiceMonitor 是 Prometheus Operator 定义的 CRD，用来以"声明式"的方式告诉 Prometheus 该抓取哪些目标（target），取代了手动改 prometheus.yml 的 scrape_configs

1.编写yaml文件

bash 复制代码

vim etcd-servicemonitor.yaml

添加以下内容：

bash 复制代码

---
  apiVersion: monitoring.coreos.com/v1
  kind: ServiceMonitor
  metadata:
    name: etcd
    namespace: monitoring
    labels:
      app: etcd                        #标签 app=etcd
  spec:
    jobLabel: k8s-etcd-cluster-apps    #用 Service 上的哪个 label 作为 job 名
    endpoints:
    - interval: 30s
      port: https-metrics
      scheme: https
      tlsConfig:
        caFile: /etc/prometheus/secrets/etcd-ssl/ca.crt
        certFile: /etc/prometheus/secrets/etcd-ssl/server.crt
        keyFile: /etc/prometheus/secrets/etcd-ssl/server.key
        insecureSkipVerify: true
    selector:
      matchLabels:
         app: etcd-prom    #匹配标签
    namespaceSelector:
      matchNames:
      - kube-system

运行yaml文件

bash 复制代码

kubectl apply -f etcd-servicemonitor.yaml
kubectl get servicemonitor etcd -n monitoring

1.1.5查看监控数据

1.1.6grafana监控界面展示

grafana导入etcd监控界面展示地址URL：

bash 复制代码

https://grafana.com/grafana/dashboards/9733-etcd-for-k8s-cn/

1.2云原生 kube-proxy 监控

说明：kube-proxy默认暴露两个端口，一个是10249，在/metrics端点暴露监控指标，另一个是10256，默认监听127.0.0.1

1.2.1查看端口

bash 复制代码

ss -tunlp | grep kube-proxy

测试能否访问资源

bash 复制代码

curl 127.0.0.1:10249/metrics

1.2.2修改监听地址

说明：修改监听端口，监听所有端口。端口为本地，Prometheus如果没有部署在这台机子上，就无法监听到该端口。

bash 复制代码

kubectl edit configmap kube-proxy -n kube-system

添加以下内容：

bash 复制代码

metricsBindAddress: "0.0.0.0:10249"

重启Pod

说明：kube-proxy采用副本的方式构建，一旦删除，会立即重建

bash 复制代码

kubectl -n kube-system get pod|grep kube-proxy|awk '{system("kubectl -n kube-system delete pod "$1"")}'
ss -tunlp | grep kube-proxy

1.2.3创建kube-proxy Service

bash 复制代码

vim kube-proxy-service.yaml

添加以下内容：

bash 复制代码

---
apiVersion: v1
kind: Service
metadata:
  labels:
    k8s-app: kube-proxy
  name: kube-proxy
  namespace: kube-system
spec:
  selector:
    k8s-app: kube-proxy    #筛选标签
  type: ClusterIP
  ports:
  - name: https-metrics
    port: 10249
    targetPort: 10249
    protocol: TCP

运行yaml

bash 复制代码

kubectl apply -f kube-proxy-service.yaml
kubectl get svc kube-proxy -n kube-system

尝试抓取服务：

bash 复制代码

curl 10.110.8.62:10249/metrics

1.2.4创建 kube-proxy servicemonitor 资源

1.编写yaml文件

bash 复制代码

vim kube-proxy-servicemonitor.yaml

添加以下内容：

bash 复制代码

---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: kube-proxy
  namespace: monitoring
  labels:
    app: kube-proxy    #Service Monitor标签
spec:
  jobLabel: kube-proxy
  endpoints:
  - interval: 30s
    port: https-metrics
    scheme: http
  selector:
    matchLabels:
      k8s-app: kube-proxy    #匹配标签
  namespaceSelector:
    matchNames:
    - kube-system

2.运行yaml文件

bash 复制代码

kubectl apply -f kube-proxy-servicemonitor.yaml
kubectl get servicemonitor kube-proxy -n monitoring

1.2.5查看监控数据

1.2.6grafana监控界面展示

说明：prometheus本身自带granfan界面

1.3云原生 calico 网络插件监控

说明：Calico 中最核心的组件就是Felix，它负责设置路由表和 ACL规则等，以便为该主机上的 endpoints 资源正常运行提供所需的网络连接 。同时它还负责提供有关网络健康状况的数据（例如，报告配置其主机时发生的错误和问题），这些数据会被写入etcd，以使其对网络中的其他组件和操作人员可见。由此可见，监控calico的核心便是监控felix，felix相当于calico的大脑。

1.3.1开启 Metrics 端口

说明：calico在启动时，默认没有打开Metrics监听

bash 复制代码

kubectl edit ds calico-node -n kube-system

添加以下内容：

bash 复制代码

    spec:
      containers:
      - env:
        - name: FELIX_PROMETHEUSMETRICSENABLED
          value: "True"
        - name: FELIX_PROMETHEUSMETRICSPORT
          value: "9091"

      ports:
        - containerPort: 9091
          name: http-metrics
          protocol: TCP

1.3.2创建podMonitor对象

**说明：**ServiceMonitor 通过 Service 发现目标，PodMonitor 直接选 Pod。前者适合有 Service 的服务，后者适合没有 Service 或不想建 Service 的场景。

1.创建yml文件

bash 复制代码

vim prometheus-podMonitorCalico.yaml

添加以下内容：

bash 复制代码

---
apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
  labels:
    k8s-app: calico-node
  name: calico-node
  namespace: monitoring
spec:
  podMetricsEndpoints:
  - interval: 15s
    path: /metrics
    port: http-metrics
  namespaceSelector:
    matchNames:
    - kube-system
  selector:
    matchLabels:
      k8s-app: calico-node    #匹配标签

2.运行yml

bash 复制代码

kubectl apply -f prometheus-podMonitorCalico.yaml
kubectl get podmonitor calico-node -n monitoring

1.3.3查看监控数据

1.3.4grafana监控界面展示

bash 复制代码

https://grafana.com/grafana/dashboards/12175-felix-dashboard-calico/

1.4云原生nginx-ingress监控

1.4.1开启 Metrics 端口

说明： helm部署时默认没有开启metrics采集服务

bash 复制代码

vim ingress-nginx.yml

添加以下内容：

bash 复制代码

global:
  image:
    registry: reg.westos.org
controller:
  image:
    image: ingress-nginx/controller
    tag: "v1.13.3"
    digest: ""
    digestChroot: ""
  ingressClassResource:
    name: nginx
    default: true
  service:
    type: LoadBalancer
  admissionWebhooks:
    patch:
      image:
        registry: reg.westos.org
        image: ingress-nginx/kube-webhook-certgen
        tag: v1.6.3
        digest: ""
  metrics:            #开启metrics采集
    enabled: true
    service:
      enabled: true
      labels:
        app: "ingress-nginx-metrics"
    serviceMonitor:
      enabled: true
      namespace: "monitoring"
defaultBackend:
  enabled: true
  name: defaultbackend
  image:
    registry: reg.westos.org
    image: ingress-nginx/defaultbackend-amd64
    tag: "1.5"

2.使用helm部署Ingress-nginx

bash 复制代码

helm -n ingress-nginx upgrade ingress-nginx -f ingress-nginx.yml ingress-nginx-4.13.3.tgz
kubectl -n ingress-nginx get svc ingress-nginx-controller-metrics

测试

bash 复制代码

curl 10.107.130.97:10254/metrics

1.4.2修改service monitor 资源

说明：helm部署service monitor，修改匹配标签

bash 复制代码

kubectl -n monitoring edit servicemonitors ingress-nginx-controller

添加以下内容：

bash 复制代码

spec:
  endpoints:
  - interval: 30s
    port: metrics
    scheme: http
  namespaceSelector:
    matchNames:
    - ingress-nginx
  selector:
    matchLabels:
      app: ingress-nginx-metrics

在prometheus监控部署之后，安装的服务会出现权限的问题，需要配置rbac授权(补充)：

bash 复制代码

kubectl apply -f rbac.yml

文件内容如下：

bash 复制代码

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: prometheus-endpoints-view
  namespace: ingress-nginx
rules:
- apiGroups: [""]
  resources: ["endpoints", "pods", "services"]
  verbs: ["get", "list", "watch"]
- apiGroups: ["discovery.k8s.io"]
  resources: ["endpointslices"]
  verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: prometheus-k8s-endpoints
  namespace: ingress-nginx
subjects:
- kind: ServiceAccount
  name: prometheus-k8s
  namespace: monitoring
roleRef:
  kind: Role
  name: prometheus-endpoints-view
  apiGroup: rbac.authorization.k8s.io

1.4.3查看监控数据

1.4.4grafana监控界面展示

bash 复制代码

https://grafana.com/grafana/dashboards/21336-nginx-ingress-controller/

二.非云原生应用监控

说明：非云原生应用不能使用metric暴露服务端口，需要使用外部的 exporter暴露

2.1 Linux 主机监控

说明：通过kube-prometheus技术栈安装的prometheus默认会以ds的形式在每一个节点上启动一个node-exporter pod，并且会监控一个9100端口。该端口用来采集宿主机的相关监控指标。

2.1.1检查暴露服务

bash 复制代码

ss -tunlp | grep 9100
kubectl  get svc node-exporter -n monitoring  -o yaml
kubectl get servicemonitor node-exporter -n monitoring  -o yaml

2.1.2查看监控数据

2.1.3grafana监控界面展示

默认面板：

中文面板

bash 复制代码

2.2非云原生应用 Redis 集群监控

2.2.1部署redis集群

1.编辑配置文件

bash 复制代码

vim redis-values-metrics.yml

添加以下内容：

bash 复制代码

global:
  imageRegistry: "reg.westos.org"
  security:
    allowInsecureImages: true
  redis:
    password: "westos"

metrics:
  enabled: true
  serviceMonitor:
    enabled: true
    namespace: "monitoring"

2.上传镜像

bash 复制代码

docker pull bitnami/redis-exporter
docker tag bitnami/redis-exporter reg.westos.org/bitnami/redis-exporter

3.部署redis集群

bash 复制代码

helm install redis-cluster -f redis-values-metrics.yml redis-23.2.6.tgz -n redis
kubectl get all -n redis

测试是否暴露服务：

bash 复制代码

curl 10.103.174.171:9121/metrics

2.2.2配置rbac权限

bash 复制代码

vim rbac.yml

添加以下内容：

bash 复制代码

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: prometheus-endpoints-view
  namespace: redis
rules:
- apiGroups: [""]
  resources: ["endpoints", "pods", "services"]
  verbs: ["get", "list", "watch"]
- apiGroups: ["discovery.k8s.io"]
  resources: ["endpointslices"]
  verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: prometheus-k8s-endpoints
  namespace: redis
subjects:
- kind: ServiceAccount
  name: prometheus-k8s
  namespace: monitoring
roleRef:
  kind: Role
  name: prometheus-endpoints-view
  apiGroup: rbac.authorization.k8s.io

运行yml

bash 复制代码

kubectl apply -f rbac.yml

2.2.3查看监控数据

2.2.4grafana监控界面展示

bash 复制代码

2.3非云原生应用 NGINX 监控

说明：nginx有自己的监控模块stub_status。

2.3.1编辑configmap文件

bash 复制代码

vim default.conf

添加以下内容：

bash 复制代码

server {
  listen       80;
  server_name  localhost;
  location / {
    root   /usr/share/nginx/html;
    index  index.html index.htm;
  }
  location /status {
    stub_status on;
  }
  error_page   500 502 503 504  /50x.html;
  location = /50x.html {
    root   /usr/share/nginx/html;
  }
}

运行yml文件

bash 复制代码

kubectl create cm nginx-conf --from-file=./default.conf

2.2.2上传nginx-exporter镜像

bash 复制代码

docker pull nginx/nginx-exporter
docker tsg nginx/nginx-exporter reg.westos.org/library/nginx-exporter
docker push reg.westos.org/library/nginx-exporter

2.3.3部署Nginx资源

bash 复制代码

vim deploy-nginx.yml

添加以下内容：

bash 复制代码

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-server
spec:
  selector:
    matchLabels:
      app: nginx
  replicas: 3
  template:
    metadata:
      labels:
        app: nginx
      annotations:
        prometheus.io/scrape: 'true'
        prometheus.io/port: '9113'
    spec:
      containers:
        - name: nginx
          image: reg.westos.org/nginx/nginx:1.29
          ports:
            - containerPort: 80
          volumeMounts:
            - name: nginxconf
              mountPath: /etc/nginx/conf.d
        - name: nginx-exporter
          image: reg.westos.org/nginx/nginx-prometheus-exporter:1.5.1
          args:
            - '-nginx.scrape-uri=http://localhost/status'
          resources:
            limits:
              memory: 128Mi
              cpu: 500m
          ports:
            - containerPort: 9113
      volumes:
      - name: nginxconf
        configMap:
          name: nginx-conf

bash 复制代码

kubectl apply -f deploy-nginx.yml
kubectl get pod

2.3.4创建service

bash 复制代码

vim service-nginx.yml

添加以下内容：

bash 复制代码

---
apiVersion: v1
kind: Service
metadata:
  name: webapp
  labels:
     app: nginx
spec:
  selector:
    app: nginx
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80
      name: web
    - protocol: TCP
      port: 9113
      targetPort: 9113
      name: metrics

运行yml文件

bash 复制代码

kubectl apply -f service-nginx.yml
kubectl get svc webapp

查看抓取数据

bash 复制代码

curl 10.108.234.20/status
curl 10.108.234.20:9113

2.3.5创建servicemonitor

bash 复制代码

vim nginx-servicemonitor.yml

添加以下内容：

bash 复制代码

---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: nginx-exporter
  namespace: monitoring
  labels:
     k8sapp: nginx-exporter
     namespace: monitoring
spec:
 jobLabel: k8s-app
 endpoints:
 - port: metrics
   interval: 30s
   scheme: http
   path: /metrics
 selector:
   matchLabels:
     app: nginx
 namespaceSelector:
   matchNames:
   - default

运行yml文件

bash 复制代码

kubectl apply -f nginx-servicemonitor.yml

2.3.6查看监控数据

2.3.7grafana监控界面展示

bash 复制代码

https://github.com/nginx/nginx-prometheus-exporter/blob/main/grafana/dashboard.json