JVM (Micrometer)监控SpringBoot（AWS EKS版）

问题

怎样使用JVM (Micrometer)面板，监控Spring？这里不涉及Prometheus和Grafana，重点介绍与Micrometer与Springboot，k8s怎样集成。

pom.xml

引入依赖，如下：

<properties>
	<micrometer.version>1.12.5</micrometer.version>
	<micrometer-jvm-extras.version>0.2.2</micrometer-jvm-extras.version>
</properties>
...
<dependency>
  <groupId>io.micrometer</groupId>
  <artifactId>micrometer-registry-prometheus</artifactId>
  <version>${micrometer.version}</version>
</dependency>

<dependency>
    <groupId>io.github.mweirauch</groupId>
    <artifactId>micrometer-jvm-extras</artifactId>
    <version>${micrometer-jvm-extras.version}</version>
</dependency>

JVMConfig.java

启用micrometer-jvm-extras库监控内存指标：

import io.github.mweirauch.micrometer.jvm.extras.ProcessMemoryMetrics;
import io.micrometer.core.instrument.binder.MeterBinder;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

@Configuration
public class JVMConfig {
    @Bean
    public MeterBinder processMemoryMetrics() {
        return new ProcessMemoryMetrics();
    }
}

application.yml

management:
  metrics:
    tags:
      application: ${spring.profiles.active}_${spring.application.name}
  endpoints:
    web:
      base-path: /actuator
      exposure:
        include: prometheus,health
server:
  tomcat:
    mbeanregistry:
      enabled: true

这里主要是三件事情：

• 标记application名称；
• 限制只能查询prometheus,health两个actuator查询；
• 启用Tomcat指标。

prometheus.yml

配置prometheus抓取程序，如下：

global:
  scrape_interval: 30s
scrape_configs:
...
  # JVM (Micrometer)
  - job_name: 'kubernetes-service-endpoints'
    kubernetes_sd_configs:
    - role: endpoints
    relabel_configs:
    - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape]
      action: keep
      regex: true
    - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scheme]
      action: replace
      target_label: __scheme__
      regex: (https?)
    - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_path]
      action: replace
      target_label: __metrics_path__
      regex: (.+)
    - source_labels: [__address__, __meta_kubernetes_service_annotation_prometheus_io_port]
      action: replace
      target_label: __address__
      regex: (.+)(?::\d+);(\d+)
      replacement: $1:$2
    - action: labelmap
      regex: __meta_kubernetes_service_label_(.+)
    - source_labels: [__meta_kubernetes_namespace]
      action: replace
      target_label: kubernetes_namespace
    - source_labels: [__meta_kubernetes_service_name]
      action: replace
      target_label: application

这里主要就是prometheus抓取程序通过k8s自动发现endpoints，找到k8s中的Spring服务暴露的prometheus指标。需要注意__meta_kubernetes_service_annotation_开头的注解，需要与k8s的service中注解保持一致。

在实践中比较完整的prometheus抓取配置内容（上面之包含了对spring中prometheus指标抓取），参考如下：

global:
  scrape_interval: 30s
  external_labels:
    clusterArn: arn:aws:eks:us-east-1:xxx:cluster/uat
    cluster: uat
scrape_configs:
  # pod metrics
  - job_name: pod_exporter
    kubernetes_sd_configs:
      - role: pod
  # container metrics
  - job_name: cadvisor
    scheme: https
    authorization:
      credentials_file: /var/run/secrets/kubernetes.io/serviceaccount/token
    kubernetes_sd_configs:
      - role: node
    relabel_configs:
      - action: labelmap
        regex: __meta_kubernetes_node_label_(.+)
      - replacement: kubernetes.default.svc:443
        target_label: __address__
      - source_labels: [__meta_kubernetes_node_name]
        regex: (.+)
        target_label: __metrics_path__
        replacement: /api/v1/nodes/$1/proxy/metrics/cadvisor
    metric_relabel_configs:
      - source_labels: [instance]
        separator: ;
        regex: (.+)
        target_label: node
        replacement: $1
        action: replace
  # apiserver metrics
  - bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
    job_name: kubernetes-apiservers
    kubernetes_sd_configs:
    - role: endpoints
    relabel_configs:
    - action: keep
      regex: default;kubernetes;https
      source_labels:
      - __meta_kubernetes_namespace
      - __meta_kubernetes_service_name
      - __meta_kubernetes_endpoint_port_name
    scheme: https
  # kube proxy metrics
  - job_name: kube-proxy
    honor_labels: true
    kubernetes_sd_configs:
    - role: pod
    relabel_configs:
    - action: keep
      source_labels:
      - __meta_kubernetes_namespace
      - __meta_kubernetes_pod_name
      separator: '/'
      regex: 'kube-system/kube-proxy.+'
    - source_labels:
      - __address__
      action: replace
      target_label: __address__
      regex: (.+?)(\\:\\d+)?
      replacement: $1:10249
  # kube-state-metrics
  - job_name: kube-state-metrics
    honor_timestamps: true
    scrape_interval: 1m
    scrape_timeout: 1m
    metrics_path: /metrics
    scheme: http
    static_configs:
    - targets:
      - kube-state-metrics.kube-system.svc.cluster.local:8080
  # node-exporter
  - job_name: 'node-exporter'
    kubernetes_sd_configs:
    - role: node
    relabel_configs:
    - action: replace
      source_labels: [__address__]
      regex: '(.*):10250'
      replacement: '${1}:9100'
      target_label: __address__
  # JVM (Micrometer)
  - job_name: 'kubernetes-service-endpoints'
    kubernetes_sd_configs:
    - role: endpoints
    relabel_configs:
    - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape]
      action: keep
      regex: true
    - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scheme]
      action: replace
      target_label: __scheme__
      regex: (https?)
    - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_path]
      action: replace
      target_label: __metrics_path__
      regex: (.+)
    - source_labels: [__address__, __meta_kubernetes_service_annotation_prometheus_io_port]
      action: replace
      target_label: __address__
      regex: (.+)(?::\d+);(\d+)
      replacement: $1:$2
    - action: labelmap
      regex: __meta_kubernetes_service_label_(.+)
    - source_labels: [__meta_kubernetes_namespace]
      action: replace
      target_label: kubernetes_namespace
    - source_labels: [__meta_kubernetes_service_name]
      action: replace
      target_label: application

k8s.yaml

k8s的service部署配置，如下：

apiVersion: v1
kind: Service
metadata:
  labels:
    app: {{ .Values.services.xxxxx.name }}
  name: {{ .Values.services.xxxxx.name }}
  namespace: {{ .Release.Namespace }}
  annotations:
    alb.ingress.kubernetes.io/healthcheck-path: {{ .Values.services.xxxxx.health.path }}
    alb.ingress.kubernetes.io/healthcheck-port: '{{ .Values.services.xxxxx.health.port }}'
    prometheus.io/path: {{ .Values.services.xxxxx.prometheus.path }}
    prometheus.io/port: '{{ .Values.services.xxxxx.prometheus.port }}'
    prometheus.io/scrape: "true"
spec:
  ports:
    - name: http
      port: {{ .Values.services.xxxxx.port }}
      targetPort: {{ .Values.services.xxxxx.port }}
  selector:
    app: {{ .Values.services.xxxxx.name }}
  type: ClusterIP

注意，这里的注解需要与prometheus抓取程序保持一致，如下图：

这里还有一个就是k8s里面的注释不规则命名，如点，斜杆符号等，在prometheus抓取程序这边都被转化成下划线符号，反正prometheus抓取程序遇到不规则的k8s注释命名符号，都被转成下划线。

三个关键注释：

• prometheus.io/path:设置springboot的暴露的prometheus指标路径，即spring actuator路径；
• prometheus.io/port:设置springboot的暴露的prometheus指标端口，即spring actuator端口；
• prometheus.io/scrape:是否启用prometheus抓取程序。

JVM (Micrometer)面板效果

总结

到这里就完成了对Spring项目添加prometheus指标过程，主要就是添加micrometer和micrometer-jvm-extras依赖，启用micrometer-jvm-extras的内存指标。放开spring actuator prometheus相关端点，启用tomcat指标，配置prometheus抓取程序，设置K8S Service的prometheus抓取程序注释配置。重新发布部署，在grafana查看效果。

参考：

• Micrometer Installing
• JVM (Micrometer)
• micrometer-jvm-extras
• Prometheus系列（4）之Springboot集成Micrometer的JVM监控
• reachlin/prometheus.yml
• Monitoring a Spring Boot application in Kubernetes with Prometheus
• prometheus 独立安装监控k8s
• 55. 监控 Kubernetes 常用资源对象