K8s Prometheus，针对 Java 的完美监控方案
GC 停顿导致接口超时，排查时发现 JVM 老年代占用率飙到 90%------如果你的 Java 应用在 K8s 上是个"黑盒"，再大的集群也算力浪费。本文用完整实战，让你的 JVM 指标在 Grafana 里一览无余。
一、一个 GC 卡顿，查了三天
"应用又超时了。"
凌晨两点，值班手机响了。监控图显示：接口 P99 延迟从 50ms 跳到了 3 秒，持续了整整两分钟，然后又恢复了。
老李查了三天，翻遍了日志、看了 CPU、查了网络，什么都没发现。最后有人提醒了一句：看一眼 GC 日志。
结果发现，Full GC 停了 2.1 秒。再深挖，每次 G1 垃圾回收老年代占用率飙升到 98% 时，就会触发一次大停顿。
问题找到了，但更扎心的是------为什么这些 JVM 指标，没有人提前看到？
这就是 Java 应用在 K8s 上的典型困境：CPU 和内存看得见，堆内存、GC 次数、线程数全是盲区。
#mermaid-svg-tLgxIgrJnVXHXueA{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-tLgxIgrJnVXHXueA .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-tLgxIgrJnVXHXueA .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-tLgxIgrJnVXHXueA .error-icon{fill:#552222;}#mermaid-svg-tLgxIgrJnVXHXueA .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-tLgxIgrJnVXHXueA .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-tLgxIgrJnVXHXueA .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-tLgxIgrJnVXHXueA .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-tLgxIgrJnVXHXueA .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-tLgxIgrJnVXHXueA .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-tLgxIgrJnVXHXueA .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-tLgxIgrJnVXHXueA .marker{fill:#333333;stroke:#333333;}#mermaid-svg-tLgxIgrJnVXHXueA .marker.cross{stroke:#333333;}#mermaid-svg-tLgxIgrJnVXHXueA svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-tLgxIgrJnVXHXueA p{margin:0;}#mermaid-svg-tLgxIgrJnVXHXueA .mermaid-main-font{font-family:"trebuchet ms",verdana,arial,sans-serif;}#mermaid-svg-tLgxIgrJnVXHXueA .exclude-range{fill:#eeeeee;}#mermaid-svg-tLgxIgrJnVXHXueA .section{stroke:none;opacity:0.2;}#mermaid-svg-tLgxIgrJnVXHXueA .section0{fill:rgba(102, 102, 255, 0.49);}#mermaid-svg-tLgxIgrJnVXHXueA .section2{fill:#fff400;}#mermaid-svg-tLgxIgrJnVXHXueA .section1,#mermaid-svg-tLgxIgrJnVXHXueA .section3{fill:white;opacity:0.2;}#mermaid-svg-tLgxIgrJnVXHXueA .sectionTitle0{fill:#333;}#mermaid-svg-tLgxIgrJnVXHXueA .sectionTitle1{fill:#333;}#mermaid-svg-tLgxIgrJnVXHXueA .sectionTitle2{fill:#333;}#mermaid-svg-tLgxIgrJnVXHXueA .sectionTitle3{fill:#333;}#mermaid-svg-tLgxIgrJnVXHXueA .sectionTitle{text-anchor:start;font-family:"trebuchet ms",verdana,arial,sans-serif;}#mermaid-svg-tLgxIgrJnVXHXueA .grid .tick{stroke:lightgrey;opacity:0.8;shape-rendering:crispEdges;}#mermaid-svg-tLgxIgrJnVXHXueA .grid .tick text{font-family:"trebuchet ms",verdana,arial,sans-serif;fill:#333;}#mermaid-svg-tLgxIgrJnVXHXueA .grid path{stroke-width:0;}#mermaid-svg-tLgxIgrJnVXHXueA .today{fill:none;stroke:red;stroke-width:2px;}#mermaid-svg-tLgxIgrJnVXHXueA .task{stroke-width:2;}#mermaid-svg-tLgxIgrJnVXHXueA .taskText{text-anchor:middle;font-family:"trebuchet ms",verdana,arial,sans-serif;}#mermaid-svg-tLgxIgrJnVXHXueA .taskTextOutsideRight{fill:black;text-anchor:start;font-family:"trebuchet ms",verdana,arial,sans-serif;}#mermaid-svg-tLgxIgrJnVXHXueA .taskTextOutsideLeft{fill:black;text-anchor:end;}#mermaid-svg-tLgxIgrJnVXHXueA .task.clickable{cursor:pointer;}#mermaid-svg-tLgxIgrJnVXHXueA .taskText.clickable{cursor:pointer;fill:#003163!important;font-weight:bold;}#mermaid-svg-tLgxIgrJnVXHXueA .taskTextOutsideLeft.clickable{cursor:pointer;fill:#003163!important;font-weight:bold;}#mermaid-svg-tLgxIgrJnVXHXueA .taskTextOutsideRight.clickable{cursor:pointer;fill:#003163!important;font-weight:bold;}#mermaid-svg-tLgxIgrJnVXHXueA .taskText0,#mermaid-svg-tLgxIgrJnVXHXueA .taskText1,#mermaid-svg-tLgxIgrJnVXHXueA .taskText2,#mermaid-svg-tLgxIgrJnVXHXueA .taskText3{fill:white;}#mermaid-svg-tLgxIgrJnVXHXueA .task0,#mermaid-svg-tLgxIgrJnVXHXueA .task1,#mermaid-svg-tLgxIgrJnVXHXueA .task2,#mermaid-svg-tLgxIgrJnVXHXueA .task3{fill:#8a90dd;stroke:#534fbc;}#mermaid-svg-tLgxIgrJnVXHXueA .taskTextOutside0,#mermaid-svg-tLgxIgrJnVXHXueA .taskTextOutside2{fill:black;}#mermaid-svg-tLgxIgrJnVXHXueA .taskTextOutside1,#mermaid-svg-tLgxIgrJnVXHXueA .taskTextOutside3{fill:black;}#mermaid-svg-tLgxIgrJnVXHXueA .active0,#mermaid-svg-tLgxIgrJnVXHXueA .active1,#mermaid-svg-tLgxIgrJnVXHXueA .active2,#mermaid-svg-tLgxIgrJnVXHXueA .active3{fill:#bfc7ff;stroke:#534fbc;}#mermaid-svg-tLgxIgrJnVXHXueA .activeText0,#mermaid-svg-tLgxIgrJnVXHXueA .activeText1,#mermaid-svg-tLgxIgrJnVXHXueA .activeText2,#mermaid-svg-tLgxIgrJnVXHXueA .activeText3{fill:black!important;}#mermaid-svg-tLgxIgrJnVXHXueA .done0,#mermaid-svg-tLgxIgrJnVXHXueA .done1,#mermaid-svg-tLgxIgrJnVXHXueA .done2,#mermaid-svg-tLgxIgrJnVXHXueA .done3{stroke:grey;fill:lightgrey;stroke-width:2;}#mermaid-svg-tLgxIgrJnVXHXueA .doneText0,#mermaid-svg-tLgxIgrJnVXHXueA .doneText1,#mermaid-svg-tLgxIgrJnVXHXueA .doneText2,#mermaid-svg-tLgxIgrJnVXHXueA .doneText3{fill:black!important;}#mermaid-svg-tLgxIgrJnVXHXueA .doneText0.taskTextOutsideLeft,#mermaid-svg-tLgxIgrJnVXHXueA .doneText0.taskTextOutsideRight,#mermaid-svg-tLgxIgrJnVXHXueA .doneText1.taskTextOutsideLeft,#mermaid-svg-tLgxIgrJnVXHXueA .doneText1.taskTextOutsideRight,#mermaid-svg-tLgxIgrJnVXHXueA .doneText2.taskTextOutsideLeft,#mermaid-svg-tLgxIgrJnVXHXueA .doneText2.taskTextOutsideRight,#mermaid-svg-tLgxIgrJnVXHXueA .doneText3.taskTextOutsideLeft,#mermaid-svg-tLgxIgrJnVXHXueA .doneText3.taskTextOutsideRight{fill:black!important;}#mermaid-svg-tLgxIgrJnVXHXueA .crit0,#mermaid-svg-tLgxIgrJnVXHXueA .crit1,#mermaid-svg-tLgxIgrJnVXHXueA .crit2,#mermaid-svg-tLgxIgrJnVXHXueA .crit3{stroke:#ff8888;fill:red;stroke-width:2;}#mermaid-svg-tLgxIgrJnVXHXueA .activeCrit0,#mermaid-svg-tLgxIgrJnVXHXueA .activeCrit1,#mermaid-svg-tLgxIgrJnVXHXueA .activeCrit2,#mermaid-svg-tLgxIgrJnVXHXueA .activeCrit3{stroke:#ff8888;fill:#bfc7ff;stroke-width:2;}#mermaid-svg-tLgxIgrJnVXHXueA .doneCrit0,#mermaid-svg-tLgxIgrJnVXHXueA .doneCrit1,#mermaid-svg-tLgxIgrJnVXHXueA .doneCrit2,#mermaid-svg-tLgxIgrJnVXHXueA .doneCrit3{stroke:#ff8888;fill:lightgrey;stroke-width:2;cursor:pointer;shape-rendering:crispEdges;}#mermaid-svg-tLgxIgrJnVXHXueA .milestone{transform:rotate(45deg) scale(0.8,0.8);}#mermaid-svg-tLgxIgrJnVXHXueA .milestoneText{font-style:italic;}#mermaid-svg-tLgxIgrJnVXHXueA .doneCritText0,#mermaid-svg-tLgxIgrJnVXHXueA .doneCritText1,#mermaid-svg-tLgxIgrJnVXHXueA .doneCritText2,#mermaid-svg-tLgxIgrJnVXHXueA .doneCritText3{fill:black!important;}#mermaid-svg-tLgxIgrJnVXHXueA .doneCritText0.taskTextOutsideLeft,#mermaid-svg-tLgxIgrJnVXHXueA .doneCritText0.taskTextOutsideRight,#mermaid-svg-tLgxIgrJnVXHXueA .doneCritText1.taskTextOutsideLeft,#mermaid-svg-tLgxIgrJnVXHXueA .doneCritText1.taskTextOutsideRight,#mermaid-svg-tLgxIgrJnVXHXueA .doneCritText2.taskTextOutsideLeft,#mermaid-svg-tLgxIgrJnVXHXueA .doneCritText2.taskTextOutsideRight,#mermaid-svg-tLgxIgrJnVXHXueA .doneCritText3.taskTextOutsideLeft,#mermaid-svg-tLgxIgrJnVXHXueA .doneCritText3.taskTextOutsideRight{fill:black!important;}#mermaid-svg-tLgxIgrJnVXHXueA .vert{stroke:navy;}#mermaid-svg-tLgxIgrJnVXHXueA .vertText{font-size:15px;text-anchor:middle;fill:navy!important;}#mermaid-svg-tLgxIgrJnVXHXueA .activeCritText0,#mermaid-svg-tLgxIgrJnVXHXueA .activeCritText1,#mermaid-svg-tLgxIgrJnVXHXueA .activeCritText2,#mermaid-svg-tLgxIgrJnVXHXueA .activeCritText3{fill:black!important;}#mermaid-svg-tLgxIgrJnVXHXueA .titleText{text-anchor:middle;font-size:18px;fill:#333;font-family:"trebuchet ms",verdana,arial,sans-serif;}#mermaid-svg-tLgxIgrJnVXHXueA :root{--mermaid-font-family:"trebuchet ms",verdana,arial,sans-serif;} 10:00 11:00 12:00 13:00 14:00 15:00 16:00 17:00 老年代占用率持续爬升 85pct 到 98pct 接口P99延迟 50ms 到 3s Full GC 暂停 2.1 秒 人工介入排查问题 查看GC日志发现根因 调整JVM参数修复 潜伏期爆发期定位恢复期 故障时间线
二、Java 监控的本质：看穿 JVM
要让 Java 应用被完美监控，核心是解决一个矛盾：
JVM 的运行时状态，藏在 JMX（Java Management Extensions）里。Prometheus 不认识 JMX，它只认 /metrics 端点。
这就需要一个"翻译官"------把 JMX 的 MBean 数据，转换成 Prometheus 的指标格式。
#mermaid-svg-Tu1PVkJZxRVNGJwh{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-Tu1PVkJZxRVNGJwh .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-Tu1PVkJZxRVNGJwh .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-Tu1PVkJZxRVNGJwh .error-icon{fill:#552222;}#mermaid-svg-Tu1PVkJZxRVNGJwh .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-Tu1PVkJZxRVNGJwh .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-Tu1PVkJZxRVNGJwh .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-Tu1PVkJZxRVNGJwh .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-Tu1PVkJZxRVNGJwh .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-Tu1PVkJZxRVNGJwh .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-Tu1PVkJZxRVNGJwh .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-Tu1PVkJZxRVNGJwh .marker{fill:#333333;stroke:#333333;}#mermaid-svg-Tu1PVkJZxRVNGJwh .marker.cross{stroke:#333333;}#mermaid-svg-Tu1PVkJZxRVNGJwh svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-Tu1PVkJZxRVNGJwh p{margin:0;}#mermaid-svg-Tu1PVkJZxRVNGJwh .label{font-family:"trebuchet ms",verdana,arial,sans-serif;color:#333;}#mermaid-svg-Tu1PVkJZxRVNGJwh .cluster-label text{fill:#333;}#mermaid-svg-Tu1PVkJZxRVNGJwh .cluster-label span{color:#333;}#mermaid-svg-Tu1PVkJZxRVNGJwh .cluster-label span p{background-color:transparent;}#mermaid-svg-Tu1PVkJZxRVNGJwh .label text,#mermaid-svg-Tu1PVkJZxRVNGJwh span{fill:#333;color:#333;}#mermaid-svg-Tu1PVkJZxRVNGJwh .node rect,#mermaid-svg-Tu1PVkJZxRVNGJwh .node circle,#mermaid-svg-Tu1PVkJZxRVNGJwh .node ellipse,#mermaid-svg-Tu1PVkJZxRVNGJwh .node polygon,#mermaid-svg-Tu1PVkJZxRVNGJwh .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-Tu1PVkJZxRVNGJwh .rough-node .label text,#mermaid-svg-Tu1PVkJZxRVNGJwh .node .label text,#mermaid-svg-Tu1PVkJZxRVNGJwh .image-shape .label,#mermaid-svg-Tu1PVkJZxRVNGJwh .icon-shape .label{text-anchor:middle;}#mermaid-svg-Tu1PVkJZxRVNGJwh .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-Tu1PVkJZxRVNGJwh .rough-node .label,#mermaid-svg-Tu1PVkJZxRVNGJwh .node .label,#mermaid-svg-Tu1PVkJZxRVNGJwh .image-shape .label,#mermaid-svg-Tu1PVkJZxRVNGJwh .icon-shape .label{text-align:center;}#mermaid-svg-Tu1PVkJZxRVNGJwh .node.clickable{cursor:pointer;}#mermaid-svg-Tu1PVkJZxRVNGJwh .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-Tu1PVkJZxRVNGJwh .arrowheadPath{fill:#333333;}#mermaid-svg-Tu1PVkJZxRVNGJwh .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-Tu1PVkJZxRVNGJwh .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-Tu1PVkJZxRVNGJwh .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-Tu1PVkJZxRVNGJwh .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-Tu1PVkJZxRVNGJwh .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-Tu1PVkJZxRVNGJwh .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-Tu1PVkJZxRVNGJwh .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-Tu1PVkJZxRVNGJwh .cluster text{fill:#333;}#mermaid-svg-Tu1PVkJZxRVNGJwh .cluster span{color:#333;}#mermaid-svg-Tu1PVkJZxRVNGJwh div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-Tu1PVkJZxRVNGJwh .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-Tu1PVkJZxRVNGJwh rect.text{fill:none;stroke-width:0;}#mermaid-svg-Tu1PVkJZxRVNGJwh .icon-shape,#mermaid-svg-Tu1PVkJZxRVNGJwh .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-Tu1PVkJZxRVNGJwh .icon-shape p,#mermaid-svg-Tu1PVkJZxRVNGJwh .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-Tu1PVkJZxRVNGJwh .icon-shape .label rect,#mermaid-svg-Tu1PVkJZxRVNGJwh .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-Tu1PVkJZxRVNGJwh .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-Tu1PVkJZxRVNGJwh .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-Tu1PVkJZxRVNGJwh :root{--mermaid-font-family:"trebuchet ms",verdana,arial,sans-serif;} 可视化
存储与告警
翻译层
Java 应用 JVM
JMX MBean
堆/GC/线程/类加载
Micrometer
SDK 方式
需改代码
JMX Exporter
Agent 方式
无侵入
Prometheus
指标存储 + 告警
Grafana 仪表盘
2.1 两种主流方案对比
| 方案 | 原理 | 侵入性 | JVM 指标 | 业务指标 | 推荐场景 |
|---|---|---|---|---|---|
| client_java | SDK 直接暴露指标 | 需要改代码 | ✅ 开箱即用 | ✅ 可自定义 | Spring Boot/新项目（首选） |
| JMX Exporter | Java Agent 无侵入 | 不改代码 | ✅ 完整暴露 | ❌ 受限 | 老应用/无法改代码 |
选型结论：能改代码用
client_java（Micrometer），不能改代码用 JMX Exporter。混着用也行，不冲突。
2.2 JVM 指标清单
打通之后，能拿到哪些核心指标？
| 分类 | 关键指标 | 什么情况该报警 |
|---|---|---|
| 堆内存 | jvm_memory_bytes_used / _max |
老年代占用率 > 85%，连续 3 个采集周期 |
| GC | jvm_gc_pause_seconds（耗时）/ jvm_gc_pause_seconds_count（次数） |
Full GC 耗时 > 1s，或 1 分钟内 Full GC 超过 3 次 |
| 线程 | jvm_threads_live_threads / jvm_threads_peak |
线程数超过阈值，或线程数持续增长不回落 |
| 类加载 | jvm_classes_loaded |
类加载数异常增长（内存泄漏征兆） |
| CPU/内存 | system_cpu_usage / jvm_memory_* |
容器 CPU 限流率过高 |
三、整体架构图
#mermaid-svg-wOcM8beDAQ41ivZF{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-wOcM8beDAQ41ivZF .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-wOcM8beDAQ41ivZF .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-wOcM8beDAQ41ivZF .error-icon{fill:#552222;}#mermaid-svg-wOcM8beDAQ41ivZF .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-wOcM8beDAQ41ivZF .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-wOcM8beDAQ41ivZF .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-wOcM8beDAQ41ivZF .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-wOcM8beDAQ41ivZF .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-wOcM8beDAQ41ivZF .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-wOcM8beDAQ41ivZF .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-wOcM8beDAQ41ivZF .marker{fill:#333333;stroke:#333333;}#mermaid-svg-wOcM8beDAQ41ivZF .marker.cross{stroke:#333333;}#mermaid-svg-wOcM8beDAQ41ivZF svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-wOcM8beDAQ41ivZF p{margin:0;}#mermaid-svg-wOcM8beDAQ41ivZF .label{font-family:"trebuchet ms",verdana,arial,sans-serif;color:#333;}#mermaid-svg-wOcM8beDAQ41ivZF .cluster-label text{fill:#333;}#mermaid-svg-wOcM8beDAQ41ivZF .cluster-label span{color:#333;}#mermaid-svg-wOcM8beDAQ41ivZF .cluster-label span p{background-color:transparent;}#mermaid-svg-wOcM8beDAQ41ivZF .label text,#mermaid-svg-wOcM8beDAQ41ivZF span{fill:#333;color:#333;}#mermaid-svg-wOcM8beDAQ41ivZF .node rect,#mermaid-svg-wOcM8beDAQ41ivZF .node circle,#mermaid-svg-wOcM8beDAQ41ivZF .node ellipse,#mermaid-svg-wOcM8beDAQ41ivZF .node polygon,#mermaid-svg-wOcM8beDAQ41ivZF .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-wOcM8beDAQ41ivZF .rough-node .label text,#mermaid-svg-wOcM8beDAQ41ivZF .node .label text,#mermaid-svg-wOcM8beDAQ41ivZF .image-shape .label,#mermaid-svg-wOcM8beDAQ41ivZF .icon-shape .label{text-anchor:middle;}#mermaid-svg-wOcM8beDAQ41ivZF .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-wOcM8beDAQ41ivZF .rough-node .label,#mermaid-svg-wOcM8beDAQ41ivZF .node .label,#mermaid-svg-wOcM8beDAQ41ivZF .image-shape .label,#mermaid-svg-wOcM8beDAQ41ivZF .icon-shape .label{text-align:center;}#mermaid-svg-wOcM8beDAQ41ivZF .node.clickable{cursor:pointer;}#mermaid-svg-wOcM8beDAQ41ivZF .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-wOcM8beDAQ41ivZF .arrowheadPath{fill:#333333;}#mermaid-svg-wOcM8beDAQ41ivZF .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-wOcM8beDAQ41ivZF .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-wOcM8beDAQ41ivZF .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-wOcM8beDAQ41ivZF .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-wOcM8beDAQ41ivZF .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-wOcM8beDAQ41ivZF .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-wOcM8beDAQ41ivZF .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-wOcM8beDAQ41ivZF .cluster text{fill:#333;}#mermaid-svg-wOcM8beDAQ41ivZF .cluster span{color:#333;}#mermaid-svg-wOcM8beDAQ41ivZF div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-wOcM8beDAQ41ivZF .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-wOcM8beDAQ41ivZF rect.text{fill:none;stroke-width:0;}#mermaid-svg-wOcM8beDAQ41ivZF .icon-shape,#mermaid-svg-wOcM8beDAQ41ivZF .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-wOcM8beDAQ41ivZF .icon-shape p,#mermaid-svg-wOcM8beDAQ41ivZF .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-wOcM8beDAQ41ivZF .icon-shape .label rect,#mermaid-svg-wOcM8beDAQ41ivZF .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-wOcM8beDAQ41ivZF .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-wOcM8beDAQ41ivZF .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-wOcM8beDAQ41ivZF :root{--mermaid-font-family:"trebuchet ms",verdana,arial,sans-serif;} 展示与告警
K8s 集群
采集
Java 应用 Pod
Micrometer 注册
K8s 服务发现 15s 抓取
Java App
Spring Boot
/actuator/prometheus
指标端点
Prometheus
ServiceMonitor 自动发现
Grafana 仪表盘
Dashboard ID: 8563
告警通知
- 蓝色：Java 应用层------代码里用 Micrometer 注册 JVM 指标，暴露 HTTP 端点
- 橙色：Prometheus 层------通过 K8s 服务发现自动抓取
- 绿色：展示层------Grafana 大盘展示，告警通知触达
四、实战：Spring Boot 应用完美接入
#mermaid-svg-wcb5iTBHbUPcQd88{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-wcb5iTBHbUPcQd88 .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-wcb5iTBHbUPcQd88 .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-wcb5iTBHbUPcQd88 .error-icon{fill:#552222;}#mermaid-svg-wcb5iTBHbUPcQd88 .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-wcb5iTBHbUPcQd88 .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-wcb5iTBHbUPcQd88 .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-wcb5iTBHbUPcQd88 .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-wcb5iTBHbUPcQd88 .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-wcb5iTBHbUPcQd88 .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-wcb5iTBHbUPcQd88 .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-wcb5iTBHbUPcQd88 .marker{fill:#333333;stroke:#333333;}#mermaid-svg-wcb5iTBHbUPcQd88 .marker.cross{stroke:#333333;}#mermaid-svg-wcb5iTBHbUPcQd88 svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-wcb5iTBHbUPcQd88 p{margin:0;}#mermaid-svg-wcb5iTBHbUPcQd88 .label{font-family:"trebuchet ms",verdana,arial,sans-serif;color:#333;}#mermaid-svg-wcb5iTBHbUPcQd88 .cluster-label text{fill:#333;}#mermaid-svg-wcb5iTBHbUPcQd88 .cluster-label span{color:#333;}#mermaid-svg-wcb5iTBHbUPcQd88 .cluster-label span p{background-color:transparent;}#mermaid-svg-wcb5iTBHbUPcQd88 .label text,#mermaid-svg-wcb5iTBHbUPcQd88 span{fill:#333;color:#333;}#mermaid-svg-wcb5iTBHbUPcQd88 .node rect,#mermaid-svg-wcb5iTBHbUPcQd88 .node circle,#mermaid-svg-wcb5iTBHbUPcQd88 .node ellipse,#mermaid-svg-wcb5iTBHbUPcQd88 .node polygon,#mermaid-svg-wcb5iTBHbUPcQd88 .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-wcb5iTBHbUPcQd88 .rough-node .label text,#mermaid-svg-wcb5iTBHbUPcQd88 .node .label text,#mermaid-svg-wcb5iTBHbUPcQd88 .image-shape .label,#mermaid-svg-wcb5iTBHbUPcQd88 .icon-shape .label{text-anchor:middle;}#mermaid-svg-wcb5iTBHbUPcQd88 .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-wcb5iTBHbUPcQd88 .rough-node .label,#mermaid-svg-wcb5iTBHbUPcQd88 .node .label,#mermaid-svg-wcb5iTBHbUPcQd88 .image-shape .label,#mermaid-svg-wcb5iTBHbUPcQd88 .icon-shape .label{text-align:center;}#mermaid-svg-wcb5iTBHbUPcQd88 .node.clickable{cursor:pointer;}#mermaid-svg-wcb5iTBHbUPcQd88 .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-wcb5iTBHbUPcQd88 .arrowheadPath{fill:#333333;}#mermaid-svg-wcb5iTBHbUPcQd88 .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-wcb5iTBHbUPcQd88 .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-wcb5iTBHbUPcQd88 .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-wcb5iTBHbUPcQd88 .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-wcb5iTBHbUPcQd88 .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-wcb5iTBHbUPcQd88 .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-wcb5iTBHbUPcQd88 .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-wcb5iTBHbUPcQd88 .cluster text{fill:#333;}#mermaid-svg-wcb5iTBHbUPcQd88 .cluster span{color:#333;}#mermaid-svg-wcb5iTBHbUPcQd88 div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-wcb5iTBHbUPcQd88 .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-wcb5iTBHbUPcQd88 rect.text{fill:none;stroke-width:0;}#mermaid-svg-wcb5iTBHbUPcQd88 .icon-shape,#mermaid-svg-wcb5iTBHbUPcQd88 .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-wcb5iTBHbUPcQd88 .icon-shape p,#mermaid-svg-wcb5iTBHbUPcQd88 .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-wcb5iTBHbUPcQd88 .icon-shape .label rect,#mermaid-svg-wcb5iTBHbUPcQd88 .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-wcb5iTBHbUPcQd88 .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-wcb5iTBHbUPcQd88 .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-wcb5iTBHbUPcQd88 :root{--mermaid-font-family:"trebuchet ms",verdana,arial,sans-serif;} ① 添加依赖
actuator + micrometer
② 开启端点
/actuator/prometheus
③ 本地验证
curl 测试指标
④ 打包部署
Docker + K8s Deployment
⑤ 配置采集
ServiceMonitor
⑥ Grafana 导入
Dashboard ID: 8563
4.1 第一步：添加依赖
xml
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-actuator</artifactId>
</dependency>
<dependency>
<groupId>io.micrometer</groupId>
<artifactId>micrometer-registry-prometheus</artifactId>
</dependency>就这两行，Spring Boot 会自动配置好 /actuator/prometheus 端点。
4.2 第二步：开启端点暴露
properties
# application.properties
management.endpoints.web.exposure.include=health,info,prometheus
management.endpoint.prometheus.enabled=true
management.metrics.export.prometheus.enabled=true如果想看更详细的线程和内存数据，加两行配置：
properties
management.metrics.enable.process.files=true
management.metrics.enable.process.uptime=true4.3 第三步：本地验证
bash
# 启动应用后，访问端点
curl http://localhost:8080/actuator/prometheus | head -30返回的数据应该包含以下 JVM 指标开头的内容：
jvm_memory_used_bytes
jvm_gc_pause_seconds
jvm_threads_live_threads有这些，就说明暴露成功了。
4.4 第四步：打包镜像并部署到 K8s
Dockerfile：
dockerfile
FROM openjdk:17-jdk-slim
COPY target/*.jar app.jar
ENTRYPOINT ["java", "-jar", "/app.jar"]K8s Deployment（关键：加注解）：
yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: java-app
spec:
template:
metadata:
annotations:
prometheus.io/scrape: "true" # Prometheus 来抓我
prometheus.io/path: "/actuator/prometheus"
prometheus.io/port: "8080"
labels:
app: java-app
spec:
containers:
- name: app
image: harbor.internal/java-app:latest
ports:
- containerPort: 8080
name: http这三行 prometheus.io 注解是核心------告诉 Prometheus："我的指标在 8080 端口的 /actuator/prometheus 路径上"。
4.5 第五步：配置 Prometheus 采集（ServiceMonitor）
如果使用 Prometheus Operator，创建一个 ServiceMonitor：
yaml
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: java-app-monitor
spec:
selector:
matchLabels:
app: java-app # 和 Deployment 的标签对齐
endpoints:
- port: http # 对应 Service 的端口名
path: /actuator/prometheus
interval: 15s
namespaceSelector:
matchNames:
- default创建后，Prometheus 会自动发现这个应用并开始抓取。
4.6 第六步：Grafana 导入 JVM 监控面板
Grafana 官方社区有现成的 JVM 仪表盘，ID 是 4701 或 8563：
登录 Grafana → 左侧 + 号 → Import
输入 Dashboard ID：8563（JVM 仪表盘）
选择 Prometheus 数据源
点击 Import能看到堆内存占用趋势、GC 耗时分布、线程数变化------排查性能问题从"猜"变成了"看"。
五、Prometheus 监控图解读（实战场景）
当应用稳定运行后，你会看到这样几个关键监控图：
#mermaid-svg-IiFROYjkQ4DSxdu0{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-IiFROYjkQ4DSxdu0 .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-IiFROYjkQ4DSxdu0 .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-IiFROYjkQ4DSxdu0 .error-icon{fill:#552222;}#mermaid-svg-IiFROYjkQ4DSxdu0 .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-IiFROYjkQ4DSxdu0 .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-IiFROYjkQ4DSxdu0 .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-IiFROYjkQ4DSxdu0 .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-IiFROYjkQ4DSxdu0 .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-IiFROYjkQ4DSxdu0 .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-IiFROYjkQ4DSxdu0 .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-IiFROYjkQ4DSxdu0 .marker{fill:#333333;stroke:#333333;}#mermaid-svg-IiFROYjkQ4DSxdu0 .marker.cross{stroke:#333333;}#mermaid-svg-IiFROYjkQ4DSxdu0 svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-IiFROYjkQ4DSxdu0 p{margin:0;}#mermaid-svg-IiFROYjkQ4DSxdu0 .label{font-family:"trebuchet ms",verdana,arial,sans-serif;color:#333;}#mermaid-svg-IiFROYjkQ4DSxdu0 .cluster-label text{fill:#333;}#mermaid-svg-IiFROYjkQ4DSxdu0 .cluster-label span{color:#333;}#mermaid-svg-IiFROYjkQ4DSxdu0 .cluster-label span p{background-color:transparent;}#mermaid-svg-IiFROYjkQ4DSxdu0 .label text,#mermaid-svg-IiFROYjkQ4DSxdu0 span{fill:#333;color:#333;}#mermaid-svg-IiFROYjkQ4DSxdu0 .node rect,#mermaid-svg-IiFROYjkQ4DSxdu0 .node circle,#mermaid-svg-IiFROYjkQ4DSxdu0 .node ellipse,#mermaid-svg-IiFROYjkQ4DSxdu0 .node polygon,#mermaid-svg-IiFROYjkQ4DSxdu0 .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-IiFROYjkQ4DSxdu0 .rough-node .label text,#mermaid-svg-IiFROYjkQ4DSxdu0 .node .label text,#mermaid-svg-IiFROYjkQ4DSxdu0 .image-shape .label,#mermaid-svg-IiFROYjkQ4DSxdu0 .icon-shape .label{text-anchor:middle;}#mermaid-svg-IiFROYjkQ4DSxdu0 .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-IiFROYjkQ4DSxdu0 .rough-node .label,#mermaid-svg-IiFROYjkQ4DSxdu0 .node .label,#mermaid-svg-IiFROYjkQ4DSxdu0 .image-shape .label,#mermaid-svg-IiFROYjkQ4DSxdu0 .icon-shape .label{text-align:center;}#mermaid-svg-IiFROYjkQ4DSxdu0 .node.clickable{cursor:pointer;}#mermaid-svg-IiFROYjkQ4DSxdu0 .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-IiFROYjkQ4DSxdu0 .arrowheadPath{fill:#333333;}#mermaid-svg-IiFROYjkQ4DSxdu0 .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-IiFROYjkQ4DSxdu0 .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-IiFROYjkQ4DSxdu0 .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-IiFROYjkQ4DSxdu0 .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-IiFROYjkQ4DSxdu0 .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-IiFROYjkQ4DSxdu0 .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-IiFROYjkQ4DSxdu0 .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-IiFROYjkQ4DSxdu0 .cluster text{fill:#333;}#mermaid-svg-IiFROYjkQ4DSxdu0 .cluster span{color:#333;}#mermaid-svg-IiFROYjkQ4DSxdu0 div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-IiFROYjkQ4DSxdu0 .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-IiFROYjkQ4DSxdu0 rect.text{fill:none;stroke-width:0;}#mermaid-svg-IiFROYjkQ4DSxdu0 .icon-shape,#mermaid-svg-IiFROYjkQ4DSxdu0 .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-IiFROYjkQ4DSxdu0 .icon-shape p,#mermaid-svg-IiFROYjkQ4DSxdu0 .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-IiFROYjkQ4DSxdu0 .icon-shape .label rect,#mermaid-svg-IiFROYjkQ4DSxdu0 .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-IiFROYjkQ4DSxdu0 .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-IiFROYjkQ4DSxdu0 .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-IiFROYjkQ4DSxdu0 :root{--mermaid-font-family:"trebuchet ms",verdana,arial,sans-serif;} 否
是
否
是
否
是
监控图解读
堆内存
锯齿底部上升?
✅ 堆内存正常
❌ 内存泄漏
需 dump 分析
Full GC
暂停 > 1s?
✅ GC 正常
⚠️ 调整 JVM 参数
增大堆内存
线程数
持续增长不回落?
✅ 线程正常
❌ 线程泄漏
检查线程池代码
5.1 堆内存监控图
- 正常模式：锯齿状，每次 GC 后内存回落
- 危险信号：锯齿底部越来越高，说明有内存泄漏
5.2 GC 耗时监控图
报警阈值：Full GC 暂停超过 1 秒，或一分钟内 Young GC 次数超过 10 次。
5.3 线程数监控图
危险信号：线程数阶梯式持续增长不回落，大概率线程泄漏。
六、告警规则配置
有了指标，就要配告警。Prometheus 告警规则如下：
#mermaid-svg-FOOIU3nky6Byes54{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-FOOIU3nky6Byes54 .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-FOOIU3nky6Byes54 .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-FOOIU3nky6Byes54 .error-icon{fill:#552222;}#mermaid-svg-FOOIU3nky6Byes54 .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-FOOIU3nky6Byes54 .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-FOOIU3nky6Byes54 .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-FOOIU3nky6Byes54 .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-FOOIU3nky6Byes54 .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-FOOIU3nky6Byes54 .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-FOOIU3nky6Byes54 .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-FOOIU3nky6Byes54 .marker{fill:#333333;stroke:#333333;}#mermaid-svg-FOOIU3nky6Byes54 .marker.cross{stroke:#333333;}#mermaid-svg-FOOIU3nky6Byes54 svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-FOOIU3nky6Byes54 p{margin:0;}#mermaid-svg-FOOIU3nky6Byes54 .label{font-family:"trebuchet ms",verdana,arial,sans-serif;color:#333;}#mermaid-svg-FOOIU3nky6Byes54 .cluster-label text{fill:#333;}#mermaid-svg-FOOIU3nky6Byes54 .cluster-label span{color:#333;}#mermaid-svg-FOOIU3nky6Byes54 .cluster-label span p{background-color:transparent;}#mermaid-svg-FOOIU3nky6Byes54 .label text,#mermaid-svg-FOOIU3nky6Byes54 span{fill:#333;color:#333;}#mermaid-svg-FOOIU3nky6Byes54 .node rect,#mermaid-svg-FOOIU3nky6Byes54 .node circle,#mermaid-svg-FOOIU3nky6Byes54 .node ellipse,#mermaid-svg-FOOIU3nky6Byes54 .node polygon,#mermaid-svg-FOOIU3nky6Byes54 .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-FOOIU3nky6Byes54 .rough-node .label text,#mermaid-svg-FOOIU3nky6Byes54 .node .label text,#mermaid-svg-FOOIU3nky6Byes54 .image-shape .label,#mermaid-svg-FOOIU3nky6Byes54 .icon-shape .label{text-anchor:middle;}#mermaid-svg-FOOIU3nky6Byes54 .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-FOOIU3nky6Byes54 .rough-node .label,#mermaid-svg-FOOIU3nky6Byes54 .node .label,#mermaid-svg-FOOIU3nky6Byes54 .image-shape .label,#mermaid-svg-FOOIU3nky6Byes54 .icon-shape .label{text-align:center;}#mermaid-svg-FOOIU3nky6Byes54 .node.clickable{cursor:pointer;}#mermaid-svg-FOOIU3nky6Byes54 .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-FOOIU3nky6Byes54 .arrowheadPath{fill:#333333;}#mermaid-svg-FOOIU3nky6Byes54 .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-FOOIU3nky6Byes54 .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-FOOIU3nky6Byes54 .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-FOOIU3nky6Byes54 .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-FOOIU3nky6Byes54 .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-FOOIU3nky6Byes54 .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-FOOIU3nky6Byes54 .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-FOOIU3nky6Byes54 .cluster text{fill:#333;}#mermaid-svg-FOOIU3nky6Byes54 .cluster span{color:#333;}#mermaid-svg-FOOIU3nky6Byes54 div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-FOOIU3nky6Byes54 .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-FOOIU3nky6Byes54 rect.text{fill:none;stroke-width:0;}#mermaid-svg-FOOIU3nky6Byes54 .icon-shape,#mermaid-svg-FOOIU3nky6Byes54 .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-FOOIU3nky6Byes54 .icon-shape p,#mermaid-svg-FOOIU3nky6Byes54 .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-FOOIU3nky6Byes54 .icon-shape .label rect,#mermaid-svg-FOOIU3nky6Byes54 .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-FOOIU3nky6Byes54 .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-FOOIU3nky6Byes54 .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-FOOIU3nky6Byes54 :root{--mermaid-font-family:"trebuchet ms",verdana,arial,sans-serif;} 是持续5分钟
否
是持续1分钟
否
是持续10分钟
否
是
否
Prometheus 告警评估
老年代占用 > 85%?
🔔 JVM_OldGenHighUsage
Full GC 暂停 > 1s?
🔔 JVM_FullGCLongPause
线程数 > 500?
🔔 JVM_HighThreadCount
CPU 限流 > 0.5?
🔔 Container_CPU_Throttling
✅ 应用健康
yaml
groups:
- name: jvm-alerts
rules:
# 老年代占用超 85%，持续 5 分钟
- alert: JVM_OldGenHighUsage
expr: (sum(jvm_memory_bytes_used{area="heap"}) / sum(jvm_memory_bytes_max{area="heap"})) > 0.85
for: 5m
annotations:
summary: "老年代占用率 {{ $value | humanizePercentage }}"
# Full GC 耗时超 1 秒
- alert: JVM_FullGCLongPause
expr: histogram_quantile(0.95, jvm_gc_pause_seconds_bucket{action="end of major GC"}) > 1
for: 1m
annotations:
summary: "Full GC 暂停 {{ $value }}秒"
# 线程数超阈值
- alert: JVM_HighThreadCount
expr: jvm_threads_live_threads > 500
for: 10m
annotations:
summary: "线程数 {{ $value }}，超过阈值 500"
# 容器 CPU 限流（重要！避免 Java 踩坑）
- alert: Container_CPU_Throttling
expr: rate(container_cpu_cfs_throttled_seconds_total[5m]) > 0.5
annotations:
summary: "Pod 被 CPU 限流，考虑调整 Requests/Limits"最后一条告警对 Java 特别重要。很多 Java 应用在容器里"感觉慢"，其实是被 Cgroup 限流了，而 JVM 默认不感知容器的 CPU 限制，需要用
-XX:ActiveProcessorCount或升级 JDK 版本解决。
七、老项目怎么办？JMX Exporter 无侵入方案
如果你的 Java 应用是老项目，不方便加依赖改代码，可以用 JMX Exporter 以 Java Agent 方式运行。
#mermaid-svg-pqnUUyGE1nf8VAYa{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-pqnUUyGE1nf8VAYa .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-pqnUUyGE1nf8VAYa .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-pqnUUyGE1nf8VAYa .error-icon{fill:#552222;}#mermaid-svg-pqnUUyGE1nf8VAYa .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-pqnUUyGE1nf8VAYa .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-pqnUUyGE1nf8VAYa .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-pqnUUyGE1nf8VAYa .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-pqnUUyGE1nf8VAYa .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-pqnUUyGE1nf8VAYa .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-pqnUUyGE1nf8VAYa .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-pqnUUyGE1nf8VAYa .marker{fill:#333333;stroke:#333333;}#mermaid-svg-pqnUUyGE1nf8VAYa .marker.cross{stroke:#333333;}#mermaid-svg-pqnUUyGE1nf8VAYa svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-pqnUUyGE1nf8VAYa p{margin:0;}#mermaid-svg-pqnUUyGE1nf8VAYa .label{font-family:"trebuchet ms",verdana,arial,sans-serif;color:#333;}#mermaid-svg-pqnUUyGE1nf8VAYa .cluster-label text{fill:#333;}#mermaid-svg-pqnUUyGE1nf8VAYa .cluster-label span{color:#333;}#mermaid-svg-pqnUUyGE1nf8VAYa .cluster-label span p{background-color:transparent;}#mermaid-svg-pqnUUyGE1nf8VAYa .label text,#mermaid-svg-pqnUUyGE1nf8VAYa span{fill:#333;color:#333;}#mermaid-svg-pqnUUyGE1nf8VAYa .node rect,#mermaid-svg-pqnUUyGE1nf8VAYa .node circle,#mermaid-svg-pqnUUyGE1nf8VAYa .node ellipse,#mermaid-svg-pqnUUyGE1nf8VAYa .node polygon,#mermaid-svg-pqnUUyGE1nf8VAYa .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-pqnUUyGE1nf8VAYa .rough-node .label text,#mermaid-svg-pqnUUyGE1nf8VAYa .node .label text,#mermaid-svg-pqnUUyGE1nf8VAYa .image-shape .label,#mermaid-svg-pqnUUyGE1nf8VAYa .icon-shape .label{text-anchor:middle;}#mermaid-svg-pqnUUyGE1nf8VAYa .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-pqnUUyGE1nf8VAYa .rough-node .label,#mermaid-svg-pqnUUyGE1nf8VAYa .node .label,#mermaid-svg-pqnUUyGE1nf8VAYa .image-shape .label,#mermaid-svg-pqnUUyGE1nf8VAYa .icon-shape .label{text-align:center;}#mermaid-svg-pqnUUyGE1nf8VAYa .node.clickable{cursor:pointer;}#mermaid-svg-pqnUUyGE1nf8VAYa .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-pqnUUyGE1nf8VAYa .arrowheadPath{fill:#333333;}#mermaid-svg-pqnUUyGE1nf8VAYa .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-pqnUUyGE1nf8VAYa .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-pqnUUyGE1nf8VAYa .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-pqnUUyGE1nf8VAYa .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-pqnUUyGE1nf8VAYa .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-pqnUUyGE1nf8VAYa .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-pqnUUyGE1nf8VAYa .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-pqnUUyGE1nf8VAYa .cluster text{fill:#333;}#mermaid-svg-pqnUUyGE1nf8VAYa .cluster span{color:#333;}#mermaid-svg-pqnUUyGE1nf8VAYa div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-pqnUUyGE1nf8VAYa .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-pqnUUyGE1nf8VAYa rect.text{fill:none;stroke-width:0;}#mermaid-svg-pqnUUyGE1nf8VAYa .icon-shape,#mermaid-svg-pqnUUyGE1nf8VAYa .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-pqnUUyGE1nf8VAYa .icon-shape p,#mermaid-svg-pqnUUyGE1nf8VAYa .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-pqnUUyGE1nf8VAYa .icon-shape .label rect,#mermaid-svg-pqnUUyGE1nf8VAYa .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-pqnUUyGE1nf8VAYa .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-pqnUUyGE1nf8VAYa .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-pqnUUyGE1nf8VAYa :root{--mermaid-font-family:"trebuchet ms",verdana,arial,sans-serif;} 指标暴露
Agent 挂载
老项目 JVM
-javaagent 启动参数
Java 应用
无法改代码
jmx_prometheus_javaagent.jar
Java Agent 无侵入
jmx-config.yaml
规则配置
/metrics 端点
端口 8088
Prometheus 采集
Dockerfile：
dockerfile
FROM openjdk:8-jdk
# 下载 JMX Exporter
ADD https://repo1.maven.org/maven2/io/prometheus/jmx/jmx_prometheus_javaagent/1.0.1/jmx_prometheus_javaagent-1.0.1.jar /agent.jar
# 配置文件（最简单的配置：暴露所有指标）
ADD jmx-config.yaml /jmx-config.yaml
# 启动时挂载 agent
ENTRYPOINT ["java", "-javaagent:/agent.jar=8088:/jmx-config.yaml", "-jar", "app.jar"]jmx-config.yaml：
yaml
rules:
- pattern: ".*"这样完全不用改代码，应用启动时自动挂载 agent，暴露 /metrics 端口。
八、总结
K8s 上监控 Java 应用，核心就是一句话：让 JVM"开口说话"，让 Prometheus"听得懂"。
#mermaid-svg-6JT6dHpE9jWQwQ7N{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-6JT6dHpE9jWQwQ7N .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-6JT6dHpE9jWQwQ7N .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-6JT6dHpE9jWQwQ7N .error-icon{fill:#552222;}#mermaid-svg-6JT6dHpE9jWQwQ7N .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-6JT6dHpE9jWQwQ7N .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-6JT6dHpE9jWQwQ7N .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-6JT6dHpE9jWQwQ7N .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-6JT6dHpE9jWQwQ7N .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-6JT6dHpE9jWQwQ7N .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-6JT6dHpE9jWQwQ7N .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-6JT6dHpE9jWQwQ7N .marker{fill:#333333;stroke:#333333;}#mermaid-svg-6JT6dHpE9jWQwQ7N .marker.cross{stroke:#333333;}#mermaid-svg-6JT6dHpE9jWQwQ7N svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-6JT6dHpE9jWQwQ7N p{margin:0;}#mermaid-svg-6JT6dHpE9jWQwQ7N .label{font-family:"trebuchet ms",verdana,arial,sans-serif;color:#333;}#mermaid-svg-6JT6dHpE9jWQwQ7N .cluster-label text{fill:#333;}#mermaid-svg-6JT6dHpE9jWQwQ7N .cluster-label span{color:#333;}#mermaid-svg-6JT6dHpE9jWQwQ7N .cluster-label span p{background-color:transparent;}#mermaid-svg-6JT6dHpE9jWQwQ7N .label text,#mermaid-svg-6JT6dHpE9jWQwQ7N span{fill:#333;color:#333;}#mermaid-svg-6JT6dHpE9jWQwQ7N .node rect,#mermaid-svg-6JT6dHpE9jWQwQ7N .node circle,#mermaid-svg-6JT6dHpE9jWQwQ7N .node ellipse,#mermaid-svg-6JT6dHpE9jWQwQ7N .node polygon,#mermaid-svg-6JT6dHpE9jWQwQ7N .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-6JT6dHpE9jWQwQ7N .rough-node .label text,#mermaid-svg-6JT6dHpE9jWQwQ7N .node .label text,#mermaid-svg-6JT6dHpE9jWQwQ7N .image-shape .label,#mermaid-svg-6JT6dHpE9jWQwQ7N .icon-shape .label{text-anchor:middle;}#mermaid-svg-6JT6dHpE9jWQwQ7N .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-6JT6dHpE9jWQwQ7N .rough-node .label,#mermaid-svg-6JT6dHpE9jWQwQ7N .node .label,#mermaid-svg-6JT6dHpE9jWQwQ7N .image-shape .label,#mermaid-svg-6JT6dHpE9jWQwQ7N .icon-shape .label{text-align:center;}#mermaid-svg-6JT6dHpE9jWQwQ7N .node.clickable{cursor:pointer;}#mermaid-svg-6JT6dHpE9jWQwQ7N .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-6JT6dHpE9jWQwQ7N .arrowheadPath{fill:#333333;}#mermaid-svg-6JT6dHpE9jWQwQ7N .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-6JT6dHpE9jWQwQ7N .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-6JT6dHpE9jWQwQ7N .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-6JT6dHpE9jWQwQ7N .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-6JT6dHpE9jWQwQ7N .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-6JT6dHpE9jWQwQ7N .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-6JT6dHpE9jWQwQ7N .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-6JT6dHpE9jWQwQ7N .cluster text{fill:#333;}#mermaid-svg-6JT6dHpE9jWQwQ7N .cluster span{color:#333;}#mermaid-svg-6JT6dHpE9jWQwQ7N div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-6JT6dHpE9jWQwQ7N .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-6JT6dHpE9jWQwQ7N rect.text{fill:none;stroke-width:0;}#mermaid-svg-6JT6dHpE9jWQwQ7N .icon-shape,#mermaid-svg-6JT6dHpE9jWQwQ7N .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-6JT6dHpE9jWQwQ7N .icon-shape p,#mermaid-svg-6JT6dHpE9jWQwQ7N .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-6JT6dHpE9jWQwQ7N .icon-shape .label rect,#mermaid-svg-6JT6dHpE9jWQwQ7N .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-6JT6dHpE9jWQwQ7N .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-6JT6dHpE9jWQwQ7N .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-6JT6dHpE9jWQwQ7N :root{--mermaid-font-family:"trebuchet ms",verdana,arial,sans-serif;} 最终效果
三类核心指标
三种方案
Micrometer 方案
2 行依赖 + 2 行配置
JMX Exporter 方案
加 agent 不改代码
① 堆内存占用率
防 OOM
② GC 暂停时间
防接口超时
③ 线程数
防线程泄漏
问题发生前提前发现 ✅
从猜变成看 ✅
- Micrometer 方案：2 行依赖 + 2 行配置 + Grafana 导入面板，Java 新项目的监控标配
- JMX Exporter 方案：不改代码，加个 agent 就搞定，老项目的救命稻草
这三类指标必须监控：
- 堆内存占用率（防 OOM）
- GC 暂停时间（防接口超时）
- 线程数（防线程泄漏）
那个 GC 卡顿查了三天的故事，后续是：
监控配好之后，团队发现------老年代占用率 85% 时就会触发一次 Full GC。他们简单调整了 JVM 参数，把堆内存从 2GB 升到 3GB，GC 停顿就消失了。如果有监控，这个问题早在两周前就会被发现和拦截。
这就是完美监控的意义------不是出了问题能快速定位，而是问题发生之前，你已经知道了。