01-06-A-JVM排查实战详解

此时不提桶，更待何时2026-09-22 16:10

01-06-A-JVM排查实战详解

📌 导读 ：阶段一的收官之作。前五篇讲了"JVM 怎么工作"（内存/对象/GC/类加载），本篇讲"出事了怎么办 "------CPU 100%、内存泄漏、频繁 GC、线程死锁、堆外泄漏、容器被杀，六大现场的完整排查 SOP。工具会过时（jstack → arthas），但"先定位现场类型 → 拿数据 → 顺引用链找根因"的方法论永不过时。

本文按六个维度组织：来源背景（排查方法论）→ 工具链 → 六大现场 SOP → 实战演练 → 对比 → 总结。全文约 5000 字（含大量命令与 SOP 清单），命令可直接复制执行。

📑 目录

  • 01-06-A-JVM排查实战详解
    • [📖 术语速查（先扫一眼，正文不迷路）](#📖 术语速查（先扫一眼，正文不迷路）)
    • 一、排查方法论：三原则与现场分诊
      • [1.1 三原则](#1.1 三原则)
      • [1.2 现场分诊表（30 秒定方向）](#1.2 现场分诊表（30 秒定方向）)
    • 二、工具链全景
      • [2.1 JDK 自带（生产机必有）](#2.1 JDK 自带（生产机必有）)
      • [2.2 Arthas（阿里开源，线上诊断首选）](#2.2 Arthas（阿里开源，线上诊断首选）)
      • [2.3 离线分析工具](#2.3 离线分析工具)
      • [2.4 生产标配的"事前准备"（出事前就要配好）](#2.4 生产标配的"事前准备"（出事前就要配好）)
    • [三、六大现场排查 SOP](#三、六大现场排查 SOP)
    • [四、实战演练：一次完整的 CPU 100% 排查](#四、实战演练：一次完整的 CPU 100% 排查)
    • 五、对比：工具选型
      • [5.1 jstack/jmap vs Arthas](#5.1 jstack/jmap vs Arthas)
      • [5.2 在线诊断 vs 离线 dump](#5.2 在线诊断 vs 离线 dump)
      • [5.3 排查思路的"三个层次"（与全系列方法论同构）](#5.3 排查思路的"三个层次"（与全系列方法论同构）)
    • 六、总结
      • [6.1 一张图回顾全文](#6.1 一张图回顾全文)
      • [6.2 核心要点浓缩（十条）](#6.2 核心要点浓缩（十条）)

📖 术语速查（先扫一眼，正文不迷路）

术语 白话解释
jps JDK 自带：列出本机所有 Java 进程及其 PID，排查第一步"先找到进程"
jstack 给 JVM 拍"线程快照"：每个线程此刻在跑哪行代码、什么状态、有没有死锁------CPU 高和卡顿排查的主力
jmap 看堆内存：histo 统计每类对象的数量/占用排行，dump 把整个堆导出成文件离线分析
jstat 实时看 GC/类加载的统计数字（每秒一行），不用重启不用 dump，生产最轻量
jinfo 查看/动态修改 JVM 参数（如临时打开 GC 日志）
jcmd JDK 8+ 的"瑞士军刀"，一个命令整合了上面所有功能（jcmd <pid> help 看全部子命令）
heap dump（堆转储） 把某一时刻整个堆的内容导出成 .hprof 文件------内存泄漏的"案发现场照片"，大堆 dump 会 STW 几十秒，慎拍
MAT Eclipse Memory Analyzer，分析 .hprof 的标配工具：自动找泄漏嫌疑、看支配树、查引用链
Dominator Tree（支配树） MAT 的核心视图：按"谁独占着最多内存"排序------找泄漏先看它，比按类统计更准
Path to GC Roots "这个对象为什么还活着"：从对象反查到 GC Roots 的完整引用链，泄漏定位的最后一公里
Leak Suspects MAT 的自动体检报告：一键分析出最可疑的泄漏点，新手友好
Arthas 阿里开源的在线诊断工具：不重启不改代码，直接看方法入参返回值（watch）、反编译线上类（jad）、动态 trace 耗时
trace/watch/stack Arthas 三板斧：trace 看方法内部每步耗时，watch 看入参出参异常，stack 看"这方法是谁调的"
火焰图 把 CPU 采样结果画成"火苗"：横轴越宽的方法越耗 CPU，一眼看出热点在哪（async-profiler 生成）
us / sy（top 命令） 进程 CPU 的两种成分：us 是业务代码烧的，sy 是内核态烧的（频繁系统调用/上下文切换）------比例本身就是线索
top -Hp top 按线程展示：先找到最耗 CPU 的线程 ID，再 printf '%x' 转十六进制去 jstack 里对号入座------CPU 100% 排查的经典三步
exit 137 容器被内核 OOM Kill 的退出码（128+9）：进程总内存超了容器限额，JVM 日志里往往干干净净------查容器内存而非堆
NMT（Native Memory Tracking） JVM 自带的"内存账本"：记录堆、元空间、线程栈、CodeCache 等各部分真实占用，堆外泄漏排查靠它（需启动参数开启，有 5%~10% 开销）
GC 日志 每次 GC 的流水账（时间/原因/前后内存/停顿时长），JDK 9+ 用 -Xlog:gc* 开启------频繁 GC 排查的第一手证据
TTSP Time To Safepoint：GC 停顿 = 等线程停 + 干活，"等线程停"过长（大循环、数大数组）时 GC 日志会显示异常
RT / QPS 响应时间 / 每秒请求数------排查的最终落点：所有 JVM 问题最后都体现为这两个业务指标的恶化
Runbook / SOP 标准排查手册：把"现象 → 命令 → 判断分支"固化成流程，出事时照着走，不靠临场发挥

一、排查方法论：三原则与现场分诊

1.1 三原则

#mermaid-svg-vr5EFo7jS7KvNSB6{font-family:Microsoft YaHei;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-vr5EFo7jS7KvNSB6 .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-vr5EFo7jS7KvNSB6 .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-vr5EFo7jS7KvNSB6 .error-icon{fill:hsl(220.5882352941, 100%, 98.3333333333%);}#mermaid-svg-vr5EFo7jS7KvNSB6 .error-text{fill:rgb(8.5000000002, 5.7500000001, 0);stroke:rgb(8.5000000002, 5.7500000001, 0);}#mermaid-svg-vr5EFo7jS7KvNSB6 .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-vr5EFo7jS7KvNSB6 .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-vr5EFo7jS7KvNSB6 .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-vr5EFo7jS7KvNSB6 .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-vr5EFo7jS7KvNSB6 .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-vr5EFo7jS7KvNSB6 .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-vr5EFo7jS7KvNSB6 .marker{fill:#0b0b0b;stroke:#0b0b0b;}#mermaid-svg-vr5EFo7jS7KvNSB6 .marker.cross{stroke:#0b0b0b;}#mermaid-svg-vr5EFo7jS7KvNSB6 svg{font-family:Microsoft YaHei;font-size:16px;}#mermaid-svg-vr5EFo7jS7KvNSB6 p{margin:0;}#mermaid-svg-vr5EFo7jS7KvNSB6 .label{font-family:Microsoft YaHei;color:#333;}#mermaid-svg-vr5EFo7jS7KvNSB6 .cluster-label text{fill:rgb(8.5000000002, 5.7500000001, 0);}#mermaid-svg-vr5EFo7jS7KvNSB6 .cluster-label span{color:rgb(8.5000000002, 5.7500000001, 0);}#mermaid-svg-vr5EFo7jS7KvNSB6 .cluster-label span p{background-color:transparent;}#mermaid-svg-vr5EFo7jS7KvNSB6 .label text,#mermaid-svg-vr5EFo7jS7KvNSB6 span{fill:#333;color:#333;}#mermaid-svg-vr5EFo7jS7KvNSB6 .node rect,#mermaid-svg-vr5EFo7jS7KvNSB6 .node circle,#mermaid-svg-vr5EFo7jS7KvNSB6 .node ellipse,#mermaid-svg-vr5EFo7jS7KvNSB6 .node polygon,#mermaid-svg-vr5EFo7jS7KvNSB6 .node path{fill:#fff4dd;stroke:hsl(40.5882352941, 60%, 83.3333333333%);stroke-width:1px;}#mermaid-svg-vr5EFo7jS7KvNSB6 .rough-node .label text,#mermaid-svg-vr5EFo7jS7KvNSB6 .node .label text,#mermaid-svg-vr5EFo7jS7KvNSB6 .image-shape .label,#mermaid-svg-vr5EFo7jS7KvNSB6 .icon-shape .label{text-anchor:middle;}#mermaid-svg-vr5EFo7jS7KvNSB6 .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-vr5EFo7jS7KvNSB6 .rough-node .label,#mermaid-svg-vr5EFo7jS7KvNSB6 .node .label,#mermaid-svg-vr5EFo7jS7KvNSB6 .image-shape .label,#mermaid-svg-vr5EFo7jS7KvNSB6 .icon-shape .label{text-align:center;}#mermaid-svg-vr5EFo7jS7KvNSB6 .node.clickable{cursor:pointer;}#mermaid-svg-vr5EFo7jS7KvNSB6 .root .anchor path{fill:#0b0b0b!important;stroke-width:0;stroke:#0b0b0b;}#mermaid-svg-vr5EFo7jS7KvNSB6 .arrowheadPath{fill:#0b0b0b;}#mermaid-svg-vr5EFo7jS7KvNSB6 .edgePath .path{stroke:#0b0b0b;stroke-width:2.0px;}#mermaid-svg-vr5EFo7jS7KvNSB6 .flowchart-link{stroke:#0b0b0b;fill:none;}#mermaid-svg-vr5EFo7jS7KvNSB6 .edgeLabel{background-color:hsl(-79.4117647059, 100%, 93.3333333333%);text-align:center;}#mermaid-svg-vr5EFo7jS7KvNSB6 .edgeLabel p{background-color:hsl(-79.4117647059, 100%, 93.3333333333%);}#mermaid-svg-vr5EFo7jS7KvNSB6 .edgeLabel rect{opacity:0.5;background-color:hsl(-79.4117647059, 100%, 93.3333333333%);fill:hsl(-79.4117647059, 100%, 93.3333333333%);}#mermaid-svg-vr5EFo7jS7KvNSB6 .labelBkg{background-color:rgba(243.9999999999, 220.9999999998, 255, 0.5);}#mermaid-svg-vr5EFo7jS7KvNSB6 .cluster rect{fill:hsl(220.5882352941, 100%, 98.3333333333%);stroke:hsl(220.5882352941, 60%, 88.3333333333%);stroke-width:1px;}#mermaid-svg-vr5EFo7jS7KvNSB6 .cluster text{fill:rgb(8.5000000002, 5.7500000001, 0);}#mermaid-svg-vr5EFo7jS7KvNSB6 .cluster span{color:rgb(8.5000000002, 5.7500000001, 0);}#mermaid-svg-vr5EFo7jS7KvNSB6 div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:Microsoft YaHei;font-size:12px;background:hsl(220.5882352941, 100%, 98.3333333333%);border:1px solid hsl(220.5882352941, 60%, 88.3333333333%);border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-vr5EFo7jS7KvNSB6 .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-vr5EFo7jS7KvNSB6 rect.text{fill:none;stroke-width:0;}#mermaid-svg-vr5EFo7jS7KvNSB6 .icon-shape,#mermaid-svg-vr5EFo7jS7KvNSB6 .image-shape{background-color:hsl(-79.4117647059, 100%, 93.3333333333%);text-align:center;}#mermaid-svg-vr5EFo7jS7KvNSB6 .icon-shape p,#mermaid-svg-vr5EFo7jS7KvNSB6 .image-shape p{background-color:hsl(-79.4117647059, 100%, 93.3333333333%);padding:2px;}#mermaid-svg-vr5EFo7jS7KvNSB6 .icon-shape .label rect,#mermaid-svg-vr5EFo7jS7KvNSB6 .image-shape .label rect{opacity:0.5;background-color:hsl(-79.4117647059, 100%, 93.3333333333%);fill:hsl(-79.4117647059, 100%, 93.3333333333%);}#mermaid-svg-vr5EFo7jS7KvNSB6 .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-vr5EFo7jS7KvNSB6 .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-vr5EFo7jS7KvNSB6 :root{--mermaid-font-family:"trebuchet ms",verdana,arial,sans-serif;} 排查三原则
① 先保现场再动手
重启前必抓
线程 dump + 堆 dump + GC 日志
重启 = 毁尸灭迹
重启是最后手段

② 先分诊后深入
30 秒定现场类型
CPU? 内存? GC? 线程? 容器?
别一头扎进细节

③ 数据说话
每个结论都要有命令输出佐证
猜出来的根因 = 没根因

方法论
定位现场类型
→ 拿数据
→ 顺引用链找根因
工具会过时
方法论永不过时

1.2 现场分诊表（30 秒定方向）

#mermaid-svg-KeUGZL6jt2JbsesC{font-family:Microsoft YaHei;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-KeUGZL6jt2JbsesC .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-KeUGZL6jt2JbsesC .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-KeUGZL6jt2JbsesC .error-icon{fill:hsl(220.5882352941, 100%, 98.3333333333%);}#mermaid-svg-KeUGZL6jt2JbsesC .error-text{fill:rgb(8.5000000002, 5.7500000001, 0);stroke:rgb(8.5000000002, 5.7500000001, 0);}#mermaid-svg-KeUGZL6jt2JbsesC .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-KeUGZL6jt2JbsesC .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-KeUGZL6jt2JbsesC .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-KeUGZL6jt2JbsesC .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-KeUGZL6jt2JbsesC .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-KeUGZL6jt2JbsesC .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-KeUGZL6jt2JbsesC .marker{fill:#0b0b0b;stroke:#0b0b0b;}#mermaid-svg-KeUGZL6jt2JbsesC .marker.cross{stroke:#0b0b0b;}#mermaid-svg-KeUGZL6jt2JbsesC svg{font-family:Microsoft YaHei;font-size:16px;}#mermaid-svg-KeUGZL6jt2JbsesC p{margin:0;}#mermaid-svg-KeUGZL6jt2JbsesC .label{font-family:Microsoft YaHei;color:#333;}#mermaid-svg-KeUGZL6jt2JbsesC .cluster-label text{fill:rgb(8.5000000002, 5.7500000001, 0);}#mermaid-svg-KeUGZL6jt2JbsesC .cluster-label span{color:rgb(8.5000000002, 5.7500000001, 0);}#mermaid-svg-KeUGZL6jt2JbsesC .cluster-label span p{background-color:transparent;}#mermaid-svg-KeUGZL6jt2JbsesC .label text,#mermaid-svg-KeUGZL6jt2JbsesC span{fill:#333;color:#333;}#mermaid-svg-KeUGZL6jt2JbsesC .node rect,#mermaid-svg-KeUGZL6jt2JbsesC .node circle,#mermaid-svg-KeUGZL6jt2JbsesC .node ellipse,#mermaid-svg-KeUGZL6jt2JbsesC .node polygon,#mermaid-svg-KeUGZL6jt2JbsesC .node path{fill:#fff4dd;stroke:hsl(40.5882352941, 60%, 83.3333333333%);stroke-width:1px;}#mermaid-svg-KeUGZL6jt2JbsesC .rough-node .label text,#mermaid-svg-KeUGZL6jt2JbsesC .node .label text,#mermaid-svg-KeUGZL6jt2JbsesC .image-shape .label,#mermaid-svg-KeUGZL6jt2JbsesC .icon-shape .label{text-anchor:middle;}#mermaid-svg-KeUGZL6jt2JbsesC .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-KeUGZL6jt2JbsesC .rough-node .label,#mermaid-svg-KeUGZL6jt2JbsesC .node .label,#mermaid-svg-KeUGZL6jt2JbsesC .image-shape .label,#mermaid-svg-KeUGZL6jt2JbsesC .icon-shape .label{text-align:center;}#mermaid-svg-KeUGZL6jt2JbsesC .node.clickable{cursor:pointer;}#mermaid-svg-KeUGZL6jt2JbsesC .root .anchor path{fill:#0b0b0b!important;stroke-width:0;stroke:#0b0b0b;}#mermaid-svg-KeUGZL6jt2JbsesC .arrowheadPath{fill:#0b0b0b;}#mermaid-svg-KeUGZL6jt2JbsesC .edgePath .path{stroke:#0b0b0b;stroke-width:2.0px;}#mermaid-svg-KeUGZL6jt2JbsesC .flowchart-link{stroke:#0b0b0b;fill:none;}#mermaid-svg-KeUGZL6jt2JbsesC .edgeLabel{background-color:hsl(-79.4117647059, 100%, 93.3333333333%);text-align:center;}#mermaid-svg-KeUGZL6jt2JbsesC .edgeLabel p{background-color:hsl(-79.4117647059, 100%, 93.3333333333%);}#mermaid-svg-KeUGZL6jt2JbsesC .edgeLabel rect{opacity:0.5;background-color:hsl(-79.4117647059, 100%, 93.3333333333%);fill:hsl(-79.4117647059, 100%, 93.3333333333%);}#mermaid-svg-KeUGZL6jt2JbsesC .labelBkg{background-color:rgba(243.9999999999, 220.9999999998, 255, 0.5);}#mermaid-svg-KeUGZL6jt2JbsesC .cluster rect{fill:hsl(220.5882352941, 100%, 98.3333333333%);stroke:hsl(220.5882352941, 60%, 88.3333333333%);stroke-width:1px;}#mermaid-svg-KeUGZL6jt2JbsesC .cluster text{fill:rgb(8.5000000002, 5.7500000001, 0);}#mermaid-svg-KeUGZL6jt2JbsesC .cluster span{color:rgb(8.5000000002, 5.7500000001, 0);}#mermaid-svg-KeUGZL6jt2JbsesC div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:Microsoft YaHei;font-size:12px;background:hsl(220.5882352941, 100%, 98.3333333333%);border:1px solid hsl(220.5882352941, 60%, 88.3333333333%);border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-KeUGZL6jt2JbsesC .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-KeUGZL6jt2JbsesC rect.text{fill:none;stroke-width:0;}#mermaid-svg-KeUGZL6jt2JbsesC .icon-shape,#mermaid-svg-KeUGZL6jt2JbsesC .image-shape{background-color:hsl(-79.4117647059, 100%, 93.3333333333%);text-align:center;}#mermaid-svg-KeUGZL6jt2JbsesC .icon-shape p,#mermaid-svg-KeUGZL6jt2JbsesC .image-shape p{background-color:hsl(-79.4117647059, 100%, 93.3333333333%);padding:2px;}#mermaid-svg-KeUGZL6jt2JbsesC .icon-shape .label rect,#mermaid-svg-KeUGZL6jt2JbsesC .image-shape .label rect{opacity:0.5;background-color:hsl(-79.4117647059, 100%, 93.3333333333%);fill:hsl(-79.4117647059, 100%, 93.3333333333%);}#mermaid-svg-KeUGZL6jt2JbsesC .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-KeUGZL6jt2JbsesC .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-KeUGZL6jt2JbsesC :root{--mermaid-font-family:"trebuchet ms",verdana,arial,sans-serif;} CPU 高
us 高
sy 高
CPU 正常
是
BLOCKED 多
WAITING 外部
进程没了
137
143 / 其他
Full GC 频繁
堆持续爬升
服务异常
top 看 CPU
us 还是 sy?
业务代码死循环
/ GC 线程狂转
→ SOP-1

系统调用
/ 上下文切换
/ 锁竞争
→ SOP-1 变体

响应慢?
线程都卡在哪?
锁竞争 / 死锁
→ SOP-4

下游 RPC / DB 慢
→ 出 JVM 排查

exit code?
容器 OOMKilled
→ SOP-5/6

K8s 驱逐
/ 健康检查失败

GC 日志异常?
→ SOP-3
→ SOP-2

二、工具链全景

2.1 JDK 自带（生产机必有）

工具 用途 高频命令
jps 找 Java 进程 PID jps -lvm
jstack 线程 dump（死锁/卡顿） jstack <pid>；卡死进程加 -F；连续抓 3 次对比
jmap 堆 dump / 直方图 jmap -histo:live <pid>（触发 Full GC，慎用）；jmap -dump:live,format=b,file=heap.hprof <pid>
jstat GC/类加载实时统计 jstat -gcutil <pid> 1000（每秒一行，看 E/O/YGC/FGC）
jcmd 万能瑞士军刀（推荐替代 jmap/jstack） jcmd <pid> help；GC.heap_dump、Thread.print、VM.native_memory、VM.classloader_stats（类统计；老命令 GC.class_stats 仅 JDK 8 且需 UnlockDiagnosticVMOptions，11+ 已移除）
jinfo 查看/修改 JVM 参数 jinfo -flags <pid>

2.2 Arthas（阿里开源，线上诊断首选）

bash 复制代码 
curl -O https://arthas.aliyun.com/arthas-boot.jar && java -jar arthas-boot.jar

dashboard          # 总览：线程/内存/GC 一屏
thread -n 3        # 最忙的 3 个线程栈（CPU 高第一命令！）
thread -b          # 直接找出阻塞其他线程的元凶（死锁排查神器）
sc -d com.yc.Order # 类从哪个 jar 加载、哪个加载器（类冲突排查）
watch com.yc.OrderService create '{params, returnObj, throwExp}' -x 2   # 方法出入参观测
trace com.yc.OrderService create    # 方法内部调用链耗时（慢在哪一步）
stack com.yc.OrderService create    # 谁在调用这个方法
profiler start / profiler stop --format html   # 火焰图（CPU/内存热点）
retransform /tmp/OrderService.class # 热替换字节码（紧急止血，事后必须补发布）

2.3 离线分析工具

工具 用途
MAT（Eclipse Memory Analyzer） heap dump 分析之王：dominator tree（谁占内存最大）、Path to GC Roots（为什么没被回收）、Leak Suspects 自动报告
GCEasy / GCViewer GC 日志可视化：停顿分布、频率、回收效率
async-profiler 低开销火焰图（CPU/alloc/lock 三种模式），Arthas profiler 的底层
JOL 对象内存布局查看
NMT（Native Memory Tracking） JVM 自身本地内存分类账：-XX:NativeMemoryTracking=detail + jcmd VM.native_memory summary

2.4 生产标配的"事前准备"（出事前就要配好）

#mermaid-svg-O1ssgXh1qYU1FolV{font-family:Microsoft YaHei;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-O1ssgXh1qYU1FolV .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-O1ssgXh1qYU1FolV .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-O1ssgXh1qYU1FolV .error-icon{fill:hsl(220.5882352941, 100%, 98.3333333333%);}#mermaid-svg-O1ssgXh1qYU1FolV .error-text{fill:rgb(8.5000000002, 5.7500000001, 0);stroke:rgb(8.5000000002, 5.7500000001, 0);}#mermaid-svg-O1ssgXh1qYU1FolV .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-O1ssgXh1qYU1FolV .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-O1ssgXh1qYU1FolV .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-O1ssgXh1qYU1FolV .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-O1ssgXh1qYU1FolV .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-O1ssgXh1qYU1FolV .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-O1ssgXh1qYU1FolV .marker{fill:#0b0b0b;stroke:#0b0b0b;}#mermaid-svg-O1ssgXh1qYU1FolV .marker.cross{stroke:#0b0b0b;}#mermaid-svg-O1ssgXh1qYU1FolV svg{font-family:Microsoft YaHei;font-size:16px;}#mermaid-svg-O1ssgXh1qYU1FolV p{margin:0;}#mermaid-svg-O1ssgXh1qYU1FolV .label{font-family:Microsoft YaHei;color:#333;}#mermaid-svg-O1ssgXh1qYU1FolV .cluster-label text{fill:rgb(8.5000000002, 5.7500000001, 0);}#mermaid-svg-O1ssgXh1qYU1FolV .cluster-label span{color:rgb(8.5000000002, 5.7500000001, 0);}#mermaid-svg-O1ssgXh1qYU1FolV .cluster-label span p{background-color:transparent;}#mermaid-svg-O1ssgXh1qYU1FolV .label text,#mermaid-svg-O1ssgXh1qYU1FolV span{fill:#333;color:#333;}#mermaid-svg-O1ssgXh1qYU1FolV .node rect,#mermaid-svg-O1ssgXh1qYU1FolV .node circle,#mermaid-svg-O1ssgXh1qYU1FolV .node ellipse,#mermaid-svg-O1ssgXh1qYU1FolV .node polygon,#mermaid-svg-O1ssgXh1qYU1FolV .node path{fill:#fff4dd;stroke:hsl(40.5882352941, 60%, 83.3333333333%);stroke-width:1px;}#mermaid-svg-O1ssgXh1qYU1FolV .rough-node .label text,#mermaid-svg-O1ssgXh1qYU1FolV .node .label text,#mermaid-svg-O1ssgXh1qYU1FolV .image-shape .label,#mermaid-svg-O1ssgXh1qYU1FolV .icon-shape .label{text-anchor:middle;}#mermaid-svg-O1ssgXh1qYU1FolV .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-O1ssgXh1qYU1FolV .rough-node .label,#mermaid-svg-O1ssgXh1qYU1FolV .node .label,#mermaid-svg-O1ssgXh1qYU1FolV .image-shape .label,#mermaid-svg-O1ssgXh1qYU1FolV .icon-shape .label{text-align:center;}#mermaid-svg-O1ssgXh1qYU1FolV .node.clickable{cursor:pointer;}#mermaid-svg-O1ssgXh1qYU1FolV .root .anchor path{fill:#0b0b0b!important;stroke-width:0;stroke:#0b0b0b;}#mermaid-svg-O1ssgXh1qYU1FolV .arrowheadPath{fill:#0b0b0b;}#mermaid-svg-O1ssgXh1qYU1FolV .edgePath .path{stroke:#0b0b0b;stroke-width:2.0px;}#mermaid-svg-O1ssgXh1qYU1FolV .flowchart-link{stroke:#0b0b0b;fill:none;}#mermaid-svg-O1ssgXh1qYU1FolV .edgeLabel{background-color:hsl(-79.4117647059, 100%, 93.3333333333%);text-align:center;}#mermaid-svg-O1ssgXh1qYU1FolV .edgeLabel p{background-color:hsl(-79.4117647059, 100%, 93.3333333333%);}#mermaid-svg-O1ssgXh1qYU1FolV .edgeLabel rect{opacity:0.5;background-color:hsl(-79.4117647059, 100%, 93.3333333333%);fill:hsl(-79.4117647059, 100%, 93.3333333333%);}#mermaid-svg-O1ssgXh1qYU1FolV .labelBkg{background-color:rgba(243.9999999999, 220.9999999998, 255, 0.5);}#mermaid-svg-O1ssgXh1qYU1FolV .cluster rect{fill:hsl(220.5882352941, 100%, 98.3333333333%);stroke:hsl(220.5882352941, 60%, 88.3333333333%);stroke-width:1px;}#mermaid-svg-O1ssgXh1qYU1FolV .cluster text{fill:rgb(8.5000000002, 5.7500000001, 0);}#mermaid-svg-O1ssgXh1qYU1FolV .cluster span{color:rgb(8.5000000002, 5.7500000001, 0);}#mermaid-svg-O1ssgXh1qYU1FolV div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:Microsoft YaHei;font-size:12px;background:hsl(220.5882352941, 100%, 98.3333333333%);border:1px solid hsl(220.5882352941, 60%, 88.3333333333%);border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-O1ssgXh1qYU1FolV .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-O1ssgXh1qYU1FolV rect.text{fill:none;stroke-width:0;}#mermaid-svg-O1ssgXh1qYU1FolV .icon-shape,#mermaid-svg-O1ssgXh1qYU1FolV .image-shape{background-color:hsl(-79.4117647059, 100%, 93.3333333333%);text-align:center;}#mermaid-svg-O1ssgXh1qYU1FolV .icon-shape p,#mermaid-svg-O1ssgXh1qYU1FolV .image-shape p{background-color:hsl(-79.4117647059, 100%, 93.3333333333%);padding:2px;}#mermaid-svg-O1ssgXh1qYU1FolV .icon-shape .label rect,#mermaid-svg-O1ssgXh1qYU1FolV .image-shape .label rect{opacity:0.5;background-color:hsl(-79.4117647059, 100%, 93.3333333333%);fill:hsl(-79.4117647059, 100%, 93.3333333333%);}#mermaid-svg-O1ssgXh1qYU1FolV .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-O1ssgXh1qYU1FolV .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-O1ssgXh1qYU1FolV :root{--mermaid-font-family:"trebuchet ms",verdana,arial,sans-serif;}
事前准备四件套
事故来临时
没有第二次配置机会

① GC 日志常开
-Xlog:gc*:file=...
:filecount=5,filesize=50m
滚动覆盖
不占满磁盘

② OOM 自动 dump
-XX:+HeapDumpOnOutOfMemoryError
-XX:HeapDumpPath=/data/dump
死无对证
不如留全尸

③ 监控四件套
堆使用率 / Full GC 频率
线程数 / RSS 与堆差值
水位告警
堆 75% · Full GC 频率
· classes_loaded

④ 排查工具就位
Arthas jar · async-profiler
MAT 本地装好
NMT 按需开启
dmesg 权限确认

三、六大现场排查 SOP

SOP-1：CPU 100%

#mermaid-svg-4p4QgcsXxXj5G5uw{font-family:Microsoft YaHei;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-4p4QgcsXxXj5G5uw .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-4p4QgcsXxXj5G5uw .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-4p4QgcsXxXj5G5uw .error-icon{fill:hsl(220.5882352941, 100%, 98.3333333333%);}#mermaid-svg-4p4QgcsXxXj5G5uw .error-text{fill:rgb(8.5000000002, 5.7500000001, 0);stroke:rgb(8.5000000002, 5.7500000001, 0);}#mermaid-svg-4p4QgcsXxXj5G5uw .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-4p4QgcsXxXj5G5uw .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-4p4QgcsXxXj5G5uw .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-4p4QgcsXxXj5G5uw .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-4p4QgcsXxXj5G5uw .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-4p4QgcsXxXj5G5uw .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-4p4QgcsXxXj5G5uw .marker{fill:#0b0b0b;stroke:#0b0b0b;}#mermaid-svg-4p4QgcsXxXj5G5uw .marker.cross{stroke:#0b0b0b;}#mermaid-svg-4p4QgcsXxXj5G5uw svg{font-family:Microsoft YaHei;font-size:16px;}#mermaid-svg-4p4QgcsXxXj5G5uw p{margin:0;}#mermaid-svg-4p4QgcsXxXj5G5uw .label{font-family:Microsoft YaHei;color:#333;}#mermaid-svg-4p4QgcsXxXj5G5uw .cluster-label text{fill:rgb(8.5000000002, 5.7500000001, 0);}#mermaid-svg-4p4QgcsXxXj5G5uw .cluster-label span{color:rgb(8.5000000002, 5.7500000001, 0);}#mermaid-svg-4p4QgcsXxXj5G5uw .cluster-label span p{background-color:transparent;}#mermaid-svg-4p4QgcsXxXj5G5uw .label text,#mermaid-svg-4p4QgcsXxXj5G5uw span{fill:#333;color:#333;}#mermaid-svg-4p4QgcsXxXj5G5uw .node rect,#mermaid-svg-4p4QgcsXxXj5G5uw .node circle,#mermaid-svg-4p4QgcsXxXj5G5uw .node ellipse,#mermaid-svg-4p4QgcsXxXj5G5uw .node polygon,#mermaid-svg-4p4QgcsXxXj5G5uw .node path{fill:#fff4dd;stroke:hsl(40.5882352941, 60%, 83.3333333333%);stroke-width:1px;}#mermaid-svg-4p4QgcsXxXj5G5uw .rough-node .label text,#mermaid-svg-4p4QgcsXxXj5G5uw .node .label text,#mermaid-svg-4p4QgcsXxXj5G5uw .image-shape .label,#mermaid-svg-4p4QgcsXxXj5G5uw .icon-shape .label{text-anchor:middle;}#mermaid-svg-4p4QgcsXxXj5G5uw .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-4p4QgcsXxXj5G5uw .rough-node .label,#mermaid-svg-4p4QgcsXxXj5G5uw .node .label,#mermaid-svg-4p4QgcsXxXj5G5uw .image-shape .label,#mermaid-svg-4p4QgcsXxXj5G5uw .icon-shape .label{text-align:center;}#mermaid-svg-4p4QgcsXxXj5G5uw .node.clickable{cursor:pointer;}#mermaid-svg-4p4QgcsXxXj5G5uw .root .anchor path{fill:#0b0b0b!important;stroke-width:0;stroke:#0b0b0b;}#mermaid-svg-4p4QgcsXxXj5G5uw .arrowheadPath{fill:#0b0b0b;}#mermaid-svg-4p4QgcsXxXj5G5uw .edgePath .path{stroke:#0b0b0b;stroke-width:2.0px;}#mermaid-svg-4p4QgcsXxXj5G5uw .flowchart-link{stroke:#0b0b0b;fill:none;}#mermaid-svg-4p4QgcsXxXj5G5uw .edgeLabel{background-color:hsl(-79.4117647059, 100%, 93.3333333333%);text-align:center;}#mermaid-svg-4p4QgcsXxXj5G5uw .edgeLabel p{background-color:hsl(-79.4117647059, 100%, 93.3333333333%);}#mermaid-svg-4p4QgcsXxXj5G5uw .edgeLabel rect{opacity:0.5;background-color:hsl(-79.4117647059, 100%, 93.3333333333%);fill:hsl(-79.4117647059, 100%, 93.3333333333%);}#mermaid-svg-4p4QgcsXxXj5G5uw .labelBkg{background-color:rgba(243.9999999999, 220.9999999998, 255, 0.5);}#mermaid-svg-4p4QgcsXxXj5G5uw .cluster rect{fill:hsl(220.5882352941, 100%, 98.3333333333%);stroke:hsl(220.5882352941, 60%, 88.3333333333%);stroke-width:1px;}#mermaid-svg-4p4QgcsXxXj5G5uw .cluster text{fill:rgb(8.5000000002, 5.7500000001, 0);}#mermaid-svg-4p4QgcsXxXj5G5uw .cluster span{color:rgb(8.5000000002, 5.7500000001, 0);}#mermaid-svg-4p4QgcsXxXj5G5uw div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:Microsoft YaHei;font-size:12px;background:hsl(220.5882352941, 100%, 98.3333333333%);border:1px solid hsl(220.5882352941, 60%, 88.3333333333%);border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-4p4QgcsXxXj5G5uw .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-4p4QgcsXxXj5G5uw rect.text{fill:none;stroke-width:0;}#mermaid-svg-4p4QgcsXxXj5G5uw .icon-shape,#mermaid-svg-4p4QgcsXxXj5G5uw .image-shape{background-color:hsl(-79.4117647059, 100%, 93.3333333333%);text-align:center;}#mermaid-svg-4p4QgcsXxXj5G5uw .icon-shape p,#mermaid-svg-4p4QgcsXxXj5G5uw .image-shape p{background-color:hsl(-79.4117647059, 100%, 93.3333333333%);padding:2px;}#mermaid-svg-4p4QgcsXxXj5G5uw .icon-shape .label rect,#mermaid-svg-4p4QgcsXxXj5G5uw .image-shape .label rect{opacity:0.5;background-color:hsl(-79.4117647059, 100%, 93.3333333333%);fill:hsl(-79.4117647059, 100%, 93.3333333333%);}#mermaid-svg-4p4QgcsXxXj5G5uw .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-4p4QgcsXxXj5G5uw .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-4p4QgcsXxXj5G5uw :root{--mermaid-font-family:"trebuchet ms",verdana,arial,sans-serif;} 业务方法
GC 线程 GC task thread
JIT 编译线程 CompilerThread
有 Arthas 更快
现象：CPU 100%
① top 看进程级 CPU
us 高 = 业务代码 / GC 线程狂转
sy 高 = 系统调用 / 上下文切换

② top -Hp pid 按线程展示
找到最耗 CPU 的线程 ID
十进制

③ printf '%x' 线程 ID
转十六进制

④ jstack pid 里搜
nid=0x 线程 ID
对号入座
看这个线程在跑哪行代码
连续抓 3 次对比
确认不是路过

栈指纹判断
死循环 / 正则回溯
/ 大计算
→ 定位到代码行修复

CPU 是 GC 烧的
不是业务
→ 转 SOP-3

正常预热
观察是否自行回落

thread -n 3 一步到位
直接给出最忙 3 个线程的栈

经典案例指纹 ：正则灾难性回溯（栈里全是 java.util.regex.Pattern$xxx.match）、HashMap 并发成环（栈里全是 HashMap.get，JDK 7，联动《集合框架源码面试与生产事故实战》事故一）、无限递归（栈里同一方法重复出现直到 StackOverflow 边缘）。

SOP-2：内存泄漏（堆持续爬升）

#mermaid-svg-vjQx8mtXxyiB4tqs{font-family:Microsoft YaHei;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-vjQx8mtXxyiB4tqs .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-vjQx8mtXxyiB4tqs .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-vjQx8mtXxyiB4tqs .error-icon{fill:hsl(220.5882352941, 100%, 98.3333333333%);}#mermaid-svg-vjQx8mtXxyiB4tqs .error-text{fill:rgb(8.5000000002, 5.7500000001, 0);stroke:rgb(8.5000000002, 5.7500000001, 0);}#mermaid-svg-vjQx8mtXxyiB4tqs .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-vjQx8mtXxyiB4tqs .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-vjQx8mtXxyiB4tqs .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-vjQx8mtXxyiB4tqs .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-vjQx8mtXxyiB4tqs .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-vjQx8mtXxyiB4tqs .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-vjQx8mtXxyiB4tqs .marker{fill:#0b0b0b;stroke:#0b0b0b;}#mermaid-svg-vjQx8mtXxyiB4tqs .marker.cross{stroke:#0b0b0b;}#mermaid-svg-vjQx8mtXxyiB4tqs svg{font-family:Microsoft YaHei;font-size:16px;}#mermaid-svg-vjQx8mtXxyiB4tqs p{margin:0;}#mermaid-svg-vjQx8mtXxyiB4tqs .label{font-family:Microsoft YaHei;color:#333;}#mermaid-svg-vjQx8mtXxyiB4tqs .cluster-label text{fill:rgb(8.5000000002, 5.7500000001, 0);}#mermaid-svg-vjQx8mtXxyiB4tqs .cluster-label span{color:rgb(8.5000000002, 5.7500000001, 0);}#mermaid-svg-vjQx8mtXxyiB4tqs .cluster-label span p{background-color:transparent;}#mermaid-svg-vjQx8mtXxyiB4tqs .label text,#mermaid-svg-vjQx8mtXxyiB4tqs span{fill:#333;color:#333;}#mermaid-svg-vjQx8mtXxyiB4tqs .node rect,#mermaid-svg-vjQx8mtXxyiB4tqs .node circle,#mermaid-svg-vjQx8mtXxyiB4tqs .node ellipse,#mermaid-svg-vjQx8mtXxyiB4tqs .node polygon,#mermaid-svg-vjQx8mtXxyiB4tqs .node path{fill:#fff4dd;stroke:hsl(40.5882352941, 60%, 83.3333333333%);stroke-width:1px;}#mermaid-svg-vjQx8mtXxyiB4tqs .rough-node .label text,#mermaid-svg-vjQx8mtXxyiB4tqs .node .label text,#mermaid-svg-vjQx8mtXxyiB4tqs .image-shape .label,#mermaid-svg-vjQx8mtXxyiB4tqs .icon-shape .label{text-anchor:middle;}#mermaid-svg-vjQx8mtXxyiB4tqs .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-vjQx8mtXxyiB4tqs .rough-node .label,#mermaid-svg-vjQx8mtXxyiB4tqs .node .label,#mermaid-svg-vjQx8mtXxyiB4tqs .image-shape .label,#mermaid-svg-vjQx8mtXxyiB4tqs .icon-shape .label{text-align:center;}#mermaid-svg-vjQx8mtXxyiB4tqs .node.clickable{cursor:pointer;}#mermaid-svg-vjQx8mtXxyiB4tqs .root .anchor path{fill:#0b0b0b!important;stroke-width:0;stroke:#0b0b0b;}#mermaid-svg-vjQx8mtXxyiB4tqs .arrowheadPath{fill:#0b0b0b;}#mermaid-svg-vjQx8mtXxyiB4tqs .edgePath .path{stroke:#0b0b0b;stroke-width:2.0px;}#mermaid-svg-vjQx8mtXxyiB4tqs .flowchart-link{stroke:#0b0b0b;fill:none;}#mermaid-svg-vjQx8mtXxyiB4tqs .edgeLabel{background-color:hsl(-79.4117647059, 100%, 93.3333333333%);text-align:center;}#mermaid-svg-vjQx8mtXxyiB4tqs .edgeLabel p{background-color:hsl(-79.4117647059, 100%, 93.3333333333%);}#mermaid-svg-vjQx8mtXxyiB4tqs .edgeLabel rect{opacity:0.5;background-color:hsl(-79.4117647059, 100%, 93.3333333333%);fill:hsl(-79.4117647059, 100%, 93.3333333333%);}#mermaid-svg-vjQx8mtXxyiB4tqs .labelBkg{background-color:rgba(243.9999999999, 220.9999999998, 255, 0.5);}#mermaid-svg-vjQx8mtXxyiB4tqs .cluster rect{fill:hsl(220.5882352941, 100%, 98.3333333333%);stroke:hsl(220.5882352941, 60%, 88.3333333333%);stroke-width:1px;}#mermaid-svg-vjQx8mtXxyiB4tqs .cluster text{fill:rgb(8.5000000002, 5.7500000001, 0);}#mermaid-svg-vjQx8mtXxyiB4tqs .cluster span{color:rgb(8.5000000002, 5.7500000001, 0);}#mermaid-svg-vjQx8mtXxyiB4tqs div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:Microsoft YaHei;font-size:12px;background:hsl(220.5882352941, 100%, 98.3333333333%);border:1px solid hsl(220.5882352941, 60%, 88.3333333333%);border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-vjQx8mtXxyiB4tqs .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-vjQx8mtXxyiB4tqs rect.text{fill:none;stroke-width:0;}#mermaid-svg-vjQx8mtXxyiB4tqs .icon-shape,#mermaid-svg-vjQx8mtXxyiB4tqs .image-shape{background-color:hsl(-79.4117647059, 100%, 93.3333333333%);text-align:center;}#mermaid-svg-vjQx8mtXxyiB4tqs .icon-shape p,#mermaid-svg-vjQx8mtXxyiB4tqs .image-shape p{background-color:hsl(-79.4117647059, 100%, 93.3333333333%);padding:2px;}#mermaid-svg-vjQx8mtXxyiB4tqs .icon-shape .label rect,#mermaid-svg-vjQx8mtXxyiB4tqs .image-shape .label rect{opacity:0.5;background-color:hsl(-79.4117647059, 100%, 93.3333333333%);fill:hsl(-79.4117647059, 100%, 93.3333333333%);}#mermaid-svg-vjQx8mtXxyiB4tqs .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-vjQx8mtXxyiB4tqs .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-vjQx8mtXxyiB4tqs :root{--mermaid-font-family:"trebuchet ms",verdana,arial,sans-serif;}
现象
堆使用率持续爬升
Full GC 后水位
一次比一次高

① jstat -gcutil pid 2000 确认
O 列 GC 后降不下去 = 泄漏
降得下去 = 对象太多 / 堆太小
转 SOP-3

② 抓 dump
纪律：大堆 dump 会 STW 数十秒
先摘流量再抓
jmap -dump:live,format=b,file=heap.hprof pid
或 jcmd pid GC.heap_dump

③ MAT 三板斧
Leak Suspects
自动体检报告
新手友好

Dominator Tree
谁独占最多内存
按保留堆排序

Path to GC Roots
这个对象为什么还活着
反查完整引用链
= 最后一公里

常见根因对号入座
静态集合只加不删
ThreadLocal 没 remove
监听器没注销
连接没关
缓存无界

④ 修复 + 验证
堆水位锯齿状稳定
Full GC 后能降回低位

SOP-3：频繁 GC / GC 停顿长

#mermaid-svg-MybI5OngCJXKLLL7{font-family:Microsoft YaHei;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-MybI5OngCJXKLLL7 .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-MybI5OngCJXKLLL7 .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-MybI5OngCJXKLLL7 .error-icon{fill:hsl(220.5882352941, 100%, 98.3333333333%);}#mermaid-svg-MybI5OngCJXKLLL7 .error-text{fill:rgb(8.5000000002, 5.7500000001, 0);stroke:rgb(8.5000000002, 5.7500000001, 0);}#mermaid-svg-MybI5OngCJXKLLL7 .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-MybI5OngCJXKLLL7 .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-MybI5OngCJXKLLL7 .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-MybI5OngCJXKLLL7 .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-MybI5OngCJXKLLL7 .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-MybI5OngCJXKLLL7 .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-MybI5OngCJXKLLL7 .marker{fill:#0b0b0b;stroke:#0b0b0b;}#mermaid-svg-MybI5OngCJXKLLL7 .marker.cross{stroke:#0b0b0b;}#mermaid-svg-MybI5OngCJXKLLL7 svg{font-family:Microsoft YaHei;font-size:16px;}#mermaid-svg-MybI5OngCJXKLLL7 p{margin:0;}#mermaid-svg-MybI5OngCJXKLLL7 .label{font-family:Microsoft YaHei;color:#333;}#mermaid-svg-MybI5OngCJXKLLL7 .cluster-label text{fill:rgb(8.5000000002, 5.7500000001, 0);}#mermaid-svg-MybI5OngCJXKLLL7 .cluster-label span{color:rgb(8.5000000002, 5.7500000001, 0);}#mermaid-svg-MybI5OngCJXKLLL7 .cluster-label span p{background-color:transparent;}#mermaid-svg-MybI5OngCJXKLLL7 .label text,#mermaid-svg-MybI5OngCJXKLLL7 span{fill:#333;color:#333;}#mermaid-svg-MybI5OngCJXKLLL7 .node rect,#mermaid-svg-MybI5OngCJXKLLL7 .node circle,#mermaid-svg-MybI5OngCJXKLLL7 .node ellipse,#mermaid-svg-MybI5OngCJXKLLL7 .node polygon,#mermaid-svg-MybI5OngCJXKLLL7 .node path{fill:#fff4dd;stroke:hsl(40.5882352941, 60%, 83.3333333333%);stroke-width:1px;}#mermaid-svg-MybI5OngCJXKLLL7 .rough-node .label text,#mermaid-svg-MybI5OngCJXKLLL7 .node .label text,#mermaid-svg-MybI5OngCJXKLLL7 .image-shape .label,#mermaid-svg-MybI5OngCJXKLLL7 .icon-shape .label{text-anchor:middle;}#mermaid-svg-MybI5OngCJXKLLL7 .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-MybI5OngCJXKLLL7 .rough-node .label,#mermaid-svg-MybI5OngCJXKLLL7 .node .label,#mermaid-svg-MybI5OngCJXKLLL7 .image-shape .label,#mermaid-svg-MybI5OngCJXKLLL7 .icon-shape .label{text-align:center;}#mermaid-svg-MybI5OngCJXKLLL7 .node.clickable{cursor:pointer;}#mermaid-svg-MybI5OngCJXKLLL7 .root .anchor path{fill:#0b0b0b!important;stroke-width:0;stroke:#0b0b0b;}#mermaid-svg-MybI5OngCJXKLLL7 .arrowheadPath{fill:#0b0b0b;}#mermaid-svg-MybI5OngCJXKLLL7 .edgePath .path{stroke:#0b0b0b;stroke-width:2.0px;}#mermaid-svg-MybI5OngCJXKLLL7 .flowchart-link{stroke:#0b0b0b;fill:none;}#mermaid-svg-MybI5OngCJXKLLL7 .edgeLabel{background-color:hsl(-79.4117647059, 100%, 93.3333333333%);text-align:center;}#mermaid-svg-MybI5OngCJXKLLL7 .edgeLabel p{background-color:hsl(-79.4117647059, 100%, 93.3333333333%);}#mermaid-svg-MybI5OngCJXKLLL7 .edgeLabel rect{opacity:0.5;background-color:hsl(-79.4117647059, 100%, 93.3333333333%);fill:hsl(-79.4117647059, 100%, 93.3333333333%);}#mermaid-svg-MybI5OngCJXKLLL7 .labelBkg{background-color:rgba(243.9999999999, 220.9999999998, 255, 0.5);}#mermaid-svg-MybI5OngCJXKLLL7 .cluster rect{fill:hsl(220.5882352941, 100%, 98.3333333333%);stroke:hsl(220.5882352941, 60%, 88.3333333333%);stroke-width:1px;}#mermaid-svg-MybI5OngCJXKLLL7 .cluster text{fill:rgb(8.5000000002, 5.7500000001, 0);}#mermaid-svg-MybI5OngCJXKLLL7 .cluster span{color:rgb(8.5000000002, 5.7500000001, 0);}#mermaid-svg-MybI5OngCJXKLLL7 div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:Microsoft YaHei;font-size:12px;background:hsl(220.5882352941, 100%, 98.3333333333%);border:1px solid hsl(220.5882352941, 60%, 88.3333333333%);border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-MybI5OngCJXKLLL7 .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-MybI5OngCJXKLLL7 rect.text{fill:none;stroke-width:0;}#mermaid-svg-MybI5OngCJXKLLL7 .icon-shape,#mermaid-svg-MybI5OngCJXKLLL7 .image-shape{background-color:hsl(-79.4117647059, 100%, 93.3333333333%);text-align:center;}#mermaid-svg-MybI5OngCJXKLLL7 .icon-shape p,#mermaid-svg-MybI5OngCJXKLLL7 .image-shape p{background-color:hsl(-79.4117647059, 100%, 93.3333333333%);padding:2px;}#mermaid-svg-MybI5OngCJXKLLL7 .icon-shape .label rect,#mermaid-svg-MybI5OngCJXKLLL7 .image-shape .label rect{opacity:0.5;background-color:hsl(-79.4117647059, 100%, 93.3333333333%);fill:hsl(-79.4117647059, 100%, 93.3333333333%);}#mermaid-svg-MybI5OngCJXKLLL7 .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-MybI5OngCJXKLLL7 .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-MybI5OngCJXKLLL7 :root{--mermaid-font-family:"trebuchet ms",verdana,arial,sans-serif;} Young GC 频繁
Full GC 频繁但回收有效
Full GC 频繁且回收无效
G1 出现 Full GC
现象：Full GC 频繁 / 停顿长
① 开 GC 日志 + GCEasy 可视化
看三指标
停顿 P99 / 频率 / 回收效率

GC 类型分型
新生代太小
/ 分配速率太高
→ 调大 -Xmn
查谁在疯狂 new

老年代太小
/ 晋升过快
→ 查大对象
调新生代与 Survivor

内存泄漏
→ 转 SOP-2

必有原因字段，对症
Humongous = 大对象
→ 改分页 / 调 RegionSize
Metadata = 元空间
→ 查类加载
Ergonomics = 晋升失败
→ 扩老年代
System.gc
→ 查显式调用 / RMI

② 验证
压测对比 GC 指标
业务修复永远优先于参数调优

SOP-4：线程死锁/卡顿

#mermaid-svg-8r5RavkI0Udfvf0d{font-family:Microsoft YaHei;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-8r5RavkI0Udfvf0d .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-8r5RavkI0Udfvf0d .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-8r5RavkI0Udfvf0d .error-icon{fill:hsl(220.5882352941, 100%, 98.3333333333%);}#mermaid-svg-8r5RavkI0Udfvf0d .error-text{fill:rgb(8.5000000002, 5.7500000001, 0);stroke:rgb(8.5000000002, 5.7500000001, 0);}#mermaid-svg-8r5RavkI0Udfvf0d .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-8r5RavkI0Udfvf0d .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-8r5RavkI0Udfvf0d .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-8r5RavkI0Udfvf0d .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-8r5RavkI0Udfvf0d .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-8r5RavkI0Udfvf0d .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-8r5RavkI0Udfvf0d .marker{fill:#0b0b0b;stroke:#0b0b0b;}#mermaid-svg-8r5RavkI0Udfvf0d .marker.cross{stroke:#0b0b0b;}#mermaid-svg-8r5RavkI0Udfvf0d svg{font-family:Microsoft YaHei;font-size:16px;}#mermaid-svg-8r5RavkI0Udfvf0d p{margin:0;}#mermaid-svg-8r5RavkI0Udfvf0d .label{font-family:Microsoft YaHei;color:#333;}#mermaid-svg-8r5RavkI0Udfvf0d .cluster-label text{fill:rgb(8.5000000002, 5.7500000001, 0);}#mermaid-svg-8r5RavkI0Udfvf0d .cluster-label span{color:rgb(8.5000000002, 5.7500000001, 0);}#mermaid-svg-8r5RavkI0Udfvf0d .cluster-label span p{background-color:transparent;}#mermaid-svg-8r5RavkI0Udfvf0d .label text,#mermaid-svg-8r5RavkI0Udfvf0d span{fill:#333;color:#333;}#mermaid-svg-8r5RavkI0Udfvf0d .node rect,#mermaid-svg-8r5RavkI0Udfvf0d .node circle,#mermaid-svg-8r5RavkI0Udfvf0d .node ellipse,#mermaid-svg-8r5RavkI0Udfvf0d .node polygon,#mermaid-svg-8r5RavkI0Udfvf0d .node path{fill:#fff4dd;stroke:hsl(40.5882352941, 60%, 83.3333333333%);stroke-width:1px;}#mermaid-svg-8r5RavkI0Udfvf0d .rough-node .label text,#mermaid-svg-8r5RavkI0Udfvf0d .node .label text,#mermaid-svg-8r5RavkI0Udfvf0d .image-shape .label,#mermaid-svg-8r5RavkI0Udfvf0d .icon-shape .label{text-anchor:middle;}#mermaid-svg-8r5RavkI0Udfvf0d .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-8r5RavkI0Udfvf0d .rough-node .label,#mermaid-svg-8r5RavkI0Udfvf0d .node .label,#mermaid-svg-8r5RavkI0Udfvf0d .image-shape .label,#mermaid-svg-8r5RavkI0Udfvf0d .icon-shape .label{text-align:center;}#mermaid-svg-8r5RavkI0Udfvf0d .node.clickable{cursor:pointer;}#mermaid-svg-8r5RavkI0Udfvf0d .root .anchor path{fill:#0b0b0b!important;stroke-width:0;stroke:#0b0b0b;}#mermaid-svg-8r5RavkI0Udfvf0d .arrowheadPath{fill:#0b0b0b;}#mermaid-svg-8r5RavkI0Udfvf0d .edgePath .path{stroke:#0b0b0b;stroke-width:2.0px;}#mermaid-svg-8r5RavkI0Udfvf0d .flowchart-link{stroke:#0b0b0b;fill:none;}#mermaid-svg-8r5RavkI0Udfvf0d .edgeLabel{background-color:hsl(-79.4117647059, 100%, 93.3333333333%);text-align:center;}#mermaid-svg-8r5RavkI0Udfvf0d .edgeLabel p{background-color:hsl(-79.4117647059, 100%, 93.3333333333%);}#mermaid-svg-8r5RavkI0Udfvf0d .edgeLabel rect{opacity:0.5;background-color:hsl(-79.4117647059, 100%, 93.3333333333%);fill:hsl(-79.4117647059, 100%, 93.3333333333%);}#mermaid-svg-8r5RavkI0Udfvf0d .labelBkg{background-color:rgba(243.9999999999, 220.9999999998, 255, 0.5);}#mermaid-svg-8r5RavkI0Udfvf0d .cluster rect{fill:hsl(220.5882352941, 100%, 98.3333333333%);stroke:hsl(220.5882352941, 60%, 88.3333333333%);stroke-width:1px;}#mermaid-svg-8r5RavkI0Udfvf0d .cluster text{fill:rgb(8.5000000002, 5.7500000001, 0);}#mermaid-svg-8r5RavkI0Udfvf0d .cluster span{color:rgb(8.5000000002, 5.7500000001, 0);}#mermaid-svg-8r5RavkI0Udfvf0d div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:Microsoft YaHei;font-size:12px;background:hsl(220.5882352941, 100%, 98.3333333333%);border:1px solid hsl(220.5882352941, 60%, 88.3333333333%);border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-8r5RavkI0Udfvf0d .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-8r5RavkI0Udfvf0d rect.text{fill:none;stroke-width:0;}#mermaid-svg-8r5RavkI0Udfvf0d .icon-shape,#mermaid-svg-8r5RavkI0Udfvf0d .image-shape{background-color:hsl(-79.4117647059, 100%, 93.3333333333%);text-align:center;}#mermaid-svg-8r5RavkI0Udfvf0d .icon-shape p,#mermaid-svg-8r5RavkI0Udfvf0d .image-shape p{background-color:hsl(-79.4117647059, 100%, 93.3333333333%);padding:2px;}#mermaid-svg-8r5RavkI0Udfvf0d .icon-shape .label rect,#mermaid-svg-8r5RavkI0Udfvf0d .image-shape .label rect{opacity:0.5;background-color:hsl(-79.4117647059, 100%, 93.3333333333%);fill:hsl(-79.4117647059, 100%, 93.3333333333%);}#mermaid-svg-8r5RavkI0Udfvf0d .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-8r5RavkI0Udfvf0d .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-8r5RavkI0Udfvf0d :root{--mermaid-font-family:"trebuchet ms",verdana,arial,sans-serif;} BLOCKED 多
Found one Java-level deadlock
WAITING 在 socket read
大量 WAITING 在线程池 get
现象：接口卡顿 / 超时

CPU 却不高
① jstack pid 抓线程快照
或 Arthas thread -b
直接找阻塞元凶

按线程状态分诊
锁竞争
找 waiting to lock 的锁地址
→ 反查 locked 该地址的线程
= 持锁者
看持锁者在干什么
为何不释放

死锁
jstack 自动检测并打印
两线程互相持有
对方要的锁
→ 修复加锁顺序 / tryLock 超时

下游 RPC / DB 慢
且无超时
→ 出 JVM 排查
查下游 + 补超时配置

任务堆积
生产快消费慢
→ 查任务耗时
扩消费者

② 连续抓 3 次对比
同一线程 3 次都在同一行 = 真卡住
位置在变 = 只是慢

SOP-5：容器 exit 137（OOMKilled）

#mermaid-svg-yKnclrM77nm32Dze{font-family:Microsoft YaHei;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-yKnclrM77nm32Dze .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-yKnclrM77nm32Dze .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-yKnclrM77nm32Dze .error-icon{fill:hsl(220.5882352941, 100%, 98.3333333333%);}#mermaid-svg-yKnclrM77nm32Dze .error-text{fill:rgb(8.5000000002, 5.7500000001, 0);stroke:rgb(8.5000000002, 5.7500000001, 0);}#mermaid-svg-yKnclrM77nm32Dze .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-yKnclrM77nm32Dze .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-yKnclrM77nm32Dze .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-yKnclrM77nm32Dze .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-yKnclrM77nm32Dze .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-yKnclrM77nm32Dze .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-yKnclrM77nm32Dze .marker{fill:#0b0b0b;stroke:#0b0b0b;}#mermaid-svg-yKnclrM77nm32Dze .marker.cross{stroke:#0b0b0b;}#mermaid-svg-yKnclrM77nm32Dze svg{font-family:Microsoft YaHei;font-size:16px;}#mermaid-svg-yKnclrM77nm32Dze p{margin:0;}#mermaid-svg-yKnclrM77nm32Dze .label{font-family:Microsoft YaHei;color:#333;}#mermaid-svg-yKnclrM77nm32Dze .cluster-label text{fill:rgb(8.5000000002, 5.7500000001, 0);}#mermaid-svg-yKnclrM77nm32Dze .cluster-label span{color:rgb(8.5000000002, 5.7500000001, 0);}#mermaid-svg-yKnclrM77nm32Dze .cluster-label span p{background-color:transparent;}#mermaid-svg-yKnclrM77nm32Dze .label text,#mermaid-svg-yKnclrM77nm32Dze span{fill:#333;color:#333;}#mermaid-svg-yKnclrM77nm32Dze .node rect,#mermaid-svg-yKnclrM77nm32Dze .node circle,#mermaid-svg-yKnclrM77nm32Dze .node ellipse,#mermaid-svg-yKnclrM77nm32Dze .node polygon,#mermaid-svg-yKnclrM77nm32Dze .node path{fill:#fff4dd;stroke:hsl(40.5882352941, 60%, 83.3333333333%);stroke-width:1px;}#mermaid-svg-yKnclrM77nm32Dze .rough-node .label text,#mermaid-svg-yKnclrM77nm32Dze .node .label text,#mermaid-svg-yKnclrM77nm32Dze .image-shape .label,#mermaid-svg-yKnclrM77nm32Dze .icon-shape .label{text-anchor:middle;}#mermaid-svg-yKnclrM77nm32Dze .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-yKnclrM77nm32Dze .rough-node .label,#mermaid-svg-yKnclrM77nm32Dze .node .label,#mermaid-svg-yKnclrM77nm32Dze .image-shape .label,#mermaid-svg-yKnclrM77nm32Dze .icon-shape .label{text-align:center;}#mermaid-svg-yKnclrM77nm32Dze .node.clickable{cursor:pointer;}#mermaid-svg-yKnclrM77nm32Dze .root .anchor path{fill:#0b0b0b!important;stroke-width:0;stroke:#0b0b0b;}#mermaid-svg-yKnclrM77nm32Dze .arrowheadPath{fill:#0b0b0b;}#mermaid-svg-yKnclrM77nm32Dze .edgePath .path{stroke:#0b0b0b;stroke-width:2.0px;}#mermaid-svg-yKnclrM77nm32Dze .flowchart-link{stroke:#0b0b0b;fill:none;}#mermaid-svg-yKnclrM77nm32Dze .edgeLabel{background-color:hsl(-79.4117647059, 100%, 93.3333333333%);text-align:center;}#mermaid-svg-yKnclrM77nm32Dze .edgeLabel p{background-color:hsl(-79.4117647059, 100%, 93.3333333333%);}#mermaid-svg-yKnclrM77nm32Dze .edgeLabel rect{opacity:0.5;background-color:hsl(-79.4117647059, 100%, 93.3333333333%);fill:hsl(-79.4117647059, 100%, 93.3333333333%);}#mermaid-svg-yKnclrM77nm32Dze .labelBkg{background-color:rgba(243.9999999999, 220.9999999998, 255, 0.5);}#mermaid-svg-yKnclrM77nm32Dze .cluster rect{fill:hsl(220.5882352941, 100%, 98.3333333333%);stroke:hsl(220.5882352941, 60%, 88.3333333333%);stroke-width:1px;}#mermaid-svg-yKnclrM77nm32Dze .cluster text{fill:rgb(8.5000000002, 5.7500000001, 0);}#mermaid-svg-yKnclrM77nm32Dze .cluster span{color:rgb(8.5000000002, 5.7500000001, 0);}#mermaid-svg-yKnclrM77nm32Dze div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:Microsoft YaHei;font-size:12px;background:hsl(220.5882352941, 100%, 98.3333333333%);border:1px solid hsl(220.5882352941, 60%, 88.3333333333%);border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-yKnclrM77nm32Dze .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-yKnclrM77nm32Dze rect.text{fill:none;stroke-width:0;}#mermaid-svg-yKnclrM77nm32Dze .icon-shape,#mermaid-svg-yKnclrM77nm32Dze .image-shape{background-color:hsl(-79.4117647059, 100%, 93.3333333333%);text-align:center;}#mermaid-svg-yKnclrM77nm32Dze .icon-shape p,#mermaid-svg-yKnclrM77nm32Dze .image-shape p{background-color:hsl(-79.4117647059, 100%, 93.3333333333%);padding:2px;}#mermaid-svg-yKnclrM77nm32Dze .icon-shape .label rect,#mermaid-svg-yKnclrM77nm32Dze .image-shape .label rect{opacity:0.5;background-color:hsl(-79.4117647059, 100%, 93.3333333333%);fill:hsl(-79.4117647059, 100%, 93.3333333333%);}#mermaid-svg-yKnclrM77nm32Dze .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-yKnclrM77nm32Dze .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-yKnclrM77nm32Dze :root{--mermaid-font-family:"trebuchet ms",verdana,arial,sans-serif;} 堆设太大
老 JDK 不感知容器
账没超还是被杀
现象
Pod 反复重启
kubectl describe 显示
OOMKilled exit 137
JVM 日志干干净净

① 确认死因
是内核击杀非 JVM OOM
exit 137 = 128 + 9 = SIGKILL
dmesg / kubectl describe pod
看 cgroup 超限

② 算容器内存账
堆 Xmx + 元空间

  • 线程数 × Xss
  • 直接内存 + CodeCache
  • GC 开销 ≤ limit?

哪一项超了
Xmx 太接近 limit
→ 改
-XX:MaxRAMPercentage=70
留 30% 给堆外

JDK 8u191 前
按宿主机算内存
→ 升级 8u191+ / JDK 11+
进容器验证
PrintFlagsFinal 的
MaxHeapSize

堆外泄漏
→ 转 SOP-6

SOP-6：堆外内存泄漏

#mermaid-svg-9vfU9i6UKGpgfR0E{font-family:Microsoft YaHei;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-9vfU9i6UKGpgfR0E .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-9vfU9i6UKGpgfR0E .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-9vfU9i6UKGpgfR0E .error-icon{fill:hsl(220.5882352941, 100%, 98.3333333333%);}#mermaid-svg-9vfU9i6UKGpgfR0E .error-text{fill:rgb(8.5000000002, 5.7500000001, 0);stroke:rgb(8.5000000002, 5.7500000001, 0);}#mermaid-svg-9vfU9i6UKGpgfR0E .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-9vfU9i6UKGpgfR0E .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-9vfU9i6UKGpgfR0E .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-9vfU9i6UKGpgfR0E .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-9vfU9i6UKGpgfR0E .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-9vfU9i6UKGpgfR0E .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-9vfU9i6UKGpgfR0E .marker{fill:#0b0b0b;stroke:#0b0b0b;}#mermaid-svg-9vfU9i6UKGpgfR0E .marker.cross{stroke:#0b0b0b;}#mermaid-svg-9vfU9i6UKGpgfR0E svg{font-family:Microsoft YaHei;font-size:16px;}#mermaid-svg-9vfU9i6UKGpgfR0E p{margin:0;}#mermaid-svg-9vfU9i6UKGpgfR0E .label{font-family:Microsoft YaHei;color:#333;}#mermaid-svg-9vfU9i6UKGpgfR0E .cluster-label text{fill:rgb(8.5000000002, 5.7500000001, 0);}#mermaid-svg-9vfU9i6UKGpgfR0E .cluster-label span{color:rgb(8.5000000002, 5.7500000001, 0);}#mermaid-svg-9vfU9i6UKGpgfR0E .cluster-label span p{background-color:transparent;}#mermaid-svg-9vfU9i6UKGpgfR0E .label text,#mermaid-svg-9vfU9i6UKGpgfR0E span{fill:#333;color:#333;}#mermaid-svg-9vfU9i6UKGpgfR0E .node rect,#mermaid-svg-9vfU9i6UKGpgfR0E .node circle,#mermaid-svg-9vfU9i6UKGpgfR0E .node ellipse,#mermaid-svg-9vfU9i6UKGpgfR0E .node polygon,#mermaid-svg-9vfU9i6UKGpgfR0E .node path{fill:#fff4dd;stroke:hsl(40.5882352941, 60%, 83.3333333333%);stroke-width:1px;}#mermaid-svg-9vfU9i6UKGpgfR0E .rough-node .label text,#mermaid-svg-9vfU9i6UKGpgfR0E .node .label text,#mermaid-svg-9vfU9i6UKGpgfR0E .image-shape .label,#mermaid-svg-9vfU9i6UKGpgfR0E .icon-shape .label{text-anchor:middle;}#mermaid-svg-9vfU9i6UKGpgfR0E .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-9vfU9i6UKGpgfR0E .rough-node .label,#mermaid-svg-9vfU9i6UKGpgfR0E .node .label,#mermaid-svg-9vfU9i6UKGpgfR0E .image-shape .label,#mermaid-svg-9vfU9i6UKGpgfR0E .icon-shape .label{text-align:center;}#mermaid-svg-9vfU9i6UKGpgfR0E .node.clickable{cursor:pointer;}#mermaid-svg-9vfU9i6UKGpgfR0E .root .anchor path{fill:#0b0b0b!important;stroke-width:0;stroke:#0b0b0b;}#mermaid-svg-9vfU9i6UKGpgfR0E .arrowheadPath{fill:#0b0b0b;}#mermaid-svg-9vfU9i6UKGpgfR0E .edgePath .path{stroke:#0b0b0b;stroke-width:2.0px;}#mermaid-svg-9vfU9i6UKGpgfR0E .flowchart-link{stroke:#0b0b0b;fill:none;}#mermaid-svg-9vfU9i6UKGpgfR0E .edgeLabel{background-color:hsl(-79.4117647059, 100%, 93.3333333333%);text-align:center;}#mermaid-svg-9vfU9i6UKGpgfR0E .edgeLabel p{background-color:hsl(-79.4117647059, 100%, 93.3333333333%);}#mermaid-svg-9vfU9i6UKGpgfR0E .edgeLabel rect{opacity:0.5;background-color:hsl(-79.4117647059, 100%, 93.3333333333%);fill:hsl(-79.4117647059, 100%, 93.3333333333%);}#mermaid-svg-9vfU9i6UKGpgfR0E .labelBkg{background-color:rgba(243.9999999999, 220.9999999998, 255, 0.5);}#mermaid-svg-9vfU9i6UKGpgfR0E .cluster rect{fill:hsl(220.5882352941, 100%, 98.3333333333%);stroke:hsl(220.5882352941, 60%, 88.3333333333%);stroke-width:1px;}#mermaid-svg-9vfU9i6UKGpgfR0E .cluster text{fill:rgb(8.5000000002, 5.7500000001, 0);}#mermaid-svg-9vfU9i6UKGpgfR0E .cluster span{color:rgb(8.5000000002, 5.7500000001, 0);}#mermaid-svg-9vfU9i6UKGpgfR0E div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:Microsoft YaHei;font-size:12px;background:hsl(220.5882352941, 100%, 98.3333333333%);border:1px solid hsl(220.5882352941, 60%, 88.3333333333%);border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-9vfU9i6UKGpgfR0E .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-9vfU9i6UKGpgfR0E rect.text{fill:none;stroke-width:0;}#mermaid-svg-9vfU9i6UKGpgfR0E .icon-shape,#mermaid-svg-9vfU9i6UKGpgfR0E .image-shape{background-color:hsl(-79.4117647059, 100%, 93.3333333333%);text-align:center;}#mermaid-svg-9vfU9i6UKGpgfR0E .icon-shape p,#mermaid-svg-9vfU9i6UKGpgfR0E .image-shape p{background-color:hsl(-79.4117647059, 100%, 93.3333333333%);padding:2px;}#mermaid-svg-9vfU9i6UKGpgfR0E .icon-shape .label rect,#mermaid-svg-9vfU9i6UKGpgfR0E .image-shape .label rect{opacity:0.5;background-color:hsl(-79.4117647059, 100%, 93.3333333333%);fill:hsl(-79.4117647059, 100%, 93.3333333333%);}#mermaid-svg-9vfU9i6UKGpgfR0E .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-9vfU9i6UKGpgfR0E .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-9vfU9i6UKGpgfR0E :root{--mermaid-font-family:"trebuchet ms",verdana,arial,sans-serif;} Internal / Other 涨
Thread 涨
Class 涨
现象
堆监控一切正常
进程 RSS 却持续上涨
/ 被 137 杀

① 确认堆外
RSS - JVM 堆已用
= 堆外规模
差值持续增长
即堆外泄漏

② NMT 定位涨在哪
启动加
-XX:NativeMemoryTracking=detail
jcmd pid VM.native_memory baseline
跑一段时间后
summary.diff 对比
看 Internal / Thread / Class
哪个区在涨

按区域对症
DirectByteBuffer 泄漏
→ 查 NIO / Netty
ByteBuf 没 release
开
-Dio.netty.leakDetection.level=paranoid
复现
日志打出 LEAK 及分配点堆栈

线程泄漏
线程数 × Xss 失控
→ 线程 dump 数线程
查线程池滥用

类加载器泄漏
→ 查动态类生成
见 03-B 篇事故二

③ 防御
MaxDirectMemorySize 显式设限
把 OS 静默击杀
变成有日志的 JVM 内 OOM

四、实战演练：一次完整的 CPU 100% 排查

把 SOP-1 走一遍全流程，含每步的真实输出解读。
#mermaid-svg-cRvkEu8P002Kt25w{font-family:Microsoft YaHei;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-cRvkEu8P002Kt25w .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-cRvkEu8P002Kt25w .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-cRvkEu8P002Kt25w .error-icon{fill:hsl(220.5882352941, 100%, 98.3333333333%);}#mermaid-svg-cRvkEu8P002Kt25w .error-text{fill:rgb(8.5000000002, 5.7500000001, 0);stroke:rgb(8.5000000002, 5.7500000001, 0);}#mermaid-svg-cRvkEu8P002Kt25w .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-cRvkEu8P002Kt25w .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-cRvkEu8P002Kt25w .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-cRvkEu8P002Kt25w .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-cRvkEu8P002Kt25w .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-cRvkEu8P002Kt25w .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-cRvkEu8P002Kt25w .marker{fill:#0b0b0b;stroke:#0b0b0b;}#mermaid-svg-cRvkEu8P002Kt25w .marker.cross{stroke:#0b0b0b;}#mermaid-svg-cRvkEu8P002Kt25w svg{font-family:Microsoft YaHei;font-size:16px;}#mermaid-svg-cRvkEu8P002Kt25w p{margin:0;}#mermaid-svg-cRvkEu8P002Kt25w .label{font-family:Microsoft YaHei;color:#333;}#mermaid-svg-cRvkEu8P002Kt25w .cluster-label text{fill:rgb(8.5000000002, 5.7500000001, 0);}#mermaid-svg-cRvkEu8P002Kt25w .cluster-label span{color:rgb(8.5000000002, 5.7500000001, 0);}#mermaid-svg-cRvkEu8P002Kt25w .cluster-label span p{background-color:transparent;}#mermaid-svg-cRvkEu8P002Kt25w .label text,#mermaid-svg-cRvkEu8P002Kt25w span{fill:#333;color:#333;}#mermaid-svg-cRvkEu8P002Kt25w .node rect,#mermaid-svg-cRvkEu8P002Kt25w .node circle,#mermaid-svg-cRvkEu8P002Kt25w .node ellipse,#mermaid-svg-cRvkEu8P002Kt25w .node polygon,#mermaid-svg-cRvkEu8P002Kt25w .node path{fill:#fff4dd;stroke:hsl(40.5882352941, 60%, 83.3333333333%);stroke-width:1px;}#mermaid-svg-cRvkEu8P002Kt25w .rough-node .label text,#mermaid-svg-cRvkEu8P002Kt25w .node .label text,#mermaid-svg-cRvkEu8P002Kt25w .image-shape .label,#mermaid-svg-cRvkEu8P002Kt25w .icon-shape .label{text-anchor:middle;}#mermaid-svg-cRvkEu8P002Kt25w .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-cRvkEu8P002Kt25w .rough-node .label,#mermaid-svg-cRvkEu8P002Kt25w .node .label,#mermaid-svg-cRvkEu8P002Kt25w .image-shape .label,#mermaid-svg-cRvkEu8P002Kt25w .icon-shape .label{text-align:center;}#mermaid-svg-cRvkEu8P002Kt25w .node.clickable{cursor:pointer;}#mermaid-svg-cRvkEu8P002Kt25w .root .anchor path{fill:#0b0b0b!important;stroke-width:0;stroke:#0b0b0b;}#mermaid-svg-cRvkEu8P002Kt25w .arrowheadPath{fill:#0b0b0b;}#mermaid-svg-cRvkEu8P002Kt25w .edgePath .path{stroke:#0b0b0b;stroke-width:2.0px;}#mermaid-svg-cRvkEu8P002Kt25w .flowchart-link{stroke:#0b0b0b;fill:none;}#mermaid-svg-cRvkEu8P002Kt25w .edgeLabel{background-color:hsl(-79.4117647059, 100%, 93.3333333333%);text-align:center;}#mermaid-svg-cRvkEu8P002Kt25w .edgeLabel p{background-color:hsl(-79.4117647059, 100%, 93.3333333333%);}#mermaid-svg-cRvkEu8P002Kt25w .edgeLabel rect{opacity:0.5;background-color:hsl(-79.4117647059, 100%, 93.3333333333%);fill:hsl(-79.4117647059, 100%, 93.3333333333%);}#mermaid-svg-cRvkEu8P002Kt25w .labelBkg{background-color:rgba(243.9999999999, 220.9999999998, 255, 0.5);}#mermaid-svg-cRvkEu8P002Kt25w .cluster rect{fill:hsl(220.5882352941, 100%, 98.3333333333%);stroke:hsl(220.5882352941, 60%, 88.3333333333%);stroke-width:1px;}#mermaid-svg-cRvkEu8P002Kt25w .cluster text{fill:rgb(8.5000000002, 5.7500000001, 0);}#mermaid-svg-cRvkEu8P002Kt25w .cluster span{color:rgb(8.5000000002, 5.7500000001, 0);}#mermaid-svg-cRvkEu8P002Kt25w div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:Microsoft YaHei;font-size:12px;background:hsl(220.5882352941, 100%, 98.3333333333%);border:1px solid hsl(220.5882352941, 60%, 88.3333333333%);border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-cRvkEu8P002Kt25w .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-cRvkEu8P002Kt25w rect.text{fill:none;stroke-width:0;}#mermaid-svg-cRvkEu8P002Kt25w .icon-shape,#mermaid-svg-cRvkEu8P002Kt25w .image-shape{background-color:hsl(-79.4117647059, 100%, 93.3333333333%);text-align:center;}#mermaid-svg-cRvkEu8P002Kt25w .icon-shape p,#mermaid-svg-cRvkEu8P002Kt25w .image-shape p{background-color:hsl(-79.4117647059, 100%, 93.3333333333%);padding:2px;}#mermaid-svg-cRvkEu8P002Kt25w .icon-shape .label rect,#mermaid-svg-cRvkEu8P002Kt25w .image-shape .label rect{opacity:0.5;background-color:hsl(-79.4117647059, 100%, 93.3333333333%);fill:hsl(-79.4117647059, 100%, 93.3333333333%);}#mermaid-svg-cRvkEu8P002Kt25w .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-cRvkEu8P002Kt25w .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-cRvkEu8P002Kt25w :root{--mermaid-font-family:"trebuchet ms",verdana,arial,sans-serif;}
① 告警
订单服务 CPU 98%
接口 RT 恶化

② top
java 进程 CPU 380%
us 占 95%
→ 业务代码或 GC 线程

③ top -Hp pid
线程 23456 占 98%
printf %x 23456
→ 0x5ba0

④ jstack pid 搜 nid=0x5ba0
连续 3 次栈都一样
= 真热点

⑤ 栈指纹
满屏
java.util.regex.Pattern$Curly.match
→ 正则灾难性回溯
用户输入的长地址串
触发指数级回溯

⑥ 止血
该接口限流

  • 匹配加超时
    根治
    重写正则去掉嵌套量词
  • 输入长度上限

⑦ 验证
压测同款输入
CPU 回落 30%
RT 恢复
复盘
正则入参必须评审

这次排查用到的方法论回顾：分诊（us/sy）→ 缩小范围（-Hp 找热点线程）→ 拿数据（jstack 栈）→ 指纹识别（满屏 Pattern$ = 正则回溯）→ 止血与根治分离 → 验证闭环。

五、对比：工具选型

5.1 jstack/jmap vs Arthas

维度 JDK 自带工具 Arthas
可用性 生产机必有（JDK 自带） 需上传 jar（一次性）
CPU 热点 top -Hp + jstack 三步换算 thread -n 3 一步
方法级观测 无（要加日志重发布） watch/trace/stack 免发布
热修复 无 retransform（应急止血）
风险 jmap -dump 会 STW 同样有（attach 机制），但操作更收敛
结论 保底能力必须会（不是所有环境让你装 Arthas） 日常首选，效率翻倍

5.2 在线诊断 vs 离线 dump

维度 在线（Arthas/jstat/profiler） 离线（heap dump + MAT）
对服务影响 低（采样式） 高（dump STW 数十秒）
信息量 运行时动态（方法耗时/热点） 全量对象图（引用链）
适用 第一现场快速分诊 泄漏根因深挖
纪律 先在线分诊，确需 dump 再摘流量抓 dump 文件含敏感数据，注意脱敏与保管

5.3 排查思路的"三个层次"（与全系列方法论同构）

层次 JVM 排查 MySQL（A 篇第九章） Redis（治理篇）
观测 GC 日志/jstat/监控四件套 慢查询日志/EXPLAIN INFO/慢日志/四层指标
定位 dump + MAT/火焰图 执行计划分析 bigkeys/hotkeys/latency
根治 代码修复优先于参数 SQL/索引优先于架构 治理制度优先于扩容

三个技术栈的排查方法论完全同构------观测 → 定位 → 根治，业务修复永远优先于参数调优。学会一次，处处适用。

六、总结

6.1 一张图回顾全文

#mermaid-svg-Kebf5hMWsPrtLetB{font-family:Microsoft YaHei;font-size:18px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-Kebf5hMWsPrtLetB .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-Kebf5hMWsPrtLetB .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-Kebf5hMWsPrtLetB .error-icon{fill:hsl(220.5882352941, 100%, 98.3333333333%);}#mermaid-svg-Kebf5hMWsPrtLetB .error-text{fill:rgb(8.5000000002, 5.7500000001, 0);stroke:rgb(8.5000000002, 5.7500000001, 0);}#mermaid-svg-Kebf5hMWsPrtLetB .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-Kebf5hMWsPrtLetB .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-Kebf5hMWsPrtLetB .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-Kebf5hMWsPrtLetB .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-Kebf5hMWsPrtLetB .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-Kebf5hMWsPrtLetB .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-Kebf5hMWsPrtLetB .marker{fill:#0b0b0b;stroke:#0b0b0b;}#mermaid-svg-Kebf5hMWsPrtLetB .marker.cross{stroke:#0b0b0b;}#mermaid-svg-Kebf5hMWsPrtLetB svg{font-family:Microsoft YaHei;font-size:18px;}#mermaid-svg-Kebf5hMWsPrtLetB p{margin:0;}#mermaid-svg-Kebf5hMWsPrtLetB .label{font-family:Microsoft YaHei;color:#333;}#mermaid-svg-Kebf5hMWsPrtLetB .cluster-label text{fill:rgb(8.5000000002, 5.7500000001, 0);}#mermaid-svg-Kebf5hMWsPrtLetB .cluster-label span{color:rgb(8.5000000002, 5.7500000001, 0);}#mermaid-svg-Kebf5hMWsPrtLetB .cluster-label span p{background-color:transparent;}#mermaid-svg-Kebf5hMWsPrtLetB .label text,#mermaid-svg-Kebf5hMWsPrtLetB span{fill:#333;color:#333;}#mermaid-svg-Kebf5hMWsPrtLetB .node rect,#mermaid-svg-Kebf5hMWsPrtLetB .node circle,#mermaid-svg-Kebf5hMWsPrtLetB .node ellipse,#mermaid-svg-Kebf5hMWsPrtLetB .node polygon,#mermaid-svg-Kebf5hMWsPrtLetB .node path{fill:#fff4dd;stroke:hsl(40.5882352941, 60%, 83.3333333333%);stroke-width:1px;}#mermaid-svg-Kebf5hMWsPrtLetB .rough-node .label text,#mermaid-svg-Kebf5hMWsPrtLetB .node .label text,#mermaid-svg-Kebf5hMWsPrtLetB .image-shape .label,#mermaid-svg-Kebf5hMWsPrtLetB .icon-shape .label{text-anchor:middle;}#mermaid-svg-Kebf5hMWsPrtLetB .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-Kebf5hMWsPrtLetB .rough-node .label,#mermaid-svg-Kebf5hMWsPrtLetB .node .label,#mermaid-svg-Kebf5hMWsPrtLetB .image-shape .label,#mermaid-svg-Kebf5hMWsPrtLetB .icon-shape .label{text-align:center;}#mermaid-svg-Kebf5hMWsPrtLetB .node.clickable{cursor:pointer;}#mermaid-svg-Kebf5hMWsPrtLetB .root .anchor path{fill:#0b0b0b!important;stroke-width:0;stroke:#0b0b0b;}#mermaid-svg-Kebf5hMWsPrtLetB .arrowheadPath{fill:#0b0b0b;}#mermaid-svg-Kebf5hMWsPrtLetB .edgePath .path{stroke:#0b0b0b;stroke-width:2.0px;}#mermaid-svg-Kebf5hMWsPrtLetB .flowchart-link{stroke:#0b0b0b;fill:none;}#mermaid-svg-Kebf5hMWsPrtLetB .edgeLabel{background-color:hsl(-79.4117647059, 100%, 93.3333333333%);text-align:center;}#mermaid-svg-Kebf5hMWsPrtLetB .edgeLabel p{background-color:hsl(-79.4117647059, 100%, 93.3333333333%);}#mermaid-svg-Kebf5hMWsPrtLetB .edgeLabel rect{opacity:0.5;background-color:hsl(-79.4117647059, 100%, 93.3333333333%);fill:hsl(-79.4117647059, 100%, 93.3333333333%);}#mermaid-svg-Kebf5hMWsPrtLetB .labelBkg{background-color:rgba(243.9999999999, 220.9999999998, 255, 0.5);}#mermaid-svg-Kebf5hMWsPrtLetB .cluster rect{fill:hsl(220.5882352941, 100%, 98.3333333333%);stroke:hsl(220.5882352941, 60%, 88.3333333333%);stroke-width:1px;}#mermaid-svg-Kebf5hMWsPrtLetB .cluster text{fill:rgb(8.5000000002, 5.7500000001, 0);}#mermaid-svg-Kebf5hMWsPrtLetB .cluster span{color:rgb(8.5000000002, 5.7500000001, 0);}#mermaid-svg-Kebf5hMWsPrtLetB div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:Microsoft YaHei;font-size:12px;background:hsl(220.5882352941, 100%, 98.3333333333%);border:1px solid hsl(220.5882352941, 60%, 88.3333333333%);border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-Kebf5hMWsPrtLetB .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-Kebf5hMWsPrtLetB rect.text{fill:none;stroke-width:0;}#mermaid-svg-Kebf5hMWsPrtLetB .icon-shape,#mermaid-svg-Kebf5hMWsPrtLetB .image-shape{background-color:hsl(-79.4117647059, 100%, 93.3333333333%);text-align:center;}#mermaid-svg-Kebf5hMWsPrtLetB .icon-shape p,#mermaid-svg-Kebf5hMWsPrtLetB .image-shape p{background-color:hsl(-79.4117647059, 100%, 93.3333333333%);padding:2px;}#mermaid-svg-Kebf5hMWsPrtLetB .icon-shape .label rect,#mermaid-svg-Kebf5hMWsPrtLetB .image-shape .label rect{opacity:0.5;background-color:hsl(-79.4117647059, 100%, 93.3333333333%);fill:hsl(-79.4117647059, 100%, 93.3333333333%);}#mermaid-svg-Kebf5hMWsPrtLetB .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-Kebf5hMWsPrtLetB .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-Kebf5hMWsPrtLetB :root{--mermaid-font-family:"trebuchet ms",verdana,arial,sans-serif;} JVM 排查
① 方法论

先保现场 · 先分诊 · 数据说话
② 工具链

JDK 自带 / Arthas / MAT / 事前监控
③ 六大 SOP
CPU 100%

top -Hp → jstack
内存泄漏

jstat → dump → MAT
频繁 GC

日志分型对症
死锁卡顿

jstack -b 三次对比
exit 137

算容器内存账
堆外泄漏

NMT diff

6.2 核心要点浓缩（十条）

  1. 三原则：先保现场再动手、先分诊后深入、数据说话------重启是最后手段。
  2. 30 秒分诊：top（CPU us/sy）→ 响应慢看线程状态 → 进程没了看 exit code → GC 日志看曲线。
  3. CPU 100% 三板斧 ：top -Hp 找热线程 → 转十六进制 → jstack 对栈（Arthas thread -n 3 一步到位）；GC 线程栈 = 转 GC 问题。
  4. 内存泄漏判定：Full GC 后老年代仍爬升才是泄漏；MAT 三板斧 = Leak Suspects + Dominator Tree + Path to GC Roots。
  5. dump 纪律 ：大堆 dump 会 STW 数十秒------先摘流量；jmap -histo:live 会触发 Full GC，垂死服务慎用。
  6. G1 的 Full GC 必有原因字段：Humongous/Metadata/Ergonomics/System.gc------按字段对症，别盲目调参。
  7. 线程卡顿看状态：BLOCKED 找锁持有者、socket read WAITING = 下游无超时、三次 dump 对比区分"卡住"与"慢"。
  8. exit 137 = 内核击杀非 JVM OOM：算容器内存账（堆+元空间+栈+直接内存+杂项 ≤ limit）。
  9. 堆外泄漏两步定位：RSS-堆差值确认 → NMT baseline/diff 定区域；Netty 用自带泄漏检测器。
  10. 事前准备决定排查速度：GC 日志、HeapDumpOnOOM、NMT、监控四件套------事故来临时没有第二次配置机会。

📌 最后一句话 ：排查能力 = 原理地图 × 工具熟练度 × 现场经验。阶段一的五篇原理文章给了你地图，本篇给了你 SOP，剩下的现场经验------希望永远用不上，但必须时刻准备着。

📌 配套阅读：

上一篇：《01-05-A-类加载机制详解.md》

 B 篇：《01-06-B-JVM排查实战面试与生产事故实战.md》

下一篇：《02-01-A-JMM与volatile-synchronized详解.md》

如果这篇文章对你有帮助，欢迎点赞、收藏、关注！

相关推荐
vipxieliang3 小时前
ValidX 在 DDD 领域驱动设计中的实践
java·spring boot
ba_pi3 小时前
mysql 查询所有表名并授权
java
ProcessOn官方账号3 小时前
java函数式编程--入门基础
java·编程·函数式编程
一嘴一个橘子4 小时前
优惠券秒杀思路，redis 分布式锁 方式
java
xufengzhu4 小时前
Java 虚拟线程实战指南（JDK 21+）
java·虚拟线程·jdk21+
大猫和小黄4 小时前
吃透Java泛型：从原理避坑到实战落地
java·泛型
Bingo_BIG5 小时前
Java Spring框架：单表的查询、新增、修改、删除、导入、导出
java·spring·前后端
程序猿_极客5 小时前
【免费】分享一套优质的基于SSM的校园失物招领系统的设计与实现，源码+文档+视频详解（讲解）
java·mysql·ssm·课程设计·失物招领系统
白远山6 小时前
上海24小时自助健身房解决方案实战指南与经验分享
java·数据库·架构·需求分析
热门推荐
01国内可直接用、免费额度/永久免费的大模型API清单（含 SiliconFlow、火山、阿里、智谱、百度、Kimi、DeepSeek、DMXAPI 等）02历年考研数学一、数学二、数学三真题试卷及答案PDF03GitHub 镜像站点04蜘蛛磁力 搜索引擎大全，如何使用蜘蛛磁力查找磁力链接05微信历史版本含下载地址（ Windows PC | 安卓 | MAC ）及设置微信不更新062026 国产 AI 大模型横评：DeepSeek、通义千问、Kimi、文心一言、星火、豆包谁更能打？07人工智能（AI）发展史：详细里程碑08ZCode 周末送额度活动开启：3 亿 Token 免费领取09Telegram APK 下载安装完整指南 — 2026年最新102026 爱知・名古屋亚运会怎么看？直播平台+赛程安排分享