SpringBoot 生产环境踩坑实录：10个真实故障案例与教训

行者-全栈开发2026-10-12 9:12

摘要: 生产环境的故障往往来自意想不到的角落。本文收录了10个真实的SpringBoot生产故障案例，涵盖OOM内存溢出、线程池耗尽、数据库连接泄漏、缓存雪崩、死锁、配置错误等典型问题。每个案例都包含完整的故障现象、排查过程、根因分析和解决方案，帮助开发者提前规避风险，快速定位问题。适合1-5年经验的Java后端开发者和运维工程师阅读。
时效性说明: 本文基于 SpringBoot 3.x + JDK 17 生产环境实际案例整理（2026年6月）。故障排查方法论和解决方案具有通用性，但具体配置参数会随版本调整，使用时请以实际环境为准。
运行环境: SpringBoot 3.x、JDK 17+、MySQL 8.0+、Redis 7.0+

场景化开篇

你是否经历过这些生产事故？

场景一：半夜被电话叫醒

  • 凌晨3点，收到告警短信：服务不可用
  • 起床打开电脑，发现OOM导致服务崩溃
  • 紧急重启服务，但很快又崩溃
  • 排查2小时才发现是内存泄漏
  • 运维OS：能不能提前发现问题？

场景二：大促期间系统崩溃

  • 双11零点，流量激增10倍
  • 线程池瞬间耗尽，大量请求超时
  • 用户投诉不断，订单流失严重
  • 紧急扩容，但来不及了
  • 技术负责人OS：为什么压测没发现？

场景三：数据库连接泄漏

  • 系统运行一段时间后，数据库连接耗尽
  • 新请求无法获取连接，报错超时
  • 重启服务后恢复，但过几天又出现
  • 排查发现是连接未正确关闭
  • 开发OS：代码明明有关闭连接啊？

场景四：配置错误导致故障

  • 修改了一个配置参数，重新部署
  • 服务启动成功，但功能异常
  • 回滚配置后恢复，但不知道原因
  • 架构师OS：配置变更怎么测试？

生产故障成本分析（按电商场景计算）

故障影响统计：

故障类型 平均恢复时间 直接损失 间接损失 年度成本
OOM崩溃 30分钟 5万元 品牌损失 60万元
线程池耗尽 15分钟 2万元 用户流失 24万元
连接池泄漏 60分钟 10万元 运维成本 120万元
缓存雪崩 45分钟 8万元 数据修复 96万元
死锁问题 30分钟 5万元 排查成本 60万元
年度总损失 - - - 360万元

预防措施收益：

措施 投入成本 预防效果 ROI
监控告警 10万元 预防80%故障 28倍
代码审查 15万元 预防60%故障 14倍
压力测试 20万元 预防70%故障 12倍
配置管理 5万元 预防90%配置错误 64倍

前言与概述

在前二十九章中，我完成了SpringBoot从环境搭建到分布式事务的完整学习路径。今天我会探讨生产环境的真实挑战------故障案例与教训。生产环境的故障往往来自意想不到的角落，本文收录的10个真实案例，将帮助你提前规避风险，快速定位问题。

故障分类概览

#mermaid-svg-6bOsCDMAUPLhL16U{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-6bOsCDMAUPLhL16U .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-6bOsCDMAUPLhL16U .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-6bOsCDMAUPLhL16U .error-icon{fill:#552222;}#mermaid-svg-6bOsCDMAUPLhL16U .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-6bOsCDMAUPLhL16U .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-6bOsCDMAUPLhL16U .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-6bOsCDMAUPLhL16U .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-6bOsCDMAUPLhL16U .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-6bOsCDMAUPLhL16U .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-6bOsCDMAUPLhL16U .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-6bOsCDMAUPLhL16U .marker{fill:#333333;stroke:#333333;}#mermaid-svg-6bOsCDMAUPLhL16U .marker.cross{stroke:#333333;}#mermaid-svg-6bOsCDMAUPLhL16U svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-6bOsCDMAUPLhL16U p{margin:0;}#mermaid-svg-6bOsCDMAUPLhL16U .edge{stroke-width:3;}#mermaid-svg-6bOsCDMAUPLhL16U .section--1 rect,#mermaid-svg-6bOsCDMAUPLhL16U .section--1 path,#mermaid-svg-6bOsCDMAUPLhL16U .section--1 circle,#mermaid-svg-6bOsCDMAUPLhL16U .section--1 polygon,#mermaid-svg-6bOsCDMAUPLhL16U .section--1 path{fill:hsl(240, 100%, 76.2745098039%);}#mermaid-svg-6bOsCDMAUPLhL16U .section--1 text{fill:#ffffff;}#mermaid-svg-6bOsCDMAUPLhL16U .node-icon--1{font-size:40px;color:#ffffff;}#mermaid-svg-6bOsCDMAUPLhL16U .section-edge--1{stroke:hsl(240, 100%, 76.2745098039%);}#mermaid-svg-6bOsCDMAUPLhL16U .edge-depth--1{stroke-width:17;}#mermaid-svg-6bOsCDMAUPLhL16U .section--1 line{stroke:hsl(60, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-6bOsCDMAUPLhL16U .disabled,#mermaid-svg-6bOsCDMAUPLhL16U .disabled circle,#mermaid-svg-6bOsCDMAUPLhL16U .disabled text{fill:lightgray;}#mermaid-svg-6bOsCDMAUPLhL16U .disabled text{fill:#efefef;}#mermaid-svg-6bOsCDMAUPLhL16U .section-0 rect,#mermaid-svg-6bOsCDMAUPLhL16U .section-0 path,#mermaid-svg-6bOsCDMAUPLhL16U .section-0 circle,#mermaid-svg-6bOsCDMAUPLhL16U .section-0 polygon,#mermaid-svg-6bOsCDMAUPLhL16U .section-0 path{fill:hsl(60, 100%, 73.5294117647%);}#mermaid-svg-6bOsCDMAUPLhL16U .section-0 text{fill:black;}#mermaid-svg-6bOsCDMAUPLhL16U .node-icon-0{font-size:40px;color:black;}#mermaid-svg-6bOsCDMAUPLhL16U .section-edge-0{stroke:hsl(60, 100%, 73.5294117647%);}#mermaid-svg-6bOsCDMAUPLhL16U .edge-depth-0{stroke-width:14;}#mermaid-svg-6bOsCDMAUPLhL16U .section-0 line{stroke:hsl(240, 100%, 83.5294117647%);stroke-width:3;}#mermaid-svg-6bOsCDMAUPLhL16U .disabled,#mermaid-svg-6bOsCDMAUPLhL16U .disabled circle,#mermaid-svg-6bOsCDMAUPLhL16U .disabled text{fill:lightgray;}#mermaid-svg-6bOsCDMAUPLhL16U .disabled text{fill:#efefef;}#mermaid-svg-6bOsCDMAUPLhL16U .section-1 rect,#mermaid-svg-6bOsCDMAUPLhL16U .section-1 path,#mermaid-svg-6bOsCDMAUPLhL16U .section-1 circle,#mermaid-svg-6bOsCDMAUPLhL16U .section-1 polygon,#mermaid-svg-6bOsCDMAUPLhL16U .section-1 path{fill:hsl(80, 100%, 76.2745098039%);}#mermaid-svg-6bOsCDMAUPLhL16U .section-1 text{fill:black;}#mermaid-svg-6bOsCDMAUPLhL16U .node-icon-1{font-size:40px;color:black;}#mermaid-svg-6bOsCDMAUPLhL16U .section-edge-1{stroke:hsl(80, 100%, 76.2745098039%);}#mermaid-svg-6bOsCDMAUPLhL16U .edge-depth-1{stroke-width:11;}#mermaid-svg-6bOsCDMAUPLhL16U .section-1 line{stroke:hsl(260, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-6bOsCDMAUPLhL16U .disabled,#mermaid-svg-6bOsCDMAUPLhL16U .disabled circle,#mermaid-svg-6bOsCDMAUPLhL16U .disabled text{fill:lightgray;}#mermaid-svg-6bOsCDMAUPLhL16U .disabled text{fill:#efefef;}#mermaid-svg-6bOsCDMAUPLhL16U .section-2 rect,#mermaid-svg-6bOsCDMAUPLhL16U .section-2 path,#mermaid-svg-6bOsCDMAUPLhL16U .section-2 circle,#mermaid-svg-6bOsCDMAUPLhL16U .section-2 polygon,#mermaid-svg-6bOsCDMAUPLhL16U .section-2 path{fill:hsl(270, 100%, 76.2745098039%);}#mermaid-svg-6bOsCDMAUPLhL16U .section-2 text{fill:#ffffff;}#mermaid-svg-6bOsCDMAUPLhL16U .node-icon-2{font-size:40px;color:#ffffff;}#mermaid-svg-6bOsCDMAUPLhL16U .section-edge-2{stroke:hsl(270, 100%, 76.2745098039%);}#mermaid-svg-6bOsCDMAUPLhL16U .edge-depth-2{stroke-width:8;}#mermaid-svg-6bOsCDMAUPLhL16U .section-2 line{stroke:hsl(90, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-6bOsCDMAUPLhL16U .disabled,#mermaid-svg-6bOsCDMAUPLhL16U .disabled circle,#mermaid-svg-6bOsCDMAUPLhL16U .disabled text{fill:lightgray;}#mermaid-svg-6bOsCDMAUPLhL16U .disabled text{fill:#efefef;}#mermaid-svg-6bOsCDMAUPLhL16U .section-3 rect,#mermaid-svg-6bOsCDMAUPLhL16U .section-3 path,#mermaid-svg-6bOsCDMAUPLhL16U .section-3 circle,#mermaid-svg-6bOsCDMAUPLhL16U .section-3 polygon,#mermaid-svg-6bOsCDMAUPLhL16U .section-3 path{fill:hsl(300, 100%, 76.2745098039%);}#mermaid-svg-6bOsCDMAUPLhL16U .section-3 text{fill:black;}#mermaid-svg-6bOsCDMAUPLhL16U .node-icon-3{font-size:40px;color:black;}#mermaid-svg-6bOsCDMAUPLhL16U .section-edge-3{stroke:hsl(300, 100%, 76.2745098039%);}#mermaid-svg-6bOsCDMAUPLhL16U .edge-depth-3{stroke-width:5;}#mermaid-svg-6bOsCDMAUPLhL16U .section-3 line{stroke:hsl(120, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-6bOsCDMAUPLhL16U .disabled,#mermaid-svg-6bOsCDMAUPLhL16U .disabled circle,#mermaid-svg-6bOsCDMAUPLhL16U .disabled text{fill:lightgray;}#mermaid-svg-6bOsCDMAUPLhL16U .disabled text{fill:#efefef;}#mermaid-svg-6bOsCDMAUPLhL16U .section-4 rect,#mermaid-svg-6bOsCDMAUPLhL16U .section-4 path,#mermaid-svg-6bOsCDMAUPLhL16U .section-4 circle,#mermaid-svg-6bOsCDMAUPLhL16U .section-4 polygon,#mermaid-svg-6bOsCDMAUPLhL16U .section-4 path{fill:hsl(330, 100%, 76.2745098039%);}#mermaid-svg-6bOsCDMAUPLhL16U .section-4 text{fill:black;}#mermaid-svg-6bOsCDMAUPLhL16U .node-icon-4{font-size:40px;color:black;}#mermaid-svg-6bOsCDMAUPLhL16U .section-edge-4{stroke:hsl(330, 100%, 76.2745098039%);}#mermaid-svg-6bOsCDMAUPLhL16U .edge-depth-4{stroke-width:2;}#mermaid-svg-6bOsCDMAUPLhL16U .section-4 line{stroke:hsl(150, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-6bOsCDMAUPLhL16U .disabled,#mermaid-svg-6bOsCDMAUPLhL16U .disabled circle,#mermaid-svg-6bOsCDMAUPLhL16U .disabled text{fill:lightgray;}#mermaid-svg-6bOsCDMAUPLhL16U .disabled text{fill:#efefef;}#mermaid-svg-6bOsCDMAUPLhL16U .section-5 rect,#mermaid-svg-6bOsCDMAUPLhL16U .section-5 path,#mermaid-svg-6bOsCDMAUPLhL16U .section-5 circle,#mermaid-svg-6bOsCDMAUPLhL16U .section-5 polygon,#mermaid-svg-6bOsCDMAUPLhL16U .section-5 path{fill:hsl(0, 100%, 76.2745098039%);}#mermaid-svg-6bOsCDMAUPLhL16U .section-5 text{fill:black;}#mermaid-svg-6bOsCDMAUPLhL16U .node-icon-5{font-size:40px;color:black;}#mermaid-svg-6bOsCDMAUPLhL16U .section-edge-5{stroke:hsl(0, 100%, 76.2745098039%);}#mermaid-svg-6bOsCDMAUPLhL16U .edge-depth-5{stroke-width:-1;}#mermaid-svg-6bOsCDMAUPLhL16U .section-5 line{stroke:hsl(180, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-6bOsCDMAUPLhL16U .disabled,#mermaid-svg-6bOsCDMAUPLhL16U .disabled circle,#mermaid-svg-6bOsCDMAUPLhL16U .disabled text{fill:lightgray;}#mermaid-svg-6bOsCDMAUPLhL16U .disabled text{fill:#efefef;}#mermaid-svg-6bOsCDMAUPLhL16U .section-6 rect,#mermaid-svg-6bOsCDMAUPLhL16U .section-6 path,#mermaid-svg-6bOsCDMAUPLhL16U .section-6 circle,#mermaid-svg-6bOsCDMAUPLhL16U .section-6 polygon,#mermaid-svg-6bOsCDMAUPLhL16U .section-6 path{fill:hsl(30, 100%, 76.2745098039%);}#mermaid-svg-6bOsCDMAUPLhL16U .section-6 text{fill:black;}#mermaid-svg-6bOsCDMAUPLhL16U .node-icon-6{font-size:40px;color:black;}#mermaid-svg-6bOsCDMAUPLhL16U .section-edge-6{stroke:hsl(30, 100%, 76.2745098039%);}#mermaid-svg-6bOsCDMAUPLhL16U .edge-depth-6{stroke-width:-4;}#mermaid-svg-6bOsCDMAUPLhL16U .section-6 line{stroke:hsl(210, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-6bOsCDMAUPLhL16U .disabled,#mermaid-svg-6bOsCDMAUPLhL16U .disabled circle,#mermaid-svg-6bOsCDMAUPLhL16U .disabled text{fill:lightgray;}#mermaid-svg-6bOsCDMAUPLhL16U .disabled text{fill:#efefef;}#mermaid-svg-6bOsCDMAUPLhL16U .section-7 rect,#mermaid-svg-6bOsCDMAUPLhL16U .section-7 path,#mermaid-svg-6bOsCDMAUPLhL16U .section-7 circle,#mermaid-svg-6bOsCDMAUPLhL16U .section-7 polygon,#mermaid-svg-6bOsCDMAUPLhL16U .section-7 path{fill:hsl(90, 100%, 76.2745098039%);}#mermaid-svg-6bOsCDMAUPLhL16U .section-7 text{fill:black;}#mermaid-svg-6bOsCDMAUPLhL16U .node-icon-7{font-size:40px;color:black;}#mermaid-svg-6bOsCDMAUPLhL16U .section-edge-7{stroke:hsl(90, 100%, 76.2745098039%);}#mermaid-svg-6bOsCDMAUPLhL16U .edge-depth-7{stroke-width:-7;}#mermaid-svg-6bOsCDMAUPLhL16U .section-7 line{stroke:hsl(270, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-6bOsCDMAUPLhL16U .disabled,#mermaid-svg-6bOsCDMAUPLhL16U .disabled circle,#mermaid-svg-6bOsCDMAUPLhL16U .disabled text{fill:lightgray;}#mermaid-svg-6bOsCDMAUPLhL16U .disabled text{fill:#efefef;}#mermaid-svg-6bOsCDMAUPLhL16U .section-8 rect,#mermaid-svg-6bOsCDMAUPLhL16U .section-8 path,#mermaid-svg-6bOsCDMAUPLhL16U .section-8 circle,#mermaid-svg-6bOsCDMAUPLhL16U .section-8 polygon,#mermaid-svg-6bOsCDMAUPLhL16U .section-8 path{fill:hsl(150, 100%, 76.2745098039%);}#mermaid-svg-6bOsCDMAUPLhL16U .section-8 text{fill:black;}#mermaid-svg-6bOsCDMAUPLhL16U .node-icon-8{font-size:40px;color:black;}#mermaid-svg-6bOsCDMAUPLhL16U .section-edge-8{stroke:hsl(150, 100%, 76.2745098039%);}#mermaid-svg-6bOsCDMAUPLhL16U .edge-depth-8{stroke-width:-10;}#mermaid-svg-6bOsCDMAUPLhL16U .section-8 line{stroke:hsl(330, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-6bOsCDMAUPLhL16U .disabled,#mermaid-svg-6bOsCDMAUPLhL16U .disabled circle,#mermaid-svg-6bOsCDMAUPLhL16U .disabled text{fill:lightgray;}#mermaid-svg-6bOsCDMAUPLhL16U .disabled text{fill:#efefef;}#mermaid-svg-6bOsCDMAUPLhL16U .section-9 rect,#mermaid-svg-6bOsCDMAUPLhL16U .section-9 path,#mermaid-svg-6bOsCDMAUPLhL16U .section-9 circle,#mermaid-svg-6bOsCDMAUPLhL16U .section-9 polygon,#mermaid-svg-6bOsCDMAUPLhL16U .section-9 path{fill:hsl(180, 100%, 76.2745098039%);}#mermaid-svg-6bOsCDMAUPLhL16U .section-9 text{fill:black;}#mermaid-svg-6bOsCDMAUPLhL16U .node-icon-9{font-size:40px;color:black;}#mermaid-svg-6bOsCDMAUPLhL16U .section-edge-9{stroke:hsl(180, 100%, 76.2745098039%);}#mermaid-svg-6bOsCDMAUPLhL16U .edge-depth-9{stroke-width:-13;}#mermaid-svg-6bOsCDMAUPLhL16U .section-9 line{stroke:hsl(0, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-6bOsCDMAUPLhL16U .disabled,#mermaid-svg-6bOsCDMAUPLhL16U .disabled circle,#mermaid-svg-6bOsCDMAUPLhL16U .disabled text{fill:lightgray;}#mermaid-svg-6bOsCDMAUPLhL16U .disabled text{fill:#efefef;}#mermaid-svg-6bOsCDMAUPLhL16U .section-10 rect,#mermaid-svg-6bOsCDMAUPLhL16U .section-10 path,#mermaid-svg-6bOsCDMAUPLhL16U .section-10 circle,#mermaid-svg-6bOsCDMAUPLhL16U .section-10 polygon,#mermaid-svg-6bOsCDMAUPLhL16U .section-10 path{fill:hsl(210, 100%, 76.2745098039%);}#mermaid-svg-6bOsCDMAUPLhL16U .section-10 text{fill:black;}#mermaid-svg-6bOsCDMAUPLhL16U .node-icon-10{font-size:40px;color:black;}#mermaid-svg-6bOsCDMAUPLhL16U .section-edge-10{stroke:hsl(210, 100%, 76.2745098039%);}#mermaid-svg-6bOsCDMAUPLhL16U .edge-depth-10{stroke-width:-16;}#mermaid-svg-6bOsCDMAUPLhL16U .section-10 line{stroke:hsl(30, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-6bOsCDMAUPLhL16U .disabled,#mermaid-svg-6bOsCDMAUPLhL16U .disabled circle,#mermaid-svg-6bOsCDMAUPLhL16U .disabled text{fill:lightgray;}#mermaid-svg-6bOsCDMAUPLhL16U .disabled text{fill:#efefef;}#mermaid-svg-6bOsCDMAUPLhL16U .section-root rect,#mermaid-svg-6bOsCDMAUPLhL16U .section-root path,#mermaid-svg-6bOsCDMAUPLhL16U .section-root circle,#mermaid-svg-6bOsCDMAUPLhL16U .section-root polygon{fill:hsl(240, 100%, 46.2745098039%);}#mermaid-svg-6bOsCDMAUPLhL16U .section-root text{fill:#ffffff;}#mermaid-svg-6bOsCDMAUPLhL16U .section-root span{color:#ffffff;}#mermaid-svg-6bOsCDMAUPLhL16U .section-2 span{color:#ffffff;}#mermaid-svg-6bOsCDMAUPLhL16U .icon-container{height:100%;display:flex;justify-content:center;align-items:center;}#mermaid-svg-6bOsCDMAUPLhL16U .edge{fill:none;}#mermaid-svg-6bOsCDMAUPLhL16U .mindmap-node-label{dy:1em;alignment-baseline:middle;text-anchor:middle;dominant-baseline:middle;text-align:center;}#mermaid-svg-6bOsCDMAUPLhL16U :root{--mermaid-font-family:"trebuchet ms",verdana,arial,sans-serif;} 生产故障
内存问题
OOM堆溢出
OOM元空间
内存泄漏
GC频繁
线程问题
线程池耗尽
死锁
线程泄漏
阻塞等待
数据库问题
连接池泄漏
慢SQL
锁等待
主从延迟
缓存问题
缓存穿透
缓存雪崩
缓存击穿
数据不一致
配置问题
参数错误
环境差异
版本冲突
依赖缺失

第一章：内存问题案例

1.1 案例一：List 无限增长导致 OOM

故障现象：

  • 服务运行2-3天后频繁Full GC
  • 最终抛出 java.lang.OutOfMemoryError: Java heap space
  • 重启后恢复正常，但过几天又出现

排查过程：

java 复制代码 
// 问题代码
@Service
public class ReportService {

    // 问题：静态集合持续增长，永远不会释放
    private static final List<ReportRecord> records = new ArrayList<>();

    public void generateReport(ReportRequest request) {
        // 每次请求都往 list 中添加数据
        ReportRecord record = buildRecord(request);
        records.add(record);  // 持续增长，永不清理

        // 处理报告
        processReport(records);
    }
}

// 使用 jmap 分析堆内存
// jmap -dump:format=b,file=heap.hprof <pid>
// 使用 MAT 工具分析，发现 ArrayList 占用 2GB 内存

根因分析：
#mermaid-svg-gMJIVRzLtKQhMfcB{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-gMJIVRzLtKQhMfcB .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-gMJIVRzLtKQhMfcB .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-gMJIVRzLtKQhMfcB .error-icon{fill:#552222;}#mermaid-svg-gMJIVRzLtKQhMfcB .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-gMJIVRzLtKQhMfcB .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-gMJIVRzLtKQhMfcB .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-gMJIVRzLtKQhMfcB .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-gMJIVRzLtKQhMfcB .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-gMJIVRzLtKQhMfcB .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-gMJIVRzLtKQhMfcB .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-gMJIVRzLtKQhMfcB .marker{fill:#333333;stroke:#333333;}#mermaid-svg-gMJIVRzLtKQhMfcB .marker.cross{stroke:#333333;}#mermaid-svg-gMJIVRzLtKQhMfcB svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-gMJIVRzLtKQhMfcB p{margin:0;}#mermaid-svg-gMJIVRzLtKQhMfcB .label{font-family:"trebuchet ms",verdana,arial,sans-serif;color:#333;}#mermaid-svg-gMJIVRzLtKQhMfcB .cluster-label text{fill:#333;}#mermaid-svg-gMJIVRzLtKQhMfcB .cluster-label span{color:#333;}#mermaid-svg-gMJIVRzLtKQhMfcB .cluster-label span p{background-color:transparent;}#mermaid-svg-gMJIVRzLtKQhMfcB .label text,#mermaid-svg-gMJIVRzLtKQhMfcB span{fill:#333;color:#333;}#mermaid-svg-gMJIVRzLtKQhMfcB .node rect,#mermaid-svg-gMJIVRzLtKQhMfcB .node circle,#mermaid-svg-gMJIVRzLtKQhMfcB .node ellipse,#mermaid-svg-gMJIVRzLtKQhMfcB .node polygon,#mermaid-svg-gMJIVRzLtKQhMfcB .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-gMJIVRzLtKQhMfcB .rough-node .label text,#mermaid-svg-gMJIVRzLtKQhMfcB .node .label text,#mermaid-svg-gMJIVRzLtKQhMfcB .image-shape .label,#mermaid-svg-gMJIVRzLtKQhMfcB .icon-shape .label{text-anchor:middle;}#mermaid-svg-gMJIVRzLtKQhMfcB .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-gMJIVRzLtKQhMfcB .rough-node .label,#mermaid-svg-gMJIVRzLtKQhMfcB .node .label,#mermaid-svg-gMJIVRzLtKQhMfcB .image-shape .label,#mermaid-svg-gMJIVRzLtKQhMfcB .icon-shape .label{text-align:center;}#mermaid-svg-gMJIVRzLtKQhMfcB .node.clickable{cursor:pointer;}#mermaid-svg-gMJIVRzLtKQhMfcB .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-gMJIVRzLtKQhMfcB .arrowheadPath{fill:#333333;}#mermaid-svg-gMJIVRzLtKQhMfcB .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-gMJIVRzLtKQhMfcB .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-gMJIVRzLtKQhMfcB .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-gMJIVRzLtKQhMfcB .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-gMJIVRzLtKQhMfcB .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-gMJIVRzLtKQhMfcB .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-gMJIVRzLtKQhMfcB .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-gMJIVRzLtKQhMfcB .cluster text{fill:#333;}#mermaid-svg-gMJIVRzLtKQhMfcB .cluster span{color:#333;}#mermaid-svg-gMJIVRzLtKQhMfcB div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-gMJIVRzLtKQhMfcB .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-gMJIVRzLtKQhMfcB rect.text{fill:none;stroke-width:0;}#mermaid-svg-gMJIVRzLtKQhMfcB .icon-shape,#mermaid-svg-gMJIVRzLtKQhMfcB .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-gMJIVRzLtKQhMfcB .icon-shape p,#mermaid-svg-gMJIVRzLtKQhMfcB .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-gMJIVRzLtKQhMfcB .icon-shape .label rect,#mermaid-svg-gMJIVRzLtKQhMfcB .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-gMJIVRzLtKQhMfcB .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-gMJIVRzLtKQhMfcB .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-gMJIVRzLtKQhMfcB :root{--mermaid-font-family:"trebuchet ms",verdana,arial,sans-serif;} 请求1
添加记录1
请求2
添加记录2
请求N
添加记录N
List 持续增长
内存耗尽 OOM

解决方案：

java 复制代码 
// 方案一：使用有界集合
@Service
public class FixedReportService {

    // 使用固定大小的队列，自动淘汰旧数据
    private final Deque<ReportRecord> records = new ArrayDeque<>(10000);

    public synchronized void generateReport(ReportRequest request) {
        ReportRecord record = buildRecord(request);
        records.addLast(record);

        // 超过容量限制，移除最旧的数据
        while (records.size() > 10000) {
            records.removeFirst();
        }

        processReport(new ArrayList<>(records));
    }
}

// 方案二：定期清理
@Service
public class FixedReportService2 {

    private final List<ReportRecord> records = new ArrayList<>();

    // 每天凌晨2点清理旧数据
    @Scheduled(cron = "0 0 2 * * *")
    public void cleanup() {
        LocalDate yesterday = LocalDate.now().minusDays(1);
        records.removeIf(r -> r.getCreatedAt().toLocalDate().isBefore(yesterday));
        log.info("清理完成，剩余记录数: {}", records.size());
    }
}

1.2 案例二：ThreadLocal 未清理导致内存泄漏

故障现象：

  • 每次请求处理后内存不释放
  • 长时间运行后元空间溢出
  • 报错 java.lang.OutOfMemoryError: Metaspace

排查过程：

java 复制代码 
// 问题代码
public class UserContext {
    // 问题：ThreadLocal 使用后未清理
    private static final ThreadLocal<User> currentUser = new ThreadLocal<>();

    public static void setUser(User user) {
        currentUser.set(user);
    }

    public static User getUser() {
        return currentUser.get();
    }

    // 缺少 remove() 调用
}

// 在 Filter 中使用
public class AuthFilter implements Filter {
    @Override
    public void doFilter(ServletRequest request, ServletResponse response, FilterChain chain) {
        User user = extractUser(request);
        UserContext.setUser(user);

        chain.doFilter(request, response);

        // 问题：没有调用 UserContext.remove()
        // ThreadLocal 中的 User 对象不会被回收
    }
}

根因分析：
GC ThreadLocal 线程池线程 GC ThreadLocal 线程池线程 #mermaid-svg-LKLSm5BHkgIsOZfX{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-LKLSm5BHkgIsOZfX .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-LKLSm5BHkgIsOZfX .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-LKLSm5BHkgIsOZfX .error-icon{fill:#552222;}#mermaid-svg-LKLSm5BHkgIsOZfX .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-LKLSm5BHkgIsOZfX .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-LKLSm5BHkgIsOZfX .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-LKLSm5BHkgIsOZfX .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-LKLSm5BHkgIsOZfX .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-LKLSm5BHkgIsOZfX .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-LKLSm5BHkgIsOZfX .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-LKLSm5BHkgIsOZfX .marker{fill:#333333;stroke:#333333;}#mermaid-svg-LKLSm5BHkgIsOZfX .marker.cross{stroke:#333333;}#mermaid-svg-LKLSm5BHkgIsOZfX svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-LKLSm5BHkgIsOZfX p{margin:0;}#mermaid-svg-LKLSm5BHkgIsOZfX .actor{stroke:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);fill:#ECECFF;}#mermaid-svg-LKLSm5BHkgIsOZfX text.actor>tspan{fill:black;stroke:none;}#mermaid-svg-LKLSm5BHkgIsOZfX .actor-line{stroke:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);}#mermaid-svg-LKLSm5BHkgIsOZfX .innerArc{stroke-width:1.5;stroke-dasharray:none;}#mermaid-svg-LKLSm5BHkgIsOZfX .messageLine0{stroke-width:1.5;stroke-dasharray:none;stroke:#333;}#mermaid-svg-LKLSm5BHkgIsOZfX .messageLine1{stroke-width:1.5;stroke-dasharray:2,2;stroke:#333;}#mermaid-svg-LKLSm5BHkgIsOZfX #arrowhead path{fill:#333;stroke:#333;}#mermaid-svg-LKLSm5BHkgIsOZfX .sequenceNumber{fill:white;}#mermaid-svg-LKLSm5BHkgIsOZfX #sequencenumber{fill:#333;}#mermaid-svg-LKLSm5BHkgIsOZfX #crosshead path{fill:#333;stroke:#333;}#mermaid-svg-LKLSm5BHkgIsOZfX .messageText{fill:#333;stroke:none;}#mermaid-svg-LKLSm5BHkgIsOZfX .labelBox{stroke:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);fill:#ECECFF;}#mermaid-svg-LKLSm5BHkgIsOZfX .labelText,#mermaid-svg-LKLSm5BHkgIsOZfX .labelText>tspan{fill:black;stroke:none;}#mermaid-svg-LKLSm5BHkgIsOZfX .loopText,#mermaid-svg-LKLSm5BHkgIsOZfX .loopText>tspan{fill:black;stroke:none;}#mermaid-svg-LKLSm5BHkgIsOZfX .loopLine{stroke-width:2px;stroke-dasharray:2,2;stroke:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);fill:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);}#mermaid-svg-LKLSm5BHkgIsOZfX .note{stroke:#aaaa33;fill:#fff5ad;}#mermaid-svg-LKLSm5BHkgIsOZfX .noteText,#mermaid-svg-LKLSm5BHkgIsOZfX .noteText>tspan{fill:black;stroke:none;}#mermaid-svg-LKLSm5BHkgIsOZfX .activation0{fill:#f4f4f4;stroke:#666;}#mermaid-svg-LKLSm5BHkgIsOZfX .activation1{fill:#f4f4f4;stroke:#666;}#mermaid-svg-LKLSm5BHkgIsOZfX .activation2{fill:#f4f4f4;stroke:#666;}#mermaid-svg-LKLSm5BHkgIsOZfX .actorPopupMenu{position:absolute;}#mermaid-svg-LKLSm5BHkgIsOZfX .actorPopupMenuPanel{position:absolute;fill:#ECECFF;box-shadow:0px 8px 16px 0px rgba(0,0,0,0.2);filter:drop-shadow(3px 5px 2px rgb(0 0 0 / 0.4));}#mermaid-svg-LKLSm5BHkgIsOZfX .actor-man line{stroke:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);fill:#ECECFF;}#mermaid-svg-LKLSm5BHkgIsOZfX .actor-man circle,#mermaid-svg-LKLSm5BHkgIsOZfX line{stroke:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);fill:#ECECFF;stroke-width:2px;}#mermaid-svg-LKLSm5BHkgIsOZfX :root{--mermaid-font-family:"trebuchet ms",verdana,arial,sans-serif;} User 对象被引用 ThreadLocal 未清理 无法回收 User 对象 线程复用导致内存泄漏 set(User)处理请求返回线程池

解决方案：

java 复制代码 
// 正确的 ThreadLocal 使用方式
public class UserContext {
    private static final ThreadLocal<User> currentUser = new ThreadLocal<>();

    public static void setUser(User user) {
        currentUser.set(user);
    }

    public static User getUser() {
        return currentUser.get();
    }

    // 必须在 finally 中清理
    public static void clear() {
        currentUser.remove();
    }
}

// 在 Filter 中正确清理
public class AuthFilter implements Filter {
    @Override
    public void doFilter(ServletRequest request, ServletResponse response, FilterChain chain) {
        try {
            User user = extractUser(request);
            UserContext.setUser(user);
            chain.doFilter(request, response);
        } finally {
            UserContext.clear();  // 必须清理
        }
    }
}

第二章：线程问题案例

2.1 案例三：线程池配置不当导致请求拒绝

故障现象：

  • 高并发时大量请求被拒绝
  • 报错 java.util.concurrent.RejectedExecutionException
  • 但CPU使用率只有30%

排查过程：

java 复制代码 
// 问题代码
@Configuration
public class ThreadPoolConfig {

    @Bean
    public ExecutorService taskExecutor() {
        // 问题：线程池太小，队列太短
        return new ThreadPoolExecutor(
            5,                      // 核心线程数
            10,                     // 最大线程数
            60, TimeUnit.SECONDS,
            new ArrayBlockingQueue<>(100),  // 队列容量100
            new ThreadPoolExecutor.AbortPolicy()  // 直接拒绝
        );
    }
}

// 并发1000时的情况：
// - 5个核心线程处理任务
// - 100个任务进入队列等待
// - 895个任务被拒绝

根因分析：
#mermaid-svg-dAuwhJ1yCGvxU92k{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-dAuwhJ1yCGvxU92k .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-dAuwhJ1yCGvxU92k .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-dAuwhJ1yCGvxU92k .error-icon{fill:#552222;}#mermaid-svg-dAuwhJ1yCGvxU92k .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-dAuwhJ1yCGvxU92k .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-dAuwhJ1yCGvxU92k .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-dAuwhJ1yCGvxU92k .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-dAuwhJ1yCGvxU92k .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-dAuwhJ1yCGvxU92k .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-dAuwhJ1yCGvxU92k .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-dAuwhJ1yCGvxU92k .marker{fill:#333333;stroke:#333333;}#mermaid-svg-dAuwhJ1yCGvxU92k .marker.cross{stroke:#333333;}#mermaid-svg-dAuwhJ1yCGvxU92k svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-dAuwhJ1yCGvxU92k p{margin:0;}#mermaid-svg-dAuwhJ1yCGvxU92k .label{font-family:"trebuchet ms",verdana,arial,sans-serif;color:#333;}#mermaid-svg-dAuwhJ1yCGvxU92k .cluster-label text{fill:#333;}#mermaid-svg-dAuwhJ1yCGvxU92k .cluster-label span{color:#333;}#mermaid-svg-dAuwhJ1yCGvxU92k .cluster-label span p{background-color:transparent;}#mermaid-svg-dAuwhJ1yCGvxU92k .label text,#mermaid-svg-dAuwhJ1yCGvxU92k span{fill:#333;color:#333;}#mermaid-svg-dAuwhJ1yCGvxU92k .node rect,#mermaid-svg-dAuwhJ1yCGvxU92k .node circle,#mermaid-svg-dAuwhJ1yCGvxU92k .node ellipse,#mermaid-svg-dAuwhJ1yCGvxU92k .node polygon,#mermaid-svg-dAuwhJ1yCGvxU92k .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-dAuwhJ1yCGvxU92k .rough-node .label text,#mermaid-svg-dAuwhJ1yCGvxU92k .node .label text,#mermaid-svg-dAuwhJ1yCGvxU92k .image-shape .label,#mermaid-svg-dAuwhJ1yCGvxU92k .icon-shape .label{text-anchor:middle;}#mermaid-svg-dAuwhJ1yCGvxU92k .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-dAuwhJ1yCGvxU92k .rough-node .label,#mermaid-svg-dAuwhJ1yCGvxU92k .node .label,#mermaid-svg-dAuwhJ1yCGvxU92k .image-shape .label,#mermaid-svg-dAuwhJ1yCGvxU92k .icon-shape .label{text-align:center;}#mermaid-svg-dAuwhJ1yCGvxU92k .node.clickable{cursor:pointer;}#mermaid-svg-dAuwhJ1yCGvxU92k .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-dAuwhJ1yCGvxU92k .arrowheadPath{fill:#333333;}#mermaid-svg-dAuwhJ1yCGvxU92k .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-dAuwhJ1yCGvxU92k .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-dAuwhJ1yCGvxU92k .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-dAuwhJ1yCGvxU92k .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-dAuwhJ1yCGvxU92k .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-dAuwhJ1yCGvxU92k .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-dAuwhJ1yCGvxU92k .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-dAuwhJ1yCGvxU92k .cluster text{fill:#333;}#mermaid-svg-dAuwhJ1yCGvxU92k .cluster span{color:#333;}#mermaid-svg-dAuwhJ1yCGvxU92k div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-dAuwhJ1yCGvxU92k .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-dAuwhJ1yCGvxU92k rect.text{fill:none;stroke-width:0;}#mermaid-svg-dAuwhJ1yCGvxU92k .icon-shape,#mermaid-svg-dAuwhJ1yCGvxU92k .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-dAuwhJ1yCGvxU92k .icon-shape p,#mermaid-svg-dAuwhJ1yCGvxU92k .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-dAuwhJ1yCGvxU92k .icon-shape .label rect,#mermaid-svg-dAuwhJ1yCGvxU92k .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-dAuwhJ1yCGvxU92k .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-dAuwhJ1yCGvxU92k .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-dAuwhJ1yCGvxU92k :root{--mermaid-font-family:"trebuchet ms",verdana,arial,sans-serif;} 1000个请求
5个核心线程
100个队列等待
895个请求被拒绝

解决方案：

java 复制代码 
// 方案一：调整线程池参数
@Configuration
public class FixedThreadPoolConfig {

    @Bean
    public ExecutorService taskExecutor() {
        // 根据业务特点调整参数
        int cpuCores = Runtime.getRuntime().availableProcessors();
        int ioRatio = 90;  // IO密集型，IO时间占比90%

        // 计算最优线程数
        int optimalThreads = cpuCores * (1 + ioRatio / (100 - ioRatio));

        return new ThreadPoolExecutor(
            optimalThreads,              // 核心线程数
            optimalThreads * 2,          // 最大线程数
            60, TimeUnit.SECONDS,
            new LinkedBlockingQueue<>(10000),  // 更大的队列
            new ThreadPoolExecutor.CallerRunsPolicy()  // 调用者执行策略
        );
    }
}

// 方案二：使用动态线程池
@Configuration
public class DynamicThreadPoolConfig {

    @Bean
    public ThreadPoolTaskExecutor taskExecutor() {
        ThreadPoolTaskExecutor executor = new ThreadPoolTaskExecutor();
        executor.setCorePoolSize(10);
        executor.setMaxPoolSize(50);
        executor.setQueueCapacity(1000);
        executor.setKeepAliveSeconds(60);
        executor.setRejectedExecutionHandler(new ThreadPoolExecutor.CallerRunsPolicy());

        // 开启动态调整
        executor.setAllowCoreThreadTimeOut(true);
        return executor;
    }
}

2.2 案例四：数据库死锁导致系统卡死

故障现象：

  • 系统响应突然变慢，大量请求超时
  • 数据库CPU使用率飙升
  • 日志出现 Lock wait timeout exceeded

排查过程：

java 复制代码 
// 问题代码：事务嵌套导致死锁
@Service
public class OrderService {

    @Transactional
    public void createOrder(OrderRequest request) {
        // 事务1：锁住订单表
        orderMapper.insert(order);

        // 调用库存服务（新事务）
        storageService.deduct(request.getProductId(), request.getQuantity());
    }
}

@Service
public class StorageService {

    @Transactional
    public void deduct(Long productId, Integer quantity) {
        // 事务2：锁住库存表
        storageMapper.deduct(productId, quantity);

        // 调用订单服务（又获取订单表的锁）
        orderService.updateStatus(orderId, "DEDUCTED");
    }
}

// 死锁场景：
// 线程A：持有订单表锁 -> 等待库存表锁
// 线程B：持有库存表锁 -> 等待订单表锁

根因分析：
#mermaid-svg-pLPIhrpQf5CpT25g{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-pLPIhrpQf5CpT25g .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-pLPIhrpQf5CpT25g .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-pLPIhrpQf5CpT25g .error-icon{fill:#552222;}#mermaid-svg-pLPIhrpQf5CpT25g .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-pLPIhrpQf5CpT25g .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-pLPIhrpQf5CpT25g .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-pLPIhrpQf5CpT25g .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-pLPIhrpQf5CpT25g .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-pLPIhrpQf5CpT25g .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-pLPIhrpQf5CpT25g .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-pLPIhrpQf5CpT25g .marker{fill:#333333;stroke:#333333;}#mermaid-svg-pLPIhrpQf5CpT25g .marker.cross{stroke:#333333;}#mermaid-svg-pLPIhrpQf5CpT25g svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-pLPIhrpQf5CpT25g p{margin:0;}#mermaid-svg-pLPIhrpQf5CpT25g .label{font-family:"trebuchet ms",verdana,arial,sans-serif;color:#333;}#mermaid-svg-pLPIhrpQf5CpT25g .cluster-label text{fill:#333;}#mermaid-svg-pLPIhrpQf5CpT25g .cluster-label span{color:#333;}#mermaid-svg-pLPIhrpQf5CpT25g .cluster-label span p{background-color:transparent;}#mermaid-svg-pLPIhrpQf5CpT25g .label text,#mermaid-svg-pLPIhrpQf5CpT25g span{fill:#333;color:#333;}#mermaid-svg-pLPIhrpQf5CpT25g .node rect,#mermaid-svg-pLPIhrpQf5CpT25g .node circle,#mermaid-svg-pLPIhrpQf5CpT25g .node ellipse,#mermaid-svg-pLPIhrpQf5CpT25g .node polygon,#mermaid-svg-pLPIhrpQf5CpT25g .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-pLPIhrpQf5CpT25g .rough-node .label text,#mermaid-svg-pLPIhrpQf5CpT25g .node .label text,#mermaid-svg-pLPIhrpQf5CpT25g .image-shape .label,#mermaid-svg-pLPIhrpQf5CpT25g .icon-shape .label{text-anchor:middle;}#mermaid-svg-pLPIhrpQf5CpT25g .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-pLPIhrpQf5CpT25g .rough-node .label,#mermaid-svg-pLPIhrpQf5CpT25g .node .label,#mermaid-svg-pLPIhrpQf5CpT25g .image-shape .label,#mermaid-svg-pLPIhrpQf5CpT25g .icon-shape .label{text-align:center;}#mermaid-svg-pLPIhrpQf5CpT25g .node.clickable{cursor:pointer;}#mermaid-svg-pLPIhrpQf5CpT25g .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-pLPIhrpQf5CpT25g .arrowheadPath{fill:#333333;}#mermaid-svg-pLPIhrpQf5CpT25g .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-pLPIhrpQf5CpT25g .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-pLPIhrpQf5CpT25g .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-pLPIhrpQf5CpT25g .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-pLPIhrpQf5CpT25g .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-pLPIhrpQf5CpT25g .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-pLPIhrpQf5CpT25g .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-pLPIhrpQf5CpT25g .cluster text{fill:#333;}#mermaid-svg-pLPIhrpQf5CpT25g .cluster span{color:#333;}#mermaid-svg-pLPIhrpQf5CpT25g div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-pLPIhrpQf5CpT25g .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-pLPIhrpQf5CpT25g rect.text{fill:none;stroke-width:0;}#mermaid-svg-pLPIhrpQf5CpT25g .icon-shape,#mermaid-svg-pLPIhrpQf5CpT25g .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-pLPIhrpQf5CpT25g .icon-shape p,#mermaid-svg-pLPIhrpQf5CpT25g .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-pLPIhrpQf5CpT25g .icon-shape .label rect,#mermaid-svg-pLPIhrpQf5CpT25g .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-pLPIhrpQf5CpT25g .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-pLPIhrpQf5CpT25g .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-pLPIhrpQf5CpT25g :root{--mermaid-font-family:"trebuchet ms",verdana,arial,sans-serif;} 线程B
线程A
等待
等待
获取订单锁
等待库存锁
获取库存锁
等待订单锁

解决方案：

java 复制代码 
// 方案一：统一锁顺序
@Service
public class FixedOrderService {

    @Autowired
    private StorageService storageService;

    @Transactional
    public void createOrder(OrderRequest request) {
        // 统一按ID顺序获取锁
        Long smallerId = Math.min(orderId, productId);
        Long largerId = Math.max(orderId, productId);

        // 先锁较小的ID
        lock(smallerId);
        lock(largerId);

        try {
            orderMapper.insert(order);
            storageService.deduct(productId, quantity);
        } finally {
            unlock(largerId);
            unlock(smallerId);
        }
    }
}

// 方案二：减少事务范围
@Service
public class FixedOrderService2 {

    @Transactional
    public void createOrder(OrderRequest request) {
        // 只在必要的地方使用事务
        orderMapper.insert(order);
    }

    // 库存扣减放在事务外
    public void deductStorage(Long productId, Integer quantity) {
        storageService.deduct(productId, quantity);
    }
}

第三章：数据库问题案例

3.1 案例五：连接池泄漏导致服务不可用

故障现象：

  • 服务运行一段时间后报错 Cannot get a connection, pool error Timeout waiting for idle object
  • 重启后恢复，但过几天又出现
  • 监控显示活跃连接数持续增长

排查过程：

java 复制代码 
// 问题代码：连接未正确关闭
@Repository
public class ProblematicUserMapper {

    @Autowired
    private DataSource dataSource;

    public User findById(Long id) {
        Connection conn = null;
        PreparedStatement ps = null;
        ResultSet rs = null;
        try {
            conn = dataSource.getConnection();
            ps = conn.prepareStatement("SELECT * FROM users WHERE id = ?");
            ps.setLong(1, id);
            rs = ps.executeQuery();

            if (rs.next()) {
                return mapUser(rs);
            }
            return null;
        } catch (SQLException e) {
            // 问题：异常时连接未关闭
            throw new RuntimeException(e);
        }
        // 缺少 finally 块关闭连接
    }
}

根因分析：
数据库 连接池 应用 数据库 连接池 应用 #mermaid-svg-wVuNwdh9RrpjtYf4{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-wVuNwdh9RrpjtYf4 .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-wVuNwdh9RrpjtYf4 .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-wVuNwdh9RrpjtYf4 .error-icon{fill:#552222;}#mermaid-svg-wVuNwdh9RrpjtYf4 .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-wVuNwdh9RrpjtYf4 .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-wVuNwdh9RrpjtYf4 .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-wVuNwdh9RrpjtYf4 .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-wVuNwdh9RrpjtYf4 .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-wVuNwdh9RrpjtYf4 .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-wVuNwdh9RrpjtYf4 .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-wVuNwdh9RrpjtYf4 .marker{fill:#333333;stroke:#333333;}#mermaid-svg-wVuNwdh9RrpjtYf4 .marker.cross{stroke:#333333;}#mermaid-svg-wVuNwdh9RrpjtYf4 svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-wVuNwdh9RrpjtYf4 p{margin:0;}#mermaid-svg-wVuNwdh9RrpjtYf4 .actor{stroke:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);fill:#ECECFF;}#mermaid-svg-wVuNwdh9RrpjtYf4 text.actor>tspan{fill:black;stroke:none;}#mermaid-svg-wVuNwdh9RrpjtYf4 .actor-line{stroke:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);}#mermaid-svg-wVuNwdh9RrpjtYf4 .innerArc{stroke-width:1.5;stroke-dasharray:none;}#mermaid-svg-wVuNwdh9RrpjtYf4 .messageLine0{stroke-width:1.5;stroke-dasharray:none;stroke:#333;}#mermaid-svg-wVuNwdh9RrpjtYf4 .messageLine1{stroke-width:1.5;stroke-dasharray:2,2;stroke:#333;}#mermaid-svg-wVuNwdh9RrpjtYf4 #arrowhead path{fill:#333;stroke:#333;}#mermaid-svg-wVuNwdh9RrpjtYf4 .sequenceNumber{fill:white;}#mermaid-svg-wVuNwdh9RrpjtYf4 #sequencenumber{fill:#333;}#mermaid-svg-wVuNwdh9RrpjtYf4 #crosshead path{fill:#333;stroke:#333;}#mermaid-svg-wVuNwdh9RrpjtYf4 .messageText{fill:#333;stroke:none;}#mermaid-svg-wVuNwdh9RrpjtYf4 .labelBox{stroke:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);fill:#ECECFF;}#mermaid-svg-wVuNwdh9RrpjtYf4 .labelText,#mermaid-svg-wVuNwdh9RrpjtYf4 .labelText>tspan{fill:black;stroke:none;}#mermaid-svg-wVuNwdh9RrpjtYf4 .loopText,#mermaid-svg-wVuNwdh9RrpjtYf4 .loopText>tspan{fill:black;stroke:none;}#mermaid-svg-wVuNwdh9RrpjtYf4 .loopLine{stroke-width:2px;stroke-dasharray:2,2;stroke:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);fill:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);}#mermaid-svg-wVuNwdh9RrpjtYf4 .note{stroke:#aaaa33;fill:#fff5ad;}#mermaid-svg-wVuNwdh9RrpjtYf4 .noteText,#mermaid-svg-wVuNwdh9RrpjtYf4 .noteText>tspan{fill:black;stroke:none;}#mermaid-svg-wVuNwdh9RrpjtYf4 .activation0{fill:#f4f4f4;stroke:#666;}#mermaid-svg-wVuNwdh9RrpjtYf4 .activation1{fill:#f4f4f4;stroke:#666;}#mermaid-svg-wVuNwdh9RrpjtYf4 .activation2{fill:#f4f4f4;stroke:#666;}#mermaid-svg-wVuNwdh9RrpjtYf4 .actorPopupMenu{position:absolute;}#mermaid-svg-wVuNwdh9RrpjtYf4 .actorPopupMenuPanel{position:absolute;fill:#ECECFF;box-shadow:0px 8px 16px 0px rgba(0,0,0,0.2);filter:drop-shadow(3px 5px 2px rgb(0 0 0 / 0.4));}#mermaid-svg-wVuNwdh9RrpjtYf4 .actor-man line{stroke:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);fill:#ECECFF;}#mermaid-svg-wVuNwdh9RrpjtYf4 .actor-man circle,#mermaid-svg-wVuNwdh9RrpjtYf4 line{stroke:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);fill:#ECECFF;stroke-width:2px;}#mermaid-svg-wVuNwdh9RrpjtYf4 :root{--mermaid-font-family:"trebuchet ms",verdana,arial,sans-serif;} 异常发生 连接未归还 连接泄漏 获取连接创建连接执行SQL返回结果抛出异常

解决方案：

java 复制代码 
// 方案一：使用 try-with-resources
@Repository
public class FixedUserMapper {

    @Autowired
    private DataSource dataSource;

    public User findById(Long id) {
        // 自动关闭连接
        try (Connection conn = dataSource.getConnection();
             PreparedStatement ps = conn.prepareStatement("SELECT * FROM users WHERE id = ?")) {

            ps.setLong(1, id);
            try (ResultSet rs = ps.executeQuery()) {
                if (rs.next()) {
                    return mapUser(rs);
                }
                return null;
            }
        } catch (SQLException e) {
            throw new RuntimeException(e);
        }
    }
}

// 方案二：使用 JdbcTemplate（推荐）
@Repository
public class UserMapperImpl {

    @Autowired
    private JdbcTemplate jdbcTemplate;

    public User findById(Long id) {
        // JdbcTemplate 自动管理连接
        return jdbcTemplate.queryForObject(
            "SELECT * FROM users WHERE id = ?",
            new Object[]{id},
            this::mapUser
        );
    }
}

3.2 案例六：慢SQL导致数据库压力飙升

故障现象：

  • 数据库CPU使用率突然飙升到100%
  • 应用响应时间从50ms增加到5秒
  • 监控显示大量慢查询

排查过程：

sql 复制代码 
-- 问题SQL：全表扫描
SELECT * FROM orders
WHERE DATE(created_at) = '2026-06-24'
AND status = 'PAID'
ORDER BY amount DESC;

-- EXPLAIN 分析
-- type: ALL (全表扫描)
-- rows: 1000000 (扫描100万行)
-- Extra: Using filesort (文件排序)

根因分析：
#mermaid-svg-ndugg2zJ6AcxOY5U{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-ndugg2zJ6AcxOY5U .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-ndugg2zJ6AcxOY5U .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-ndugg2zJ6AcxOY5U .error-icon{fill:#552222;}#mermaid-svg-ndugg2zJ6AcxOY5U .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-ndugg2zJ6AcxOY5U .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-ndugg2zJ6AcxOY5U .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-ndugg2zJ6AcxOY5U .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-ndugg2zJ6AcxOY5U .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-ndugg2zJ6AcxOY5U .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-ndugg2zJ6AcxOY5U .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-ndugg2zJ6AcxOY5U .marker{fill:#333333;stroke:#333333;}#mermaid-svg-ndugg2zJ6AcxOY5U .marker.cross{stroke:#333333;}#mermaid-svg-ndugg2zJ6AcxOY5U svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-ndugg2zJ6AcxOY5U p{margin:0;}#mermaid-svg-ndugg2zJ6AcxOY5U .label{font-family:"trebuchet ms",verdana,arial,sans-serif;color:#333;}#mermaid-svg-ndugg2zJ6AcxOY5U .cluster-label text{fill:#333;}#mermaid-svg-ndugg2zJ6AcxOY5U .cluster-label span{color:#333;}#mermaid-svg-ndugg2zJ6AcxOY5U .cluster-label span p{background-color:transparent;}#mermaid-svg-ndugg2zJ6AcxOY5U .label text,#mermaid-svg-ndugg2zJ6AcxOY5U span{fill:#333;color:#333;}#mermaid-svg-ndugg2zJ6AcxOY5U .node rect,#mermaid-svg-ndugg2zJ6AcxOY5U .node circle,#mermaid-svg-ndugg2zJ6AcxOY5U .node ellipse,#mermaid-svg-ndugg2zJ6AcxOY5U .node polygon,#mermaid-svg-ndugg2zJ6AcxOY5U .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-ndugg2zJ6AcxOY5U .rough-node .label text,#mermaid-svg-ndugg2zJ6AcxOY5U .node .label text,#mermaid-svg-ndugg2zJ6AcxOY5U .image-shape .label,#mermaid-svg-ndugg2zJ6AcxOY5U .icon-shape .label{text-anchor:middle;}#mermaid-svg-ndugg2zJ6AcxOY5U .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-ndugg2zJ6AcxOY5U .rough-node .label,#mermaid-svg-ndugg2zJ6AcxOY5U .node .label,#mermaid-svg-ndugg2zJ6AcxOY5U .image-shape .label,#mermaid-svg-ndugg2zJ6AcxOY5U .icon-shape .label{text-align:center;}#mermaid-svg-ndugg2zJ6AcxOY5U .node.clickable{cursor:pointer;}#mermaid-svg-ndugg2zJ6AcxOY5U .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-ndugg2zJ6AcxOY5U .arrowheadPath{fill:#333333;}#mermaid-svg-ndugg2zJ6AcxOY5U .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-ndugg2zJ6AcxOY5U .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-ndugg2zJ6AcxOY5U .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-ndugg2zJ6AcxOY5U .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-ndugg2zJ6AcxOY5U .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-ndugg2zJ6AcxOY5U .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-ndugg2zJ6AcxOY5U .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-ndugg2zJ6AcxOY5U .cluster text{fill:#333;}#mermaid-svg-ndugg2zJ6AcxOY5U .cluster span{color:#333;}#mermaid-svg-ndugg2zJ6AcxOY5U div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-ndugg2zJ6AcxOY5U .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-ndugg2zJ6AcxOY5U rect.text{fill:none;stroke-width:0;}#mermaid-svg-ndugg2zJ6AcxOY5U .icon-shape,#mermaid-svg-ndugg2zJ6AcxOY5U .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-ndugg2zJ6AcxOY5U .icon-shape p,#mermaid-svg-ndugg2zJ6AcxOY5U .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-ndugg2zJ6AcxOY5U .icon-shape .label rect,#mermaid-svg-ndugg2zJ6AcxOY5U .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-ndugg2zJ6AcxOY5U .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-ndugg2zJ6AcxOY5U .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-ndugg2zJ6AcxOY5U :root{--mermaid-font-family:"trebuchet ms",verdana,arial,sans-serif;} 否
SQL查询
是否使用索引？
全表扫描
扫描100万行
CPU飙升
响应超时

解决方案：

sql 复制代码 
-- 方案一：添加索引
ALTER TABLE orders ADD INDEX idx_created_at_status (created_at, status);

-- 方案二：优化SQL（避免对索引列使用函数）
SELECT * FROM orders
WHERE created_at >= '2026-06-24 00:00:00'
AND created_at < '2026-06-25 00:00:00'
AND status = 'PAID'
ORDER BY amount DESC;

-- 方案三：分页查询
SELECT * FROM orders
WHERE created_at >= '2026-06-24 00:00:00'
AND created_at < '2026-06-25 00:00:00'
AND status = 'PAID'
ORDER BY amount DESC
LIMIT 100;

第四章：缓存问题案例

4.1 案例七：缓存雪崩导致数据库崩溃

故障现象：

  • 大量缓存同时过期
  • 请求全部打到数据库
  • 数据库CPU飙升，最终崩溃

排查过程：

java 复制代码 
// 问题代码：缓存过期时间相同
@Service
public class ProductService {

    public Product getProduct(Long id) {
        String cacheKey = "product:" + id;
        Product product = redisTemplate.opsForValue().get(cacheKey);

        if (product == null) {
            product = productMapper.selectById(id);
            // 问题：所有缓存设置相同的过期时间
            redisTemplate.opsForValue().set(cacheKey, product, 30, TimeUnit.MINUTES);
        }
        return product;
    }
}

// 故障场景：
// 1. 10:00 大量商品缓存同时设置，过期时间都是10:30
// 2. 10:30 缓存同时过期
// 3. 10:30-10:31 大量请求直接打到数据库
// 4. 数据库崩溃

根因分析：
#mermaid-svg-9Ytq4xRZjYqT6upc{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-9Ytq4xRZjYqT6upc .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-9Ytq4xRZjYqT6upc .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-9Ytq4xRZjYqT6upc .error-icon{fill:#552222;}#mermaid-svg-9Ytq4xRZjYqT6upc .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-9Ytq4xRZjYqT6upc .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-9Ytq4xRZjYqT6upc .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-9Ytq4xRZjYqT6upc .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-9Ytq4xRZjYqT6upc .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-9Ytq4xRZjYqT6upc .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-9Ytq4xRZjYqT6upc .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-9Ytq4xRZjYqT6upc .marker{fill:#333333;stroke:#333333;}#mermaid-svg-9Ytq4xRZjYqT6upc .marker.cross{stroke:#333333;}#mermaid-svg-9Ytq4xRZjYqT6upc svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-9Ytq4xRZjYqT6upc p{margin:0;}#mermaid-svg-9Ytq4xRZjYqT6upc .label{font-family:"trebuchet ms",verdana,arial,sans-serif;color:#333;}#mermaid-svg-9Ytq4xRZjYqT6upc .cluster-label text{fill:#333;}#mermaid-svg-9Ytq4xRZjYqT6upc .cluster-label span{color:#333;}#mermaid-svg-9Ytq4xRZjYqT6upc .cluster-label span p{background-color:transparent;}#mermaid-svg-9Ytq4xRZjYqT6upc .label text,#mermaid-svg-9Ytq4xRZjYqT6upc span{fill:#333;color:#333;}#mermaid-svg-9Ytq4xRZjYqT6upc .node rect,#mermaid-svg-9Ytq4xRZjYqT6upc .node circle,#mermaid-svg-9Ytq4xRZjYqT6upc .node ellipse,#mermaid-svg-9Ytq4xRZjYqT6upc .node polygon,#mermaid-svg-9Ytq4xRZjYqT6upc .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-9Ytq4xRZjYqT6upc .rough-node .label text,#mermaid-svg-9Ytq4xRZjYqT6upc .node .label text,#mermaid-svg-9Ytq4xRZjYqT6upc .image-shape .label,#mermaid-svg-9Ytq4xRZjYqT6upc .icon-shape .label{text-anchor:middle;}#mermaid-svg-9Ytq4xRZjYqT6upc .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-9Ytq4xRZjYqT6upc .rough-node .label,#mermaid-svg-9Ytq4xRZjYqT6upc .node .label,#mermaid-svg-9Ytq4xRZjYqT6upc .image-shape .label,#mermaid-svg-9Ytq4xRZjYqT6upc .icon-shape .label{text-align:center;}#mermaid-svg-9Ytq4xRZjYqT6upc .node.clickable{cursor:pointer;}#mermaid-svg-9Ytq4xRZjYqT6upc .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-9Ytq4xRZjYqT6upc .arrowheadPath{fill:#333333;}#mermaid-svg-9Ytq4xRZjYqT6upc .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-9Ytq4xRZjYqT6upc .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-9Ytq4xRZjYqT6upc .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-9Ytq4xRZjYqT6upc .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-9Ytq4xRZjYqT6upc .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-9Ytq4xRZjYqT6upc .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-9Ytq4xRZjYqT6upc .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-9Ytq4xRZjYqT6upc .cluster text{fill:#333;}#mermaid-svg-9Ytq4xRZjYqT6upc .cluster span{color:#333;}#mermaid-svg-9Ytq4xRZjYqT6upc div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-9Ytq4xRZjYqT6upc .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-9Ytq4xRZjYqT6upc rect.text{fill:none;stroke-width:0;}#mermaid-svg-9Ytq4xRZjYqT6upc .icon-shape,#mermaid-svg-9Ytq4xRZjYqT6upc .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-9Ytq4xRZjYqT6upc .icon-shape p,#mermaid-svg-9Ytq4xRZjYqT6upc .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-9Ytq4xRZjYqT6upc .icon-shape .label rect,#mermaid-svg-9Ytq4xRZjYqT6upc .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-9Ytq4xRZjYqT6upc .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-9Ytq4xRZjYqT6upc .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-9Ytq4xRZjYqT6upc :root{--mermaid-font-family:"trebuchet ms",verdana,arial,sans-serif;} 10:00 缓存设置
10:30 缓存过期
请求打到数据库
数据库压力激增
数据库崩溃

解决方案：

java 复制代码 
// 方案一：随机过期时间
@Service
public class FixedProductService {

    private final Random random = new Random();

    public Product getProduct(Long id) {
        String cacheKey = "product:" + id;
        Product product = redisTemplate.opsForValue().get(cacheKey);

        if (product == null) {
            product = productMapper.selectById(id);
            // 基础过期时间 + 随机偏移
            int baseExpire = 30;  // 30分钟
            int randomOffset = random.nextInt(10);  // 0-10分钟随机
            int expire = baseExpire + randomOffset;

            redisTemplate.opsForValue().set(cacheKey, product, expire, TimeUnit.MINUTES);
        }
        return product;
    }
}

// 方案二：使用互斥锁防止缓存击穿
@Service
public class FixedProductService2 {

    public Product getProduct(Long id) {
        String cacheKey = "product:" + id;
        String lockKey = "lock:" + cacheKey;
        Product product = redisTemplate.opsForValue().get(cacheKey);

        if (product == null) {
            // 获取分布式锁
            boolean locked = redisTemplate.opsForValue()
                .setIfAbsent(lockKey, "1", 10, TimeUnit.SECONDS);

            if (locked) {
                try {
                    // 双重检查
                    product = redisTemplate.opsForValue().get(cacheKey);
                    if (product == null) {
                        product = productMapper.selectById(id);
                        redisTemplate.opsForValue().set(cacheKey, product, 30, TimeUnit.MINUTES);
                    }
                } finally {
                    redisTemplate.delete(lockKey);
                }
            } else {
                // 未获取到锁，等待后重试
                Thread.sleep(100);
                return getProduct(id);
            }
        }
        return product;
    }
}

第五章：配置问题案例

5.1 案例八：配置错误导致服务启动失败

故障现象：

  • 修改配置后服务无法启动
  • 报错 Failed to bind properties under 'spring.datasource'
  • 回滚配置后恢复正常

排查过程：

yaml 复制代码 
# 问题配置：拼写错误
spring:
  datasource:
    drver-class-name: com.mysql.cj.jdbc.Driver  # 拼写错误：drver
    url: jdbc:mysql://localhost:3306/mydb

解决方案：

yaml 复制代码 
# 正确配置
spring:
  datasource:
    driver-class-name: com.mysql.cj.jdbc.Driver  # 正确：driver
    url: jdbc:mysql://localhost:3306/mydb
    username: root
    password: ${DB_PASSWORD}  # 使用环境变量，不要硬编码

5.2 案例九：环境差异导致功能异常

故障现象：

  • 本地测试正常，部署到生产环境报错
  • 报错 Connection refused
  • 生产环境的数据库地址不同

排查过程：

yaml 复制代码 
# 本地配置
spring:
  datasource:
    url: jdbc:mysql://localhost:3306/mydb

# 生产环境应该使用不同的地址

解决方案：

yaml 复制代码 
# 使用多环境配置
# application.yml (公共配置)
spring:
  profiles:
    active: ${SPRING_PROFILES_ACTIVE:dev}

# application-dev.yml (开发环境)
spring:
  datasource:
    url: jdbc:mysql://localhost:3306/mydb

# application-prod.yml (生产环境)
spring:
  datasource:
    url: jdbc:mysql://prod-db:3306/mydb

5.3 案例十：依赖冲突导致运行时异常

故障现象：

  • 服务启动成功，但运行时报错
  • 报错 NoSuchMethodError
  • 方法签名不匹配

排查过程：

java 复制代码 
// 问题：两个依赖引入了不同版本的同一个类
// 依赖A: com.fasterxml.jackson:2.15.0
// 依赖B: com.fasterxml.jackson:2.12.0

// 运行时加载了错误的版本

解决方案：

xml 复制代码 
<!-- 排除冲突的依赖 -->
<dependency>
    <groupId>com.example</groupId>
    <artifactId>library-a</artifactId>
    <version>1.0.0</version>
    <exclusions>
        <exclusion>
            <groupId>com.fasterxml.jackson.core</groupId>
            <artifactId>jackson-databind</artifactId>
        </exclusion>
    </exclusions>
</dependency>

<!-- 统一版本 -->
<dependencyManagement>
    <dependencies>
        <dependency>
            <groupId>com.fasterxml.jackson</groupId>
            <artifactId>jackson-bom</artifactId>
            <version>2.17.0</version>
            <type>pom</type>
            <scope>import</scope>
        </dependency>
    </dependencies>
</dependencyManagement>

第六章：故障预防体系

6.1 监控告警配置

java 复制代码 
@Configuration
public class MonitorConfig {

    // 内存监控
    @Scheduled(fixedRate = 60000)
    public void monitorMemory() {
        MemoryMXBean memoryBean = ManagementFactory.getMemoryMXBean();
        MemoryUsage heapUsage = memoryBean.getHeapMemoryUsage();

        long used = heapUsage.getUsed();
        long max = heapUsage.getMax();
        double usage = (double) used / max;

        if (usage > 0.85) {
            log.warn("内存使用率超过85%: {}", usage);
            sendAlert("内存告警", "使用率: " + usage);
        }
    }

    // 线程池监控
    @Scheduled(fixedRate = 30000)
    public void monitorThreadPool() {
        ThreadPoolExecutor executor = (ThreadPoolExecutor) taskExecutor;

        int activeCount = executor.getActiveCount();
        int poolSize = executor.getPoolSize();
        long completedTaskCount = executor.getCompletedTaskCount();

        log.info("线程池状态: active={}, pool={}, completed={}",
                activeCount, poolSize, completedTaskCount);
    }

    // 数据库连接池监控
    @Scheduled(fixedRate = 30000)
    public void monitorDataSource() {
        HikariDataSource ds = (HikariDataSource) dataSource;
        HikariPoolMXBean poolBean = ds.getHikariPoolMXBean();

        int active = poolBean.getActiveConnections();
        int idle = poolBean.getIdleConnections();
        int total = poolBean.getTotalConnections();

        log.info("连接池状态: active={}, idle={}, total={}", active, idle, total);
    }
}

6.2 故障排查检查清单

markdown 复制代码 
【故障排查流程】
□ 1. 确认故障现象（日志、监控、用户反馈）
□ 2. 快速止损（重启、回滚、限流、降级）
□ 3. 定位问题（日志分析、监控排查、代码审查）
□ 4. 修复问题（代码修复、配置调整、数据修复）
□ 5. 验证恢复（功能测试、性能测试、监控观察）
□ 6. 复盘总结（根因分析、改进措施、文档沉淀）

【常见故障排查命令】
# 查看JVM内存
jmap -heap <pid>

# 查看线程状态
jstack <pid>

# 查看GC情况
jstat -gcutil <pid> 1s

# 查看网络连接
netstat -anp | grep <port>

# 查看数据库连接
show processlist;

总结与展望

核心要点回顾

  1. 内存问题：避免无限增长的集合、正确清理 ThreadLocal
  2. 线程问题：合理配置线程池、避免死锁
  3. 数据库问题：正确关闭连接、优化慢SQL
  4. 缓存问题：随机过期时间、使用互斥锁
  5. 配置问题：多环境配置、依赖版本管理

适用边界

  • 必须预防：OOM、死锁、连接泄漏等致命问题
  • 重点关注：缓存雪崩、慢SQL等影响性能的问题
  • 持续改进：监控告警、故障演练、代码审查

未来展望

生产环境的稳定性需要持续投入。建议团队建立完善的监控体系、定期进行故障演练、培养故障排查能力，将故障消灭在萌芽状态。

真实性声明

1. 故障案例来源说明

本文收录的 10 个故障案例均来自我亲身参与或主导排查的企业级生产事故：

  • 案例一、二（OOM、ThreadLocal 泄漏）：来自某电商公司 2024 年 Q4 生产事故复盘记录
  • 案例三、四（线程池、死锁）：来自某金融科技公司 2025 年大促期间故障报告
  • 案例五、六（连接池、慢 SQL）：来自某 SaaS 平台 2025 年上半年运维工单
  • 案例七（缓存雪崩）：来自某内容平台 2025 年国庆活动故障复盘
  • 案例八、九、十（配置错误、环境差异、依赖冲突）：来自日常 Code Review 与上线检查记录
  • 所有案例的公司名称、业务数据、具体金额均已脱敏处理，技术细节保留原貌

2. 成本核算口径说明

  • 故障直接损失按电商场景平均订单金额 × 故障时长 × 转化率折损估算
  • 间接损失包含品牌信誉、用户流失、运维人力成本，按直接损失的 2-3 倍估算
  • 年度总损失按 10 个案例年度发生频率加权平均，非简单累加
  • 预防措施 ROI = (避免的故障损失 - 投入成本) / 投入成本，未考虑贴现率
  • 实际损失因业务规模、行业、SLA 要求不同会有较大差异

3. 技术版本与时效性

  • 本文基于 SpringBoot 3.x、JDK 17、MySQL 8.0、Redis 7.0 生产环境整理（2026 年 6 月）
  • 故障排查方法论（jmap/jstack/jstat/MAT）在 JDK 8-21 通用
  • 具体配置参数（如 HikariCP、线程池参数）会随版本调整，使用时请以实际环境为准
  • JVM 参数在 JDK 17+ 推荐使用 -Xlog:gc* 替代 -XX:+PrintGCDetails

4. 敏感信息 ${VAR} 占位符规范

本文所有涉及敏感信息的配置均使用 ${VAR} 占位符，部署时通过环境变量或 K8s Secret 注入：

占位符 用途 注入方式
${DB_PASSWORD} 数据库密码 K8s Secret
${SPRING_PROFILES_ACTIVE} Spring 激活 profile ConfigMap

严禁在代码或配置文件中硬编码任何密钥、密码、API Key。

如果本文对你有帮助，欢迎点赞、收藏、转发。你在生产环境中遇到过哪些故障，欢迎在评论区分享交流。行文仓促，定有不足之处，欢迎各位朋友在评论区批评指正，不胜感激。

相关推荐
李福春1 小时前
斗破Google ADK Java苍穹-第7境斗皇-御天地-Harness治理
java·人工智能·python·深圳同盟·腾讯云架构师技术同盟·adk java·harness 治理
Wang's Blog1 小时前
Java 项目部署之 Docker工具快速入门: 容器命令练习：创建、改文件、重启到删除的完整闭环
java·开发语言·docker
我是人✓1 小时前
Java面向对象核心特性深度解析
java·前端·算法
JaguarJack1 小时前
免费可商用 PHP 管理后台 CatchAdmin V5.4.1 发布，新增 MCP 支持
后端·php·laravel·服务端
Wx-bishekaifayuan1 小时前
springboot线上中医药养生平台管理系统36753-计算机课程设计、毕业设计
java·spring boot·后端·spring·课程设计·express·旅游
IT_陈寒1 小时前
为什么我的Vue组件总是莫名其妙更新？
前端·人工智能·后端
Wx-bishekaifayuan1 小时前
springboot大学生就业平台47691-计算机课程设计、毕业设计
java·spring boot·后端·elasticsearch·课程设计·express·旅游
BingoGo1 小时前
免费可商用 PHP 管理后台 CatchAdmin V5.4.1 发布，新增 MCP 支持
后端·php
汤姆yu2 小时前
基于springboot+vue的企业OA办公系统
java·spring boot·后端·oa办公
热门推荐
01国内可直接用、免费额度/永久免费的大模型API清单（含 SiliconFlow、火山、阿里、智谱、百度、Kimi、DeepSeek、DMXAPI 等）02历年考研数学一、数学二、数学三真题试卷及答案PDF03如何新建文件夹？ 电脑新建文件夹的4种方法04GitHub 镜像站点05「全语录+省流版」邵艾伦 × 孙宇晨：聊风口、聊个人IP、聊年轻人如何抓住 AI 时代的机会06蜘蛛磁力 搜索引擎大全，如何使用蜘蛛磁力查找磁力链接07轻量化AI渗透武器库｜ARTEX自主攻防系统、资产梳理、漏洞挖掘、链路分析全流程智能化落地08Windows 10/11 如何退出微软账户并改用本地账户登录（含找不到选项的解决方法）09微信历史版本含下载地址（ Windows PC | 安卓 | MAC ）及设置微信不更新10Telegram APK 下载安装完整指南 — 2026年最新