JVM G1GC参数调优实战

最近几年的工作中,一直都使用JDK1.8,虽然JDK1.8并未把G1GC作为默认的垃圾回收器(JDK9+默认为G1收集器),但是我们的系统都设置为G1GC。G1GC的核心优点是:可预测停顿时间以及非常适合大堆和多核机器,很适合高并发、低延迟的系统。今天总结下我的一次JVM G1GC参数优化案例。

问题背景

我们的业务系统是广告检索系统,高峰期的业务流量可达9亿/天,又分为站内流量和站外流量。其中站内流量约占30%,占总收入的50%~60%;站外流量约占70%,占总收入的40%~60%。站内和站外是同一套代码,业务上来讲站内广告检索更优质,收入相对更高。

站内日均请求量约为2亿+,部署在16核16G的机器,20台机器(高峰期单机的qps可达600左右),下图是一天的超时量10348次,算下站内服务的可用性也是99.9948%,虽然是满足我们最初的设定目标,满足SLA目标(4个9)的预期。但是还是发现有可优化的空间,降低超时量,尽可能提升到SLA五个9(毕竟对于广告检索系统,一条检索信息里可能会包含50条广告,对于尤其CPT类的广告,C端用户点击就会带来收入)

下图是一台站内服务的实例,平时CPU基本都维持在不超过10%,但是一天会有几次突刺。而且发现CPU的突刺和接口超时是比较吻合的,于是就朝着优化CPU突刺的方向思考。

问题分析

公司有现成的工具可以将线上单台机器的全部流量拷贝到沙箱环境(即测试环境),于是我写了一个脚本,用来监控CPU的突刺,每30s执行top命令获取CPU的占比,当CPU>25%的时候,利用async工具dump下CPU的火焰图来进行分析.

#!/bin/bash 复制代码
# 目标Java进程PID(修改为你的进程号)
PID=5567

# CPU使用率阈值(百分比)
CPU_THRESHOLD=400

# async-profiler路径(修改为你的async-profiler目录)
PROFILER_DIR=/home/work/async-profiler-2.9-linux-x64

# 采样时长(秒)
DURATION=30

while true; do
  # 获取进程CPU使用率,取总CPU核数百分比,比如一核100%,4核400%
  # CPU_USAGE=$(top -b -n1 -p $PID | awk '/^[0-9]/{print $9}')
  CPU_USAGE=$(top -b -n1 -p $PID | awk '{gsub(/^ +/, ""); if ($1 ~ /^[0-9]+$/) print $9}')
  # 如果获取失败(进程不存在),退出脚本
  if [ -z "$CPU_USAGE" ]; then
    echo "进程 $PID 不存在,脚本退出"
    exit 1
  fi

  CPU_INT=${CPU_USAGE%.*}  # 取整数部分

  echo "$(date '+%Y-%m-%d %H:%M:%S') - PID $PID CPU: $CPU_USAGE%"

  if [ "$CPU_INT" -gt "$CPU_THRESHOLD" ]; then
    echo "CPU超出阈值,开始采样..."
    TIMESTAMP=$(date '+%Y%m%d_%H%M%S')
    OUTPUT_FILE="cpu-profile-$TIMESTAMP.html"
    $PROFILER_DIR/profiler.sh -d $DURATION -f $OUTPUT_FILE $PID
    echo "采样完成,火焰图保存为 $OUTPUT_FILE"
  fi

  sleep 10
done

CPU突刺的火焰图(右侧红色和黄色为G1GC相关信息) CPU正常的火焰图(右侧红色和黄色为G1GC相关信息) 从上面的火焰图可以看出来,发生CPU突刺的时候GC占用CPU较多。我们系统的检索接口大部分是RPC调下游的接口,是偏IO型CPU,而不是计算型CPU,理论上不太需要太多CPU。因此下面的优化方向主要放在G1GC上。

如下是JVM参数配置(16核16G,JDK1.8.65版本),站内检索接口超时时间为230ms,站外检索接口超时时间为115ms。

⚠️注意:我们线上是并没有打印任何GC信息的,咨询了下给出如下理由:线上一般很少有GC问题,另一方面是打印GC日志也会影响接口耗时。

ini 复制代码
-XX:+UseG1GC 
-Xmx10g 
-Xms10g 
-XX:ConcGCThreads=8 
-XX:GCTimeRatio=39 
-XX:G1HeapRegionSize=8m 
-XX:MaxGCPauseMillis=40 
-XX:ParallelGCThreads=10 
-XX:G1RSetUpdatingPauseTimePercent=40 
-XX:MetaspaceSize=512M 
-XX:MaxMetaspaceSize=1024M 
-XX:MaxTenuringThreshold=15 
-XX:InitiatingHeapOccupancyPercent=70 
-XX:-UseBiasedLocking 

先补充了GC日志信息

ruby 复制代码
    -Xloggc:/opt/XX/XX_gc.log 
    -XX:+PrintGCDetails 
    -XX:+PrintGCTimeStamps 
    -XX:+PrintClassHistogramAfterFullGC 
    -XX:+PrintClassHistogramBeforeFullGC 
    -XX:+PrintGCApplicationConcurrentTime 
    -XX:+PrintGCApplicationStoppedTime 
    -XX:+PrintGCDateStamps
    -XX:+UseGCLogFileRotation 
    -XX:NumberOfGCLogFiles=3 
    -XX:GCLogFileSize=100M 
    -XX:+PrintTenuringDistribution 
    -XX:+UnlockDiagnosticVMOptions 
    -XX:+G1SummarizeRSetStats 
    -XX:+G1SummarizeConcMark 
    -XX:+PrintHeapAtGC 
    -XX:+PrintReferenceGC 
    -XX:+PrintAdaptiveSizePolicy

然后再沙箱环境Copy线上单台实例流量,观察GC日志信息

js 复制代码
2025-11-25T10:22:13.893+0800: 54834.370: Total time for which application threads were stopped: 0.0399467 seconds, Stopping threads took: 0.0003971 seconds
2025-11-25T10:22:18.360+0800: 54838.837: Application time: 4.4673903 seconds
{Heap before GC invocations=3371 (full 0):
 garbage-first heap   total 10485760K, used 9293743K [0x0000000540000000, 0x0000000540802800, 0x00000007c0000000) // 堆10G,已使用9G
  region size 8192K, 249 young (2039808K), 14 survivors (114688K) // region大小为8M,249个young region占2G,14个survivor region占114M
 Metaspace       used 438233K, capacity 443533K, committed 446720K, reserved 1425408K // 元空间,已使用438M,JVM向OS申请的预留内存443M,潜在最大可用内存1.4G(虚拟内存)
  class space    used 69311K, capacity 70187K, committed 70912K, reserved 1048576K // 类空间,已使用69M,JVM向OS申请的预留内存70M,潜在最大可用内存1.0G(虚拟内存)
2025-11-25T10:22:18.364+0800: 54838.841: [GC pause (G1 Evacuation Pause) (young) // 54838表示自JVM启动到现在的时间,单位为s,STW,存活对象拷贝转移到新Region上,young gc
Desired survivor size 134217728 bytes, new threshold 15 (max 15) // 理想survivor区大小为134M,最大阈值15
- age   1:   22631736 bytes,   22631736 total // 存活1次young gc,22M
- age   2:    3914072 bytes,   26545808 total
- age   3:   11398312 bytes,   37944120 total
- age   4:    4621616 bytes,   42565736 total
- age   5:    4064352 bytes,   46630088 total
- age   6:    2745080 bytes,   49375168 total
- age   7:    6074624 bytes,   55449792 total
- age   8:   10838784 bytes,   66288576 total
- age   9:    3798888 bytes,   70087464 total
- age  10:    6146800 bytes,   76234264 total
- age  11:    5616944 bytes,   81851208 total
- age  12:    2407672 bytes,   84258880 total
- age  13:    2391048 bytes,   86649928 total 
- age  14:    1436968 bytes,   88086896 total // 存活14次young gc,2M,age1~age14累积88M
 54838.841: [G1Ergonomics (CSet Construction) start choosing CSet, _pending_cards: 39031, predicted base time: 22.03 ms, remaining time: 17.97 ms, target pause time: 40.00 ms] // 选择CSet,待处理的卡片39031,预测基础暂停时间22ms,剩余时间17ms
 54838.841: [G1Ergonomics (CSet Construction) add young regions to CSet, eden: 235 regions, survivors: 14 regions, predicted young region time: 12.66 ms] // 添加region至CSet,eden有235个region,survivor有14个region,预测回收eden + survivor需要12ms
 54838.841: [G1Ergonomics (CSet Construction) finish choosing CSet, eden: 235 regions, survivors: 14 regions, old: 0 regions, predicted pause time: 34.69 ms, target pause time: 40.00 ms] // 完成CSet的选择,eden:235,survivor:14,old:0,预测暂停时间34ms
2025-11-25T10:22:18.398+0800: 54838.875: [SoftReference, 0 refs, 0.0000507 secs]2025-11-25T10:22:18.398+0800: 54838.875: [WeakReference, 0 refs, 0.0000070 secs]2025-11-25T10:22:18.398+0800: 54838.875: [FinalReference, 330 refs, 0.0000523 secs]2025-11-25T10:22:18.398+0800: 54838.875: [PhantomReference, 0 refs, 0 refs, 0.0000063 secs]2025-11-25T10:22:18.398+0800: 54838.875: [JNI Weak Reference, 0.0002955 secs], 0.0362511 secs] // 终结引用有330个
   [Parallel Time: 32.2 ms, GC Workers: 10] // 并行GC worker,10个GC线程同时回收,总暂停时间32ms
      [GC Worker Start (ms): Min: 54838842.1, Avg: 54838842.2, Max: 54838842.3, Diff: 0.1] 
      [Ext Root Scanning (ms): Min: 1.9, Avg: 3.3, Max: 9.1, Diff: 7.2, Sum: 32.6] // 扫描外部GC Roots(静态对象、线程栈,总耗时32ms)
      [Update RS (ms): Min: 10.1, Avg: 15.7, Max: 17.1, Diff: 7.0, Sum: 157.3] // 更新记忆集RSet,总耗时157ms
         [Processed Buffers: Min: 58, Avg: 104.7, Max: 203, Diff: 145, Sum: 1047]
      [Scan RS (ms): Min: 2.3, Avg: 2.4, Max: 2.5, Diff: 0.1, Sum: 24.0] // 扫描RSet,找到存活对象,总耗时24ms
      [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] // 扫描JIT代码根对象,总耗时0ms
      [Object Copy (ms): Min: 10.4, Avg: 10.5, Max: 10.6, Diff: 0.2, Sum: 105.4] // 对存活对象copy+转移,总耗时105ms
      [Termination (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.1] // 并行线程终止同步
         [Termination Attempts: Min: 1, Avg: 4.1, Max: 8, Diff: 7, Sum: 41]
      [GC Worker Other (ms): Min: 0.0, Avg: 0.0, Max: 0.1, Diff: 0.1, Sum: 0.4]
      [GC Worker Total (ms): Min: 31.9, Avg: 32.0, Max: 32.0, Diff: 0.1, Sum: 319.8] // 由于是10个线程并发执行,总耗时319ms
      [GC Worker End (ms): Min: 54838874.2, Avg: 54838874.2, Max: 54838874.2, Diff: 0.1]
   [Code Root Fixup: 0.2 ms]
   [Code Root Purge: 0.0 ms]
   [Clear CT: 0.6 ms] // 清理卡表 clear card table
   [Other: 3.3 ms]
      [Choose CSet: 0.0 ms]
      [Ref Proc: 0.7 ms]
      [Ref Enq: 0.0 ms]
      [Redirty Cards: 0.2 ms] 
      [Humongous Register: 0.1 ms]
      [Humongous Reclaim: 0.0 ms]
      [Free CSet: 0.4 ms]
   [Eden: 1880.0M(1880.0M)->0.0B(1880.0M) Survivors: 112.0M->112.0M Heap: 9075.9M(10.0G)->7199.9M(10.0G)] // eden由1.8G->0,survivor搬迁后112M保持不变,堆9.0G->7.1G
Heap after GC invocations=3372 (full 0):
 garbage-first heap   total 10485760K, used 7372719K [0x0000000540000000, 0x0000000540802800, 0x00000007c0000000)
  region size 8192K, 14 young (114688K), 14 survivors (114688K)
 Metaspace       used 438233K, capacity 443533K, committed 446720K, reserved 1425408K
  class space    used 69311K, capacity 70187K, committed 70912K, reserved 1048576K
}
 [Times: user=0.32 sys=0.01, real=0.04 secs] // 10个GC work累积起来CPU使用了0.32s,系统CPU时间0.01s,GC整体耗时40ms(即STW时间为40ms)
 
 
......此处省略很多日志信息,主要是抓住上面的一次young gc,下面的是一次mixed gc
 

2025-11-25T10:23:09.261+0800: 54889.738: [GC concurrent-root-region-scan-start]
2025-11-25T10:23:09.261+0800: 54889.738: Total time for which application threads were stopped: 0.0541826 seconds, Stopping threads took: 0.0003794 seconds
2025-11-25T10:23:09.272+0800: 54889.749: [GC concurrent-root-region-scan-end, 0.0105196 secs]
2025-11-25T10:23:09.272+0800: 54889.749: [GC concurrent-mark-start]
2025-11-25T10:23:09.764+0800: 54890.241: [GC concurrent-mark-end, 0.4918518 secs]
2025-11-25T10:23:09.764+0800: 54890.241: Application time: 0.5022052 seconds
2025-11-25T10:23:09.767+0800: 54890.244: [GC remark 2025-11-25T10:23:09.767+0800: 54890.244: [Finalize Marking, 0.0011891 secs] 2025-11-25T10:23:09.769+0800: 54890.246: [GC ref-proc2025-11-25T10:23:09.769+0800: 54890.246: [SoftReference, 231 refs, 0.0001804 secs]2025-11-25T10:23:09.769+0800: 54890.246: [WeakReference, 1410 refs, 0.0005327 secs]2025-11-25T10:23:09.769+0800: 54890.246: [FinalReference, 142834 refs, 0.3074949 secs]2025-11-25T10:23:10.077+0800: 54890.554: [PhantomReference, 0 refs, 75 refs, 0.0001334 secs]2025-11-25T10:23:10.077+0800: 54890.554: [JNI Weak Reference, 0.0027464 secs], 0.3186995 secs] 2025-11-25T10:23:10.087+0800: 54890.564: [Unloading, 0.0765362 secs], 0.4024319 secs]
 [Times: user=0.70 sys=0.37, real=0.41 secs] 
2025-11-25T10:23:10.170+0800: 54890.647: Total time for which application threads were stopped: 0.4066764 seconds, Stopping threads took: 0.0004158 seconds
2025-11-25T10:23:10.170+0800: 54890.647: Application time: 0.0001122 seconds
2025-11-25T10:23:10.174+0800: 54890.651: [GC cleanup 7471M->7463M(10G), 0.0331351 secs]
 [Times: user=0.32 sys=0.00, real=0.03 secs] 
2025-11-25T10:23:10.207+0800: 54890.684: Total time for which application threads were stopped: 0.0370230 seconds, Stopping threads took: 0.0002807 seconds
2025-11-25T10:23:10.207+0800: 54890.685: [GC concurrent-cleanup-start]
2025-11-25T10:23:10.208+0800: 54890.685: [GC concurrent-cleanup-end, 0.0000349 secs]
2025-11-25T10:23:10.226+0800: 54890.703: Application time: 0.0186830 seconds
2025-11-25T10:23:10.232+0800: 54890.709: Total time for which application threads were stopped: 0.0059369 seconds, Stopping threads took: 0.0007812 seconds
2025-11-25T10:23:10.264+0800: 54890.741: Application time: 0.0323801 seconds
{Heap before GC invocations=3386 (full 0):
 garbage-first heap   total 10485760K, used 7831345K [0x0000000540000000, 0x0000000540802800, 0x00000007c0000000)
  region size 8192K, 64 young (524288K), 15 survivors (122880K)
 Metaspace       used 438233K, capacity 443533K, committed 446720K, reserved 1425408K
  class space    used 69311K, capacity 70187K, committed 70912K, reserved 1048576K
2025-11-25T10:23:10.269+0800: 54890.746: [GC pause (G1 Evacuation Pause) (young)
Desired survivor size 33554432 bytes, new threshold 4 (max 15)
- age   1:   25594440 bytes,   25594440 total
- age   2:    2250768 bytes,   27845208 total
- age   3:    4198992 bytes,   32044200 total
- age   4:    4975912 bytes,   37020112 total
- age   5:    4380272 bytes,   41400384 total
- age   6:    5889984 bytes,   47290368 total
- age   7:   11192072 bytes,   58482440 total
- age   8:    1776304 bytes,   60258744 total
- age   9:    7334440 bytes,   67593184 total
- age  10:    1684488 bytes,   69277672 total
- age  11:    2989192 bytes,   72266864 total
- age  12:    3355072 bytes,   75621936 total
- age  13:    2714968 bytes,   78336904 total
- age  14:    7023864 bytes,   85360768 total
- age  15:    7301376 bytes,   92662144 total
 54890.747: [G1Ergonomics (CSet Construction) start choosing CSet, _pending_cards: 37502, predicted base time: 28.25 ms, remaining time: 11.75 ms, target pause time: 40.00 ms]
 54890.747: [G1Ergonomics (CSet Construction) add young regions to CSet, eden: 49 regions, survivors: 15 regions, predicted young region time: 13.33 ms]
 54890.747: [G1Ergonomics (CSet Construction) finish choosing CSet, eden: 49 regions, survivors: 15 regions, old: 0 regions, predicted pause time: 41.58 ms, target pause time: 40.00 ms]
2025-11-25T10:23:10.304+0800: 54890.782: [SoftReference, 0 refs, 0.0000483 secs]2025-11-25T10:23:10.305+0800: 54890.782: [WeakReference, 0 refs, 0.0000071 secs]2025-11-25T10:23:10.305+0800: 54890.782: [FinalReference, 2054 refs, 0.0002449 secs]2025-11-25T10:23:10.305+0800: 54890.782: [PhantomReference, 0 refs, 0 refs, 0.0000070 secs]2025-11-25T10:23:10.305+0800: 54890.782: [JNI Weak Reference, 0.0003081 secs] 54890.784: [G1Ergonomics (Mixed GCs) start mixed GCs, reason: candidate old regions available, candidate old regions: 824 regions, reclaimable: 6012264304 bytes (55.99 %), threshold: 5.00 %]
, 0.0376695 secs]
   [Parallel Time: 33.7 ms, GC Workers: 10]
      [GC Worker Start (ms): Min: 54890747.3, Avg: 54890747.3, Max: 54890747.4, Diff: 0.1]
      [Ext Root Scanning (ms): Min: 1.9, Avg: 3.3, Max: 9.7, Diff: 7.8, Sum: 33.3]
      [Update RS (ms): Min: 7.5, Avg: 13.8, Max: 15.6, Diff: 8.0, Sum: 137.8]
         [Processed Buffers: Min: 38, Avg: 73.2, Max: 100, Diff: 62, Sum: 732]
      [Scan RS (ms): Min: 3.2, Avg: 3.6, Max: 3.8, Diff: 0.6, Sum: 36.4]
      [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0]
      [Object Copy (ms): Min: 12.4, Avg: 12.7, Max: 12.8, Diff: 0.4, Sum: 126.5]
      [Termination (ms): Min: 0.0, Avg: 0.0, Max: 0.1, Diff: 0.0, Sum: 0.4]
         [Termination Attempts: Min: 1, Avg: 30.0, Max: 43, Diff: 42, Sum: 300]
      [GC Worker Other (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.2]
      [GC Worker Total (ms): Min: 33.4, Avg: 33.5, Max: 33.5, Diff: 0.1, Sum: 334.6]
      [GC Worker End (ms): Min: 54890780.8, Avg: 54890780.8, Max: 54890780.8, Diff: 0.0]
   [Code Root Fixup: 0.4 ms]
   [Code Root Purge: 0.0 ms]
   [Clear CT: 0.5 ms]
   [Other: 3.1 ms]
      [Choose CSet: 0.0 ms]
      [Ref Proc: 0.8 ms]
      [Ref Enq: 0.0 ms]
      [Redirty Cards: 0.2 ms]
      [Humongous Register: 0.1 ms]
      [Humongous Reclaim: 0.0 ms]
      [Free CSet: 0.2 ms]
   [Eden: 392.0M(392.0M)->0.0B(464.0M) Survivors: 120.0M->48.0M Heap: 7647.8M(10.0G)->7247.8M(10.0G)]
Heap after GC invocations=3387 (full 0):
 garbage-first heap   total 10485760K, used 7421697K [0x0000000540000000, 0x0000000540802800, 0x00000007c0000000)
  region size 8192K, 6 young (49152K), 6 survivors (49152K)
 Metaspace       used 438233K, capacity 443533K, committed 446720K, reserved 1425408K
  class space    used 69311K, capacity 70187K, committed 70912K, reserved 1048576K
}
 [Times: user=0.33 sys=0.01, real=0.03 secs] 
2025-11-25T10:23:10.308+0800: 54890.785: Total time for which application threads were stopped: 0.0431743 seconds, Stopping threads took: 0.0004376 seconds
2025-11-25T10:23:11.165+0800: 54891.642: Application time: 0.8572387 seconds
{Heap before GC invocations=3387 (full 0):
 garbage-first heap   total 10485760K, used 7896833K [0x0000000540000000, 0x0000000540802800, 0x00000007c0000000)
  region size 8192K, 64 young (524288K), 6 survivors (49152K)
 Metaspace       used 438233K, capacity 443533K, committed 446720K, reserved 1425408K
  class space    used 69311K, capacity 70187K, committed 70912K, reserved 1048576K
2025-11-25T10:23:11.169+0800: 54891.646: [GC pause (G1 Evacuation Pause) (mixed)
Desired survivor size 33554432 bytes, new threshold 15 (max 15)
- age   1:   11818032 bytes,   11818032 total
- age   2:   14113656 bytes,   25931688 total
- age   3:    2097312 bytes,   28029000 total
- age   4:    4122952 bytes,   32151952 total
 54891.646: [G1Ergonomics (CSet Construction) start choosing CSet, _pending_cards: 34128, predicted base time: 25.46 ms, remaining time: 14.54 ms, target pause time: 40.00 ms]
 54891.646: [G1Ergonomics (CSet Construction) add young regions to CSet, eden: 58 regions, survivors: 6 regions, predicted young region time: 8.09 ms]
 54891.647: [G1Ergonomics (CSet Construction) finish adding old regions to CSet, reason: predicted time is too high, predicted time: 0.25 ms, remaining time: 0.00 ms, old: 103 regions, min: 103 regions]
 54891.647: [G1Ergonomics (CSet Construction) added expensive regions to CSet, reason: old CSet region num not reached min, old: 103 regions, expensive: 68 regions, min: 103 regions, remaining time: 0.00 ms]
 54891.647: [G1Ergonomics (CSet Construction) finish choosing CSet, eden: 58 regions, survivors: 6 regions, old: 103 regions, predicted pause time: 55.30 ms, target pause time: 40.00 ms]
2025-11-25T10:23:11.211+0800: 54891.688: [SoftReference, 1 refs, 0.0000831 secs]2025-11-25T10:23:11.211+0800: 54891.688: [WeakReference, 0 refs, 0.0000073 secs]2025-11-25T10:23:11.211+0800: 54891.688: [FinalReference, 163 refs, 0.0000294 secs]2025-11-25T10:23:11.211+0800: 54891.688: [PhantomReference, 0 refs, 0 refs, 0.0000067 secs]2025-11-25T10:23:11.211+0800: 54891.688: [JNI Weak Reference, 0.0002868 secs] 54891.691: [G1Ergonomics (Mixed GCs) continue mixed GCs, reason: candidate old regions available, candidate old regions: 721 regions, reclaimable: 5170979216 bytes (48.16 %), threshold: 5.00 %]
, 0.0459042 secs]
   [Parallel Time: 40.4 ms, GC Workers: 10]
      [GC Worker Start (ms): Min: 54891647.2, Avg: 54891647.3, Max: 54891647.3, Diff: 0.1]
      [Ext Root Scanning (ms): Min: 2.0, Avg: 3.3, Max: 9.0, Diff: 7.1, Sum: 33.5]
      [Update RS (ms): Min: 3.8, Avg: 9.4, Max: 11.0, Diff: 7.1, Sum: 93.9]
         [Processed Buffers: Min: 51, Avg: 78.9, Max: 118, Diff: 67, Sum: 789]
      [Scan RS (ms): Min: 19.6, Avg: 19.9, Max: 20.0, Diff: 0.4, Sum: 198.6]
      [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0]
      [Object Copy (ms): Min: 7.4, Avg: 7.5, Max: 7.6, Diff: 0.2, Sum: 75.4]
      [Termination (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.2]
         [Termination Attempts: Min: 1, Avg: 17.1, Max: 25, Diff: 24, Sum: 171]
      [GC Worker Other (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.1]
      [GC Worker Total (ms): Min: 40.1, Avg: 40.2, Max: 40.2, Diff: 0.1, Sum: 401.8]
      [GC Worker End (ms): Min: 54891687.4, Avg: 54891687.4, Max: 54891687.4, Diff: 0.0]
   [Code Root Fixup: 0.2 ms]
   [Code Root Purge: 0.0 ms]
   [Clear CT: 0.5 ms]
   [Other: 4.8 ms]
      [Choose CSet: 0.6 ms]
      [Ref Proc: 0.6 ms]
      [Ref Enq: 0.0 ms]
      [Redirty Cards: 0.2 ms]
      [Humongous Register: 0.1 ms]
      [Humongous Reclaim: 0.0 ms]
      [Free CSet: 1.5 ms]
   [Eden: 464.0M(464.0M)->0.0B(456.0M) Survivors: 48.0M->56.0M Heap: 7711.8M(10.0G)->6476.0M(10.0G)]
Heap after GC invocations=3388 (full 0):
 garbage-first heap   total 10485760K, used 6631424K [0x0000000540000000, 0x0000000540802800, 0x00000007c0000000)
  region size 8192K, 7 young (57344K), 7 survivors (57344K)
 Metaspace       used 438233K, capacity 443533K, committed 446720K, reserved 1425408K
  class space    used 69311K, capacity 70187K, committed 70912K, reserved 1048576K
}
 [Times: user=0.40 sys=0.01, real=0.05 secs] 
2025-11-25T10:23:11.215+0800: 54891.692: Total time for which application threads were stopped: 0.0503583 seconds, Stopping threads took: 0.0003777 seconds
2025-11-25T10:23:12.215+0800: 54892.692: Application time: 1.0000975 seconds
2025-11-25T10:23:12.219+0800: 54892.696: Total time for which application threads were stopped: 0.0038770 seconds, Stopping threads took: 0.0004035 seconds
2025-11-25T10:23:12.419+0800: 54892.896: Application time: 0.1999900 seconds
{Heap before GC invocations=3388 (full 0): // 从JVM启动开始,一共发生过3388次GC(这里指的是young gc + mixed gc),full gc的次数为0
 garbage-first heap   total 10485760K, used 7098368K [0x0000000540000000, 0x0000000540802800, 0x00000007c0000000) // 堆大小10G,已使用7.0G
  region size 8192K, 64 young (524288K), 7 survivors (57344K) // region大小为8M,64个eden region占比524M,7个survivor region占比57M
 Metaspace       used 438233K, capacity 443533K, committed 446720K, reserved 1425408K
  class space    used 69311K, capacity 70187K, committed 70912K, reserved 1048576K
2025-11-25T10:23:12.423+0800: 54892.900: [GC pause (G1 Evacuation Pause) (mixed) // STW,触发mixed gc
Desired survivor size 33554432 bytes, new threshold 15 (max 15) // 理想的survivor区33M
- age   1:   11871704 bytes,   11871704 total // age1:11M
- age   2:    2059688 bytes,   13931392 total // age2:2M
- age   3:   13506584 bytes,   27437976 total // age3:13M
- age   4:    2072984 bytes,   29510960 total
- age   5:    3819240 bytes,   33330200 total
 54892.900: [G1Ergonomics (CSet Construction) start choosing CSet, _pending_cards: 34397, predicted base time: 23.79 ms, remaining time: 16.21 ms, target pause time: 40.00 ms]
 54892.900: [G1Ergonomics (CSet Construction) add young regions to CSet, eden: 57 regions, survivors: 7 regions, predicted young region time: 6.72 ms]
 54892.901: [G1Ergonomics (CSet Construction) finish adding old regions to CSet, reason: predicted time is too high, predicted time: 0.39 ms, remaining time: 0.00 ms, old: 103 regions, min: 103 regions] 
    // 将old region添加至CSet,但是由于预测时间太高,至少要选择103个old region(性价比高的,即存活率低、RS少、回收成本低)添加进CSet,预测时间为0.39ms(另:这0.39ms是这103个old region增加的边际预测时间,不代表完整回收成本)
    // min:103region的含义:根据G1的评估模型,如果本次gc不回收103个region,未来mixed gc压力加重,甚至可能逼近full gc
 54892.901: [G1Ergonomics (CSet Construction) added expensive regions to CSet, reason: old CSet region num not reached min, old: 103 regions, expensive: 74 regions, min: 103 regions, remaining time: 0.00 ms]
    // 由于选择了部分便宜的region,但是还是未能达到,至少回收103old region的要求,不得不再选择74个昂贵的old region至CSet
 54892.901: [G1Ergonomics (CSet Construction) finish choosing CSet, eden: 57 regions, survivors: 7 regions, old: 103 regions, predicted pause time: 66.73 ms, target pause time: 40.00 ms]
    // 完成选择,57个eden,7个survivor,103个old,预测暂停时间66ms
2025-11-25T10:23:12.476+0800: 54892.953: [SoftReference, 0 refs, 0.0000462 secs]2025-11-25T10:23:12.476+0800: 54892.953: [WeakReference, 0 refs, 0.0000069 secs]2025-11-25T10:23:12.476+0800: 54892.953: [FinalReference, 7 refs, 0.0000158 secs]2025-11-25T10:23:12.476+0800: 54892.953: [PhantomReference, 0 refs, 0 refs, 0.0000066 secs]2025-11-25T10:23:12.476+0800: 54892.953: [JNI Weak Reference, 0.0002829 secs] 54892.957: [G1Ergonomics (Mixed GCs) continue mixed GCs, reason: candidate old regions available, candidate old regions: 618 regions, reclaimable: 4331449584 bytes (40.34 %), threshold: 5.00 %]
, 0.0572976 secs]
   [Parallel Time: 50.8 ms, GC Workers: 10]
      [GC Worker Start (ms): Min: 54892902.1, Avg: 54892902.2, Max: 54892902.2, Diff: 0.1]
      [Ext Root Scanning (ms): Min: 1.9, Avg: 3.3, Max: 9.1, Diff: 7.2, Sum: 33.1]
      [Update RS (ms): Min: 7.5, Avg: 13.2, Max: 14.6, Diff: 7.1, Sum: 132.1]
         [Processed Buffers: Min: 56, Avg: 82.9, Max: 118, Diff: 62, Sum: 829]
      [Scan RS (ms): Min: 25.6, Avg: 25.6, Max: 25.7, Diff: 0.1, Sum: 255.9]
      [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0]
      [Object Copy (ms): Min: 8.3, Avg: 8.4, Max: 8.5, Diff: 0.2, Sum: 84.1]
      [Termination (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.1]
         [Termination Attempts: Min: 1, Avg: 4.9, Max: 10, Diff: 9, Sum: 49]
      [GC Worker Other (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.2]
      [GC Worker Total (ms): Min: 50.5, Avg: 50.6, Max: 50.6, Diff: 0.1, Sum: 505.5] // 10个线程并行处理,总耗时505ms
      [GC Worker End (ms): Min: 54892952.7, Avg: 54892952.7, Max: 54892952.7, Diff: 0.0]
   [Code Root Fixup: 0.2 ms]
   [Code Root Purge: 0.0 ms]
   [Clear CT: 0.5 ms]
   [Other: 5.8 ms]
      [Choose CSet: 1.1 ms]
      [Ref Proc: 0.6 ms]
      [Ref Enq: 0.0 ms]
      [Redirty Cards: 0.2 ms]
      [Humongous Register: 0.1 ms]
      [Humongous Reclaim: 0.0 ms]
      [Free CSet: 1.9 ms]
   [Eden: 456.0M(456.0M)->0.0B(464.0M) Survivors: 56.0M->48.0M Heap: 6932.0M(10.0G)->5656.8M(10.0G)]
Heap after GC invocations=3389 (full 0):
 garbage-first heap   total 10485760K, used 5792607K [0x0000000540000000, 0x0000000540802800, 0x00000007c0000000)
  region size 8192K, 6 young (49152K), 6 survivors (49152K)
 Metaspace       used 438233K, capacity 443533K, committed 446720K, reserved 1425408K
  class space    used 69311K, capacity 70187K, committed 70912K, reserved 1048576K
}
 [Times: user=0.51 sys=0.01, real=0.06 secs] 
2025-11-25T10:23:12.481+0800: 54892.958: Total time for which application threads were stopped: 0.0616297 seconds, Stopping threads took: 0.0003992 seconds
2025-11-25T10:23:13.443+0800: 54893.920: Application time: 0.9618601 seconds
{Heap before GC invocations=3389 (full 0):
 garbage-first heap   total 10485760K, used 6267743K [0x0000000540000000, 0x0000000540802800, 0x00000007c0000000)
  region size 8192K, 64 young (524288K), 6 survivors (49152K)
 Metaspace       used 438233K, capacity 443533K, committed 446720K, reserved 1425408K
  class space    used 69311K, capacity 70187K, committed 70912K, reserved 1048576K
2025-11-25T10:23:13.447+0800: 54893.924: [GC pause (G1 Evacuation Pause) (mixed)
Desired survivor size 33554432 bytes, new threshold 15 (max 15)
- age   1:    7724688 bytes,    7724688 total
- age   2:     702288 bytes,    8426976 total
- age   3:    1943536 bytes,   10370512 total
- age   4:   12413992 bytes,   22784504 total
- age   5:    2068200 bytes,   24852704 total
- age   6:    3396752 bytes,   28249456 total
 54893.924: [G1Ergonomics (CSet Construction) start choosing CSet, _pending_cards: 38929, predicted base time: 24.78 ms, remaining time: 15.22 ms, target pause time: 40.00 ms]
 54893.924: [G1Ergonomics (CSet Construction) add young regions to CSet, eden: 58 regions, survivors: 6 regions, predicted young region time: 7.68 ms]
 54893.926: [G1Ergonomics (CSet Construction) finish adding old regions to CSet, reason: predicted time is too high, predicted time: 0.50 ms, remaining time: 0.00 ms, old: 103 regions, min: 103 regions]
 54893.926: [G1Ergonomics (CSet Construction) added expensive regions to CSet, reason: old CSet region num not reached min, old: 103 regions, expensive: 86 regions, min: 103 regions, remaining time: 0.00 ms]
 54893.926: [G1Ergonomics (CSet Construction) finish choosing CSet, eden: 58 regions, survivors: 6 regions, old: 103 regions, predicted pause time: 80.06 ms, target pause time: 40.00 ms]
2025-11-25T10:23:13.507+0800: 54893.984: [SoftReference, 0 refs, 0.0000529 secs]2025-11-25T10:23:13.507+0800: 54893.984: [WeakReference, 0 refs, 0.0000068 secs]2025-11-25T10:23:13.507+0800: 54893.984: [FinalReference, 59 refs, 0.0000457 secs]2025-11-25T10:23:13.507+0800: 54893.985: [PhantomReference, 0 refs, 0 refs, 0.0000066 secs]2025-11-25T10:23:13.507+0800: 54893.985: [JNI Weak Reference, 0.0003339 secs] 54893.989: [G1Ergonomics (Mixed GCs) continue mixed GCs, reason: candidate old regions available, candidate old regions: 515 regions, reclaimable: 3491747192 bytes (32.52 %), threshold: 5.00 %]
, 0.0653969 secs]
   [Parallel Time: 57.8 ms, GC Workers: 10]
      [GC Worker Start (ms): Min: 54893926.1, Avg: 54893926.2, Max: 54893926.3, Diff: 0.1]
      [Ext Root Scanning (ms): Min: 1.9, Avg: 3.1, Max: 8.1, Diff: 6.3, Sum: 31.4]
      [Update RS (ms): Min: 9.1, Avg: 14.0, Max: 15.2, Diff: 6.2, Sum: 140.0]
         [Processed Buffers: Min: 41, Avg: 83.0, Max: 124, Diff: 83, Sum: 830]
      [Scan RS (ms): Min: 29.9, Avg: 29.9, Max: 30.0, Diff: 0.1, Sum: 299.4]
      [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0]
      [Object Copy (ms): Min: 10.4, Avg: 10.5, Max: 10.6, Diff: 0.2, Sum: 105.2]
      [Termination (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0]
         [Termination Attempts: Min: 1, Avg: 1.7, Max: 3, Diff: 2, Sum: 17]
      [GC Worker Other (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.2]
      [GC Worker Total (ms): Min: 57.6, Avg: 57.6, Max: 57.7, Diff: 0.1, Sum: 576.2]
      [GC Worker End (ms): Min: 54893983.8, Avg: 54893983.8, Max: 54893983.8, Diff: 0.0]
   [Code Root Fixup: 0.2 ms]
   [Code Root Purge: 0.0 ms]
   [Clear CT: 0.6 ms]
   [Other: 6.7 ms]
      [Choose CSet: 1.6 ms]
      [Ref Proc: 0.7 ms]
      [Ref Enq: 0.0 ms]
      [Redirty Cards: 0.2 ms]
      [Humongous Register: 0.1 ms]
      [Humongous Reclaim: 0.0 ms]
      [Free CSet: 2.3 ms]
   [Eden: 464.0M(464.0M)->0.0B(456.0M) Survivors: 48.0M->56.0M Heap: 6120.8M(10.0G)->4861.8M(10.0G)]
Heap after GC invocations=3390 (full 0):
 garbage-first heap   total 10485760K, used 4978445K [0x0000000540000000, 0x0000000540802800, 0x00000007c0000000)
  region size 8192K, 7 young (57344K), 7 survivors (57344K)
 Metaspace       used 438233K, capacity 443533K, committed 446720K, reserved 1425408K
  class space    used 69311K, capacity 70187K, committed 70912K, reserved 1048576K
}
 [Times: user=0.58 sys=0.01, real=0.07 secs] 
2025-11-25T10:23:13.513+0800: 54893.990: Total time for which application threads were stopped: 0.0698825 seconds, Stopping threads took: 0.0004164 seconds
2025-11-25T10:23:14.108+0800: 54894.585: Application time: 0.5956209 seconds
{Heap before GC invocations=3390 (full 0):
 garbage-first heap   total 10485760K, used 5445389K [0x0000000540000000, 0x0000000540802800, 0x00000007c0000000)
  region size 8192K, 64 young (524288K), 7 survivors (57344K)
 Metaspace       used 438233K, capacity 443533K, committed 446720K, reserved 1425408K
  class space    used 69311K, capacity 70187K, committed 70912K, reserved 1048576K
2025-11-25T10:23:14.112+0800: 54894.589: [GC pause (G1 Evacuation Pause) (mixed)
Desired survivor size 33554432 bytes, new threshold 7 (max 15)
- age   1:   15958424 bytes,   15958424 total
- age   2:    1214920 bytes,   17173344 total
- age   3:     634608 bytes,   17807952 total
- age   4:    1703320 bytes,   19511272 total
- age   5:   12131752 bytes,   31643024 total
- age   6:    1741312 bytes,   33384336 total
- age   7:    3366880 bytes,   36751216 total
 54894.589: [G1Ergonomics (CSet Construction) start choosing CSet, _pending_cards: 40514, predicted base time: 24.46 ms, remaining time: 15.54 ms, target pause time: 40.00 ms]
 54894.589: [G1Ergonomics (CSet Construction) add young regions to CSet, eden: 57 regions, survivors: 7 regions, predicted young region time: 11.11 ms]
 54894.592: [G1Ergonomics (CSet Construction) finish adding old regions to CSet, reason: predicted time is too high, predicted time: 0.66 ms, remaining time: 0.00 ms, old: 103 regions, min: 103 regions]
 54894.592: [G1Ergonomics (CSet Construction) added expensive regions to CSet, reason: old CSet region num not reached min, old: 103 regions, expensive: 95 regions, min: 103 regions, remaining time: 0.00 ms]
 54894.592: [G1Ergonomics (CSet Construction) finish choosing CSet, eden: 57 regions, survivors: 7 regions, old: 103 regions, predicted pause time: 96.55 ms, target pause time: 40.00 ms]
2025-11-25T10:23:14.180+0800: 54894.657: [SoftReference, 0 refs, 0.0000496 secs]2025-11-25T10:23:14.180+0800: 54894.657: [WeakReference, 0 refs, 0.0000154 secs]2025-11-25T10:23:14.180+0800: 54894.657: [FinalReference, 5 refs, 0.0000208 secs]2025-11-25T10:23:14.180+0800: 54894.657: [PhantomReference, 0 refs, 1 refs, 0.0000076 secs]2025-11-25T10:23:14.180+0800: 54894.657: [JNI Weak Reference, 0.0002843 secs] 54894.662: [G1Ergonomics (Mixed GCs) continue mixed GCs, reason: candidate old regions available, candidate old regions: 412 regions, reclaimable: 2654644096 bytes (24.72 %), threshold: 5.00 %]
, 0.0732945 secs]
   [Parallel Time: 64.6 ms, GC Workers: 10]
      [GC Worker Start (ms): Min: 54894592.2, Avg: 54894592.3, Max: 54894592.4, Diff: 0.1]
      [Ext Root Scanning (ms): Min: 1.8, Avg: 3.2, Max: 8.7, Diff: 6.8, Sum: 32.1]
      [Update RS (ms): Min: 8.2, Avg: 13.7, Max: 15.3, Diff: 7.1, Sum: 136.9]
         [Processed Buffers: Min: 41, Avg: 88.3, Max: 126, Diff: 85, Sum: 883]
      [Scan RS (ms): Min: 34.0, Avg: 34.3, Max: 34.4, Diff: 0.4, Sum: 342.8]
      [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0]
      [Object Copy (ms): Min: 13.1, Avg: 13.2, Max: 13.3, Diff: 0.2, Sum: 132.4]
      [Termination (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.1]
         [Termination Attempts: Min: 1, Avg: 6.4, Max: 11, Diff: 10, Sum: 64]
      [GC Worker Other (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.3]
      [GC Worker Total (ms): Min: 64.4, Avg: 64.5, Max: 64.5, Diff: 0.1, Sum: 644.5]
      [GC Worker End (ms): Min: 54894656.7, Avg: 54894656.8, Max: 54894656.8, Diff: 0.0]
   [Code Root Fixup: 0.3 ms]
   [Code Root Purge: 0.0 ms]
   [Clear CT: 0.5 ms]
   [Other: 7.8 ms]
      [Choose CSet: 2.2 ms]
      [Ref Proc: 0.6 ms]
      [Ref Enq: 0.0 ms]
      [Redirty Cards: 0.3 ms]
      [Humongous Register: 0.1 ms]
      [Humongous Reclaim: 0.0 ms]
      [Free CSet: 2.8 ms]
   [Eden: 456.0M(456.0M)->0.0B(448.0M) Survivors: 56.0M->64.0M Heap: 5317.8M(10.0G)->4078.3M(10.0G)]
Heap after GC invocations=3391 (full 0):
 garbage-first heap   total 10485760K, used 4176182K [0x0000000540000000, 0x0000000540802800, 0x00000007c0000000)
  region size 8192K, 8 young (65536K), 8 survivors (65536K)
 Metaspace       used 438233K, capacity 443533K, committed 446720K, reserved 1425408K
  class space    used 69311K, capacity 70187K, committed 70912K, reserved 1048576K
}
 [Times: user=0.65 sys=0.01, real=0.07 secs] 
2025-11-25T10:23:14.186+0800: 54894.663: Total time for which application threads were stopped: 0.0777673 seconds, Stopping threads took: 0.0003897 seconds
2025-11-25T10:23:14.882+0800: 54895.360: Application time: 0.6965141 seconds
{Heap before GC invocations=3391 (full 0):
 garbage-first heap   total 10485760K, used 4634934K [0x0000000540000000, 0x0000000540802800, 0x00000007c0000000)
  region size 8192K, 64 young (524288K), 8 survivors (65536K)
 Metaspace       used 438233K, capacity 443533K, committed 446720K, reserved 1425408K
  class space    used 69311K, capacity 70187K, committed 70912K, reserved 1048576K
2025-11-25T10:23:14.886+0800: 54895.363: [GC pause (G1 Evacuation Pause) (mixed)
Desired survivor size 33554432 bytes, new threshold 6 (max 15)
- age   1:   18416568 bytes,   18416568 total
- age   2:    4592136 bytes,   23008704 total
- age   3:    1010608 bytes,   24019312 total
- age   4:     574408 bytes,   24593720 total
- age   5:    1526056 bytes,   26119776 total
- age   6:   11596080 bytes,   37715856 total
- age   7:    1584672 bytes,   39300528 total
 54895.364: [G1Ergonomics (CSet Construction) start choosing CSet, _pending_cards: 41932, predicted base time: 24.04 ms, remaining time: 15.96 ms, target pause time: 40.00 ms]
 54895.364: [G1Ergonomics (CSet Construction) add young regions to CSet, eden: 56 regions, survivors: 8 regions, predicted young region time: 9.41 ms]
 54895.367: [G1Ergonomics (CSet Construction) finish adding old regions to CSet, reason: predicted time is too high, predicted time: 1.05 ms, remaining time: 0.00 ms, old: 103 regions, min: 103 regions]
 54895.367: [G1Ergonomics (CSet Construction) added expensive regions to CSet, reason: old CSet region num not reached min, old: 103 regions, expensive: 94 regions, min: 103 regions, remaining time: 0.00 ms]
 54895.367: [G1Ergonomics (CSet Construction) finish choosing CSet, eden: 56 regions, survivors: 8 regions, old: 103 regions, predicted pause time: 117.26 ms, target pause time: 40.00 ms]
2025-11-25T10:23:14.965+0800: 54895.442: [SoftReference, 0 refs, 0.0000558 secs]2025-11-25T10:23:14.965+0800: 54895.442: [WeakReference, 0 refs, 0.0000069 secs]2025-11-25T10:23:14.965+0800: 54895.442: [FinalReference, 23 refs, 0.0000112 secs]2025-11-25T10:23:14.965+0800: 54895.442: [PhantomReference, 0 refs, 0 refs, 0.0000061 secs]2025-11-25T10:23:14.965+0800: 54895.442: [JNI Weak Reference, 0.0002860 secs] 54895.448: [G1Ergonomics (Mixed GCs) continue mixed GCs, reason: candidate old regions available, candidate old regions: 309 regions, reclaimable: 1832794208 bytes (17.07 %), threshold: 5.00 %]
, 0.0849832 secs]
    // 继续进行mixed gc,可回收的old region有309个,可回收17.07%的内存。阈值为5%,即可回收内存>阈值时,会继续回收
   [Parallel Time: 74.7 ms, GC Workers: 10]
      [GC Worker Start (ms): Min: 54895367.3, Avg: 54895367.4, Max: 54895367.5, Diff: 0.1]
      [Ext Root Scanning (ms): Min: 1.6, Avg: 3.0, Max: 8.3, Diff: 6.6, Sum: 29.7]
      [Update RS (ms): Min: 8.6, Avg: 13.7, Max: 15.0, Diff: 6.4, Sum: 137.0]
         [Processed Buffers: Min: 45, Avg: 84.0, Max: 113, Diff: 68, Sum: 840]
      [Scan RS (ms): Min: 38.8, Avg: 39.0, Max: 39.1, Diff: 0.3, Sum: 389.7]
      [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0]
      [Object Copy (ms): Min: 18.6, Avg: 18.8, Max: 19.0, Diff: 0.4, Sum: 187.8]
      [Termination (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.1]
         [Termination Attempts: Min: 1, Avg: 4.2, Max: 9, Diff: 8, Sum: 42]
      [GC Worker Other (ms): Min: 0.0, Avg: 0.0, Max: 0.1, Diff: 0.0, Sum: 0.3]
      [GC Worker Total (ms): Min: 74.4, Avg: 74.5, Max: 74.5, Diff: 0.1, Sum: 744.6]
      [GC Worker End (ms): Min: 54895441.9, Avg: 54895441.9, Max: 54895441.9, Diff: 0.0]
   [Code Root Fixup: 0.2 ms]
   [Code Root Purge: 0.0 ms]
   [Clear CT: 0.5 ms]
   [Other: 9.5 ms]
      [Choose CSet: 3.2 ms]
      [Ref Proc: 0.6 ms]
      [Ref Enq: 0.0 ms]
      [Redirty Cards: 0.3 ms]
      [Humongous Register: 0.1 ms]
      [Humongous Reclaim: 0.0 ms]
      [Free CSet: 3.5 ms]
   [Eden: 448.0M(448.0M)->0.0B(464.0M) Survivors: 64.0M->48.0M Heap: 4526.3M(10.0G)->3296.4M(10.0G)]
Heap after GC invocations=3392 (full 0):
 garbage-first heap   total 10485760K, used 3375564K [0x0000000540000000, 0x0000000540802800, 0x00000007c0000000)
  region size 8192K, 6 young (49152K), 6 survivors (49152K)
 Metaspace       used 438233K, capacity 443533K, committed 446720K, reserved 1425408K
  class space    used 69311K, capacity 70187K, committed 70912K, reserved 1048576K
}
 [Times: user=0.75 sys=0.00, real=0.09 secs] 

在沙箱环境中,统计工具每5分钟聚合一次检索接口的性能数据,重点关注 P99、P99.9、P99.99、P99.999高分位耗时以及最大耗时(Max)。从统计结果可以看到,在 10:25 时间点接口耗时出现明显抖动,多项高分位指标均超过了 230ms 的超时时间阈值

进一步结合 GC 日志分析发现,在 10:23 左右发生了多次 Mixed GC 。这些 Mixed GC 的 STW(Stop-The-World)暂停时间 分别达到了 60ms、90ms 等 ,显著超过了 JVM 配置的 MaxGCPauseTime=40ms 。接口在 10:25 统计数据的高延迟现象,与前后时间段内频繁且耗时较长的 Mixed GC 具有较强的时间相关性。

推断链路:由于苛刻的 MaxGCPauseTime=40 ,垃圾收集器会倾向于通过 抢占更多 CPU 资源 来尽可能压缩 STW 暂停时间。但是Mixed GC的STW时间仍然远超设定的MaxGCPauseTime,GC 线程对 CPU 的抢占引发了明显的 CPU 突刺,进一步挤压了业务线程的执行资源,最终导致该时间窗口内业务接口响应时间显著上升,并出现大量超时。

结论:通过调整JVM的参数来降低Young GC和Mixed GC的次数【理想情况下降低GC次数,也即降低GC总耗时】

为此,采用对比实验法【用控制变量的手段,将线上版本作为baseline,其他版本每次只更改一个参数,copy线上单台实例流量到测试环境,效果如下】

备注:本次调整保证MaxGCPauseTime=40的条件不能调整,原因是站内和站外代码是同一套,站外检索接口的耗时为115ms,改动MaxGCPauseTime将会对站外产生较大影响

对比实验一【请求流量和机器配置完全一致,仅保证某一个变量变化】:

参数变更 Young GC次数 Mixed GC次数 备注
12275 28 baseline
-Xmx10g -Xms10g -> -Xmx12g -Xms12g 10517 15
-XX:ParallelGCThreads=10->12 11991 8
-XX:ParallelGCThreads=10->8 ? ? 实验数据丢失,印象中是负向效果
-XX:InitatingHeapOccupancyPercent=70->50 10405 21
-XX:MaxTenuringThreshold=15->5 13290 85

对比实验二【请求流量和机器配置完全一致,仅保证某一个变量变化】:

参数变更 Young GC次数 Mixed GC次数 备注
JDK1.8.65 11875 23 baseline
JDK1.8.341 7484 21

对比实验三【请求流量和机器配置完全一致,仅保证某一个变量变化】:

参数变更 Young GC次数 Mixed GC次数 备注
ConcGCThread=8 8332 8 baseline
ConcGCThread=4 5725 8

经过上面几个实验的数据对比,将young gc和mixed gc均下降的参数调整可视为正向效果。

【舍弃堆内存调大至12G】虽然堆内存调整为12G的确有正向效果,但是机器内存大小为16G,也就是说堆外内存【元空间、直接内存、JIT编译后的机器码、JVM自身内存、线程栈等信息】仅剩4G。元空间已经占用了1G,我们假设机器有1024个线程(1个线程默认1M),线程栈占用1G,JVM自身内存占用500M,加起来已经占用了2.5G。我们本身是广告检索系统,很可能出现qps突刺,为防止OOM,保守起见堆内存仍保持不变

【最终结论】

js 复制代码
ConcGCThreads=8 -> 4 // 并发标记阶段的线程数。降低该值,降低对GC线程抢占业务线程,降低CPU突刺
ParallelGCThreads=10 -> 12 // 并行垃圾回收过程中的线程数。增加改值,加快回收速度,降低young gc和mixed gc
JDK version:1.8.65 -> 1.8.341 // 341版本是2022年发布的,相比65版本是2015发布的,在GC方面算法层面做了一些优化
删除-XX:GCTimeRatio=39 
删除-XX:G1RSetUpdatingPauseTimePercent=40 

详细参数如下

js 复制代码
// 优化前(JVM日志为本次添加的,用于JVM日志分析用的,实际上线时不带上线)
-XX:+UseG1GC 
-Xmx10g 
-Xms10g 
-XX:ConcGCThreads=8 
-XX:GCTimeRatio=39 
-XX:G1HeapRegionSize=8m 
-XX:MaxGCPauseMillis=40 
-XX:ParallelGCThreads=10 
-XX:G1RSetUpdatingPauseTimePercent=40 
-XX:MetaspaceSize=512M 
-XX:MaxMetaspaceSize=1024M 
-XX:MaxTenuringThreshold=15 
-XX:InitiatingHeapOccupancyPercent=70 
-XX:-UseBiasedLocking 
    -Xloggc:/opt/XX/XX_gc.log 
    -XX:+PrintGCDetails 
    -XX:+PrintGCTimeStamps 
    -XX:+PrintClassHistogramAfterFullGC 
    -XX:+PrintClassHistogramBeforeFullGC 
    -XX:+PrintGCApplicationConcurrentTime 
    -XX:+PrintGCApplicationStoppedTime 
    -XX:+PrintGCDateStamps
    -XX:+UseGCLogFileRotation 
    -XX:NumberOfGCLogFiles=3 
    -XX:GCLogFileSize=100M 
    -XX:+PrintTenuringDistribution 
    -XX:+UnlockDiagnosticVMOptions 
    -XX:+G1SummarizeRSetStats 
    -XX:+G1SummarizeConcMark 
    -XX:+PrintHeapAtGC 
    -XX:+PrintReferenceGC 
    -XX:+PrintAdaptiveSizePolicy


// 优化后
-XX:+UseG1GC
-Xmx10g 
-Xms10g
-XX:ConcGCThreads=4
-XX:G1HeapRegionSize=8m 
-XX:MaxGCPauseMillis=40 
-XX:ParallelGCThreads=12 
-XX:MetaspaceSize=512M 
-XX:MaxMetaspaceSize=1024M 
-XX:MaxTenuringThreshold=15
-XX:InitiatingHeapOccupancyPercent=70 
-XX:-UseBiasedLocking 
    -Xloggc:/opt/XX/XX_gc.log 
    -XX:+PrintGCDetails 
    -XX:+PrintGCTimeStamps 
    -XX:+PrintClassHistogramAfterFullGC
    -XX:+PrintClassHistogramBeforeFullGC
    -XX:+PrintGCApplicationConcurrentTime
    -XX:+PrintGCApplicationStoppedTime
    -XX:+PrintGCDateStamps
    -XX:+UseGCLogFileRotation
    -XX:NumberOfGCLogFiles=3
    -XX:GCLogFileSize=100M
    -XX:+PrintTenuringDistribution
    -XX:+UnlockDiagnosticVMOptions
    -XX:+G1SummarizeRSetStats
    -XX:+G1SummarizeConcMark
    -XX:+PrintHeapAtGC
    -XX:+PrintReferenceGC
    -XX:+PrintAdaptiveSizePolicy
JDK版本升级:1.8.65 -> 1.8.341

【沙箱环境验证】(站内和站外分别实验)

站内环境

参数变更 Young GC次数 Mixed GC次数 备注
online版本 12907 35 baseline
优化版本 5554 8

结论:站内机器的GC次数明显下降,CPU突刺下降,99.99%分位的超时突刺下降

站外耗时图

站外JVM监控图

online版本 优化版本

结论:站外机器的GC次数下降,整体也呈现下降趋势(约下降5ms左右)

综上:准备上线验证

问题解决

【上线效果分析】2025-12-10推全代码

1.CPU突刺减少(由于MaxGCPauseTime=40的目标太严苛,很难完全消灭CPU突刺)

2.GC次数和GC耗时下降

3.CPU突刺带给站内流量的超时下降(之前每天超时量在5000+

问题总结

总结:经过本次的调优学到了很多G1GC的知识,也看了很多官网和G1GC的书籍,以及和ChatGPT的大量讨论,收获颇丰。

如果您觉得有收获的话,欢迎点赞、评价、收藏,以及关注github:github.com/HangboQuan

后续计划分享更多G1GC的笔记

相关推荐
我认不到你2 小时前
paxos一致性算法(大白话+图解)
分布式·后端
文心快码BaiduComate2 小时前
插件开发实录:我用Comate在VS Code里造了一场“能被代码融化”的初雪
前端·后端·前端框架
austin流川枫2 小时前
深度解析六大Java微服务框架
java·后端·微服务
martin10172 小时前
基于Spring Boot + Thymeleaf + Flying Saucer实现PDF导出功能
后端
程序员爱钓鱼2 小时前
Node.js 编程实战:路由与中间件
前端·后端·node.js
程序员爱钓鱼2 小时前
Node.js 编程实战:Express 基础
前端·后端·node.js
CosMr2 小时前
【QT】【FFmpeg】 Qt 中FFmpeg环境搭建以及D__STDC_FORMAT_MACROS、PRId64解答
后端
Knight_AL2 小时前
Spring Boot 的主要特性与传统 Spring 项目的区别
spring boot·后端·spring
回家路上绕了弯2 小时前
一文读懂分布式事务:核心原理、解决方案与实践思考
分布式·后端