最近几年的工作中,一直都使用JDK1.8,虽然JDK1.8并未把G1GC作为默认的垃圾回收器(JDK9+默认为G1收集器),但是我们的系统都设置为G1GC。G1GC的核心优点是:可预测停顿时间以及非常适合大堆和多核机器,很适合高并发、低延迟的系统。今天总结下我的一次JVM G1GC参数优化案例。
问题背景
我们的业务系统是广告检索系统,高峰期的业务流量可达9亿/天,又分为站内流量和站外流量。其中站内流量约占30%,占总收入的50%~60%;站外流量约占70%,占总收入的40%~60%。站内和站外是同一套代码,业务上来讲站内广告检索更优质,收入相对更高。
站内日均请求量约为2亿+,部署在16核16G的机器,20台机器(高峰期单机的qps可达600左右),下图是一天的超时量10348次,算下站内服务的可用性也是99.9948%,虽然是满足我们最初的设定目标,满足SLA目标(4个9)的预期。但是还是发现有可优化的空间,降低超时量,尽可能提升到SLA五个9(毕竟对于广告检索系统,一条检索信息里可能会包含50条广告,对于尤其CPT类的广告,C端用户点击就会带来收入)
下图是一台站内服务的实例,平时CPU基本都维持在不超过10%,但是一天会有几次突刺。而且发现CPU的突刺和接口超时是比较吻合的,于是就朝着优化CPU突刺的方向思考。
问题分析
公司有现成的工具可以将线上单台机器的全部流量拷贝到沙箱环境(即测试环境),于是我写了一个脚本,用来监控CPU的突刺,每30s执行top命令获取CPU的占比,当CPU>25%的时候,利用async工具dump下CPU的火焰图来进行分析.
#!/bin/bash
# 目标Java进程PID(修改为你的进程号)
PID=5567
# CPU使用率阈值(百分比)
CPU_THRESHOLD=400
# async-profiler路径(修改为你的async-profiler目录)
PROFILER_DIR=/home/work/async-profiler-2.9-linux-x64
# 采样时长(秒)
DURATION=30
while true; do
# 获取进程CPU使用率,取总CPU核数百分比,比如一核100%,4核400%
# CPU_USAGE=$(top -b -n1 -p $PID | awk '/^[0-9]/{print $9}')
CPU_USAGE=$(top -b -n1 -p $PID | awk '{gsub(/^ +/, ""); if ($1 ~ /^[0-9]+$/) print $9}')
# 如果获取失败(进程不存在),退出脚本
if [ -z "$CPU_USAGE" ]; then
echo "进程 $PID 不存在,脚本退出"
exit 1
fi
CPU_INT=${CPU_USAGE%.*} # 取整数部分
echo "$(date '+%Y-%m-%d %H:%M:%S') - PID $PID CPU: $CPU_USAGE%"
if [ "$CPU_INT" -gt "$CPU_THRESHOLD" ]; then
echo "CPU超出阈值,开始采样..."
TIMESTAMP=$(date '+%Y%m%d_%H%M%S')
OUTPUT_FILE="cpu-profile-$TIMESTAMP.html"
$PROFILER_DIR/profiler.sh -d $DURATION -f $OUTPUT_FILE $PID
echo "采样完成,火焰图保存为 $OUTPUT_FILE"
fi
sleep 10
done
CPU突刺的火焰图(右侧红色和黄色为G1GC相关信息)
CPU正常的火焰图(右侧红色和黄色为G1GC相关信息)
从上面的火焰图可以看出来,发生CPU突刺的时候GC占用CPU较多。我们系统的检索接口大部分是RPC调下游的接口,是偏IO型CPU,而不是计算型CPU,理论上不太需要太多CPU。因此下面的优化方向主要放在G1GC上。
如下是JVM参数配置(16核16G,JDK1.8.65版本),站内检索接口超时时间为230ms,站外检索接口超时时间为115ms。
⚠️注意:我们线上是并没有打印任何GC信息的,咨询了下给出如下理由:线上一般很少有GC问题,另一方面是打印GC日志也会影响接口耗时。
ini
-XX:+UseG1GC
-Xmx10g
-Xms10g
-XX:ConcGCThreads=8
-XX:GCTimeRatio=39
-XX:G1HeapRegionSize=8m
-XX:MaxGCPauseMillis=40
-XX:ParallelGCThreads=10
-XX:G1RSetUpdatingPauseTimePercent=40
-XX:MetaspaceSize=512M
-XX:MaxMetaspaceSize=1024M
-XX:MaxTenuringThreshold=15
-XX:InitiatingHeapOccupancyPercent=70
-XX:-UseBiasedLocking
先补充了GC日志信息
ruby
-Xloggc:/opt/XX/XX_gc.log
-XX:+PrintGCDetails
-XX:+PrintGCTimeStamps
-XX:+PrintClassHistogramAfterFullGC
-XX:+PrintClassHistogramBeforeFullGC
-XX:+PrintGCApplicationConcurrentTime
-XX:+PrintGCApplicationStoppedTime
-XX:+PrintGCDateStamps
-XX:+UseGCLogFileRotation
-XX:NumberOfGCLogFiles=3
-XX:GCLogFileSize=100M
-XX:+PrintTenuringDistribution
-XX:+UnlockDiagnosticVMOptions
-XX:+G1SummarizeRSetStats
-XX:+G1SummarizeConcMark
-XX:+PrintHeapAtGC
-XX:+PrintReferenceGC
-XX:+PrintAdaptiveSizePolicy
然后再沙箱环境Copy线上单台实例流量,观察GC日志信息
js
2025-11-25T10:22:13.893+0800: 54834.370: Total time for which application threads were stopped: 0.0399467 seconds, Stopping threads took: 0.0003971 seconds
2025-11-25T10:22:18.360+0800: 54838.837: Application time: 4.4673903 seconds
{Heap before GC invocations=3371 (full 0):
garbage-first heap total 10485760K, used 9293743K [0x0000000540000000, 0x0000000540802800, 0x00000007c0000000) // 堆10G,已使用9G
region size 8192K, 249 young (2039808K), 14 survivors (114688K) // region大小为8M,249个young region占2G,14个survivor region占114M
Metaspace used 438233K, capacity 443533K, committed 446720K, reserved 1425408K // 元空间,已使用438M,JVM向OS申请的预留内存443M,潜在最大可用内存1.4G(虚拟内存)
class space used 69311K, capacity 70187K, committed 70912K, reserved 1048576K // 类空间,已使用69M,JVM向OS申请的预留内存70M,潜在最大可用内存1.0G(虚拟内存)
2025-11-25T10:22:18.364+0800: 54838.841: [GC pause (G1 Evacuation Pause) (young) // 54838表示自JVM启动到现在的时间,单位为s,STW,存活对象拷贝转移到新Region上,young gc
Desired survivor size 134217728 bytes, new threshold 15 (max 15) // 理想survivor区大小为134M,最大阈值15
- age 1: 22631736 bytes, 22631736 total // 存活1次young gc,22M
- age 2: 3914072 bytes, 26545808 total
- age 3: 11398312 bytes, 37944120 total
- age 4: 4621616 bytes, 42565736 total
- age 5: 4064352 bytes, 46630088 total
- age 6: 2745080 bytes, 49375168 total
- age 7: 6074624 bytes, 55449792 total
- age 8: 10838784 bytes, 66288576 total
- age 9: 3798888 bytes, 70087464 total
- age 10: 6146800 bytes, 76234264 total
- age 11: 5616944 bytes, 81851208 total
- age 12: 2407672 bytes, 84258880 total
- age 13: 2391048 bytes, 86649928 total
- age 14: 1436968 bytes, 88086896 total // 存活14次young gc,2M,age1~age14累积88M
54838.841: [G1Ergonomics (CSet Construction) start choosing CSet, _pending_cards: 39031, predicted base time: 22.03 ms, remaining time: 17.97 ms, target pause time: 40.00 ms] // 选择CSet,待处理的卡片39031,预测基础暂停时间22ms,剩余时间17ms
54838.841: [G1Ergonomics (CSet Construction) add young regions to CSet, eden: 235 regions, survivors: 14 regions, predicted young region time: 12.66 ms] // 添加region至CSet,eden有235个region,survivor有14个region,预测回收eden + survivor需要12ms
54838.841: [G1Ergonomics (CSet Construction) finish choosing CSet, eden: 235 regions, survivors: 14 regions, old: 0 regions, predicted pause time: 34.69 ms, target pause time: 40.00 ms] // 完成CSet的选择,eden:235,survivor:14,old:0,预测暂停时间34ms
2025-11-25T10:22:18.398+0800: 54838.875: [SoftReference, 0 refs, 0.0000507 secs]2025-11-25T10:22:18.398+0800: 54838.875: [WeakReference, 0 refs, 0.0000070 secs]2025-11-25T10:22:18.398+0800: 54838.875: [FinalReference, 330 refs, 0.0000523 secs]2025-11-25T10:22:18.398+0800: 54838.875: [PhantomReference, 0 refs, 0 refs, 0.0000063 secs]2025-11-25T10:22:18.398+0800: 54838.875: [JNI Weak Reference, 0.0002955 secs], 0.0362511 secs] // 终结引用有330个
[Parallel Time: 32.2 ms, GC Workers: 10] // 并行GC worker,10个GC线程同时回收,总暂停时间32ms
[GC Worker Start (ms): Min: 54838842.1, Avg: 54838842.2, Max: 54838842.3, Diff: 0.1]
[Ext Root Scanning (ms): Min: 1.9, Avg: 3.3, Max: 9.1, Diff: 7.2, Sum: 32.6] // 扫描外部GC Roots(静态对象、线程栈,总耗时32ms)
[Update RS (ms): Min: 10.1, Avg: 15.7, Max: 17.1, Diff: 7.0, Sum: 157.3] // 更新记忆集RSet,总耗时157ms
[Processed Buffers: Min: 58, Avg: 104.7, Max: 203, Diff: 145, Sum: 1047]
[Scan RS (ms): Min: 2.3, Avg: 2.4, Max: 2.5, Diff: 0.1, Sum: 24.0] // 扫描RSet,找到存活对象,总耗时24ms
[Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] // 扫描JIT代码根对象,总耗时0ms
[Object Copy (ms): Min: 10.4, Avg: 10.5, Max: 10.6, Diff: 0.2, Sum: 105.4] // 对存活对象copy+转移,总耗时105ms
[Termination (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.1] // 并行线程终止同步
[Termination Attempts: Min: 1, Avg: 4.1, Max: 8, Diff: 7, Sum: 41]
[GC Worker Other (ms): Min: 0.0, Avg: 0.0, Max: 0.1, Diff: 0.1, Sum: 0.4]
[GC Worker Total (ms): Min: 31.9, Avg: 32.0, Max: 32.0, Diff: 0.1, Sum: 319.8] // 由于是10个线程并发执行,总耗时319ms
[GC Worker End (ms): Min: 54838874.2, Avg: 54838874.2, Max: 54838874.2, Diff: 0.1]
[Code Root Fixup: 0.2 ms]
[Code Root Purge: 0.0 ms]
[Clear CT: 0.6 ms] // 清理卡表 clear card table
[Other: 3.3 ms]
[Choose CSet: 0.0 ms]
[Ref Proc: 0.7 ms]
[Ref Enq: 0.0 ms]
[Redirty Cards: 0.2 ms]
[Humongous Register: 0.1 ms]
[Humongous Reclaim: 0.0 ms]
[Free CSet: 0.4 ms]
[Eden: 1880.0M(1880.0M)->0.0B(1880.0M) Survivors: 112.0M->112.0M Heap: 9075.9M(10.0G)->7199.9M(10.0G)] // eden由1.8G->0,survivor搬迁后112M保持不变,堆9.0G->7.1G
Heap after GC invocations=3372 (full 0):
garbage-first heap total 10485760K, used 7372719K [0x0000000540000000, 0x0000000540802800, 0x00000007c0000000)
region size 8192K, 14 young (114688K), 14 survivors (114688K)
Metaspace used 438233K, capacity 443533K, committed 446720K, reserved 1425408K
class space used 69311K, capacity 70187K, committed 70912K, reserved 1048576K
}
[Times: user=0.32 sys=0.01, real=0.04 secs] // 10个GC work累积起来CPU使用了0.32s,系统CPU时间0.01s,GC整体耗时40ms(即STW时间为40ms)
......此处省略很多日志信息,主要是抓住上面的一次young gc,下面的是一次mixed gc
2025-11-25T10:23:09.261+0800: 54889.738: [GC concurrent-root-region-scan-start]
2025-11-25T10:23:09.261+0800: 54889.738: Total time for which application threads were stopped: 0.0541826 seconds, Stopping threads took: 0.0003794 seconds
2025-11-25T10:23:09.272+0800: 54889.749: [GC concurrent-root-region-scan-end, 0.0105196 secs]
2025-11-25T10:23:09.272+0800: 54889.749: [GC concurrent-mark-start]
2025-11-25T10:23:09.764+0800: 54890.241: [GC concurrent-mark-end, 0.4918518 secs]
2025-11-25T10:23:09.764+0800: 54890.241: Application time: 0.5022052 seconds
2025-11-25T10:23:09.767+0800: 54890.244: [GC remark 2025-11-25T10:23:09.767+0800: 54890.244: [Finalize Marking, 0.0011891 secs] 2025-11-25T10:23:09.769+0800: 54890.246: [GC ref-proc2025-11-25T10:23:09.769+0800: 54890.246: [SoftReference, 231 refs, 0.0001804 secs]2025-11-25T10:23:09.769+0800: 54890.246: [WeakReference, 1410 refs, 0.0005327 secs]2025-11-25T10:23:09.769+0800: 54890.246: [FinalReference, 142834 refs, 0.3074949 secs]2025-11-25T10:23:10.077+0800: 54890.554: [PhantomReference, 0 refs, 75 refs, 0.0001334 secs]2025-11-25T10:23:10.077+0800: 54890.554: [JNI Weak Reference, 0.0027464 secs], 0.3186995 secs] 2025-11-25T10:23:10.087+0800: 54890.564: [Unloading, 0.0765362 secs], 0.4024319 secs]
[Times: user=0.70 sys=0.37, real=0.41 secs]
2025-11-25T10:23:10.170+0800: 54890.647: Total time for which application threads were stopped: 0.4066764 seconds, Stopping threads took: 0.0004158 seconds
2025-11-25T10:23:10.170+0800: 54890.647: Application time: 0.0001122 seconds
2025-11-25T10:23:10.174+0800: 54890.651: [GC cleanup 7471M->7463M(10G), 0.0331351 secs]
[Times: user=0.32 sys=0.00, real=0.03 secs]
2025-11-25T10:23:10.207+0800: 54890.684: Total time for which application threads were stopped: 0.0370230 seconds, Stopping threads took: 0.0002807 seconds
2025-11-25T10:23:10.207+0800: 54890.685: [GC concurrent-cleanup-start]
2025-11-25T10:23:10.208+0800: 54890.685: [GC concurrent-cleanup-end, 0.0000349 secs]
2025-11-25T10:23:10.226+0800: 54890.703: Application time: 0.0186830 seconds
2025-11-25T10:23:10.232+0800: 54890.709: Total time for which application threads were stopped: 0.0059369 seconds, Stopping threads took: 0.0007812 seconds
2025-11-25T10:23:10.264+0800: 54890.741: Application time: 0.0323801 seconds
{Heap before GC invocations=3386 (full 0):
garbage-first heap total 10485760K, used 7831345K [0x0000000540000000, 0x0000000540802800, 0x00000007c0000000)
region size 8192K, 64 young (524288K), 15 survivors (122880K)
Metaspace used 438233K, capacity 443533K, committed 446720K, reserved 1425408K
class space used 69311K, capacity 70187K, committed 70912K, reserved 1048576K
2025-11-25T10:23:10.269+0800: 54890.746: [GC pause (G1 Evacuation Pause) (young)
Desired survivor size 33554432 bytes, new threshold 4 (max 15)
- age 1: 25594440 bytes, 25594440 total
- age 2: 2250768 bytes, 27845208 total
- age 3: 4198992 bytes, 32044200 total
- age 4: 4975912 bytes, 37020112 total
- age 5: 4380272 bytes, 41400384 total
- age 6: 5889984 bytes, 47290368 total
- age 7: 11192072 bytes, 58482440 total
- age 8: 1776304 bytes, 60258744 total
- age 9: 7334440 bytes, 67593184 total
- age 10: 1684488 bytes, 69277672 total
- age 11: 2989192 bytes, 72266864 total
- age 12: 3355072 bytes, 75621936 total
- age 13: 2714968 bytes, 78336904 total
- age 14: 7023864 bytes, 85360768 total
- age 15: 7301376 bytes, 92662144 total
54890.747: [G1Ergonomics (CSet Construction) start choosing CSet, _pending_cards: 37502, predicted base time: 28.25 ms, remaining time: 11.75 ms, target pause time: 40.00 ms]
54890.747: [G1Ergonomics (CSet Construction) add young regions to CSet, eden: 49 regions, survivors: 15 regions, predicted young region time: 13.33 ms]
54890.747: [G1Ergonomics (CSet Construction) finish choosing CSet, eden: 49 regions, survivors: 15 regions, old: 0 regions, predicted pause time: 41.58 ms, target pause time: 40.00 ms]
2025-11-25T10:23:10.304+0800: 54890.782: [SoftReference, 0 refs, 0.0000483 secs]2025-11-25T10:23:10.305+0800: 54890.782: [WeakReference, 0 refs, 0.0000071 secs]2025-11-25T10:23:10.305+0800: 54890.782: [FinalReference, 2054 refs, 0.0002449 secs]2025-11-25T10:23:10.305+0800: 54890.782: [PhantomReference, 0 refs, 0 refs, 0.0000070 secs]2025-11-25T10:23:10.305+0800: 54890.782: [JNI Weak Reference, 0.0003081 secs] 54890.784: [G1Ergonomics (Mixed GCs) start mixed GCs, reason: candidate old regions available, candidate old regions: 824 regions, reclaimable: 6012264304 bytes (55.99 %), threshold: 5.00 %]
, 0.0376695 secs]
[Parallel Time: 33.7 ms, GC Workers: 10]
[GC Worker Start (ms): Min: 54890747.3, Avg: 54890747.3, Max: 54890747.4, Diff: 0.1]
[Ext Root Scanning (ms): Min: 1.9, Avg: 3.3, Max: 9.7, Diff: 7.8, Sum: 33.3]
[Update RS (ms): Min: 7.5, Avg: 13.8, Max: 15.6, Diff: 8.0, Sum: 137.8]
[Processed Buffers: Min: 38, Avg: 73.2, Max: 100, Diff: 62, Sum: 732]
[Scan RS (ms): Min: 3.2, Avg: 3.6, Max: 3.8, Diff: 0.6, Sum: 36.4]
[Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0]
[Object Copy (ms): Min: 12.4, Avg: 12.7, Max: 12.8, Diff: 0.4, Sum: 126.5]
[Termination (ms): Min: 0.0, Avg: 0.0, Max: 0.1, Diff: 0.0, Sum: 0.4]
[Termination Attempts: Min: 1, Avg: 30.0, Max: 43, Diff: 42, Sum: 300]
[GC Worker Other (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.2]
[GC Worker Total (ms): Min: 33.4, Avg: 33.5, Max: 33.5, Diff: 0.1, Sum: 334.6]
[GC Worker End (ms): Min: 54890780.8, Avg: 54890780.8, Max: 54890780.8, Diff: 0.0]
[Code Root Fixup: 0.4 ms]
[Code Root Purge: 0.0 ms]
[Clear CT: 0.5 ms]
[Other: 3.1 ms]
[Choose CSet: 0.0 ms]
[Ref Proc: 0.8 ms]
[Ref Enq: 0.0 ms]
[Redirty Cards: 0.2 ms]
[Humongous Register: 0.1 ms]
[Humongous Reclaim: 0.0 ms]
[Free CSet: 0.2 ms]
[Eden: 392.0M(392.0M)->0.0B(464.0M) Survivors: 120.0M->48.0M Heap: 7647.8M(10.0G)->7247.8M(10.0G)]
Heap after GC invocations=3387 (full 0):
garbage-first heap total 10485760K, used 7421697K [0x0000000540000000, 0x0000000540802800, 0x00000007c0000000)
region size 8192K, 6 young (49152K), 6 survivors (49152K)
Metaspace used 438233K, capacity 443533K, committed 446720K, reserved 1425408K
class space used 69311K, capacity 70187K, committed 70912K, reserved 1048576K
}
[Times: user=0.33 sys=0.01, real=0.03 secs]
2025-11-25T10:23:10.308+0800: 54890.785: Total time for which application threads were stopped: 0.0431743 seconds, Stopping threads took: 0.0004376 seconds
2025-11-25T10:23:11.165+0800: 54891.642: Application time: 0.8572387 seconds
{Heap before GC invocations=3387 (full 0):
garbage-first heap total 10485760K, used 7896833K [0x0000000540000000, 0x0000000540802800, 0x00000007c0000000)
region size 8192K, 64 young (524288K), 6 survivors (49152K)
Metaspace used 438233K, capacity 443533K, committed 446720K, reserved 1425408K
class space used 69311K, capacity 70187K, committed 70912K, reserved 1048576K
2025-11-25T10:23:11.169+0800: 54891.646: [GC pause (G1 Evacuation Pause) (mixed)
Desired survivor size 33554432 bytes, new threshold 15 (max 15)
- age 1: 11818032 bytes, 11818032 total
- age 2: 14113656 bytes, 25931688 total
- age 3: 2097312 bytes, 28029000 total
- age 4: 4122952 bytes, 32151952 total
54891.646: [G1Ergonomics (CSet Construction) start choosing CSet, _pending_cards: 34128, predicted base time: 25.46 ms, remaining time: 14.54 ms, target pause time: 40.00 ms]
54891.646: [G1Ergonomics (CSet Construction) add young regions to CSet, eden: 58 regions, survivors: 6 regions, predicted young region time: 8.09 ms]
54891.647: [G1Ergonomics (CSet Construction) finish adding old regions to CSet, reason: predicted time is too high, predicted time: 0.25 ms, remaining time: 0.00 ms, old: 103 regions, min: 103 regions]
54891.647: [G1Ergonomics (CSet Construction) added expensive regions to CSet, reason: old CSet region num not reached min, old: 103 regions, expensive: 68 regions, min: 103 regions, remaining time: 0.00 ms]
54891.647: [G1Ergonomics (CSet Construction) finish choosing CSet, eden: 58 regions, survivors: 6 regions, old: 103 regions, predicted pause time: 55.30 ms, target pause time: 40.00 ms]
2025-11-25T10:23:11.211+0800: 54891.688: [SoftReference, 1 refs, 0.0000831 secs]2025-11-25T10:23:11.211+0800: 54891.688: [WeakReference, 0 refs, 0.0000073 secs]2025-11-25T10:23:11.211+0800: 54891.688: [FinalReference, 163 refs, 0.0000294 secs]2025-11-25T10:23:11.211+0800: 54891.688: [PhantomReference, 0 refs, 0 refs, 0.0000067 secs]2025-11-25T10:23:11.211+0800: 54891.688: [JNI Weak Reference, 0.0002868 secs] 54891.691: [G1Ergonomics (Mixed GCs) continue mixed GCs, reason: candidate old regions available, candidate old regions: 721 regions, reclaimable: 5170979216 bytes (48.16 %), threshold: 5.00 %]
, 0.0459042 secs]
[Parallel Time: 40.4 ms, GC Workers: 10]
[GC Worker Start (ms): Min: 54891647.2, Avg: 54891647.3, Max: 54891647.3, Diff: 0.1]
[Ext Root Scanning (ms): Min: 2.0, Avg: 3.3, Max: 9.0, Diff: 7.1, Sum: 33.5]
[Update RS (ms): Min: 3.8, Avg: 9.4, Max: 11.0, Diff: 7.1, Sum: 93.9]
[Processed Buffers: Min: 51, Avg: 78.9, Max: 118, Diff: 67, Sum: 789]
[Scan RS (ms): Min: 19.6, Avg: 19.9, Max: 20.0, Diff: 0.4, Sum: 198.6]
[Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0]
[Object Copy (ms): Min: 7.4, Avg: 7.5, Max: 7.6, Diff: 0.2, Sum: 75.4]
[Termination (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.2]
[Termination Attempts: Min: 1, Avg: 17.1, Max: 25, Diff: 24, Sum: 171]
[GC Worker Other (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.1]
[GC Worker Total (ms): Min: 40.1, Avg: 40.2, Max: 40.2, Diff: 0.1, Sum: 401.8]
[GC Worker End (ms): Min: 54891687.4, Avg: 54891687.4, Max: 54891687.4, Diff: 0.0]
[Code Root Fixup: 0.2 ms]
[Code Root Purge: 0.0 ms]
[Clear CT: 0.5 ms]
[Other: 4.8 ms]
[Choose CSet: 0.6 ms]
[Ref Proc: 0.6 ms]
[Ref Enq: 0.0 ms]
[Redirty Cards: 0.2 ms]
[Humongous Register: 0.1 ms]
[Humongous Reclaim: 0.0 ms]
[Free CSet: 1.5 ms]
[Eden: 464.0M(464.0M)->0.0B(456.0M) Survivors: 48.0M->56.0M Heap: 7711.8M(10.0G)->6476.0M(10.0G)]
Heap after GC invocations=3388 (full 0):
garbage-first heap total 10485760K, used 6631424K [0x0000000540000000, 0x0000000540802800, 0x00000007c0000000)
region size 8192K, 7 young (57344K), 7 survivors (57344K)
Metaspace used 438233K, capacity 443533K, committed 446720K, reserved 1425408K
class space used 69311K, capacity 70187K, committed 70912K, reserved 1048576K
}
[Times: user=0.40 sys=0.01, real=0.05 secs]
2025-11-25T10:23:11.215+0800: 54891.692: Total time for which application threads were stopped: 0.0503583 seconds, Stopping threads took: 0.0003777 seconds
2025-11-25T10:23:12.215+0800: 54892.692: Application time: 1.0000975 seconds
2025-11-25T10:23:12.219+0800: 54892.696: Total time for which application threads were stopped: 0.0038770 seconds, Stopping threads took: 0.0004035 seconds
2025-11-25T10:23:12.419+0800: 54892.896: Application time: 0.1999900 seconds
{Heap before GC invocations=3388 (full 0): // 从JVM启动开始,一共发生过3388次GC(这里指的是young gc + mixed gc),full gc的次数为0
garbage-first heap total 10485760K, used 7098368K [0x0000000540000000, 0x0000000540802800, 0x00000007c0000000) // 堆大小10G,已使用7.0G
region size 8192K, 64 young (524288K), 7 survivors (57344K) // region大小为8M,64个eden region占比524M,7个survivor region占比57M
Metaspace used 438233K, capacity 443533K, committed 446720K, reserved 1425408K
class space used 69311K, capacity 70187K, committed 70912K, reserved 1048576K
2025-11-25T10:23:12.423+0800: 54892.900: [GC pause (G1 Evacuation Pause) (mixed) // STW,触发mixed gc
Desired survivor size 33554432 bytes, new threshold 15 (max 15) // 理想的survivor区33M
- age 1: 11871704 bytes, 11871704 total // age1:11M
- age 2: 2059688 bytes, 13931392 total // age2:2M
- age 3: 13506584 bytes, 27437976 total // age3:13M
- age 4: 2072984 bytes, 29510960 total
- age 5: 3819240 bytes, 33330200 total
54892.900: [G1Ergonomics (CSet Construction) start choosing CSet, _pending_cards: 34397, predicted base time: 23.79 ms, remaining time: 16.21 ms, target pause time: 40.00 ms]
54892.900: [G1Ergonomics (CSet Construction) add young regions to CSet, eden: 57 regions, survivors: 7 regions, predicted young region time: 6.72 ms]
54892.901: [G1Ergonomics (CSet Construction) finish adding old regions to CSet, reason: predicted time is too high, predicted time: 0.39 ms, remaining time: 0.00 ms, old: 103 regions, min: 103 regions]
// 将old region添加至CSet,但是由于预测时间太高,至少要选择103个old region(性价比高的,即存活率低、RS少、回收成本低)添加进CSet,预测时间为0.39ms(另:这0.39ms是这103个old region增加的边际预测时间,不代表完整回收成本)
// min:103region的含义:根据G1的评估模型,如果本次gc不回收103个region,未来mixed gc压力加重,甚至可能逼近full gc
54892.901: [G1Ergonomics (CSet Construction) added expensive regions to CSet, reason: old CSet region num not reached min, old: 103 regions, expensive: 74 regions, min: 103 regions, remaining time: 0.00 ms]
// 由于选择了部分便宜的region,但是还是未能达到,至少回收103old region的要求,不得不再选择74个昂贵的old region至CSet
54892.901: [G1Ergonomics (CSet Construction) finish choosing CSet, eden: 57 regions, survivors: 7 regions, old: 103 regions, predicted pause time: 66.73 ms, target pause time: 40.00 ms]
// 完成选择,57个eden,7个survivor,103个old,预测暂停时间66ms
2025-11-25T10:23:12.476+0800: 54892.953: [SoftReference, 0 refs, 0.0000462 secs]2025-11-25T10:23:12.476+0800: 54892.953: [WeakReference, 0 refs, 0.0000069 secs]2025-11-25T10:23:12.476+0800: 54892.953: [FinalReference, 7 refs, 0.0000158 secs]2025-11-25T10:23:12.476+0800: 54892.953: [PhantomReference, 0 refs, 0 refs, 0.0000066 secs]2025-11-25T10:23:12.476+0800: 54892.953: [JNI Weak Reference, 0.0002829 secs] 54892.957: [G1Ergonomics (Mixed GCs) continue mixed GCs, reason: candidate old regions available, candidate old regions: 618 regions, reclaimable: 4331449584 bytes (40.34 %), threshold: 5.00 %]
, 0.0572976 secs]
[Parallel Time: 50.8 ms, GC Workers: 10]
[GC Worker Start (ms): Min: 54892902.1, Avg: 54892902.2, Max: 54892902.2, Diff: 0.1]
[Ext Root Scanning (ms): Min: 1.9, Avg: 3.3, Max: 9.1, Diff: 7.2, Sum: 33.1]
[Update RS (ms): Min: 7.5, Avg: 13.2, Max: 14.6, Diff: 7.1, Sum: 132.1]
[Processed Buffers: Min: 56, Avg: 82.9, Max: 118, Diff: 62, Sum: 829]
[Scan RS (ms): Min: 25.6, Avg: 25.6, Max: 25.7, Diff: 0.1, Sum: 255.9]
[Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0]
[Object Copy (ms): Min: 8.3, Avg: 8.4, Max: 8.5, Diff: 0.2, Sum: 84.1]
[Termination (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.1]
[Termination Attempts: Min: 1, Avg: 4.9, Max: 10, Diff: 9, Sum: 49]
[GC Worker Other (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.2]
[GC Worker Total (ms): Min: 50.5, Avg: 50.6, Max: 50.6, Diff: 0.1, Sum: 505.5] // 10个线程并行处理,总耗时505ms
[GC Worker End (ms): Min: 54892952.7, Avg: 54892952.7, Max: 54892952.7, Diff: 0.0]
[Code Root Fixup: 0.2 ms]
[Code Root Purge: 0.0 ms]
[Clear CT: 0.5 ms]
[Other: 5.8 ms]
[Choose CSet: 1.1 ms]
[Ref Proc: 0.6 ms]
[Ref Enq: 0.0 ms]
[Redirty Cards: 0.2 ms]
[Humongous Register: 0.1 ms]
[Humongous Reclaim: 0.0 ms]
[Free CSet: 1.9 ms]
[Eden: 456.0M(456.0M)->0.0B(464.0M) Survivors: 56.0M->48.0M Heap: 6932.0M(10.0G)->5656.8M(10.0G)]
Heap after GC invocations=3389 (full 0):
garbage-first heap total 10485760K, used 5792607K [0x0000000540000000, 0x0000000540802800, 0x00000007c0000000)
region size 8192K, 6 young (49152K), 6 survivors (49152K)
Metaspace used 438233K, capacity 443533K, committed 446720K, reserved 1425408K
class space used 69311K, capacity 70187K, committed 70912K, reserved 1048576K
}
[Times: user=0.51 sys=0.01, real=0.06 secs]
2025-11-25T10:23:12.481+0800: 54892.958: Total time for which application threads were stopped: 0.0616297 seconds, Stopping threads took: 0.0003992 seconds
2025-11-25T10:23:13.443+0800: 54893.920: Application time: 0.9618601 seconds
{Heap before GC invocations=3389 (full 0):
garbage-first heap total 10485760K, used 6267743K [0x0000000540000000, 0x0000000540802800, 0x00000007c0000000)
region size 8192K, 64 young (524288K), 6 survivors (49152K)
Metaspace used 438233K, capacity 443533K, committed 446720K, reserved 1425408K
class space used 69311K, capacity 70187K, committed 70912K, reserved 1048576K
2025-11-25T10:23:13.447+0800: 54893.924: [GC pause (G1 Evacuation Pause) (mixed)
Desired survivor size 33554432 bytes, new threshold 15 (max 15)
- age 1: 7724688 bytes, 7724688 total
- age 2: 702288 bytes, 8426976 total
- age 3: 1943536 bytes, 10370512 total
- age 4: 12413992 bytes, 22784504 total
- age 5: 2068200 bytes, 24852704 total
- age 6: 3396752 bytes, 28249456 total
54893.924: [G1Ergonomics (CSet Construction) start choosing CSet, _pending_cards: 38929, predicted base time: 24.78 ms, remaining time: 15.22 ms, target pause time: 40.00 ms]
54893.924: [G1Ergonomics (CSet Construction) add young regions to CSet, eden: 58 regions, survivors: 6 regions, predicted young region time: 7.68 ms]
54893.926: [G1Ergonomics (CSet Construction) finish adding old regions to CSet, reason: predicted time is too high, predicted time: 0.50 ms, remaining time: 0.00 ms, old: 103 regions, min: 103 regions]
54893.926: [G1Ergonomics (CSet Construction) added expensive regions to CSet, reason: old CSet region num not reached min, old: 103 regions, expensive: 86 regions, min: 103 regions, remaining time: 0.00 ms]
54893.926: [G1Ergonomics (CSet Construction) finish choosing CSet, eden: 58 regions, survivors: 6 regions, old: 103 regions, predicted pause time: 80.06 ms, target pause time: 40.00 ms]
2025-11-25T10:23:13.507+0800: 54893.984: [SoftReference, 0 refs, 0.0000529 secs]2025-11-25T10:23:13.507+0800: 54893.984: [WeakReference, 0 refs, 0.0000068 secs]2025-11-25T10:23:13.507+0800: 54893.984: [FinalReference, 59 refs, 0.0000457 secs]2025-11-25T10:23:13.507+0800: 54893.985: [PhantomReference, 0 refs, 0 refs, 0.0000066 secs]2025-11-25T10:23:13.507+0800: 54893.985: [JNI Weak Reference, 0.0003339 secs] 54893.989: [G1Ergonomics (Mixed GCs) continue mixed GCs, reason: candidate old regions available, candidate old regions: 515 regions, reclaimable: 3491747192 bytes (32.52 %), threshold: 5.00 %]
, 0.0653969 secs]
[Parallel Time: 57.8 ms, GC Workers: 10]
[GC Worker Start (ms): Min: 54893926.1, Avg: 54893926.2, Max: 54893926.3, Diff: 0.1]
[Ext Root Scanning (ms): Min: 1.9, Avg: 3.1, Max: 8.1, Diff: 6.3, Sum: 31.4]
[Update RS (ms): Min: 9.1, Avg: 14.0, Max: 15.2, Diff: 6.2, Sum: 140.0]
[Processed Buffers: Min: 41, Avg: 83.0, Max: 124, Diff: 83, Sum: 830]
[Scan RS (ms): Min: 29.9, Avg: 29.9, Max: 30.0, Diff: 0.1, Sum: 299.4]
[Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0]
[Object Copy (ms): Min: 10.4, Avg: 10.5, Max: 10.6, Diff: 0.2, Sum: 105.2]
[Termination (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0]
[Termination Attempts: Min: 1, Avg: 1.7, Max: 3, Diff: 2, Sum: 17]
[GC Worker Other (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.2]
[GC Worker Total (ms): Min: 57.6, Avg: 57.6, Max: 57.7, Diff: 0.1, Sum: 576.2]
[GC Worker End (ms): Min: 54893983.8, Avg: 54893983.8, Max: 54893983.8, Diff: 0.0]
[Code Root Fixup: 0.2 ms]
[Code Root Purge: 0.0 ms]
[Clear CT: 0.6 ms]
[Other: 6.7 ms]
[Choose CSet: 1.6 ms]
[Ref Proc: 0.7 ms]
[Ref Enq: 0.0 ms]
[Redirty Cards: 0.2 ms]
[Humongous Register: 0.1 ms]
[Humongous Reclaim: 0.0 ms]
[Free CSet: 2.3 ms]
[Eden: 464.0M(464.0M)->0.0B(456.0M) Survivors: 48.0M->56.0M Heap: 6120.8M(10.0G)->4861.8M(10.0G)]
Heap after GC invocations=3390 (full 0):
garbage-first heap total 10485760K, used 4978445K [0x0000000540000000, 0x0000000540802800, 0x00000007c0000000)
region size 8192K, 7 young (57344K), 7 survivors (57344K)
Metaspace used 438233K, capacity 443533K, committed 446720K, reserved 1425408K
class space used 69311K, capacity 70187K, committed 70912K, reserved 1048576K
}
[Times: user=0.58 sys=0.01, real=0.07 secs]
2025-11-25T10:23:13.513+0800: 54893.990: Total time for which application threads were stopped: 0.0698825 seconds, Stopping threads took: 0.0004164 seconds
2025-11-25T10:23:14.108+0800: 54894.585: Application time: 0.5956209 seconds
{Heap before GC invocations=3390 (full 0):
garbage-first heap total 10485760K, used 5445389K [0x0000000540000000, 0x0000000540802800, 0x00000007c0000000)
region size 8192K, 64 young (524288K), 7 survivors (57344K)
Metaspace used 438233K, capacity 443533K, committed 446720K, reserved 1425408K
class space used 69311K, capacity 70187K, committed 70912K, reserved 1048576K
2025-11-25T10:23:14.112+0800: 54894.589: [GC pause (G1 Evacuation Pause) (mixed)
Desired survivor size 33554432 bytes, new threshold 7 (max 15)
- age 1: 15958424 bytes, 15958424 total
- age 2: 1214920 bytes, 17173344 total
- age 3: 634608 bytes, 17807952 total
- age 4: 1703320 bytes, 19511272 total
- age 5: 12131752 bytes, 31643024 total
- age 6: 1741312 bytes, 33384336 total
- age 7: 3366880 bytes, 36751216 total
54894.589: [G1Ergonomics (CSet Construction) start choosing CSet, _pending_cards: 40514, predicted base time: 24.46 ms, remaining time: 15.54 ms, target pause time: 40.00 ms]
54894.589: [G1Ergonomics (CSet Construction) add young regions to CSet, eden: 57 regions, survivors: 7 regions, predicted young region time: 11.11 ms]
54894.592: [G1Ergonomics (CSet Construction) finish adding old regions to CSet, reason: predicted time is too high, predicted time: 0.66 ms, remaining time: 0.00 ms, old: 103 regions, min: 103 regions]
54894.592: [G1Ergonomics (CSet Construction) added expensive regions to CSet, reason: old CSet region num not reached min, old: 103 regions, expensive: 95 regions, min: 103 regions, remaining time: 0.00 ms]
54894.592: [G1Ergonomics (CSet Construction) finish choosing CSet, eden: 57 regions, survivors: 7 regions, old: 103 regions, predicted pause time: 96.55 ms, target pause time: 40.00 ms]
2025-11-25T10:23:14.180+0800: 54894.657: [SoftReference, 0 refs, 0.0000496 secs]2025-11-25T10:23:14.180+0800: 54894.657: [WeakReference, 0 refs, 0.0000154 secs]2025-11-25T10:23:14.180+0800: 54894.657: [FinalReference, 5 refs, 0.0000208 secs]2025-11-25T10:23:14.180+0800: 54894.657: [PhantomReference, 0 refs, 1 refs, 0.0000076 secs]2025-11-25T10:23:14.180+0800: 54894.657: [JNI Weak Reference, 0.0002843 secs] 54894.662: [G1Ergonomics (Mixed GCs) continue mixed GCs, reason: candidate old regions available, candidate old regions: 412 regions, reclaimable: 2654644096 bytes (24.72 %), threshold: 5.00 %]
, 0.0732945 secs]
[Parallel Time: 64.6 ms, GC Workers: 10]
[GC Worker Start (ms): Min: 54894592.2, Avg: 54894592.3, Max: 54894592.4, Diff: 0.1]
[Ext Root Scanning (ms): Min: 1.8, Avg: 3.2, Max: 8.7, Diff: 6.8, Sum: 32.1]
[Update RS (ms): Min: 8.2, Avg: 13.7, Max: 15.3, Diff: 7.1, Sum: 136.9]
[Processed Buffers: Min: 41, Avg: 88.3, Max: 126, Diff: 85, Sum: 883]
[Scan RS (ms): Min: 34.0, Avg: 34.3, Max: 34.4, Diff: 0.4, Sum: 342.8]
[Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0]
[Object Copy (ms): Min: 13.1, Avg: 13.2, Max: 13.3, Diff: 0.2, Sum: 132.4]
[Termination (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.1]
[Termination Attempts: Min: 1, Avg: 6.4, Max: 11, Diff: 10, Sum: 64]
[GC Worker Other (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.3]
[GC Worker Total (ms): Min: 64.4, Avg: 64.5, Max: 64.5, Diff: 0.1, Sum: 644.5]
[GC Worker End (ms): Min: 54894656.7, Avg: 54894656.8, Max: 54894656.8, Diff: 0.0]
[Code Root Fixup: 0.3 ms]
[Code Root Purge: 0.0 ms]
[Clear CT: 0.5 ms]
[Other: 7.8 ms]
[Choose CSet: 2.2 ms]
[Ref Proc: 0.6 ms]
[Ref Enq: 0.0 ms]
[Redirty Cards: 0.3 ms]
[Humongous Register: 0.1 ms]
[Humongous Reclaim: 0.0 ms]
[Free CSet: 2.8 ms]
[Eden: 456.0M(456.0M)->0.0B(448.0M) Survivors: 56.0M->64.0M Heap: 5317.8M(10.0G)->4078.3M(10.0G)]
Heap after GC invocations=3391 (full 0):
garbage-first heap total 10485760K, used 4176182K [0x0000000540000000, 0x0000000540802800, 0x00000007c0000000)
region size 8192K, 8 young (65536K), 8 survivors (65536K)
Metaspace used 438233K, capacity 443533K, committed 446720K, reserved 1425408K
class space used 69311K, capacity 70187K, committed 70912K, reserved 1048576K
}
[Times: user=0.65 sys=0.01, real=0.07 secs]
2025-11-25T10:23:14.186+0800: 54894.663: Total time for which application threads were stopped: 0.0777673 seconds, Stopping threads took: 0.0003897 seconds
2025-11-25T10:23:14.882+0800: 54895.360: Application time: 0.6965141 seconds
{Heap before GC invocations=3391 (full 0):
garbage-first heap total 10485760K, used 4634934K [0x0000000540000000, 0x0000000540802800, 0x00000007c0000000)
region size 8192K, 64 young (524288K), 8 survivors (65536K)
Metaspace used 438233K, capacity 443533K, committed 446720K, reserved 1425408K
class space used 69311K, capacity 70187K, committed 70912K, reserved 1048576K
2025-11-25T10:23:14.886+0800: 54895.363: [GC pause (G1 Evacuation Pause) (mixed)
Desired survivor size 33554432 bytes, new threshold 6 (max 15)
- age 1: 18416568 bytes, 18416568 total
- age 2: 4592136 bytes, 23008704 total
- age 3: 1010608 bytes, 24019312 total
- age 4: 574408 bytes, 24593720 total
- age 5: 1526056 bytes, 26119776 total
- age 6: 11596080 bytes, 37715856 total
- age 7: 1584672 bytes, 39300528 total
54895.364: [G1Ergonomics (CSet Construction) start choosing CSet, _pending_cards: 41932, predicted base time: 24.04 ms, remaining time: 15.96 ms, target pause time: 40.00 ms]
54895.364: [G1Ergonomics (CSet Construction) add young regions to CSet, eden: 56 regions, survivors: 8 regions, predicted young region time: 9.41 ms]
54895.367: [G1Ergonomics (CSet Construction) finish adding old regions to CSet, reason: predicted time is too high, predicted time: 1.05 ms, remaining time: 0.00 ms, old: 103 regions, min: 103 regions]
54895.367: [G1Ergonomics (CSet Construction) added expensive regions to CSet, reason: old CSet region num not reached min, old: 103 regions, expensive: 94 regions, min: 103 regions, remaining time: 0.00 ms]
54895.367: [G1Ergonomics (CSet Construction) finish choosing CSet, eden: 56 regions, survivors: 8 regions, old: 103 regions, predicted pause time: 117.26 ms, target pause time: 40.00 ms]
2025-11-25T10:23:14.965+0800: 54895.442: [SoftReference, 0 refs, 0.0000558 secs]2025-11-25T10:23:14.965+0800: 54895.442: [WeakReference, 0 refs, 0.0000069 secs]2025-11-25T10:23:14.965+0800: 54895.442: [FinalReference, 23 refs, 0.0000112 secs]2025-11-25T10:23:14.965+0800: 54895.442: [PhantomReference, 0 refs, 0 refs, 0.0000061 secs]2025-11-25T10:23:14.965+0800: 54895.442: [JNI Weak Reference, 0.0002860 secs] 54895.448: [G1Ergonomics (Mixed GCs) continue mixed GCs, reason: candidate old regions available, candidate old regions: 309 regions, reclaimable: 1832794208 bytes (17.07 %), threshold: 5.00 %]
, 0.0849832 secs]
// 继续进行mixed gc,可回收的old region有309个,可回收17.07%的内存。阈值为5%,即可回收内存>阈值时,会继续回收
[Parallel Time: 74.7 ms, GC Workers: 10]
[GC Worker Start (ms): Min: 54895367.3, Avg: 54895367.4, Max: 54895367.5, Diff: 0.1]
[Ext Root Scanning (ms): Min: 1.6, Avg: 3.0, Max: 8.3, Diff: 6.6, Sum: 29.7]
[Update RS (ms): Min: 8.6, Avg: 13.7, Max: 15.0, Diff: 6.4, Sum: 137.0]
[Processed Buffers: Min: 45, Avg: 84.0, Max: 113, Diff: 68, Sum: 840]
[Scan RS (ms): Min: 38.8, Avg: 39.0, Max: 39.1, Diff: 0.3, Sum: 389.7]
[Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0]
[Object Copy (ms): Min: 18.6, Avg: 18.8, Max: 19.0, Diff: 0.4, Sum: 187.8]
[Termination (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.1]
[Termination Attempts: Min: 1, Avg: 4.2, Max: 9, Diff: 8, Sum: 42]
[GC Worker Other (ms): Min: 0.0, Avg: 0.0, Max: 0.1, Diff: 0.0, Sum: 0.3]
[GC Worker Total (ms): Min: 74.4, Avg: 74.5, Max: 74.5, Diff: 0.1, Sum: 744.6]
[GC Worker End (ms): Min: 54895441.9, Avg: 54895441.9, Max: 54895441.9, Diff: 0.0]
[Code Root Fixup: 0.2 ms]
[Code Root Purge: 0.0 ms]
[Clear CT: 0.5 ms]
[Other: 9.5 ms]
[Choose CSet: 3.2 ms]
[Ref Proc: 0.6 ms]
[Ref Enq: 0.0 ms]
[Redirty Cards: 0.3 ms]
[Humongous Register: 0.1 ms]
[Humongous Reclaim: 0.0 ms]
[Free CSet: 3.5 ms]
[Eden: 448.0M(448.0M)->0.0B(464.0M) Survivors: 64.0M->48.0M Heap: 4526.3M(10.0G)->3296.4M(10.0G)]
Heap after GC invocations=3392 (full 0):
garbage-first heap total 10485760K, used 3375564K [0x0000000540000000, 0x0000000540802800, 0x00000007c0000000)
region size 8192K, 6 young (49152K), 6 survivors (49152K)
Metaspace used 438233K, capacity 443533K, committed 446720K, reserved 1425408K
class space used 69311K, capacity 70187K, committed 70912K, reserved 1048576K
}
[Times: user=0.75 sys=0.00, real=0.09 secs]
在沙箱环境中,统计工具每5分钟聚合一次检索接口的性能数据,重点关注 P99、P99.9、P99.99、P99.999高分位耗时以及最大耗时(Max)。从统计结果可以看到,在 10:25 时间点接口耗时出现明显抖动,多项高分位指标均超过了 230ms 的超时时间阈值 。 
进一步结合 GC 日志分析发现,在 10:23 左右发生了多次 Mixed GC 。这些 Mixed GC 的 STW(Stop-The-World)暂停时间 分别达到了 60ms、90ms 等 ,显著超过了 JVM 配置的 MaxGCPauseTime=40ms 。接口在 10:25 统计数据的高延迟现象,与前后时间段内频繁且耗时较长的 Mixed GC 具有较强的时间相关性。
推断链路:由于苛刻的 MaxGCPauseTime=40 ,垃圾收集器会倾向于通过 抢占更多 CPU 资源 来尽可能压缩 STW 暂停时间。但是Mixed GC的STW时间仍然远超设定的MaxGCPauseTime,GC 线程对 CPU 的抢占引发了明显的 CPU 突刺,进一步挤压了业务线程的执行资源,最终导致该时间窗口内业务接口响应时间显著上升,并出现大量超时。
结论:通过调整JVM的参数来降低Young GC和Mixed GC的次数【理想情况下降低GC次数,也即降低GC总耗时】
为此,采用对比实验法【用控制变量的手段,将线上版本作为baseline,其他版本每次只更改一个参数,copy线上单台实例流量到测试环境,效果如下】
备注:本次调整保证MaxGCPauseTime=40的条件不能调整,原因是站内和站外代码是同一套,站外检索接口的耗时为115ms,改动MaxGCPauseTime将会对站外产生较大影响
对比实验一【请求流量和机器配置完全一致,仅保证某一个变量变化】:
| 参数变更 | Young GC次数 | Mixed GC次数 | 备注 |
|---|---|---|---|
| 无 | 12275 | 28 | baseline |
| -Xmx10g -Xms10g -> -Xmx12g -Xms12g | 10517 | 15 | |
| -XX:ParallelGCThreads=10->12 | 11991 | 8 | |
| -XX:ParallelGCThreads=10->8 | ? | ? | 实验数据丢失,印象中是负向效果 |
| -XX:InitatingHeapOccupancyPercent=70->50 | 10405 | 21 | |
| -XX:MaxTenuringThreshold=15->5 | 13290 | 85 |
对比实验二【请求流量和机器配置完全一致,仅保证某一个变量变化】:
| 参数变更 | Young GC次数 | Mixed GC次数 | 备注 |
|---|---|---|---|
| JDK1.8.65 | 11875 | 23 | baseline |
| JDK1.8.341 | 7484 | 21 |
对比实验三【请求流量和机器配置完全一致,仅保证某一个变量变化】:
| 参数变更 | Young GC次数 | Mixed GC次数 | 备注 |
|---|---|---|---|
| ConcGCThread=8 | 8332 | 8 | baseline |
| ConcGCThread=4 | 5725 | 8 |
经过上面几个实验的数据对比,将young gc和mixed gc均下降的参数调整可视为正向效果。
【舍弃堆内存调大至12G】虽然堆内存调整为12G的确有正向效果,但是机器内存大小为16G,也就是说堆外内存【元空间、直接内存、JIT编译后的机器码、JVM自身内存、线程栈等信息】仅剩4G。元空间已经占用了1G,我们假设机器有1024个线程(1个线程默认1M),线程栈占用1G,JVM自身内存占用500M,加起来已经占用了2.5G。我们本身是广告检索系统,很可能出现qps突刺,为防止OOM,保守起见堆内存仍保持不变
【最终结论】
js
ConcGCThreads=8 -> 4 // 并发标记阶段的线程数。降低该值,降低对GC线程抢占业务线程,降低CPU突刺
ParallelGCThreads=10 -> 12 // 并行垃圾回收过程中的线程数。增加改值,加快回收速度,降低young gc和mixed gc
JDK version:1.8.65 -> 1.8.341 // 341版本是2022年发布的,相比65版本是2015发布的,在GC方面算法层面做了一些优化
删除-XX:GCTimeRatio=39
删除-XX:G1RSetUpdatingPauseTimePercent=40
详细参数如下
js
// 优化前(JVM日志为本次添加的,用于JVM日志分析用的,实际上线时不带上线)
-XX:+UseG1GC
-Xmx10g
-Xms10g
-XX:ConcGCThreads=8
-XX:GCTimeRatio=39
-XX:G1HeapRegionSize=8m
-XX:MaxGCPauseMillis=40
-XX:ParallelGCThreads=10
-XX:G1RSetUpdatingPauseTimePercent=40
-XX:MetaspaceSize=512M
-XX:MaxMetaspaceSize=1024M
-XX:MaxTenuringThreshold=15
-XX:InitiatingHeapOccupancyPercent=70
-XX:-UseBiasedLocking
-Xloggc:/opt/XX/XX_gc.log
-XX:+PrintGCDetails
-XX:+PrintGCTimeStamps
-XX:+PrintClassHistogramAfterFullGC
-XX:+PrintClassHistogramBeforeFullGC
-XX:+PrintGCApplicationConcurrentTime
-XX:+PrintGCApplicationStoppedTime
-XX:+PrintGCDateStamps
-XX:+UseGCLogFileRotation
-XX:NumberOfGCLogFiles=3
-XX:GCLogFileSize=100M
-XX:+PrintTenuringDistribution
-XX:+UnlockDiagnosticVMOptions
-XX:+G1SummarizeRSetStats
-XX:+G1SummarizeConcMark
-XX:+PrintHeapAtGC
-XX:+PrintReferenceGC
-XX:+PrintAdaptiveSizePolicy
// 优化后
-XX:+UseG1GC
-Xmx10g
-Xms10g
-XX:ConcGCThreads=4
-XX:G1HeapRegionSize=8m
-XX:MaxGCPauseMillis=40
-XX:ParallelGCThreads=12
-XX:MetaspaceSize=512M
-XX:MaxMetaspaceSize=1024M
-XX:MaxTenuringThreshold=15
-XX:InitiatingHeapOccupancyPercent=70
-XX:-UseBiasedLocking
-Xloggc:/opt/XX/XX_gc.log
-XX:+PrintGCDetails
-XX:+PrintGCTimeStamps
-XX:+PrintClassHistogramAfterFullGC
-XX:+PrintClassHistogramBeforeFullGC
-XX:+PrintGCApplicationConcurrentTime
-XX:+PrintGCApplicationStoppedTime
-XX:+PrintGCDateStamps
-XX:+UseGCLogFileRotation
-XX:NumberOfGCLogFiles=3
-XX:GCLogFileSize=100M
-XX:+PrintTenuringDistribution
-XX:+UnlockDiagnosticVMOptions
-XX:+G1SummarizeRSetStats
-XX:+G1SummarizeConcMark
-XX:+PrintHeapAtGC
-XX:+PrintReferenceGC
-XX:+PrintAdaptiveSizePolicy
JDK版本升级:1.8.65 -> 1.8.341
【沙箱环境验证】(站内和站外分别实验)
站内环境
| 参数变更 | Young GC次数 | Mixed GC次数 | 备注 |
|---|---|---|---|
| online版本 | 12907 | 35 | baseline |
| 优化版本 | 5554 | 8 |
结论:站内机器的GC次数明显下降,CPU突刺下降,99.99%分位的超时突刺下降
站外耗时图

站外JVM监控图
online版本
优化版本

结论:站外机器的GC次数下降,整体也呈现下降趋势(约下降5ms左右)
综上:准备上线验证
问题解决
【上线效果分析】2025-12-10推全代码
1.CPU突刺减少(由于MaxGCPauseTime=40的目标太严苛,很难完全消灭CPU突刺)

2.GC次数和GC耗时下降

3.CPU突刺带给站内流量的超时下降(之前每天超时量在5000+)

问题总结
总结:经过本次的调优学到了很多G1GC的知识,也看了很多官网和G1GC的书籍,以及和ChatGPT的大量讨论,收获颇丰。
如果您觉得有收获的话,欢迎点赞、评价、收藏,以及关注github:github.com/HangboQuan
后续计划分享更多G1GC的笔记