一份GC日志的解读

原始日志

cpp 复制代码

numactl -N 0 -m 0 /home/test/spark-3.3.3-bin-hadoop3/bin/spark-sql \
        --master local[14] \
		--driver-java-options "-Xloggc:/home/test/profiler/driver_gc.log  -XX:+PrintGCDetails -XX:+PrintGCTimeStamps" \
        --driver-memory 90g \
        --database tpcds_parquet_tpcds_scale_200_db \
        -f /home/test/sql/q83.sql

cpp 复制代码

$ cat driver_gc.log
OpenJDK 64-Bit Server VM (25.372-b07) for linux-amd64 JRE (1.8.0_372-b07), built on Apr 19 2023 08:57:57 by "jenkins" with gcc 7.5.0
Memory: 4k page, physical 788161268k(744451844k free), swap 0k(0k free)
CommandLine flags: -XX:+IgnoreUnrecognizedVMOptions -XX:InitialHeapSize=2147483648 -XX:MaxHeapSize=96636764160 -XX:+PrintGC -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+UseParallelGC
4.014: [GC (Metadata GC Threshold) [PSYoungGen: 188929K->16120K(611840K)] 188929K->16144K(2010112K), 0.0119549 secs] [Times: user=0.04 sys=0.01, real=0.01 secs]
4.026: [Full GC (Metadata GC Threshold) [PSYoungGen: 16120K->0K(611840K)] [ParOldGen: 24K->15249K(540160K)] 16144K->15249K(1152000K), [Metaspace: 20356K->20356K(22528K)], 0.0197266 secs] [Times: user=0.12 sys=0.01, real=0.02 secs]
6.637: [GC (Metadata GC Threshold) [PSYoungGen: 141761K->16338K(611840K)] 157011K->31660K(1152000K), 0.0111804 secs] [Times: user=0.03 sys=0.00, real=0.00 secs]
6.649: [Full GC (Metadata GC Threshold) [PSYoungGen: 16338K->0K(611840K)] [ParOldGen: 15321K->21851K(899072K)] 31660K->21851K(1510912K), [Metaspace: 33903K->33903K(36864K)], 0.0263242 secs] [Times: user=0.09 sys=0.03, real=0.03 secs]
8.640: [GC (Metadata GC Threshold) [PSYoungGen: 314953K->27022K(611840K)] 336804K->48881K(1510912K), 0.0138599 secs] [Times: user=0.04 sys=0.02, real=0.01 secs]
8.654: [Full GC (Metadata GC Threshold) [PSYoungGen: 27022K->0K(611840K)] [ParOldGen: 21859K->36404K(1327104K)] 48881K->36404K(1938944K), [Metaspace: 56265K->56255K(59392K)], 0.0656241 secs] [Times: user=0.41 sys=0.00, real=0.07 secs]
12.778: [GC (Allocation Failure) [PSYoungGen: 524800K->45799K(691200K)] 561204K->82212K(2018304K), 0.0232692 secs] [Times: user=0.06 sys=0.02, real=0.02 secs]
13.651: [GC (Metadata GC Threshold) [PSYoungGen: 177058K->26954K(762880K)] 213471K->63391K(2089984K), 0.0127219 secs] [Times: user=0.09 sys=0.02, real=0.01 secs]
13.664: [Full GC (Metadata GC Threshold) [PSYoungGen: 26954K->0K(762880K)] [ParOldGen: 36436K->47198K(1857024K)] 63391K->47198K(2619904K), [Metaspace: 94603K->94603K(100352K)], 0.0567371 secs] [Times: user=0.33 sys=0.02, real=0.06 secs]
19.404: [GC (Allocation Failure) [PSYoungGen: 675840K->42995K(871424K)] 723038K->112450K(2728448K), 0.0311273 secs] [Times: user=0.13 sys=0.02, real=0.03 secs]
19.952: [GC (Allocation Failure) [PSYoungGen: 871411K->51701K(890880K)] 940866K->121172K(2747904K), 0.0296181 secs] [Times: user=0.18 sys=0.01, real=0.03 secs]
20.218: [GC (Allocation Failure) [PSYoungGen: 880117K->67582K(1347584K)] 949588K->144965K(3204608K), 0.1164347 secs] [Times: user=0.78 sys=0.03, real=0.11 secs]
20.679: [GC (Allocation Failure) [PSYoungGen: 1347582K->81398K(1361408K)] 1424965K->174117K(3218432K), 0.1862208 secs] [Times: user=1.04 sys=0.05, real=0.18 secs]
21.187: [GC (GCLocker Initiated GC) [PSYoungGen: 1361398K->75568K(2079232K)] 1454117K->199370K(3936256K), 0.1962536 secs] [Times: user=1.12 sys=0.04, real=0.20 secs]
21.931: [GC (Allocation Failure) [PSYoungGen: 2054960K->90928K(2084864K)] 2178767K->221756K(3941888K), 0.2904077 secs] [Times: user=1.58 sys=0.04, real=0.29 secs]
22.735: [GC (GCLocker Initiated GC) [PSYoungGen: 2070320K->115184K(3198464K)] 2201162K->252834K(5055488K), 0.2870335 secs] [Times: user=1.62 sys=0.05, real=0.29 secs]
24.994: [GC (Allocation Failure) [PSYoungGen: 3198448K->134140K(3217408K)] 3336103K->292387K(5074432K), 0.4149455 secs] [Times: user=2.31 sys=0.05, real=0.42 secs]
26.166: [GC (Allocation Failure) [PSYoungGen: 3217404K->139280K(4978688K)] 3375651K->342576K(6835712K), 0.4512913 secs] [Times: user=2.49 sys=0.06, real=0.46 secs]
27.426: [GC (Metadata GC Threshold) [PSYoungGen: 2227994K->100740K(4995072K)] 2431289K->314205K(6852096K), 0.2786383 secs] [Times: user=1.50 sys=0.04, real=0.28 secs]
27.704: [Full GC (Metadata GC Threshold) [PSYoungGen: 100740K->0K(4995072K)] [ParOldGen: 213464K->191716K(2071552K)] 314205K->191716K(7066624K), [Metaspace: 145232K->137818K(165888K)], 0.1685947 secs] [Times: user=1.01 sys=0.01, real=0.17 secs]
29.746: [GC (GCLocker Initiated GC) [PSYoungGen: 4816896K->177147K(7703040K)] 5008616K->398346K(9774592K), 0.6594221 secs] [Times: user=3.70 sys=0.08, real=0.66 secs]
35.203: [GC (Allocation Failure) [PSYoungGen: 7703035K->214527K(7740416K)] 7924269K->1964935K(9811968K), 0.3887560 secs] [Times: user=3.17 sys=0.89, real=0.39 secs]
35.592: [Full GC (Ergonomics) [PSYoungGen: 214527K->0K(7740416K)] [ParOldGen: 1750407K->1180998K(3698176K)] 1964935K->1180998K(11438592K), [Metaspace: 168023K->164336K(188416K)], 0.4731631 secs] [Times: user=2.58 sys=0.07, real=0.47 secs]
Heap
 PSYoungGen      total 7740416K, used 2221660K [0x00001503dd000000, 0x00001506b8900000, 0x0000150b5d000000)
  eden space 7525888K, 29% used [0x00001503dd000000,0x0000150464997320,0x00001505a8580000)
  from space 214528K, 0% used [0x00001505a8580000,0x00001505a8580000,0x00001505b5700000)
  to   space 813568K, 0% used [0x0000150686e80000,0x0000150686e80000,0x00001506b8900000)
 ParOldGen       total 3698176K, used 1180998K [0x000014f4dd000000, 0x000014f5beb80000, 0x00001503dd000000)
  object space 3698176K, 31% used [0x000014f4dd000000,0x000014f525151870,0x000014f5beb80000)
 Metaspace       used 166650K, capacity 181271K, committed 188160K, reserved 188416K

Metadata GC Threshold

只要 GC 原因是 Metadata GC Threshold，Minor GC 之后必跟一次 Full GC。