简单的springboot应用部署后内存占用量过大问题排查

1.问题背景

需要部署一个演示环境。所有组件都要部署到一台服务器，采用Docker容器部署，发现多个简单的springboot应用占用内存高达2G，后续的应用因为内存不足就部署不了了。排查下内存占用大的原因：

docker stats命令：

2.排查步骤：

不想看过程的可以直接查看文末总结

jstats查下应用堆内存情况：

[root@968204e40b opt]# jstat -gc 1 1000 10
 S0C    S1C    S0U    S1U      EC       EU        OC         OU       MC     MU    CCSC   CCSU   YGC     YGCT    FGC    FGCT     GCT
27136.0 26624.0  0.0    0.0   2508800.0 1676722.1  320512.0   23699.8   59160.0 56533.7 8320.0 7819.9     13    0.414   3      0.280
27136.0 26624.0  0.0    0.0   2508800.0 1676722.1  320512.0   23699.8   59160.0 56533.7 8320.0 7819.9     13    0.414   3      0.280
27136.0 26624.0  0.0    0.0   2508800.0 1676722.1  320512.0   23699.8   59160.0 56533.7 8320.0 7819.9     13    0.414   3      0.280

EC EU区居然比OC OU区还大,青年代大于老年代 也就是说青年代没有设置或者设置的过大了。青年代过大触发GC频率过低。造成大量垃圾对象堆积在青年代不回收。而Eden区多次扩容导致。

NGCMN：新生代最小容量

NGCMX：新生代最大容量

NGC：当前新生代容量

S0CMX：最大幸存1区大小

S0C：当前幸存1区大小

S1CMX：最大幸存2区大小

S1C：当前幸存2区大小

ECMX：最大伊甸园区大小

EC：当前伊甸园区大小

YGC：年轻代垃圾回收次数

FGC：老年代回收次数

jmap查看应用内存详情：

-- 问题发现 eden区内存大
[root@576aea1126d9 opt]# jmap -heap 1
Attaching to process ID 1, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 25.191-b12

using thread-local object allocation.
Parallel GC with 4 thread(s)

Heap Configuration:
   MinHeapFreeRatio         = 0
   MaxHeapFreeRatio         = 100
   MaxHeapSize              = 8317304832 (7932.0MB)
   NewSize                  = 173015040 (165.0MB)
   MaxNewSize               = 2772434944 (2644.0MB)
   OldSize                  = 347078656 (331.0MB)
   NewRatio                 = 2
   SurvivorRatio            = 8
   MetaspaceSize            = 21807104 (20.796875MB)
   CompressedClassSpaceSize = 1073741824 (1024.0MB)
   MaxMetaspaceSize         = 17592186044415 MB
   G1HeapRegionSize         = 0 (0.0MB)

Heap Usage:
PS Young Generation
Eden Space:
   capacity = 2402811904 (2291.5MB)
   used     = 1686518448 (1608.3893280029297MB) #应用刚启动 eden区就使用了1.6G内存  
   free     = 716293456 (683.1106719970703MB)
   70.18936626676542% used
From Space:
   capacity = 27262976 (26.0MB)
   used     = 0 (0.0MB)
   free     = 27262976 (26.0MB)
   0.0% used
To Space:
   capacity = 27787264 (26.5MB)
   used     = 0 (0.0MB)
   free     = 27787264 (26.5MB)
   0.0% used
PS Old Generation
   capacity = 350224384 (334.0MB)
   used     = 25793584 (24.598678588867188MB)
   free     = 324430800 (309.4013214111328MB)
   7.364873829002152% used

27924 interned Strings occupying 2735400 bytes.

查看启动命令：

docker run -d --cap-add=SYS_PTRACE -e JAVA_OPTS=' -XX:MetaspaceSize=128m -XX:MaxMetaspaceSize=256m -Xms128m -Xmx128m -Xmn32m -Xss256k -XX:SurvivorRatio=6 -XX:+UseConcMarkSweepGC -Djava.net.preferIPv4Stack=true -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djava.security.egd=file:/dev/./urandom -Dlog4j2.formatMsgNoLookups=true' 【镜像：版本】

--cap-add=SYS_PTRACE 用来解决docker容器内不能运行 jmap命令

说明启动参数：JAVA_OPTS 并没有生效。

一顿百度加实验后只有：将JVM参数加入Dockerfile中才生效

FROM med/oraclejdk:8u191
ADD hcp-terminal-app-1.0.0.jar /opt/app.jar
WORKDIR /opt
EXPOSE 8080
ENTRYPOINT ["java", "-jar","-XX:MaxMetaspaceSize=256m","-Xms128m","-Xmx128m","-Xmn32m","-XX:SurvivorRatio=6","-Dlog4j2.configuration=/opt/log4j2.properties","app.jar","--spring.config.location=/opt/application.yaml"]

重新构建镜像后启动

修改参数后内存使用量情况：

[root@15972b64bff3 opt]# jps
1 jar
205 Jps
[root@15972b64bff3 opt]# jstat -gc 1 1000 2
 S0C    S1C    S0U    S1U      EC       EU        OC         OU       MC     MU    CCSC   CCSU   YGC     YGCT    FGC    FGCT     GCT
1536.0 1536.0 256.0   0.0   29696.0   5130.0   98304.0    76430.8   83712.0 79187.7 11520.0 10663.5    406    1.784   3      0.329    2.113
1536.0 1536.0 256.0   0.0   29696.0   5130.0   98304.0    76430.8   83712.0 79187.7 11520.0 10663.5    406    1.784   3      0.329    2.113

#eden区使用量[EU]下去了,OU使用量也是合理的

[root@15972b64bff3 opt]# jmap -heap 1
Attaching to process ID 1, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 25.191-b12

using thread-local object allocation.
Parallel GC with 4 thread(s)

Heap Configuration:
   MinHeapFreeRatio         = 0
   MaxHeapFreeRatio         = 100
   MaxHeapSize              = 134217728 (128.0MB)
   NewSize                  = 33554432 (32.0MB)
   MaxNewSize               = 33554432 (32.0MB)
   OldSize                  = 100663296 (96.0MB)
   NewRatio                 = 2
   SurvivorRatio            = 6
   MetaspaceSize            = 21807104 (20.796875MB)
   CompressedClassSpaceSize = 260046848 (248.0MB)
   MaxMetaspaceSize         = 268435456 (256.0MB)
   G1HeapRegionSize         = 0 (0.0MB)

Heap Usage:
PS Young Generation
Eden Space:
   capacity = 30408704 (29.0MB)
   used     = 7501416 (7.153907775878906MB)
   free     = 22907288 (21.846092224121094MB)
   24.668647503030712% used
From Space:
   capacity = 1572864 (1.5MB)
   used     = 753664 (0.71875MB)
   free     = 819200 (0.78125MB)
   47.916666666666664% used
To Space:
   capacity = 1572864 (1.5MB)
   used     = 0 (0.0MB)
   free     = 1572864 (1.5MB)
   0.0% used
PS Old Generation
   capacity = 100663296 (96.0MB)
   used     = 71246592 (67.946044921875MB)
   free     = 29416704 (28.053955078125MB)
   70.77713012695312% used

27238 interned Strings occupying 2666864 bytes.

至此：问题解决

3.总结：

问题排查步骤：

1. docker stats 命令查看docker容器内存和CPU使用情况
2. 高内存使用量应用，进入容器 jps查询java 进程id
3. jstat 命令查看年轻代和老年代内存使用量和分布。
4. jmap查看内存详情，分析问题原因为：青年代没有设置或者设置的过大了。青年代过大触发GC频率过低。造成大量垃圾对象堆积在青年代不回收。而Eden区多次扩容导致。
5. 设置JVM参数限制内存大小：-XX:MetaspaceSize=128m -XX:MaxMetaspaceSize=256m -Xms128m -Xmx128m -Xmn32m -Xss256k -XX:SurvivorRatio=6 -XX:+UseConcMarkSweepGC【数值可以自己设置不必照抄】
6. docker容器运行时设置参数-e JAVA_OPTS 没有生效。
7. 为设置JVM参数多次实验 Dockerfile中设置的参数可以生效：ENTRYPOINT "java", "-jar","-XX:MaxMetaspaceSize=256m","-Xms128m","-Xmx128m","-Xmn32m","-XX:SurvivorRatio=6","app.jar","--spring.config.location=/opt/application.yaml"
8. 重新构建应用镜像后部署问题解决。