jvm-接口调用排查

问题描述

线上碰到个问题,某个接口调用时间特别长,线上调用接口直接报gateway time out

分析处理

1、先关闭该功能 (该功能是非核心功能)

2、本地起服务连环境排查,发现本地正常。并且线上其他接口正常,只有这个接口异常。跳过gateway直接调用服务发现可以调通但是耗时140s。排除是gateway的问题,定位是服务接口问题。

3、排查代码

java 复制代码
    public List<OaClassScheduleChartVO> oaClassScheduleChart(OaClassScheduleQuery query, UserBaseDTO currentUser) {
        log.info("OaClassScheduleService->oaClassSchedulChart  query:{}", query);
        query.setOrderBy("schedule_date desc");
        List<OaClassScheduleChartVO> oaClassScheduleChartVOS = new ArrayList<>();
        SimpleDateFormat sdf = new SimpleDateFormat("yyyy-MM-dd");
        List<OaClassSchedule> oaClassSchedules = this.queryList(query);
        Map<String, OaClassScheduleChartVO> oaClassSchedulePictureVOMap = new HashMap<>();
        List<String> dates = com.wpg.waterwork.common.util.DateUtil.getMonthFullDay(Integer.parseInt(query.getQueryDate().substring(0, 4)), Integer.parseInt(query.getQueryDate().substring(5, 7)));
        //按名称分组存入map
        oaClassSchedules.forEach(oaClassSchedule -> {
            List<ClassGroupAndUserVO> classGroupAndUserVOS = JSON.parseArray(oaClassSchedule.getClassGroupUserVal(), ClassGroupAndUserVO.class);
                log.info("OaClassScheduleService->oaClassSchedulChart->oaClassSchedules.forEach  classGroupAndUserVOS:{}", JSON.toJSONString(classGroupAndUserVOS));
            }

        });

        //返回数据
        oaClassSchedulePictureVOMap.forEach((userId, pictureVO) -> {
            //根据用户id和班组名称查询
            boolean have = (StringUtils.isBlank(query.getUserId()) || userId.equals(query.getUserId())) &&
                    (StringUtils.isBlank(query.getClassGroupName()) || pictureVO.getClassGroupName().contains(query.getClassGroupName()));
            if (have) {
                //按班次顺序排序,便于前端处理
                pictureVO.getDateClassList().forEach(dateClass -> {
                    if (dateClass.getClassValVOList() != null) {
                        dateClass.setClassValVOList(dateClass.getClassValVOList().stream().sorted(Comparator.comparing(ClassValVO::getSort)).collect(Collectors.toList()));
                    }
                });
                oaClassScheduleChartVOS.add(pictureVO);
            }
        });
        log.info("OaClassScheduleService->oaClassSchedulChart->oaClassSchedulePictureVOMap.forEach  oaClassSchedulePictureVOMap:{}", oaClassSchedulePictureVOMap);
        return oaClassScheduleChartVOS;
    }

代码非常简单就没有想太多,也没有外部接口调用,只看到查询数据库和对数据操作。直接考虑是线上内存问题

4、查看服务器内存(共64G)占用60%左右不算多,进入docker容器排查内存

jps

jcmd 1 VM.flags

查到堆空间256M,年轻代78M,from和to区各3.5M,新生代71M

查看从数据查到的数据

从数据库查到的对象特别大有2.6M,考虑是大对象导致的问题

排查调用时的jvm情况

jps -l

jstat -gcutil 1 1000

S0 --- Heap上的 Survivor space 0 区已使用空间的百分比

S1 --- Heap上的 Survivor space 1 区已使用空间的百分比

E --- Heap上的 Eden space 区已使用空间的百分比

O --- Heap上的 Old space 区已使用空间的百分比

P --- Perm space 区已使用空间的百分比

YGC --- 从应用程序启动到采样时发生 Young GC 的次数

YGCT-- 从应用程序启动到采样时 Young GC 所用的时间(单位秒)

FGC --- 从应用程序启动到采样时发生 Full GC 的次数

FGCT-- 从应用程序启动到采样时 Full GC 所用的时间(单位秒)

GCT --- 从应用程序启动到采样时用于垃圾回收的总时间(单位秒)

bash 复制代码
 S0     S1      E      O      M    CCS    YGC     YGCT     FGC    FGCT     GCT
 53.75   0.00  66.96  79.73  90.31  86.66 153136 1508.284 30787 6199.465 7707.749
 53.75   0.00  67.02  79.73  90.31  86.66 153136 1508.284 30787 6199.465 7707.749
  0.00  98.39   2.19  87.75  90.31  86.66 153147 1508.391 30787 6199.465 7707.856
 28.41   0.00   0.00  90.10  90.31  86.66 153170 1508.587 30787 6199.465 7708.052
  0.00  98.94  81.74  85.13  90.31  86.65 153188 1508.740 30788 6199.661 7708.401
  0.00   6.99  72.82  93.67  90.31  86.65 153213 1508.999 30788 6199.661 7708.660
  0.00  17.23   0.00  97.21  90.31  86.65 153241 1509.245 30788 6199.661 7708.906
 70.84   0.00   0.00  97.08  90.31  86.65 153259 1509.406 30790 6199.826 7709.231
  5.57  45.17  97.56  90.30  90.31  86.65 153282 1509.640 30790 6200.007 7709.646
  0.00   0.00 100.00  98.95  90.31  86.65 153292 1509.725 30793 6200.359 7710.084
  0.00   0.00  98.44  99.58  90.31  86.65 153294 1509.759 30797 6201.100 7710.859
  0.00  97.51  99.48  98.28  90.31  86.65 153296 1509.776 30801 6201.884 7711.661
  0.00   0.00  98.71  99.58  90.31  86.65 153297 1509.784 30805 6202.638 7712.422
  0.00   0.00  98.15  99.58  90.31  86.65 153297 1509.784 30809 6203.393 7713.177
  0.00   0.00  13.59  83.53  90.31  86.65 153300 1509.811 30812 6204.198 7714.009
  0.00   0.00  99.49  98.81  90.31  86.65 153301 1509.825 30816 6204.835 7714.659
  0.00  40.83  98.37  99.59  90.31  86.65 153304 1509.866 30820 6205.574 7715.440
  0.00   0.92   0.00  94.98  90.31  86.65 153310 1509.941 30823 6206.304 7716.245
  0.00   0.00   0.00  85.23  90.31  86.65 153330 1510.130 30824 6206.523 7716.653
  0.00  36.15   0.00  93.73  90.31  86.65 153355 1510.363 30824 6206.523 7716.886
  0.00  40.50   0.00  88.64  90.31  86.65 153372 1510.548 30825 6206.733 7717.281
 70.34   0.00   0.00  88.65  90.31  86.65 153391 1510.728 30826 6206.911 7717.639
  0.00   8.70   0.00  90.35  90.31  86.65 153416 1510.964 30826 6206.911 7717.875
  1.00   0.00 100.00  98.82  90.31  86.65 153437 1511.200 30827 6206.911 7718.110
  0.00  52.46  50.21  85.29  90.31  86.65 153442 1511.242 30830 6207.645 7718.887
 21.53   0.00  97.62  98.19  90.31  86.65 153447 1511.292 30834 6208.192 7719.485
  0.00   0.00  32.20  85.29  90.31  86.65 153448 1511.317 30838 6209.003 7720.320
  0.00   0.00  68.29  85.29  90.31  86.65 153449 1511.327 30842 6209.769 7721.096
 89.31   0.00  94.45  98.44  90.31  86.65 153451 1511.362 30845 6210.355 7721.717
  0.00  20.80   0.00  97.20  90.31  86.65 153456 1511.412 30848 6211.200 7722.612
  0.00   0.00   2.67  83.60  90.31  86.65 153458 1511.430 30852 6211.941 7723.371
  0.00  45.68   0.00  97.99  90.31  86.65 153466 1511.528 30855 6212.469 7723.997
 34.73   0.00   0.00  88.32  90.31  86.65 153482 1511.683 30857 6212.874 7724.557
 62.36   5.16  96.71  96.81  90.31  86.65 153504 1511.885 30858 6212.874 7724.759
  0.00  21.65  98.46  90.36  90.31  86.65 153529 1512.106 30858 6213.052 7725.158
 73.27   0.00   0.00  97.28  90.31  86.65 153553 1512.350 30858 6213.052 7725.402
  0.00  37.09  53.81  90.35  90.31  86.65 153573 1512.522 30859 6213.242 7725.763
  0.00   0.75 100.00  94.95  90.31  86.65 153593 1512.702 30860 6213.413 7726.116
 76.46  93.71  98.18  98.84  90.31  86.65 153599 1512.771 30864 6213.962 7726.733
  0.00  97.69  97.99  98.49  90.31  86.65 153600 1512.779 30868 6214.705 7727.485
  0.00  98.78  90.35  90.72  90.31  86.65 153602 1512.799 30872 6215.449 7728.248
  0.00   0.00   0.00  83.62  90.31  86.65 153605 1512.833 30875 6216.210 7729.043
  0.00   0.00  97.90  98.79  90.31  86.65 153613 1512.922 30878 6216.610 7729.532
  0.00   0.00  97.92  98.80  90.31  86.65 153613 1512.922 30882 6217.441 7730.363
  0.00   0.00  97.25  99.64  90.31  86.65 153613 1512.922 30886 6218.287 7731.208
 50.59   0.00  22.11  98.05  90.31  86.65 153621 1513.018 30889 6218.910 7731.928
  0.00  38.80   0.00  81.64  90.31  86.65 153633 1513.151 30891 6219.465 7732.617
 17.89   0.00   0.00  88.44  90.31  86.65 153662 1513.424 30891 6219.465 7732.890
  0.00   5.57   0.00  91.86  90.31  86.65 153689 1513.650 30891 6219.465 7733.116
  0.00  65.59   0.00  85.01  90.31  86.65 153709 1513.824 30892 6219.638 7733.462
  0.00  98.73   0.00  96.85  90.31  86.65 153735 1514.052 30892 6219.638 7733.690
  0.00  24.77   0.00  93.99  90.31  86.65 153754 1514.228 30893 6219.823 7734.051
  0.00  38.29  67.46  86.72  90.31  86.65 153765 1514.316 30896 6220.376 7734.692
  0.00  98.14 100.00  99.24  90.31  86.65 153774 1514.408 30899 6220.763 7735.170
 22.51  68.29 100.00  98.95  90.31  86.65 153793 1514.602 30901 6221.131 7735.732
 14.89  36.61  97.16  99.34  90.31  86.65 153812 1514.806 30903 6221.507 7736.313
  0.00  13.84  98.90  99.47  90.31  86.65 153816 1514.877 30905 6222.251 7737.127
  0.00  47.63  98.38  99.47  90.31  86.65 153822 1514.945 30909 6222.821 7737.766
  0.00  39.96   0.00  98.62  90.31  86.65 153837 1515.088 30911 6223.401 7738.489
  0.00   0.00   0.00  81.36  90.31  86.65 153855 1515.278 30913 6223.813 7739.091
  0.00  16.92   0.00  92.57  90.31  86.65 153890 1515.626 30913 6223.813 7739.439
 41.41   0.00   6.65  96.00  90.31  86.65 153915 1515.856 30913 6223.813 7739.669
 71.23   0.00   0.00  85.09  90.31  86.65 153933 1516.024 30914 6224.028 7740.051
  0.00  64.14  19.92  93.57  90.31  86.65 153958 1516.298 30914 6224.028 7740.325
  0.00  95.82  51.30  83.42  90.31  86.65 153972 1516.429 30916 6224.395 7740.825
 57.41   0.00  84.44  97.00  90.31  86.65 153980 1516.498 30919 6224.993 7741.491
  0.00  56.23  75.79  97.20  90.31  86.65 153982 1516.523 30923 6225.714 7742.237
 63.98   0.00  98.68  99.57  90.31  86.65 153986 1516.564 30927 6226.267 7742.830
  0.00   0.00  18.69  83.54  90.31  86.65 153990 1516.600 30930 6227.009 7743.608
  0.00   0.00  99.40  98.72  90.31  86.65 153995 1516.672 30934 6227.551 7744.222
  0.00   0.00   0.00  85.20  90.31  86.65 154000 1516.717 30937 6228.296 7745.012
  0.00  97.00  96.32  98.76  90.31  86.65 154013 1516.847 30939 6228.659 7745.506
  0.00  26.48  48.61  91.58  90.31  86.65 154027 1517.001 30941 6229.022 7746.023
  0.00  98.04   0.00  98.40  90.31  86.65 154049 1517.250 30942 6229.022 7746.272
  3.86   0.00  82.27  86.85  90.31  86.65 154068 1517.440 30942 6229.235 7746.675
  0.00  26.34   0.00  90.40  90.31  86.65 154093 1517.678 30942 6229.235 7746.913
 65.59   0.00   0.00  94.12  90.31  86.65 154118 1517.910 30942 6229.235 7747.145
  0.00  98.39   0.00  98.78  90.31  86.65 154138 1518.081 30943 6229.415 7747.495
  0.00  98.93  97.82  98.80  90.31  86.65 154151 1518.211 30945 6229.599 7747.810
 90.29   0.00  82.16  90.58  90.31  86.65 154154 1518.258 30949 6230.395 7748.653
 95.25  72.81  97.51  97.46  90.31  86.65 154157 1518.306 30952 6230.977 7749.283
  0.00   0.00  97.33  97.96  90.31  86.65 154157 1518.306 30957 6231.895 7750.201
  0.00  97.34  99.56  99.01  90.31  86.65 154162 1518.347 30960 6232.451 7750.798
 37.88   0.00  98.24  99.77  90.31  86.65 154169 1518.425 30963 6233.086 7751.511
 85.23  66.06  93.56  98.79  90.31  86.65 154179 1518.528 30966 6233.626 7752.154
  0.00  61.12   0.00  92.17  90.31  86.65 154200 1518.719 30966 6233.804 7752.523
 19.80   0.00   0.00  95.76  90.31  86.65 154222 1518.949 30966 6233.804 7752.753
  0.00  56.52  20.74  88.37  90.30  86.65 154240 1519.113 30967 6234.095 7753.208
 41.50   0.00   0.00  84.99  90.30  86.65 154258 1519.322 30968 6234.296 7753.619
  0.00  23.16   0.00  90.19  90.30  86.65 154279 1519.586 30968 6234.296 7753.882
 76.18   0.00  62.22  90.43  90.30  86.65 154300 1519.867 30968 6234.296 7754.163
  0.00  98.03  98.01  99.91  90.30  86.65 154305 1519.912 30972 6234.879 7754.792
  0.00   0.00   0.00  80.13  90.30  86.65 154307 1519.936 30975 6235.647 7755.584
  0.00  97.13  98.82  99.87  90.30  86.65 154315 1520.011 30978 6236.030 7756.041
  0.00   0.00  98.47  99.46  90.30  86.65 154319 1520.053 30982 6236.793 7756.845
  0.00   0.00   0.00  83.40  90.30  86.65 154321 1520.086 30985 6237.620 7757.707
  0.00   0.00  99.77  98.62  90.30  86.65 154324 1520.122 30989 6238.181 7758.303
  0.00   0.00  99.46  98.64  90.30  86.65 154324 1520.122 30994 6239.090 7759.212
  0.00   0.00  97.99  99.47  90.30  86.65 154327 1520.159 30997 6239.679 7759.839
  0.00   0.00  98.00  99.47  90.30  86.65 154327 1520.159 31001 6240.503 7760.663
 13.17   0.00   0.00  80.06  90.30  86.65 154332 1520.216 31004 6241.282 7761.498
  0.00  99.95   0.00  85.10  90.30  86.65 154349 1520.372 31006 6241.696 7762.067
  0.00  17.89   0.00  86.79  90.30  86.65 154376 1520.743 31006 6241.696 7762.438
  0.00  27.40   0.00  86.85  90.30  86.65 154400 1520.963 31006 6241.696 7762.658
  2.08  74.58  97.07  97.01  90.30  86.65 154422 1521.188 31007 6241.696 7762.884
 54.99   0.00   0.00  93.66  90.30  86.65 154444 1521.398 31007 6241.912 7763.310
  0.00   2.53   0.00  98.27  90.30  86.65 154471 1521.634 31007 6241.912 7763.546
  0.00   0.00  36.10  85.11  90.30  86.65 154481 1521.720 31010 6242.429 7764.149
 71.37  18.08  98.30  99.40  90.30  86.65 154498 1521.886 31012 6242.618 7764.504
  0.00   0.00  24.57  83.42  90.30  86.65 154510 1522.013 31014 6243.182 7765.195
 81.06  98.70  93.97  99.14  90.30  86.65 154519 1522.109 31017 6243.597 7765.707
 25.23  16.03 100.00  99.56  90.30  86.65 154542 1522.318 31018 6243.940 7766.258
  0.00  97.53   0.00  90.19  90.30  86.65 154560 1522.492 31020 6244.296 7766.788
 15.87   0.00   0.00  91.93  90.30  86.65 154589 1522.757 31020 6244.296 7767.053
  0.00  37.35   0.00  95.32  90.30  86.65 154614 1522.996 31020 6244.296 7767.292
 14.56   0.00   0.00  90.41  90.30  86.65 154635 1523.189 31021 6244.484 7767.672
 46.73   0.00  18.87  93.84  90.30  86.65 154658 1523.475 31021 6244.484 7767.958
 73.01   0.00  96.70  98.93  90.30  86.65 154671 1523.750 31022 6244.484 7768.233
  0.00   0.00   0.00  80.12  90.30  86.65 154679 1523.826 31024 6245.109 7768.935
 65.13  63.02  97.23  98.62  90.30  86.65 154688 1523.928 31027 6245.478 7769.405
  0.00   0.00  98.87  99.46  90.30  86.65 154696 1524.033 31030 6246.069 7770.102
  0.00  56.27  97.59  99.54  90.30  86.65 154698 1524.091 31033 6246.681 7770.772
 36.43  77.00  94.89  98.63  90.30  86.65 154710 1524.203 31036 6247.253 7771.456
 93.07  98.39   8.80  98.62  90.30  86.65 154716 1524.265 31039 6247.888 7772.153
 41.68   0.00  47.40  98.65  90.30  86.65 154723 1524.330 31041 6248.468 7772.798
 37.09   0.00  66.82  85.09  90.30  86.65 154738 1524.466 31043 6248.849 7773.315
  0.00  60.86   0.00  93.59  90.30  86.65 154764 1524.707 31043 6248.849 7773.555
 22.55   0.00  45.11  93.55  90.30  86.65 154783 1524.901 31044 6249.059 7773.960
  1.30  73.53  96.24  96.97  90.30  86.65 154802 1525.099 31045 6249.059 7774.158
 97.55   0.00  76.20  88.44  90.30  86.65 154810 1525.176 31045 6249.274 7774.449
 97.55   0.00  76.32  88.44  90.30  86.65 154810 1525.176 31045 6249.274 7774.449
 97.55   0.00  76.90  88.44  90.30  86.65 154810 1525.176 31045 6249.274 7774.449
 97.55   0.00  76.96  88.44  90.30  86.65 154810 1525.176 31045 6249.274 7774.449
 97.55   0.00  77.22  88.44  90.30  86.65 154810 1525.176 31045 6249.274 7774.449

可以看到老年代在疯狂的gc 用7774-7707,gc时间70s

这是不正常的,并且老年代长期占比90 即使gc之后也占比90左右,没有清理太多空间,调用完成之后开始下降。

5、在本地设置jvm参数和线上一致

bash 复制代码
-Xmx256m -Xms256m -Xmn78m -XX:SurvivorRatio=20

本地调用的jvm情况

bash 复制代码
   S0     S1      E      O      M    CCS    YGC     YGCT     FGC    FGCT     GCT
 58.41   0.00  84.56  26.21  93.96  91.76   1280   15.835    25    5.309   21.144
 58.41   0.00  84.57  26.21  93.96  91.76   1280   15.835    25    5.309   21.144
 58.41   0.00  84.58  26.21  93.96  91.76   1280   15.835    25    5.309   21.144
 58.41   0.00  86.61  26.21  93.96  91.76   1280   15.835    25    5.309   21.144
 58.41   0.00  86.72  26.21  93.96  91.76   1280   15.835    25    5.309   21.144
 58.41   0.00  86.77  26.21  93.96  91.76   1280   15.835    25    5.309   21.144
 58.41   0.00  86.80  26.21  93.96  91.76   1280   15.835    25    5.309   21.144
 58.41   0.00  86.80  26.21  93.96  91.76   1280   15.835    25    5.309   21.144
 58.41   0.00  86.81  26.21  93.96  91.76   1280   15.835    25    5.309   21.144
 58.41   0.00  86.82  26.21  93.96  91.76   1280   15.835    25    5.309   21.144
 58.41   0.00  86.83  26.21  93.96  91.76   1280   15.835    25    5.309   21.144
  0.00  40.06  66.26  26.21  94.05  91.81   1281   15.844    25    5.309   21.153
  1.49   0.00  27.64  26.84  94.05  91.81   1284   15.877    25    5.309   21.187
  2.76   0.00  40.50  29.33  94.05  91.81   1286   15.908    25    5.309   21.217
  4.65   0.00  77.50  31.92  94.05  91.81   1288   15.960    25    5.309   21.269
  0.00   4.57   0.00  35.82  94.05  91.82   1291   15.988    25    5.309   21.297
  0.00  98.38  80.18  35.82  94.05  91.82   1293   16.011    25    5.309   21.321
 71.34   0.00  45.67  35.83  94.05  91.82   1296   16.046    25    5.309   21.355
  0.00  69.84  22.24  35.83  94.05  91.82   1299   16.087    25    5.309   21.396
  0.00  37.59  96.33  36.17  94.05  91.82   1301   16.104    25    5.309   21.414
  1.09   0.00  43.51  39.61  94.05  91.82   1304   16.138    25    5.309   21.447
  3.91   0.00  81.61  42.33  94.05  91.82   1306   16.165    25    5.309   21.474
  0.00   3.67   7.38  45.88  94.05  91.82   1309   16.226    25    5.309   21.535
  0.00  43.29  72.63  45.88  94.06  91.82   1311   16.255    25    5.309   21.565
 87.92   0.00  38.35  45.89  94.06  91.82   1314   16.282    25    5.309   21.591
  0.00  97.00   2.64  46.32  94.06  91.82   1317   16.337    25    5.309   21.646
  0.00  93.98  64.06  46.54  94.06  91.82   1319   16.362    25    5.309   21.672
  0.62   0.00  16.52  49.46  94.06  91.82   1322   16.408    25    5.309   21.718
  0.99   0.00  34.07  51.86  94.06  91.82   1324   16.430    25    5.309   21.739
 92.23   0.00  75.27  53.04  94.06  91.82   1326   16.471    25    5.309   21.780
  0.00  97.35  35.03  53.47  94.06  91.82   1329   16.502    25    5.309   21.811
  9.79   0.00   6.42  53.47  94.06  91.82   1332   16.540    25    5.309   21.849
 26.21   0.00  76.94  53.69  94.06  91.82   1334   16.561    25    5.309   21.870
  0.00  25.94  34.39  56.24  94.07  91.82   1337   16.594    25    5.309   21.904
  0.00   0.78  94.30  57.01  94.07  91.82   1339   16.618    25    5.309   21.927
  0.00   1.50  58.88  61.69  94.08  91.82   1343   16.650    25    5.309   21.959
 10.65   0.00   7.98  61.70  94.08  91.82   1346   16.688    25    5.309   21.997
 42.16   0.00  79.46  62.13  94.08  91.82   1348   16.714    25    5.309   22.023
  0.00  24.11  37.41  62.35  94.08  91.82   1351   16.773    25    5.309   22.083
  0.00   2.60  92.89  68.13  94.08  91.82   1353   16.800    25    5.309   22.109
  0.00   1.20  32.21  73.67  94.08  91.82   1357   16.853    25    5.309   22.163
  0.00   3.16  58.36  76.66  94.08  91.82   1359   16.876    25    5.309   22.186
 70.59   0.00  13.87  79.99  94.08  91.82   1362   16.906    25    5.309   22.215
 20.30   0.00  80.71  80.00  94.08  91.82   1364   16.935    25    5.309   22.244
  0.00  23.86  44.47  80.00  94.08  91.82   1367   16.974    25    5.309   22.283
  0.89   0.00   5.56  80.61  94.08  91.82   1370   16.999    25    5.309   22.308
  1.74   0.00  58.97  83.29  94.08  91.82   1372   17.020    25    5.309   22.330
  0.00   4.33  26.48  86.73  94.08  91.82   1375   17.075    25    5.309   22.384
  0.00   1.84  88.47  88.72  94.08  91.82   1377   17.112    25    5.309   22.421
 37.28   0.00  51.16  88.72  94.08  91.82   1380   17.148    25    5.309   22.457
  0.00   9.51   3.29  88.72  94.08  91.82   1383   17.187    25    5.309   22.496
  0.00  98.92  45.94  89.58  94.08  91.82   1385   17.242    25    5.309   22.551
  0.00   4.83  90.43  92.60  94.08  91.82   1387   17.264    25    5.309   22.574
  6.03   0.00  18.62  96.38  94.08  91.82   1390   17.294    25    5.309   22.603
  0.00  84.79  78.89  98.79  94.08  91.82   1393   17.342    25    5.309   22.651
 74.18   0.00  78.54  11.12  94.08  91.82   1394   17.350    26    5.523   22.873
  0.00  58.74  51.49  11.34  94.08  91.82   1397   17.376    26    5.523   22.899
 65.59   0.00  14.89  11.56  94.08  91.82   1400   17.410    26    5.523   22.933
  0.74   0.00  58.64  15.73  94.08  91.82   1402   17.441    26    5.523   22.964
  0.00   0.74  16.73  18.86  94.08  91.82   1405   17.462    26    5.523   22.985
  0.00   5.53  96.17  20.50  94.08  91.82   1407   17.475    26    5.523   22.998
  2.60   0.00  69.61  22.88  94.08  91.82   1410   17.495    26    5.523   23.018
  0.00  34.18  19.42  23.10  94.08  91.82   1413   17.532    26    5.523   23.055
  0.00  77.70  99.52  23.11  94.08  91.82   1415   17.553    26    5.523   23.076
 27.64   0.00  55.97  23.13  94.08  91.82   1418   17.623    26    5.523   23.146
  0.00   0.69   0.00  25.52  94.08  91.82   1421   17.677    26    5.523   23.200
  0.00   1.84  30.09  27.28  94.08  91.82   1423   17.696    26    5.523   23.219
  0.00   2.46  96.91  29.00  94.08  91.82   1426   17.715    26    5.523   23.238
 39.77   0.00  28.65  29.87  94.08  91.82   1428   17.772    26    5.523   23.295
 45.40   0.00  93.87  29.87  94.08  91.82   1430   17.791    26    5.523   23.314
  0.00  90.83  43.78  30.51  94.08  91.82   1433   17.835    26    5.523   23.358
  0.00   4.47  77.05  32.54  94.08  91.82   1435   17.856    26    5.523   23.379
 98.80   0.00  25.08  34.37  94.08  91.82   1438   17.882    26    5.523   23.406
 98.80   0.00  28.14  34.37  94.08  91.82   1438   17.882    26    5.523   23.406
 98.80   0.00  28.57  34.37  94.08  91.82   1438   17.882    26    5.523   23.406
 98.80   0.00  29.64  34.37  94.08  91.82   1438   17.882    26    5.523   23.406
 98.80   0.00  29.66  34.37  94.08  91.82   1438   17.882    26    5.523   23.406
 98.80   0.00  29.66  34.37  94.08  91.82   1438   17.882    26    5.523   23.406
 98.80   0.00  29.67  34.37  94.08  91.82   1438   17.882    26    5.523   23.406
 98.80   0.00  29.68  34.37  94.08  91.82   1438   17.882    26    5.523   23.406
 98.80   0.00  29.69  34.37  94.08  91.82   1438   17.882    26    5.523   23.406
 98.80   0.00  29.70  34.37  94.08  91.82   1438   17.882    26    5.523   23.406
 98.80   0.00  29.89  34.37  94.08  91.82   1438   17.882    26    5.523   23.406
 98.80   0.00  32.10  34.37  94.08  91.82   1438   17.882    26    5.523   23.406
 98.80   0.00  32.14  34.37  94.08  91.82   1438   17.882    26    5.523   23.406
 98.80   0.00  32.16  34.37  94.08  91.82   1438   17.882    26    5.523   23.406
 98.80   0.00  32.22  34.37  94.08  91.82   1438   17.882    26    5.523   23.406
 98.80   0.00  32.38  34.37  94.08  91.82   1438   17.882    26    5.523   23.406
 98.80   0.00  32.38  34.37  94.08  91.82   1438   17.882    26    5.523   23.406
 98.80   0.00  32.57  34.37  94.08  91.82   1438   17.882    26    5.523   23.406
 98.80   0.00  32.58  34.37  94.08  91.82   1438   17.882    26    5.523   23.406
 98.80   0.00  32.59  34.37  94.08  91.82   1438   17.882    26    5.523   23.406
 98.80   0.00  32.88  34.37  94.08  91.82   1438   17.882    26    5.523   23.406
 98.80   0.00  34.99  34.37  94.08  91.82   1438   17.882    26    5.523   23.406
 98.80   0.00  35.03  34.37  94.08  91.82   1438   17.882    26    5.523   23.406
 98.80   0.00  35.07  34.37  94.08  91.82   1438   17.882    26    5.523   23.406
 98.80   0.00  35.10  34.37  94.08  91.82   1438   17.882    26    5.523   23.406
 98.80   0.00  35.11  34.37  94.08  91.82   1438   17.882    26    5.523   23.406
 98.80   0.00  35.11  34.37  94.08  91.82   1438   17.882    26    5.523   23.406
 98.80   0.00  35.13  34.37  94.08  91.82   1438   17.882    26    5.523   23.406
 98.80   0.00  35.13  34.37  94.08  91.82   1438   17.882    26    5.523   23.406
 98.80   0.00  35.14  34.37  94.08  91.82   1438   17.882    26    5.523   23.406
 98.80   0.00  35.15  34.37  94.08  91.82   1438   17.882    26    5.523   23.406
 98.80   0.00  37.23  34.37  94.08  91.82   1438   17.882    26    5.523   23.406
 98.80   0.00  37.43  34.37  94.08  91.82   1438   17.882    26    5.523   23.406
 98.80   0.00  37.45  34.37  94.08  91.82   1438   17.882    26    5.523   23.406
 98.80   0.00  37.47  34.37  94.08  91.82   1438   17.882    26    5.523   23.406
 98.80   0.00  37.47  34.37  94.08  91.82   1438   17.882    26    5.523   23.406
 98.80   0.00  83.89  34.37  94.08  91.82   1438   17.882    26    5.523   23.406

可以看到本地fullgc次数只有一次,调用时gc花费1s,最大的区别是

在本地初始老年代占比只有20%且gc一次就几乎已经全部清理干净,清理一次占比从98.79->11.12

bash 复制代码
0.00  84.79  78.89  98.79  94.08  91.82   1393   17.342    25    5.309   22.651
74.18   0.00  78.54  11.12  94.08  91.82   1394   17.350    26    5.523   22.873

在线上老年代频繁清理且清理出来的空间并不多。

6、考虑是线上环境其他大对象占用了老年代,出现了内存泄漏。但是本地jvm参数和线上一致却没有出现这个问题。

然后排查是否有定时任务之类的其他任务占用内存,发现也没有。

然后比较本地和线上的环境和配置的差异,发现线上和本地日志级别不一样,并且线上不断打日志

7、修改日志级别后正常,原来代码有一段在双重循环里面打大对象的info日志。

问题总结

1、打日志代码不应该在循环里面打印大对象,非常占用内存,导致老年代的对象无法及时回收,出现内存泄露。

2、线上配置的日志级别错误,线上应该配置日志较少的级别。

相关推荐
东阳马生架构16 小时前
JVM简介—3.JVM的执行子系统
jvm
程序员志哥1 天前
JVM系列(十三) -常用调优工具介绍
jvm
后台技术汇1 天前
JavaAgent技术应用和原理:JVM持久化监控
jvm
程序员志哥1 天前
JVM系列(十二) -常用调优命令汇总
jvm
黄油饼卷咖喱鸡就味增汤拌孜然羊肉炒饭1 天前
聊聊volatile的实现原理?
java·jvm·redis
_LiuYan_1 天前
JVM执行引擎JIT深度剖析
java·jvm
王佑辉1 天前
【jvm】内存泄漏的8种情况
jvm
工业甲酰苯胺1 天前
JVM简介—1.Java内存区域
java·jvm·python
yuanbenshidiaos2 天前
c++---------数据类型
java·jvm·c++