后端返回前端的数据量过大解决方案

后端返回前端的数据量过大解决方案

性能面板(Performance) · chrome调试指南

原因

遇到一个页面有好几个表格,部分表格采用虚拟滚动条

数据量有点大

接近快60s了,看一下是哪里导致的慢

后台请求方法执行并不慢

复制代码
2024-12-04 15:21:52.889  INFO 69948 --- [nio-8004-exec-7] s.d.e.i.SecurityEventMonitorDaoHibernate : build sql time: 4ms
2024-12-04 15:21:52.889  INFO 69948 --- [nio-8004-exec-5] s.d.e.i.SecurityEventMonitorDaoHibernate : build sql time: 4ms
2024-12-04 15:21:52.889  INFO 69948 --- [nio-8004-exec-8] s.d.e.i.SecurityEventMonitorDaoHibernate : build sql time: 4ms
2024-12-04 15:21:52.889  INFO 69948 --- [io-8004-exec-10] s.d.e.i.SecurityEventMonitorDaoHibernate : build sql time: 4ms
2024-12-04 15:21:58.276  INFO 69948 --- [nio-8004-exec-8] s.d.e.i.SecurityEventMonitorDaoHibernate : exec sql [select * from  ( select * from SECURITY_EVENT_MONITOR_202411 union all  select * from SECURITY_EVENT_MONITOR_202412)  where 1=1  and alarmTime >= to_date('2024-11-30','yyyy-MM-dd') and alarmTime <= to_date('2024-12-03','yyyy-MM-dd')] time: 5392ms
2024-12-04 15:21:58.276  INFO 69948 --- [nio-8004-exec-5] s.d.e.i.SecurityEventMonitorDaoHibernate : exec sql [select * from  ( select * from SECURITY_EVENT_MONITOR_202411 union all  select * from SECURITY_EVENT_MONITOR_202412)  where 1=1  and alarmTime >= to_date('2024-11-30','yyyy-MM-dd') and alarmTime <= to_date('2024-12-03','yyyy-MM-dd')] time: 5392ms
2024-12-04 15:21:58.283  INFO 69948 --- [nio-8004-exec-7] s.d.e.i.SecurityEventMonitorDaoHibernate : exec sql [select * from  ( select * from SECURITY_EVENT_MONITOR_202411 union all  select * from SECURITY_EVENT_MONITOR_202412)  where 1=1  and alarmTime >= to_date('2024-11-30','yyyy-MM-dd') and alarmTime <= to_date('2024-12-03','yyyy-MM-dd')] time: 5398ms
2024-12-04 15:21:58.297  INFO 69948 --- [io-8004-exec-10] s.d.e.i.SecurityEventMonitorDaoHibernate : exec sql [select * from  ( select * from SECURITY_EVENT_MONITOR_202411 union all  select * from SECURITY_EVENT_MONITOR_202412)  where 1=1  and alarmTime >= to_date('2024-11-30','yyyy-MM-dd') and alarmTime <= to_date('2024-12-03','yyyy-MM-dd')] time: 5412ms
2024-12-04 15:21:58.412  INFO 69948 --- [io-8004-exec-10] s.d.e.i.SecurityEventMonitorDaoHibernate : transform data time: 5527ms
2024-12-04 15:21:58.414  INFO 69948 --- [nio-8004-exec-5] s.d.e.i.SecurityEventMonitorDaoHibernate : transform data time: 5527ms
2024-12-04 15:21:58.481  INFO 69948 --- [nio-8004-exec-8] s.d.e.i.SecurityEventMonitorDaoHibernate : transform data time: 5595ms
2024-12-04 15:21:58.482  INFO 69948 --- [nio-8004-exec-7] s.d.e.i.SecurityEventMonitorDaoHibernate : transform data time: 5597ms
2024-12-04 15:21:58.485  INFO 69948 --- [nio-8004-exec-8] .s.s.e.i.SecurityEventMonitorServiceImpl : build queryGrid4 data cost time:2ms
2024-12-04 15:21:58.553  INFO 69948 --- [nio-8004-exec-7] .s.s.e.i.SecurityEventMonitorServiceImpl : build queryGrid1 data cost time:70ms
2024-12-04 15:21:58.562  INFO 69948 --- [nio-8004-exec-7] com.wisdom.facade.BizWebServiceSkeleton  : Biz web service skeleton process request done
2024-12-04 15:21:58.562  WARN 69948 --- [nio-8004-exec-7] com.wisdom.facade.BizWebServiceSkeleton  : request =  com.wisdom.imd.wypap.security.dto.event.SecurityEventMonitorDto, method = queryGrid1, cost = 5690 ms
2024-12-04 15:21:58.562  INFO 69948 --- [io-8004-exec-10] .s.s.e.i.SecurityEventMonitorServiceImpl : build queryGrid3 data cost time:149ms
2024-12-04 15:21:58.564  INFO 69948 --- [io-8004-exec-10] com.wisdom.facade.BizWebServiceSkeleton  : Biz web service skeleton process request done
2024-12-04 15:21:58.564  WARN 69948 --- [io-8004-exec-10] com.wisdom.facade.BizWebServiceSkeleton  : request =  com.wisdom.imd.wypap.security.dto.event.SecurityEventMonitorDto, method = queryGrid3, cost = 5692 ms
exec method : queryGrid3 time: 5709ms
exec method : queryGrid1 time: 5709ms
2024-12-04 15:21:58.571  INFO 69948 --- [nio-8004-exec-8] com.wisdom.facade.BizWebServiceSkeleton  : Biz web service skeleton process request done
2024-12-04 15:21:58.571  WARN 69948 --- [nio-8004-exec-8] com.wisdom.facade.BizWebServiceSkeleton  : request =  com.wisdom.imd.wypap.security.dto.event.SecurityEventMonitorDto, method = queryGrid4, cost = 5699 ms
exec method : queryGrid4 time: 5712ms
2024-12-04 15:21:59.711  INFO 69948 --- [nio-8004-exec-5] .s.s.e.i.SecurityEventMonitorServiceImpl : build queryGrid2 data cost time:1297ms
2024-12-04 15:21:59.713  INFO 69948 --- [nio-8004-exec-5] com.wisdom.facade.BizWebServiceSkeleton  : Biz web service skeleton process request done
2024-12-04 15:21:59.713  WARN 69948 --- [nio-8004-exec-5] com.wisdom.facade.BizWebServiceSkeleton  : request =  com.wisdom.imd.wypap.security.dto.event.SecurityEventMonitorDto, method = queryGrid2, cost = 6841 ms
exec method : queryGrid2 time: 6854ms

后台并行请求了4分方法,这个最慢接近60s,直接看一下这个为什么,找到占比最多的时间在执行什么方法

我靠,返回这么大,主要是解析后台加密的json数据过大导致

json会存储额外的结构,易读,但是数据量过大时性能不好

序列化使用json肯定说改不了了,只能想办法去节省传输的内容大小

后端使用GZIPOutputStream

java 复制代码
package com.wisdom.framework.web.utils;

import java.io.BufferedReader;
import java.io.ByteArrayInputStream;
import java.io.ByteArrayOutputStream;
import java.io.InputStreamReader;
import java.util.Base64;
import java.util.zip.GZIPInputStream;
import java.util.zip.GZIPOutputStream;

public class GzipUtils {

    /****
     * 字符串压缩
     * @param str
     * @return
     */
    public static String compress(String str){
        try {
            if (str == null || str.length() == 0) {
                return str;
            }
            ByteArrayOutputStream obj=new ByteArrayOutputStream();
            GZIPOutputStream gzip = new GZIPOutputStream(obj);
            gzip.write(str.getBytes("UTF-8"));
            gzip.close();
            return Base64.getEncoder().encodeToString(obj.toByteArray());
        }catch (Exception ex){
            ex.printStackTrace();
        }
        return str;
    }

    /***
     * 字符串解压
     * @param str
     * @return
     */
    public static String decompress(String str) {
        if (str == null || str.length() == 0) {
            return str;
        }
        try {
            byte[] compressed = Base64.getDecoder().decode(str);
            GZIPInputStream gis = new GZIPInputStream(new ByteArrayInputStream(compressed));
            StringBuilder outStr = new StringBuilder();
            BufferedReader br = new BufferedReader(new InputStreamReader(gis, "UTF-8"));

            String line;
            while ((line = br.readLine()) != null) {
                outStr.append(line);
            }
            gis.close();
            return outStr.toString();
        }catch (Exception ex){
            ex.printStackTrace();
        }
        return str;
    }

}

使用

判断json字符串长度大于200,000,就把json压缩一下

java 复制代码
String json = JsonUtils.serialize(wri, jsonConfig);
if(json.length() > 200000){
    json = GzipUtils.compress(json);
}
return json;

前端使用pako工具解压

https://github.com/nodeca/pako

放在lib目录下

引入js

html 复制代码
<script src="../../../scripts/lib/zip/pako.es5.min.js" type="text/javascript"></script>
js 复制代码
//压缩字符
                if(typeof data == "string" && data.indexOf("H4sIAAAAA") == 0){
                    let gzipArr = atob(data).split('');
                    let gzipArr_add = [];
                    for(let i = 0;i<gzipArr.length;i++){
                        let ch_ = gzipArr[i];
                        gzipArr_add.push(ch_.charCodeAt(0));
                    };
                    let gzipData = new Uint8Array(gzipArr_add);
                    data = pako.inflate(gzipData, { to: 'string' });
                }

测试

传输内容大小缩小了很多,时间也将近减少了一半

再看看后台方法的执行时间

复制代码
2024-12-04 15:51:16.434  INFO 69948 --- [nio-8004-exec-2] s.d.e.i.SecurityEventMonitorDaoHibernate : build sql time: 0ms
2024-12-04 15:51:16.436  INFO 69948 --- [nio-8004-exec-1] s.d.e.i.SecurityEventMonitorDaoHibernate : build sql time: 0ms
2024-12-04 15:51:22.100  INFO 69948 --- [nio-8004-exec-3] s.d.e.i.SecurityEventMonitorDaoHibernate : exec sql [select * from  SECURITY_EVENT_MONITOR_202412  where 1=1  and alarmTime >= to_date('2024-12-01','yyyy-MM-dd') and alarmTime <= to_date('2024-12-04','yyyy-MM-dd')] time: 5681ms
2024-12-04 15:51:22.110  INFO 69948 --- [nio-8004-exec-3] s.d.e.i.SecurityEventMonitorDaoHibernate : transform data time: 5690ms
2024-12-04 15:51:22.117  INFO 69948 --- [nio-8004-exec-3] .s.s.e.i.SecurityEventMonitorServiceImpl : build queryGrid1 data cost time:6ms
2024-12-04 15:51:22.119  INFO 69948 --- [nio-8004-exec-3] com.wisdom.facade.BizWebServiceSkeleton  : Biz web service skeleton process request done
2024-12-04 15:51:22.119  WARN 69948 --- [nio-8004-exec-3] com.wisdom.facade.BizWebServiceSkeleton  : request =  com.wisdom.imd.wypap.security.dto.event.SecurityEventMonitorDto, method = queryGrid1, cost = 5703 ms
exec method : queryGrid1 time: 5707ms
2024-12-04 15:51:22.313  INFO 69948 --- [nio-8004-exec-7] s.d.e.i.SecurityEventMonitorDaoHibernate : exec sql [select * from  SECURITY_EVENT_MONITOR_202412  where 1=1  and alarmTime >= to_date('2024-12-01','yyyy-MM-dd') and alarmTime <= to_date('2024-12-04','yyyy-MM-dd')] time: 5892ms
2024-12-04 15:51:22.334  INFO 69948 --- [nio-8004-exec-7] s.d.e.i.SecurityEventMonitorDaoHibernate : transform data time: 5914ms
2024-12-04 15:51:22.457  INFO 69948 --- [nio-8004-exec-1] s.d.e.i.SecurityEventMonitorDaoHibernate : exec sql [select * from  SECURITY_EVENT_MONITOR_202412  where 1=1  and alarmTime >= to_date('2024-12-01','yyyy-MM-dd') and alarmTime <= to_date('2024-12-04','yyyy-MM-dd')] time: 6022ms
2024-12-04 15:51:22.457  INFO 69948 --- [nio-8004-exec-2] s.d.e.i.SecurityEventMonitorDaoHibernate : exec sql [select * from  SECURITY_EVENT_MONITOR_202412  where 1=1  and alarmTime >= to_date('2024-12-01','yyyy-MM-dd') and alarmTime <= to_date('2024-12-04','yyyy-MM-dd')] time: 6024ms
2024-12-04 15:51:22.497  INFO 69948 --- [nio-8004-exec-1] s.d.e.i.SecurityEventMonitorDaoHibernate : transform data time: 6061ms
2024-12-04 15:51:22.497  INFO 69948 --- [nio-8004-exec-2] s.d.e.i.SecurityEventMonitorDaoHibernate : transform data time: 6063ms
2024-12-04 15:51:22.498  INFO 69948 --- [nio-8004-exec-1] .s.s.e.i.SecurityEventMonitorServiceImpl : build queryGrid4 data cost time:2ms
2024-12-04 15:51:22.500  INFO 69948 --- [nio-8004-exec-1] com.wisdom.facade.BizWebServiceSkeleton  : Biz web service skeleton process request done
2024-12-04 15:51:22.500  WARN 69948 --- [nio-8004-exec-1] com.wisdom.facade.BizWebServiceSkeleton  : request =  com.wisdom.imd.wypap.security.dto.event.SecurityEventMonitorDto, method = queryGrid4, cost = 6067 ms
exec method : queryGrid4 time: 6069ms
2024-12-04 15:51:22.512  INFO 69948 --- [nio-8004-exec-2] .s.s.e.i.SecurityEventMonitorServiceImpl : build queryGrid3 data cost time:14ms
2024-12-04 15:51:22.514  INFO 69948 --- [nio-8004-exec-2] com.wisdom.facade.BizWebServiceSkeleton  : Biz web service skeleton process request done
2024-12-04 15:51:22.515  WARN 69948 --- [nio-8004-exec-2] com.wisdom.facade.BizWebServiceSkeleton  : request =  com.wisdom.imd.wypap.security.dto.event.SecurityEventMonitorDto, method = queryGrid3, cost = 6085 ms
exec method : queryGrid3 time: 6086ms
2024-12-04 15:51:22.816  INFO 69948 --- [nio-8004-exec-7] .s.s.e.i.SecurityEventMonitorServiceImpl : build queryGrid2 data cost time:480ms
2024-12-04 15:51:22.817  INFO 69948 --- [nio-8004-exec-7] com.wisdom.facade.BizWebServiceSkeleton  : Biz web service skeleton process request done
2024-12-04 15:51:22.818  WARN 69948 --- [nio-8004-exec-7] com.wisdom.facade.BizWebServiceSkeleton  : request =  com.wisdom.imd.wypap.security.dto.event.SecurityEventMonitorDto, method = queryGrid2, cost = 6401 ms
exec method : queryGrid2 time: 6403ms

4个方法最长耗时也不超过7s,前端还是有10-20s的,有个20s作业的空闲时间,等待服务器数据返回时间

这个应该是传输过慢,等待服务器数据返回有点慢

相关推荐
青皮桔28 分钟前
CSS实现百分比水柱图
前端·css
失落的多巴胺28 分钟前
使用deepseek制作“喝什么奶茶”随机抽签小网页
javascript·css·css3·html5
DataGear32 分钟前
如何在DataGear 5.4.1 中快速制作SQL服务端分页的数据表格看板
javascript·数据库·sql·信息可视化·数据分析·echarts·数据可视化
影子信息34 分钟前
vue 前端动态导入文件 import.meta.glob
前端·javascript·vue.js
青阳流月35 分钟前
1.vue权衡的艺术
前端·vue.js·开源
样子201839 分钟前
Vue3 之dialog弹框简单制作
前端·javascript·vue.js·前端框架·ecmascript
kevin_水滴石穿40 分钟前
Vue 中报错 TypeError: crypto$2.getRandomValues is not a function
前端·javascript·vue.js
翻滚吧键盘40 分钟前
vue文本插值
javascript·vue.js·ecmascript
孤水寒月2 小时前
给自己网站增加一个免费的AI助手,纯HTML
前端·人工智能·html
CoderLiu2 小时前
用这个MCP,只给大模型一个figma链接就能直接导出图片,还能自动压缩上传?
前端·llm·mcp