后端返回前端的数据量过大解决方案

后端返回前端的数据量过大解决方案

性能面板(Performance) · chrome调试指南

原因

遇到一个页面有好几个表格,部分表格采用虚拟滚动条

数据量有点大

接近快60s了,看一下是哪里导致的慢

后台请求方法执行并不慢

2024-12-04 15:21:52.889  INFO 69948 --- [nio-8004-exec-7] s.d.e.i.SecurityEventMonitorDaoHibernate : build sql time: 4ms
2024-12-04 15:21:52.889  INFO 69948 --- [nio-8004-exec-5] s.d.e.i.SecurityEventMonitorDaoHibernate : build sql time: 4ms
2024-12-04 15:21:52.889  INFO 69948 --- [nio-8004-exec-8] s.d.e.i.SecurityEventMonitorDaoHibernate : build sql time: 4ms
2024-12-04 15:21:52.889  INFO 69948 --- [io-8004-exec-10] s.d.e.i.SecurityEventMonitorDaoHibernate : build sql time: 4ms
2024-12-04 15:21:58.276  INFO 69948 --- [nio-8004-exec-8] s.d.e.i.SecurityEventMonitorDaoHibernate : exec sql [select * from  ( select * from SECURITY_EVENT_MONITOR_202411 union all  select * from SECURITY_EVENT_MONITOR_202412)  where 1=1  and alarmTime >= to_date('2024-11-30','yyyy-MM-dd') and alarmTime <= to_date('2024-12-03','yyyy-MM-dd')] time: 5392ms
2024-12-04 15:21:58.276  INFO 69948 --- [nio-8004-exec-5] s.d.e.i.SecurityEventMonitorDaoHibernate : exec sql [select * from  ( select * from SECURITY_EVENT_MONITOR_202411 union all  select * from SECURITY_EVENT_MONITOR_202412)  where 1=1  and alarmTime >= to_date('2024-11-30','yyyy-MM-dd') and alarmTime <= to_date('2024-12-03','yyyy-MM-dd')] time: 5392ms
2024-12-04 15:21:58.283  INFO 69948 --- [nio-8004-exec-7] s.d.e.i.SecurityEventMonitorDaoHibernate : exec sql [select * from  ( select * from SECURITY_EVENT_MONITOR_202411 union all  select * from SECURITY_EVENT_MONITOR_202412)  where 1=1  and alarmTime >= to_date('2024-11-30','yyyy-MM-dd') and alarmTime <= to_date('2024-12-03','yyyy-MM-dd')] time: 5398ms
2024-12-04 15:21:58.297  INFO 69948 --- [io-8004-exec-10] s.d.e.i.SecurityEventMonitorDaoHibernate : exec sql [select * from  ( select * from SECURITY_EVENT_MONITOR_202411 union all  select * from SECURITY_EVENT_MONITOR_202412)  where 1=1  and alarmTime >= to_date('2024-11-30','yyyy-MM-dd') and alarmTime <= to_date('2024-12-03','yyyy-MM-dd')] time: 5412ms
2024-12-04 15:21:58.412  INFO 69948 --- [io-8004-exec-10] s.d.e.i.SecurityEventMonitorDaoHibernate : transform data time: 5527ms
2024-12-04 15:21:58.414  INFO 69948 --- [nio-8004-exec-5] s.d.e.i.SecurityEventMonitorDaoHibernate : transform data time: 5527ms
2024-12-04 15:21:58.481  INFO 69948 --- [nio-8004-exec-8] s.d.e.i.SecurityEventMonitorDaoHibernate : transform data time: 5595ms
2024-12-04 15:21:58.482  INFO 69948 --- [nio-8004-exec-7] s.d.e.i.SecurityEventMonitorDaoHibernate : transform data time: 5597ms
2024-12-04 15:21:58.485  INFO 69948 --- [nio-8004-exec-8] .s.s.e.i.SecurityEventMonitorServiceImpl : build queryGrid4 data cost time:2ms
2024-12-04 15:21:58.553  INFO 69948 --- [nio-8004-exec-7] .s.s.e.i.SecurityEventMonitorServiceImpl : build queryGrid1 data cost time:70ms
2024-12-04 15:21:58.562  INFO 69948 --- [nio-8004-exec-7] com.wisdom.facade.BizWebServiceSkeleton  : Biz web service skeleton process request done
2024-12-04 15:21:58.562  WARN 69948 --- [nio-8004-exec-7] com.wisdom.facade.BizWebServiceSkeleton  : request =  com.wisdom.imd.wypap.security.dto.event.SecurityEventMonitorDto, method = queryGrid1, cost = 5690 ms
2024-12-04 15:21:58.562  INFO 69948 --- [io-8004-exec-10] .s.s.e.i.SecurityEventMonitorServiceImpl : build queryGrid3 data cost time:149ms
2024-12-04 15:21:58.564  INFO 69948 --- [io-8004-exec-10] com.wisdom.facade.BizWebServiceSkeleton  : Biz web service skeleton process request done
2024-12-04 15:21:58.564  WARN 69948 --- [io-8004-exec-10] com.wisdom.facade.BizWebServiceSkeleton  : request =  com.wisdom.imd.wypap.security.dto.event.SecurityEventMonitorDto, method = queryGrid3, cost = 5692 ms
exec method : queryGrid3 time: 5709ms
exec method : queryGrid1 time: 5709ms
2024-12-04 15:21:58.571  INFO 69948 --- [nio-8004-exec-8] com.wisdom.facade.BizWebServiceSkeleton  : Biz web service skeleton process request done
2024-12-04 15:21:58.571  WARN 69948 --- [nio-8004-exec-8] com.wisdom.facade.BizWebServiceSkeleton  : request =  com.wisdom.imd.wypap.security.dto.event.SecurityEventMonitorDto, method = queryGrid4, cost = 5699 ms
exec method : queryGrid4 time: 5712ms
2024-12-04 15:21:59.711  INFO 69948 --- [nio-8004-exec-5] .s.s.e.i.SecurityEventMonitorServiceImpl : build queryGrid2 data cost time:1297ms
2024-12-04 15:21:59.713  INFO 69948 --- [nio-8004-exec-5] com.wisdom.facade.BizWebServiceSkeleton  : Biz web service skeleton process request done
2024-12-04 15:21:59.713  WARN 69948 --- [nio-8004-exec-5] com.wisdom.facade.BizWebServiceSkeleton  : request =  com.wisdom.imd.wypap.security.dto.event.SecurityEventMonitorDto, method = queryGrid2, cost = 6841 ms
exec method : queryGrid2 time: 6854ms

后台并行请求了4分方法,这个最慢接近60s,直接看一下这个为什么,找到占比最多的时间在执行什么方法

我靠,返回这么大,主要是解析后台加密的json数据过大导致

json会存储额外的结构,易读,但是数据量过大时性能不好

序列化使用json肯定说改不了了,只能想办法去节省传输的内容大小

后端使用GZIPOutputStream

java 复制代码
package com.wisdom.framework.web.utils;

import java.io.BufferedReader;
import java.io.ByteArrayInputStream;
import java.io.ByteArrayOutputStream;
import java.io.InputStreamReader;
import java.util.Base64;
import java.util.zip.GZIPInputStream;
import java.util.zip.GZIPOutputStream;

public class GzipUtils {

    /****
     * 字符串压缩
     * @param str
     * @return
     */
    public static String compress(String str){
        try {
            if (str == null || str.length() == 0) {
                return str;
            }
            ByteArrayOutputStream obj=new ByteArrayOutputStream();
            GZIPOutputStream gzip = new GZIPOutputStream(obj);
            gzip.write(str.getBytes("UTF-8"));
            gzip.close();
            return Base64.getEncoder().encodeToString(obj.toByteArray());
        }catch (Exception ex){
            ex.printStackTrace();
        }
        return str;
    }

    /***
     * 字符串解压
     * @param str
     * @return
     */
    public static String decompress(String str) {
        if (str == null || str.length() == 0) {
            return str;
        }
        try {
            byte[] compressed = Base64.getDecoder().decode(str);
            GZIPInputStream gis = new GZIPInputStream(new ByteArrayInputStream(compressed));
            StringBuilder outStr = new StringBuilder();
            BufferedReader br = new BufferedReader(new InputStreamReader(gis, "UTF-8"));

            String line;
            while ((line = br.readLine()) != null) {
                outStr.append(line);
            }
            gis.close();
            return outStr.toString();
        }catch (Exception ex){
            ex.printStackTrace();
        }
        return str;
    }

}

使用

判断json字符串长度大于200,000,就把json压缩一下

java 复制代码
String json = JsonUtils.serialize(wri, jsonConfig);
if(json.length() > 200000){
    json = GzipUtils.compress(json);
}
return json;

前端使用pako工具解压

https://github.com/nodeca/pako

放在lib目录下

引入js

html 复制代码
<script src="../../../scripts/lib/zip/pako.es5.min.js" type="text/javascript"></script>
js 复制代码
//压缩字符
                if(typeof data == "string" && data.indexOf("H4sIAAAAA") == 0){
                    let gzipArr = atob(data).split('');
                    let gzipArr_add = [];
                    for(let i = 0;i<gzipArr.length;i++){
                        let ch_ = gzipArr[i];
                        gzipArr_add.push(ch_.charCodeAt(0));
                    };
                    let gzipData = new Uint8Array(gzipArr_add);
                    data = pako.inflate(gzipData, { to: 'string' });
                }

测试

传输内容大小缩小了很多,时间也将近减少了一半

再看看后台方法的执行时间

2024-12-04 15:51:16.434  INFO 69948 --- [nio-8004-exec-2] s.d.e.i.SecurityEventMonitorDaoHibernate : build sql time: 0ms
2024-12-04 15:51:16.436  INFO 69948 --- [nio-8004-exec-1] s.d.e.i.SecurityEventMonitorDaoHibernate : build sql time: 0ms
2024-12-04 15:51:22.100  INFO 69948 --- [nio-8004-exec-3] s.d.e.i.SecurityEventMonitorDaoHibernate : exec sql [select * from  SECURITY_EVENT_MONITOR_202412  where 1=1  and alarmTime >= to_date('2024-12-01','yyyy-MM-dd') and alarmTime <= to_date('2024-12-04','yyyy-MM-dd')] time: 5681ms
2024-12-04 15:51:22.110  INFO 69948 --- [nio-8004-exec-3] s.d.e.i.SecurityEventMonitorDaoHibernate : transform data time: 5690ms
2024-12-04 15:51:22.117  INFO 69948 --- [nio-8004-exec-3] .s.s.e.i.SecurityEventMonitorServiceImpl : build queryGrid1 data cost time:6ms
2024-12-04 15:51:22.119  INFO 69948 --- [nio-8004-exec-3] com.wisdom.facade.BizWebServiceSkeleton  : Biz web service skeleton process request done
2024-12-04 15:51:22.119  WARN 69948 --- [nio-8004-exec-3] com.wisdom.facade.BizWebServiceSkeleton  : request =  com.wisdom.imd.wypap.security.dto.event.SecurityEventMonitorDto, method = queryGrid1, cost = 5703 ms
exec method : queryGrid1 time: 5707ms
2024-12-04 15:51:22.313  INFO 69948 --- [nio-8004-exec-7] s.d.e.i.SecurityEventMonitorDaoHibernate : exec sql [select * from  SECURITY_EVENT_MONITOR_202412  where 1=1  and alarmTime >= to_date('2024-12-01','yyyy-MM-dd') and alarmTime <= to_date('2024-12-04','yyyy-MM-dd')] time: 5892ms
2024-12-04 15:51:22.334  INFO 69948 --- [nio-8004-exec-7] s.d.e.i.SecurityEventMonitorDaoHibernate : transform data time: 5914ms
2024-12-04 15:51:22.457  INFO 69948 --- [nio-8004-exec-1] s.d.e.i.SecurityEventMonitorDaoHibernate : exec sql [select * from  SECURITY_EVENT_MONITOR_202412  where 1=1  and alarmTime >= to_date('2024-12-01','yyyy-MM-dd') and alarmTime <= to_date('2024-12-04','yyyy-MM-dd')] time: 6022ms
2024-12-04 15:51:22.457  INFO 69948 --- [nio-8004-exec-2] s.d.e.i.SecurityEventMonitorDaoHibernate : exec sql [select * from  SECURITY_EVENT_MONITOR_202412  where 1=1  and alarmTime >= to_date('2024-12-01','yyyy-MM-dd') and alarmTime <= to_date('2024-12-04','yyyy-MM-dd')] time: 6024ms
2024-12-04 15:51:22.497  INFO 69948 --- [nio-8004-exec-1] s.d.e.i.SecurityEventMonitorDaoHibernate : transform data time: 6061ms
2024-12-04 15:51:22.497  INFO 69948 --- [nio-8004-exec-2] s.d.e.i.SecurityEventMonitorDaoHibernate : transform data time: 6063ms
2024-12-04 15:51:22.498  INFO 69948 --- [nio-8004-exec-1] .s.s.e.i.SecurityEventMonitorServiceImpl : build queryGrid4 data cost time:2ms
2024-12-04 15:51:22.500  INFO 69948 --- [nio-8004-exec-1] com.wisdom.facade.BizWebServiceSkeleton  : Biz web service skeleton process request done
2024-12-04 15:51:22.500  WARN 69948 --- [nio-8004-exec-1] com.wisdom.facade.BizWebServiceSkeleton  : request =  com.wisdom.imd.wypap.security.dto.event.SecurityEventMonitorDto, method = queryGrid4, cost = 6067 ms
exec method : queryGrid4 time: 6069ms
2024-12-04 15:51:22.512  INFO 69948 --- [nio-8004-exec-2] .s.s.e.i.SecurityEventMonitorServiceImpl : build queryGrid3 data cost time:14ms
2024-12-04 15:51:22.514  INFO 69948 --- [nio-8004-exec-2] com.wisdom.facade.BizWebServiceSkeleton  : Biz web service skeleton process request done
2024-12-04 15:51:22.515  WARN 69948 --- [nio-8004-exec-2] com.wisdom.facade.BizWebServiceSkeleton  : request =  com.wisdom.imd.wypap.security.dto.event.SecurityEventMonitorDto, method = queryGrid3, cost = 6085 ms
exec method : queryGrid3 time: 6086ms
2024-12-04 15:51:22.816  INFO 69948 --- [nio-8004-exec-7] .s.s.e.i.SecurityEventMonitorServiceImpl : build queryGrid2 data cost time:480ms
2024-12-04 15:51:22.817  INFO 69948 --- [nio-8004-exec-7] com.wisdom.facade.BizWebServiceSkeleton  : Biz web service skeleton process request done
2024-12-04 15:51:22.818  WARN 69948 --- [nio-8004-exec-7] com.wisdom.facade.BizWebServiceSkeleton  : request =  com.wisdom.imd.wypap.security.dto.event.SecurityEventMonitorDto, method = queryGrid2, cost = 6401 ms
exec method : queryGrid2 time: 6403ms

4个方法最长耗时也不超过7s,前端还是有10-20s的,有个20s作业的空闲时间,等待服务器数据返回时间

这个应该是传输过慢,等待服务器数据返回有点慢

相关推荐
m0_748240255 分钟前
前端如何检测用户登录状态是否过期
前端
black^sugar6 分钟前
纯前端实现更新检测
开发语言·前端·javascript
寻找沙漠的人1 小时前
前端知识补充—CSS
前端·css
GISer_Jing1 小时前
2025前端面试热门题目——计算机网络篇
前端·计算机网络·面试
m0_748245521 小时前
吉利前端、AI面试
前端·面试·职场和发展
理想不理想v1 小时前
webpack最基础的配置
前端·webpack·node.js
pubuzhixing1 小时前
开源白板新方案:Plait 同时支持 Angular 和 React 啦!
前端·开源·github
2401_857600951 小时前
SSM 与 Vue 共筑电脑测评系统:精准洞察电脑世界
前端·javascript·vue.js
2401_857600951 小时前
数字时代的医疗挂号变革:SSM+Vue 系统设计与实现之道
前端·javascript·vue.js
GDAL1 小时前
vue入门教程:组件透传 Attributes
前端·javascript·vue.js