后端返回前端的数据量过大解决方案
性能面板(Performance) · chrome调试指南
原因
遇到一个页面有好几个表格，部分表格采用虚拟滚动条
数据量有点大
接近快60s了，看一下是哪里导致的慢
后台请求方法执行并不慢
2024-12-04 15:21:52.889 INFO 69948 --- [nio-8004-exec-7] s.d.e.i.SecurityEventMonitorDaoHibernate : build sql time: 4ms
2024-12-04 15:21:52.889 INFO 69948 --- [nio-8004-exec-5] s.d.e.i.SecurityEventMonitorDaoHibernate : build sql time: 4ms
2024-12-04 15:21:52.889 INFO 69948 --- [nio-8004-exec-8] s.d.e.i.SecurityEventMonitorDaoHibernate : build sql time: 4ms
2024-12-04 15:21:52.889 INFO 69948 --- [io-8004-exec-10] s.d.e.i.SecurityEventMonitorDaoHibernate : build sql time: 4ms
2024-12-04 15:21:58.276 INFO 69948 --- [nio-8004-exec-8] s.d.e.i.SecurityEventMonitorDaoHibernate : exec sql [select * from ( select * from SECURITY_EVENT_MONITOR_202411 union all select * from SECURITY_EVENT_MONITOR_202412) where 1=1 and alarmTime >= to_date('2024-11-30','yyyy-MM-dd') and alarmTime <= to_date('2024-12-03','yyyy-MM-dd')] time: 5392ms
2024-12-04 15:21:58.276 INFO 69948 --- [nio-8004-exec-5] s.d.e.i.SecurityEventMonitorDaoHibernate : exec sql [select * from ( select * from SECURITY_EVENT_MONITOR_202411 union all select * from SECURITY_EVENT_MONITOR_202412) where 1=1 and alarmTime >= to_date('2024-11-30','yyyy-MM-dd') and alarmTime <= to_date('2024-12-03','yyyy-MM-dd')] time: 5392ms
2024-12-04 15:21:58.283 INFO 69948 --- [nio-8004-exec-7] s.d.e.i.SecurityEventMonitorDaoHibernate : exec sql [select * from ( select * from SECURITY_EVENT_MONITOR_202411 union all select * from SECURITY_EVENT_MONITOR_202412) where 1=1 and alarmTime >= to_date('2024-11-30','yyyy-MM-dd') and alarmTime <= to_date('2024-12-03','yyyy-MM-dd')] time: 5398ms
2024-12-04 15:21:58.297 INFO 69948 --- [io-8004-exec-10] s.d.e.i.SecurityEventMonitorDaoHibernate : exec sql [select * from ( select * from SECURITY_EVENT_MONITOR_202411 union all select * from SECURITY_EVENT_MONITOR_202412) where 1=1 and alarmTime >= to_date('2024-11-30','yyyy-MM-dd') and alarmTime <= to_date('2024-12-03','yyyy-MM-dd')] time: 5412ms
2024-12-04 15:21:58.412 INFO 69948 --- [io-8004-exec-10] s.d.e.i.SecurityEventMonitorDaoHibernate : transform data time: 5527ms
2024-12-04 15:21:58.414 INFO 69948 --- [nio-8004-exec-5] s.d.e.i.SecurityEventMonitorDaoHibernate : transform data time: 5527ms
2024-12-04 15:21:58.481 INFO 69948 --- [nio-8004-exec-8] s.d.e.i.SecurityEventMonitorDaoHibernate : transform data time: 5595ms
2024-12-04 15:21:58.482 INFO 69948 --- [nio-8004-exec-7] s.d.e.i.SecurityEventMonitorDaoHibernate : transform data time: 5597ms
2024-12-04 15:21:58.485 INFO 69948 --- [nio-8004-exec-8] .s.s.e.i.SecurityEventMonitorServiceImpl : build queryGrid4 data cost time:2ms
2024-12-04 15:21:58.553 INFO 69948 --- [nio-8004-exec-7] .s.s.e.i.SecurityEventMonitorServiceImpl : build queryGrid1 data cost time:70ms
2024-12-04 15:21:58.562 INFO 69948 --- [nio-8004-exec-7] com.wisdom.facade.BizWebServiceSkeleton : Biz web service skeleton process request done
2024-12-04 15:21:58.562 WARN 69948 --- [nio-8004-exec-7] com.wisdom.facade.BizWebServiceSkeleton : request = com.wisdom.imd.wypap.security.dto.event.SecurityEventMonitorDto, method = queryGrid1, cost = 5690 ms
2024-12-04 15:21:58.562 INFO 69948 --- [io-8004-exec-10] .s.s.e.i.SecurityEventMonitorServiceImpl : build queryGrid3 data cost time:149ms
2024-12-04 15:21:58.564 INFO 69948 --- [io-8004-exec-10] com.wisdom.facade.BizWebServiceSkeleton : Biz web service skeleton process request done
2024-12-04 15:21:58.564 WARN 69948 --- [io-8004-exec-10] com.wisdom.facade.BizWebServiceSkeleton : request = com.wisdom.imd.wypap.security.dto.event.SecurityEventMonitorDto, method = queryGrid3, cost = 5692 ms
exec method : queryGrid3 time: 5709ms
exec method : queryGrid1 time: 5709ms
2024-12-04 15:21:58.571 INFO 69948 --- [nio-8004-exec-8] com.wisdom.facade.BizWebServiceSkeleton : Biz web service skeleton process request done
2024-12-04 15:21:58.571 WARN 69948 --- [nio-8004-exec-8] com.wisdom.facade.BizWebServiceSkeleton : request = com.wisdom.imd.wypap.security.dto.event.SecurityEventMonitorDto, method = queryGrid4, cost = 5699 ms
exec method : queryGrid4 time: 5712ms
2024-12-04 15:21:59.711 INFO 69948 --- [nio-8004-exec-5] .s.s.e.i.SecurityEventMonitorServiceImpl : build queryGrid2 data cost time:1297ms
2024-12-04 15:21:59.713 INFO 69948 --- [nio-8004-exec-5] com.wisdom.facade.BizWebServiceSkeleton : Biz web service skeleton process request done
2024-12-04 15:21:59.713 WARN 69948 --- [nio-8004-exec-5] com.wisdom.facade.BizWebServiceSkeleton : request = com.wisdom.imd.wypap.security.dto.event.SecurityEventMonitorDto, method = queryGrid2, cost = 6841 ms
exec method : queryGrid2 time: 6854ms后台并行请求了4分方法，这个最慢接近60s，直接看一下这个为什么，找到占比最多的时间在执行什么方法
我靠，返回这么大，主要是解析后台加密的json数据过大导致
json会存储额外的结构，易读，但是数据量过大时性能不好
序列化使用json肯定说改不了了，只能想办法去节省传输的内容大小
后端使用GZIPOutputStream
java
package com.wisdom.framework.web.utils;
import java.io.BufferedReader;
import java.io.ByteArrayInputStream;
import java.io.ByteArrayOutputStream;
import java.io.InputStreamReader;
import java.util.Base64;
import java.util.zip.GZIPInputStream;
import java.util.zip.GZIPOutputStream;
public class GzipUtils {
/****
* 字符串压缩
* @param str
* @return
*/
public static String compress(String str){
try {
if (str == null || str.length() == 0) {
return str;
}
ByteArrayOutputStream obj=new ByteArrayOutputStream();
GZIPOutputStream gzip = new GZIPOutputStream(obj);
gzip.write(str.getBytes("UTF-8"));
gzip.close();
return Base64.getEncoder().encodeToString(obj.toByteArray());
}catch (Exception ex){
ex.printStackTrace();
}
return str;
}
/***
* 字符串解压
* @param str
* @return
*/
public static String decompress(String str) {
if (str == null || str.length() == 0) {
return str;
}
try {
byte[] compressed = Base64.getDecoder().decode(str);
GZIPInputStream gis = new GZIPInputStream(new ByteArrayInputStream(compressed));
StringBuilder outStr = new StringBuilder();
BufferedReader br = new BufferedReader(new InputStreamReader(gis, "UTF-8"));
String line;
while ((line = br.readLine()) != null) {
outStr.append(line);
}
gis.close();
return outStr.toString();
}catch (Exception ex){
ex.printStackTrace();
}
return str;
}
}使用
判断json字符串长度大于200,000，就把json压缩一下
java
String json = JsonUtils.serialize(wri, jsonConfig);
if(json.length() > 200000){
json = GzipUtils.compress(json);
}
return json;前端使用pako工具解压
https://github.com/nodeca/pako
放在lib目录下
引入js
html
<script src="../../../scripts/lib/zip/pako.es5.min.js" type="text/javascript"></script>
js
//压缩字符
if(typeof data == "string" && data.indexOf("H4sIAAAAA") == 0){
let gzipArr = atob(data).split('');
let gzipArr_add = [];
for(let i = 0;i<gzipArr.length;i++){
let ch_ = gzipArr[i];
gzipArr_add.push(ch_.charCodeAt(0));
};
let gzipData = new Uint8Array(gzipArr_add);
data = pako.inflate(gzipData, { to: 'string' });
}测试
传输内容大小缩小了很多，时间也将近减少了一半
再看看后台方法的执行时间
2024-12-04 15:51:16.434 INFO 69948 --- [nio-8004-exec-2] s.d.e.i.SecurityEventMonitorDaoHibernate : build sql time: 0ms
2024-12-04 15:51:16.436 INFO 69948 --- [nio-8004-exec-1] s.d.e.i.SecurityEventMonitorDaoHibernate : build sql time: 0ms
2024-12-04 15:51:22.100 INFO 69948 --- [nio-8004-exec-3] s.d.e.i.SecurityEventMonitorDaoHibernate : exec sql [select * from SECURITY_EVENT_MONITOR_202412 where 1=1 and alarmTime >= to_date('2024-12-01','yyyy-MM-dd') and alarmTime <= to_date('2024-12-04','yyyy-MM-dd')] time: 5681ms
2024-12-04 15:51:22.110 INFO 69948 --- [nio-8004-exec-3] s.d.e.i.SecurityEventMonitorDaoHibernate : transform data time: 5690ms
2024-12-04 15:51:22.117 INFO 69948 --- [nio-8004-exec-3] .s.s.e.i.SecurityEventMonitorServiceImpl : build queryGrid1 data cost time:6ms
2024-12-04 15:51:22.119 INFO 69948 --- [nio-8004-exec-3] com.wisdom.facade.BizWebServiceSkeleton : Biz web service skeleton process request done
2024-12-04 15:51:22.119 WARN 69948 --- [nio-8004-exec-3] com.wisdom.facade.BizWebServiceSkeleton : request = com.wisdom.imd.wypap.security.dto.event.SecurityEventMonitorDto, method = queryGrid1, cost = 5703 ms
exec method : queryGrid1 time: 5707ms
2024-12-04 15:51:22.313 INFO 69948 --- [nio-8004-exec-7] s.d.e.i.SecurityEventMonitorDaoHibernate : exec sql [select * from SECURITY_EVENT_MONITOR_202412 where 1=1 and alarmTime >= to_date('2024-12-01','yyyy-MM-dd') and alarmTime <= to_date('2024-12-04','yyyy-MM-dd')] time: 5892ms
2024-12-04 15:51:22.334 INFO 69948 --- [nio-8004-exec-7] s.d.e.i.SecurityEventMonitorDaoHibernate : transform data time: 5914ms
2024-12-04 15:51:22.457 INFO 69948 --- [nio-8004-exec-1] s.d.e.i.SecurityEventMonitorDaoHibernate : exec sql [select * from SECURITY_EVENT_MONITOR_202412 where 1=1 and alarmTime >= to_date('2024-12-01','yyyy-MM-dd') and alarmTime <= to_date('2024-12-04','yyyy-MM-dd')] time: 6022ms
2024-12-04 15:51:22.457 INFO 69948 --- [nio-8004-exec-2] s.d.e.i.SecurityEventMonitorDaoHibernate : exec sql [select * from SECURITY_EVENT_MONITOR_202412 where 1=1 and alarmTime >= to_date('2024-12-01','yyyy-MM-dd') and alarmTime <= to_date('2024-12-04','yyyy-MM-dd')] time: 6024ms
2024-12-04 15:51:22.497 INFO 69948 --- [nio-8004-exec-1] s.d.e.i.SecurityEventMonitorDaoHibernate : transform data time: 6061ms
2024-12-04 15:51:22.497 INFO 69948 --- [nio-8004-exec-2] s.d.e.i.SecurityEventMonitorDaoHibernate : transform data time: 6063ms
2024-12-04 15:51:22.498 INFO 69948 --- [nio-8004-exec-1] .s.s.e.i.SecurityEventMonitorServiceImpl : build queryGrid4 data cost time:2ms
2024-12-04 15:51:22.500 INFO 69948 --- [nio-8004-exec-1] com.wisdom.facade.BizWebServiceSkeleton : Biz web service skeleton process request done
2024-12-04 15:51:22.500 WARN 69948 --- [nio-8004-exec-1] com.wisdom.facade.BizWebServiceSkeleton : request = com.wisdom.imd.wypap.security.dto.event.SecurityEventMonitorDto, method = queryGrid4, cost = 6067 ms
exec method : queryGrid4 time: 6069ms
2024-12-04 15:51:22.512 INFO 69948 --- [nio-8004-exec-2] .s.s.e.i.SecurityEventMonitorServiceImpl : build queryGrid3 data cost time:14ms
2024-12-04 15:51:22.514 INFO 69948 --- [nio-8004-exec-2] com.wisdom.facade.BizWebServiceSkeleton : Biz web service skeleton process request done
2024-12-04 15:51:22.515 WARN 69948 --- [nio-8004-exec-2] com.wisdom.facade.BizWebServiceSkeleton : request = com.wisdom.imd.wypap.security.dto.event.SecurityEventMonitorDto, method = queryGrid3, cost = 6085 ms
exec method : queryGrid3 time: 6086ms
2024-12-04 15:51:22.816 INFO 69948 --- [nio-8004-exec-7] .s.s.e.i.SecurityEventMonitorServiceImpl : build queryGrid2 data cost time:480ms
2024-12-04 15:51:22.817 INFO 69948 --- [nio-8004-exec-7] com.wisdom.facade.BizWebServiceSkeleton : Biz web service skeleton process request done
2024-12-04 15:51:22.818 WARN 69948 --- [nio-8004-exec-7] com.wisdom.facade.BizWebServiceSkeleton : request = com.wisdom.imd.wypap.security.dto.event.SecurityEventMonitorDto, method = queryGrid2, cost = 6401 ms
exec method : queryGrid2 time: 6403ms4个方法最长耗时也不超过7s，前端还是有10-20s的，有个20s作业的空闲时间，等待服务器数据返回时间
这个应该是传输过慢，等待服务器数据返回有点慢