一种excel多线程并发写sheet的方案

喵喵队摆大烂2023-12-02 9:32

一、背景

有一次项目的需求要求导出excel，并且将不同的数据分别写到不同的sheet中。

二、 方案概述

首先一开始使用easyexcel去导出excel，结果发现导出时间需要3秒左右。于是想着能不能缩短excel导出时间，于是第一次尝试使用异步线程去查询数据库，却发现接口的时间并没有明显缩短，于是自己就开始排查耗时的操作，于是发现是写sheet的时候是串行执行，并且每个写sheet的时间并不短，尤其在sheet比较多的时候，会导致导出的时间比较长。于是，想着能不能使用异步线程并发去写sheet，但是，使用的时候报错。后来去找报错的原因，是因为easyexcel并不支持并发写。于是，我就转战POI。尝试是否能够并发写入多个sheet。

使用easyexcel写入多个sheet

java 复制代码 
        try {
            response.setContentType("application/vnd.openxmlformats-officedocument.spreadsheetml.sheet");
            response.setCharacterEncoding("utf-8");
            // 这里URLEncoder.encode可以防止中文乱码 当然和easyexcel没有关系
            String fileName = URLEncoder.encode(EXCEL, "UTF-8").replaceAll("\\+", "%20");
            response.setHeader("Content-disposition", "attachment;filename*=utf-8''" + fileName + ".xlsx");
            AtomicInteger atomicInteger = new AtomicInteger(0);
            ExcelWriter build = EasyExcel.write(response.getOutputStream(),VirtualInstanceDataPoint.class).build();
            list.stream().map(i -> CompletableFuture.supplyAsync(() -> {
                        return service.list();
                    }, executor)).collect(Collectors.toList()).stream()
                    .map(CompletableFuture::join).collect(Collectors.toList()).forEach(r->{
                        int andIncrement = atomicInteger.getAndIncrement();
                        WriteSheet build1 = EasyExcel.writerSheet(andIncrement, r.get(0).getDevice() + andIncrement).build();
                        build.write(r, build1);
                    });
            build.finish();
            response.flushBuffer();
        } catch (Exception e) {
            // 重置response
            response.reset();
            response.setContentType("application/json");
            response.setCharacterEncoding("utf-8");
            response.getWriter().println(JSON.toJSONString(R.error().message(e.getMessage()).code(20007)));
}

并发写入多个sheet会报

org.apache.poi.openxml4j.exceptions.PartAlreadyExistsException: A part with the name '/xl/worksheets/sheet1.xml' already exists : Packages shall not contain equivalent part names and package implementers shall neither create nor recognize packages with equivalent part names. [M1.12]

POI写入多个sheet

java 复制代码 
    String[] EXPORT_HEADER = {"head1","head2"};
    @GetMapping("export3")
    @ApiOperation(value = "excel导出信息")
    @SneakyThrows
    public void export3(HttpServletResponse response) {
        OutputStream outputStream = response.getOutputStream();
        response.reset();
        response.setContentType("application/vnd.ms-excel");
        response.setHeader("Content-disposition", "attachment;filename=template.xls");
        AtomicInteger atomicInteger = new AtomicInteger();
        HSSFWorkbook workbook = new HSSFWorkbook();
        Font font = workbook.createFont();
        font.setBold(true);
        HSSFCellStyle cellStyle = workbook.createCellStyle();
        cellStyle.setAlignment(HorizontalAlignment.CENTER);
        cellStyle.setVerticalAlignment(VerticalAlignment.CENTER);
        cellStyle.setFont(font);
        List<IndexData> indexDatas = new ArrayList<>();
        indexDatas.add(new IndexData("1",1.1));
        indexDatas.add(new IndexData("2",2.2));
        indexDatas.add(new IndexData("3",3.3));
        for (IndexData indexData : indexDatas) {
            HSSFSheet sheet = workbook.createSheet(indexData.getStr());
            HSSFRow row = sheet.createRow(0);
            // 创建表头
            for (int i = 0; i < EXPORT_HEADER.length; i++) {
                HSSFCell cell = row.createCell(i);
                cell.setCellValue(EXPORT_HEADER[i]);
                cell.setCellStyle(cellStyle);
            }
            row = sheet.createRow(1);
            row.createCell(0).setCellValue(indexData.getStr());
            row.createCell(1).setCellValue(indexData.getDoubleData());
        }

        workbook.write(outputStream);
        outputStream.flush();
        outputStream.close();
    }
    static  class IndexData {
        public IndexData(String string, Double doubleData) {
            this.str = string;
            this.doubleData = doubleData;
        }

        public String getStr() {
            return str;
        }

        public Double getDoubleData() {
            return doubleData;
        }

        private String str;

        private Double doubleData;
    }

POI多线程写多个sheet

java 复制代码 
    String[] EXPORT_HEADER = {"head1","head2"};
    @GetMapping("export3")
    @ApiOperation(value = "excel导出")
    @SneakyThrows
    public void export3(HttpServletResponse response) {
        OutputStream outputStream = response.getOutputStream();
        response.reset();
        response.setContentType("application/vnd.ms-excel");
        response.setHeader("Content-disposition", "attachment;filename=template.xls");
        AtomicInteger atomicInteger = new AtomicInteger();
        HSSFWorkbook workbook = new HSSFWorkbook();
        Font font = workbook.createFont();
        font.setBold(true);
        HSSFCellStyle cellStyle = workbook.createCellStyle();
        cellStyle.setAlignment(HorizontalAlignment.CENTER);
        cellStyle.setVerticalAlignment(VerticalAlignment.CENTER);
        cellStyle.setFont(font);
        List<IndexData> indexDatas = new ArrayList<>();
        indexDatas.add(new IndexData("1",1.1));
        indexDatas.add(new IndexData("2",2.2));
        indexDatas.add(new IndexData("3",3.3));
        indexDatas.stream().map(data ->CompletableFuture.runAsync(() ->{
            HSSFSheet sheet = workbook.createSheet(data.getStr());
            HSSFRow row = sheet.createRow(0);
            // 创建表头
            for (int i = 0; i < EXPORT_HEADER.length; i++) {
                HSSFCell cell = row.createCell(i);
                cell.setCellValue(EXPORT_HEADER[i]);
                cell.setCellStyle(cellStyle);
            }
            row = sheet.createRow(1);
            row.createCell(0).setCellValue(data.getStr());
            row.createCell(1).setCellValue(data.getDoubleData());
        })).collect(Collectors.toList()).stream().map(CompletableFuture::join).collect(Collectors.toList());
        workbook.write(outputStream);
        outputStream.flush();
        outputStream.close();
    }
    static  class IndexData {
        public IndexData(String string, Double doubleData) {
            this.str = string;
            this.doubleData = doubleData;
        }

        public String getStr() {
            return str;
        }

        public Double getDoubleData() {
            return doubleData;
        }

        private String str;

        private Double doubleData;
    }

但是有时候会报错

java.lang.IllegalArgumentException: calculated end index (2576) is out of allowable range (2564..2569)

是因为在 子线程中创建sheet产生并发。

一个解决方案是主线程预先创建sheet

另一个方案是为创建sheet的操作加锁

java 复制代码 
    @GetMapping("export1")
    @ApiOperation(value = "excel导出信息")
    @SneakyThrows
    public void export2(HttpServletResponse response) {
        OutputStream outputStream = response.getOutputStream();
        response.reset();
        response.setContentType("application/vnd.ms-excel");
        response.setHeader("Content-disposition", "attachment;filename=template.xls");
        AtomicInteger atomicInteger = new AtomicInteger();
        HSSFWorkbook workbook = new HSSFWorkbook();
        Font font = workbook.createFont();
        font.setBold(true);
        HSSFCellStyle cellStyle = workbook.createCellStyle();
        cellStyle.setAlignment(HorizontalAlignment.CENTER);
        cellStyle.setVerticalAlignment(VerticalAlignment.CENTER);
        cellStyle.setFont(font);
        list.stream().map(instanceId -> CompletableFuture.runAsync(() -> {
                    List<> collect =  service.list();
                    HSSFSheet sheet = workbook.createSheet(collect.get(0).getDevice()+ atomicInteger.getAndIncrement());
                    HSSFRow row = sheet.createRow(0);
                    // 创建表头
                    for (int i = 0; i < EXPORT_HEADER.length; i++) {
                        HSSFCell cell = row.createCell(i);
                        cell.setCellValue(EXPORT_HEADER[i]);
                        cell.setCellStyle(cellStyle);
                    }
                    for (int i = 0; i < collect.size(); i++) {
                        row = sheet.createRow(i + 1);
                        row.createCell(0).setCellValue(collect.get(i).getInstanceId());
                        row.createCell(1).setCellValue(collect.get(i).getDevice());
                        row.createCell(2).setCellValue(collect.get(i).getDataId());
                        row.createCell(3).setCellValue(collect.get(i).getDataName());
                    }
                }, executor)).collect(Collectors.toList()).stream()
                .map(CompletableFuture::join).collect(Collectors.toList());
        workbook.write(outputStream);
        outputStream.flush();
        outputStream.close();
        
    }

以下使用加锁方式

java 复制代码 
    String[] EXPORT_HEADER = {"head1","head2"};
    @GetMapping("export3")
    @ApiOperation(value = "excel导出信息")
    @SneakyThrows
    public void export3(HttpServletResponse response) {
        OutputStream outputStream = response.getOutputStream();
        response.reset();
        response.setContentType("application/vnd.ms-excel");
        response.setHeader("Content-disposition", "attachment;filename=template.xls");
        AtomicInteger atomicInteger = new AtomicInteger();
        HSSFWorkbook workbook = new HSSFWorkbook();
        Font font = workbook.createFont();
        font.setBold(true);
        HSSFCellStyle cellStyle = workbook.createCellStyle();
        cellStyle.setAlignment(HorizontalAlignment.CENTER);
        cellStyle.setVerticalAlignment(VerticalAlignment.CENTER);
        cellStyle.setFont(font);
        List<IndexData> indexDatas = new ArrayList<>();
        indexDatas.add(new IndexData("1",1.1));
        indexDatas.add(new IndexData("2",2.2));
        indexDatas.add(new IndexData("3",3.3));
        indexDatas.stream().map(data ->CompletableFuture.runAsync(() ->{
            HSSFSheet sheet = getSheet(data, workbook);
            HSSFRow row = sheet.createRow(0);
            // 创建表头
            for (int i = 0; i < EXPORT_HEADER.length; i++) {
                HSSFCell cell = row.createCell(i);
                cell.setCellValue(EXPORT_HEADER[i]);
                cell.setCellStyle(cellStyle);
            }
            row = sheet.createRow(1);
            row.createCell(0).setCellValue(data.getStr());
            row.createCell(1).setCellValue(data.getDoubleData());
        })).collect(Collectors.toList()).stream().map(CompletableFuture::join).collect(Collectors.toList());
        workbook.write(outputStream);
        outputStream.flush();
        outputStream.close();
    }

    @org.jetbrains.annotations.NotNull
    private synchronized static HSSFSheet getSheet(IndexData data, HSSFWorkbook workbook) {
        HSSFSheet sheet = workbook.createSheet(data.getStr());
        return sheet;
    }

但是这种方式还是会有一些并发带来的错误。

java.lang.NullPointerException: null

at org.apache.poi.hssf.record.SSTSerializer.serialize(SSTSerializer.java:70)

at org.apache.poi.hssf.record.SSTRecord.serialize(SSTRecord.java:279)

at org.apache.poi.hssf.record.cont.ContinuableRecord.getRecordSize(ContinuableRecord.java:60)

at org.apache.poi.hssf.model.InternalWorkbook.getSize(InternalWorkbook.java:1072)

at org.apache.poi.hssf.usermodel.HSSFWorkbook.getBytes(HSSFWorkbook.java:1474)

at org.apache.poi.hssf.usermodel.HSSFWorkbook.write(HSSFWorkbook.java:1386)

at org.apache.poi.hssf.usermodel.HSSFWorkbook.write(HSSFWorkbook.java:1374)

但是在本机实测100个线程10个循环出错的个数在20-30之间

我们可以捕获这些错误使用do while循环，当出错的时候可以清空状态再次重试。

总结：

该方法只是本菜鸡的愚见，使用这种方式的确可以完成并发写sheet，缩短接口的相应速度，将3秒左右的接口降低到50ms左右。应该能适合sheet略多，但并发量、数据量不多的excel导出，但本人也是第一次使用POI,所以可能有错误，希望大佬们能够多多指点。

相关推荐
m0_7482552613 分钟前
easyExcel导出大数据量EXCEL文件，前端实现进度条或者遮罩层
前端·excel
小张认为的测试36 分钟前
Linux性能监控命令_nmon 安装与使用以及生成分析Excel图表
linux·服务器·测试工具·自动化·php·excel·压力测试
PieroPc7 小时前
Python 写的 智慧记 进销存 辅助 程序 导入导出 excel 可打印
开发语言·python·excel
不吃鱼的羊1 天前
Excel生成DBC脚本源文件
服务器·网络·excel
chenchihwen1 天前
数据分析时的json to excel 转换的好用小工具
数据分析·json·excel
lxxxxl1 天前
C#调用OpenXml，读取excel行数据，遇到空单元跳过现象处理
excel
m0_748246351 天前
前端通过new Blob下载文档流（下载zip或excel）
前端·excel
不吃鱼不吃鱼2 天前
Excel加载项入门：原理、安装卸载流程与常见问题
excel·wps
深耕AI2 天前
在Excel中绘制ActiveX控件：解决文本编辑框定位问题
java·前端·excel
热门推荐
01〔AI 绘画〕Stable Diffusion 之 解决绘制多人或面部很小的人物时面部崩坏问题 篇02ARM学习（31）编译器对overlay方式的支持03玄机平台应急响应—webshell查杀04DisplayPort AltMode On Type-c协议_Ver2.0(学习笔记)05校验 GPT-4 真实性的三个经典问题：快速区分 GPT-3.5 与 GPT-4，并提供免费测试网站06Docker 夺命连环 15 问076个主流的工业3D管道设计软件08基于大语言模型（LLM）的合成数据生成、策展和评估的综述09(欧拉)openEuler系统添加网卡文件配置流程、(欧拉)openEuler系统手动配置ipv6地址流程、(欧拉)openEuler系统网络管理说明10红米手机使用google play