从CSV到数据库(简易)

需求:客户上传CSV文档,要求CSV文档内容查重/插入/更新相关数据。

框架:jdbcTemplate、commons-io、

DB:oracle

相关依赖:

这里本来打算用的2.11.0,无奈正式项目那边用老版本1.3.1,新版本对类型支持和转换好一点。不过无伤大雅。

xml 复制代码
        <dependency>
            <groupId>commons-io</groupId>
            <artifactId>commons-io</artifactId>
            <version>1.3.1</version>
        </dependency>

CSV文档格式:

csv 复制代码
Xxx Code,Yerial,OP
600001,2024082400305, OP20240818_XDFD
600001,2024082400306, OP20240818_XDFD
600001,2024082400307, OP20240818_XDFD
600001,2024082400308, OP20240818_XDFD
600001,2024082400309, OP20240818_XDFD
600001,2024082400310, OP20240818_XDFD
600001,2024082400311, OP20240818_XDFD
600001,2024082400312, OP20240818_XDFD
600001,2024082400313, OP20240818_XDFD
600001,2024082400314, OP20240818_XDFD
600001,2024082400315, OP20240818_XDFD
600001,2024082400316, OP20240818_XDFD
600001,2024082400317, OP20240818_XDFD
600001,2024082400318, OP20240818_XDFD
600001,2024082400319, OP20240818_XDFD
600001,2024082400320, OP20240818_XDFD
600001,2024082400321, OP20240818_XDFD
600001,2024082400322, OP20240818_XDFD
600001,2024082400323, OP20240818_XDFD
600001,2024082400324, OP20240818_XDFD
600001,2024082400325, OP20240818_XDFD

接口:

MultipartFile接受CSV文件

java 复制代码
    @PostMapping("/import")
    public ResponseEntity<BaseResponse<?>> importCSV(@RequestBody MultipartFile files) {
        xxxService.importSerial(files);
        return new ResponseEntity<>(new BaseResponse<>(), HttpStatus.OK);
    }

前段需要在Request-Body中以form-data的形式上传文档

CSV解析

简单转成Json

java 复制代码
    private String convertCsvToJson(MultipartFile multipartFile) throws IOException {
        // read csv as list
        List<String> lines = IOUtils.readLines(multipartFile.getInputStream(), "UTF-8");

        List<List<String>> data = lines.stream()
                .skip(1) // skip label line
                .filter(line -> !line.trim().isEmpty()) // filtering empty line
                .map(line -> Arrays.asList(line.split(",")))
                .collect(Collectors.toList());

        return objectMapper.writeValueAsString(data);
    }

转换后是这样的:

注意这是print出来的json对象,本来应该是json字符串。

json 复制代码
[
    ["600001","2024082400305","OP20240818_XDFD"],
    ["600001","2024082400306","OP20240818_XDFD"],
    ["600001","2024082400307","OP20240818_XDFD"],
    ["600001","2024082400308","OP20240818_XDFD"],
    ["600001","2024082400309","OP20240818_XDFD"],
    ["600001","2024082400310","OP20240818_XDFD"],
    ["600001","2024082400311","OP20240818_XDFD"],
    ["600001","2024082400312","OP20240818_XDFD"],
    ["600001","2024082400313","OP20240818_XDFD"],
    ["600001","2024082400314","OP20240818_XDFD"],
    ["600001","2024082400315","OP20240818_XDFD"],
    ["600001","2024082400316","OP20240818_XDFD"],
    ["600001","2024082400317","OP20240818_XDFD"],
    ["600001","2024082400318","OP20240818_XDFD"],
    ["600001","2024082400319","OP20240818_XDFD"],
    ["600001","2024082400320","OP20240818_XDFD"],
    ["600001","2024082400321","OP20240818_XDFD"],
    ["600001","2024082400322","OP20240818_XDFD"],
    ["600001","2024082400323","OP20240818_XDFD"],
    ["600001","2024082400324","OP20240818_XDFD"],
    ["600001","2024082400325","OP20240818_XDFD"]
]

以clob参数的形式传到oracle存储过程中处理

java 复制代码
    public void import(MultipartFile files) {
        final int[] status = new int[1];
        Object result = jdbcTemplate.execute(new ConnectionCallback<Object>() {
            @Override
            public Object doInConnection(Connection con) throws SQLException, DataAccessException {
                CallableStatement cs = con.prepareCall("{call TEST_PACKAGE.pro_add_csv_data(?, ?)}");
                Clob clob = con.createClob(); // 创建一个Clob对象
                try {
                    String s = convertCsvToJson(files); // csv 转 json字符串
                    clob.setString(1, s); // 把json字符串封装进clob对象中
                    cs.setClob(1, clob);  // 入参
                    cs.registerOutParameter(2, Types.INTEGER); // 出参
                    cs.execute();
                    status[0] = cs.getInt(2); // 取结果
                } catch (IOException e) {
                    throw new RuntimeException("Import failed.");
                }
                return null;
            }
        });
    }

存储过程

sql 复制代码
PROCEDURE pro_add_csv_data(v_data_list IN CLOB,v_status OUT NUMBER) 
AS 
    v_code VARCHAR2(10);  
    v_yerial VARCHAR2(20);
    v_op VARCHAR2(20);
    v_cur SYS_REFCURSOR;
    EXC_EXIST EXCEPTION;
    v_count NUMBER;
    v_ref VARCHAR2(30) := TO_CHAR(SYSDATE, 'YYYYMMDDHH24MI');
BEGIN    
	-- 解析成表
    OPEN v_cur FOR    
        SELECT jt.*      
        FROM (    
            SELECT j.*    
            FROM JSON_TABLE(    
                v_data_list , '$[*]' COLUMNS (
                    code VARCHAR2(10) PATH '$[0]',        
                    yerial VARCHAR2(20) PATH '$[1]',        
                    op VARCHAR2(20) PATH '$[2]'        
                )              
            ) j    
        ) jt;  
    -- 遍历插入
    LOOP FETCH v_cur INTO v_code, v_yerial, v_op;  
    EXIT WHEN v_cur%NOTFOUND;
            INSERT INTO table_one
            VALUES( v_code,
                    v_yerial,
                    v_op
                   );
    END LOOP;
    
    COMMIT;
    CLOSE v_cur;
    v_status := 0;
EXCEPTION  
    WHEN EXC_EXIST THEN 
        ROLLBACK;
        v_status:=2;
    WHEN OTHERS THEN
        ROLLBACK;
        v_status:=1;
END pro_add_csv_data;

完美

相关推荐
←か淡定☆ ヾ9 分钟前
SQL Server 2008R2 到 2012 数据库迁移完整指南
数据库·sql server
瀚高PG实验室16 分钟前
Arcgis连接HGDB报错
数据库·arcgis·瀚高数据库
IT小辉同学1 小时前
PostgreSQL 与 MySQL 获取字段注释并转换为驼峰命名教程
数据库·mysql·postgresql
xinghunzhiye20101 小时前
redis升级
数据库·redis·缓存
一只fish2 小时前
MySQL 8.0 OCP 1Z0-908 题目解析(21)
数据库·mysql
涛思数据(TDengine)2 小时前
时序数据库 TDengine × SSRS:专为工业、能源场景打造的报表解决方案
大数据·数据库·物联网·时序数据库·tdengine
打鱼又晒网2 小时前
Lecture #20:Database Logging
数据库
白仑色2 小时前
Oracle 数据库管理与维护实战指南(用户权限、备份恢复、性能调优)
数据库·oracle·数据库管理·性能调优·备份恢复
wx_ywyy67982 小时前
分布式推客系统全栈开发指南:SpringCloud+Neo4j+Redis实战解析
数据库·oracle·推客系统·推客小程序·推客系统开发·推客小程序开发·推客分销系统
isNotNullX2 小时前
实时数仓和离线数仓还分不清楚?看完就懂了
大数据·数据库·数据仓库·人工智能·数据分析