hive sql 处理多层 json 数组

第一片心意2024-08-16 8:34

1. 背景

json 字符串值数据示例：

json 复制代码 
{
    "score": 1,
    "submitTime": 1712491933,
    "answerFlag": 1,
    "groupId": 1755547960,
    "answers": [
        {
            "value": "[1, 2, 3]",
            "ids": [
                4,
                5,
                6
            ],
            "isPic": 0,
            "duration": 22314,
            "status": 1,
            "tid": 1
        },
        {
            "value": "aabbcc",
            "lessons": [
                44,
                55,
                66
            ],
            "isPic": 0,
            "duration": 22314,
            "status": 2,
            "tid": 2
        }
    ],
    "questionType": 65
}

现在这个 json 字符串形式的字段值在 hive 表的某个字段中，我需要获取到 "answers" 这个 json 数组，然后将其按照数组长度，列转行到多行数据，每行数据一个子 json ，并且从中获取到每个子 json 的 "tid" 和 "status" 值，理想情况下，我需要这行数据处理完之后，结果如下表所示。

tid status
1 1
2 2

2. 常见方案

通过 ge_json_object 函数，先获取到 "answers" 对应的 json 数组字符串，然后通过正则替换掉 [ 和 ] 符号，之后将 },{ 符号替换为 }我是分隔符{，最后将 我是分隔符 作为 split 函数的分隔符号，将字符串分割，再通过 lateral view explode() 语法，将数组放到多行。多行数据，都是处理好的 json 对象，之后通过 get_json_object 函数后取需要的字段值即可。具体代码示例如下。

  1. 获取 "answers" 对应的 json 数组。

    sql 复制代码 
    with table1 as (
    select '{"score":1,"submitTime":1712491933,"answerFlag":1,"groupId":1755547960,"answers":[{"value":"[1, 2, 3]","ids":[4,5,6],"isPic":0,"duration":22314,"status":1,"tid":1},{"value":"aabbcc","lessons":[44,55,66],"isPic":0,"duration":22314,"status":2,"tid":2}],"questionType":65}' as col
)

select get_json_object(col, '$.answers') as answers
from table1

    结果如下所示，正常获取到 "answers" 下的 json 数组，结果为字符串。

    answers
    [{"value":"[1, 2, 3]","ids":[4,5,6],"isPic":0,"duration":22314,"status":1,"tid":1},{"value":"aabbcc","lessons":[44,55,66],"isPic":0,"duration":22314,"status":2,"tid":2}]

  2. 将最外层的 [ 和 ] 符号去掉

    sql 复制代码 
    with table1 as (
    select '{"score":1,"submitTime":1712491933,"answerFlag":1,"groupId":1755547960,"answers":[{"value":"[1, 2, 3]","ids":[4,5,6],"isPic":0,"duration":22314,"status":1,"tid":1},{"value":"aabbcc","lessons":[44,55,66],"isPic":0,"duration":22314,"status":2,"tid":2}],"questionType":65}' as col
)

select regexp_replace(answers, '\\[|\\]', '') as answers
from (
    select get_json_object(col, '$.answers') as answers
    from table1
    ) as a

    结果如下所示。

    answers
    {"value":"1, 2, 3","ids":4,5,6,"isPic":0,"duration":22314,"status":1,"tid":1},{"value":"aabbcc","lessons":44,55,66,"isPic":0,"duration":22314,"status":2,"tid":2}

    由于第一步处理的结果最外层是 json 数组，左右有 [ 和 ] 符号，但是由于内层还有子 json 数组，这种直接全局替换的方式，会将内层子 json 数组的 [、] 符号也一并去掉，可以查看下面的结果，"ids":4,5,6,，就是因为全局替换，造成整个 json 结构被破坏，之后将无法使用 get_json_object() 函数来获取想要的 key 对应的值了。

由此可见，这种方式只适合于 "answers" 下的 json 数组内的每个 json 对象中都只包含 json 对象才行，不能再包含 json 数组，否则就会造成处理错误，拿不到想要的数据。

3. 推荐方案

3.1 具体步骤

  1. 获取 "answers" 对应的 json 数组。

    sql 复制代码 
    with table1 as (
    select '{"score":1,"submitTime":1712491933,"answerFlag":1,"groupId":1755547960,"answers":[{"value":"[1, 2, 3]","ids":[4,5,6],"isPic":0,"duration":22314,"status":1,"tid":1},{"value":"aabbcc","lessons":[44,55,66],"isPic":0,"duration":22314,"status":2,"tid":2}],"questionType":65}' as col
)

select get_json_object(col, '$.answers') as answers
from table1

    结果如下所示，正常获取到 "answers" 下的 json 数组，结果为字符串。

    answers
    [{"value":"[1, 2, 3]","ids":[4,5,6],"isPic":0,"duration":22314,"status":1,"tid":1},{"value":"aabbcc","lessons":[44,55,66],"isPic":0,"duration":22314,"status":2,"tid":2}]

  2. 通过字符串截取的方式，将第一步的结果最前面和最后面的 [、] 符号去掉

    sql 复制代码 
    with table1 as (
    select '{"score":1,"submitTime":1712491933,"answerFlag":1,"groupId":1755547960,"answers":[{"value":"[1, 2, 3]","ids":[4,5,6],"isPic":0,"duration":22314,"status":1,"tid":1},{"value":"aabbcc","lessons":[44,55,66],"isPic":0,"duration":22314,"status":2,"tid":2}],"questionType":65}' as col
)

select substring(answers, 2, length(answers) - 2) as answers
from (
    select get_json_object(col, '$.answers') as answers
    from table1
    ) as a

    结果如下所示，最前面和最后面的 [、] 符号已经被去掉。

    answers
    {"value":"[1, 2, 3]","ids":[4,5,6],"isPic":0,"duration":22314,"status":1,"tid":1},{"value":"aabbcc","lessons":[44,55,66],"isPic":0,"duration":22314,"status":2,"tid":2}

  3. 最重要的一步 ：通过正则替换，只要匹配到 [.*] 内容，就直接替换为数字 0。

    sql 复制代码 
    with table1 as (
    select '{"score":1,"submitTime":1712491933,"answerFlag":1,"groupId":1755547960,"answers":[{"value":"[1, 2, 3]","ids":[4,5,6],"isPic":0,"duration":22314,"status":1,"tid":1},{"value":"aabbcc","lessons":[44,55,66],"isPic":0,"duration":22314,"status":2,"tid":2}],"questionType":65}' as col
)

select substring(answers, 1, length(answers) - 3) as answers
from (
    select get_json_object(col, '$.answers') as answers
    from table1
    ) as a

    结果如下所示，字符串中所有 [.*] 的部分，都已经被替换为数字 0

    answers
    {"value":"0","ids":0,"isPic":0,"duration":22314,"status":1,"tid":1},{"value":"aabbcc","lessons":0,"isPic":0,"duration":22314,"status":2,"tid":2}

    将 json 数字替换为数字 0，是为了兼容 "[.*]" 和 [.*] 两种情况，不管 json 数组是不是被英文双引号包围，替换为数字 0，都是没问题的，都可以保证 json 的格式不被破坏。

  4. 由于已经去掉了所有的子 json 数组，之后就可以按照传统的方式，将 },{ 替换为 }我是分隔符号{。

    sql 复制代码 
    with table1 as (
    select '{"score":1,"submitTime":1712491933,"answerFlag":1,"groupId":1755547960,"answers":[{"value":"[1, 2, 3]","ids":[4,5,6],"duration":22314,"status":1,"tid":1},{"value":"aabbcc","lessons":[44,55,66],"isPic":0,"duration":22314,"status":2,"tid":2}],"questionType":65}' as col
)

select regexp_replace(a.answers, '\\}\\,\\{', '\\}我是分隔符号{') as answers
from (
    select regexp_replace(answers, '\\[.*\\]', '0') as answers
    from (
        select substring(answers, 2, length(answers) - 2) as answers
        from (
            select get_json_object(col, '$.answers') as answers
            from table1
            ) as a
        ) as a
    ) as a
    answers
    {"value":"0","ids":0,"isPic":0,"duration":22314,"status":1,"tid":1}我是分隔符号{"value":"aabbcc","lessons":0,"isPic":0,"duration":22314,"status":2,"tid":2}

  5. 通过 我是分隔符号 切换字符串，再通过 lateral view explode() 语法将 json 数组展开，最后通过 get_json_object() 函数，获取需要的值即可。

    sql 复制代码 
    with table1 as (
    select '{"score":1,"submitTime":1712491933,"answerFlag":1,"groupId":1755547960,"answers":[{"value":"[1, 2, 3]","ids":[4,5,6],"duration":22314,"status":1,"tid":1},{"value":"aabbcc","lessons":[44,55,66],"isPic":0,"duration":22314,"status":2,"tid":2}],"questionType":65}' as col
)

select get_json_object(b.answer, '$.tid') as tid, get_json_object(b.answer, '$.status') as status
from (
    select regexp_replace(a.answers, '\\}\\,\\{', '\\}我是分隔符号{') as answers
    from (
        select regexp_replace(answers, '\\[.*\\]', '0') as answers
        from (
            select substring(answers, 2, length(answers) - 2) as answers
            from (
                select get_json_object(col, '$.answers') as answers
                from table1
                ) as a
            ) as a
        ) as a
    ) as a
lateral view explode(split(regexp_replace(a.answers, '\\}\\,\\{', '\\}我是分隔符号{'), '我是分隔符号')) b as answer

    结果如下，可以看到，数据处理结果符合预期

    tid status
    1 1
    2 2

3.2 注意事项

上面的步骤 3，只适用于你想要展开最外层的这个 json 数组，并且完全不需要内部嵌套的子 json 数组才行，否则将内部的子 json 数组全部替换为数字 0 之后，你就获取不到子 json 数组数据了。

如果还想要内部的子 json 数组，单纯的 sql 应该是实现不了的，需要去自定义 udf，然后通过 java 代码一层一层解析了。

相关推荐
Edingbrugh.南空13 小时前
Apache Iceberg与Hive集成：分区表篇
大数据·hive·hadoop
Cachel wood14 小时前
后端开发：计算机网络、数据库常识
android·大数据·数据库·数据仓库·sql·计算机网络·mysql
Lx35216 小时前
SQL参数化查询：防注入与计划缓存的双重优势
后端·sql·mysql
BillKu17 小时前
sql中like and not like的优化
数据库·sql
kooboo china.1 天前
JSON 编辑器：从语法到数据处理（二）
编辑器·json
夜光小兔纸1 天前
SQL Server 查询数据库中所有表中所有字段的数据类型及长度
数据库·sql·sql server
ifanatic1 天前
[每周一更]-(第147期)：使用 Go 语言实现 JSON Web Token (JWT)
前端·golang·json
Humbunklung1 天前
JavaScript 将一个带K-V特征的JSON数组转换为JSON对象
开发语言·javascript·json
vvilkim1 天前
Flutter JSON解析全攻略：使用json_serializable实现高效序列化
flutter·json
AAA建材批发王师傅1 天前
Hive 序列化与反序列化：数据的 “打包“ 与 “拆箱“ 艺术
数据仓库·hive·hadoop
热门推荐
01Coze扣子平台完整体验和实践（附国内和国际版对比）02DeepSeek各版本说明与优缺点分析03KGG转MP3工具|非KGM文件|解密音频04扣子（coze）实战|我用扣子搭建了一个自动分析小红薯笔记内容的AI应用|详细步骤拆解05零代码入门 | Coze——让大模型接入自己的数据库06从零安装 LLaMA-Factory 微调 Qwen 大模型成功及所有的坑07电工电子课程设计---四路病房呼叫系统08YOLOv8入门 | 重要性能衡量指标、训练结果评价及分析及影响mAP的因素【发论文关注的指标】09【SpeedAI科研小助手】2分钟极速解决知网维普重复率、AIGC率过高，一键全文降！文件格式不变，公式都保留的！10AI Agent | Coze 插件使用指南：从功能解析到实操步骤