union all 优化案例

小至尖尖2024-03-17 0:32

遇到个子查询嵌套 UNION ALL 的SQL语句很慢，谓词过滤条件不能内推进去，需要优化这段 UNION ALL这块的内容。

UNION ALL 慢SQL：

复制代码 
    SELECT * FROM ((SELECT 
                                         jfw03.id            AS newid,
                                         jfw03.emc_person_id AS bizPersonOid,
                                         jfw03.jfw03001      AS jfw03001,
                                         jfw03.jfw03002      AS jfw03002,
                                         jfw03.jfw03003      AS jfw03003,
                                         jfw03.jfw03004      AS jfw03004,
                                         jfw03.jfw03005      AS jfw03005,
                                         jfw03.jfw03006      AS jfw03006,
                                         jfw03.jfw03007      AS jfw03007,
                                         jfw03.remark        AS remark,
                                         old03.id            AS oldId,
                                         old03.jfw03001      AS jfw03001pre,
                                         old03.jfw03002      AS jfw03002pre,
                                         old03.jfw03003      AS jfw03003pre,
                                         old03.jfw03004      AS jfw03004pre,
                                         old03.jfw03005      AS jfw03005pre,
                                         old03.jfw03006      AS jfw03006pre,
                                         old03.jfw03007      AS jfw03007pre,
                                         old03.remark        AS remarkpre
                                    FROM emc jfw03
                                    LEFT JOIN emc old03
                                      ON jfw03.emc_person_id =
                                         old03.emc_person_id
                                   WHERE jfw03.jfw03001 = old03.jfw03001
                                     AND jfw03.handle_mark = '2'
                                     AND old03.handle_mark IN ('1', '3'))
                      UNION ALL (SELECT jfw03.id            AS newid,
                                       jfw03.emc_person_id AS bizPersonOid,
                                       jfw03.jfw03001,
                                       jfw03.jfw03002,
                                       jfw03.jfw03003,
                                       jfw03.jfw03004,
                                       jfw03.jfw03005,
                                       jfw03.jfw03006,
                                       jfw03.jfw03007,
                                       jfw03.remark,
                                       NULL                AS oldId,
                                       NULL                AS jfw03001pre,
                                       NULL                AS jfw03002pre,
                                       NULL                AS jfw03003pre,
                                       NULL                AS jfw03004pre,
                                       NULL                AS jfw03005pre,
                                       NULL                AS jfw03006pre,
                                       NULL                AS jfw03007pre,
                                       NULL                AS remarkpre
                                  FROM emc jfw03
                                 WHERE jfw03.handle_mark = '2'
                                   AND NOT EXISTS
                                 (SELECT 1
                                          FROM emc old03
                                         WHERE jfw03.emc_person_id =
                                               old03.emc_person_id
                                           AND jfw03.jfw03001 = old03.jfw03001
                                           AND old03.handle_mark IN ('1', '3')))
                      UNION ALL (SELECT NULL                AS newid,
                                       old03.emc_person_id AS bizPersonOid,
                                       NULL                AS jfw03001,
                                       NULL                AS jfw03002,
                                       NULL                AS jfw03003,
                                       NULL                AS jfw03004,
                                       NULL                AS jfw03005,
                                       NULL                AS jfw03006,
                                       NULL                AS jfw03007,
                                       NULL                AS remark,
                                       old03.id            AS oldId,
                                       old03.jfw03001      AS jfw03001pre,
                                       old03.jfw03002      AS jfw03002pre,
                                       old03.jfw03003      AS jfw03003pre,
                                       old03.jfw03004      AS jfw03004pre,
                                       old03.jfw03005      AS jfw03005pre,
                                       old03.jfw03006      AS jfw03006pre,
                                       old03.jfw03007      AS jfw03007pre,
                                       old03.remark        AS remarkpre
                                  FROM emc old03
                                 WHERE old03.handle_mark IN ('1', '3')
                                   AND NOT EXISTS
                                 (SELECT 1
                                          FROM emc jfw03
                                         WHERE jfw03.emc_person_id =
                                               old03.emc_person_id
                                           AND jfw03.jfw03001 = old03.jfw03001
                                           AND jfw03.handle_mark = '2'))) jfw03;

执行计划：

复制代码 
                                                                     QUERY PLAN                                                                     
----------------------------------------------------------------------------------------------------------------------------------------------------
 Append  (cost=0.69..7253737.08 rows=49711 width=274) (actual time=0.252..60906.535 rows=3291407 loops=1)
   ->  Nested Loop  (cost=0.69..2753734.17 rows=49709 width=274) (actual time=0.251..16535.465 rows=49558 loops=1)
         ->  Seq Scan on emc jfw03  (cost=0.00..347258.58 rows=1132320 width=139) (actual time=0.010..5103.042 rows=1114280 loops=1)
               Filter: (handle_mark = '2'::bpchar)
               Rows Removed by Filter: 8885720
         ->  Index Scan using idx_1_2_3_emc on emc old03  (cost=0.69..2.12 rows=1 width=139) (actual time=0.010..0.010 rows=0 loops=1114280)
               Index Cond: ((jfw03001 = jfw03.jfw03001) AND (emc_person_id = jfw03.emc_person_id))
               Filter: (handle_mark = ANY ('{1,3}'::bpchar[]))
               Rows Removed by Filter: 1
   ->  Nested Loop Anti Join  (cost=0.69..1864694.18 rows=1 width=399) (actual time=0.032..16606.236 rows=1065808 loops=1)
         ->  Seq Scan on emc jfw03_1  (cost=0.00..347258.58 rows=1132320 width=139) (actual time=0.013..5171.111 rows=1114280 loops=1)
               Filter: (handle_mark = '2'::bpchar)
               Rows Removed by Filter: 8885720
         ->  Index Only Scan using idx_1_2_3_emc on emc old03_1  (cost=0.69..1.33 rows=1 width=105) (actual time=0.009..0.009 rows=0 loops=1114280)
               Index Cond: ((jfw03001 = jfw03_1.jfw03001) AND (emc_person_id = jfw03_1.emc_person_id))
               Filter: (handle_mark = ANY ('{1,3}'::bpchar[]))
               Rows Removed by Filter: 1
               Heap Fetches: 0
   ->  Nested Loop Anti Join  (cost=0.69..2634563.07 rows=1 width=399) (actual time=0.033..27638.050 rows=2176041 loops=1)
         ->  Seq Scan on emc old03_2  (cost=0.00..347258.58 rows=2205308 width=139) (actual time=0.014..5510.884 rows=2225077 loops=1)
               Filter: (handle_mark = ANY ('{1,3}'::bpchar[]))
               Rows Removed by Filter: 7774923
         ->  Index Only Scan using idx_1_2_3_emc on emc jfw03_2  (cost=0.69..1.03 rows=1 width=105) (actual time=0.009..0.009 rows=0 loops=2225077)
               Index Cond: ((jfw03001 = old03_2.jfw03001) AND (emc_person_id = old03_2.emc_person_id) AND (handle_mark = '2'::bpchar))
               Heap Fetches: 0
 Planning Time: 0.318 ms
 Execution Time: 60987.916 ms
(27 行记录)

加个索引看看速度

复制代码 
CREATE INDEX idx_emc_1 ON emc ( handle_mark );

加索引后执行计划

复制代码 
                                                                     QUERY PLAN                                                                     
----------------------------------------------------------------------------------------------------------------------------------------------------
 Append  (cost=21059.67..7011218.65 rows=49969 width=278) (actual time=115.683..46578.675 rows=3291407 loops=1)
   ->  Nested Loop  (cost=21059.67..2655195.15 rows=49967 width=278) (actual time=115.682..11939.294 rows=49558 loops=1)
         ->  Bitmap Heap Scan on emc jfw03  (cost=21058.98..257323.10 rows=1120329 width=141) (actual time=115.458..417.906 rows=1114280 loops=1)
               Recheck Cond: (handle_mark = '2'::bpchar)
               Heap Blocks: exact=221151
               ->  Bitmap Index Scan on idx_emc_1  (cost=0.00..20778.90 rows=1120329 width=0) (actual time=92.887..92.887 rows=1114280 loops=1)
                     Index Cond: (handle_mark = '2'::bpchar)
         ->  Index Scan using idx_1_2_3_emc on emc old03  (cost=0.69..2.13 rows=1 width=141) (actual time=0.010..0.010 rows=0 loops=1114280)
               Index Cond: ((jfw03001 = jfw03.jfw03001) AND (emc_person_id = jfw03.emc_person_id))
               Filter: (handle_mark = ANY ('{1,3}'::bpchar[]))
               Rows Removed by Filter: 1
   ->  Nested Loop Anti Join  (cost=21059.67..1766155.15 rows=1 width=401) (actual time=127.721..11690.213 rows=1065808 loops=1)
         ->  Bitmap Heap Scan on emc jfw03_1  (cost=21058.98..257323.10 rows=1120329 width=141) (actual time=127.640..423.128 rows=1114280 loops=1)
               Recheck Cond: (handle_mark = '2'::bpchar)
               Heap Blocks: exact=221151
               ->  Bitmap Index Scan on idx_emc_1  (cost=0.00..20778.90 rows=1120329 width=0) (actual time=105.117..105.117 rows=1114280 loops=1)
                     Index Cond: (handle_mark = '2'::bpchar)
         ->  Index Only Scan using idx_1_2_3_emc on emc old03_1  (cost=0.69..1.34 rows=1 width=105) (actual time=0.009..0.009 rows=0 loops=1114280)
               Index Cond: ((jfw03001 = jfw03_1.jfw03001) AND (emc_person_id = jfw03_1.emc_person_id))
               Filter: (handle_mark = ANY ('{1,3}'::bpchar[]))
               Rows Removed by Filter: 1
               Heap Fetches: 0
   ->  Nested Loop Anti Join  (cost=39323.15..2589118.81 rows=1 width=401) (actual time=201.187..22828.908 rows=2176041 loops=1)
         ->  Bitmap Heap Scan on emc old03_2  (cost=39322.47..289365.69 rows=2222658 width=141) (actual time=201.135..603.170 rows=2225077 loops=1)
               Recheck Cond: (handle_mark = ANY ('{1,3}'::bpchar[]))
               Heap Blocks: exact=222258
               ->  Bitmap Index Scan on idx_emc_1  (cost=0.00..38766.81 rows=2222658 width=0) (actual time=179.770..179.770 rows=2225077 loops=1)
                     Index Cond: (handle_mark = ANY ('{1,3}'::bpchar[]))
         ->  Index Only Scan using idx_1_2_3_emc on emc jfw03_2  (cost=0.69..1.02 rows=1 width=105) (actual time=0.009..0.009 rows=0 loops=2225077)
               Index Cond: ((jfw03001 = old03_2.jfw03001) AND (emc_person_id = old03_2.emc_person_id) AND (handle_mark = '2'::bpchar))
               Heap Fetches: 0
 Planning Time: 0.373 ms
 Execution Time: 46666.296 ms
(33 行记录)

快了14秒，没达到目的效果，无解，只能改写SQL，可以改 FULL JOIN。

等价改写：

复制代码 
SELECT * FROM (
with x as (SELECT * FROM emc where handle_mark IN ('1', '3','2'))
    SELECT 
    jfw03.id                                          AS newid,
    jfw03.emc_person_id                                  AS bizPersonOid,
    jfw03.jfw03001                                       AS jfw03001,
    jfw03.jfw03002                                       AS jfw03002,
    jfw03.jfw03003                                       AS jfw03003,
    jfw03.jfw03004                                       AS jfw03004,
    jfw03.jfw03005                                       AS jfw03005,
    jfw03.jfw03006                                       AS jfw03006,
    jfw03.jfw03007                                       AS jfw03007,
    jfw03.remark                                         AS remark,
    old03.id                                             AS oldId,
    old03.jfw03001                                       AS jfw03001pre,
    old03.jfw03002                                       AS jfw03002pre,
    old03.jfw03003                                       AS jfw03003pre,
    old03.jfw03004                                       AS jfw03004pre,
    old03.jfw03005                                       AS jfw03005pre,
    old03.jfw03006                                       AS jfw03006pre,
    old03.jfw03007                                       AS jfw03007pre,
    old03.remark                                         AS remarkpre
    FROM (SELECT * FROM x WHERE handle_mark IN ('2')) jfw03 
        full join (SELECT * FROM x WHERE handle_mark IN ('1', '3')) old03 
        ON jfw03.emc_person_id = old03.emc_person_id AND jfw03.jfw03001 = old03.jfw03001);

等价改写计划：

复制代码 
                                           
                                                                  QUERY PLAN                                                                  
----------------------------------------------------------------------------------------------------------------------------------------------
 Merge Full Join  (cost=10000479890.42..10000480406.20 rows=33430 width=1844) (actual time=9433.530..10954.254 rows=3291407 loops=1)
   Merge Cond: ((x.emc_person_id = x_1.emc_person_id) AND (x.jfw03001 = x_1.jfw03001))
   CTE x
     ->  Bitmap Heap Scan on emc  (cost=57545.45..325771.51 rows=3342986 width=145) (actual time=268.766..721.294 rows=3339357 loops=1)
           Recheck Cond: (handle_mark = ANY ('{1,3,2}'::bpchar[]))
           Heap Blocks: exact=222260
           ->  Bitmap Index Scan on idx_emc_1  (cost=0.00..56709.71 rows=3342986 width=0) (actual time=246.952..246.953 rows=3339357 loops=1)
                 Index Cond: (handle_mark = ANY ('{1,3,2}'::bpchar[]))
   ->  Sort  (cost=76389.65..76431.43 rows=16715 width=924) (actual time=3818.370..4021.726 rows=1114280 loops=1)
         Sort Key: x.emc_person_id, x.jfw03001
         Sort Method: quicksort  Memory: 345133kB
         ->  CTE Scan on x  (cost=0.00..75217.18 rows=16715 width=924) (actual time=268.780..1391.594 rows=1114280 loops=1)
               Filter: (handle_mark = '2'::bpchar)
               Rows Removed by Filter: 2225077
   ->  Sort  (cost=77729.26..77812.83 rows=33430 width=924) (actual time=5615.144..5949.562 rows=2225599 loops=1)
         Sort Key: x_1.emc_person_id, x_1.jfw03001
         Sort Method: quicksort  Memory: 676056kB
         ->  CTE Scan on x x_1  (cost=0.00..75217.18 rows=33430 width=924) (actual time=0.007..360.353 rows=2225077 loops=1)
               Filter: (handle_mark = ANY ('{1,3}'::bpchar[]))
               Rows Removed by Filter: 1114280
 Planning Time: 0.138 ms
 Execution Time: 11101.829 ms
(22 行记录)

改写后11s就能跑出结果，符合性能优化目标。

**　　这条改写SQL留个坑，不完全等价，完全等价的没放出来，有兴趣的同学可以思考下，自己做做实验。**

相关推荐
freejackman17 小时前
【实战二】字段级血缘追踪：拆解 AST 迷局，精准穿透函数链与子查询
大数据·数据库·sql
Peter-OK20 小时前
PostgreSQL学习笔记（入门-进阶-精通说明）
笔记·学习·postgresql
骑着蜗牛撵大象3272 天前
MySQL 8.0 迁移后 SQL 变慢：执行计划里 3 类被忽略的成本差异
android·数据库·sql·mysql·优化器·执行计划
zhengfei6112 天前
Go-fastdfs /reload 未授权访问： 有效PoC
数据库·sql
大维2 天前
DBX 实测：从 SQLite 查询到 MiniMax-M3 自然语言生成 SQL
数据库·sql·sqlite
景煜拾年2 天前
InnoAI SQL 助手
运维·数据库·笔记·sql
Decay__3 天前
22_MySQL 运维自动化平台 —— 部署编排 + 监控 + 巡检 + 告警 + 工单闭环一体化
运维·数据库·sql·mysql·架构
FungLeo3 天前
成为全栈·基础补充·数据库索引：为什么加了索引还是慢
mysql·postgresql·sqlite·sql优化·数据库索引·成为全栈
Decay__3 天前
21_MySQL 优化案例集 —— 慢 SQL 优化、索引重构与参数调整收益对比
运维·数据库·sql·mysql·架构
挖掘狂人3 天前
MySQL 深分页优化：limit 100000, 10 慢在哪，怎么救
后端·sql·mysql
热门推荐
01如何新建文件夹？ 电脑新建文件夹的4种方法02国内可直接用、免费额度/永久免费的大模型API清单（含 SiliconFlow、火山、阿里、智谱、百度、Kimi、DeepSeek、DMXAPI 等）03历年考研数学一、数学二、数学三真题试卷及答案PDF04几个好用的ip纯净度检测网站05GitHub 镜像站点06苹果iPA游戏软件资源下载网站07微信历史版本含下载地址（ Windows PC | 安卓 | MAC ）及设置微信不更新08蜘蛛磁力 搜索引擎大全，如何使用蜘蛛磁力查找磁力链接09昇腾910B系列芯片完全指南：四款型号对比、服务器选型与大模型部署实战10Dify：一站式AI应用开发平台全解析与Windows部署指南