union all 优化案例

小至尖尖2024-03-17 0:32

遇到个子查询嵌套 UNION ALL 的SQL语句很慢，谓词过滤条件不能内推进去，需要优化这段 UNION ALL这块的内容。

UNION ALL 慢SQL：

复制代码 
    SELECT * FROM ((SELECT 
                                         jfw03.id            AS newid,
                                         jfw03.emc_person_id AS bizPersonOid,
                                         jfw03.jfw03001      AS jfw03001,
                                         jfw03.jfw03002      AS jfw03002,
                                         jfw03.jfw03003      AS jfw03003,
                                         jfw03.jfw03004      AS jfw03004,
                                         jfw03.jfw03005      AS jfw03005,
                                         jfw03.jfw03006      AS jfw03006,
                                         jfw03.jfw03007      AS jfw03007,
                                         jfw03.remark        AS remark,
                                         old03.id            AS oldId,
                                         old03.jfw03001      AS jfw03001pre,
                                         old03.jfw03002      AS jfw03002pre,
                                         old03.jfw03003      AS jfw03003pre,
                                         old03.jfw03004      AS jfw03004pre,
                                         old03.jfw03005      AS jfw03005pre,
                                         old03.jfw03006      AS jfw03006pre,
                                         old03.jfw03007      AS jfw03007pre,
                                         old03.remark        AS remarkpre
                                    FROM emc jfw03
                                    LEFT JOIN emc old03
                                      ON jfw03.emc_person_id =
                                         old03.emc_person_id
                                   WHERE jfw03.jfw03001 = old03.jfw03001
                                     AND jfw03.handle_mark = '2'
                                     AND old03.handle_mark IN ('1', '3'))
                      UNION ALL (SELECT jfw03.id            AS newid,
                                       jfw03.emc_person_id AS bizPersonOid,
                                       jfw03.jfw03001,
                                       jfw03.jfw03002,
                                       jfw03.jfw03003,
                                       jfw03.jfw03004,
                                       jfw03.jfw03005,
                                       jfw03.jfw03006,
                                       jfw03.jfw03007,
                                       jfw03.remark,
                                       NULL                AS oldId,
                                       NULL                AS jfw03001pre,
                                       NULL                AS jfw03002pre,
                                       NULL                AS jfw03003pre,
                                       NULL                AS jfw03004pre,
                                       NULL                AS jfw03005pre,
                                       NULL                AS jfw03006pre,
                                       NULL                AS jfw03007pre,
                                       NULL                AS remarkpre
                                  FROM emc jfw03
                                 WHERE jfw03.handle_mark = '2'
                                   AND NOT EXISTS
                                 (SELECT 1
                                          FROM emc old03
                                         WHERE jfw03.emc_person_id =
                                               old03.emc_person_id
                                           AND jfw03.jfw03001 = old03.jfw03001
                                           AND old03.handle_mark IN ('1', '3')))
                      UNION ALL (SELECT NULL                AS newid,
                                       old03.emc_person_id AS bizPersonOid,
                                       NULL                AS jfw03001,
                                       NULL                AS jfw03002,
                                       NULL                AS jfw03003,
                                       NULL                AS jfw03004,
                                       NULL                AS jfw03005,
                                       NULL                AS jfw03006,
                                       NULL                AS jfw03007,
                                       NULL                AS remark,
                                       old03.id            AS oldId,
                                       old03.jfw03001      AS jfw03001pre,
                                       old03.jfw03002      AS jfw03002pre,
                                       old03.jfw03003      AS jfw03003pre,
                                       old03.jfw03004      AS jfw03004pre,
                                       old03.jfw03005      AS jfw03005pre,
                                       old03.jfw03006      AS jfw03006pre,
                                       old03.jfw03007      AS jfw03007pre,
                                       old03.remark        AS remarkpre
                                  FROM emc old03
                                 WHERE old03.handle_mark IN ('1', '3')
                                   AND NOT EXISTS
                                 (SELECT 1
                                          FROM emc jfw03
                                         WHERE jfw03.emc_person_id =
                                               old03.emc_person_id
                                           AND jfw03.jfw03001 = old03.jfw03001
                                           AND jfw03.handle_mark = '2'))) jfw03;

执行计划：

复制代码 
                                                                     QUERY PLAN                                                                     
----------------------------------------------------------------------------------------------------------------------------------------------------
 Append  (cost=0.69..7253737.08 rows=49711 width=274) (actual time=0.252..60906.535 rows=3291407 loops=1)
   ->  Nested Loop  (cost=0.69..2753734.17 rows=49709 width=274) (actual time=0.251..16535.465 rows=49558 loops=1)
         ->  Seq Scan on emc jfw03  (cost=0.00..347258.58 rows=1132320 width=139) (actual time=0.010..5103.042 rows=1114280 loops=1)
               Filter: (handle_mark = '2'::bpchar)
               Rows Removed by Filter: 8885720
         ->  Index Scan using idx_1_2_3_emc on emc old03  (cost=0.69..2.12 rows=1 width=139) (actual time=0.010..0.010 rows=0 loops=1114280)
               Index Cond: ((jfw03001 = jfw03.jfw03001) AND (emc_person_id = jfw03.emc_person_id))
               Filter: (handle_mark = ANY ('{1,3}'::bpchar[]))
               Rows Removed by Filter: 1
   ->  Nested Loop Anti Join  (cost=0.69..1864694.18 rows=1 width=399) (actual time=0.032..16606.236 rows=1065808 loops=1)
         ->  Seq Scan on emc jfw03_1  (cost=0.00..347258.58 rows=1132320 width=139) (actual time=0.013..5171.111 rows=1114280 loops=1)
               Filter: (handle_mark = '2'::bpchar)
               Rows Removed by Filter: 8885720
         ->  Index Only Scan using idx_1_2_3_emc on emc old03_1  (cost=0.69..1.33 rows=1 width=105) (actual time=0.009..0.009 rows=0 loops=1114280)
               Index Cond: ((jfw03001 = jfw03_1.jfw03001) AND (emc_person_id = jfw03_1.emc_person_id))
               Filter: (handle_mark = ANY ('{1,3}'::bpchar[]))
               Rows Removed by Filter: 1
               Heap Fetches: 0
   ->  Nested Loop Anti Join  (cost=0.69..2634563.07 rows=1 width=399) (actual time=0.033..27638.050 rows=2176041 loops=1)
         ->  Seq Scan on emc old03_2  (cost=0.00..347258.58 rows=2205308 width=139) (actual time=0.014..5510.884 rows=2225077 loops=1)
               Filter: (handle_mark = ANY ('{1,3}'::bpchar[]))
               Rows Removed by Filter: 7774923
         ->  Index Only Scan using idx_1_2_3_emc on emc jfw03_2  (cost=0.69..1.03 rows=1 width=105) (actual time=0.009..0.009 rows=0 loops=2225077)
               Index Cond: ((jfw03001 = old03_2.jfw03001) AND (emc_person_id = old03_2.emc_person_id) AND (handle_mark = '2'::bpchar))
               Heap Fetches: 0
 Planning Time: 0.318 ms
 Execution Time: 60987.916 ms
(27 行记录)

加个索引看看速度

复制代码 
CREATE INDEX idx_emc_1 ON emc ( handle_mark );

加索引后执行计划

复制代码 
                                                                     QUERY PLAN                                                                     
----------------------------------------------------------------------------------------------------------------------------------------------------
 Append  (cost=21059.67..7011218.65 rows=49969 width=278) (actual time=115.683..46578.675 rows=3291407 loops=1)
   ->  Nested Loop  (cost=21059.67..2655195.15 rows=49967 width=278) (actual time=115.682..11939.294 rows=49558 loops=1)
         ->  Bitmap Heap Scan on emc jfw03  (cost=21058.98..257323.10 rows=1120329 width=141) (actual time=115.458..417.906 rows=1114280 loops=1)
               Recheck Cond: (handle_mark = '2'::bpchar)
               Heap Blocks: exact=221151
               ->  Bitmap Index Scan on idx_emc_1  (cost=0.00..20778.90 rows=1120329 width=0) (actual time=92.887..92.887 rows=1114280 loops=1)
                     Index Cond: (handle_mark = '2'::bpchar)
         ->  Index Scan using idx_1_2_3_emc on emc old03  (cost=0.69..2.13 rows=1 width=141) (actual time=0.010..0.010 rows=0 loops=1114280)
               Index Cond: ((jfw03001 = jfw03.jfw03001) AND (emc_person_id = jfw03.emc_person_id))
               Filter: (handle_mark = ANY ('{1,3}'::bpchar[]))
               Rows Removed by Filter: 1
   ->  Nested Loop Anti Join  (cost=21059.67..1766155.15 rows=1 width=401) (actual time=127.721..11690.213 rows=1065808 loops=1)
         ->  Bitmap Heap Scan on emc jfw03_1  (cost=21058.98..257323.10 rows=1120329 width=141) (actual time=127.640..423.128 rows=1114280 loops=1)
               Recheck Cond: (handle_mark = '2'::bpchar)
               Heap Blocks: exact=221151
               ->  Bitmap Index Scan on idx_emc_1  (cost=0.00..20778.90 rows=1120329 width=0) (actual time=105.117..105.117 rows=1114280 loops=1)
                     Index Cond: (handle_mark = '2'::bpchar)
         ->  Index Only Scan using idx_1_2_3_emc on emc old03_1  (cost=0.69..1.34 rows=1 width=105) (actual time=0.009..0.009 rows=0 loops=1114280)
               Index Cond: ((jfw03001 = jfw03_1.jfw03001) AND (emc_person_id = jfw03_1.emc_person_id))
               Filter: (handle_mark = ANY ('{1,3}'::bpchar[]))
               Rows Removed by Filter: 1
               Heap Fetches: 0
   ->  Nested Loop Anti Join  (cost=39323.15..2589118.81 rows=1 width=401) (actual time=201.187..22828.908 rows=2176041 loops=1)
         ->  Bitmap Heap Scan on emc old03_2  (cost=39322.47..289365.69 rows=2222658 width=141) (actual time=201.135..603.170 rows=2225077 loops=1)
               Recheck Cond: (handle_mark = ANY ('{1,3}'::bpchar[]))
               Heap Blocks: exact=222258
               ->  Bitmap Index Scan on idx_emc_1  (cost=0.00..38766.81 rows=2222658 width=0) (actual time=179.770..179.770 rows=2225077 loops=1)
                     Index Cond: (handle_mark = ANY ('{1,3}'::bpchar[]))
         ->  Index Only Scan using idx_1_2_3_emc on emc jfw03_2  (cost=0.69..1.02 rows=1 width=105) (actual time=0.009..0.009 rows=0 loops=2225077)
               Index Cond: ((jfw03001 = old03_2.jfw03001) AND (emc_person_id = old03_2.emc_person_id) AND (handle_mark = '2'::bpchar))
               Heap Fetches: 0
 Planning Time: 0.373 ms
 Execution Time: 46666.296 ms
(33 行记录)

快了14秒，没达到目的效果，无解，只能改写SQL，可以改 FULL JOIN。

等价改写：

复制代码 
SELECT * FROM (
with x as (SELECT * FROM emc where handle_mark IN ('1', '3','2'))
    SELECT 
    jfw03.id                                          AS newid,
    jfw03.emc_person_id                                  AS bizPersonOid,
    jfw03.jfw03001                                       AS jfw03001,
    jfw03.jfw03002                                       AS jfw03002,
    jfw03.jfw03003                                       AS jfw03003,
    jfw03.jfw03004                                       AS jfw03004,
    jfw03.jfw03005                                       AS jfw03005,
    jfw03.jfw03006                                       AS jfw03006,
    jfw03.jfw03007                                       AS jfw03007,
    jfw03.remark                                         AS remark,
    old03.id                                             AS oldId,
    old03.jfw03001                                       AS jfw03001pre,
    old03.jfw03002                                       AS jfw03002pre,
    old03.jfw03003                                       AS jfw03003pre,
    old03.jfw03004                                       AS jfw03004pre,
    old03.jfw03005                                       AS jfw03005pre,
    old03.jfw03006                                       AS jfw03006pre,
    old03.jfw03007                                       AS jfw03007pre,
    old03.remark                                         AS remarkpre
    FROM (SELECT * FROM x WHERE handle_mark IN ('2')) jfw03 
        full join (SELECT * FROM x WHERE handle_mark IN ('1', '3')) old03 
        ON jfw03.emc_person_id = old03.emc_person_id AND jfw03.jfw03001 = old03.jfw03001);

等价改写计划：

复制代码 
                                           
                                                                  QUERY PLAN                                                                  
----------------------------------------------------------------------------------------------------------------------------------------------
 Merge Full Join  (cost=10000479890.42..10000480406.20 rows=33430 width=1844) (actual time=9433.530..10954.254 rows=3291407 loops=1)
   Merge Cond: ((x.emc_person_id = x_1.emc_person_id) AND (x.jfw03001 = x_1.jfw03001))
   CTE x
     ->  Bitmap Heap Scan on emc  (cost=57545.45..325771.51 rows=3342986 width=145) (actual time=268.766..721.294 rows=3339357 loops=1)
           Recheck Cond: (handle_mark = ANY ('{1,3,2}'::bpchar[]))
           Heap Blocks: exact=222260
           ->  Bitmap Index Scan on idx_emc_1  (cost=0.00..56709.71 rows=3342986 width=0) (actual time=246.952..246.953 rows=3339357 loops=1)
                 Index Cond: (handle_mark = ANY ('{1,3,2}'::bpchar[]))
   ->  Sort  (cost=76389.65..76431.43 rows=16715 width=924) (actual time=3818.370..4021.726 rows=1114280 loops=1)
         Sort Key: x.emc_person_id, x.jfw03001
         Sort Method: quicksort  Memory: 345133kB
         ->  CTE Scan on x  (cost=0.00..75217.18 rows=16715 width=924) (actual time=268.780..1391.594 rows=1114280 loops=1)
               Filter: (handle_mark = '2'::bpchar)
               Rows Removed by Filter: 2225077
   ->  Sort  (cost=77729.26..77812.83 rows=33430 width=924) (actual time=5615.144..5949.562 rows=2225599 loops=1)
         Sort Key: x_1.emc_person_id, x_1.jfw03001
         Sort Method: quicksort  Memory: 676056kB
         ->  CTE Scan on x x_1  (cost=0.00..75217.18 rows=33430 width=924) (actual time=0.007..360.353 rows=2225077 loops=1)
               Filter: (handle_mark = ANY ('{1,3}'::bpchar[]))
               Rows Removed by Filter: 1114280
 Planning Time: 0.138 ms
 Execution Time: 11101.829 ms
(22 行记录)

改写后11s就能跑出结果，符合性能优化目标。

**　　这条改写SQL留个坑，不完全等价，完全等价的没放出来，有兴趣的同学可以思考下，自己做做实验。**

相关推荐
Hacker_LaoYi2 小时前
SQL注入的那些面试题总结
数据库·sql
Hacker_LaoYi4 小时前
【渗透技术总结】SQL手工注入总结
数据库·sql
独行soc4 小时前
#渗透测试#漏洞挖掘#红蓝攻防#护网#sql注入介绍06-基于子查询的SQL注入（Subquery-Based SQL Injection）
数据库·sql·安全·web安全·漏洞挖掘·hw
独行soc6 小时前
#渗透测试#漏洞挖掘#红蓝攻防#护网#sql注入介绍08-基于时间延迟的SQL注入（Time-Based SQL Injection）
数据库·sql·安全·渗透测试·漏洞挖掘
清平乐的技术专栏6 小时前
Hive SQL 查询所有函数
hive·hadoop·sql
cmdch201711 小时前
Mybatis加密解密查询操作（sql前），where要传入加密后的字段时遇到的问题
数据库·sql·mybatis
程序员学习随笔11 小时前
PostgreSQL技术内幕21：SysLogger日志收集器的工作原理
数据库·postgresql
秦时明月之君临天下11 小时前
PostgreSQL标识符长度限制不能超过63字节
数据库·postgresql
程序猿小柒11 小时前
【Spark】Spark SQL执行计划-精简版
大数据·sql·spark
听见~13 小时前
SQL优化
数据库·sql
热门推荐
01〔AI 绘画〕Stable Diffusion 之 解决绘制多人或面部很小的人物时面部崩坏问题 篇02ARM学习（31）编译器对overlay方式的支持03玄机平台应急响应—webshell查杀04DisplayPort AltMode On Type-c协议_Ver2.0(学习笔记)05校验 GPT-4 真实性的三个经典问题：快速区分 GPT-3.5 与 GPT-4，并提供免费测试网站06Docker 夺命连环 15 问076个主流的工业3D管道设计软件08基于大语言模型（LLM）的合成数据生成、策展和评估的综述09(欧拉)openEuler系统添加网卡文件配置流程、(欧拉)openEuler系统手动配置ipv6地址流程、(欧拉)openEuler系统网络管理说明10树莓派 5 AI 套件（Hailo-8L）使用教程