union all 优化案例

小至尖尖2024-03-17 0:32

遇到个子查询嵌套 UNION ALL 的SQL语句很慢，谓词过滤条件不能内推进去，需要优化这段 UNION ALL这块的内容。

UNION ALL 慢SQL：

复制代码 
    SELECT * FROM ((SELECT 
                                         jfw03.id            AS newid,
                                         jfw03.emc_person_id AS bizPersonOid,
                                         jfw03.jfw03001      AS jfw03001,
                                         jfw03.jfw03002      AS jfw03002,
                                         jfw03.jfw03003      AS jfw03003,
                                         jfw03.jfw03004      AS jfw03004,
                                         jfw03.jfw03005      AS jfw03005,
                                         jfw03.jfw03006      AS jfw03006,
                                         jfw03.jfw03007      AS jfw03007,
                                         jfw03.remark        AS remark,
                                         old03.id            AS oldId,
                                         old03.jfw03001      AS jfw03001pre,
                                         old03.jfw03002      AS jfw03002pre,
                                         old03.jfw03003      AS jfw03003pre,
                                         old03.jfw03004      AS jfw03004pre,
                                         old03.jfw03005      AS jfw03005pre,
                                         old03.jfw03006      AS jfw03006pre,
                                         old03.jfw03007      AS jfw03007pre,
                                         old03.remark        AS remarkpre
                                    FROM emc jfw03
                                    LEFT JOIN emc old03
                                      ON jfw03.emc_person_id =
                                         old03.emc_person_id
                                   WHERE jfw03.jfw03001 = old03.jfw03001
                                     AND jfw03.handle_mark = '2'
                                     AND old03.handle_mark IN ('1', '3'))
                      UNION ALL (SELECT jfw03.id            AS newid,
                                       jfw03.emc_person_id AS bizPersonOid,
                                       jfw03.jfw03001,
                                       jfw03.jfw03002,
                                       jfw03.jfw03003,
                                       jfw03.jfw03004,
                                       jfw03.jfw03005,
                                       jfw03.jfw03006,
                                       jfw03.jfw03007,
                                       jfw03.remark,
                                       NULL                AS oldId,
                                       NULL                AS jfw03001pre,
                                       NULL                AS jfw03002pre,
                                       NULL                AS jfw03003pre,
                                       NULL                AS jfw03004pre,
                                       NULL                AS jfw03005pre,
                                       NULL                AS jfw03006pre,
                                       NULL                AS jfw03007pre,
                                       NULL                AS remarkpre
                                  FROM emc jfw03
                                 WHERE jfw03.handle_mark = '2'
                                   AND NOT EXISTS
                                 (SELECT 1
                                          FROM emc old03
                                         WHERE jfw03.emc_person_id =
                                               old03.emc_person_id
                                           AND jfw03.jfw03001 = old03.jfw03001
                                           AND old03.handle_mark IN ('1', '3')))
                      UNION ALL (SELECT NULL                AS newid,
                                       old03.emc_person_id AS bizPersonOid,
                                       NULL                AS jfw03001,
                                       NULL                AS jfw03002,
                                       NULL                AS jfw03003,
                                       NULL                AS jfw03004,
                                       NULL                AS jfw03005,
                                       NULL                AS jfw03006,
                                       NULL                AS jfw03007,
                                       NULL                AS remark,
                                       old03.id            AS oldId,
                                       old03.jfw03001      AS jfw03001pre,
                                       old03.jfw03002      AS jfw03002pre,
                                       old03.jfw03003      AS jfw03003pre,
                                       old03.jfw03004      AS jfw03004pre,
                                       old03.jfw03005      AS jfw03005pre,
                                       old03.jfw03006      AS jfw03006pre,
                                       old03.jfw03007      AS jfw03007pre,
                                       old03.remark        AS remarkpre
                                  FROM emc old03
                                 WHERE old03.handle_mark IN ('1', '3')
                                   AND NOT EXISTS
                                 (SELECT 1
                                          FROM emc jfw03
                                         WHERE jfw03.emc_person_id =
                                               old03.emc_person_id
                                           AND jfw03.jfw03001 = old03.jfw03001
                                           AND jfw03.handle_mark = '2'))) jfw03;

执行计划：

复制代码 
                                                                     QUERY PLAN                                                                     
----------------------------------------------------------------------------------------------------------------------------------------------------
 Append  (cost=0.69..7253737.08 rows=49711 width=274) (actual time=0.252..60906.535 rows=3291407 loops=1)
   ->  Nested Loop  (cost=0.69..2753734.17 rows=49709 width=274) (actual time=0.251..16535.465 rows=49558 loops=1)
         ->  Seq Scan on emc jfw03  (cost=0.00..347258.58 rows=1132320 width=139) (actual time=0.010..5103.042 rows=1114280 loops=1)
               Filter: (handle_mark = '2'::bpchar)
               Rows Removed by Filter: 8885720
         ->  Index Scan using idx_1_2_3_emc on emc old03  (cost=0.69..2.12 rows=1 width=139) (actual time=0.010..0.010 rows=0 loops=1114280)
               Index Cond: ((jfw03001 = jfw03.jfw03001) AND (emc_person_id = jfw03.emc_person_id))
               Filter: (handle_mark = ANY ('{1,3}'::bpchar[]))
               Rows Removed by Filter: 1
   ->  Nested Loop Anti Join  (cost=0.69..1864694.18 rows=1 width=399) (actual time=0.032..16606.236 rows=1065808 loops=1)
         ->  Seq Scan on emc jfw03_1  (cost=0.00..347258.58 rows=1132320 width=139) (actual time=0.013..5171.111 rows=1114280 loops=1)
               Filter: (handle_mark = '2'::bpchar)
               Rows Removed by Filter: 8885720
         ->  Index Only Scan using idx_1_2_3_emc on emc old03_1  (cost=0.69..1.33 rows=1 width=105) (actual time=0.009..0.009 rows=0 loops=1114280)
               Index Cond: ((jfw03001 = jfw03_1.jfw03001) AND (emc_person_id = jfw03_1.emc_person_id))
               Filter: (handle_mark = ANY ('{1,3}'::bpchar[]))
               Rows Removed by Filter: 1
               Heap Fetches: 0
   ->  Nested Loop Anti Join  (cost=0.69..2634563.07 rows=1 width=399) (actual time=0.033..27638.050 rows=2176041 loops=1)
         ->  Seq Scan on emc old03_2  (cost=0.00..347258.58 rows=2205308 width=139) (actual time=0.014..5510.884 rows=2225077 loops=1)
               Filter: (handle_mark = ANY ('{1,3}'::bpchar[]))
               Rows Removed by Filter: 7774923
         ->  Index Only Scan using idx_1_2_3_emc on emc jfw03_2  (cost=0.69..1.03 rows=1 width=105) (actual time=0.009..0.009 rows=0 loops=2225077)
               Index Cond: ((jfw03001 = old03_2.jfw03001) AND (emc_person_id = old03_2.emc_person_id) AND (handle_mark = '2'::bpchar))
               Heap Fetches: 0
 Planning Time: 0.318 ms
 Execution Time: 60987.916 ms
(27 行记录)

加个索引看看速度

复制代码 
CREATE INDEX idx_emc_1 ON emc ( handle_mark );

加索引后执行计划

复制代码 
                                                                     QUERY PLAN                                                                     
----------------------------------------------------------------------------------------------------------------------------------------------------
 Append  (cost=21059.67..7011218.65 rows=49969 width=278) (actual time=115.683..46578.675 rows=3291407 loops=1)
   ->  Nested Loop  (cost=21059.67..2655195.15 rows=49967 width=278) (actual time=115.682..11939.294 rows=49558 loops=1)
         ->  Bitmap Heap Scan on emc jfw03  (cost=21058.98..257323.10 rows=1120329 width=141) (actual time=115.458..417.906 rows=1114280 loops=1)
               Recheck Cond: (handle_mark = '2'::bpchar)
               Heap Blocks: exact=221151
               ->  Bitmap Index Scan on idx_emc_1  (cost=0.00..20778.90 rows=1120329 width=0) (actual time=92.887..92.887 rows=1114280 loops=1)
                     Index Cond: (handle_mark = '2'::bpchar)
         ->  Index Scan using idx_1_2_3_emc on emc old03  (cost=0.69..2.13 rows=1 width=141) (actual time=0.010..0.010 rows=0 loops=1114280)
               Index Cond: ((jfw03001 = jfw03.jfw03001) AND (emc_person_id = jfw03.emc_person_id))
               Filter: (handle_mark = ANY ('{1,3}'::bpchar[]))
               Rows Removed by Filter: 1
   ->  Nested Loop Anti Join  (cost=21059.67..1766155.15 rows=1 width=401) (actual time=127.721..11690.213 rows=1065808 loops=1)
         ->  Bitmap Heap Scan on emc jfw03_1  (cost=21058.98..257323.10 rows=1120329 width=141) (actual time=127.640..423.128 rows=1114280 loops=1)
               Recheck Cond: (handle_mark = '2'::bpchar)
               Heap Blocks: exact=221151
               ->  Bitmap Index Scan on idx_emc_1  (cost=0.00..20778.90 rows=1120329 width=0) (actual time=105.117..105.117 rows=1114280 loops=1)
                     Index Cond: (handle_mark = '2'::bpchar)
         ->  Index Only Scan using idx_1_2_3_emc on emc old03_1  (cost=0.69..1.34 rows=1 width=105) (actual time=0.009..0.009 rows=0 loops=1114280)
               Index Cond: ((jfw03001 = jfw03_1.jfw03001) AND (emc_person_id = jfw03_1.emc_person_id))
               Filter: (handle_mark = ANY ('{1,3}'::bpchar[]))
               Rows Removed by Filter: 1
               Heap Fetches: 0
   ->  Nested Loop Anti Join  (cost=39323.15..2589118.81 rows=1 width=401) (actual time=201.187..22828.908 rows=2176041 loops=1)
         ->  Bitmap Heap Scan on emc old03_2  (cost=39322.47..289365.69 rows=2222658 width=141) (actual time=201.135..603.170 rows=2225077 loops=1)
               Recheck Cond: (handle_mark = ANY ('{1,3}'::bpchar[]))
               Heap Blocks: exact=222258
               ->  Bitmap Index Scan on idx_emc_1  (cost=0.00..38766.81 rows=2222658 width=0) (actual time=179.770..179.770 rows=2225077 loops=1)
                     Index Cond: (handle_mark = ANY ('{1,3}'::bpchar[]))
         ->  Index Only Scan using idx_1_2_3_emc on emc jfw03_2  (cost=0.69..1.02 rows=1 width=105) (actual time=0.009..0.009 rows=0 loops=2225077)
               Index Cond: ((jfw03001 = old03_2.jfw03001) AND (emc_person_id = old03_2.emc_person_id) AND (handle_mark = '2'::bpchar))
               Heap Fetches: 0
 Planning Time: 0.373 ms
 Execution Time: 46666.296 ms
(33 行记录)

快了14秒，没达到目的效果，无解，只能改写SQL，可以改 FULL JOIN。

等价改写：

复制代码 
SELECT * FROM (
with x as (SELECT * FROM emc where handle_mark IN ('1', '3','2'))
    SELECT 
    jfw03.id                                          AS newid,
    jfw03.emc_person_id                                  AS bizPersonOid,
    jfw03.jfw03001                                       AS jfw03001,
    jfw03.jfw03002                                       AS jfw03002,
    jfw03.jfw03003                                       AS jfw03003,
    jfw03.jfw03004                                       AS jfw03004,
    jfw03.jfw03005                                       AS jfw03005,
    jfw03.jfw03006                                       AS jfw03006,
    jfw03.jfw03007                                       AS jfw03007,
    jfw03.remark                                         AS remark,
    old03.id                                             AS oldId,
    old03.jfw03001                                       AS jfw03001pre,
    old03.jfw03002                                       AS jfw03002pre,
    old03.jfw03003                                       AS jfw03003pre,
    old03.jfw03004                                       AS jfw03004pre,
    old03.jfw03005                                       AS jfw03005pre,
    old03.jfw03006                                       AS jfw03006pre,
    old03.jfw03007                                       AS jfw03007pre,
    old03.remark                                         AS remarkpre
    FROM (SELECT * FROM x WHERE handle_mark IN ('2')) jfw03 
        full join (SELECT * FROM x WHERE handle_mark IN ('1', '3')) old03 
        ON jfw03.emc_person_id = old03.emc_person_id AND jfw03.jfw03001 = old03.jfw03001);

等价改写计划：

复制代码 
                                           
                                                                  QUERY PLAN                                                                  
----------------------------------------------------------------------------------------------------------------------------------------------
 Merge Full Join  (cost=10000479890.42..10000480406.20 rows=33430 width=1844) (actual time=9433.530..10954.254 rows=3291407 loops=1)
   Merge Cond: ((x.emc_person_id = x_1.emc_person_id) AND (x.jfw03001 = x_1.jfw03001))
   CTE x
     ->  Bitmap Heap Scan on emc  (cost=57545.45..325771.51 rows=3342986 width=145) (actual time=268.766..721.294 rows=3339357 loops=1)
           Recheck Cond: (handle_mark = ANY ('{1,3,2}'::bpchar[]))
           Heap Blocks: exact=222260
           ->  Bitmap Index Scan on idx_emc_1  (cost=0.00..56709.71 rows=3342986 width=0) (actual time=246.952..246.953 rows=3339357 loops=1)
                 Index Cond: (handle_mark = ANY ('{1,3,2}'::bpchar[]))
   ->  Sort  (cost=76389.65..76431.43 rows=16715 width=924) (actual time=3818.370..4021.726 rows=1114280 loops=1)
         Sort Key: x.emc_person_id, x.jfw03001
         Sort Method: quicksort  Memory: 345133kB
         ->  CTE Scan on x  (cost=0.00..75217.18 rows=16715 width=924) (actual time=268.780..1391.594 rows=1114280 loops=1)
               Filter: (handle_mark = '2'::bpchar)
               Rows Removed by Filter: 2225077
   ->  Sort  (cost=77729.26..77812.83 rows=33430 width=924) (actual time=5615.144..5949.562 rows=2225599 loops=1)
         Sort Key: x_1.emc_person_id, x_1.jfw03001
         Sort Method: quicksort  Memory: 676056kB
         ->  CTE Scan on x x_1  (cost=0.00..75217.18 rows=33430 width=924) (actual time=0.007..360.353 rows=2225077 loops=1)
               Filter: (handle_mark = ANY ('{1,3}'::bpchar[]))
               Rows Removed by Filter: 1114280
 Planning Time: 0.138 ms
 Execution Time: 11101.829 ms
(22 行记录)

改写后11s就能跑出结果，符合性能优化目标。

**　　这条改写SQL留个坑，不完全等价，完全等价的没放出来，有兴趣的同学可以思考下，自己做做实验。**

相关推荐
夜光小兔纸22 分钟前
oracle查询出表中某几个字段值不唯一的数据
数据库·sql·oracle
穷苦书生_万事愁5 小时前
如何在Oracle、MySQL、PostgreSQL中改变SQL提示格式
mysql·postgresql·oracle
小流年 °6 小时前
SQL Server时间转换
数据库·sql·sqlserver
TOR-NADO11 小时前
数据库详细复习第三章SQL语句
数据库·sql·oracle
吹牛不交税12 小时前
查询 条件列值用notepad++批量添加单引号和逗号
sql·notepad++
张念磊13 小时前
MySQL与Oracle 执行计划对比
sql·mysql·oracle
T.Y.Bao14 小时前
树型结构数据存储实践
sql
PangPiLoLo17 小时前
数据库与SQL
数据库·sql·mysql
0nePiece17 小时前
SQL中字符串类型中char和varchar之间的区别是什么
数据库·sql
好记忆不如烂笔头abc17 小时前
PostgreSQL 16.3+pg_rman 1.3.16的测试
数据库·postgresql
热门推荐
01Coze扣子平台完整体验和实践（附国内和国际版对比）02组基轨迹建模 GBTM的介绍与实现（Stata 或 R）03【经验分享】Ubuntu22.04安装微信(linux官方版)042024亚太赛（中文赛）数学建模竞赛选题建议+初步分析05yolov8实战第五天——yolov8+ffmpg实时视频流检测并进行实时推流——（推流，保姆教学）06【立创EDA-PCB设计基础】6.布线铺铜实战及细节详解07YOLOv5改进 | 添加CA注意力机制 + 增加预测层 + 更换损失函数之GIoU082025张宇考研数学，百度网盘视频课+36讲PDF讲义+真题09通达信神奇九转指标原理及选股公式，无未来函数，数字不消失10【极空间NAS】使用WebDAV服务 + DDNSTO内网穿透 实现思源笔记的内外网同步