PostgreSQL、KingBase 数据库 ORDER BY LIMIT 查询缓慢案例

好久没写博客了,最近从人大金仓离职了,新公司入职了蚂蚁集团,正在全力学习 OcenaBase数据库的体系结构中。

以后分享的案例知识基本上都是以 OcenaBase 分布式数据库为主了,呦西。😁

昨天帮朋友看了个金仓 KES数据库的 SQL 案例,废话不说,直接贴SQL:

慢SQL(执行时间 8s ,限制返回 30 行)

复制代码
explain analyze 
SELECT GI.ID,
       GI.MODULE_ID,
       GI.BT,
       GI.WH,
       GI.JJCD_TEXT,
       GI.CREATE_DEPTNAME,
       GI.CREATE_TIME,
       GI.MODULE_NAME
FROM gifgifgif GI
         INNER JOIN gufgufguf GUF ON (GUF.ifid = GI.ID)
WHERE GI.ROWSTATE > - 1
  AND (GUF.usid = '0' OR GUF.usid = '210317100256if6gVcTb3Ado1o2ytLs')
  AND ((GI.BT LIKE '%签%') OR (GI.MODULE_NAME LIKE '%签%') OR (GI.WH LIKE '%签%') OR (GI.JJCD_TEXT LIKE '%签%') OR
       (GI.CREATE_DEPTNAME LIKE '%签%'))
ORDER BY GI.CREATE_TIME DESC LIMIT 30;

****慢SQL执行计划

复制代码
                                                                                                                        QUERY PLAN                                                                                                           
             
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
-------------
 Limit  (cost=1001.05..17578.06 rows=30 width=240) (actual time=6458.263..8763.733 rows=7 loops=1)
   ->  Gather Merge  (cost=1001.05..3879467.79 rows=7019 width=240) (actual time=6458.261..8763.728 rows=7 loops=1)
         Workers Planned: 4
         Workers Launched: 4
         ->  Nested Loop  (cost=0.99..3877631.71 rows=1755 width=240) (actual time=2843.144..8274.217 rows=1 loops=5)
               ->  Parallel Index Scan Backward using gifgifgif_CREATE_TIME1 on gifgifgif GI  (cost=0.43..1158925.09 rows=433728 width=240) (actual time=0.043..2159.037 rows=350466 loops=5)
                     Filter: ((ROWSTATE > '-1'::numeric) AND (((BT)::text ~~ '%签%'::text) OR ((MODULE_NAME)::text ~~ '%签%'::text) OR ((WH)::text ~~ '%签%'::text) OR ((JJCD_TEXT)::text ~~ '%签%'::text) OR ((CREATE_DEPTNAME)::text ~~ '%
%'::text)))
                     Rows Removed by Filter: 423271
               ->  Index Only Scan using idx_gufgufguf_1_2_3 on gufgufguf GUF  (cost=0.56..6.26 rows=1 width=32) (actual time=0.017..0.017 rows=0 loops=1752329)  -- 慢:(1752329/5) * 0.017 / 1000  = 5.95s
                     Index Cond: (ifid = (GI.ID)::text)
                     Filter: (((usid)::text = '0'::text) OR ((usid)::text = '210317100256if6gVcTb3Ado1o2ytLs'::text))
                     Rows Removed by Filter: 3
                     Heap Fetches: 0
 Planning Time: 0.832 ms
 Execution Time: 8763.803 ms
(15 行记录)

我看到这计划简直无语,这种SQL不能 300 ms出来就绝对有问题,而且这么简单的语句都能用上并行,真的服了。

**  Index Only Scan using idx_gufgufguf_1_2_3 on gufgufguf GUF 每个并行进程执行 5.95s 这也太拉跨了。**

看执行计划基本都是用 Index Scan 或者是 Index Only Scan,但是本SQL 谓词过滤条件很多 or ,其实优化器如果执行位图扫描才是最优解计划,但是CBO偏偏没执行!!!

SQL去掉 LIMIT 30限制条件:

复制代码
explain analyze 
SELECT GI.ID,
       GI.MODULE_ID,
       GI.BT,
       GI.WH,
       GI.JJCD_TEXT,
       GI.CREATE_DEPTNAME,
       GI.CREATE_TIME,
       GI.MODULE_NAME
FROM gifgifgif GI
         INNER JOIN gufgufguf GUF ON (GUF.ifid = GI.ID)
WHERE GI.ROWSTATE > - 1
  AND (GUF.usid = '0' OR GUF.usid = '210317100256if6gVcTb3Ado1o2ytLs')
  AND ((GI.BT LIKE '%签%') OR (GI.MODULE_NAME LIKE '%签%') OR (GI.WH LIKE '%签%') OR (GI.JJCD_TEXT LIKE '%签%') OR
       (GI.CREATE_DEPTNAME LIKE '%签%'))
ORDER BY GI.CREATE_TIME DESC ;

去掉 LIMIT 30限制条件SQL执行计划:

复制代码
                                                                                                                        QUERY PLAN                                                                                                           
             
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
-------------
 Gather Merge  (cost=98222.89..99026.61 rows=6792 width=240) (actual time=33.640..35.974 rows=7 loops=1)
   Workers Planned: 3
   Workers Launched: 3
   ->  Sort  (cost=97222.85..97228.51 rows=2264 width=240) (actual time=26.724..26.725 rows=2 loops=4)
         Sort Key: GI.CREATE_TIME DESC
         Sort Method: quicksort  Memory: 25kB
         Worker 0:  Sort Method: quicksort  Memory: 25kB
         Worker 1:  Sort Method: quicksort  Memory: 25kB
         Worker 2:  Sort Method: quicksort  Memory: 26kB
         ->  Nested Loop  (cost=510.90..97096.70 rows=2264 width=240) (actual time=11.118..26.693 rows=2 loops=4)
               ->  Parallel Bitmap Heap Scan on gufgufguf GUF  (cost=510.35..59045.81 rows=5049 width=32) (actual time=0.480..3.498 rows=1178 loops=4)
                     Recheck Cond: (((usid)::text = '0'::text) OR ((usid)::text = '210317100256if6gVcTb3Ado1o2ytLs'::text))
                     Heap Blocks: exact=1464
                     ->  BitmapOr  (cost=510.35..510.35 rows=15652 width=0) (actual time=1.567..1.568 rows=0 loops=1)
                           ->  Bitmap Index Scan on gufgufguf_usid  (cost=0.00..251.26 rows=7826 width=0) (actual time=0.022..0.022 rows=0 loops=1)
                                 Index Cond: ((usid)::text = '0'::text)
                           ->  Bitmap Index Scan on gufgufguf_usid  (cost=0.00..251.26 rows=7826 width=0) (actual time=1.545..1.545 rows=4713 loops=1)
                                 Index Cond: ((usid)::text = '210317100256if6gVcTb3Ado1o2ytLs'::text)
               ->  Index Scan using gifgifgif_PKEY1 on gifgifgif GI  (cost=0.56..7.54 rows=1 width=240) (actual time=0.019..0.019 rows=0 loops=4713)
                     Index Cond: ((ID)::text = (GUF.ifid)::text)
                     Filter: ((ROWSTATE > '-1'::numeric) AND (((BT)::text ~~ '%签%'::text) OR ((MODULE_NAME)::text ~~ '%签%'::text) OR ((WH)::text ~~ '%签%'::text) OR ((JJCD_TEXT)::text ~~ '%签%'::text) OR ((CREATE_DEPTNAME)::text ~~ '%
%'::text)))
                     Rows Removed by Filter: 1
 Planning Time: 0.815 ms
 Execution Time: 36.060 ms
(24 行记录)

可以看到去掉LIMIT 30 以后,CBO能正常使用上 Bitmap Index Scan + BitmapOr 的查询策略,SQL只需要 36ms就能跑出结果。

PG比较牛逼的地方是B+树索引能基于SQL的查询条件,自动能转换成位图索引的查询策略。

像这种情况就简单了,只需要改变下限制SQL返回条数的逻辑即可,kingbase也兼容Oracle rownum 的语法,我们可以将上面SQL等价改成 rownum来进行现在。

LIMIT 改写成 rownum :

复制代码
explain analyze 
SELECT * FROM (
SELECT GI.ID,
       GI.MODULE_ID,
       GI.BT,
       GI.WH,
       GI.JJCD_TEXT,
       GI.CREATE_DEPTNAME,
       GI.CREATE_TIME,
       GI.MODULE_NAME
FROM gifgifgif GI
         INNER JOIN gufgufguf GUF ON (GUF.ifid = GI.ID)
WHERE GI.ROWSTATE > - 1
  AND (GUF.usid = '0' OR GUF.usid = '210317100256if6gVcTb3Ado1o2ytLs')
  AND ((GI.BT LIKE '%签%') OR (GI.MODULE_NAME LIKE '%签%') OR (GI.WH LIKE '%签%') OR (GI.JJCD_TEXT LIKE '%签%') OR
       (GI.CREATE_DEPTNAME LIKE '%签%'))
ORDER BY GI.CREATE_TIME DESC) WHERE ROWNUM <= 30;

LIMIT 改写成 rownum 执行计划:

复制代码
                                                                                                                           QUERY PLAN                                                                                                        
                   
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
-------------------
 Count  (cost=98222.89..99162.45 rows=0 width=240) (actual time=31.418..33.691 rows=7 loops=1)
   Stop Keys: (ROWNUM <= 30)
   ->  Gather Merge  (cost=98222.89..99026.61 rows=6792 width=240) (actual time=31.415..33.686 rows=7 loops=1)
         Workers Planned: 3
         Workers Launched: 3
         ->  Sort  (cost=97222.85..97228.51 rows=2264 width=240) (actual time=26.497..26.498 rows=2 loops=4)
               Sort Key: GI.CREATE_TIME DESC
               Sort Method: quicksort  Memory: 25kB
               Worker 0:  Sort Method: quicksort  Memory: 25kB
               Worker 1:  Sort Method: quicksort  Memory: 27kB
               Worker 2:  Sort Method: quicksort  Memory: 25kB
               ->  Nested Loop  (cost=510.90..97096.70 rows=2264 width=240) (actual time=14.246..26.465 rows=2 loops=4)
                     ->  Parallel Bitmap Heap Scan on gufgufguf GUF  (cost=510.35..59045.81 rows=5049 width=32) (actual time=0.513..3.401 rows=1178 loops=4)
                           Recheck Cond: (((usid)::text = '0'::text) OR ((usid)::text = '210317100256if6gVcTb3Ado1o2ytLs'::text))
                           Heap Blocks: exact=1373
                           ->  BitmapOr  (cost=510.35..510.35 rows=15652 width=0) (actual time=1.664..1.664 rows=0 loops=1)
                                 ->  Bitmap Index Scan on gufgufguf_usid  (cost=0.00..251.26 rows=7826 width=0) (actual time=0.024..0.024 rows=0 loops=1)
                                       Index Cond: ((usid)::text = '0'::text)
                                 ->  Bitmap Index Scan on gufgufguf_usid  (cost=0.00..251.26 rows=7826 width=0) (actual time=1.639..1.639 rows=4713 loops=1)
                                       Index Cond: ((usid)::text = '210317100256if6gVcTb3Ado1o2ytLs'::text)
                     ->  Index Scan using gifgifgif_PKEY1 on gifgifgif GI  (cost=0.56..7.54 rows=1 width=240) (actual time=0.019..0.019 rows=0 loops=4713)
                           Index Cond: ((ID)::text = (GUF.ifid)::text)
                           Filter: ((ROWSTATE > '-1'::numeric) AND (((BT)::text ~~ '%签%'::text) OR ((MODULE_NAME)::text ~~ '%签%'::text) OR ((WH)::text ~~ '%签%'::text) OR ((JJCD_TEXT)::text ~~ '%签%'::text) OR ((CREATE_DEPTNAME)::text 
~~ '%签%'::text)))
                           Rows Removed by Filter: 1
 Planning Time: 0.897 ms
 Execution Time: 33.778 ms
(26 行记录)

可以看到SQL通过将LIMIT 改写成 rownum 以后,原来执行时间 8s 降低到 33ms 就能跑出结果了,本条SQL到此已经优化完毕。

最后问题:那为什么原SQL使用 limit 会慢?改成 rownum 后速度能秒出,通常情况下来说 limit 是PG提供原生的语法,性能应该更好才是?

解答:是因为在PostgreSQL中,LIMIT子句本身不直接与索引类型相关联,而是用于指定返回的记录数。然而,当LIMIT与ORDER BY结合使用时,PostgreSQL的查询优化器可能会利用B+树索引来加速查询。

**   这是因为B+树索引能够有效地支持有序数据的检索,使得数据库能够快速地定位到需要的记录而不必扫描整个表或索引。**

**   然而需要通过索引进行排序的话,必然要通过 Index Scan 或者 Index Only Scan 扫描才可以对数据进行升序或者降序排序,而位图索引是不支持对数据进行排序功能的。**

**   所以为什么一开始SQL会使用**Index Scan 和 Index Only Scan 而不使用 Bitmap Index Scan + BitmapOr 的查询策略。****

****   各位读者以后在kingbase数据库进行业务开发,如果需要谓词过滤条件中有 or 排序限制条件中有 order by + limit 的需求,尽量对业务SQL进行评估,从而选择使用 rownum 还是 limit 语句来进行限制数据。****

****   如果在postgresql 进行开发的话遇到这种需求(pg不支持rownum写法),还需要在外面再包一层查询,使用 row_number() over() 窗口函数来进行限制。****

相关推荐
Mephisto.java5 小时前
【大数据学习 | HBASE高级】storeFile文件的合并
大数据·sql·oracle·json·hbase·database
不二人生5 小时前
SQL面试题——连续出现次数
hive·sql·面试
kahnyao7 小时前
PostgreSQL加密连接SSL配置
postgresql
在下方方8 小时前
【快捷入门笔记】mysql基本操作大全-SQL表
笔记·sql·mysql
Ttang238 小时前
数据库&SQL——嵌套子查询(IN/NOT IN/SOME/ALL/EXIST/NOT EXIST)
数据库·sql
Denny辉10 小时前
Flink使用SQL Gateway提交SQL Job到远程集群
sql·flink·gateway
shix .14 小时前
数据库sql初识以及-增删改查
数据库·sql·oracle
jokerest12317 小时前
web——sqliabs靶场——第三关
数据库·sql
Moshow郑锴18 小时前
PostgreSQL中如果有Left Join的时候索引怎么加
数据库·postgresql
Navicat中国19 小时前
Navicat 17 功能简介 | 单元格编辑器
数据库·sql·mysql·postgresql·oracle·编辑器·navicat