https://db.in.tum.de上的 QueryBrew: System-Agnostic SQL-to-SQL Query Optimization 论文中介绍对SQL语句优化时举了这个例子
select i_category, count(distinct i_color)
from item group by i_category;
It counts the number of distinct colors for each category in the item
table from the TPC-DS benchmark.
Umbra's optimized query plan translated to SQL looks as follows:
WITH scan_1 AS (
SELECT i_category AS v5, i_color AS v6 FROM item
), groupby_2 AS (
SELECT v5 AS v3, v6 AS v4 FROM scan_1 s GROUP BY v5, v6
), groupby_3 AS (
SELECT v3 AS v1, count(v4) AS v2
FROM groupby_2 s GROUP BY v3)
SELECT v1 AS i_category, v2 AS count FROM groupby_3;我把后者简化成
select i_category, count(i_color) from (select i_category, i_color from item group by i_category, i_color )group by i_category然后对各种数据库软件做同样的测试，采用fsqlite生成的同一份sqlite3格式数据库。其中duckdb分别测试1、4、8线程。
C:\d>fsqlite0410 dist.db
fsqlite[dist.db]> .timer on
fsqlite[dist.db]> create table item as select value%7 i_category,value%11 i_color from generate_series(1,1000000);
Run Time: real 2.701
fsqlite[dist.db]> select i_category, count(i_color) from (select i_category, i_color from item group by i_category, i_color )group by i_category;
0|11
1|11
2|11
3|11
4|11
5|11
6|11
Run Time: real 0.640
fsqlite[dist.db]> select i_category, count(distinct i_color) from item group by i_category;
0|11
1|11
2|11
3|11
4|11
5|11
6|11
Run Time: real 1.074
fsqlite[dist.db]> .exit
C:\d>tursodb082 dist.db
Turso v0.8.2
Enter ".help" for usage hints.
Did you know that Turso supports vector search? Type .manual vector to learn more.
turso> .timer on
turso> select i_category, count(i_color) from (select i_category, i_color from item group by i_category, i_color )group
by i_category;
┌────────────┬────────────────┐
│ i_category │ count(i_color) │
├────────────┼────────────────┤
│ 0 │ 11 │
│ 1 │ 11 │
│ 2 │ 11 │
│ 3 │ 11 │
│ 4 │ 11 │
│ 5 │ 11 │
│ 6 │ 11 │
└────────────┴────────────────┘
Command stats:
----------------------------
total: 449 ms (this includes parsing/coloring of cli app)
query execution stats:
----------------------------
Execution: avg=55 ms, total=445 ms
I/O: No samples available
turso> select i_category, count(distinct i_color) from item group by i_category;
┌────────────┬─────────────────────────┐
│ i_category │ count(distinct i_color) │
├────────────┼─────────────────────────┤
│ 0 │ 11 │
│ 1 │ 11 │
│ 2 │ 11 │
│ 3 │ 11 │
│ 4 │ 11 │
│ 5 │ 11 │
│ 6 │ 11 │
└────────────┴─────────────────────────┘
Command stats:
----------------------------
total: 269 ms (this includes parsing/coloring of cli app)
query execution stats:
----------------------------
Execution: avg=33 ms, total=266 ms
I/O: No samples available
turso> explain select i_category, count(distinct i_color) from item group by i_category;
addr opcode p1 p2 p3 p4 p5 comment
---- ----------------- ---- ---- ---- ------------- -- -------
0 Init 0 46 0 0 Start at 46
1 Null 0 8 0 0 r[8]=NULL
2 SorterOpen 0 2 0 k(1,B) 0 cursor=0
3 Integer 0 5 0 0 r[5]=0; clear group by abort flag
4 Null 0 6 0 0 r[6]=NULL; initialize group by comparison registers to NULL
5 Gosub 12 41 0 0 ; go to clear accumulator subroutine
6 HashClear 1073741824 0 0 0
7 OpenRead 2 2 0 k(3,B,B) 0 table=item, root=2, iDb=0
8 Rewind 2 13 0 0 Rewind table item
9 ColumnRange 2 0 10 2 0 r[10..11]=item.i_category..i_color
10 MakeRecord 10 2 9 0 r[9]=mkrec(r[10..11])
11 SorterInsert 0 9 0 0 0 key=r[9]
12 Next 2 9 0 1
13 OpenPseudo 1 9 2 0 2 columns in r[9]
14 SorterSort 0 30 0 0
15 SorterData 0 9 1 0 r[9]=data
16 Column 1 0 13 0 r[13]=pseudo.column 0
17 Compare 6 13 1 k(1, Binary) 0 r[6..6]==r[13..13]
18 Jump 19 23 19 0 ; start new group if comparison is not equal
19 Gosub 3 34 0 0 ; check if ended group had data, and output if so
20 Move 13 6 1 0 r[6..6]=r[13..13]
21 IfPos 5 45 0 0 r[5]>0 -> r[5]-=0, goto 45; check abort flag
22 Gosub 12 41 0 0 ; goto clear accumulator subroutine
23 Column 1 1 14 0 r[14]=pseudo.column 1
24 HashDistinct 1073741824 14 1 jmp=26 0
turso>.exit
C:\d>sqlite35304 dist.db
SQLite version 3.53.4 2026-07-24 19:02:57
Enter ".help" for usage hints.
sqlite> .timer on
sqlite> select i_category, count(i_color) from (select i_category, i_color from item group by i_category, i_color )group
...> by i_category;
╭────────────┬────────────────╮
│ i_category │ count(i_color) │
╞════════════╪════════════════╡
│ 0 │ 11 │
│ 1 │ 11 │
│ 2 │ 11 │
│ 3 │ 11 │
│ 4 │ 11 │
│ 5 │ 11 │
│ 6 │ 11 │
╰────────────┴────────────────╯
Run Time: real 0.341434 user 0.250000 sys 0.046875
sqlite> select i_category, count(distinct i_color) from item group by i_category;
╭────────────┬──────────────────────╮
│ i_category │ count(distinct i_... │
╞════════════╪══════════════════════╡
│ 0 │ 11 │
│ 1 │ 11 │
│ 2 │ 11 │
│ 3 │ 11 │
│ 4 │ 11 │
│ 5 │ 11 │
│ 6 │ 11 │
╰────────────┴──────────────────────╯
Run Time: real 0.192112 user 0.156250 sys 0.031250
sqlite> .exit
C:\d>duckdb155 dist.db
DuckDB v1.5.5 (Variegata)
Enter ".help" for usage hints.
dist D .timer on
dist D set threads=1;
Run Time (s): real 0.003 user 0.000000 sys 0.000000
dist D explain select i_category, count(distinct i_color) from item group by i_category;
┌─────────────────────────────┐
│┌───────────────────────────┐│
││ Physical Plan ││
│└───────────────────────────┘│
└─────────────────────────────┘
┌───────────────────────────┐
│ HASH_GROUP_BY │
│ ──────────────────── │
│ Groups: #0 │
│ │
│ Aggregates: │
│ count(DISTINCT #1) │
│ │
│ ~632,119 rows │
└─────────────┬─────────────┘
┌─────────────┴─────────────┐
│ PROJECTION │
│ ──────────────────── │
│ i_category │
│ i_color │
│ │
│ ~999,999 rows │
└─────────────┬─────────────┘
┌─────────────┴─────────────┐
│ SQLITE_SCAN │
│ ──────────────────── │
│ Table: item │
│ File: dist.db │
│ │
│ Projections: │
│ i_category │
│ i_color │
│ │
│ ~999,999 rows │
└───────────────────────────┘
dist D select i_category, count(i_color) from (select i_category, i_color from item group by i_category, i_color )group by i_category;
┌────────────┬────────────────┐
│ i_category │ count(i_color) │
│ blob │ int64 │
├────────────┼────────────────┤
│ 1 │ 11 │
│ 2 │ 11 │
│ 3 │ 11 │
│ 4 │ 11 │
│ 5 │ 11 │
│ 6 │ 11 │
│ 0 │ 11 │
└────────────┴────────────────┘
Run Time (s): real 0.140 user 0.125000 sys 0.015625
dist D select i_category, count(distinct i_color) from item group by i_category;
┌────────────┬─────────────────────────┐
│ i_category │ count(DISTINCT i_color) │
│ blob │ int64 │
├────────────┼─────────────────────────┤
│ 1 │ 11 │
│ 2 │ 11 │
│ 3 │ 11 │
│ 4 │ 11 │
│ 5 │ 11 │
│ 6 │ 11 │
│ 0 │ 11 │
└────────────┴─────────────────────────┘
Run Time (s): real 0.142 user 0.140625 sys 0.000000
dist D set threads=4;
Run Time (s): real 0.003 user 0.000000 sys 0.000000
dist D select i_category, count(distinct i_color) from item group by i_category;
┌────────────┬─────────────────────────┐
│ i_category │ count(DISTINCT i_color) │
│ blob │ int64 │
├────────────┼─────────────────────────┤
│ 0 │ 11 │
│ 5 │ 11 │
│ 6 │ 11 │
│ 1 │ 11 │
│ 4 │ 11 │
│ 3 │ 11 │
│ 2 │ 11 │
└────────────┴─────────────────────────┘
Run Time (s): real 0.047 user 0.109375 sys 0.031250
dist D select i_category, count(i_color) from (select i_category, i_color from item group by i_category, i_color )group by i_category;
┌────────────┬────────────────┐
│ i_category │ count(i_color) │
│ blob │ int64 │
├────────────┼────────────────┤
│ 2 │ 11 │
│ 6 │ 11 │
│ 1 │ 11 │
│ 5 │ 11 │
│ 0 │ 11 │
│ 4 │ 11 │
│ 3 │ 11 │
└────────────┴────────────────┘
Run Time (s): real 0.046 user 0.046875 sys 0.078125
dist D set threads=8;
Run Time (s): real 0.001 user 0.000000 sys 0.000000
dist D select i_category, count(distinct i_color) from item group by i_category;
┌────────────┬─────────────────────────┐
│ i_category │ count(DISTINCT i_color) │
│ blob │ int64 │
├────────────┼─────────────────────────┤
│ 6 │ 11 │
│ 5 │ 11 │
│ 4 │ 11 │
│ 3 │ 11 │
│ 1 │ 11 │
│ 2 │ 11 │
│ 0 │ 11 │
└────────────┴─────────────────────────┘
Run Time (s): real 0.037 user 0.078125 sys 0.062500
dist D select i_category, count(i_color) from (select i_category, i_color from item group by i_category, i_color )group by i_category;
┌────────────┬────────────────┐
│ i_category │ count(i_color) │
│ blob │ int64 │
├────────────┼────────────────┤
│ 3 │ 11 │
│ 4 │ 11 │
│ 0 │ 11 │
│ 5 │ 11 │
│ 6 │ 11 │
│ 1 │ 11 │
│ 2 │ 11 │
└────────────┴────────────────┘
Run Time (s): real 0.034 user 0.093750 sys 0.046875clickhouse不能直接读sqlite, 单独测试，未关多线程，把数据放大了10倍
:) create table item as select i%7 i_category, i%11 i_color from (select generate_series i from generate_series(1,1000000));
CREATE TABLE item
AS SELECT
i % 7 AS i_category,
i % 11 AS i_color
FROM
(
SELECT generate_series AS i
FROM generate_series(1, 1000000)
)
Query id: e3af87a0-f7c3-4ba7-99f8-9fbe1c2dbdd9
Ok.
1000000 rows in set. Elapsed: 0.136 sec. Processed 1.00 million rows, 8.00 MB (7.38 million rows/s., 59.01 MB/s.)
Peak memory usage: 4.08 MiB.
:) insert into item select i%7 i_category, i%11 i_color from (select generate_series i from generate_series(1,10000000));
INSERT INTO item SELECT
i % 7 AS i_category,
i % 11 AS i_color
FROM
(
SELECT generate_series AS i
FROM generate_series(1, 10000000)
)
Query id: f80e99d9-33a4-4bf8-b34c-6cf428038405
Ok.
10000000 rows in set. Elapsed: 0.050 sec. Processed 10.00 million rows, 80.00 MB (200.09 million rows/s., 1.60 GB/s.)
:) select i_category, count(distinct i_color) from item group by i_category;
SELECT
i_category,
countDistinct(i_color)
FROM item
GROUP BY i_category
Query id: 64b66948-8a40-45b2-84d2-6757aa152fb7
┌─i_category─┬─countDistinct(i_color)─┐
1. │ 0 │ 11 │
2. │ 1 │ 11 │
3. │ 2 │ 11 │
4. │ 3 │ 11 │
5. │ 4 │ 11 │
6. │ 5 │ 11 │
7. │ 6 │ 11 │
└────────────┴────────────────────────┘
7 rows in set. Elapsed: 0.046 sec. Processed 11.00 million rows, 22.00 MB (238.03 million rows/s., 476.07 MB/s.)
Peak memory usage: 114.28 KiB.
:) select i_category, count(i_color) from (select i_category, i_color from item group by i_category, i_color
)group by i_category;
SELECT
i_category,
count(i_color)
FROM
(
SELECT
i_category,
i_color
FROM item
GROUP BY
i_category,
i_color
)
GROUP BY i_category
Query id: 44838165-6a68-4d47-b5de-c9af54fecfe2
┌─i_category─┬─count(i_color)─┐
1. │ 0 │ 11 │
2. │ 1 │ 11 │
3. │ 2 │ 11 │
4. │ 3 │ 11 │
5. │ 4 │ 11 │
6. │ 5 │ 11 │
7. │ 6 │ 11 │
└────────────┴────────────────┘
7 rows in set. Elapsed: 0.010 sec. Processed 11.00 million rows, 22.00 MB (1.11 billion rows/s., 2.22 GB/s.)
Peak memory usage: 4.01 MiB.
:)
:) explain select i_category, count(distinct i_color) from item group by i_category;
EXPLAIN
SELECT
i_category,
countDistinct(i_color)
FROM item
GROUP BY i_category
Query id: c11416a6-6d0c-44cd-a9e5-4485f012585f
┌─explain────────────────────────────────┐
1. │ Output: i_category, uniqExact(i_color) │
2. │ │
3. │ Aggregating │
4. │ │ Keys: i_category │
5. │ │ Aggregates: uniqExact(i_color) │
6. │ │ Skip merging: 0 │
7. │ └──ReadFromMergeTree (default.item) │
8. │ Read type: Default │
9. │ Parts: 1 | Granules: 1343 │
10. │ Output: i_category, i_color │
└────────────────────────────────────────┘
10 rows in set. Elapsed: 0.009 sec.
:) explain select i_category, count(i_color) from (select i_category, i_color from item group by i_category, i_color
)group by i_category;
EXPLAIN
SELECT
i_category,
count(i_color)
FROM
(
SELECT
i_category,
i_color
FROM item
GROUP BY
i_category,
i_color
)
GROUP BY i_category
Query id: 258e88a6-842e-4066-b899-776b8355515e
┌─explain────────────────────────────────┐
1. │ Output: i_category, count(i_color) │
2. │ │
3. │ Aggregating │
4. │ │ Keys: i_category │
5. │ │ Aggregates: count(i_color) │
6. │ │ Skip merging: 0 │
7. │ └──Aggregating │
8. │ │ Keys: i_category, i_color │
9. │ │ Skip merging: 0 │
10. │ └──ReadFromMergeTree (default.item) │
11. │ Read type: Default │
12. │ Parts: 1 | Granules: 1343 │
13. │ Output: i_category, i_color │
└────────────────────────────────────────┘
13 rows in set. Elapsed: 0.002 sec.可见，tursodb和sqlite的count(distinct)性能超过count()from group by子查询, 其他数据库都是后者性能更高，这是因为tursodb采用了hash distinct。