union+distinct去重改写经验

1、问题

项目中有这样的语句写法

select count(1) from (
select distinct * from (
select t1.c2,t2.c1,t4.c1 as t4c1,t4.c2 as t4c2 from t1,t2,t4
where t1.c1=t2.c1 and t2.c2=t4.c2 and t2.status in (1,2,3)
union 
select t1.c2,t2.c1,t4.c1 as t4c1,t4.c2 as t4c2 from t1,t3,t2,t4
where t1.c1=t2.c1 and t2.c2=t3.c2 and t4.c2=t3.c2 and t3.status=1 and t2.status in (1,2,3)
)t)tt

计划：

语句的突破点在于去重，union 合并结果集，一般都是有重复的部分，根据这点，我们可以考虑三个思路：

（1）提取公共部分

（2）先去重后关联

（3）减少冗余去重，union去重后，没有增加查询项，就不用再distinct去重了。

2、改写

（1）提取公共部分，t1,t2,t4都是重复的，只是union 第二部分多了t3,可以将其拆成两部分；

（2）先去重后关联，重复值来自关联产生的，关联后产生的重复值是成倍的，因此可以思考先去重。

with temp as(
select  distinct t1.c2,t2.c1,t2.c2 t2c2
from  t1,t2
where t1.c1=t2.c1  and t2.status in (1,2,3)
),
temp2 as
(select t4.c1 as t4c1,t4.c2 as t4c2,t4.c2 as tc2
from t4
union 
select t4.c1 as t4c1,t4.c2 as t4c2,t3.c2
from t4,(select c2 from t3 where t3.status=1 group by c2)t3 where t4.c2=t3.c2)

然后再将两部分关联

select distinct t.c2,t.c1,t4c1,t4c2 from temp t,temp2 tt where t.t2c2=tt.tc2

值得注意是这里还需要加distinct，因为两个with 中查询项多查询一个项，最终是不需要求关联列的，那查询项有变化，就不能去掉distinct，举个例（查询项：1，2，3，4，（5）和1，2，3，4，（6））是没有重复的，但最终求前面四列此时就有重复的（1，2，3，4），因此需要去重。整体改写：

整体改写：

select count(1) from (
with temp as(
select  distinct t1.c2,t2.c1,t2.c2 t2c2
from  t1,t2
where t1.c1=t2.c1  and t2.status in (1,2,3)
),
temp2 as
(select t4.c1 as t4c1,t4.c2 as t4c2,t4.c2 as tc2
from t4
union 
select t4.c1 as t4c1,t4.c2 as t4c2,t3.c2
from t4,(select c2 from t3 where t3.status=1 group by c2)t3 where t4.c2=t3.c2)
select distinct t.c2,t.c1,t4c1,t4c2 from  temp t,temp2 tt where t.t2c2=tt.tc2
)t

计划：

语句从原来的8.9s下降到4.9s，减少一倍时间。

3、小结

这次想分享的思路是如何从去重角度出发，先减少重复值再关联处理。