Flink SQL 中两个频繁变化 Topic 的 Join 行为分析

会不会产生重复数据？

会的，确实可能会关联出两条一模一样的数据。这是 Flink SQL 中流流 Join 的典型行为。

产生重复数据的原因

1. 数据更新机制

当两个 Topic 中的数据频繁更新时，每次更新都会产生新的数据记录。如果 Join 条件匹配，每次更新都会触发新的 Join 结果。

2. 示例场景

假设有两个表：

sql 复制代码

-- 用户表 (频繁更新)
CREATE TABLE user_topic (
    user_id BIGINT,
    user_name STRING,
    update_time TIMESTAMP(3),
    WATERMARK FOR update_time AS update_time - INTERVAL '5' SECOND
);
-- 订单表 (频繁更新) 
CREATE TABLE order_topic (
    order_id BIGINT,
    user_id BIGINT,
    order_amount DECIMAL(10, 2),
    update_time TIMESTAMP(3),
    WATERMARK FOR update_time AS update_time - INTERVAL '5' SECOND
);

3. 重复数据产生过程

```

时间点 T1:

用户表: (user_id=1, name='Alice', update_time=T1)

订单表: (order_id=100, user_id=1, amount=100, update_time=T1)

Join 结果: (user_id=1, name='Alice', order_id=100, amount=100)

时间点 T2:

用户表: (user_id=1, name='Alice Updated', update_time=T2)

订单表数据不变

Join 结果: (user_id=1, name='Alice Updated', order_id=100, amount=100)

→ 产生"重复"的Join结果（除了name字段更新）

```

解决方案

方案一：使用窗口 Join（推荐）

sql 复制代码

SELECT
    u.user_id,
    u.user_name,
    o.order_id,
    o.order_amount,
    u.update_time AS user_update_time,
    o.update_time AS order_update_time
FROM user_topic u
JOIN order_topic o ON u.user_id = o.user_id
AND u.update_time BETWEEN o.update_time - INTERVAL '1' HOUR AND o.update_time + INTERVAL '1' HOUR

方案二：使用最新状态 Join

sql 复制代码

-- 先获取每个用户的最新状态
CREATE TEMPORARY VIEW latest_users AS
SELECT
    user_id,
    user_name,
    update_time
FROM (
    SELECT *,
        ROW_NUMBER() OVER (PARTITION BY user_id ORDER BY update_time DESC) as rn
    FROM user_topic
) WHERE rn = 1;
-- 再与订单表Join
SELECT
    u.user_id,
    u.user_name,
    o.order_id,
    o.order_amount,
    o.update_time
FROM latest_users u
JOIN order_topic o ON u.user_id = o.user_id;

方案三：使用时态表 Join

sql 复制代码

-- 将用户表定义为时态表
CREATE TEMPORARY VIEW user_temporal AS
SELECT
    user_id,
    user_name,
    update_time,
    PROCTIME() AS proc_time
FROM user_topic;
-- 使用时态表Join
SELECT
    o.order_id,
    o.user_id,
    u.user_name,
    o.order_amount,
    o.update_time
FROM order_topic o
JOIN user_temporal FOR SYSTEM_TIME AS OF o.proc_time u
ON o.user_id = u.user_id;

方案四：使用去重处理

sql 复制代码

-- 在最终结果上做去重
CREATE TEMPORARY VIEW joined_result AS
SELECT
    u.user_id,
    u.user_name,
    o.order_id,
    o.order_amount,
    u.update_time AS user_update_time,
    o.update_time AS order_update_time
FROM user_topic u
JOIN order_topic o ON u.user_id = o.user_id;
-- 只保留每个order_id的最新Join结果
SELECT
    user_id,
    user_name,
    order_id,
    order_amount,
    user_update_time,
    order_update_time
FROM (
    SELECT *,
        ROW_NUMBER() OVER (PARTITION BY order_id ORDER BY GREATEST(user_update_time, order_update_time) DESC) as rn
    FROM joined_result
) WHERE rn = 1;

完整示例：避免重复数据的 Join

sql 复制代码

-- 1. 创建源表
CREATE TABLE user_updates (
    user_id BIGINT,
    user_name STRING,
    department STRING,
    update_time TIMESTAMP(3),
    WATERMARK FOR update_time AS update_time - INTERVAL '5' SECOND
) WITH (
    'connector' = 'kafka',
    'topic' = 'user_updates',
    'properties.bootstrap.servers' = 'localhost:9092',
    'format' = 'json'
);
CREATE TABLE order_updates (
    order_id BIGINT,
    user_id BIGINT,
    order_status STRING,
    update_time TIMESTAMP(3),
    WATERMARK FOR update_time AS update_time - INTERVAL '5' SECOND
) WITH (
    'connector' = 'kafka',
    'topic' = 'order_updates',
    'properties.bootstrap.servers' = 'localhost:9092',
    'format' = 'json'
);
-- 2. 使用窗口Join避免重复
SELECT
    u.user_id,
    u.user_name,
    u.department,
    o.order_id,
    o.order_status,
    u.update_time AS user_update_time,
    o.update_time AS order_update_time
FROM user_updates u
JOIN order_updates o ON u.user_id = o.user_id
AND u.update_time BETWEEN o.update_time - INTERVAL '10' MINUTE AND o.update_time + INTERVAL '10' MINUTE;
-- 3. 或者使用状态去重
CREATE TEMPORARY VIEW distinct_join_result AS
SELECT
    user_id,
    user_name,
    order_id,
    order_status,
    user_update_time,
    order_update_time,
    ROW_NUMBER() OVER (
        PARTITION BY user_id, order_id
        ORDER BY GREATEST(user_update_time, order_update_time) DESC
    ) as rn
FROM (
    SELECT
        u.user_id,
        u.user_name,
        o.order_id,
        o.order_status,
        u.update_time AS user_update_time,
        o.update_time AS order_update_time
    FROM user_updates u
    JOIN order_updates o ON u.user_id = o.user_id
);
SELECT
    user_id,
    user_name,
    order_id,
    order_status,
    user_update_time,
    order_update_time
FROM distinct_join_result
WHERE rn = 1;

注意事项

状态管理：流流 Join 会占用大量状态，需要合理设置状态 TTL
水位线设置：正确设置 WATERMARK 以避免数据乱序问题
性能考虑：频繁更新的 Topic Join 可能产生大量中间结果
去重策略：根据业务需求选择合适的去重粒度

总结：是的，两个频繁变化的 Topic 做 Inner Join 确实会产生"重复"数据，需要通过窗口 Join、状态去重或时态表等技术来处理这种重复问题。