Apache Paimon 使用之 Lookup Joins 解析

Lookup Join 是流式查询中的一种 Join，Join 要求一个表具有处理时间属性，另一个表由lookup source connector支持。

Paimon支持在主键表和附加表上进行Lookup Join。

a) 准备

创建一个Paimon表并实时更新它。

-- Create a paimon catalog
CREATE CATALOG my_catalog WITH (
  'type'='paimon',
  'warehouse'='hdfs://nn:8020/warehouse/path' -- or 'file://tmp/foo/bar'
);

USE CATALOG my_catalog;

-- Create a table in paimon catalog
CREATE TABLE customers (
    id INT PRIMARY KEY NOT ENFORCED,
    name STRING,
    country STRING,
    zip STRING
);

-- Launch a streaming job to update customers table
INSERT INTO customers ...

-- Create a temporary left table, like from kafka
CREATE TEMPORARY TABLE Orders (
    order_id INT,
    total INT,
    customer_id INT,
    proc_time AS PROCTIME()
) WITH (
    'connector' = 'kafka',
    'topic' = '...',
    'properties.bootstrap.servers' = '...',
    'format' = 'csv'
    ...
);

b) Normal Lookup（正常查找）

可以在lookup join query中使用customers。

-- enrich each order with customer information
SELECT o.order_id, o.total, c.country, c.zip
FROM Orders AS o
JOIN customers
FOR SYSTEM_TIME AS OF o.proc_time AS c
ON o.customer_id = c.id;

c) Retry Lookup（重试查找）

在 Flink 1.16+ ，如果Orders记录（主表）没有 Join 上，是因为相应的customers数据（查找表）尚未准备就绪，可以使用Flink的延迟重试策略进行查找。

-- enrich each order with customer information
SELECT /*+ LOOKUP('table'='c', 'retry-predicate'='lookup_miss', 'retry-strategy'='fixed_delay', 'fixed-delay'='1s', 'max-attempts'='600') */
o.order_id, o.total, c.country, c.zip
FROM Orders AS o
JOIN customers
FOR SYSTEM_TIME AS OF o.proc_time AS c
ON o.customer_id = c.id;

d) Async Retry Lookup（异步重试查找）

同步重试的问题是，一条记录没返回会阻塞后续记录，导致整个作业被阻塞，可以使用async + allow_unordered以避免阻塞。

-- enrich each order with customer information
SELECT /*+ LOOKUP('table'='c', 'retry-predicate'='lookup_miss', 'output-mode'='allow_unordered', 'retry-strategy'='fixed_delay', 'fixed-delay'='1s', 'max-attempts'='600') */
o.order_id, o.total, c.country, c.zip
FROM Orders AS o
JOIN customers /*+ OPTIONS('lookup.async'='true', 'lookup.async-thread-number'='16') */
FOR SYSTEM_TIME AS OF o.proc_time AS c
ON o.customer_id = c.id;

如果主表（Orders）是CDC流，allow_unordered将被Flink SQL忽略（仅支持附加流），可能阻塞流式任务，可以尝试使用Paimon的audit_log系统表功能（将CDC流转换为附加流）。

8）Query Service

可以运行Flink流作业来启动表的查询服务，当QueryService存在时，Flink Lookup Join将优先从中获取数据，这将有效地提高查询性能。

Flink SQL

CALL sys.query_service('database_name.table_name', parallelism);

Flink Action

<FLINK_HOME>/bin/flink run \
    /path/to/paimon-flink-action-0.7.0-incubating.jar \
    query_service \
    --warehouse <warehouse-path> \
    --database <database-name> \
    --table <table-name> \
    [--parallelism <parallelism>] \
    [--catalog_conf <paimon-catalog-conf> [--catalog_conf <paimon-catalog-conf> ...]]