霸王餐用户行为埋点：Kafka Connect+ClickHouse实时OLAP分析

整体架构概述

"吃喝不愁"App需对用户点击、下单、分享等行为进行毫秒级追踪与分析，以支撑运营决策与推荐策略。系统采用三层架构：

前端/客户端：通过 SDK 上报 JSON 格式埋点事件至 Kafka；
Kafka Connect：使用官方或自研 Sink Connector 将消息写入 ClickHouse；
ClickHouse：提供亚秒级 OLAP 查询能力。

该方案避免了传统批处理延迟，实现端到端秒级数据可见。

埋点事件格式定义

每条事件包含公共字段与业务字段：

json 复制代码

{
  "event_id": "evt_123456",
  "user_id": "u_7890",
  "event_type": "click_claim_button",
  "timestamp": 1717020800000,
  "app_version": "2.3.1",
  "device_id": "d_abc123",
  "properties": {
    "restaurant_id": "r_456",
    "campaign_id": "camp_789"
  }
}

ClickHouse 表结构设计

使用 MergeTree 引擎，按日期分区，按用户ID和事件类型排序：

sql 复制代码

CREATE TABLE eatfree.user_events (
    event_id String,
    user_id String,
    event_type LowCardinality(String),
    timestamp DateTime64(3),
    app_version String,
    device_id String,
    restaurant_id String,
    campaign_id String
) ENGINE = MergeTree()
PARTITION BY toYYYYMMDD(timestamp)
ORDER BY (user_id, event_type, timestamp)
SETTINGS index_granularity = 8192;

Kafka Connect Sink 配置

使用开源 clickhouse-kafka-connect 插件，配置如下：

properties 复制代码

name=eatfree-clickhouse-sink
connector.class=com.clickhouse.kafka.connect.ClickHouseSinkConnector
tasks.max=3
topics=user_events

key.converter=org.apache.kafka.connect.storage.StringConverter
value.converter=org.apache.kafka.connect.json.JsonConverter
value.converter.schemas.enable=false

clickhouse.server.url=http://clickhouse:8123
clickhouse.database=eatfree
clickhouse.table=user_events
clickhouse.username=default
clickhouse.password=

batch.size=1000
linger.ms=1000

部署后，Kafka 中 user_events 主题的消息将自动批量写入 ClickHouse。

Java 埋点上报示例（SDK 端）

java 复制代码

package juwatech.cn.tracking;

import org.apache.kafka.clients.producer.*;
import java.util.Properties;

public class EventTracker {

    private final KafkaProducer<String, String> producer;
    private static final String TOPIC = "user_events";

    public EventTracker() {
        Properties props = new Properties();
        props.put("bootstrap.servers", "kafka:9092");
        props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
        props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");
        this.producer = new KafkaProducer<>(props);
    }

    public void track(String userId, String eventType, String propertiesJson) {
        long ts = System.currentTimeMillis();
        String eventId = "evt_" + ts + "_" + userId.hashCode();
        String payload = String.format(
            "{\"event_id\":\"%s\",\"user_id\":\"%s\",\"event_type\":\"%s\",\"timestamp\":%d,\"app_version\":\"2.3.1\",\"device_id\":\"d_abc123\",%s}",
            eventId, userId, eventType, ts, propertiesJson.substring(1)
        );
        ProducerRecord<String, String> record = new ProducerRecord<>(TOPIC, userId, payload);
        producer.send(record, (metadata, exception) -> {
            if (exception != null) {
                System.err.println("Failed to send event: " + exception.getMessage());
            }
        });
    }

    public void close() {
        producer.close();
    }
}

调用示例：

java 复制代码

EventTracker tracker = new EventTracker();
tracker.track("u_7890", "click_claim_button", 
    "{\"restaurant_id\":\"r_456\",\"campaign_id\":\"camp_789\"}");

实时查询场景示例

每小时霸王餐领取次数：

sql 复制代码

SELECT
    toStartOfHour(timestamp) AS hour,
    count(*) AS claim_count
FROM eatfree.user_events
WHERE event_type = 'click_claim_button'
  AND timestamp >= now() - INTERVAL 24 HOUR
GROUP BY hour
ORDER BY hour DESC;

活跃用户地域分布（假设 device_id 关联 IP 地理库）：

sql 复制代码

SELECT
    multiIf(
        startsWith(device_id, 'BJ'), 'Beijing',
        startsWith(device_id, 'SH'), 'Shanghai',
        'Other'
    ) AS region,
    uniq(user_id) AS dau
FROM eatfree.user_events
WHERE timestamp >= today()
GROUP BY region;

容错与性能保障

Kafka 设置副本数 ≥ 2，确保消息不丢失；
ClickHouse 使用 ReplicatedMergeTree 实现集群高可用；
Kafka Connect 启用 exactly.once 语义（需 broker 支持）避免重复写入；
对高频事件（如曝光）可采样上报，降低吞吐压力。

扩展：自定义 Transformer（可选）

若需在 Connect 层清洗数据，可实现 Transformation 接口：

java 复制代码

package juwatech.cn.connect;

import org.apache.kafka.connect.connector.ConnectRecord;
import org.apache.kafka.connect.transforms.Transformation;
import java.util.Map;

public class EnrichEventTransformer<R extends ConnectRecord<R>> implements Transformation<R> {

    @Override
    public void configure(Map<String, ?> configs) {}

    @Override
    public R apply(R record) {
        // 示例：添加处理时间字段
        // 实际中可解析 JSON 并注入新字段
        return record;
    }

    @Override
    public void close() {}
}

注册后在 connector 配置中启用：

properties 复制代码

transforms=enrich
transforms.enrich.type=juwatech.cn.connect.EnrichEventTransformer

本文著作权归吃喝不愁app开发者团队，转载请注明出处！