探索ClickHouse——使用MaterializedView存储kafka传递的数据

《探索ClickHouse------连接Kafka和Clickhouse》中,我们讲解了如何使用kafka engin连接kafka,并读取topic中的数据。但是遇到了一个问题,就是数据只能读取一次,即使后面还有新数据发送到该topic,该表也读不出来。

为了解决这个问题,我们引入MaterializedView。

创建表

该表结构直接借用了《探索ClickHouse------使用Projection加速查询》中的表结构。

bash 复制代码
CREATE TABLE materialized_uk_price_paid_from_kafka ( price UInt32, date Date, postcode1 LowCardinality(String), postcode2 LowCardinality(String), type Enum8('terraced' = 1, 'semi-detached' = 2, 'detached' = 3, 'flat' = 4, 'other' = 0), is_new UInt8, duration Enum8('freehold' = 1, 'leasehold' = 2, 'unknown' = 0), addr1 String, addr2 String, street LowCardinality(String), locality LowCardinality(String), town LowCardinality(String), district LowCardinality(String), county LowCardinality(String) ) ENGINE = MergeTree ORDER BY (postcode1, postcode2, addr1, addr2);

CREATE TABLE materialized_uk_price_paid_from_kafka

(
price UInt32,
date Date,
postcode1 LowCardinality(String),
postcode2 LowCardinality(String),
type Enum8('terraced' = 1, 'semi-detached' = 2, 'detached' = 3, 'flat' = 4, 'other' = 0),
is_new UInt8,
duration Enum8('freehold' = 1, 'leasehold' = 2, 'unknown' = 0),
addr1 String,
addr2 String,
street LowCardinality(String),
locality LowCardinality(String),
town LowCardinality(String),
district LowCardinality(String),
county LowCardinality(String)

)

ENGINE = MergeTree

ORDER BY (postcode1, postcode2, addr1, addr2)

Query id: 55b16049-a865-4d54-9333-d661c6280a09

Ok.

0 rows in set. Elapsed: 0.005 sec.

创建MaterializedView

bash 复制代码
CREATE MATERIALIZED VIEW uk_price_paid_from_kafka_consumer_view TO materialized_uk_price_paid_from_kafka AS SELECT splitByChar(' ', postcode) AS p, toUInt32(price_string) AS price, parseDateTimeBestEffortUS(time) AS date, p[1] AS postcode1, p[2] AS postcode2, transform(a, ['T', 'S', 'D', 'F', 'O'], ['terraced', 'semi-detached', 'detached', 'flat', 'other']) AS type, b = 'Y' AS is_new, transform(c, ['F', 'L', 'U'], ['freehold', 'leasehold', 'unknown']) AS duration, addr1, addr2, street, locality, town, district, county FROM uk_price_paid_from_kafka;

这样kafka topic中的数据被清洗到materialized_uk_price_paid_from_kafka表中。

查询

bash 复制代码
select * from materialized_uk_price_paid_from_kafka;

我们在给topic发送下面的内容

"{5FA8692E-537B-4278-8C67-5A060540506D}","19500","1995-01-27 00:00","SK10 2QW","T","N","L","38","","GARDEN STREET","MACCLESFIELD","MACCLESFIELD","MACCLESFIELD","CHESHIRE","A","A"

再查询表

bash 复制代码
select * from materialized_uk_price_paid_from_kafka;
相关推荐
互联网搬砖老肖2 小时前
运维打铁: MongoDB 数据库集群搭建与管理
运维·数据库·mongodb
典学长编程3 小时前
数据库Oracle从入门到精通!第四天(并发、锁、视图)
数据库·oracle
积跬步,慕至千里4 小时前
clickhouse数据库表和doris数据库表迁移starrocks数据库时建表注意事项总结
数据库·clickhouse
极限实验室4 小时前
搭建持久化的 INFINI Console 与 Easysearch 容器环境
数据库
白仑色5 小时前
Oracle PL/SQL 编程基础详解(从块结构到游标操作)
数据库·oracle·数据库开发·存储过程·plsql编程
程序猿小D6 小时前
[附源码+数据库+毕业论文]基于Spring+MyBatis+MySQL+Maven+jsp实现的个人财务管理系统,推荐!
java·数据库·mysql·spring·毕业论文·ssm框架·个人财务管理系统
钢铁男儿7 小时前
C# 接口(什么是接口)
java·数据库·c#
__风__8 小时前
PostgreSQL kv(jsonb)存储
数据库·postgresql
轩情吖8 小时前
Qt的第一个程序(2)
服务器·数据库·qt·qt creator·qlineedit·hello world·编辑框
Databend8 小时前
Databend 产品月报(2025年6月)
数据库