探索ClickHouse——使用MaterializedView存储kafka传递的数据

《探索ClickHouse------连接Kafka和Clickhouse》中,我们讲解了如何使用kafka engin连接kafka,并读取topic中的数据。但是遇到了一个问题,就是数据只能读取一次,即使后面还有新数据发送到该topic,该表也读不出来。

为了解决这个问题,我们引入MaterializedView。

创建表

该表结构直接借用了《探索ClickHouse------使用Projection加速查询》中的表结构。

bash 复制代码
CREATE TABLE materialized_uk_price_paid_from_kafka ( price UInt32, date Date, postcode1 LowCardinality(String), postcode2 LowCardinality(String), type Enum8('terraced' = 1, 'semi-detached' = 2, 'detached' = 3, 'flat' = 4, 'other' = 0), is_new UInt8, duration Enum8('freehold' = 1, 'leasehold' = 2, 'unknown' = 0), addr1 String, addr2 String, street LowCardinality(String), locality LowCardinality(String), town LowCardinality(String), district LowCardinality(String), county LowCardinality(String) ) ENGINE = MergeTree ORDER BY (postcode1, postcode2, addr1, addr2);

CREATE TABLE materialized_uk_price_paid_from_kafka

(
price UInt32,
date Date,
postcode1 LowCardinality(String),
postcode2 LowCardinality(String),
type Enum8('terraced' = 1, 'semi-detached' = 2, 'detached' = 3, 'flat' = 4, 'other' = 0),
is_new UInt8,
duration Enum8('freehold' = 1, 'leasehold' = 2, 'unknown' = 0),
addr1 String,
addr2 String,
street LowCardinality(String),
locality LowCardinality(String),
town LowCardinality(String),
district LowCardinality(String),
county LowCardinality(String)

)

ENGINE = MergeTree

ORDER BY (postcode1, postcode2, addr1, addr2)

Query id: 55b16049-a865-4d54-9333-d661c6280a09

Ok.

0 rows in set. Elapsed: 0.005 sec.

创建MaterializedView

bash 复制代码
CREATE MATERIALIZED VIEW uk_price_paid_from_kafka_consumer_view TO materialized_uk_price_paid_from_kafka AS SELECT splitByChar(' ', postcode) AS p, toUInt32(price_string) AS price, parseDateTimeBestEffortUS(time) AS date, p[1] AS postcode1, p[2] AS postcode2, transform(a, ['T', 'S', 'D', 'F', 'O'], ['terraced', 'semi-detached', 'detached', 'flat', 'other']) AS type, b = 'Y' AS is_new, transform(c, ['F', 'L', 'U'], ['freehold', 'leasehold', 'unknown']) AS duration, addr1, addr2, street, locality, town, district, county FROM uk_price_paid_from_kafka;

这样kafka topic中的数据被清洗到materialized_uk_price_paid_from_kafka表中。

查询

bash 复制代码
select * from materialized_uk_price_paid_from_kafka;

我们在给topic发送下面的内容

"{5FA8692E-537B-4278-8C67-5A060540506D}","19500","1995-01-27 00:00","SK10 2QW","T","N","L","38","","GARDEN STREET","MACCLESFIELD","MACCLESFIELD","MACCLESFIELD","CHESHIRE","A","A"

再查询表

bash 复制代码
select * from materialized_uk_price_paid_from_kafka;
相关推荐
沧澜sincerely2 分钟前
Redis 缓存模式与注解缓存
数据库·redis·缓存
Elastic 中国社区官方博客41 分钟前
Elasticsearch 推理 API 增加了开放的可定制服务
大数据·数据库·人工智能·elasticsearch·搜索引擎·ai·全文检索
埃泽漫笔1 小时前
Kafka、ActiveMQ、RabbitMQ、RocketMQ 对比
kafka·rabbitmq·activemq
nzxzn1 小时前
MYSQL第二次作业
数据库·mysql
核桃杏仁粉2 小时前
excel拼接数据库
数据库·oracle·excel
TiAmo zhang2 小时前
SQL Server 2019实验 │ 设计数据库的完整性
数据库·sqlserver
冻咸鱼2 小时前
MySQL的CRUD
数据库·mysql·oracle
Funny Valentine-js2 小时前
团队作业——概要设计和数据库设计
数据库
CodeJourney.3 小时前
SQL提数与数据分析指南
数据库·信息可视化·数据分析
whn19773 小时前
oracle数据库seg$的type#含义
数据库·oracle