1 序：实验背景
- 湖仓一体架构，在当前大数据行业、AIGC时代背景下，成为企业底层数据架构的必然选择。
- 之前一直是听各种湖仓一体的概念和教程，终归是没有动手亲身体验。因此，下定决心，终于在今天把整个过程实验通了------------基于 Docker + Flink + Iceberg + MinIO 构建湖仓一体架构。
诚然，也是踩了几个坑，目前都已趟过去了~哈哈
- 好了，小小总结一二，先讲一下选型思路 和实验过程吧。
Iceberg 在湖仓一体架构演进中的作用与地位
1. 湖仓一体的核心矛盾
- 传统数仓架构里：
- 数据湖 （HDFS/S3/OSS）擅长存原始数据、成本低、格式灵活，但缺少 ACID、缺 schema 约束、查询慢、难做增量更新
- 数据仓库 （Hive/ClickHouse/Redshift/Doris等）强一致、查询快，但贵、封闭、不易存原始明细（仅支持存储【结构化数据】，不支持存储【非结构化数据】、【半结构化数据】）
- 湖仓一体 （
lakehouse）要解决的问题就是：用一套存储，同时具备"湖的开放廉价"和"仓的可靠高效"。
2. Iceberg 扮演的角色
Apache Iceberg 是表格式层（Table Format），不是【存储引擎】，也不是【计算引擎】。它坐在"存储（对象存储/ HDFS）"和"计算（Flink/Spark/Trino）"中间，提供：
- ACID 事务：并发写不脏读，Flink 流式写入可 Exactly-Once
- 隐藏分区 / 时间旅行 / 快照隔离：查某历史版本、回滚、增量读取
- 开放格式：Parquet/ORC/Avro 存数据，元数据自己管，不绑定 Hive Metastore
- Schema Evolution：加列、改类型不影响老数据
- 与计算引擎解耦：Flink 写、Spark 批修、Trino 查、StarRocks 加速，都行
3. 行业地位
在湖仓一体演进中，Iceberg 和 Delta Lake、Hudi 并列三大表格式，但趋势上：
- Iceberg 因完全开源、引擎中立、社区跨云活跃 ，逐渐成为事实标准
- 国内大厂（阿里、腾讯、字节）及海外（Netflix 起源、AWS/GCP 支持）均主推
- 它把"数据湖"升级成了**"可治理的湖"**，是湖仓一体的关键拼图
总结：没有 Iceberg/Hudi/Delta，湖仓一体只是概念；有了 Iceberg，湖才真正具备仓的能力。
基于 Docker + Flink + Iceberg + MinIO 构建湖仓一体的意义
1. 各组件分工
- MinIO：S3 兼容的对象存储，模拟 AWS S3 / 阿里 OSS，承担"湖"的底层存储
- Iceberg：表格式层，让 MinIO 上的 Parquet 文件变成"有事务的表"
- Flink：流批一体计算引擎，负责 Kafka/CDC 实时入湖、流式 ETL、写 Iceberg
- Docker：容器化编排，一键拉起整套环境，去掉 Hadoop 全家桶负担
2. 这套组合的典型架构
Kafka / CDC
↓
Flink (SQL / DataStream)
↓ 写 Iceberg 表（事务提交）
MinIO (s3://warehouse/db/table/)
↓
Trino / Spark / StarRocks 查询Iceberg 元数据 + 数据文件都在 MinIO，Flink 通过 catalog 直接管理表。
3. 意义解读（为什么这么搭有价值）
（1）零云厂商绑定，成本可控
- MinIO 替代 S3/OSS，本地 IDC、笔记本、内网都能跑
- 存储与计算彻底分离，扩容只加节点，不走 HDFS 老路
（2）真正"轻量湖仓"可落地
- 不需要部署 Hive MetaStore、HDFS、YARN
- Docker Compose 几分钟起一套：Flink 1.18 + Iceberg 1.x + MinIO + Trino
- 适合验证 POC、教学、中小团队生产
（3）流式入湖闭环
- Fllink Checkpoint + Iceberg Commit 联动，实现端到端 Exactly-Once
- 迟到数据、CDC 更新（Merge-on-Read）都能处理
- 批流共用同一张 Iceberg 表，避免"实时一条链路、离线一条链路"
（4）开放生态，查询加速灵活
- 同一份 Iceberg 表：Flink 写、Trino 交互式查、Spark 补数、StarRocks 物化加速
- 不锁死某个引擎，符合"湖仓一体=开放"的初衷
（5）契合行业演进方向
当前大数据趋势是：
Hadoop 退场 → 对象存储 + 表格式 + 流批一体计算
Docker+Flink+Iceberg+MinIO 正是这个趋势的最小可用范式，也是很多公司从"Hive 数仓"迁向"湖仓一体"的试验田。
小结
- Iceberg 的地位：湖仓一体中的"表格式标准"，让数据湖获得事务、演化、快照能力，是湖变仓的关键层。
- Docker+Flink+Iceberg+MinIO 的意义：用最小依赖、全开源、可容器化的方式，把"实时入湖 + ACID 表 + 对象存储"跑通，既是学习范式，也是低成本生产落地的现实路径。
如果你愿意，我可以给你一份可直接 docker compose up 的 Flink+Icerberg+MinIO 配置示例，包含 Kafka 入湖 SQL 和 Trino 查询验证步骤。
2 实验过程
组件版本
- Docker: 24.0.2
- Docker Compose : v2.20.1-6047-g6817716
- MinIO: minio version RELEASE.2025-02-07T23-21-09Z (commit-id=703f51164d3d0c44af41b0d86075a1f61e4779e7)
- Flink: 2.0.2 (发布日期: 2026.4.28)
docker/iceberg-flink-quickstart/README.md#Build Arguments
- Iceberg: 1.10.1
docker/iceberg-flink-quickstart/README.md#Build Arguments
- Iceberg-Flink-Runtime: 2.0
docker/iceberg-flink-quickstart/README.md#Build Arguments
- Hadoop: 3.4.2
docker/iceberg-flink-quickstart/README.md#Build Arguments解释：iceberg-flink-quickstart Demo 项目下的 Dockerfile 在构建 Flink+Icerberg 镜像时，会把 Hadoop 的 client/common 类打进镜像里，而 3.4.2 是当前 Iceberg 主线（对应 quickstart 所在分支）锁定的 Hadoop 基线版本。并不是说 quickstart 项目在运行时一定要连 HDFS，而是 Iceberg 自身对 Hadoop 有编译期/运行期依赖。
1、Iceberg 内核一直带着 Hadoop 依赖
Apache Iceberg 的
iceberg-core里大量用到org.apache.hadoop.*（FileSystem、Path、Configuration、Hadoop catalog 实现等）。即使你用的是 REST Catalog + MinIO/S3 （quickstart 默认就是这套，不碰 HDFS），Iceberg 在 JVM 里依然要走 Hadoop 的Configuration和FileSystem抽象层来做 IO 适配。所以 Flink 镜像里必须有一套 Hadoop jars 在 classpath 上，版本就由构建参数 HADOOP_VERSION 控制。
2、为什么是 3.4.2 而不是 3.3.x？
Iceberg 1.x 后期（大致 1.7+ 起）把默认 Hadoop 基线从 3.3.4 / 3.3.6 往 3.4.x 推，目的是拿 Hadoop 3.4 的 S3A/ABFS 改进和 CVE 修复。
iceberg-flink-quickstart这个目录跟 Iceberg 父工程的gradle.properties/hadoopVersion属性联动，父工程改了基线，quickstart 的 Dockerfile 用ARG HADOOP_VERSION接同一个值，README 里就同步写成3.4.2。3.4.2 是 Hadoop 3.4 线的一个稳定补丁版，不是随便挑的------Hadoop 3.4.0 有些早期回归，社区 quickstart 一般锁到 3.4.2。
3、容易误解的点
3.1 写
HADOOP_VERSION≠ quickstart 会启动 HDFS 。默认 docker compose 里只有 Flink + REST Catalog + MinIO，没有 NameNode/DataNode。hadoop在这里只是"Java 依赖库"，不是"集群服务"。3.2 REST Catalog 路径下 Hadoop 版本基本不影响功能 ，因为 IO 走的是 Iceberg 的
S3FileIO+ AWS SDK，不是hadoop s3a。Hadoop 类只在 Iceberg 内部做 path 解析、metric、catalog 抽象时用。3.3 如果你把 catalog 换成
type=hadoop（HDFS 仓库）或 Hive catalog，那这个 Hadoop 版本才真正和文件系统语义相关，3.4.2 的 S3A/HDFS client 行为就得出差异了。
安装过程
构建镜像
确认已安装 docker / docker compose
shell
ash-4.4# docker-compose --version
Docker Compose version v2.20.1-6047-g6817716
ash-4.4# docker version
Client:
Version: 24.0.2
API version: 1.43
Go version: go1.20.4
Git commit: 610b8d0
Built: Thu Aug 1 07:07:08 2024
OS/Arch: linux/amd64
Context: default
Server:
Engine:
Version: 24.0.2
API version: 1.43 (minimum version 1.12)
Go version: go1.20.4
Git commit: b5710a2
Built: Thu Aug 1 07:07:31 2024
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: v1.7.1
GitCommit: 067f5021280b8de2059026fb5c43c4adb0f3f244
runc:
Version: v1.1.7
GitCommit: adc1b13
docker-init:
Version: 0.19.0
GitCommit: ed96d00下载源码
shell
ash-4.4# cd /volume1/docker/
ash-4.4# git clone https://github.com/apache/iceberg.git
ash-4.4# cd iceberg修改配置（可选步骤，已执行）
本步骤的目的：
通过修改 iceberg-flink-quickstart/docker-compose.yml 文件，实现：1、使用在同一宿主机的 minio （也是一个Docker容器：minio）服务，使得 docker compose up -d 时 不再单独创建 iceberg-flink-quickstart-minio-1 的容器
2、将 jobmanager 容器的 Flink Web UI 的8081端口暴露到宿主机（28081）上；
3、Flink 正常对接宿主机已有的 MinIO
备份原始文件: docker-compose.yml
shell
cd /volume1/docker/iceberg/docker/iceberg-flink-quickstart
cp docker-compose.yml docker-compose.bakup-at-202607291828.yml
ls -la | grep -i docker前提1： 宿主机已部署有 MinIO 服务
容器服务
| 项目 | 值 |
|---|---|
| 容器名 | minio => minio-server |
| MinIO API 端口 | 9000 => 9000 |
| MinIO Console | 9001 => 9001 |
| Access Key | minioadmin => minioadmin |
| Secret Key | minioadmin => （略） |
| Bucket及目录 | warehouse => lakehouse/iceberg（提前人工创建） |
| 网络 | NetworkdName = bridge NetworkID = 9f1b13e4e49c98838a8f887a43a2e49d6ce0845c949fa1892aae1fccb9dd4b78 （通过docker inspect minio-server grep -A 20 Networks获得） |
- 确认 minio 的部署运行情况、容器名称
shell
ash-4.4# docker ps --format "table {{.Names}}\t{{.Image}}"
NAMES IMAGE
mysql-V8.4 mysql:8
trino trinodb/trino:399
minio-server minio/minio:latest
synology_docviewer_2 synology/docviewer:1.3.0.0125
synology_docviewer_1 synology/docviewer:1.3.0.0125
centos7 centos:centos7如果叫别的，比如
minio-server，那就要改成：S3_ENDPOINT=http://minio-server:9000
- 确认 MinIO bucket 存在
shell
docker exec -it minio-server sh
mc alias set local http://my-nas-device.com:9000 minioadmin <密码>
mc mb local/warehouse或者：
shelldocker exec -it minio-server mc mb data/warehouse docker exec -it minio-server mc ls /data/
- 扩展：进入 minio 容器内 (本步骤，了解即可)
shell
ash-4.4# docker exec -it minio-server sh
sh-5.1# exit
exit
ash-4.4# 前提2：MinIO 容器与 Flink 容器需要在同一个 Docker 网络
这是关键，否则
jobmanager无法通过容器名访问minio。
txt
ash-4.4# docker inspect minio-server | grep -A 20 Networks
"Networks": {
"bridge": {
"IPAMConfig": null,
"Links": null,
"Aliases": null,
"NetworkID": "9f1b13e4e49c98838a8f887a43a2e49d6ce0845c949fa1892aae1fccb9dd4b78",
"EndpointID": "2cb0f72ee6acf114dfda35890c24e3fb0aff19bb628020cb6d589884aec5f27d",
"Gateway": "172.17.0.1",
"IPAddress": "172.17.0.2",
"IPPrefixLen": 16,
"IPv6Gateway": "",
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"MacAddress": "02:42:ac:11:00:02",
"DriverOpts": null
}
}
}
}
]
ash-4.4# docker network ls
NETWORK ID NAME DRIVER SCOPE
9f1b13e4e49c bridge bridge local
3056155d0119 host host local
a9ee6b998801 none null local
9d19e872f309 ubuntu_default bridge local
74ae7857cbca wps-office_default bridge local
- NetworkName = "bridge"
- NetworkID = 是 Docker 内部的唯一 ID
- =
9f1b13e4e49c98838a8f887a43a2e49d6ce0845c949fa1892aae1fccb9dd4b78- =
9f1b13e4e49c
创建自定义网络
为解决使用
bridge网络来启动时(docker-compose up -d)的错误:Error response from daemon: network-scoped alias is supported only for containers in user defined networks
bridge是 Docker 的"内置默认网络"- Docker 对它有很多限制，其中一条就是：
- ❌ 不支持 aliases
- ❌ 不支持 network-scoped alias
- ✅ 只允许最基础的容器通信
- 而 docker-compose.yml 中 有对 services.jobmanager/taskmanager/... . networks.iceberg_net.aliases.jobmanager 等别名的设置
- 创建网络
shell
ash-4.4# docker network create iceberg-net
25a6367209e888a60a3e7c4c5beb24cbe044367b10e90821c9c1c612f16f6219
ash-4.4# docker network ls
NETWORK ID NAME DRIVER SCOPE
9f1b13e4e49c bridge bridge local
3056155d0119 host host local
25a6367209e8 iceberg-net bridge local
a9ee6b998801 none null local
9d19e872f309 ubuntu_default bridge local
74ae7857cbca wps-office_default bridge local- 把 MinIO 连到这个网络
shekk
docker network connect iceberg-net minio-server- 修改 docker-compose.yml（关键）
yml
networks:
iceberg_net:
external: true
# name: bridge
name: iceberg-net- 延申：在 整个应用运行起来后，验证 Flink / REST Catalog 是否同一个网络？
- 在 各个 容器里验证：
shell
ash-4.4# docker-compose ps
NAME IMAGE COMMAND SERVICE CREATED STATUS PORTS
iceberg-flink-quickstart-iceberg-rest-1 apache/iceberg-rest-fixture "java -jar iceberg-r..." iceberg-rest 22 minutes ago Up 22 minutes (healthy) 8181/tcp
iceberg-flink-quickstart-taskmanager-1 iceberg-flink-quickstart-taskmanager "/docker-entrypoint...." taskmanager 22 minutes ago Up 21 minutes 6123/tcp, 8081/tcp
jobmanager iceberg-flink-quickstart-jobmanager "/docker-entrypoint...." jobmanager 22 minutes ago Up 21 minutes (healthy) 6123/tcp, 0.0.0.0:28081->8081/tcp, :::28081->8081/tcp
ash-4.4# docker exec -it iceberg-flink-quickstart-iceberg-rest-1 curl -Iv http://minio-server:9000
* Trying 172.21.0.2:9000...
* Connected to minio-server (172.21.0.2) port 9000 (#0)
> HEAD / HTTP/1.1
> Host: minio-server:9000
> User-Agent: curl/7.81.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 400 Bad Request
HTTP/1.1 400 Bad Request
< Accept-Ranges: bytes
Accept-Ranges: bytes
< Content-Length: 225
Content-Length: 225
< Content-Type: application/xml
Content-Type: application/xml
< Server: MinIO
Server: MinIO
< Vary: Origin
Vary: Origin
< Date: Wed, 29 Jul 2026 12:48:59 GMT
Date: Wed, 29 Jul 2026 12:48:59 GMT
<
* Connection #0 to host minio-server left intact
ash-4.4# docker exec -it iceberg-flink-quickstart-taskmanager-1 wget -S http://minio-server:9000
--2026-07-29 12:46:54-- http://minio-server:9000/
Resolving minio-server (minio-server)... 172.21.0.2
Connecting to minio-server (minio-server)|172.21.0.2|:9000... connected.
HTTP request sent, awaiting response...
HTTP/1.1 403 Forbidden
Accept-Ranges: bytes
Content-Length: 254
Content-Type: application/xml
Server: MinIO
Strict-Transport-Security: max-age=31536000; includeSubDomains
Vary: Origin
Vary: Accept-Encoding
X-Amz-Id-2: 5609df5e3dbc5a3fb996975c12f6250ce62549250ad5af51770e85f842f9d23c
X-Amz-Request-Id: 18C6C3761E0EBB53
X-Content-Type-Options: nosniff
X-Ratelimit-Limit: 304
X-Ratelimit-Remaining: 304
X-Xss-Protection: 1; mode=block
Date: Wed, 29 Jul 2026 12:46:54 GMT
2026-07-29 12:46:54 ERROR 403: Forbidden.
ash-4.4# docker exec -it jobmanager wget -S http://minio-server:9000
--2026-07-29 12:46:22-- http://minio-server:9000/
Resolving minio-server (minio-server)... 172.21.0.2
Connecting to minio-server (minio-server)|172.21.0.2|:9000... connected.
HTTP request sent, awaiting response...
HTTP/1.1 403 Forbidden
Accept-Ranges: bytes
Content-Length: 254
Content-Type: application/xml
Server: MinIO
Strict-Transport-Security: max-age=31536000; includeSubDomains
Vary: Origin
Vary: Accept-Encoding
X-Amz-Id-2: 5609df5e3dbc5a3fb996975c12f6250ce62549250ad5af51770e85f842f9d23c
X-Amz-Request-Id: 18C6C36E981B7F62
X-Content-Type-Options: nosniff
X-Ratelimit-Limit: 304
X-Ratelimit-Remaining: 304
X-Xss-Protection: 1; mode=block
Date: Wed, 29 Jul 2026 12:46:22 GMT
2026-07-29 12:46:22 ERROR 403: Forbidden.编辑配置文件
- 修改文件权限，解决文件无编辑权限的问题
shell
ash-4.4# chmod -R 755 /volume1/docker/iceberg/
ash-4.4# vim docker-compose.yml- 让 quickstart 使用这个网络
yml
networks:
iceberg_net:
external: true
name: bridge # ← 改成实际的 MinIO 网络名
- 确保在所有 service 中均有引用它：
shell
services:
jobmanager:
networks:
- iceberg_net
taskmanager:
networks:
- iceberg_net
# Iceberg REST Catalog
iceberg-rest:
networks:
- iceberg_net
- 直接删掉整个
minio:service 定义
yml
# # MinIO for S3-compatible object storage
# minio:
# image: minio/minio
# hostname: minio
# environment:
# MINIO_ROOT_USER: admin
# MINIO_ROOT_PASSWORD: password
# MINIO_DOMAIN: minio
# networks:
# iceberg_net:
# aliases:
# - warehouse.minio
# command: server /data --console-address ":9001"
# healthcheck:
# test: ["CMD", "mc", "ready", "local"]
# interval: 5s
# timeout: 5s
# retries: 5
# # Create the warehouse bucket
# create-bucket:
# image: minio/mc
# depends_on:
# minio:
# condition: service_healthy
# networks:
# iceberg_net:
# entrypoint: |
# /bin/sh -c "
# until (/usr/bin/mc alias set minio http://minio:9000 admin password) do echo '...waiting...' && sleep 1; done;
# /usr/bin/mc rm -r --force minio/warehouse;
# /usr/bin/mc mb minio/warehouse;
# /usr/bin/mc policy set public minio/warehouse;
# "
- 删除掉
iceberg-rest容器对create-bucket容器的依赖
shell
iceberg-rest:
...
#depends_on:
# create-bucket:
# condition: service_completed_successfully
...- 为什么这样 Flink 就能访问 MinIO？
- 网络拓扑
txt┌───────────────────────────────┐ │ Docker default bridge network │ │ (bridge) │ │ │ │ minio-server :9000 │ │ jobmanager :8081 │ │ taskmanager │ │ rest │ └───────────────────────────────┘✅ 容器间通信规则
✅ 可以通过容器名 互相访问。例如： taskmanager / jobmanager / iceberg-rest
✅ minio-server:9000 是合法的 DNS
❌ localhost:9000 不行（那是宿主机）
- 所以在 Flink / REST Catalog 中必须写成：
http://minio-server:9000，而非http://localhost:9000
- 暴露 JobManager 的 8081 到宿主机 28081
找到
jobmanagerservice，加上或修改ports：
yml
services:
jobmanager:
...
ports:
- "28081:8081" # ← 宿主机:容器 (Flink JobManager Web UI : http://localhost:28081 )
networks:
iceberg_net:- 修改 S3 的环境变量配置
- service =
yml
environment:
AWS_REGION: us-east-1
# AWS_ACCESS_KEY_ID: admin
AWS_ACCESS_KEY_ID: minioadmin
# AWS_SECRET_ACCESS_KEY: password
AWS_SECRET_ACCESS_KEY: 《密码》
# S3_ENDPOINT: http://minio:9000
S3_ENDPOINT: http://minio-server:9000
S3_PATH_STYLE_ACCESS: true
- service =
shell
environment:
AWS_REGION: us-east-1
#CATALOG_WAREHOUSE: s3://warehouse/
CATALOG_WAREHOUSE: s3://lakehouse/iceberg/
CATALOG_IO__IMPL: org.apache.iceberg.aws.s3.S3FileIO
#CATALOG_S3_ENDPOINT: http://minio:9000
CATALOG_S3_ENDPOINT: http://minio-server:9000
#CATALOG_S3_ACCESS__KEY__ID: admin
CATALOG_S3_ACCESS__KEY__ID: minioadmin
#CATALOG_S3_SECRET__ACCESS__KEY: password
CATALOG_S3_SECRET__ACCESS__KEY: 【密码】
# 配置 REST 服务本身去访问 S3 时的 Path Style 属性
CATALOG_S3_PATH__STYLE__ACCESS: "true"
# 强制 REST 服务把这个属性下发（Override）给客户端（Flink/Spark）
CATALOG_REST_S3_PATH__STYLE__ACCESS: "true"
#CATALOG_REST_OPTION_S3_PATH__STYLE__ACCESS: "true"
iceberg-rest容器服务所属的apache/iceberg-rest-fixture镜像在把环境变量转换为REST Catalog的配置时，只认2种环境变量命名格式：
CATALOG_前缀双下划线模式： 环境变量名中的点（.）需要替换为双下划线（__）。因此：s3.path-style-access应该写成CATALOG_S3_PATH__STYLE__ACCESS（双下划线）。如果写成了单下划线，镜像将无法识别。
REST_/ICEBERG_原生属性前缀。
清理先前已构建的基础镜像
如果先前有构建、而本次有可能涉及到更新的基础镜像，则清理之
shell
ash-4.4# cd /volume1/docker/iceberg/docker/iceberg-flink-quickstart
ash-4.4# docker-compose down
ash-4.4# docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
iceberg-flink-quickstart-taskmanager latest 41b9341e61b9 2 hours ago 1.06GB
iceberg-flink-quickstart-jobmanager latest 2cdc84320ddb 2 hours ago 1.06GB
apache/iceberg-flink-quickstart latest 948f09d0c182 3 hours ago 1.06GB
mysql 8 9cffaceb9b62 5 days ago 813MB
trinodb/trino latest 1a95bd13a377 11 days ago 1.38GB
alpine latest d529dd0c6e55 6 weeks ago 8.41MB
apache/flink 2.0-java21 5c6266043d12 2 months ago 905MB
apache/iceberg-rest-fixture latest 7ea2ad86a944 3 months ago 640MB
ubuntu latest f794f40ddfff 5 months ago 78.1MB
minio/mc latest c2ad77420d33 10 months ago 84.9MB
minio/minio latest 4ab91b236c0e 17 months ago 180MB
langgenius/dify-web 0.15.3 f0fae584255f 17 months ago 436MB
nginx latest 97662d24417b 17 months ago 192MB
postgres 15-alpine f2404745c576 18 months ago 273MB
redis 6-alpine 8d7a968b2baf 18 months ago 30.2MB
langgenius/dify-sandbox 0.2.10 4328059557e8 21 months ago 567MB
ubuntu/squid latest 87507c4542d0 22 months ago 242MB
mysql 5.7 5107333e08a8 2 years ago 501MB
nginx 1.24.0 6c0218f16876 3 years ago 142MB
trinodb/trino 399 cafc1b551d6d 3 years ago 1.29GB
...
ash-4.4# docker rmi iceberg-flink-quickstart-taskmanager
Untagged: iceberg-flink-quickstart-taskmanager:latest
Deleted: sha256:41b9341e61b9a7c7d55bd3d3495aaa1e1c859905f7a162cdac7198f7f5415f43
ash-4.4# docker rmi iceberg-flink-quickstart-jobmanager
Untagged: iceberg-flink-quickstart-jobmanager:latest
Deleted: sha256:2cdc84320ddb3da8c89cf10f6bae716b7c3892d3faf82f2ee28acb2d8d84825a
ash-4.4# docker rmi apache/iceberg-flink-quickstart
Untagged: apache/iceberg-flink-quickstart:latest
Deleted: sha256:948f09d0c18293cfba38be9c4a8246688aa37eb5a237de93bfb50a4dd00cc723
Deleted: sha256:fa8f98101272cda325362ce03c439ed19ed1db3768343d7dde41a0c528424975
Deleted: sha256:f168e9267fc322f8cd20c9dfe5a84507ad71ef03febea4ddd1ff28123bfb2674
Deleted: sha256:0f0c3aef14e3614f455d3ac5c448928cc56769ae41689d6b018b20465176213e
Deleted: sha256:2f1ac1a3ca9bcfac3401d83af7d22ec3d425eb034e0ff8169cdba45ee0184ca7
Deleted: sha256:c27f28551c3480347a9bfd0be41a8f934a766ce7aac623b7930260a2be7cdb53
Deleted: sha256:75d292c3f8979a52fc8931eebb7e87edc5e2fa32cdb128260bff18fcfce199c5
Deleted: sha256:b7ef879d4a7568530ecfda7d08b3757a04829c3c69af087fea408501da552428
Deleted: sha256:a93e796de99518d0340e86b4e980a969aff0895d51ee850adbeeae9865fbcad2
Deleted: sha256:90f75e1c7a98c82dad68d5bc9c109cd3999a6b1b677302bdd01afb11500bd808构建镜像
shell
cd /volume1/docker/iceberg
docker build \
--build-arg FLINK_VERSION=2.0 \
--build-arg ICEBERG_VERSION=1.10.1 \
-t apache/iceberg-flink-quickstart \
docker/iceberg-flink-quickstart/
# 或 默认的构建策略：docker-compose -f docker/iceberg-flink-quickstart/docker-compose.yml up -d --build
//查看构建完成的镜像
ash-4.4# docker images | grep -i iceberg
apache/iceberg-flink-quickstart latest 948f09d0c182 5 minutes ago 1.06GB假设修改了配置，则：
logsash-4.4# docker build \ > --build-arg FLINK_VERSION=2.0 \ > --build-arg ICEBERG_VERSION=1.10.1 \ > -t apache/iceberg-flink-quickstart \ > docker/iceberg-flink-quickstart/ DEPRECATED: The legacy builder is deprecated and will be removed in a future release. Install the buildx component to build images with BuildKit: https://docs.docker.com/go/buildx/ Sending build context to Docker daemon 21.5kB Step 1/10 : ARG FLINK_VERSION=2.0 Step 2/10 : FROM apache/flink:${FLINK_VERSION}-java21 ---> 5c6266043d12 Step 3/10 : SHELL ["/bin/bash", "-c"] ---> Running in 0e31264e09af Removing intermediate container 0e31264e09af ---> 05753f4fe14b Step 4/10 : ARG ICEBERG_FLINK_RUNTIME_VERSION=2.0 ---> Running in 61ca77d7cf7b Removing intermediate container 61ca77d7cf7b ---> 9b617a078acc Step 5/10 : ARG ICEBERG_VERSION=1.10.1 ---> Running in b09dda0a5f4e Removing intermediate container b09dda0a5f4e ---> 166535165f91 Step 6/10 : ARG HADOOP_VERSION=3.4.2 ---> Running in cb9513015672 Removing intermediate container cb9513015672 ---> 4305b5872cb6 Step 7/10 : USER flink ---> Running in 5f48ba19a0fc Removing intermediate container 5f48ba19a0fc ---> 6297b48757d2 Step 8/10 : WORKDIR /opt/flink ---> Running in 969fcad0f36b Removing intermediate container 969fcad0f36b ---> 8bac594ee43e Step 9/10 : RUN echo "-> Install JARs: Dependencies for Iceberg" && mkdir -p ./lib/iceberg && pushd $_ && curl -fO https://repo.maven.apache.org/maven2/org/apache/iceberg/iceberg-flink-runtime-${ICEBERG_FLINK_RUNTIME_VERSION}/${ICEBERG_VERSION}/iceberg-flink-runtime-${ICEBERG_FLINK_RUNTIME_VERSION}-${ICEBERG_VERSION}.jar && curl -fO https://repo.maven.apache.org/maven2/org/apache/iceberg/iceberg-aws-bundle/${ICEBERG_VERSION}/iceberg-aws-bundle-${ICEBERG_VERSION}.jar && popd ---> Running in a86fdb2cb7a0 -> Install JARs: Dependencies for Iceberg ~/lib/iceberg ~ % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 36.6M 100 36.6M 0 0 4644k 0 0:00:08 0:00:08 --:--:-- 6565k % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 59.7M 100 59.7M 0 0 5465k 0 0:00:11 0:00:11 --:--:-- 6997k ~ Removing intermediate container a86fdb2cb7a0 ---> b2b1afce06be Step 10/10 : RUN echo "-> Install JARs: Hadoop" && mkdir -p ./lib/hadoop && pushd $_ && curl -fO https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-client-api/${HADOOP_VERSION}/hadoop-client-api-${HADOOP_VERSION}.jar && curl -fO https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-client-runtime/${HADOOP_VERSION}/hadoop-client-runtime-${HADOOP_VERSION}.jar && popd ---> Running in 3dab295d3a8e -> Install JARs: Hadoop ~/lib/hadoop ~ % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 18.7M 100 18.7M 0 0 3628k 0 0:00:05 0:00:05 --:--:-- 4784k % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 29.0M 100 29.0M 0 0 4383k 0 0:00:06 0:00:06 --:--:-- 6228k ~ Removing intermediate container 3dab295d3a8e ---> 06ccc1ea3d01 Successfully built 06ccc1ea3d01 Successfully tagged apache/iceberg-flink-quickstart:latest ash-4.4#
创建并启动容器
创建并启动容器
- 基于已构建的本地镜像，创建并启动容器
启动后容器里会有一个叫
jobmanager的服务。
shell
ash-4.4# cd /volume1/docker/
ash-4.4# cd iceberg/docker/iceberg-flink-quickstart/
//启动（因本地已 build 过镜像，所以直接用本地镜像）
ash-4.4# docker-compose up -d
或者 这个命令： docker-compose up -d --force-recreate （没有亲测过）
[+] Running 13/13
✔ create-bucket 6 layers [⣿⣿⣿⣿⣿⣿] 0B/0B Pulled 24.3s
✔ b83ce1c86227 Pull complete 8.8s
✔ 4a93952ab9ce Pull complete 10.0s
✔ b268f447c871 Pull complete 10.7s
✔ f29310fc4a8a Pull complete 11.5s
✔ 9404e0df6bd3 Pull complete 13.5s
✔ b7c1efd1d2b4 Pull complete 18.3s
✔ iceberg-rest 5 layers [⣿⣿⣿⣿⣿] 0B/0B Pulled 146.7s
✔ f63eb04151bc Already exists 0.0s
✔ f70b96e55c3e Pull complete 67.9s
✔ 5de027cc8599 Pull complete 72.9s
✔ a68fefff9d4a Pull complete 74.5s
✔ ffe41b99943f Pull complete 139.1s
[+] Building 82.8s (11/11) FINISHED
=> [jobmanager internal] load .dockerignore 6.1s
=> => transferring context: 2B 0.0s
=> [jobmanager internal] load build definition from Dockerfile 7.0s
=> => transferring dockerfile: 2.22kB 0.0s
=> [taskmanager internal] load metadata for docker.io/apache/flink:2.0-java21 0.0s
=> [taskmanager 1/4] FROM docker.io/apache/flink:2.0-java21 21.6s
=> CACHED [taskmanager 2/4] WORKDIR /opt/flink 2.9s
=> CACHED [taskmanager 3/4] RUN echo "-> Install JARs: Dependencies for Iceberg" && mkdir -p ./lib/iceberg && pushd $_ && curl -fO https://repo.maven.apache. 18.8s
=> CACHED [taskmanager 4/4] RUN echo "-> Install JARs: Hadoop" && mkdir -p ./lib/hadoop && pushd $_ && curl -fO https://repo.maven.apache.org/maven2/org/apac 16.2s
=> [jobmanager] exporting to image 6.0s
=> => exporting layers 5.9s
=> => writing image sha256:2cdc84320ddb3da8c89cf10f6bae716b7c3892d3faf82f2ee28acb2d8d84825a 0.1s
=> => naming to docker.io/library/iceberg-flink-quickstart-jobmanager 0.1s
=> [taskmanager internal] load build definition from Dockerfile 2.6s
=> => transferring dockerfile: 2.22kB 0.0s
=> [taskmanager internal] load .dockerignore 2.3s
=> => transferring context: 2B 0.0s
=> [taskmanager] exporting to image 0.2s
=> => exporting layers 0.0s
=> => writing image sha256:41b9341e61b9a7c7d55bd3d3495aaa1e1c859905f7a162cdac7198f7f5415f43 0.1s
=> => naming to docker.io/library/iceberg-flink-quickstart-taskmanager 0.2s
[+] Running 6/6
✔ Network iceberg-flink-quickstart_iceberg_net Created 1.2s
✔ Container iceberg-flink-quickstart-minio-1 Healthy 11.5s
✔ Container iceberg-flink-quickstart-create-bucket-1 Exited 15.9s
✔ Container iceberg-flink-quickstart-iceberg-rest-1 Healthy 25.0s
✔ Container jobmanager Healthy 43.5s
✔ Container iceberg-flink-quickstart-taskmanager-1 Started 48.1s查验容器运行状态
shell
//查看是否有运行异常的容器 （最好是没有）
ash-4.4# docker-compose ps | grep -v "Up"
NAME IMAGE COMMAND SERVICE CREATED STATUS PORTS
//查看整体运行状态 || 查看创建并运行的容器： minio / iceberg-rest / iceberg-taskmanager / iceberg-jobnamager
ash-4.4# docker-compose ps
NAME IMAGE COMMAND SERVICE CREATED STATUS PORTS
iceberg-flink-quickstart-iceberg-rest-1 apache/iceberg-rest-fixture "java -jar iceberg-r..." iceberg-rest 17 minutes ago Up 17 minutes (healthy) 8181/tcp
iceberg-flink-quickstart-minio-1 minio/minio "/usr/bin/docker-ent..." minio 17 minutes ago Up 17 minutes (healthy) 9000/tcp
iceberg-flink-quickstart-taskmanager-1 iceberg-flink-quickstart-taskmanager "/docker-entrypoint...." taskmanager 17 minutes ago Up 16 minutes 6123/tcp, 8081/tcp
jobmanager iceberg-flink-quickstart-jobmanager "/docker-entrypoint...." jobmanager 17 minutes ago Up 17 minutes (healthy) 6123/tcp, 8081/tcp
注： State 列：
Up → 正常运行
Up (health: starting) → 正在做健康检查，等一会儿再看
Up (unhealthy) → 健康检查失败，容器在跑但服务不正常
Restarting → 容器反复重启，典型异常
Exit 0 / Exit 1 / ... → 容器已经退出，需要根据退出码排查
//如果是修改了配置后：
ash-4.4# docker-compose up -d
[+] Running 3/3
✔ Container iceberg-flink-quickstart-iceberg-rest-1 Healthy 10.7s
✔ Container jobmanager Healthy 29.4s
✔ Container iceberg-flink-quickstart-taskmanager-1 Started 32.8s
ash-4.4# docker-compose ps
NAME IMAGE COMMAND SERVICE CREATED STATUS PORTS
iceberg-flink-quickstart-iceberg-rest-1 apache/iceberg-rest-fixture "java -jar iceberg-r..." iceberg-rest About a minute ago Up About a minute (healthy) 8181/tcp
iceberg-flink-quickstart-taskmanager-1 iceberg-flink-quickstart-taskmanager "/docker-entrypoint...." taskmanager About a minute ago Up 31 seconds 6123/tcp, 8081/tcp
jobmanager iceberg-flink-quickstart-jobmanager "/docker-entrypoint...." jobmanager About a minute ago Up 51 seconds (healthy) 6123/tcp, 0.0.0.0:28081->8081/tcp, :::28081->8081/tcp
ash-4.4# docker ps | grep -i iceberg
d74cf38010cf iceberg-flink-quickstart-taskmanager "/docker-entrypoint...." 7 minutes ago Up 6 minutes 6123/tcp, 8081/tcp iceberg-flink-quickstart-taskmanager-1
639f603e89fe iceberg-flink-quickstart-jobmanager "/docker-entrypoint...." 7 minutes ago Up 7 minutes (healthy) 6123/tcp, 8081/tcp jobmanager
7371a3721017 apache/iceberg-rest-fixture "java -jar iceberg-r..." 7 minutes ago Up 7 minutes (healthy) 8181/tcp iceberg-flink-quickstart-iceberg-rest-1
7f2e224a8a85 minio/minio "/usr/bin/docker-ent..." 7 minutes ago Up 7 minutes (healthy) 9000/tcp iceberg-flink-quickstart-minio-1
// 看所有服务的最近日志
docker compose logs --tail=100
// 看某个服务的日志
docker compose logs <service_name>
// 实时跟踪某个服务的日志
docker compose logs -f <service_name>
// 如果容器一直在重启，加 --since 看历史：
docker compose logs --since 10m <service_name>查验容器服务: iceberg-rest
shell
ash-4.4# cd /volume1/docker/iceberg/docker/iceberg-flink-quickstart
ash-4.4# docker-compose exec -it iceberg-rest curl -s http://localhost:8181/v1/config
{
"defaults": {},
"overrides": {
"namespace-separator": "%2E"
},
"endpoints": [
"POST v1/oauth/tokens",
"POST https://auth-server.com/token",
"GET v1/config",
"GET /v1/{prefix}/namespaces",
"POST /v1/{prefix}/namespaces",
"HEAD /v1/{prefix}/namespaces/{namespace}",
"GET /v1/{prefix}/namespaces/{namespace}",
"DELETE /v1/{prefix}/namespaces/{namespace}",
"POST /v1/{prefix}/namespaces/{namespace}/properties",
"GET /v1/{prefix}/namespaces/{namespace}/tables",
"POST /v1/{prefix}/namespaces/{namespace}/tables",
"HEAD /v1/{prefix}/namespaces/{namespace}/tables/{table}",
"GET /v1/{prefix}/namespaces/{namespace}/tables/{table}",
"POST /v1/{prefix}/namespaces/{namespace}/register",
"POST /v1/{prefix}/namespaces/{namespace}/tables/{table}",
"DELETE /v1/{prefix}/namespaces/{namespace}/tables/{table}",
"POST /v1/{prefix}/tables/rename",
"POST /v1/{prefix}/namespaces/{namespace}/tables/{table}/metrics",
"POST /v1/{prefix}/transactions/commit",
"GET /v1/{prefix}/namespaces/{namespace}/views",
"HEAD /v1/{prefix}/namespaces/{namespace}/views/{view}",
"GET /v1/{prefix}/namespaces/{namespace}/views/{view}",
"POST /v1/{prefix}/namespaces/{namespace}/views",
"POST /v1/{prefix}/namespaces/{namespace}/views/{view}",
"POST /v1/{prefix}/views/rename",
"DELETE /v1/{prefix}/namespaces/{namespace}/views/{view}",
"POST /v1/{prefix}/namespaces/{namespace}/register-view",
"POST /v1/{prefix}/namespaces/{namespace}/tables/{table}/plan",
"GET /v1/{prefix}/namespaces/{namespace}/tables/{table}/plan/{plan-id}",
"POST /v1/{prefix}/namespaces/{namespace}/tables/{table}/tasks",
"DELETE /v1/{prefix}/namespaces/{namespace}/tables/{table}/plan/{plan-id}"
]
}进 Flink SQL Client
shell
ash-4.4# docker exec -it jobmanager ./bin/sql-client.sh
WARNING: Unknown module: jdk.compiler specified to --add-exports
WARNING: Unknown module: jdk.compiler specified to --add-exports
WARNING: Unknown module: jdk.compiler specified to --add-exports
WARNING: Unknown module: jdk.compiler specified to --add-exports
WARNING: Unknown module: jdk.compiler specified to --add-exports
▒▓██▓██▒
▓████▒▒█▓▒▓███▓▒
▓███▓░░ ▒▒▒▓██▒ ▒
░██▒ ▒▒▓▓█▓▓▒░ ▒████
██▒ ░▒▓███▒ ▒█▒█▒
░▓█ ███ ▓░▒██
▓█ ▒▒▒▒▒▓██▓░▒░▓▓█
█░ █ ▒▒░ ███▓▓█ ▒█▒▒▒
████░ ▒▓█▓ ██▒▒▒ ▓███▒
░▒█▓▓██ ▓█▒ ▓█▒▓██▓ ░█░
▓░▒▓████▒ ██ ▒█ █▓░▒█▒░▒█▒
███▓░██▓ ▓█ █ █▓ ▒▓█▓▓█▒
░██▓ ░█░ █ █▒ ▒█████▓▒ ██▓░▒
███░ ░ █░ ▓ ░█ █████▒░░ ░█░▓ ▓░
██▓█ ▒▒▓▒ ▓███████▓░ ▒█▒ ▒▓ ▓██▓
▒██▓ ▓█ █▓█ ░▒█████▓▓▒░ ██▒▒ █ ▒ ▓█▒
▓█▓ ▓█ ██▓ ░▓▓▓▓▓▓▓▒ ▒██▓ ░█▒
▓█ █ ▓███▓▒░ ░▓▓▓███▓ ░▒░ ▓█
██▓ ██▒ ░▒▓▓███▓▓▓▓▓██████▓▒ ▓███ █
▓███▒ ███ ░▓▓▒░░ ░▓████▓░ ░▒▓▒ █▓
█▓▒▒▓▓██ ░▒▒░░░▒▒▒▒▓██▓░ █▓
██ ▓░▒█ ▓▓▓▓▒░░ ▒█▓ ▒▓▓██▓ ▓▒ ▒▒▓
▓█▓ ▓▒█ █▓░ ░▒▓▓██▒ ░▓█▒ ▒▒▒░▒▒▓█████▒
██░ ▓█▒█▒ ▒▓▓▒ ▓█ █░ ░░░░ ░█▒
▓█ ▒█▓ ░ █░ ▒█ █▓
█▓ ██ █░ ▓▓ ▒█▓▓▓▒█░
█▓ ░▓██░ ▓▒ ▓█▓▒░░░▒▓█░ ▒█
██ ▓█▓░ ▒ ░▒█▒██▒ ▓▓
▓█▒ ▒█▓▒░ ▒▒ █▒█▓▒▒░░▒██
░██▒ ▒▓▓▒ ▓██▓▒█▒ ░▓▓▓▓▒█▓
░▓██▒ ▓░ ▒█▓█ ░░▒▒▒
▒▓▓▓▓▓▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒░░▓▓ ▓░▒█░
______ _ _ _ _____ ____ _ _____ _ _ _ BETA
| ____| (_) | | / ____|/ __ \| | / ____| (_) | |
| |__ | |_ _ __ | | __ | (___ | | | | | | | | |_ ___ _ __ | |_
| __| | | | '_ \| |/ / \___ \| | | | | | | | | |/ _ \ '_ \| __|
| | | | | | | | < ____) | |__| | |____ | |____| | | __/ | | | |_
|_| |_|_|_| |_|_|\_\ |_____/ \___\_\______| \_____|_|_|\___|_| |_|\__|
Welcome! Enter 'HELP;' to list all available commands. 'QUIT;' to exit.
Command history file path: /opt/flink/.flink-sql-history
Flink SQL> help;
The following commands are available:
HELP Prints the available commands.
QUIT/EXIT Quits the SQL CLI client.
CLEAR Clears the current terminal.
SET Sets a session configuration property. Syntax: "SET '<key>'='<value>';". Use "SET;" for listing all properties.
RESET Resets a session configuration property. Syntax: "RESET '<key>';". Use "RESET;" for reset all session properties.
INSERT INTO Inserts the results of a SQL SELECT query into a declared table sink.
INSERT OVERWRITE Inserts the results of a SQL SELECT query into a declared table sink and overwrite existing data.
SELECT Executes a SQL SELECT query on the Flink cluster.
EXPLAIN Describes the execution plan of a query or table with the given name.
BEGIN STATEMENT SET Begins a statement set. Syntax: "BEGIN STATEMENT SET;"
END Ends a statement set. Syntax: "END;"
ADD JAR Adds the specified jar file to the submitted jobs' classloader. Syntax: "ADD JAR '<path_to_filename>.jar'"
REMOVE JAR Removes the specified jar file from the submitted jobs' classloader. Syntax: "REMOVE JAR '<path_to_filename>.jar'"
SHOW JARS Shows the list of user-specified jar dependencies. This list is impacted by the --jar and --library startup options as well as the ADD/REMOVE JAR commands.
Hint: Make sure that a statement ends with ";" for finalizing (multi-line) statements.
You can also type any Flink SQL statement, please visit https://nightlies.apache.org/flink/flink-docs-stable/docs/dev/table/sql/overview/ for more details.
Flink SQL>
Flink SQL>
> select 1 as id, 'jack' as name union all select 2 as id , 'johnny' as name;
SQL Query Result (Table)
Table program finished. Page: Last of 1 Updated: 06:22:58.908
id name
1 jack
2 johnny
Flink SQL> exit;
[INFO] Exiting Flink SQL CLI Client...
Shutting down the session...
done.
ash-4.4# 看到
Flink SQL>提示符就成功了，可以直接建 Iceberg catalog / table 试。
- Flink Web UI：http://localhost:8081
- MinIO Console：http://localhost:9001 （admin/password，看 warehouse bucket）
Demo ： Flink SQL + Iceberg（含: Iceberg-Rest-Catalog） + MinIO
正式开始 Demo 体验。
首先，需要进入 Flink SQL Client 窗口。
在 Flink 中创建 Iceberg Catalog 和 Database
-
Iceberg 支持多种可用于跟踪表的 Catalog 后端，例如 JDBC、Hive MetaStore 和 Glue。在本指南中，我们使用由 S3 支持的 REST Catalog。要了解更多信息，请查看 Flink 章节中的 Catalog 页面。
-
首先，我们需要定义一个 Flink Catalog。此 Catalog 中的表将存储在 S3 对象存储中。
sql
CREATE CATALOG iceberg_catalog WITH (
'type' = 'iceberg',
'catalog-impl' = 'org.apache.iceberg.rest.RESTCatalog',
'uri' = 'http://iceberg-rest:8181',
'warehouse' = 's3://lakehouse/icebgerg/',
'io-impl' = 'org.apache.iceberg.aws.s3.S3FileIO',
's3.endpoint' = 'http://minio-server:9000',
's3.access-key-id' = 'minioadmin',
's3.secret-access-key' = 'password_xxx',
's3.path-style-access' = 'true'
);
'warehouse' = 's3://warehouse/'==>'warehouse' = 's3://lakehouse/icebgerg/'
- 在 Catalog 中创建一个数据库
shell
Flink SQL> CREATE DATABASE IF NOT EXISTS iceberg_catalog.nyc;
[INFO] Execute statement succeeded.可以理解为 数据库的完整库名为:
<catalogName>.<databaseName>，例如：iceberg_catalog.nyc
创建表
- 要在 Flink 中创建您的第一个 Iceberg 表，请运行
CREATE TABLE命令。让我们使用iceberg_catalog.nyc.taxis创建一个表，其中iceberg_catalog是 Catalog 名称，nyc是数据库名称，taxis是表名称。
shell
CREATE TABLE iceberg_catalog.nyc.taxis
(
vendor_id BIGINT,
trip_id BIGINT,
trip_distance FLOAT,
fare_amount DOUBLE,
store_and_fwd_flag STRING
);此时，对象存储的桶中将存在类似的目录与文件:
warehouse/nyc/taxis/metadata/00000-ef520259-980a-4535-87b0-8ec9dbcf26a3.metadata.json文件内容：
format-version属性：Iceberg目前有两个表版本（V1和V2），根据数据选择合适的表版本：
V1表只支持增量数据插入，适合做纯增量写入场景，如埋点数据。
V2表才支持行级更新，适合做状态变化的更新，如订单表同步。
json{ "format-version": 2, "table-uuid": "0e98eefa-f368-4b17-a459-17edca936410", "location": "s3://lakehouse/iceberg/nyc/taxis", "last-sequence-number": 0, "last-updated-ms": 1785398232040, "last-column-id": 5, "current-schema-id": 0, "schemas": [ { "type": "struct", "schema-id": 0, "fields": [ { "id": 1, "name": "vendor_id", "required": false, "type": "long" }, { "id": 2, "name": "trip_id", "required": false, "type": "long" }, { "id": 3, "name": "trip_distance", "required": false, "type": "float" }, { "id": 4, "name": "fare_amount", "required": false, "type": "double" }, { "id": 5, "name": "store_and_fwd_flag", "required": false, "type": "string" } ] } ], "default-spec-id": 0, "partition-specs": [ { "spec-id": 0, "fields": [] } ], "last-partition-id": 999, "default-sort-order-id": 0, "sort-orders": [ { "order-id": 0, "fields": [] } ], "properties": { "write.parquet.compression-codec": "zstd" }, "current-snapshot-id": -1, "refs": {}, "snapshots": [], "statistics": [], "partition-statistics": [], "snapshot-log": [], "metadata-log": [] }
-
Iceberg Catalog 支持完整的 Flink SQL DDL 命令，包括：
向表中写入数据
表创建完成后，您就可以插入记录了。
-
Flink 使用 Checkpoint 来确保数据持久性 和精确一次（exactly-once）语义。如果没有开启 Checkpoint，Iceberg 的数据和元数据可能无法完整提交到存储中。
Flink SQL> SET 'execution.checkpointing.interval' = '10s';
[INFO] Execute statement succeeded. -
然后，可以写入一些数据：
vendor_id / trip_id / trip_distance / fare_amount / store_and_flag
INSERT INTO iceberg_catalog.nyc.taxis
VALUES
(1, 1000371, 1.8, 15.32, 'N')
, (2, 1000372, 2.5, 22.15, 'N')
, (2, 1000373, 0.9, 9.01, 'N')
, (1, 1000374, 8.4, 42.13, 'Y');响应内容：
logFlink SQL> > INSERT INTO iceberg_catalog.nyc.taxis > ... [INFO] Submitting SQL update statement to the cluster... [INFO] SQL update statement has been successfully submitted to the cluster: Job ID: 293e34bec6f528d96b7f709bf9b0a617当 checkpoint 完成周期性的提交后，Flink Job Manager - Web UI - Jobs - Completed Jobs 中将存在1条 JobName=
insert-into_iceberg_catalog.nyc.taxis、JobID=293e34bec6f528d96b7f709bf9b0a617、JobType=STREAMING的记录。该作业的详情可访问: http://my-nas-device.com:28081/#/job/completed/293e34bec6f528d96b7f709bf9b0a617/overview此时，对象存储的桶中将存在类似的目录与文件:
shellwarehouse/nyc/taxis /data/00000-0-78f59a93-e76d-4f7e-862f-98a6443dfb6e-00001.parquet /metadata/ 00000-ef520259-980a-4535-87b0-8ec9dbcf26a3.metadata.json (此文件在建表时即会存在) 00001-e53e7252-f757-469f-9cfe-07fdc6c5aef6.metadata.json 50b28b70-a9f6-4fb8-a5cc-765a22abded9-m0.avro snap-257888749078048997-1-50b28b70-a9f6-4fb8-a5cc-765a22abded9.avro
读取表数据
- 要读取表，请使用 Iceberg 表的名称
shell
//错误示范1
Flink SQL> SELECT * FROM taxis;
[ERROR] Could not execute SQL statement. Reason:
org.apache.calcite.sql.validate.SqlValidatorException: Object 'taxis' not found
//错误示范2
Flink SQL> SELECT * FROM nyc.taxis;
[ERROR] Could not execute SQL statement. Reason:
org.apache.calcite.sql.validate.SqlValidatorException: Object 'nyc' not found
//正确示范
> SELECT * FROM iceberg_catalog.nyc.taxis;
SQL Query Result (Table)
Table program finished. Page: Last of 1 Updated: 07:43:54.543
vendor_id trip_id trip_distance fare_amount store_and_fwd_flag
1 1000371 1.8 15.32 N
2 1000372 2.5 22.15 N
2 1000373 0.9 9.01 N
1 1000374 8.4 42.13 Y通过内联配置创建表
如上所示创建一个由 Iceberg REST Catalog 支持的 Flink Catalog 是在 Flink 中使用 Iceberg 的一种方式。另一种方式是使用 Flink connector 并在表定义中直接指定 Catalog 连接详细信息。这种方式仍然连接到同一个外部 Iceberg REST Catalog，区别在于你不需要单独执行
CREATE CATALOG语句。
- 使用内联配置创建表：
这里的 Flink 表定义在 Flink 默认的内存 Catalog (
default_catalog) 中注册，但连接器属性 会告诉 Flink 将该 Iceberg 表及其数据存储在与之前相同的 REST Catalog 和 S3 存储中，对应的 Catalog / Database 分别为: default_catalog / default_database。
sql
CREATE TABLE taxis_inline_config (
vendor_id BIGINT,
trip_id BIGINT,
trip_distance FLOAT,
fare_amount DOUBLE,
store_and_fwd_flag STRING
) WITH (
'connector' = 'iceberg',
'catalog-name' = 'foo', -- Required by Flink connector but value doesn't matter for inline config
'catalog-type' = 'rest',
'uri' = 'http://iceberg-rest:8181',
'warehouse' = 's3://lakehouse/iceberg/',
'io-impl' = 'org.apache.iceberg.aws.s3.S3FileIO',
's3.endpoint' = 'http://minio-server:9000',
's3.access-key-id' = 'minioadmin',
's3.secret-access-key' = 'password_xxx',
's3.path-style-access' = 'true'
);
//写数据
INSERT INTO default_database.taxis_inline_config
VALUES
(1, 1000371, 1.8, 15.32, 'N')
, (2, 1000372, 2.5, 22.15, 'N')
, (2, 1000373, 0.9, 9.01, 'N')
, (1, 1000374, 8.4, 42.13, 'Y');
-- 响应内容:
[INFO] Submitting SQL update statement to the cluster...
[INFO] SQL update statement has been successfully submitted to the cluster:
Job ID: 5360c5c5f785a57e4322a2c159c1391e
//查询表数据 （查询方式稍有不同，可对比）
Flink SQL> select * from iceberg_catalog.default_database.taxis_inline_config; -- 查询方式0
Flink SQL> select * from default_database.taxis_inline_config; -- 查询方式1
Flink SQL> select * from taxis_inline_config; -- 查询方式2此时，这张表即会在对应的目录下创建:
shelllakehouse/iceberg/default_database/taxis_inline_config/metadata/ 00000-5a8c92fe-533d-4121-a597-7226a093ae51.metadata.json其文件内容：
json{ "format-version": 2, "table-uuid": "79720fbb-9fb9-4b42-9f6e-baf75c1f2d7d", "location": "s3://lakehouse/iceberg/default_database/taxis_inline_config", "last-sequence-number": 0, "last-updated-ms": 1785399057242, "last-column-id": 5, "current-schema-id": 0, "schemas": [ { "type": "struct", "schema-id": 0, "fields": [ { "id": 1, "name": "vendor_id", "required": false, "type": "long" }, { "id": 2, "name": "trip_id", "required": false, "type": "long" }, { "id": 3, "name": "trip_distance", "required": false, "type": "float" }, { "id": 4, "name": "fare_amount", "required": false, "type": "double" }, { "id": 5, "name": "store_and_fwd_flag", "required": false, "type": "string" } ] } ], "default-spec-id": 0, "partition-specs": [ { "spec-id": 0, "fields": [] } ], "last-partition-id": 999, "default-sort-order-id": 0, "sort-orders": [ { "order-id": 0, "fields": [] } ], "properties": { "s3.path-style-access": "true", "s3.access-key-id": "minioadmin", "s3.secret-access-key": "password_xxx", "s3.endpoint": "http://minio-server:9000", "io-impl": "org.apache.iceberg.aws.s3.S3FileIO", "write.parquet.compression-codec": "zstd", "catalog-type": "rest", "catalog-name": "foo", "warehouse": "s3://lakehouse/iceberg/", "uri": "http://iceberg-rest:8181" }, "current-snapshot-id": -1, "refs": {}, "snapshots": [], "statistics": [], "partition-statistics": [], "snapshot-log": [], "metadata-log": [] }
常见操作
进入容器内部
shell
ash-4.4# docker run --rm -it apache/iceberg-flink-quickstart:latest /bin/bash
flink@2a505077f4db:~$
flink@2a505077f4db:~$ cat /etc/os-release
PRETTY_NAME="Ubuntu 22.04.5 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.5 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy
flink@2a505077f4db:~$ exit
exit
ash-4.4# 停止运行
shell
cd /volume1/docker/iceberg/docker/iceberg-flink-quickstart
docker-compose down
或 docker-compose -f docker/iceberg-flink-quickstart/docker-compose.yml down3 使用指南(Flink SQL Client)
Catalog / Database 管理
- 使用指定的 Catalog、Database
shell
Flink SQL> USE CATALOG default_catalog;
[INFO] Execute statement succeeded.
Flink SQL> USE default_database;
[INFO] Execute statement succeeded.- 查看当前有哪些 Catalog、Database；查看当前使用的哪个 Catalog 、Database？
sql
Flink SQL> SHOW CURRENT CATALOG;
+----------------------+
| current catalog name |
+----------------------+
| default_catalog |
+----------------------+
1 row in set
Flink SQL> SHOW CURRENT DATABASE;
+-----------------------+
| current database name |
+-----------------------+
| default_database |
+-----------------------+
1 row in set
Flink SQL> show databases;
+------------------+
| database name |
+------------------+
| default_database |
+------------------+
1 row in set
Flink SQL> show catalogs;
+-----------------+
| catalog name |
+-----------------+
| default_catalog |
| iceberg_catalog |
+-----------------+
2 rows in set
//Flink 支持带全限定名或 IN 子句，不用 USE 也能列：
-- 列出指定 catalog 下的库
Flink SQL> SHOW DATABASES FROM iceberg_catalog;
+------------------+
| database name |
+------------------+
| default_database |
| nyc |
+------------------+
2 rows in set
-- 列出指定 catalog.db 下的表
Flink SQL> SHOW TABLES FROM iceberg_catalog.nyc;
+------------+
| table name |
+------------+
| taxis |
+------------+
1 row in set
或:
Flink SQL> SHOW TABLES IN iceberg_catalog.nyc;
+------------+
| table name |
+------------+
| taxis |
+------------+
1 row in set
-- 模糊匹配表名
Flink SQL> SHOW TABLES FROM iceberg_catalog.nyc LIKE '%ta%';
+------------+
| table name |
+------------+
| taxis |
+------------+
1 row in set- 查看 指定 Catalog 的定义
shell
Flink SQL> SHOW CREATE CATALOG iceberg_catalog;
+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| result |
+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| CREATE CATALOG `iceberg_catalog`
WITH (
'catalog-impl' = 'org.apache.iceberg.rest.RESTCatalog',
'io-impl' = 'org.apache.iceberg.aws.s3.S3FileIO',
's3.access-key-id' = 'minioadmin',
's3.endpoint' = 'http://minio-server:9000',
's3.path-style-access' = 'true',
's3.secret-access-key' = 'password_xxx',
'type' = 'iceberg',
'uri' = 'http://iceberg-rest:8181',
'warehouse' = 's3://lakehouse/icebgerg/'
)
|
+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
1 row in setTable 管理
- 查看表的定义
sql
-- 直接查表结构
Flink SQL> DESCRIBE iceberg_catalog.nyc.taxis;
+--------------------+--------+------+-----+--------+-----------+
| name | type | null | key | extras | watermark |
+--------------------+--------+------+-----+--------+-----------+
| vendor_id | BIGINT | TRUE | | | |
| trip_id | BIGINT | TRUE | | | |
| trip_distance | FLOAT | TRUE | | | |
| fare_amount | DOUBLE | TRUE | | | |
| store_and_fwd_flag | STRING | TRUE | | | |
+--------------------+--------+------+-----+--------+-----------+
5 rows in setF 附件
F.1 docker/iceberg-flink-quickstart/docker-compose.yml (原始配置)
/volume1/docker/iceberg/docker/iceberg-flink-quickstart/docker-compose.yml
yml
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
#
# Flink Quickstart with Apache Iceberg
#
# Usage:
# docker compose -f docker/iceberg-flink-quickstart/docker-compose.yml up -d --build
#
# Connect to SQL client:
# docker exec -it jobmanager ./bin/sql-client.sh
services:
# Flink JobManager
jobmanager:
build: .
hostname: jobmanager
container_name: jobmanager
depends_on:
iceberg-rest:
condition: service_healthy
networks:
iceberg_net:
command: jobmanager
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:8081/overview"]
start_period: 30s
interval: 5s
timeout: 3s
retries: 5
environment:
FLINK_PROPERTIES: |
jobmanager.rpc.address: jobmanager
taskmanager.numberOfTaskSlots: 2
parallelism.default: 2
AWS_REGION: us-east-1
AWS_ACCESS_KEY_ID: admin
AWS_SECRET_ACCESS_KEY: password
S3_ENDPOINT: http://minio:9000
# Flink TaskManager
taskmanager:
build: .
hostname: taskmanager
depends_on:
jobmanager:
condition: service_healthy
networks:
iceberg_net:
command: taskmanager
deploy:
replicas: 1
environment:
FLINK_PROPERTIES: |
jobmanager.rpc.address: jobmanager
taskmanager.numberOfTaskSlots: 2
parallelism.default: 2
AWS_REGION: us-east-1
AWS_ACCESS_KEY_ID: admin
AWS_SECRET_ACCESS_KEY: password
S3_ENDPOINT: http://minio:9000
# Iceberg REST Catalog
iceberg-rest:
image: apache/iceberg-rest-fixture
hostname: iceberg-rest
depends_on:
create-bucket:
condition: service_completed_successfully
networks:
iceberg_net:
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:8181/v1/config"]
interval: 5s
timeout: 3s
retries: 5
environment:
AWS_REGION: us-east-1
CATALOG_WAREHOUSE: s3://warehouse/
CATALOG_IO__IMPL: org.apache.iceberg.aws.s3.S3FileIO
CATALOG_S3_ENDPOINT: http://minio:9000
CATALOG_S3_ACCESS__KEY__ID: admin
CATALOG_S3_SECRET__ACCESS__KEY: password
# MinIO for S3-compatible object storage
minio:
image: minio/minio
hostname: minio
environment:
MINIO_ROOT_USER: admin
MINIO_ROOT_PASSWORD: password
MINIO_DOMAIN: minio
networks:
iceberg_net:
aliases:
- warehouse.minio
command: server /data --console-address ":9001"
healthcheck:
test: ["CMD", "mc", "ready", "local"]
interval: 5s
timeout: 5s
retries: 5
# Create the warehouse bucket
create-bucket:
image: minio/mc
depends_on:
minio:
condition: service_healthy
networks:
iceberg_net:
entrypoint: |
/bin/sh -c "
until (/usr/bin/mc alias set minio http://minio:9000 admin password) do echo '...waiting...' && sleep 1; done;
/usr/bin/mc rm -r --force minio/warehouse;
/usr/bin/mc mb minio/warehouse;
/usr/bin/mc policy set public minio/warehouse;
"
networks:
iceberg_net:Y 推荐文献
- Apache Iceberg
- Docker Docker Compose 基础教程（概念/基础操作）](