通过 docker-compose 部署 Flink

概要

通过 docker-compose 以 Session Mode 部署 flink

前置依赖

  • Docker、docker-compose
  • flink 客户端
  • docker-compose.yml
yaml 复制代码
version: "2.2"
services:
  jobmanager:
    image: flink:1.17.2
    ports:
      - "8081:8081"
    command: jobmanager
    volumes:
      - ${PWD}/checkpoint:/opt/flink/checkpoint
      - ${PWD}/savepoint:/opt/flink/savepoint
    environment:
      - |
        FLINK_PROPERTIES=
        jobmanager.rpc.address: jobmanager

  taskmanager:
    image: flink:1.17.2
    depends_on:
      - jobmanager
    command: taskmanager
    scale: 1
    volumes:
      - ${PWD}/checkpoint:/opt/flink/checkpoint
      - ${PWD}/savepoint:/opt/flink/savepoint
    environment:
      - |
        FLINK_PROPERTIES=
        jobmanager.rpc.address: jobmanager
        taskmanager.numberOfTaskSlots: 3

部署

shell 复制代码
# 下载 docker、docker-compose

# docker-compose 部署
# 复制上述 docker-compose.yml 内容
$ vim docker-compose.yml

# 创建 savepoint、checkpoint 目录用于保存状态数据
$ mkdir savepoint checkpoint

$ docker-compose up -d

提交任务

shell 复制代码
# 下载 flink 客户端
$ wget https://archive.apache.org/dist/flink/flink-1.17.0/flink-1.17.0-bin-scala_2.12.tgz
$ tar xvf flink-1.17.0-bin-scala_2.12.tgz
$ cd flink-1.17.0-bin-scala_2.12

# 提交任务
$ bin/flink run -d -m 192.168.56.103:8081 -c com.xchenhao.Demo flink-cdc-1.0-SNAPSHOT-jar-with-dependencies.jar

# 查看任务列表
$ bin/flink list -m 192.168.56.103:8081
Waiting for response...
------------------ Running/Restarting Jobs -------------------
02.02.2024 16:45:01 : 9f70a7c1430d609eaa3434c4f30d1e93 : Flink-CDC (RUNNING)
--------------------------------------------------------------
No scheduled jobs

Web UI 查看

浏览器访问目标端口 192.168.56.103:8081

其它常用命令

  • 从指定 checkpoint 恢复任务
shell 复制代码
bin/flink run -d -m 192.168.56.103:8081 -s file:///opt/flink/checkpoint/47bea16a8ffcb51beff21ed1c7297dc1/chk-129773 -c com.xchenhao.Demo flink-cdc-1.0-SNAPSHOT-jar-with-dependencies.jar
  • 从指定 savepoint 恢复任务
shell 复制代码
bin/flink run -d -m 192.168.56.103:8081 -s file:///opt/flink/savepoint/savepoint-63712d-d7cbfee1ede9  -c com.xchenhao.Demo flink-cdc-1.0-SNAPSHOT-jar-with-dependencies.jar
  • 保存 savepoint
shell 复制代码
bin/flink savepoint 9f70a7c1430d609eaa3434c4f30d1e93 file:///opt/flink/savepoint -m 192.168.56.103:8081
  • 取消任务
shell 复制代码
bin/flink cancel 9f70a7c1430d609eaa3434c4f30d1e93 -m 192.168.56.103:8081
  • 停止任务并保存 savepoint
shell 复制代码
bin/flink stop 9f70a7c1430d609eaa3434c4f30d1e93 -p file:///opt/flink/savepoint -m 192.168.56.103:8081

参考

相关推荐
今晚务必早点睡23 分钟前
从零到上线:Docker、Docker Compose 与 Runtime 安装部署全指南(含实战示例与应用场景)
运维·docker·容器
zyplanke2 小时前
Kubernetes(四):Service
云原生·容器·kubernetes·k8s
Android小码家4 小时前
Vscode + docker + qt 网络监听小工具
vscode·qt·docker
虚伪的空想家4 小时前
K8S删除命名空间卡住一直Terminating状态
云原生·容器·kubernetes·删除·卡顿·delete·命名空间
衍余未了6 小时前
k8s除了主server服务器可正常使用kubectl命令,其他节点不能使用原因,以及如何在其他k8s节点正常使用kubectl命令??
云原生·容器·kubernetes
Clownseven6 小时前
Mattermost教程:用Docker搭建自己的开源Slack替代品 (团队聊天)
docker·容器·开源
❀͜͡傀儡师6 小时前
Docker部署Drawnix开源白板工具
docker·容器·开源·drawnix
❀͜͡傀儡师6 小时前
Docker部署Lunalytics开源监控工具
docker·容器·开源·lunalytics
To_再飞行6 小时前
K8s 存储配置资源
linux·云原生·容器·kubernetes
鸿儒之观8 小时前
hadoop 框架 jar下载
大数据·hadoop·jar