docker 部署Flink和传统部署

简单介绍

Flink 是一个开源的、分布式的、高性能的流处理框架，主要特性有：状态管理与容错，事件时间与水位线，先进的窗口模型，轻量级的分布式快照，端到端的精确一次交付，流批一体与统一的 API，复杂的流处理拓扑与高级 API（侧输出，CEP：复杂事件处理库，Table API & SQL）等

下载地址: https://mirrors.aliyun.com/apache/flink/ z根据需求选版本

下载安装包

bash 复制代码

https://mirrors.aliyun.com/apache/flink/flink-1.20.3/flink-1.20.3-bin-scala_2.12.tgz

解压

bash 复制代码

tra -zxf flink-1.20.3-bin-scala_2.12.tgz

修改配置文件

bash 复制代码

jobmanager.rpc.address: master
jobmanager.rpc.port: 6123
jobmanager.bind-host: 0.0.0.0
jobmanager.memory.process.size: 1600m
taskmanager.bind-host: 0.0.0.0
taskmanager.host: master
taskmanager.memory.process.size: 1728m
taskmanager.numberOfTaskSlots: 2
parallelism.default: 2
jobmanager.execution.failover-strategy: region
rest.address: localhost
rest.bind-address: 0.0.0.0

启动

bash 复制代码

bin/start-cluster.sh

登录首页： http://192.168.115.102:8081

Docker Run

拉取镜像

bash 复制代码

x86架构
docker pull registry.cn-hangzhou.aliyuncs.com/qiluo-images/flink
arm架构
docker pull registry.cn-hangzhou.aliyuncs.com/qiluo-images/linux_arm64_flink

创建网络

bash 复制代码

docker network create flink-network

启动JobManager

bash 复制代码

docker run -d \
  --name=flink-jobmanager \
  --network=flink-network \
  -p 8081:8081 \
  -e FLINK_PROPERTIES="jobmanager.rpc.address: flink-jobmanager" \
  -v /data/flink:/opt/flink/data \
  registry.cn-hangzhou.aliyuncs.com/qiluo-images/flink \
  jobmanager

启动TaskManager

bash 复制代码

docker run -d \
  --name=flink-taskmanager \
  --network=flink-network \
  -e FLINK_PROPERTIES="jobmanager.rpc.address: flink-jobmanager" \
  -v /data/flink:/opt/flink/data \
  registry.cn-hangzhou.aliyuncs.com/qiluo-images/flink \
  taskmanager

带依赖的完整版本

docker-compose-full.yml

bash 复制代码

version: "3.8"
services:
  jobmanager:
    image: registry.cn-hangzhou.aliyuncs.com/qiluo-images/flink
    container_name: jobmanager
    ports:
      - "8081:8081"
      - "6123:6123"
    command: jobmanager
    environment:
      - FLINK_PROPERTIES=jobmanager.rpc.address: jobmanager
    volumes:
      - ./data/jobmanager:/opt/flink/data
      - ./logs/jobmanager:/opt/flink/log
      - ./apps:/opt/flink/apps
    networks:
      - flink-network

  taskmanager:
    image: flink:1.17.2
    container_name: taskmanager
    depends_on:
      - jobmanager
    command: taskmanager
    scale: 2
    environment:
      - FLINK_PROPERTIES=jobmanager.rpc.address: jobmanager
      - TASK_MANAGER_NUMBER_OF_TASK_SLOTS=4
    volumes:
      - ./data/taskmanager:/opt/flink/data
      - ./logs/taskmanager:/opt/flink/log
      - ./apps:/opt/flink/apps
    networks:
      - flink-network

  zookeeper:
    image: registry.cn-hangzhou.aliyuncs.com/qiluo-images/zookeeper
    container_name: zookeeper
    environment:
      - ALLOW_ANONYMOUS_LOGIN=yes
    ports:
      - "2181:2181"
    networks:
      - flink-network

  kafka:
    image: registry.cn-hangzhou.aliyuncs.com/qiluo-images/kafka
    container_name: kafka
    depends_on:
      - zookeeper
    environment:
      - KAFKA_CFG_ZOOKEEPER_CONNECT=zookeeper:2181
      - KAFKA_CFG_ADVERTISED_LISTENERS=PLAINTEXT://kafka:9092
      - ALLOW_PLAINTEXT_LISTENER=yes
    ports:
      - "9092:9092"
    networks:
      - flink-network

networks:
  flink-network:
    driver: bridge

volumes:
  flink-data:

使用SQL Client的完整环境

docker-compose-sql.yml

bash 复制代码

version: "3.8"
services:
  jobmanager:
    image: registry.cn-hangzhou.aliyuncs.com/qiluo-images/flink
    container_name: flink-jobmanager
    ports:
      - "8081:8081"
      - "6123:6123"
    command: jobmanager
    environment:
      - FLINK_PROPERTIES=
          jobmanager.rpc.address: jobmanager
          taskmanager.numberOfTaskSlots: 4
          parallelism.default: 2
    volumes:
      - ./data/jobmanager:/opt/flink/data
      - ./logs/jobmanager:/opt/flink/log
    networks:
      - flink-network

  taskmanager:
    image: registry.cn-hangzhou.aliyuncs.com/qiluo-images/flink
    container_name: flink-taskmanager
    depends_on:
      - jobmanager
    command: taskmanager
    scale: 2
    environment:
      - FLINK_PROPERTIES=
          jobmanager.rpc.address: jobmanager
          taskmanager.numberOfTaskSlots: 4
          parallelism.default: 2
    volumes:
      - ./data/taskmanager:/opt/flink/data
      - ./logs/taskmanager:/opt/flink/log
    networks:
      - flink-network

  sql-client:
    image: registry.cn-hangzhou.aliyuncs.com/qiluo-images/flink
    container_name: flink-sql-client
    depends_on:
      - jobmanager
    command: |
      bash -c "
      echo 'Waiting for Flink cluster to be ready...'
      while ! nc -z jobmanager 8081; do
        sleep 1
      done
      echo 'Starting SQL Client...'
      ./bin/sql-client.sh"
    volumes:
      - ./sql-scripts:/opt/flink/sql-scripts
    networks:
      - flink-network
    tty: true
    stdin_open: true

networks:
  flink-network:
    driver: bridge