Apache Flink 流处理-[CentOS|Rocky] 镜像

Flink Docker仓库包含了Dockerfiles用于为Flink构建docker images使用,这些 Dockerfile 由 Apache Flink 社区维护,但 Docker 社区负责在 Docker Hub 上构建和托管映像。目前市面上流行的Flink镜像都是基于Ubuntu镜像构建,由于项目需求变化,需要在容器中进行HDFS文件挂载(京东云海存储,适配了HDFS,目前驱动适配CentOS8)我们需要在CentOS8的镜像中重构Flink的官方镜像,一下示例按照Flink-1.19.0版本镜像制作过程示例如下:

CentOS8 镜像

shell 复制代码
FROM centos:centos8

# Install dependencies
RUN sed -i 's/mirrorlist/#mirrorlist/g' /etc/yum.repos.d/CentOS-*
RUN sed -i 's|#baseurl=http://mirror.centos.org|baseurl=http://vault.centos.org|g' /etc/yum.repos.d/CentOS-*
RUN set -ex; \
    dnf update -y; \
    dnf install -y epel-release; \
    dnf install -y gpg snappy gettext-devel jemalloc wget java-1.8.0-openjdk bind-utils; \
    yum clean all;

ENV GOSU_VERSION 1.17
ENV ARCH=arm64

RUN set -ex; \
  wget -nv -O /usr/local/bin/gosu "https://github.com/tianon/gosu/releases/download/$GOSU_VERSION/gosu-$ARCH"; \
  wget -nv -O /usr/local/bin/gosu.asc "https://github.com/tianon/gosu/releases/download/$GOSU_VERSION/gosu-$ARCH.asc"; \
  export GNUPGHOME="$(mktemp -d)"; \
  for server in ha.pool.sks-keyservers.net $(shuf -e \
                          hkp://p80.pool.sks-keyservers.net:80 \
                          keyserver.ubuntu.com \
                          hkp://keyserver.ubuntu.com:80 \
                          pgp.mit.edu) ; do \
      gpg --batch --keyserver "$server" --recv-keys B42F6819007F00F88E364FD4036A9C25BF357DD4 && break || : ; \
  done && \
  gpg --batch --verify /usr/local/bin/gosu.asc /usr/local/bin/gosu; \
  gpgconf --kill all; \
  rm -rf "$GNUPGHOME" /usr/local/bin/gosu.asc; \
  chmod +x /usr/local/bin/gosu; \
  gosu nobody true

# Configure Flink version
ENV FLINK_TGZ_URL=https://www.apache.org/dyn/closer.cgi?action=download&filename=flink/flink-1.19.0/flink-1.19.0-bin-scala_2.12.tgz \
    FLINK_ASC_URL=https://downloads.apache.org/flink/flink-1.19.0/flink-1.19.0-bin-scala_2.12.tgz.asc \
    GPG_KEY=028B6605F51BC296B56A5042E57D30ABEE75CA06 \
    CHECK_GPG=true

# Prepare environment
ENV FLINK_HOME=/opt/flink
ENV PATH=$FLINK_HOME/bin:$PATH
RUN groupadd --system --gid=9999 flink && \
    useradd --system --home-dir $FLINK_HOME --uid=9999 --gid=flink flink
WORKDIR $FLINK_HOME

# Install Flink
RUN set -ex; \
  wget -nv -O flink.tgz "$FLINK_TGZ_URL"; \
  \
  if [ "$CHECK_GPG" = "true" ]; then \
    wget -nv -O flink.tgz.asc "$FLINK_ASC_URL"; \
    export GNUPGHOME="$(mktemp -d)"; \
    for server in ha.pool.sks-keyservers.net $(shuf -e \
                            hkp://p80.pool.sks-keyservers.net:80 \
                            keyserver.ubuntu.com \
                            hkp://keyserver.ubuntu.com:80 \
                            pgp.mit.edu) ; do \
        gpg --batch --keyserver "$server" --recv-keys "$GPG_KEY" && break || : ; \
    done && \
    gpg --batch --verify flink.tgz.asc flink.tgz; \
    gpgconf --kill all; \
    rm -rf "$GNUPGHOME" flink.tgz.asc; \
  fi; \
  \
  tar -xf flink.tgz --strip-components=1; \
  rm flink.tgz; \
  \
  chown -R flink:flink .; \
  \
  # Replace default REST/RPC endpoint bind address to use the container's network interface \
  CONF_FILE="$FLINK_HOME/conf/flink-conf.yaml"; \
  if [ ! -e "$FLINK_HOME/conf/flink-conf.yaml" ]; then \
    CONF_FILE="${FLINK_HOME}/conf/config.yaml"; \
    /bin/bash "$FLINK_HOME/bin/config-parser-utils.sh" "${FLINK_HOME}/conf" "${FLINK_HOME}/bin" "${FLINK_HOME}/lib" \
        "-repKV" "rest.address,localhost,0.0.0.0" \
        "-repKV" "rest.bind-address,localhost,0.0.0.0" \
        "-repKV" "jobmanager.bind-host,localhost,0.0.0.0" \
        "-repKV" "taskmanager.bind-host,localhost,0.0.0.0" \
        "-rmKV" "taskmanager.host=localhost"; \
  else \
    sed -i 's/rest.address: localhost/rest.address: 0.0.0.0/g' "$CONF_FILE"; \
    sed -i 's/rest.bind-address: localhost/rest.bind-address: 0.0.0.0/g' "$CONF_FILE"; \
    sed -i 's/jobmanager.bind-host: localhost/jobmanager.bind-host: 0.0.0.0/g' "$CONF_FILE"; \
    sed -i 's/taskmanager.bind-host: localhost/taskmanager.bind-host: 0.0.0.0/g' "$CONF_FILE"; \
    sed -i '/taskmanager.host: localhost/d' "$CONF_FILE"; \
  fi;

# Configure container
COPY docker-entrypoint.sh /
ENTRYPOINT ["/docker-entrypoint.sh"]
EXPOSE 6123 8081
CMD ["help"]

需要注意小编的电脑是Apple M1 Pro,因此在进行镜像编译的时候,需要将ARCH环境变量修改为arm64,如果是Intel芯片的x86架构,请将ARCH修改为amd64即可,修改结束后可以执行一下命令实现Flink镜像打包等工作:

shell 复制代码
jiangzhongzhou@ZBMac-C02CW08SM scala_2.12-java8-centos % docker build -t flink:1.19-scala_2.12-centos8-java8 .
[+] Building 311.6s (14/14) FINISHED                                                                                                                                                                                                                                                            docker:desktop-linux
 => [internal] load .dockerignore                                                                                                                                                                                                                                                                               0.0s
 => => transferring context: 2B                                                                                                                                                                                                                                                                                 0.0s
 => [internal] load build definition from Dockerfile                                                                                                                                                                                                                                                            0.0s
 => => transferring dockerfile: 4.89kB                                                                                                                                                                                                                                                                          0.0s
 => [internal] load metadata for docker.io/library/centos:centos8                                                                                                                                                                                                                                              15.3s
 => [1/9] FROM docker.io/library/centos:centos8@sha256:a27fd8080b517143cbbbab9dfb7c8571c40d67d534bbdee55bd6c473f432b177                                                                                                                                                                                         0.0s
 => [internal] load build context                                                                                                                                                                                                                                                                               0.0s
 => => transferring context: 42B                                                                                                                                                                                                                                                                                0.0s
 => CACHED [2/9] RUN sed -i 's/mirrorlist/#mirrorlist/g' /etc/yum.repos.d/CentOS-*                                                                                                                                                                                                                              0.0s
 => CACHED [3/9] RUN sed -i 's|#baseurl=http://mirror.centos.org|baseurl=http://vault.centos.org|g' /etc/yum.repos.d/CentOS-*                                                                                                                                                                                   0.0s
 => CACHED [4/9] RUN set -ex;     dnf install -y epel-release;     dnf update -y;     dnf install -y gpg snappy gettext-devel jemalloc wget java-1.8.0-openjdk bind-utils;     yum clean all;                                                                                                                   0.0s
 => [5/9] RUN set -ex;   wget -nv -O /usr/local/bin/gosu "https://github.com/tianon/gosu/releases/download/1.17/gosu-arm64";   wget -nv -O /usr/local/bin/gosu.asc "https://github.com/tianon/gosu/releases/download/1.17/gosu-arm64.asc";   export GNUPGHOME="$(mktemp -d)";   for server in ha.pool.sks-key  18.9s
 => [6/9] RUN groupadd --system --gid=9999 flink &&     useradd --system --home-dir /opt/flink --uid=9999 --gid=flink flink                                                                                                                                                                                     0.2s
 => [7/9] WORKDIR /opt/flink                                                                                                                                                                                                                                                                                    0.0s
 => [8/9] RUN set -ex;   wget -nv -O flink.tgz "https://www.apache.org/dyn/closer.cgi?action=download&filename=flink/flink-1.19.0/flink-1.19.0-bin-scala_2.12.tgz";     if [ "true" = "true" ]; then     wget -nv -O flink.tgz.asc "https://downloads.apache.org/flink/flink-1.19.0/flink-1.19.0-bin-scala_2  276.1s
 => [9/9] COPY docker-entrypoint.sh /                                                                                                                                                                                                                                                                           0.0s
 => exporting to image                                                                                                                                                                                                                                                                                          0.9s
 => => exporting layers                                                                                                                                                                                                                                                                                         0.9s
 => => writing image sha256:ea57baecdc1d1fa59ac697062c4a59b3d0ed3abfc698b0f4ecfa4f6692009810                                                                                                                                                                                                                    0.0s
 => => naming to docker.io/library/flink:1.19-scala_2.12-centos8-java8                                                                                                                                                                                                                                          0.0s

What's Next?
  View a summary of image vulnerabilities and recommendations → docker scout quickview

Rocky 镜像

shell 复制代码
FROM arm64v8/rockylinux:9.3

# Install dependencies
RUN set -ex; \
    dnf update -y; \
    dnf install -y epel-release; \
    dnf install -y gpg snappy gettext-devel jemalloc wget java-1.8.0-openjdk bind-utils; \
    dnf clean all;

ENV GOSU_VERSION 1.17
ENV ARCH=arm64

RUN set -ex; \
  wget -nv -O /usr/local/bin/gosu "https://github.com/tianon/gosu/releases/download/$GOSU_VERSION/gosu-$ARCH"; \
  wget -nv -O /usr/local/bin/gosu.asc "https://github.com/tianon/gosu/releases/download/$GOSU_VERSION/gosu-$ARCH.asc"; \
  export GNUPGHOME="$(mktemp -d)"; \
  for server in ha.pool.sks-keyservers.net $(shuf -e \
                          hkp://p80.pool.sks-keyservers.net:80 \
                          keyserver.ubuntu.com \
                          hkp://keyserver.ubuntu.com:80 \
                          pgp.mit.edu) ; do \
      gpg --batch --keyserver "$server" --recv-keys B42F6819007F00F88E364FD4036A9C25BF357DD4 && break || : ; \
  done && \
  gpg --batch --verify /usr/local/bin/gosu.asc /usr/local/bin/gosu; \
  gpgconf --kill all; \
  rm -rf "$GNUPGHOME" /usr/local/bin/gosu.asc; \
  chmod +x /usr/local/bin/gosu; \
  gosu nobody true

# Configure Flink version
ENV FLINK_TGZ_URL=https://www.apache.org/dyn/closer.cgi?action=download&filename=flink/flink-1.19.0/flink-1.19.0-bin-scala_2.12.tgz \
    FLINK_ASC_URL=https://downloads.apache.org/flink/flink-1.19.0/flink-1.19.0-bin-scala_2.12.tgz.asc \
    GPG_KEY=028B6605F51BC296B56A5042E57D30ABEE75CA06 \
    CHECK_GPG=true

# Prepare environment
ENV FLINK_HOME=/opt/flink
ENV PATH=$FLINK_HOME/bin:$PATH
RUN groupadd --system --gid=9999 flink && \
    useradd --system --home-dir $FLINK_HOME --uid=9999 --gid=flink flink
WORKDIR $FLINK_HOME

# Install Flink
RUN set -ex; \
  wget -nv -O flink.tgz "$FLINK_TGZ_URL"; \
  \
  if [ "$CHECK_GPG" = "true" ]; then \
    wget -nv -O flink.tgz.asc "$FLINK_ASC_URL"; \
    export GNUPGHOME="$(mktemp -d)"; \
    for server in ha.pool.sks-keyservers.net $(shuf -e \
                            hkp://p80.pool.sks-keyservers.net:80 \
                            keyserver.ubuntu.com \
                            hkp://keyserver.ubuntu.com:80 \
                            pgp.mit.edu) ; do \
        gpg --batch --keyserver "$server" --recv-keys "$GPG_KEY" && break || : ; \
    done && \
    gpg --batch --verify flink.tgz.asc flink.tgz; \
    gpgconf --kill all; \
    rm -rf "$GNUPGHOME" flink.tgz.asc; \
  fi; \
  \
  tar -xf flink.tgz --strip-components=1; \
  rm flink.tgz; \
  \
  chown -R flink:flink .; \
  \
  # Replace default REST/RPC endpoint bind address to use the container's network interface \
  CONF_FILE="$FLINK_HOME/conf/flink-conf.yaml"; \
  if [ ! -e "$FLINK_HOME/conf/flink-conf.yaml" ]; then \
    CONF_FILE="${FLINK_HOME}/conf/config.yaml"; \
    /bin/bash "$FLINK_HOME/bin/config-parser-utils.sh" "${FLINK_HOME}/conf" "${FLINK_HOME}/bin" "${FLINK_HOME}/lib" \
        "-repKV" "rest.address,localhost,0.0.0.0" \
        "-repKV" "rest.bind-address,localhost,0.0.0.0" \
        "-repKV" "jobmanager.bind-host,localhost,0.0.0.0" \
        "-repKV" "taskmanager.bind-host,localhost,0.0.0.0" \
        "-rmKV" "taskmanager.host=localhost"; \
  else \
    sed -i 's/rest.address: localhost/rest.address: 0.0.0.0/g' "$CONF_FILE"; \
    sed -i 's/rest.bind-address: localhost/rest.bind-address: 0.0.0.0/g' "$CONF_FILE"; \
    sed -i 's/jobmanager.bind-host: localhost/jobmanager.bind-host: 0.0.0.0/g' "$CONF_FILE"; \
    sed -i 's/taskmanager.bind-host: localhost/taskmanager.bind-host: 0.0.0.0/g' "$CONF_FILE"; \
    sed -i '/taskmanager.host: localhost/d' "$CONF_FILE"; \
  fi;

# Configure container
COPY docker-entrypoint.sh /
ENTRYPOINT ["/docker-entrypoint.sh"]
EXPOSE 6123 8081
CMD ["help"]

需要注意小编的电脑是Apple M1 Pro,因此在进行镜像编译的时候,需要将ARCH环境变量修改为arm64,如果是Intel芯片的x86架构,请将ARCH修改为amd64即可,修改结束后可以执行一下命令实现Flink镜像打包等工作:

shell 复制代码
jiangzhongzhou@ZBMac-C02CW08SM scala_2.12-java8-rocky % docker build -t flink:1.19-scala_2.12-rocky-java8 .
[+] Building 297.9s (12/12) FINISHED                                                                                                                                                                                                                                                            docker:desktop-linux
 => [internal] load .dockerignore                                                                                                                                                                                                                                                                               0.0s
 => => transferring context: 2B                                                                                                                                                                                                                                                                                 0.0s
 => [internal] load build definition from Dockerfile                                                                                                                                                                                                                                                            0.0s
 => => transferring dockerfile: 4.72kB                                                                                                                                                                                                                                                                          0.0s
 => [internal] load metadata for docker.io/arm64v8/rockylinux:9.3                                                                                                                                                                                                                                               0.0s
 => [1/7] FROM docker.io/arm64v8/rockylinux:9.3                                                                                                                                                                                                                                                                 0.0s
 => [internal] load build context                                                                                                                                                                                                                                                                               0.0s
 => => transferring context: 42B                                                                                                                                                                                                                                                                                0.0s
 => CACHED [2/7] RUN set -ex;     dnf update -y;     dnf install -y epel-release;     dnf install -y gpg snappy gettext-devel jemalloc wget java-1.8.0-openjdk bind-utils;     dnf clean all;                                                                                                                   0.0s
 => [3/7] RUN set -ex;   wget -nv -O /usr/local/bin/gosu "https://github.com/tianon/gosu/releases/download/1.17/gosu-arm64";   wget -nv -O /usr/local/bin/gosu.asc "https://github.com/tianon/gosu/releases/download/1.17/gosu-arm64.asc";   export GNUPGHOME="$(mktemp -d)";   for server in ha.pool.sks-key  16.0s
 => [4/7] RUN groupadd --system --gid=9999 flink &&     useradd --system --home-dir /opt/flink --uid=9999 --gid=flink flink                                                                                                                                                                                     0.3s 
 => [5/7] WORKDIR /opt/flink                                                                                                                                                                                                                                                                                    0.0s 
 => [6/7] RUN set -ex;   wget -nv -O flink.tgz "https://www.apache.org/dyn/closer.cgi?action=download&filename=flink/flink-1.19.0/flink-1.19.0-bin-scala_2.12.tgz";     if [ "true" = "true" ]; then     wget -nv -O flink.tgz.asc "https://downloads.apache.org/flink/flink-1.19.0/flink-1.19.0-bin-scala_2  277.3s 
 => [7/7] COPY docker-entrypoint.sh /                                                                                                                                                                                                                                                                           0.0s 
 => exporting to image                                                                                                                                                                                                                                                                                          4.1s 
 => => exporting layers                                                                                                                                                                                                                                                                                         4.1s 
 => => writing image sha256:eb38aa8f37e1a920e46efd4792d5803fed9ba95698e3d465ed365f98b67cad30                                                                                                                                                                                                                    0.0s 
 => => naming to docker.io/library/flink:1.19-scala_2.12-rocky-java8                                                                                                                                                                                                                                            0.0s 
                                                                                                                                                                                                                                                                                                                     
What's Next?
  View a summary of image vulnerabilities and recommendations → docker scout quickview
相关推荐
上辈子杀猪这辈子学IT1 小时前
【Zookeeper集群搭建】安装zookeeper、zookeeper集群配置、zookeeper启动与关闭、zookeeper的shell命令操作
linux·hadoop·zookeeper·centos·debian
minihuabei1 小时前
linux centos 安装redis
linux·redis·centos
W Y4 小时前
【架构-37】Spark和Flink
架构·flink·spark
ycsdn104 小时前
Caused by: org.apache.flink.api.common.io.ParseException: Row too short:
大数据·flink
Diamond技术流16 小时前
从0开始学习Linux——远程连接工具
linux·学习·centos·ssh·xshell·ftp
188_djh1 天前
# Python基础到实战一飞冲天(一)--linux基础(一)
linux·开发语言·python·ubuntu·centos·os·operationsystem
晨航1 天前
Flink本地模式安装详解
大数据·flink
sf_www1 天前
flink 内存配置(五):网络缓存调优
大数据·网络·flink
我的K84091 天前
Flink滑动窗口(Sliding)中window和windowAll的区别
flink
inori12561 天前
FlinkSql读取外部Mysql和HBase数据库的方法(scala)
mysql·flink·hbase