k8s 部署 canal 集群,RocketMQ 模式

k8s 部署 canal 集群,RocketMQ 模式

  • [k8s 部署 canal 集群,RocketMQ 模式](#k8s 部署 canal 集群,RocketMQ 模式)
    • 前提
      • MySQL
      • RocketMQ
      • [制作 canal-admin、canal-server 镜像](#制作 canal-admin、canal-server 镜像)
    • [部署 zookeeper](#部署 zookeeper)
    • [部署 canal-admin](#部署 canal-admin)
    • [部署 canal-server](#部署 canal-server)
    • 测试

k8s 部署 canal 集群,RocketMQ 模式

前提

MySQL

开启 binlog 的 MySQL 服务器:192.168.59.1:3306

ini 复制代码
[mysqld]
log-bin=mysql-bin # 开启 binlog
binlog-format=ROW # 选择 ROW 模式
server_id=1 # 配置 MySQL replaction 需要定义,不要和 canal 的 slaveId 重复

具有 slave 权限的账号:用户名 canal,密码:canal

sql 复制代码
CREATE USER canal IDENTIFIED BY 'canal';  
-- GRANT SELECT, REPLICATION SLAVE, REPLICATION CLIENT ON *.* TO 'canal'@'%';
GRANT ALL PRIVILEGES ON *.* TO 'canal'@'%' ;
FLUSH PRIVILEGES;

RocketMQ

RocketMQ 地址:192.168.56.1:9876

创建 tipoc:canal_test

RocketMQ-Dashboard 地址:192.168.56.1:8080,便于查看消息

制作 canal-admin、canal-server 镜像

由于 canal:v1.1.7 有一个 bug com.alibaba.otter.canal.server.embedded.CanalServerWithEmbedded isn't start , please check · Issue #4930,而官方已经在 Release 1.1.8-alpha 中修复了这个问题,但没有发布官方的 docker 镜像,所以需要自己制作镜像。以下是我制作的镜像,已发布到 dockerhub,可直接使用

zhaobingshuang/canal-admin:v1.1.8-alpha
zhaobingshuang/canal-server:v1.1.8-alpha

部署 zookeeper

zk-configmap.yaml

数据目录配置为 /data

yaml 复制代码
apiVersion: v1
kind: ConfigMap
metadata:
  name: zk-configmap
data:
  "zoo.cfg": |
    tickTime=2000
    dataDir=/data
    clientPort=2181

zk-pvc.yaml

数据目录持久化

yaml 复制代码
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: zk-data-pvc
spec:
  resources:
    requests:
      storage: 2Gi
  accessModes:
    - ReadWriteOnce

zk-deployment.yaml

无状态服务

将 zk-data-pvc 挂载到数据目录 /data,将 zk-configmap 的 zoo.cfg 挂载到 /conf/zoo.cfg 文件

yaml 复制代码
apiVersion: apps/v1
kind: Deployment
metadata:
  name: zk
spec:
  replicas: 1
  selector:
    matchLabels:
      app: zk
  template:
    metadata:
      labels:
        app: zk
    spec:
      containers:
        - name: zk
          image: zookeeper
          ports:
            - containerPort: 2181
              name: serve
          volumeMounts:
            - mountPath: /data
              name: zk-persistent-storage
            - mountPath: /conf
              name: zk-config
              readOnly: true
      volumes:
        - name: zk-persistent-storage
          persistentVolumeClaim:
            claimName: zk-data-pvc
        - name: zk-config
          configMap:
            name: zk-configmap
            items:
              - key: zoo.cfg
                path: zoo.cfg

zk-service.yaml

yaml 复制代码
apiVersion: v1
kind: Service
metadata:
  name: zk
spec:
  selector:
    app: zk
  ports:
    - protocol: TCP
      port: 2181
      targetPort: 2181
  type: ClusterIP
  clusterIP: None

执行以下命令

shell 复制代码
kubectl apply -f zk-configmap.yaml
kubectl apply -f zk-pvc.yaml
kubectl apply -f zk-deployment.yaml
kubectl apply -f zk-service.yaml

打开 kubernetes-dashboard,进入 zk pod 验证。

执行 create /cluster_1 创建 znode,为之后 canal-admin 创建集群使用。

部署 canal-admin

执行 manager.sql 初始化数据库。

canal-admin-configmap.yaml

yaml 复制代码
apiVersion: v1
kind: ConfigMap
metadata:
  name: canal-admin-configmap
data:
  "server.port": "8089"
  "spring.datasource.address": "192.168.59.1:3306"
  "spring.datasource.database": "canal_manager"
  "spring.datasource.username": "canal"
  "spring.datasource.password": "canal"
  "canal.adminPasswd": "admin"

canal-admin-deployment.yaml

yaml 复制代码
apiVersion: apps/v1
kind: Deployment
metadata:
  name: canal-admin
spec:
  selector:
    matchLabels:
      app: canal-admin
  template:
    metadata:
      labels:
        app: canal-admin
    spec:
      containers:
        - name: canal-admin
          image: zhaobingshuang/canal-admin:v1.1.8-alpha
          ports:
            - containerPort: 8089
          envFrom:
            - configMapRef:
                name: canal-admin-configmap

canal-admin-service.yaml

yaml 复制代码
apiVersion: v1
kind: Service
metadata:
  name: canal-admin
spec:
  selector:
    app: canal-admin
  ports:
    - protocol: TCP
      port: 8089
      targetPort: 8089
  type: ClusterIP

canal-admin-ingress.yaml

yaml 复制代码
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: canal-admin
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /
spec:
  ingressClassName: nginx
  rules:
    - host: canal-admin.com
      http:
        paths:
        - path: /
          pathType: Prefix
          backend:
            service:
              name: canal-admin
              port:
                number: 8089

执行以下命令

shell 复制代码
kubectl apply -f canal-admin-configmap.yaml
kubectl apply -f canal-admin-deployment.yaml
kubectl apply -f canal-admin-service.yaml
kubectl apply -f canal-admin-ingress.yaml

zhaobingshuang/canal-admin:v1.1.8-alpha 镜像比较大,需要等待一段时间等待,等 pod canal-admin 启动后,浏览器访问 http://canal-admin.com/。

用户名:admin,密码:123456

创建集群,集群名称:cluster_1 ,ZK 地址:zk:2181/cluster_1

点击主配置 -> 载入模板

修改以下配置:

properties 复制代码
canal.zkServers =zk:2181/cluster_1
canal.serverMode = rocketMQ
canal.instance.global.spring.xml = classpath:spring/default-instance.xml

rocketmq.namesrv.addr = 192.168.56.1:9876
rocketmq.tag =canal

部署 canal-server

canal-server-configmap.yaml

yaml 复制代码
apiVersion: v1
kind: ConfigMap
metadata:
  name: canal-server-configmap
data:
  canal.admin.manager: "canal-admin:8089"
  canal.admin.port: "11110"
  canal.admin.user: "admin"
  canal.admin.passwd: "4ACFE3202A5FF5CF467898FC58AAB1D615029441"
  canal.admin.register.auto: "true"
  canal.admin.register.cluster: "cluster_1"
  

canal-server-service.yaml

yaml 复制代码
apiVersion: v1
kind: Service
metadata:
  name: canal-server-headless
spec:
  selector:
    app: canal-server
  ports:
    - port: 11110
      name: admin
    - port: 11111
      name: tcp
    - port: 11112
      name: metric
  type: ClusterIP
  clusterIP: None

canal-server-statefulset.yaml

yaml 复制代码
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: canal-server
spec:
  replicas: 1
  selector:
    matchLabels:
      app: canal-server
  serviceName: canal-server-headless
  template:
    metadata:
      labels:
        app: canal-server
    spec:
      containers:
        - name: canal-server
          image: zhaobingshuang/canal-server:v1.1.8-alpha
          envFrom:
            - configMapRef:
                name: canal-server-configmap
          env:
            - name: POD_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: "metadata.namespace"
            - name: POD_HOST
              valueFrom:
                fieldRef:
                  fieldPath: "metadata.name"
            - name: RAW_NAME
              valueFrom:
                fieldRef:
                  fieldPath: "metadata.labels['app']"
            - name: canal.register.ip
              value: $(POD_HOST).$(RAW_NAME)-headless.$(POD_NAMESPACE)
            - name: canal.admin.register.name
              value: $(POD_HOST)
          ports:
            - containerPort: 11110
              name: admin
            - containerPort: 11111
              name: tcp
            - containerPort: 11112
              name: metric

执行以下命令:

shell 复制代码
kubectl apply -f canal-server-configmap.yaml
kubectl apply -f canal-server-service.yaml
kubectl apply -f canal-server-statefulset.yaml

等 pod canal-server启动后,查看 Server 管理

测试

载入模板,修改以下配置之后保存:

properties 复制代码
canal.instance.master.address=192.168.59.1:3306
canal.instance.dbUsername=canal
canal.instance.dbPassword=canal
canal.instance.filter.regex=test\\..*
canal.mq.topic=canal_test

启动实例

执行以下 SQL,创建数据库、创建表、插入数据:

sql 复制代码
CREATE DATABASE IF NOT EXISTS test;
USE test;
CREATE TABLE `table_1` (
  `id` int NOT NULL,
  `name` varchar(50) DEFAULT NULL,
  PRIMARY KEY (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_0900_ai_ci;
INSERT INTO table_1 VALUES(1, 'tom');
INSERT INTO table_1 VALUES(2, 'jack')

浏览器访问 http://192.168.56.1:8080/ 打开 RocketMQ-Dashboard 查看 canal_test 的消息:

相关推荐
懒洋洋大魔王33 分钟前
RocketMQ的使⽤
java·rocketmq·java-rocketmq
wuxingge3 小时前
k8s1.30.0高可用集群部署
云原生·容器·kubernetes
志凌海纳SmartX4 小时前
趋势洞察|AI 能否带动裸金属 K8s 强势崛起?
云原生·容器·kubernetes
锅总4 小时前
nacos与k8s service健康检查详解
云原生·容器·kubernetes
BUG弄潮儿4 小时前
k8s 集群安装
云原生·容器·kubernetes
颜淡慕潇6 小时前
【K8S系列】kubectl describe pod显示ImagePullBackOff,如何进一步排查?
后端·云原生·容器·kubernetes
Linux运维日记6 小时前
k8s1.31版本最新版本集群使用容器镜像仓库Harbor
linux·docker·云原生·容器·kubernetes
AI_小站8 小时前
RAG 示例:使用 langchain、Redis、llama.cpp 构建一个 kubernetes 知识库问答
人工智能·程序人生·langchain·kubernetes·llama·知识库·rag
长囧鹿12 小时前
云原生之k8s服务管理
云原生·容器·kubernetes
怡雪~12 小时前
centos7.9搭建k8s集群
云原生·容器·kubernetes