k8s 部署 canal 集群,RocketMQ 模式

k8s 部署 canal 集群,RocketMQ 模式

  • [k8s 部署 canal 集群,RocketMQ 模式](#k8s 部署 canal 集群,RocketMQ 模式)
    • 前提
      • MySQL
      • RocketMQ
      • [制作 canal-admin、canal-server 镜像](#制作 canal-admin、canal-server 镜像)
    • [部署 zookeeper](#部署 zookeeper)
    • [部署 canal-admin](#部署 canal-admin)
    • [部署 canal-server](#部署 canal-server)
    • 测试

k8s 部署 canal 集群,RocketMQ 模式

前提

MySQL

开启 binlog 的 MySQL 服务器:192.168.59.1:3306

ini 复制代码
[mysqld]
log-bin=mysql-bin # 开启 binlog
binlog-format=ROW # 选择 ROW 模式
server_id=1 # 配置 MySQL replaction 需要定义,不要和 canal 的 slaveId 重复

具有 slave 权限的账号:用户名 canal,密码:canal

sql 复制代码
CREATE USER canal IDENTIFIED BY 'canal';  
-- GRANT SELECT, REPLICATION SLAVE, REPLICATION CLIENT ON *.* TO 'canal'@'%';
GRANT ALL PRIVILEGES ON *.* TO 'canal'@'%' ;
FLUSH PRIVILEGES;

RocketMQ

RocketMQ 地址:192.168.56.1:9876

创建 tipoc:canal_test

RocketMQ-Dashboard 地址:192.168.56.1:8080,便于查看消息

制作 canal-admin、canal-server 镜像

由于 canal:v1.1.7 有一个 bug com.alibaba.otter.canal.server.embedded.CanalServerWithEmbedded isn't start , please check · Issue #4930,而官方已经在 Release 1.1.8-alpha 中修复了这个问题,但没有发布官方的 docker 镜像,所以需要自己制作镜像。以下是我制作的镜像,已发布到 dockerhub,可直接使用

zhaobingshuang/canal-admin:v1.1.8-alpha
zhaobingshuang/canal-server:v1.1.8-alpha

部署 zookeeper

zk-configmap.yaml

数据目录配置为 /data

yaml 复制代码
apiVersion: v1
kind: ConfigMap
metadata:
  name: zk-configmap
data:
  "zoo.cfg": |
    tickTime=2000
    dataDir=/data
    clientPort=2181

zk-pvc.yaml

数据目录持久化

yaml 复制代码
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: zk-data-pvc
spec:
  resources:
    requests:
      storage: 2Gi
  accessModes:
    - ReadWriteOnce

zk-deployment.yaml

无状态服务

将 zk-data-pvc 挂载到数据目录 /data,将 zk-configmap 的 zoo.cfg 挂载到 /conf/zoo.cfg 文件

yaml 复制代码
apiVersion: apps/v1
kind: Deployment
metadata:
  name: zk
spec:
  replicas: 1
  selector:
    matchLabels:
      app: zk
  template:
    metadata:
      labels:
        app: zk
    spec:
      containers:
        - name: zk
          image: zookeeper
          ports:
            - containerPort: 2181
              name: serve
          volumeMounts:
            - mountPath: /data
              name: zk-persistent-storage
            - mountPath: /conf
              name: zk-config
              readOnly: true
      volumes:
        - name: zk-persistent-storage
          persistentVolumeClaim:
            claimName: zk-data-pvc
        - name: zk-config
          configMap:
            name: zk-configmap
            items:
              - key: zoo.cfg
                path: zoo.cfg

zk-service.yaml

yaml 复制代码
apiVersion: v1
kind: Service
metadata:
  name: zk
spec:
  selector:
    app: zk
  ports:
    - protocol: TCP
      port: 2181
      targetPort: 2181
  type: ClusterIP
  clusterIP: None

执行以下命令

shell 复制代码
kubectl apply -f zk-configmap.yaml
kubectl apply -f zk-pvc.yaml
kubectl apply -f zk-deployment.yaml
kubectl apply -f zk-service.yaml

打开 kubernetes-dashboard,进入 zk pod 验证。

执行 create /cluster_1 创建 znode,为之后 canal-admin 创建集群使用。

部署 canal-admin

执行 manager.sql 初始化数据库。

canal-admin-configmap.yaml

yaml 复制代码
apiVersion: v1
kind: ConfigMap
metadata:
  name: canal-admin-configmap
data:
  "server.port": "8089"
  "spring.datasource.address": "192.168.59.1:3306"
  "spring.datasource.database": "canal_manager"
  "spring.datasource.username": "canal"
  "spring.datasource.password": "canal"
  "canal.adminPasswd": "admin"

canal-admin-deployment.yaml

yaml 复制代码
apiVersion: apps/v1
kind: Deployment
metadata:
  name: canal-admin
spec:
  selector:
    matchLabels:
      app: canal-admin
  template:
    metadata:
      labels:
        app: canal-admin
    spec:
      containers:
        - name: canal-admin
          image: zhaobingshuang/canal-admin:v1.1.8-alpha
          ports:
            - containerPort: 8089
          envFrom:
            - configMapRef:
                name: canal-admin-configmap

canal-admin-service.yaml

yaml 复制代码
apiVersion: v1
kind: Service
metadata:
  name: canal-admin
spec:
  selector:
    app: canal-admin
  ports:
    - protocol: TCP
      port: 8089
      targetPort: 8089
  type: ClusterIP

canal-admin-ingress.yaml

yaml 复制代码
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: canal-admin
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /
spec:
  ingressClassName: nginx
  rules:
    - host: canal-admin.com
      http:
        paths:
        - path: /
          pathType: Prefix
          backend:
            service:
              name: canal-admin
              port:
                number: 8089

执行以下命令

shell 复制代码
kubectl apply -f canal-admin-configmap.yaml
kubectl apply -f canal-admin-deployment.yaml
kubectl apply -f canal-admin-service.yaml
kubectl apply -f canal-admin-ingress.yaml

zhaobingshuang/canal-admin:v1.1.8-alpha 镜像比较大,需要等待一段时间等待,等 pod canal-admin 启动后,浏览器访问 http://canal-admin.com/。

用户名:admin,密码:123456

创建集群,集群名称:cluster_1 ,ZK 地址:zk:2181/cluster_1

点击主配置 -> 载入模板

修改以下配置:

properties 复制代码
canal.zkServers =zk:2181/cluster_1
canal.serverMode = rocketMQ
canal.instance.global.spring.xml = classpath:spring/default-instance.xml

rocketmq.namesrv.addr = 192.168.56.1:9876
rocketmq.tag =canal

部署 canal-server

canal-server-configmap.yaml

yaml 复制代码
apiVersion: v1
kind: ConfigMap
metadata:
  name: canal-server-configmap
data:
  canal.admin.manager: "canal-admin:8089"
  canal.admin.port: "11110"
  canal.admin.user: "admin"
  canal.admin.passwd: "4ACFE3202A5FF5CF467898FC58AAB1D615029441"
  canal.admin.register.auto: "true"
  canal.admin.register.cluster: "cluster_1"
  

canal-server-service.yaml

yaml 复制代码
apiVersion: v1
kind: Service
metadata:
  name: canal-server-headless
spec:
  selector:
    app: canal-server
  ports:
    - port: 11110
      name: admin
    - port: 11111
      name: tcp
    - port: 11112
      name: metric
  type: ClusterIP
  clusterIP: None

canal-server-statefulset.yaml

yaml 复制代码
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: canal-server
spec:
  replicas: 1
  selector:
    matchLabels:
      app: canal-server
  serviceName: canal-server-headless
  template:
    metadata:
      labels:
        app: canal-server
    spec:
      containers:
        - name: canal-server
          image: zhaobingshuang/canal-server:v1.1.8-alpha
          envFrom:
            - configMapRef:
                name: canal-server-configmap
          env:
            - name: POD_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: "metadata.namespace"
            - name: POD_HOST
              valueFrom:
                fieldRef:
                  fieldPath: "metadata.name"
            - name: RAW_NAME
              valueFrom:
                fieldRef:
                  fieldPath: "metadata.labels['app']"
            - name: canal.register.ip
              value: $(POD_HOST).$(RAW_NAME)-headless.$(POD_NAMESPACE)
            - name: canal.admin.register.name
              value: $(POD_HOST)
          ports:
            - containerPort: 11110
              name: admin
            - containerPort: 11111
              name: tcp
            - containerPort: 11112
              name: metric

执行以下命令:

shell 复制代码
kubectl apply -f canal-server-configmap.yaml
kubectl apply -f canal-server-service.yaml
kubectl apply -f canal-server-statefulset.yaml

等 pod canal-server启动后,查看 Server 管理

测试

载入模板,修改以下配置之后保存:

properties 复制代码
canal.instance.master.address=192.168.59.1:3306
canal.instance.dbUsername=canal
canal.instance.dbPassword=canal
canal.instance.filter.regex=test\\..*
canal.mq.topic=canal_test

启动实例

执行以下 SQL,创建数据库、创建表、插入数据:

sql 复制代码
CREATE DATABASE IF NOT EXISTS test;
USE test;
CREATE TABLE `table_1` (
  `id` int NOT NULL,
  `name` varchar(50) DEFAULT NULL,
  PRIMARY KEY (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_0900_ai_ci;
INSERT INTO table_1 VALUES(1, 'tom');
INSERT INTO table_1 VALUES(2, 'jack')

浏览器访问 http://192.168.56.1:8080/ 打开 RocketMQ-Dashboard 查看 canal_test 的消息:

相关推荐
景天科技苑8 小时前
【云原生开发】K8S多集群资源管理平台架构设计
云原生·容器·kubernetes·k8s·云原生开发·k8s管理系统
wclass-zhengge9 小时前
K8S篇(基本介绍)
云原生·容器·kubernetes
颜淡慕潇9 小时前
【K8S问题系列 |1 】Kubernetes 中 NodePort 类型的 Service 无法访问【已解决】
后端·云原生·容器·kubernetes·问题解决
昌sit!17 小时前
K8S node节点没有相应的pod镜像运行故障处理办法
云原生·容器·kubernetes
A ?Charis19 小时前
Gitlab-runner running on Kubernetes - hostAliases
容器·kubernetes·gitlab
北漂IT民工_程序员_ZG20 小时前
k8s集群安装(minikube)
云原生·容器·kubernetes
2301_806131361 天前
Kubernetes的基本构建块和最小可调度单元pod-0
云原生·容器·kubernetes
SilentCodeY1 天前
containerd配置私有仓库registry
容器·kubernetes·containerd·镜像·crictl
idealzouhu1 天前
【canal 中间件】canal 实时监听 binlog
mysql·canal
小安运维日记1 天前
Linux云计算 |【第五阶段】CLOUD-DAY8
linux·运维·docker·云计算·k8s·学习方法