canal同步mysql数据到elasticsearch

本文使用docker环境安装mysql、canal、elasticsearch，基于binlog利用canal实现mysql的数据同步到elasticsearch中。

实验中间件版本说明：

centos 8
mysql 5.7.36
es 7.16.2
cannal.server: 1.1.5
canal.adapter: 1.1.5
postman

1、安装docker

基本命令：

csharp 复制代码

#centos 7 安装 docker
yum install docker

#centos 8 安装docker
yum erase podman buildah 
yum install -y yum-utils
yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
yum install docker-ce docker-ce-cli containerd.io

#检验安装是否成功
[root@localhost opt]# docker --version
Docker version 20.10.12, build e91ed57

#启动
systemctl start docker

#换镜像源
sudo vim /etc/docker/daemon.json
内容如下：
{
 "registry-mirrors": ["https://m9r2r2uj.mirror.aliyuncs.com"]
}
保存退出，重启docker

#重启
sudo service docker restart

#列出镜像
docker images

#查看运行进程
docker ps

2、安装mysql

方法1

sql 复制代码

docker pull mysql:5.7.36
docker run --name mysql5736 -p 3306:3306 -e MYSQL_ROOT_PASSWORD=admin -d mysql:5.7.36

docker exec -it mysql5736 /bin/bash
apt-get update
apt-get install vim
cd /etc/mysql/mysql.conf.d
vim mysqld.cnf  // 修改mysql配置

查看配置情况

dart 复制代码

mysql -uroot -p
show master status  // binlog日志文件
reset master; // 重启日志

方法2 或者docker-compose-es -f docker-compose-mysql.yml up -d

yaml 复制代码

version: '3.9'
services:
    mysql:
        image: 'mysql:5.7.36'
        environment:
            - MYSQL_ROOT_PASSWORD=123456
        ports:
            - '3306:3306'
        volumes:
            - './mysql/mysql.conf.d/mysqld.cnf:/etc/mysql/mysql.conf.d/mysqld.cnf'
        container_name: mysql5736

新建文件 [root@c1 bk]# vi./mysql/mysql.conf.d/mysqld.cnf

ini 复制代码

# Copyright (c) 2014, 2021, Oracle and/or its affiliates.
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License, version 2.0,
# as published by the Free Software Foundation.
#
# This program is also distributed with certain software (including
# but not limited to OpenSSL) that is licensed under separate terms,
# as designated in a particular file or component or in included license
# documentation.  The authors of MySQL hereby grant you an additional
# permission to link the program and your derivative works with the
# separately licensed software that they have included with MySQL.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License, version 2.0, for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA  02110-1301 USA

#
# The MySQL  Server configuration file.
#
# For explanations see
# http://dev.mysql.com/doc/mysql/en/server-system-variables.html

[mysqld]
pid-file	= /var/run/mysqld/mysqld.pid
socket		= /var/run/mysqld/mysqld.sock
datadir		= /var/lib/mysql
#log-error	= /var/log/mysql/error.log
log-bin=/var/log/mysql/mysql-bin
binlog-format=ROW
server-id=1
# By default we only accept connections from localhost
#bind-address	= 127.0.0.1
# Disabling symbolic-links is recommended to prevent assorted security risks
symbolic-links=0

查看日志文件：

c 复制代码

cd /var/log/mysql/  // 进入日志文件目录
mysqlbinlog -vv mysql-bin.000001  // row格式查看日志

3、安装es

方法1

arduino 复制代码

docker pull elasticsearch:7.16.2
docker run -p 9200:9200 -p 9300:9300 -e "discovery.type=single-node" --name='es7162' -d elasticsearch:7.16.2

然后我们需要配置一下es的信息：

bash 复制代码

docker exec -ites es7162 /bin/bash
cd config
vi elasticsearch.yml

配置文件：

yaml 复制代码

cluster.name: dailyhub-es
network.host: 0.0.0.0

node.name: node-1
http.port: 9200
http.cors.enabled: true
http.cors.allow-origin: "*"
node.master: true
node.data: true

方法2 或者 [root@c1 bk]# docker-compose-es -f docker-compose-es.yml up -d

vbnet 复制代码

version: '3.9'
services:
    elasticsearch:
        image: 'elasticsearch:7.16.2'
        container_name: es7162
        environment:
            - discovery.type=single-node
        volumes:
            - './elasticsearch/config/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml'
        ports:
            - '9300:9300'
            - '9200:9200'

配置文件

yaml 复制代码

[root@c1 bk]# cat elasticsearch/config/elasticsearch.yml 
cluster.name: "docker-cluster"
network.host: 0.0.0.0
node.name: node-1
http.port: 9200
http.cors.enabled: true
http.cors.allow-origin: "*"
node.master: true
node.data: true

// 查询es所有mapping

ini 复制代码

http://119.45.25.164:9200/_mapping?pretty=true

1、在线安装分词器

bash 复制代码

docker exec -ites es7162 /bin/bash

./bin/elasticsearch-plugin install https://github.com/medcl/elasticsearch-analysis-ik/releases/download/v7.16.2/elasticsearch-analysis-ik-7.16.2.zip

2、离线安装中文分词器:

首先打开这个链接：github.com/medcl/elast...，把分词器插件下载下来，

bash 复制代码

# 把插件复制到容器内
docker cp elasticsearch-analysis-ik-7.16.2.zip es7162:/usr/share/elasticsearch/plugins

docker exec -it es7162 /bin/bash
cd /usr/share/elasticsearch/plugins/
mkdir ik
unzip elasticsearch-analysis-ik-7.16.2.zip -d ik
rm -rf elasticsearch-analysis-ik-7.16.2.zip

查看是否安装成功、当你看到日志中有输出analysis-ik，说明已经安装成功。

复制代码

docker restart es7162
docker logs es7162

4、安装canal-server

方法1

拉取镜像并启动：

bash 复制代码

docker pull canal/canal-server:v1.1.5

docker run --name canal115 -p 11111:11111  --link mysql5736:mysql5736 -id canal/canal-server:v1.1.5

修改对应的配置：

ini 复制代码

docker exec -it canal115 /bin/bash
cd canal-server/conf/example/
vi instance.properties  // 修改配置

# 把0改成10，只要不和mysql的id相同就行
canal.instance.mysql.slaveId=10
# 修改成mysql对应的账号密码，mysql5736就是mysql镜像的链接别名
canal.instance.master.address=mysql5736:3306
canal.instance.dbUsername=root
canal.instance.dbPassword=admin

验证配置是否成功：

bash 复制代码

#首先重启一下canal
docker restart  canal115

docker exec -it canal115 /bin/bash
cd canal-server/logs/example/
tail -100f example.log  // 查看日志

截图如下，说明已经链接上了mysql主机，此时mysql中的数据变化，都会在canal中有同步。

方法二：