canal同步mysql数据到elasticsearch

本文使用docker环境安装mysql、canal、elasticsearch，基于binlog利用canal实现mysql的数据同步到elasticsearch中。

实验中间件版本说明：

• centos 8
• mysql 5.7.36
• es 7.16.2
• cannal.server: 1.1.5
• canal.adapter: 1.1.5
• postman

1、安装docker

基本命令：

#centos 7 安装 docker
yum install docker

#centos 8 安装docker
yum erase podman buildah 
yum install -y yum-utils
yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
yum install docker-ce docker-ce-cli containerd.io

#检验安装是否成功
[root@localhost opt]# docker --version
Docker version 20.10.12, build e91ed57

#启动
systemctl start docker

#换镜像源
sudo vim /etc/docker/daemon.json
内容如下：
{
 "registry-mirrors": ["https://m9r2r2uj.mirror.aliyuncs.com"]
}
保存退出，重启docker

#重启
sudo service docker restart

#列出镜像
docker images

#查看运行进程
docker ps

2、安装mysql

方法1

docker pull mysql:5.7.36
docker run --name mysql5736 -p 3306:3306 -e MYSQL_ROOT_PASSWORD=admin -d mysql:5.7.36

docker exec -it mysql5736 /bin/bash
apt-get update
apt-get install vim
cd /etc/mysql/mysql.conf.d
vim mysqld.cnf  // 修改mysql配置

查看配置情况

mysql -uroot -p
show master status  // binlog日志文件
reset master; // 重启日志

方法2 或者docker-compose-es -f docker-compose-mysql.yml up -d

version: '3.9'
services:
    mysql:
        image: 'mysql:5.7.36'
        environment:
            - MYSQL_ROOT_PASSWORD=123456
        ports:
            - '3306:3306'
        volumes:
            - './mysql/mysql.conf.d/mysqld.cnf:/etc/mysql/mysql.conf.d/mysqld.cnf'
        container_name: mysql5736

新建文件 [root@c1 bk]# vi./mysql/mysql.conf.d/mysqld.cnf

# Copyright (c) 2014, 2021, Oracle and/or its affiliates.
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License, version 2.0,
# as published by the Free Software Foundation.
#
# This program is also distributed with certain software (including
# but not limited to OpenSSL) that is licensed under separate terms,
# as designated in a particular file or component or in included license
# documentation.  The authors of MySQL hereby grant you an additional
# permission to link the program and your derivative works with the
# separately licensed software that they have included with MySQL.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License, version 2.0, for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA  02110-1301 USA

#
# The MySQL  Server configuration file.
#
# For explanations see
# http://dev.mysql.com/doc/mysql/en/server-system-variables.html

[mysqld]
pid-file	= /var/run/mysqld/mysqld.pid
socket		= /var/run/mysqld/mysqld.sock
datadir		= /var/lib/mysql
#log-error	= /var/log/mysql/error.log
log-bin=/var/log/mysql/mysql-bin
binlog-format=ROW
server-id=1
# By default we only accept connections from localhost
#bind-address	= 127.0.0.1
# Disabling symbolic-links is recommended to prevent assorted security risks
symbolic-links=0

查看日志文件：

cd /var/log/mysql/  // 进入日志文件目录
mysqlbinlog -vv mysql-bin.000001  // row格式查看日志

3、安装es

方法1

docker pull elasticsearch:7.16.2
docker run -p 9200:9200 -p 9300:9300 -e "discovery.type=single-node" --name='es7162' -d elasticsearch:7.16.2

然后我们需要配置一下es的信息：

docker exec -ites es7162 /bin/bash
cd config
vi elasticsearch.yml

配置文件：

cluster.name: dailyhub-es
network.host: 0.0.0.0

node.name: node-1
http.port: 9200
http.cors.enabled: true
http.cors.allow-origin: "*"
node.master: true
node.data: true

方法2 或者 [root@c1 bk]# docker-compose-es -f docker-compose-es.yml up -d

version: '3.9'
services:
    elasticsearch:
        image: 'elasticsearch:7.16.2'
        container_name: es7162
        environment:
            - discovery.type=single-node
        volumes:
            - './elasticsearch/config/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml'
        ports:
            - '9300:9300'
            - '9200:9200'

配置文件

[root@c1 bk]# cat elasticsearch/config/elasticsearch.yml 
cluster.name: "docker-cluster"
network.host: 0.0.0.0
node.name: node-1
http.port: 9200
http.cors.enabled: true
http.cors.allow-origin: "*"
node.master: true
node.data: true

// 查询es所有mapping

http://119.45.25.164:9200/_mapping?pretty=true

1、在线安装分词器

docker exec -ites es7162 /bin/bash

./bin/elasticsearch-plugin install https://github.com/medcl/elasticsearch-analysis-ik/releases/download/v7.16.2/elasticsearch-analysis-ik-7.16.2.zip

2、离线安装中文分词器:

首先打开这个链接：github.com/medcl/elast...，把分词器插件下载下来，

# 把插件复制到容器内
docker cp elasticsearch-analysis-ik-7.16.2.zip es7162:/usr/share/elasticsearch/plugins

docker exec -it es7162 /bin/bash
cd /usr/share/elasticsearch/plugins/
mkdir ik
unzip elasticsearch-analysis-ik-7.16.2.zip -d ik
rm -rf elasticsearch-analysis-ik-7.16.2.zip

查看是否安装成功、当你看到日志中有输出analysis-ik，说明已经安装成功。

docker restart es7162
docker logs es7162

4、安装canal-server

方法1

拉取镜像并启动：

docker pull canal/canal-server:v1.1.5

docker run --name canal115 -p 11111:11111  --link mysql5736:mysql5736 -id canal/canal-server:v1.1.5

修改对应的配置：

docker exec -it canal115 /bin/bash
cd canal-server/conf/example/
vi instance.properties  // 修改配置

# 把0改成10，只要不和mysql的id相同就行
canal.instance.mysql.slaveId=10
# 修改成mysql对应的账号密码，mysql5736就是mysql镜像的链接别名
canal.instance.master.address=mysql5736:3306
canal.instance.dbUsername=root
canal.instance.dbPassword=admin

验证配置是否成功：

#首先重启一下canal
docker restart  canal115

docker exec -it canal115 /bin/bash
cd canal-server/logs/example/
tail -100f example.log  // 查看日志

截图如下，说明已经链接上了mysql主机，此时mysql中的数据变化，都会在canal中有同步。

方法二：

[root@c1 bk]# docker-compose-es -f docker-compose-canal-server.yml up -d

version: '3.9'
services:
    canal-server:
        image: 'canal/canal-server:v1.1.5'
        stdin_open: true
        links:
            - 'mysql5736:mysql5736'
        ports:
            - '11111:11111'
        volumes:
            - './canal-server/conf/example/instance.properties:/home/admin/canal-server/conf/example/instance.properties'
        container_name: canal115

配置文件

cat canal-server/conf/example/instance.properties 
#################################################
## mysql serverId , v1.0.26+ will autoGen
canal.instance.mysql.slaveId=10

# enable gtid use true/false
canal.instance.gtidon=false

# position info
canal.instance.master.address=192.168.8.93:3306
canal.instance.master.journal.name=
canal.instance.master.position=
canal.instance.master.timestamp=
canal.instance.master.gtid=

# rds oss binlog
canal.instance.rds.accesskey=
canal.instance.rds.secretkey=
canal.instance.rds.instanceId=

# table meta tsdb info
canal.instance.tsdb.enable=true
#canal.instance.tsdb.url=jdbc:mysql://127.0.0.1:3306/canal_tsdb
#canal.instance.tsdb.dbUsername=canal
#canal.instance.tsdb.dbPassword=canal

#canal.instance.standby.address =
#canal.instance.standby.journal.name =
#canal.instance.standby.position =
#canal.instance.standby.timestamp =
#canal.instance.standby.gtid=

# username/password
canal.instance.dbUsername=root
canal.instance.dbPassword=123@456
canal.instance.connectionCharset = UTF-8
# enable druid Decrypt database password
canal.instance.enableDruid=false
#canal.instance.pwdPublicKey=MFwwDQYJKoZIhvcNAQEBBQADSwAwSAJBALK4BUxdDltRRE5/zXpVEVPUgunvscYFtEip3pmLlhrWpacX7y7GCMo2/JM6LeHmiiNdH1FWgGCpUfircSwlWKUCAwEAAQ==

# table regex
canal.instance.filter.regex=.*\\..*
# table black regex
canal.instance.filter.black.regex=mysql\\.slave_.*
# table field filter(format: schema1.tableName1:field1/field2,schema2.tableName2:field1/field2)
#canal.instance.filter.field=test1.t_product:id/subject/keywords,test2.t_company:id/name/contact/ch
# table field black filter(format: schema1.tableName1:field1/field2,schema2.tableName2:field1/field2)
#canal.instance.filter.black.field=test1.t_product:subject/product_image,test2.t_company:id/name/contact/ch

# mq config
canal.mq.topic=example
# dynamic topic route by schema or table regex
#canal.mq.dynamicTopic=mytest1.user,mytest2\\..*,.*\\..*
canal.mq.partition=0
# hash partition config
#canal.mq.partitionsNum=3
#canal.mq.partitionHash=test.table:id^name,.*\\..*
#canal.mq.dynamicTopicPartitionNum=test.*:4,mycanal:6
#################################################

java验证canal-server

可以通过Java程序测试有没连接上mysql：

导入canal-client包

<!-- 为了测试canal-server是否连接mysql成功，1.1.5版本少包，所以用1.1.4版本 -->
<dependency>
   <groupId>com.alibaba.otter</groupId>
   <artifactId>canal.client</artifactId>
   <version>1.1.4</version>
</dependency>

springboot版本

 
<parent>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-parent</artifactId>
    <version>2.2.3.RELEASE</version>
    <relativePath/> <!-- lookup parent from repository -->
</parent>
 

• com.markerhub.SimpleCanalClientExample

/**
 * 公众号：MarkerHub
 *
 * 说明：用于测试canal是否已经连接上了mysql
 */
public class SimpleCanalClientExample {
    public static void main(String args[]) {
        // 创建链接
        CanalConnector connector = CanalConnectors.newSingleConnector(new InetSocketAddress("119.45.25.164",
                11111), "example", "", "");
        int batchSize = 1000;
        int emptyCount = 0;
        try {
            connector.connect();
            connector.subscribe(".*\..*");
            connector.rollback();
            int totalEmptyCount = 120;
            while (emptyCount < totalEmptyCount) {
                Message message = connector.getWithoutAck(batchSize); // 获取指定数量的数据
                long batchId = message.getId();
                int size = message.getEntries().size();
                if (batchId == -1 || size == 0) {
                    emptyCount++;
                    System.out.println("empty count : " + emptyCount);
                    try {
                        Thread.sleep(1000);
                    } catch (InterruptedException e) {
                    }
                } else {
                    emptyCount = 0;
                    // System.out.printf("message[batchId=%s,size=%s] \n", batchId, size);
                    printEntry(message.getEntries());
                }
                connector.ack(batchId); // 提交确认
                // connector.rollback(batchId); // 处理失败, 回滚数据
            }
            System.out.println("empty too many times, exit");
        } finally {
            connector.disconnect();
        }
    }
    private static void printEntry(List<Entry> entrys) {
        for (Entry entry : entrys) {
            if (entry.getEntryType() == EntryType.TRANSACTIONBEGIN || entry.getEntryType() == EntryType.TRANSACTIONEND) {
                continue;
            }
            RowChange rowChage = null;
            try {
                rowChage = RowChange.parseFrom(entry.getStoreValue());
            } catch (Exception e) {
                throw new RuntimeException("ERROR ## parser of eromanga-event has an error , data:" + entry.toString(),
                        e);
            }
            EventType eventType = rowChage.getEventType();
            System.out.println(String.format("================&gt; binlog[%s:%s] , name[%s,%s] , eventType : %s",
                    entry.getHeader().getLogfileName(), entry.getHeader().getLogfileOffset(),
                    entry.getHeader().getSchemaName(), entry.getHeader().getTableName(),
                    eventType));
            for (RowData rowData : rowChage.getRowDatasList()) {
                if (eventType == EventType.DELETE) {
                    printColumn(rowData.getBeforeColumnsList());
                } else if (eventType == EventType.INSERT) {
                    printColumn(rowData.getAfterColumnsList());
                } else {
                    System.out.println("-------&gt; before");
                    printColumn(rowData.getBeforeColumnsList());
                    System.out.println("-------&gt; after");
                    printColumn(rowData.getAfterColumnsList());
                }
            }
        }
    }
    private static void printColumn(List<Column> columns) {
        for (Column column : columns) {
            System.out.println(column.getName() + " : " + column.getValue() + "    update=" + column.getUpdated());
        }
    }
}

当mysql的数据更新时候效果如下：

注意当后面canal-adapter也连接上canal-server后，程序就监听不到数据变化了。

5、安装canal-adapter

方法1：

由于目前canal-adapter没有官方docker镜像，所以拉去一个非官方的

docker pull slpcat/canal-adapter:v1.1.5

docker run --name adapter115 -p 8081:8081 --link mysql5736:mysql5736 --link canal115:canal115 --link es7162:es7162 -d slpcat/canal-adapter:v1.1.5

修改配置：

docker exec -it adapter115 /bin/bash
cd conf/
vi application.yml

配置修改如下，一些不需要的配置或者注释掉的配置可以删除掉：

server:
  port: 8081
spring:
  jackson:
    date-format: yyyy-MM-dd HH:mm:ss
    time-zone: GMT+8
    default-property-inclusion: non_null

canal.conf:
  mode: tcp #tcp kafka rocketMQ rabbitMQ
  flatMessage: true
  zookeeperHosts:
  syncBatchSize: 1000
  retries: 0
  timeout:
  accessKey:
  secretKey:
  consumerProperties:
    # canal tcp consumer
    canal.tcp.server.host: canal115:11111
    canal.tcp.zookeeper.hosts:
    canal.tcp.batch.size: 500
    canal.tcp.username:
    canal.tcp.password:
  srcDataSources:
    defaultDS:
      url: jdbc:mysql://mysql5736:3306/dailyhub?useUnicode=true
      username: root
      password: admin
  canalAdapters:
  - instance: example # canal instance Name or mq topic name
    groups:
    - groupId: g1
      outerAdapters:
      - name: logger
      - name: es7
        hosts: es7162:9200 # 127.0.0.1:9200 for rest mode
        properties:
          mode: rest
          # security.auth: test:123456 #  only used for rest mode
          cluster.name: dailyhub-es

接下来是修改表映射索引文件：

docker exec -it adapter115 /bin/bash
cd conf/es7

cp -v mytest_user.yml dailyhub_collect.yml
# 删除其他多余的
rm -rf biz_order.yml customer.yml mytest_user.yml
vi dailyhub_collect.yml

配置文件：

dataSourceKey: defaultDS
destination: example
groupId: g1
esMapping:
  _index: dailyhub_collect
  _id: _id
  _type: _doc
  upsert: true
#  pk: id
  sql: "
SELECT
        c.id AS _id,
        c.user_id AS userId,
        c.title AS title,
        c.url AS url,
        c.note AS note,
        c.collected AS collected,
        c.created AS created,
        c.personal AS personal,
        u.username AS username,
        u.avatar AS userAvatar
FROM
        m_collect c
LEFT JOIN m_user u ON c.user_id = u.id

"
#  objFields:
#    _labels: array:;
#   etlCondition: "where c.c_time>={}"
  commitBatch: 3000

注意对于时间类型，在后端一定要使用LocalDateTime或者LocalDate类型，如果是Date类型，需要自己手动设置格式。

方法二：

[root@c1 bk]# docker-compose-es -f docker-compose-canal-adapter.yml up -d

version: '3.9'
services:
    canal-adapter:
        image: 'slpcat/canal-adapter:v1.1.5'
        links:
            - 'es7162:es7162'
            - 'canal115:canal115'
            - 'mysql5736:mysql5736'
        ports:
            - '8081:8081'
        volumes:
            - './canal-adapter/conf/es7/mytest_user.yml:/opt/canal-adapter/conf/es7/mytest_user.yml'
            - './canal-adapter/conf/application.yml:/opt/canal-adapter/conf/application.yml'
        container_name: adapter115

配置文件[root@c1 bk]# cat ./canal-adapter/conf/es7/mytest_user.yml

dataSourceKey: defaultDS
destination: example
groupId: g1
esMapping:
  _index: dailyhub_collect
  _id: _id
  _type: _doc
  upsert: true
#  pk: id
  sql: "SELECT p.id AS _id, p.NAME AS NAME, p.branch_bank_id AS branchBankId, p.city_id AS cityId, p.city_id AS cityIdList, p.district_id AS districtId, p.district_id AS districtIdList, p.province_id AS provinceId, p.label_ids AS labelIdList, p.term_range_start AS termRangeStart, p.term_range_end AS termRangeEnd, 
p.annual_interest_range_start AS annualInterestRangeStart, p.annual_interest_range_end AS annualInterestRangeEnd, p.quota_range_end AS quotaRangeEnd, p.acceptance_time_start AS acceptanceTimeStart, p.acceptance_time_end AS acceptanceTimeEnd, p.loan_use AS loanUse, p.guarantee_means_ids AS guaranteeMeansIdList, p.repayment_means_ids AS repaymentMeansIdList, p.introduce AS introduce, p.characteristic AS characteristic, p.enterprise_access_requirement_json AS enterpriseAccessRequirementJson, p.apply_requirement AS applyRequirement, p.requirement_material AS requirementMaterial, p.sale_status AS saleStatus, p.order_quantity AS orderQuantity, p.bank_id AS bankId, p.branch_bank_name AS branchBankName, p.bank_name AS bankName, p.bank_logo AS bankLogo, p.area_json AS areaJson, p.whilte_list_last_import_time AS whilteListLastImportTime, p.financing_category AS financingCategory, p.approval_time AS approvalTime, p.submit_time AS submitTime, p.reason_rejection AS reasonRejection, p.reason_force_off_sale AS reasonForceOffSale, p.force_off_sale_user_id AS forceOffSaleUserId, p.force_off_sale_time AS forceOffSaleTime, p.reason_off_sale AS reasonOffSale, p.off_sale_user_id AS offSaleUserId, p.off_sale_time AS offSaleTime, p.approval_user_id AS approvalUserId, p.on_sale_time AS onSaleTime, p.label_names AS labelList, p.repayment_means_names AS repaymentMeansList, p.guarantee_means_names AS guaranteeMeansList, p.create_time AS createTime, p.update_time AS updateTime, p.create_user_id AS createUserId, p.update_user_id AS updateUserId, p.deleted AS deleted, p.version AS version, p.data_type AS dataType, p.applicable_objects AS applicableObjects, p.applicable_objects AS applicableObjectsList, p.interest_rate_type AS interestRateType, p.interest_rate_type_name AS interestRateTypeName, p.applicable_bank_ids AS applicableBankIds FROM loan_product p"  objFields:
    cityIdList: array:,
    districtIdList: array:,
    labelIdList: array:,
    guaranteeMeansIdList: array:,
    repaymentMeansIdList: array:,
    applicableObjectsList: array:,
    labelList: array:,
    repaymentMeansList: array:,
    guaranteeMeansList: array:,
    etlCondition: "where p.create_time>={}"
  commitBatch: 3000

[root@c1 bk]# cat ./canal-adapter/conf/application.yml

server:
port: 8081
spring:
jackson:
  date-format: yyyy-MM-dd HH:mm:ss
  time-zone: GMT+8
  default-property-inclusion: non_null

canal.conf:
mode: tcp #tcp kafka rocketMQ rabbitMQ
flatMessage: true
zookeeperHosts:
syncBatchSize: 1000
retries: 0
timeout:
accessKey:
secretKey:
consumerProperties:
  # canal tcp consumer
  canal.tcp.server.host: canal115:11111
  canal.tcp.zookeeper.hosts:
  canal.tcp.batch.size: 500
  canal.tcp.username:
  canal.tcp.password:
  # kafka consumer
  kafka.bootstrap.servers: 127.0.0.1:9092
  kafka.enable.auto.commit: false
  kafka.auto.commit.interval.ms: 1000
  kafka.auto.offset.reset: latest
  kafka.request.timeout.ms: 40000
  kafka.session.timeout.ms: 30000
  kafka.isolation.level: read_committed
  kafka.max.poll.records: 1000
  # rocketMQ consumer
  rocketmq.namespace:
  rocketmq.namesrv.addr: 127.0.0.1:9876
  rocketmq.batch.size: 1000
  rocketmq.enable.message.trace: false
  rocketmq.customized.trace.topic:
  rocketmq.access.channel:
  rocketmq.subscribe.filter:
  # rabbitMQ consumer
  rabbitmq.host:
  rabbitmq.virtual.host:
  rabbitmq.username:
  rabbitmq.password:
  rabbitmq.resource.ownerId:

srcDataSources:
  defaultDS:
    url: jdbc:mysql://mysql5736:3306/dailyhub?useUnicode=true
    username: root
    password: 123@456
canalAdapters:
- instance: example # canal instance Name or mq topic name
  groups:
  - groupId: g1
    outerAdapters:
    - name: logger
    - name: es7
      hosts: es7162:9200
      properties:
        mode: rest
        cluster.name: docker-cluster

6、联合测试

然后就可以直接测试了，准备测试条件：

• 在数据库中生成表和字段，
• 然后elasticsearch中生成索引。 先新建数据库dailyhub。然后数据表结构：

SET FOREIGN_KEY_CHECKS=0;

-- ----------------------------
-- Table structure for m_collect
-- ----------------------------
DROP TABLE IF EXISTS `m_collect`;
CREATE TABLE `m_collect` (
  `id` bigint(20) NOT NULL AUTO_INCREMENT,
  `collected` date DEFAULT NULL,
  `created` datetime(6) DEFAULT NULL,
  `note` varchar(255) DEFAULT NULL,
  `personal` int(11) DEFAULT NULL,
  `title` varchar(255) DEFAULT NULL,
  `url` varchar(255) DEFAULT NULL,
  `user_id` bigint(20) DEFAULT NULL,
  PRIMARY KEY (`id`),
  KEY `FK6yx2mr7fgvv204y8jw5ubsn7h` (`user_id`),
  CONSTRAINT `FK6yx2mr7fgvv204y8jw5ubsn7h` FOREIGN KEY (`user_id`) REFERENCES `m_user` (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=19 DEFAULT CHARSET=utf8mb4;

-- ----------------------------
-- Records of m_collect
-- ----------------------------

-- ----------------------------
-- Table structure for m_user
-- ----------------------------
DROP TABLE IF EXISTS `m_user`;
CREATE TABLE `m_user` (
  `id` bigint(20) NOT NULL AUTO_INCREMENT,
  `avatar` varchar(255) DEFAULT NULL,
  `created` datetime(6) DEFAULT NULL,
  `lasted` datetime(6) DEFAULT NULL,
  `open_id` varchar(255) DEFAULT NULL,
  `statu` int(11) DEFAULT NULL,
  `username` varchar(255) DEFAULT NULL,
  PRIMARY KEY (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=2 DEFAULT CHARSET=utf8mb4;

-- ----------------------------
-- Records of m_user
-- ----------------------------
INSERT INTO `m_user` VALUES ('1', 'https://image-1300566513.cos.ap-guangzhou.myqcloud.com/upload/images/5a9f48118166308daba8b6da7e466aab.jpg', '2022-01-05 16:08:40.042000', '2022-01-06 13:07:45.153000', 'ozWZ-uAOY2iecT-byynO382u01zg', '0', '公众号：MarkerHub');

接下来借postman来新建elasticsearch的索引：

# 创建索引并添加映射字段
PUT http://119.45.25.164:9200/dailyhub_collect

{
    "mappings": {
        "properties": {
            "collected": {
                "type": "date",
                "format": "date_optional_time||epoch_millis"
            },
            "created": {
                "type": "date",
                "format": "date_optional_time||epoch_millis"
            },
            "note": {
                "type": "text",
                "analyzer": "ik_max_word",
                "search_analyzer": "ik_smart"
            },
            "personal": {
                "type": "integer"
            },
            "title": {
                "type": "text",
                "analyzer": "ik_max_word",
                "search_analyzer": "ik_smart"
            },
            "url": {
                "type": "text"
            },
            "userAvatar": {
                "type": "text"
            },
            "userId": {
                "type": "long"
            },
            "username": {
                "type": "keyword"
            }
        }
    }
}

其他常用操作：

# 删除索引
PUT  http://119.45.25.164:9200/dailyhub_collect

# 查看素有索引映射
GET  http://119.45.25.164:9200/_mapping?pretty=true

# 搜索文档
GET http://119.45.25.164:9200/dailyhub_collect/_search

# 删除ID为1的文档
DELETE  http://119.45.25.164:9200/dailyhub_collect/_doc/1

然后我们打开canal-adapter的输入日志：

docker logs --tail 100  -f adapter115

然后我们在mysql的m_collect中新添加一条记录，可以看到日志输出如下：

然后搜索全部文档，发现es中有数据啦。

如果看到adaptar115一直出现这种异常，说明启动顺序不对，启动顺序应该是：mysql、es、canal、adapar

2022-01-11 10:43:15.278 [Thread-2] ERROR c.a.otter.canal.adapter.launcher.loader.AdapterProcessor - com.alibaba.otter.canal.protocol.exception.CanalClientException: java.io.IOException: Broken pipe Error sync but ACK!