【ElasticSearch】Linux安装ES、LogStash、Kibana、Canal

【ElasticSearch】docker安装es、kibana、canal

【一】环境初始化
【二】安装
- 【1】离线安装
- 【2】docker安装
- 【3】docker-compose安装
- 【2】安装分词器
- 【3】Kibana部署
- [【4】LogStash 部署](#【4】LogStash 部署)
- [【5】Canal 部署](#【5】Canal 部署)
- - - 2-安装Canal-adapter
    - 4-测试同步

【一】环境初始化

【1】Linux服务器初始化

查看ip地址：ifconfig eth0

使用yum安装报错：Could not retrieve mirrorlist http://mirrorlist.centos.org/?release=7\&arch=x86_64\&repo=os\&

出现这个错误是因为使用的 CentOS 7 仓库已经被归档，当前的镜像地址无法找到所需的文件。CentOS 7 的官方支持已经结束，部分仓库已被移至归档库。这导致了你的 yum 命令无法找到所需的元数据文件。CentOS 7 的官方仓库在 2024 年 6 月 30 日之后已经停止维护。因此，使用最新的 CentOS 7 官方仓库可能会遇到问题。

修改仓库指向，镜像更改为阿里云

java 复制代码

curl -o /etc/yum.repos.d/CentOS-Base.repo https://mirrors.aliyun.com/repo/Centos-7.repo

（1）查看系统的内核版本

powershell 复制代码

uname -r 4.18.0-80.11.2.el8_0.x86_64

执行命令：uname -r

注意：x86 64位系统，如果是32位是不能安装 docker 的

（2）yum更新到最新版本

powershell 复制代码

sudo yum -y update

执行命令：sudo yum update

注意：看到显示 Complete 就代表完成了，整个过程需要 5-10 分钟左右

更新gcc（C语言的编译环境）

powershell 复制代码

yum -y install gcc
yum -y install gcc-c++

【2】安装宝塔面板

powershell 复制代码

Centos安装脚本
yum install -y wget && wget -O install.sh https://download.bt.cn/install/install_6.0.sh && sh install.sh ed8484bec
Ubuntu/Deepin安装脚本
wget -O install.sh https://download.bt.cn/install/install-ubuntu_6.0.sh && sudo bash install.sh ed8484bec

【3】安装docker和docker-compose

使用宝塔页面直接安装

配置镜像

bash 复制代码

cat /etc/docker/daemon.json

sudo tee /etc/docker/daemon.json <<'EOF'
{
  "registry-mirrors": [
    "https://docker.1ms.run",
    "https://docker.1panel.live",
    "https://hub.rat.dev"
  ]
}
EOF
 
# 重载配置并重启 Docker
sudo systemctl daemon-reload
sudo systemctl restart docker
 
# 验证配置
docker info | grep -A 5 "Registry Mirrors"

国内下载镜像的网站：https://docker.aityp.com/#google_vignette

【4】elk架构

【二】安装

版本统一

elasticsearch:8.13.3

kibana:8.13.3

logstash:8.13.3

canal-server:v1.1.7

【1】离线安装

（1）es安装

下载地址：

bash 复制代码

#解压到相应目录
tar -zxvf elasticsearch-7.10.2-linux-x86_64.tar.gz -C /usr/local

#修改配置
cd /usr/local/elasticsearch-7.10.2/config/
vim elasticsearch.yml

#配置内容
node.name: node-1
path.data: /usr/local/elasticsearch-7.10.2/data
path.logs: /usr/local/elasticsearch-7.10.2/logs
network.host: 127.0.0.1
http.host: 0.0.0.0
http.port: 9200
discovery.seed_hosts: ["127.0.0.1"]
cluster.initial_master_nodes: ["node-1"]

#创建es用户
useradd es
chown -R es:es /usr/local/elasticsearch-7.10.2

#切换用户成es用户进行操作
su - es
/usr/local/elasticsearch-7.10.2/bin/elasticsearch

#后台启动
/usr/local/elasticsearch-7.10.2/bin/elasticsearch -d

在浏览器打开9200端口地址： (http://120.78.129.95:9200/ (opens new window))，如果出现了下面的信息，就表示已经成功启动了

（2）LogStash安装

下载地址：

bash 复制代码

#解压到相应目录
tar -zxvf logstash-7.10.2.tar.gz -C /usr/local

#新增配置文件
cd /usr/local/logstash-7.10.2/bin
vim logstash-elasticsearch.conf

#内容如下
input {
	stdin {}
}
output {
	elasticsearch {
		hosts => '120.78.129.95:9200'
	}
	stdout {
		codec => rubydebug
	}
}

#启动
./logstash -f logstash-elasticsearch.conf

（3）Kibana安装

下载地址：

bash 复制代码

#解压到相应目录
tar -zxvf kibana-7.10.2-linux-x86_64.tar.gz -C /usr/local
mv /usr/local/kibana-7.10.2-linux-x86_64 /usr/local/kibana-7.10.2

#修改配置
cd /usr/local/kibana-7.10.2/config
vim kibana.yml

#内容如下
server.port: 5601 
server.host: "0.0.0.0" 
elasticsearch.hosts: ["http://120.78.129.95:9200"] 
kibana.index: ".kibana"

#授权es用户
chown -R es:es /usr/local/kibana-7.10.2/

#切换用户成es用户进行操作
su - es
/usr/local/kibana-7.10.2/bin/kibana 

#后台启动
/usr/local/kibana-7.10.2/bin/kibana &

在浏览器打开5601端口地址： (http://120.78.129.95:5601/ (opens new window))，如果出现了下面的信息，就表示已经成功启动了

切换中文在config/kibana.yml添加

bash 复制代码

i18n.locale: "zh-CN"

（4）Canal安装

用的Canal-deployer是1.1.7版本。

下载地址：https://github.com/alibaba/canal/releases

下载canal.deployer，可以直接监听MySQL的binlog，把自己伪装成MySQL的从库，只负责接收数据，并不做处理。

bash 复制代码

#下载
wget https://github.com/alibaba/canal/releases/download/canal-1.1.7/canal.deployer-1.1.7.tar.gz

#解压
mkdir /tmp/canal
tar zxvf canal.deployer-1.1.7.tar.gz  -C /tmp/canal
#解压完成后，进入 /tmp/canal 目录，可以看到如下结构
drwxr-xr-x 2 jianghang jianghang  136 2013-02-05 21:51 bin
drwxr-xr-x 4 jianghang jianghang  160 2013-02-05 21:51 conf
drwxr-xr-x 2 jianghang jianghang 1.3K 2013-02-05 21:51 lib
drwxr-xr-x 2 jianghang jianghang   48 2013-02-05 21:29 logs

配置修改canal.properties

修改canal.properties，增加canal.ip = 127.0.0.1

powershell 复制代码

cd /tmp/canal/conf
vim canal.properties

修改example/instance.properties

（1）配置mysql数据库服务器的ip和端口

（2）配置监听mysql的用户和密码

powershell 复制代码

cd /tmp/canal/conf/example
vi instance.properties

bash 复制代码

## mysql serverId
canal.instance.mysql.slaveId = 1234
#position info，需要改成自己的数据库信息
canal.instance.master.address = 127.0.0.1:3306 
canal.instance.master.journal.name = 
canal.instance.master.position = 
canal.instance.master.timestamp = 
#canal.instance.standby.address = 
#canal.instance.standby.journal.name =
#canal.instance.standby.position = 
#canal.instance.standby.timestamp = 
#username/password，需要改成自己的数据库信息
canal.instance.dbUsername = canal  
canal.instance.dbPassword = canal
canal.instance.defaultDatabaseName =
canal.instance.connectionCharset = UTF-8
#table regex
canal.instance.filter.regex = .\*\\\\..\*

canal.instance.connectionCharset 代表数据库的编码方式对应到 java 中的编码类型，比如 UTF-8，GBK , ISO-8859-1

如果系统是1个 cpu，需要将 canal.instance.parser.parallel 设置为 false

到 bin 目录下，执行命令：./startup.sh

powershell 复制代码

cd /tmp/canal/bin && ./stop.sh && ./startup.sh

有报错

把jdk安装上，再执行

（1）查看 server 日志

启动成功，检查canal/logs/canal.log

看到the canal server is running now ...表示启动成功。

powershell 复制代码

tail -f /tmp/canal/logs/canal/canal.log

bash 复制代码

2013-02-05 22:45:27.967 [main] INFO  com.alibaba.otter.canal.deployer.CanalLauncher - ## start the canal server.
2013-02-05 22:45:28.113 [main] INFO  com.alibaba.otter.canal.deployer.CanalController - ## start the canal server[10.1.29.120:11111]
2013-02-05 22:45:28.210 [main] INFO  com.alibaba.otter.canal.deployer.CanalLauncher - ## the canal server is running now ......

（2）查看 instance 的日志

bash 复制代码

tail -f /tmp/canal/logs/example/example.log

bash 复制代码

2013-02-05 22:50:45.636 [main] INFO  c.a.o.c.i.spring.support.PropertyPlaceholderConfigurer - Loading properties file from class path resource [canal.properties]
2013-02-05 22:50:45.641 [main] INFO  c.a.o.c.i.spring.support.PropertyPlaceholderConfigurer - Loading properties file from class path resource [example/instance.properties]
2013-02-05 22:50:45.803 [main] INFO  c.a.otter.canal.instance.spring.CanalInstanceWithSpring - start CannalInstance for 1-example 
2013-02-05 22:50:45.810 [main] INFO  c.a.otter.canal.instance.spring.CanalInstanceWithSpring - start successful....

SHOW MASTER STATUS;语句报错了，因为这是mysql8.0的语法，用来查看主库状态信息，mysql8.4应该使用

sql 复制代码

SHOW BINARY LOG STATUS;

【2】docker安装

（1）es安装

（1）配置要求

CPU：1核

内存：4G

bash 复制代码

#docker pull elasticsearch:8.13.0
docker pull swr.cn-north-4.myhuaweicloud.com/ddn-k8s/docker.elastic.co/elasticsearch/elasticsearch:8.13.3

mkdir -p /root/package/es/config
vim /root/package/es/config/elasticsearch.yml

#内容如下
cluster.name: "nfturbo-cluster"
network.host: 0.0.0.0
http.cors.enabled: true
http.cors.allow-origin: "*"
xpack.security.enabled: false

#授权
cd /home/package/es
chmod 777 -R ./*

# 启动镜像
docker run --name elasticsearch -p 9200:9200  -p 9300:9300  -e "discovery.type=single-node" -e ES_JAVA_OPTS="-Xms256m -Xmx512m" -v /root/package/es/config/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml -v /home/package/es/data:/usr/share/elasticsearch/data -v /home/package/es/plugins:/usr/share/elasticsearch8/plugins -d swr.cn-north-4.myhuaweicloud.com/ddn-k8s/docker.elastic.co/elasticsearch/elasticsearch:8.13.3

检查是否启动成功

首先通过 docker logs elasticsearch 查看是否有报错。

然后通过curl检查是否成功：

powershell 复制代码

curl localhost:9200
{
  "name" : "3981174ed6df",
  "cluster_name" : "nfturbo-cluster",
  "cluster_uuid" : "9blwCZsiSiOapFjQEF42cA",
  "version" : {
    "number" : "8.13.0",
    "build_flavor" : "default",
    "build_type" : "docker",
    "build_hash" : "09df99393193b2c53d92899662a8b8b3c55b45cd",
    "build_date" : "2024-03-22T03:35:46.757803203Z",
    "build_snapshot" : false,
    "lucene_version" : "9.10.0",
    "minimum_wire_compatibility_version" : "7.17.0",
    "minimum_index_compatibility_version" : "7.0.0"
  },
  "tagline" : "You Know, for Search"
}

（7）通过9200端口访问

http://192.168.220.128:9200/

springboot项目连接报错

powershell 复制代码

Caused by: org.elasticsearch.ElasticsearchStatusException: Elasticsearch exception [type=mapper_parsing_exception, reason=Failed to parse mapping: analyzer [chinese_analyzer] has not been configured in mappings]

（2）LogStash安装

bash 复制代码

#docker pull logstash:8.13.0
docker pull swr.cn-north-4.myhuaweicloud.com/ddn-k8s/docker.elastic.co/logstash/logstash:8.13.3

#创建文件夹
mkdir /root/package/logstash

#先run出来一个容器
docker run -d --rm -it swr.cn-north-4.myhuaweicloud.com/ddn-k8s/docker.elastic.co/logstash/logstash:8.13.3 --name logstash

#将config和pipeline从容器cp到宿主机
#查看容器id为f03b8ad3bd691d4ddb9fa88c046b99003aaa0f285bcc4e15b303901f604aaba7
docker cp f03b8ad3bd691d4ddb9fa88c046b99003aaa0f285bcc4e15b303901f604aaba7:/usr/share/logstash/config /root/package/logstash/

docker cp f03b8ad3bd691d4ddb9fa88c046b99003aaa0f285bcc4e15b303901f604aaba7:/usr/share/logstash/pipeline /root/package/logstash/

#停止容器(启动时配置了 --rm ，停止后会自动删除)
docker stop f03b8ad3bd691d4ddb9fa88c046b99003aaa0f285bcc4e15b303901f604aaba7

#启动容器
docker run --name logstash \
-p 5044:5044 \
-v /root/package/logstash/pipeline/:/usr/share/logstash/pipeline/ \
-v /root/package/logstash/config/:/usr/share/logstash/config/ \
--restart=always \
-d swr.cn-north-4.myhuaweicloud.com/ddn-k8s/docker.elastic.co/logstash/logstash:8.13.3

修改配置文件

修改挂载出来的默认 logstash.yml 文件，这里位于 /root/package /logstash/config/ 目录，修改后配置如下：

bash 复制代码

vim /root/package/logstash/config/logstash.yml


http.host: "0.0.0.0"
xpack.monitoring.elasticsearch.hosts: [ "http://es:9200" ]
xpack.monitoring.elasticsearch.username: "elastic"
xpack.monitoring.elasticsearch.password: "123456"

修改默认的 logstash.conf 文件，创建日志处理规则，位于 /root/package/logstash/pipeline/ 目录，修改后如下：

bash 复制代码

vim /root/package/logstash/pipeline/logstash.conf

input {  
  tcp {  
    port => 5044  
    codec => json_lines  
  }  
}  
  
output {  
  elasticsearch {  
    hosts => ["es:9200"]  
    index => "iov-%{+YYYY.MM.dd}"  
    document_type => "_doc"
    user => "elastic"
    password => "123456"  
  }
}

修改后重启 logstash 服务。

（3）Kibana安装

bash 复制代码

#docker pull kibana:8.13.0
docker pull swr.cn-north-4.myhuaweicloud.com/ddn-k8s/docker.elastic.co/kibana/kibana:8.13.3

#添加配置
vi /root/package/es/config/kibana.yml

#内容如下
server.name: kibana
#server.port: 5601
server.host: 0.0.0.0
elasticsearch.hosts: [ "http://192.168.220.130:9200" ] # 改成 es 的内网 ip
#elasticsearch.username: "elastic"
#elasticsearch.password: "123456"
xpack.monitoring.ui.container.elasticsearch.enabled: true
i18n.locale: "zh-CN"

#启动Kibana
sudo docker run --name kibana -d -p 5601:5601 -v /root/package/es/config/kibana.yml:/usr/share/kibana/config/kibana.yml swr.cn-north-4.myhuaweicloud.com/ddn-k8s/docker.elastic.co/kibana/kibana:8.13.3

http://192.168.220.130:5601/app/home#/

输入用户名（elastic）、密码（123456））即可访问

（8）初始化es索引

powershell 复制代码

PUT nfturbo_collection
{
  "mappings": {
    "properties": {
      "name": {
        "type": "text"
      },
      "cover": {
        "type": "text"
      },
      "class_id": {
        "type": "text"
      },
      "price": {
        "type": "text"
      },
      "quantity": {
        "type": "long"
      },
      "detail": {
        "type": "text"
      },
      "saleable_inventory": {
        "type": "long"
      },
      "occupied_inventory": {
        "type": "long"
      },
      "state": {
        "type": "keyword"
      },
      "create_time": {
        "type": "date",
        "format": "yyyy-MM-dd HH:mm:ss || yyyy-MM-dd'T'HH:mm:ss+08:00 || strict_date_optional_time || epoch_millis"
      },
      "sale_time": {
        "type": "date",
        "format": "yyyy-MM-dd HH:mm:ss || yyyy-MM-dd'T'HH:mm:ss+08:00 || strict_date_optional_time || epoch_millis"
      },
      "sync_chain_time": {
        "type": "date",
        "format": "yyyy-MM-dd HH:mm:ss || yyyy-MM-dd'T'HH:mm:ss+08:00 || strict_date_optional_time || epoch_millis"
      },
	"book_start_time": {
	        "type": "date",
	        "format": "yyyy-MM-dd HH:mm:ss || yyyy-MM-dd'T'HH:mm:ss+08:00 || strict_date_optional_time || epoch_millis"
	      },
	"book_end_time": {
	        "type": "date",
	        "format": "yyyy-MM-dd HH:mm:ss || yyyy-MM-dd'T'HH:mm:ss+08:00 || strict_date_optional_time || epoch_millis"
	      },
	"can_book": {
        "type": "integer"
      },

      "deleted": {
        "type": "keyword"
      }
    }
  }
}

（9）重启后kibana打不开

日志内容如下

powershell 复制代码

[2025-11-30T14:38:28.637+00:00][INFO ][plugins.assetManager] Server is NOT enabled
[2025-11-30T14:38:28.849+00:00][ERROR][elasticsearch-service] Unable to retrieve version information from Elasticsearch nodes. connect ECONNREFUSED 172.17.0.2:9200
[2025-11-30T14:38:29.665+00:00][INFO ][plugins.screenshotting.chromium] Browser executable: /usr/share/kibana/node_modules/@kbn/screenshotting-plugin/chromium/headless_shell-linux_x64/headless_shell

原因是es的内网ip自动改了

不要使用动态的ip，直接写死服务器的ip

重启后解决

（4）Canal安装

bash 复制代码

# 远程拉取
# docker pull canal/canal-server:v1.1.7
docker pull swr.cn-north-4.myhuaweicloud.com/ddn-k8s/docker.io/canal/canal-server:v1.1.7

#创建文件夹
mkdir /root/package/canal && cd /root/package/canal

# 下载脚本
wget https://raw.githubusercontent.com/alibaba/canal/master/docker/run.sh 

# 构建一个destination name为test的队列，配置自己mysql的数据源信息
sh run.sh -e canal.auto.scan=false \
		  -e canal.destinations=test \
		  -e canal.instance.master.address=192.168.220.130:3306  \
		  -e canal.instance.dbUsername=canal  \
		  -e canal.instance.dbPassword=canal  \
		  -e canal.instance.connectionCharset=UTF-8 \
		  -e canal.instance.tsdb.enable=true \
		  -e canal.instance.gtidon=false  \

【3】docker-compose安装

（1）创建部署目录 & 配置文件夹

bash 复制代码

mkdir -p /data/canal-es-stack
cd /data/canal-es-stack

# 持久化目录
mkdir -p ./es/data ./es/plugins ./logstash/config ./logstash/pipeline
mkdir -p ./canal/deployer/conf ./canal/adapter/conf ./canal/logs

# 权限修复 ES专用
chmod 777 ./es/data

（2）docker-compose.yml 完整文件

bash 复制代码

version: '3.8'
services:
  # Elasticsearch 8.13.0 单节点
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:8.13.0
    container_name: es8
    restart: always
    environment:
      - discovery.type=single-node
      - "ES_JAVA_OPTS=-Xms1g -Xmx1g"
      # 关闭安全、ssl，内网使用
      - xpack.security.enabled=false
      - xpack.security.enrollment.enabled=false
      - xpack.security.http.ssl.enabled=false
      - xpack.security.transport.ssl.enabled=false
      - http.host=0.0.0.0
      - transport.host=0.0.0.0
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - ./es/data:/usr/share/elasticsearch/data
      - ./es/plugins:/usr/share/elasticsearch/plugins
    ports:
      - "9200:9200"
      - "9300:9300"
    networks:
      canal_es_net:
        ipv4_address: 172.20.0.10

  # Kibana 8.13.0
  kibana:
    image: docker.elastic.co/kibana/kibana:8.13.0
    container_name: kibana8
    restart: always
    environment:
      - ELASTICSEARCH_HOSTS=http://elasticsearch:9200
      - SERVER_HOST=0.0.0.0
    ports:
      - "5601:5601"
    depends_on:
      - elasticsearch
    networks:
      canal_es_net:
        ipv4_address: 172.20.0.11

  # Logstash 8.13.0
  logstash:
    image: docker.elastic.co/logstash/logstash:8.13.0
    container_name: logstash8
    restart: always
    environment:
      - "LS_JAVA_OPTS=-Xms512m -Xmx512m"
    volumes:
      - ./logstash/config:/usr/share/logstash/config
      - ./logstash/pipeline:/usr/share/logstash/pipeline
    ports:
      - "5044:5044"
    depends_on:
      - elasticsearch
    networks:
      canal_es_net:
        ipv4_address: 172.20.0.12

  # Canal Deployer 1.1.7 捕获MySQL binlog
  canal-deployer:
    image: canal/canal-server:v1.1.7
    container_name: canal-deployer
    restart: always
    volumes:
      - ./canal/deployer/conf:/usr/local/canal/conf
      - ./canal/logs:/usr/local/canal/logs
    ports:
      - "11111:11111"
    environment:
      - canal.instance.mysql.slaveId=12345
    networks:
      canal_es_net:
        ipv4_address: 172.20.0.20

  # Canal Adapter 1.1.7 同步数据到ES
  canal-adapter:
    image: canal/canal-adapter:v1.1.7
    container_name: canal-adapter
    restart: always
    volumes:
      - ./canal/adapter/conf:/usr/local/canal-adapter/conf
    ports:
      - "8081:8081"
    depends_on:
      - canal-deployer
      - elasticsearch
    networks:
      canal_es_net:
        ipv4_address: 172.20.0.21

networks:
  canal_es_net:
    driver: bridge
    ipam:
      config:
        - subnet: 172.20.0.0/16

（3）Logstash 基础配置（可选采集日志使用）

logstash/config/logstash.yml

bash 复制代码

http.host: "0.0.0.0"
xpack.monitoring.enabled: false

logstash/pipeline/log.conf

bash 复制代码

input {
  beats {
    port => 5044
  }
}
filter {
}
output {
  elasticsearch {
    hosts => ["http://elasticsearch:9200"]
    index => "app-log-%{+YYYY.MM.dd}"
  }
}

【2】安装分词器

安装IK分词器:8.13.0

（1）下载地址

https://release.infinilabs.com/analysis-ik/stable/

（2）把压缩包放到es的plugins文件夹下

（3）解压，删除压缩包

（4）重启es和Kibana

查看日志

powershell 复制代码

docker logs -f es|grep -E 'loaded plugin'

（5）测试ik分词器

ik_smart分词器

powershell 复制代码

GET _analyze
{
  "analyzer": "ik_smart",
  "text": "中华人民共和国万岁"
}

安装拼音检索插件:8.13.0

（1）下载地址

https://release.infinilabs.com/analysis-pinyin/stable/

（2）解压

现在的下载包可以直接使用

（3）放在es的plugins目录下

（4）重启es和Kibana

（5）测试拼音插件

powershell 复制代码

GET _analyze
{
  "analyzer": "pinyin",
  "text": "中华人民共和国万岁"
}

验证插件是否安装成功

powershell 复制代码

GET /_cat/plugins?v

【3】Kibana部署

【4】LogStash 部署

【5】Canal 部署

（1）Canal-deployer下载

用的Canal-deployer是1.1.7版本。

下载地址：https://github.com/alibaba/canal/releases

下载canal.deployer，可以直接监听MySQL的binlog，把自己伪装成MySQL的从库，只负责接收数据，并不做处理。

（2）解压

（3）配置修改

修改canal.properties，增加

canal.ip = 127.0.0.1

powershell 复制代码

cd /home/allensun/develop/canal/canal_deployer/conf
vim canal.properties

（4）修改example/instance.properties

powershell 复制代码

cd /home/allensun/develop/canal/canal_deployer/conf/example
vi instance.properties

（5）启动 Canal

到 bin 目录下，执行命令：./startup.sh

powershell 复制代码

cd /home/allensun/develop/canal/canal_deployer/bin && ./stop.sh && ./startup.sh

有报错

把jdk安装上，再执行

（6）启动成功检查

启动成功，检查canal/logs/canal.log

看到the canal server is running now ...表示启动成功。

powershell 复制代码

tail -f /home/allensun/develop/canal/canal_deployer/logs/canal/canal.log

（7）查看主库状态的时候有报错

powershell 复制代码

tail -f /home/allensun/develop/canal/canal_deployer/logs/example/example.log

SHOW MASTER STATUS;语句报错了，因为这是mysql8.0的语法，用来查看主库状态信息，mysql8.4应该使用

sql 复制代码

SHOW BINARY LOG STATUS;

2-安装Canal-adapter

（1）下载canal.adapter

同样是使用1.1.7版本，和deployer 保持一致。

canal.adapter，相当于canal的客户端，会从canal-deployer中获取数据，然后对数据进行同步，可以同步到MySQL、Elasticsearch和HBase等存储中去。

到https://github.com/alibaba/canal/releases 下载

（2）解压

（3）修改配置

修改 application.properties：

powershell 复制代码

cd /root/canal_adapter/conf
vi application.yml

yml 复制代码

server:
  port: 8081
spring:
  jackson:
    date-format: yyyy-MM-dd HH:mm:ss
    time-zone: GMT+8
    default-property-inclusion: non_null

canal.conf:
  mode: tcp #tcp kafka rocketMQ rabbitMQ
  flatMessage: true
  zookeeperHosts:
  syncBatchSize: 1000
  retries: -1
  timeout:
  accessKey:
  secretKey:
  consumerProperties:
    # canal tcp consumer
    canal.tcp.server.host: 127.0.0.1:11111
    canal.tcp.zookeeper.hosts:
    canal.tcp.batch.size: 500
    canal.tcp.username:
    canal.tcp.password:
    # kafka consumer
    kafka.bootstrap.servers: 127.0.0.1:9092
    kafka.enable.auto.commit: false
    kafka.auto.commit.interval.ms: 1000
    kafka.auto.offset.reset: latest
    kafka.request.timeout.ms: 40000
    kafka.session.timeout.ms: 30000
    kafka.isolation.level: read_committed
    kafka.max.poll.records: 1000
    # rocketMQ consumer
    rocketmq.namespace:
    rocketmq.namesrv.addr: 127.0.0.1:9876
    rocketmq.batch.size: 1000
    rocketmq.enable.message.trace: false
    rocketmq.customized.trace.topic:
    rocketmq.access.channel:
    rocketmq.subscribe.filter:
    # rabbitMQ consumer
    rabbitmq.host:
    rabbitmq.virtual.host:
    rabbitmq.username:
    rabbitmq.password:
    rabbitmq.resource.ownerId:

  srcDataSources:
    defaultDS:
      url: jdbc:mysql://rm-xxx.com:3306/nfturbo?useUnicode=true
      username: xxx
      password: xxx
  canalAdapters:
  - instance: example # canal instance Name or mq topic name
    groups:
    - groupId: g1
      outerAdapters:       
      - name: logger
      - name: es8
        hosts: localhost:9200 # 127.0.0.1:9200 for rest mode
        properties:
          mode: transport # or rest
          # security.auth: test:123456 #  only used for rest mode
          cluster.name: nfturbo-cluster

因为在outerAdapters下面，我们配置的 name 是 es8（如果你自己安装的是 es7，记得修改成对应的版本。），所以adapter将会自动加载 conf/es8 下的所有.yml结尾的配置文件，适配器表映射文件，创建并修改 conf/es8/mytest_user.yml文件：

powershell 复制代码

cd /root/canal_adapter/conf/es8
vi mytest_user.yml

powershell 复制代码

dataSourceKey: defaultDS
destination: example
groupId: g1
esMapping:
  _index: nfturbo_users
  _id: _id
  #  upsert: true
  #  pk: id
  sql: "select t.id as _id, t.nick_name as  nick_name, t.state as state,t.telephone as telephone  from users as t"
  #  objFields:
  #    _labels: array:;
  #etlCondition: "where a.c_time>={}"
  commitBatch: 3000

（4）启动canal.adapter

同样到 bin 目录下，执行命令：./startup.sh

powershell 复制代码

cd /root/canal_adapter/bin && ./stop.sh && ./startup.sh
tail -f /root/canal_adapter/logs/adapter/adapter.log

4-测试同步

（1）到 es 上创建 ES 索引

访问 es（http://ip:5601/app/home#/ ），然后进入开发工具，在控制台创建如下索引：

http://192.168.220.128:5601/app/dev_tools#/console

powershell 复制代码

PUT nfturbo_users
{
  "mappings": {
    "properties": {
      "nickname": {
        "type": "text"
      },
      "telephone": {
        "type": "text"
      },
      "state": {
        "type": "text"
      }
    }
  }
}

（2）数据库执行INSERT

sql 复制代码

INSERT INTO `nfturbo`.`users` (`id`,`gmt_create`,`gmt_modified`,`nick_name`,`password_hash`,`state`,`telephone`,`user_role`) VALUES (14,'2024-04-18 17:47:40','2024-04-18 17:47:42','test11111','c2975f0faec10adca0ecd729c8cbc0aa','INIT','13000000000','CUSTOMER')

（3）从 ES 查询：

powershell 复制代码

GET nfturbo_users/_search
{"_source": ["nick_name","telephone","state"],
  "query": {
    "match": {
      "nick_name": "test11111"
    }
  }
}