Es集群部署 - 技术栈

组件全家套

版本说明

主机准备

1.解压安装

2.运行环境配置

2.1修改每个节点linux系统限制

[2.2 修改每个节点 linux 系统配置](#2.2 修改每个节点 linux 系统配置)

[2.3 调整vm.max_map_count的大小](#2.3 调整vm.max_map_count的大小)

[2.4 重启验证配置](#2.4 重启验证配置)

[3. 配置ES](#3. 配置ES)

[3.1 每个节点创建ES用户，ES不能使用root启动](#3.1 每个节点创建ES用户，ES不能使用root启动)

[3.2 每个节点创建数据目录，日志目录](#3.2 每个节点创建数据目录，日志目录)

[3.3 配置es文件目录，跨域请求,修改每个节点特殊的内容](#3.3 配置es文件目录，跨域请求,修改每个节点特殊的内容)

3.4修改JVM启动参数

[3.5 配置开机启动，启动服务](#3.5 配置开机启动，启动服务)

组件全家套

LogStash数据同步组件相关
Lostash同步Mysql数据到Elasticsearch（一）服务介绍及环境配置
 Lostash同步Mysql数据到ElasticSearch（二）logstash脚本配置和常见坑点
 Lostash同步Mysql数据到Elasticsearch（三）Elasticsearch模板与索引设置
 Lostash同步Mysql数据到Elasticsearch（四）通过kibana辅助查看同步情况

kibana开发工具相关
Kibana开发工具安装

版本说明

|---------------|-----------------------------------------|--------|
| 组件 | 版本 | 备注 |
| Elasticsearch | elasticsearch-7.7.1-linux-x86_64.tar.gz | |

主机准备

|---------|---------------|---|
| 主机名 | ip | |
| node144 | 172.168.9.144 | |
| node145 | 172.168.9.146 | |
| node146 | 172.168.9.146 | |

1.解压安装

注意：Elasticsearch需要安装JDK环境，配置前请先行安装JDK环境

复制代码

tar -zxvf elasticsearch-7.7.1-linux-x86_64.tar.gz -C /usr/local/

2.运行环境配置

2.1修改每个节点linux系统限制

复制代码

 vim /etc/security/limits.conf

复制代码

* hard nproc 4096
* soft nproc 4096
* hard nofile 655300
* soft nofile 655300

* soft memlock unlimited
* hard memlokc unlimited

2.2 修改每个节点 linux 系统配置

复制代码

编辑文件，查找如下参数进行配置
vim /etc/systemd/system.conf

DefaultLimitNOFILE=65536

DefaultLimitNPROC=32000

DefaultLimitMEMLOCK=infinity

复制代码

ulimit -a  #查看重启前系统配置

2.3 调整vm.max_map_count的大小

复制代码

sysctl -a|grep vm.max_map_count  #查看当前系统配置大小，可以设置为262144 建议设置为262144,最大655360
vim /etc/sysctl.conf

vm.max_map_count=655360

复制代码

sysctl -p  #生效配置

2.4 重启验证配置

复制代码

ulimit -a  #重启后查看配置是否生效
sysctl -a|grep vm.max_map_count #重启后查看配置是否生效

3. 配置ES

3.1 每个节点创建ES用户，ES不能使用root启动

|---------------------------------------------------------------|
| groupadd elasticsearch useradd elasticsearch -g elasticsearch |

|----------------------------------------------------------------------------------------------------------------------------------------------------|
| mkdir -p /mnt/elk/elasticsearch/{data,logs} chown -R elasticsearch. /mnt/elk/elasticsearch chown -R elasticsearch. /usr/local/elasticsearch-7.7.1/ |

3.3 配置es文件目录，跨域请求,修改每个节点特殊的内容

复制代码

vi /usr/local/elasticsearch-7.7.1/config/elasticsearch.yml

复制代码

# ======================== Elasticsearch Configuration =========================
#
# NOTE: Elasticsearch comes with reasonable defaults for most settings.
#       Before you set out to tweak and tune the configuration, make sure you
#       understand what are you trying to accomplish and the consequences.
#
# The primary way of configuring a node is via this file. This template lists
# the most important settings you may want to configure for a production cluster.
#
# Please consult the documentation for further information on configuration options:
# https://www.elastic.co/guide/en/elasticsearch/reference/index.html
#
# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
#cluster.name: my-application

cluster.name: es_cluster


#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
#node.name: node-1

node.name: es_znyg104
node.master: true
node.data: true

#
# Add custom attributes to the node:
#
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#

path.data: /mnt/elk/elasticsearch/data

#
# Path to log files:
#

path.logs: /mnt/elk/elasticsearch/logs

#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
#bootstrap.memory_lock: true

bootstrap.memory_lock: true

#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# Set the bind address to a specific IP (IPv4 or IPv6):
#
#network.host: 192.168.0.1

network.host: 0.0.0.0


#
# Set a custom port for HTTP:
#
#http.port: 9200

http.port: 9200

http.cors.enabled: true
http.cors.allow-origin: "*"
http.cors.allow-credentials: true

#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when this node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
#discovery.seed_hosts: ["host1", "host2"]

discovery.zen.minimum_master_nodes: 2
discovery.zen.ping_timeout: 3s
discovery.seed_hosts: ["172.168.9.104", "172.168.9.105", "172.168.9.203", "172.168.9.204", "172.168.9.205"]


#
# Bootstrap the cluster using an initial set of master-eligible nodes:
#
#cluster.initial_master_nodes: ["node-1", "node-2"]

cluster.initial_master_nodes: ["es_znyg104", "es_znyg204"]

#
# For more information, consult the discovery and cluster formation module documentation.
#
# ---------------------------------- Gateway -----------------------------------
#
# Block initial recovery after a full cluster restart until N nodes are started:
#
#gateway.recover_after_nodes: 3
#
# For more information, consult the gateway module documentation.
#
# ---------------------------------- Various -----------------------------------
#
# Require explicit names when deleting indices:
#
#action.destructive_requires_name: true

transport.tcp.port: 9300

3.4修改JVM启动参数

复制代码

vi /usr/local/elasticsearch-7.7.1/config/jvm.options

主要修改的是运行内存，日志文件路径等，按照自己定义的进行处理

|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| -xms2g -xmx2g -XX:HeapDumpPath=/mnt/elk/elasticsearch -XX:ErrorFile=/mnt/elk/elasticsearch/hs_err_pid%p.log 8:-xloggc:/mnt/elk/elasticsearch/logs/gc.log 9-:-xlog:gc*,gc+age=trace,safepoint:file=/mnt/elk/elasticsearch/logs/gc.log:utctime,pid,tags :filecount=32,filesize=64m |

3.5 配置开机启动，启动服务

复制代码

cd /etc/init.d
vi elasticsearch

复制代码

#!/bin/bash
#chkconfig: 345 63 37
#description: elasticsearch
#processname: elasticsearch-7.7.1

export ES_HOME=/usr/local/elasticsearch-7.7.1     # elasticsearch所在目录

case $1 in
        start)
                su elasticsearch<<!        # smarthome对应的是启动elasticsearch的账号
                cd $ES_HOME
                ./bin/elasticsearch -d -p pid
                exit
!
                echo "elasticsearch is started"
                ;;
        stop)
                pid=`cat $ES_HOME/pid`
                kill -9 $pid
                echo "elasticsearch is stopped"
                ;;
        restart)
                pid=`cat $ES_HOME/pid`
                kill -9 $pid
                echo "elasticsearch is stopped"
                sleep 1
                su elasticsearch<<!    # 启动elasticsearch的账户/用户
                cd $ES_HOME
                ./bin/elasticsearch -d -p pid
                exit
!
                echo "elasticsearch is started"
        ;;
    *)
        echo "start|stop|restart"
        ;;
esac
exit 0

保存退出，赋予执行权限

chmod +x elasticsearch

添加到开机启动任务

chkconfig --add elasticsearch

启动，重启，停止

复制代码

service elasticsearch start/stop/restart

遇到could not find java in bundled jdk at /usr/local/elasticsearch-7.7.1/jdk/bin/java，查看自己的es安装目录是否是root权限，修改授权为ES

验证服务启动

复制代码

ps aux|grep elasticsearch
curl 127.0.0.1:9200