全面了解 ElasticStack 技术栈：RPM、二进制、多实例及 Docker 部署

一、ElasticStack 技术栈

1、什么是ElasticStack 技术栈

ElasticsStak是elatic公司推出的一些列技术栈，早期有一个比较响亮的名字叫"ELK"用于日志采集系统，后续由于对该组件的扩充，引入很多新的组件，比如beats，xpack及云原生相关的组件，最终统称为elasticstack。

2、EFK 架构

3、ELFK 架构

3、ELFK 加价购升级

二、ElasticSearch集群部署

节点准备

ElasticStack

----> 2C 4G内存，磁盘50G+

elk91 10.0.0.91

elk92 10.0.0.92

elk93 10.0.0.93

1、ElaticSeach 的RPM的单点部署

1.1 下载Elcticsearch

wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.17.5-x86_64.rpm

1.2 使用rpm包安装

rpm -ivh elasticsearch-7.17.5-x86_ 64.rpm

1.3 设置别名

#vim ~/.bashrc

...

alias yy=`egrep -v "^#|^$"`



#source ~/.bashrc alias yy='egrep -v "^#|^$"'

1. 4 修该配置文件

#yy /etc/elasticesearch/elaticsearch.yml

path.data: /var/lib/elasticsearch

path.logs: /var/log/elasticsearch

network.host: 10.0.0.91

discovery.seed_hosts:["10.0.0.91"]

1.5 启动 elasticsearch 服务

systemctl enable --now elasticsearch

1.6 访问 elasticsearch 的 WebUI

curl http://10.0.0.91:9200/

2、Elasticsearch 的 rpm 集群部署

2.1 下载 Elastic search

wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.17.5-x86_64.rpm

2.2 所有节点配置主机解析

cat >> /etc/hosts <<EOF

10.0.0.91 elk91

10.0.0.92 elk92

10.0.0.93 elk93

EOF

2.3 安装 elasticsearch

rpm -ivh elasticsearch-7.17.5-x86_64.rpm

2.4修改配置文件

# yy /etc/elasticsearch/elasticsearch.yml

cluster.name: oldboyedu-linux87

path.data: /var/lib/elasticsearch

path.logs: /var/log/elasticsearch

network.host:  0.0.0.0

discovery.seed_hosts: ["elk91","elk92","elk93"]

cluster.initial_master_nodes: ["elk91","elk92","elk93"]

...



# scp /etc/elasticsearch/elasticsearch.yml  elk92:/etc/elasticsearch/elasticsearch.yml

# scp /etc/elasticsearch/elasticsearch.yml  elk93:/etc/elasticsearch/elasticsearch.yml

2.5 启动集群

# systemctl stop elasticsearch

systemctl enable --now elasticsearch

# systemctl restart elasticsearch

2.6 检查集群状态

# curl 10.0.0.91:9200/_cat/nodes

10.0.0.92 13 91 8 0.09 0.20 0.15 cdfhilmrstw - elk92

10.0.0.91 13 96 9 0.11 0.23 0.18 cdfhilmrstw * elk91

10.0.0.93 14 71 7 0.21 0.25 0.13 cdfhilmrstw - elk93

2.7 集群关机拍照

init 0

温馨提示:

如果你的集群不正常工作，执行如下操作

(1)集群所有节点停止服务

systemctl stop elasticsearch

pkill java

(2)确保停止服务完成，可以执行"ss -ntl"查看监听端口是否存在或者是查看java

ss -ntl

ps -ef | grep java

(3)删除集群默认的数据

rm -rf /var/lib/elasticsearch/* /var/log/elasticsearch/* /tmp/*

(4)重新启动集群即可

systemctl restart elasticsearch

3、ElasticSearch 二进制单点部署

3.1 下载二进制的elasticsearch软件包

wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.17.5-linux-x86_64.tar.gz

3.2 创建运行elasticsearch服务的普通用户

useadd -u 1000 xiaomeng

3.3 创建工作目录

mkdir -pv /xiaomeng/{ softwares,data,logs}

3.4 解压软件包

tar xf elasticsearch-7.17.5-linux-x86_64.tar.gz -C /xiaomeng/softwares/

3.5 创符号链接

cd /oldboyedu/softwares/ && ln -svf elasticsearch-7.17.5 elasticsearch

3.6 修改配置文件

# yy /xiaomeng/softwares/elasticsearch/config/elasticsearch.yml

cluster.name: xiaomeng

path.data: /xiaomeng/data/es7

path.logs: /xiaomeng/logs/es7

network.host: 0.0.0.0

discovery.seed_hosts: ["10.0.0.91"]

cluster.initial_master_nodes: ["10.0.0.91"]

...

参数说明：

cluster.name: 指定ES集群名称

path.data: 指定数据目录。

path.logs: 指定日志目录。

network.host: 指定监听的地址。

discovery.seed_hosts: 当前集群的地址列表。

cluster.initial_master_nodes: 指定集群的master选举列表。

3.7 修改权限

install -d /xaiomeng/{logs,data}/es7 -o xaiomeng -g xiaomeng
chown xiaomeng:xiaoemng -R /xiaomeng/softwares/elasticsearch/*

3.8 配置资源限制

cat > /etc/security/limits.d/es.conf <<EOF
*  soft nofile 65535
*  hard nofile 131070
*  soft nproc 4096
*  hard nproc 8192
EOF
ctrl + D  # 重连后生效

3.9 修改内核参数

cat > /etc/sysctl.d/es.conf <<EOF
vm.max_map_count=262144
EOF

sysctl -p  /etc/sysctl.d/es.conf

3.10 配置环境变量

cat > /etc/profile.d/elk.sh  <<'EOF'
#!/bin/bash

export ES_HOME=/xiaomeng/softwares/elasticsearch
export PATH=$PATH:$ES_HOME/bin
EOF

source  /etc/profile.d/elk.sh

3.11 启动服务

su - oldboyedu -c "elasticsearch -d"

3.12 验证节点是否正常

[root@elk93 ~]# curl 10.0.0.91:9200
{
  "name" : "elk91",
  "cluster_name" : "xiaomeng",
  "cluster_uuid" : "Tbz4V4g_QkmcKAaLH3g2gg",
  "version" : {
    "number" : "7.17.5",
    "build_flavor" : "default",
    "build_type" : "tar",
    "build_hash" : "8d61b4f7ddf931f219e3745f295ed2bbc50c8e84",
    "build_date" : "2022-06-23T21:57:28.736740635Z",
    "build_snapshot" : false,
    "lucene_version" : "8.11.1",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}
[root@elk93 ~]#

温馨提示:

如果你的集群不正常工作，执行如下操作

（1）集群所有节点停止服务

pkill java

（2）确保停止服务完成，可以执行"ss -ntl"查看端口是否存在或者是查看 java

ss -ntl

ps -ef |grep java

（3）删除集群默认的数据

rm -rf /xiaomeng/logs/es7/* /xaiomeng/data/es7/*/tmp/*

（4）重新启动集群即可

su - oldboyedu -c "elasticsearch -d"

4、 Elasticsearch 二进制集群部署

4.1 下载二进制的elasticsearch 软件包

wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.17.5-linux-x86_64.tar.gz

4.2 所有节点创建工作目录并添加用户

mkdir -pv /oldboyedu/{softwares,data,logs}
useradd -u 1000 oldboyedu
install -d /oldboyedu/{data,logs}/es7 -o oldboyedu -g oldboyedu

4.3 修改 elk91 的配置文件

# yy /xaiomeng/softwares/elasticsearch/config/elasticsearch.yml

cluster.name: xiaomeng-linux87

path.data: /xiaomeng/data/es7

path.logs: /xiaomeng/logs/es7

network.host: 0.0.0.0

discovery.seed_hosts: ["10.0.0.91","10.0.0.92","10.0.0.93"]

cluster.initial_master_nodes: ["10.0.0.91","10.0.0.92","10.0.0.93"]

4.4 停止 elk91 的服务

kill `ps -ef | grep 'elasticsearch' | awk '$3==1 {print $2}'`

4.5 编写启动脚本，使用systemctl 管理

cat > /usr/lib/systemd/system/es7.service <<EOF
[Unit]
Description=Oldboyedu linux87 ES7 server daemon
Documentation=www.oldboyedu.com
After=network.target

[Service]
User=oldboyedu
LimitNOFILE=131070
LimitNPROC=8192
ExecStart=/oldboyedu/softwares/elasticsearch/bin/elasticsearch

[Install]
WantedBy=multi-user.target
EOF
systemctl daemon-reload
systemctl enable --now es7

4.6 将 elk91 的软件包同步到其他节点

scp -rp /xiaomeng/softwares/elasticsearch 10.0.0.92:/xiaomeng/softwares/

scp -rp /xiaomeng/softwares/elasticsearch 10.0.0.93:/xaiomeng/softwares/



scp /usr/lib/systemd/system/es7.service 10.0.0.92:/usr/lib/systemd/es7.service

scp /usr/lib/systemd/system/es7.service 10.0.0.92:/usr/lib/systemd/es7.service



scp /etc/sysctl.d/es.conf 10.0.0.92:/etc/sysctl.d/es.conf

scp /etc/sysctl.d/es.conf 10.0.0.93:/etc/sysctl.d/es.conf



scp /etc/profile.d/elk.sh 10.0.0.92:/etc/profile.d/elk.sh

scp /etc/profile.d/elk.sh 10.0.0.93:/etc/profile.d/elk.sh

4.7 其他节点修改权限

chown xiaomeng:xiaomeng -R /xaiomeng/softwares/elasticsearch/*

4.8 所有节点启动服务

source /etc/profile.d/elk.sh

sysctl -f /etc/sysctl.d/es.conf

sys temctl daemon-reload

systemctl enable --now es7

4.9 检查集群是否可用

[root@elk91 ~]# curl 10.0.0.92:9200/_cat/nodes

10.0.0.91 17 94 1 0.26 0.23 0.18 cdfhilmrstw * elk91

10.0.0.92  7 96 1 0.80 0.31 0.15 cdfhilmrstw - elk92

10.0.0.93  8 83 1 0.60 0.25 0.13 cdfhilmrstw - elk93

[root@elk91 ~]#

5.、Elasticsearch 多实例部署【了解即可】

5.1 下载二进制的·elasticsearch 软件包

elasticsearch-7.17.5-linux-x86_64.tar.gz

elasticsearch-6.8.23.tar.gz

5.2 解压软件包

tar xf elasticsearch-6.8.23.tar.gz -C /xiaomeng/sohtwares

5.3 创建数据和目录日志

install -d /xiaomeng/(data,logs)/es6 -o xiaomeng --g xiaomeng

5.4 修改配置文件

# yy /xiaomeng/softwares/elasticsearch-6.8.23/config/elasticsearch.yml 
cluster.name: xiaomeng
node.name: elk91
path.data: /xiaomeng/data/es6
path.logs: /xiaomeng/logs/es6
network.host: 0.0.0.0
http.port: 19200
transport.tcp.port: 19300
discovery.zen.ping.unicast.hosts: ["10.0.0.91", "10.0.0.92","10.0.0.93"]
discovery.zen.minimum_master_nodes: 2
...

	参数说明:
cluster.name: 集群名称
node.name: 节点的名称
path.data: 数据目录
path.logs: 日志目录
network.host: 监听地址
http.port: 监听的端口号，web页面，走的http/https协议。
transport.tcp.port: ES集群内部数据传输端口，走的是tcp协议。
discovery.zen.ping.unicast.hosts:  集群的数据广播节点。
discovery.zen.minimum_master_nodes: 参与master选举的投票数量，建议是集群的半数以上，以防止脑裂。

5.5 所有节点配置JDk环境

jdk-8u291-linux-x64.tar.gz
tar xf jdk-8u291-linux-x64.tar.gz -C /oldboyedu/softwares/

# scp  -r /oldboyedu/softwares/jdk1.8.0_291/ 10.0.0.92:/oldboyedu/softwares/
# scp  -r /oldboyedu/softwares/jdk1.8.0_291/ 10.0.0.93:/oldboyedu/softwares/

5.6 编写启动脚本，使用systemctl管理

cat > /usr/lib/systemd/system/es6.service <<EOF

[Unit]

Description=xiaomeng linux ES6 server daemon

Documentation=www.xiaomeng.com

After=network.target



[Service]

User=xiaomeng

LimitNOFILE=131070

LimitNPROC=8192

Environment=JAVA_HOME=/xiaomeng/softwares/jdk1.8.0_291

ExecStart=/xiaomeng/softwares/elasticsearch-6.8.23/bin/elasticsearch



[Install]

WantedBy=multi-user.target

EOF

5.7 同步软件到其他节点

scp -rp /xiaomeng/softwares/elasticsearch-6.8.23/ 10.0.0.92:/xiaomeng/softwares/
scp -rp /xiaomeng/softwares/elasticsearch-6.8.23/ 10.0.0.93:/xiaomeng/softwares/
	
scp /usr/lib/systemd/system/es6.service  10.0.0.92:/usr/lib/systemd/system/es6.service 
scp /usr/lib/systemd/system/es6.service  10.0.0.93:/usr/lib/systemd/system/es6.service

5.8 所有节点修改主机名和权限

sed -ri "/^node.name:/s#(node.name:) elk91#\1 `hostname`#" /xiaomeng/softwares/elasticsearch-6.8.23/config/elasticsearch.yml
chown xiaomeng:xiaomeng -R /xiaomeng/softwares/elasticsearch-6.8.23/

5.9 所有节点启动服务

systemctl daemon-reload
systemctl enable --now es6

5.10 访问 ES 6的WebUI

[root@elk91 ~]# curl 10.0.0.91:19200/_cat/nodes
10.0.0.91 20 97 26 0.62 0.21 0.10 mdi - elk91
10.0.0.92 18 97 24 0.54 0.27 0.17 mdi * elk92
10.0.0.93 14 96 21 0.31 0.14 0.08 mdi - elk93
[root@elk91 ~]#

6、 Ela taicSearch 基于docker部署

6.1安装docker环境

docker-compose-binary.tar.gz

tar xf oldboyedu-docker-compose-binary.tar.gz 

./install-docker.sh install

6.2 基于 docker部署ES服务

docker run -p 29200:9200 \

   -e "discovery.type=single-node" \

   --name "xiaomeng-linux-es7" \

   --restart always \

   -d \

   docker.elastic.co/elasticsearch/elasticsearch:7.17.5