一、ElasticStack 技术栈
1、什么是ElasticStack 技术栈
ElasticsStak是elatic公司推出的一些列技术栈，早期有一个比较响亮的名字叫"ELK"用于日志采集系统，后续由于对该组件的扩充，引入很多新的组件，比如beats，xpack及云原生相关的组件，最终统称为elasticstack。
2、EFK 架构
3、ELFK 架构
3、ELFK 加价购升级
二、ElasticSearch集群部署
节点准备
ElasticStack
----> 2C 4G内存，磁盘50G+
elk91 10.0.0.91
elk92 10.0.0.92
elk93 10.0.0.93
1、ElaticSeach 的RPM的单点部署
1.1 下载Elcticsearch
wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.17.5-x86_64.rpm1.2 使用rpm包安装
rpm -ivh elasticsearch-7.17.5-x86_ 64.rpm1.3 设置别名
#vim ~/.bashrc
...
alias yy=`egrep -v "^#|^$"`
#source ~/.bashrc alias yy='egrep -v "^#|^$"'1. 4 修该配置文件
#yy /etc/elasticesearch/elaticsearch.yml
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: 10.0.0.91
discovery.seed_hosts:["10.0.0.91"]1.5 启动 elasticsearch 服务
systemctl enable --now elasticsearch1.6 访问 elasticsearch 的 WebUI
curl http://10.0.0.91:9200/
2、Elasticsearch 的 rpm 集群部署
2.1 下载 Elastic search
wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.17.5-x86_64.rpm2.2 所有节点配置主机解析
cat >> /etc/hosts <<EOF
10.0.0.91 elk91
10.0.0.92 elk92
10.0.0.93 elk93
EOF2.3 安装 elasticsearch
rpm -ivh elasticsearch-7.17.5-x86_64.rpm2.4修改配置文件
# yy /etc/elasticsearch/elasticsearch.yml
cluster.name: oldboyedu-linux87
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: 0.0.0.0
discovery.seed_hosts: ["elk91","elk92","elk93"]
cluster.initial_master_nodes: ["elk91","elk92","elk93"]
...
# scp /etc/elasticsearch/elasticsearch.yml elk92:/etc/elasticsearch/elasticsearch.yml
# scp /etc/elasticsearch/elasticsearch.yml elk93:/etc/elasticsearch/elasticsearch.yml2.5 启动集群
# systemctl stop elasticsearch
systemctl enable --now elasticsearch
# systemctl restart elasticsearch2.6 检查集群状态
# curl 10.0.0.91:9200/_cat/nodes
10.0.0.92 13 91 8 0.09 0.20 0.15 cdfhilmrstw - elk92
10.0.0.91 13 96 9 0.11 0.23 0.18 cdfhilmrstw * elk91
10.0.0.93 14 71 7 0.21 0.25 0.13 cdfhilmrstw - elk93
2.7 集群关机拍照
init 0温馨提示:
如果你的集群不正常工作，执行如下操作
(1)集群所有节点停止服务
systemctl stop elasticsearch
pkill java(2)确保停止服务完成，可以执行"ss -ntl"查看监听端口是否存在或者是查看java
ss -ntl
ps -ef | grep java(3)删除集群默认的数据
rm -rf /var/lib/elasticsearch/* /var/log/elasticsearch/* /tmp/*(4)重新启动集群即可
systemctl restart elasticsearch3、ElasticSearch 二进制单点部署
3.1 下载二进制的elasticsearch软件包
wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.17.5-linux-x86_64.tar.gz3.2 创建运行elasticsearch服务的普通用户
useadd -u 1000 xiaomeng3.3 创建工作目录
mkdir -pv /xiaomeng/{ softwares,data,logs}3.4 解压软件包
tar xf elasticsearch-7.17.5-linux-x86_64.tar.gz -C /xiaomeng/softwares/3.5 创符号链接
cd /oldboyedu/softwares/ && ln -svf elasticsearch-7.17.5 elasticsearch3.6 修改配置文件
# yy /xiaomeng/softwares/elasticsearch/config/elasticsearch.yml
cluster.name: xiaomeng
path.data: /xiaomeng/data/es7
path.logs: /xiaomeng/logs/es7
network.host: 0.0.0.0
discovery.seed_hosts: ["10.0.0.91"]
cluster.initial_master_nodes: ["10.0.0.91"]
...参数说明：
cluster.name: 指定ES集群名称
path.data: 指定数据目录。
path.logs: 指定日志目录。
network.host: 指定监听的地址。
discovery.seed_hosts: 当前集群的地址列表。
cluster.initial_master_nodes: 指定集群的master选举列表。
3.7 修改权限
install -d /xaiomeng/{logs,data}/es7 -o xaiomeng -g xiaomeng
chown xiaomeng:xiaoemng -R /xiaomeng/softwares/elasticsearch/*3.8 配置资源限制
cat > /etc/security/limits.d/es.conf <<EOF
* soft nofile 65535
* hard nofile 131070
* soft nproc 4096
* hard nproc 8192
EOF
ctrl + D # 重连后生效3.9 修改内核参数
cat > /etc/sysctl.d/es.conf <<EOF
vm.max_map_count=262144
EOF
sysctl -p /etc/sysctl.d/es.conf3.10 配置环境变量
cat > /etc/profile.d/elk.sh <<'EOF'
#!/bin/bash
export ES_HOME=/xiaomeng/softwares/elasticsearch
export PATH=$PATH:$ES_HOME/bin
EOF
source /etc/profile.d/elk.sh3.11 启动服务
su - oldboyedu -c "elasticsearch -d"3.12 验证节点是否正常
[root@elk93 ~]# curl 10.0.0.91:9200
{
"name" : "elk91",
"cluster_name" : "xiaomeng",
"cluster_uuid" : "Tbz4V4g_QkmcKAaLH3g2gg",
"version" : {
"number" : "7.17.5",
"build_flavor" : "default",
"build_type" : "tar",
"build_hash" : "8d61b4f7ddf931f219e3745f295ed2bbc50c8e84",
"build_date" : "2022-06-23T21:57:28.736740635Z",
"build_snapshot" : false,
"lucene_version" : "8.11.1",
"minimum_wire_compatibility_version" : "6.8.0",
"minimum_index_compatibility_version" : "6.0.0-beta1"
},
"tagline" : "You Know, for Search"
}
[root@elk93 ~]#温馨提示:
如果你的集群不正常工作，执行如下操作
（1）集群所有节点停止服务
pkill java
（2）确保停止服务完成，可以执行"ss -ntl"查看端口是否存在或者是查看 java
ss -ntl
ps -ef |grep java
（3）删除集群默认的数据
rm -rf /xiaomeng/logs/es7/* /xaiomeng/data/es7/*/tmp/*
（4）重新启动集群即可
su - oldboyedu -c "elasticsearch -d"
4、 Elasticsearch 二进制集群部署
4.1 下载二进制的elasticsearch 软件包
wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.17.5-linux-x86_64.tar.gz4.2 所有节点创建工作目录并添加用户
mkdir -pv /oldboyedu/{softwares,data,logs}
useradd -u 1000 oldboyedu
install -d /oldboyedu/{data,logs}/es7 -o oldboyedu -g oldboyedu4.3 修改 elk91 的配置文件
# yy /xaiomeng/softwares/elasticsearch/config/elasticsearch.yml
cluster.name: xiaomeng-linux87
path.data: /xiaomeng/data/es7
path.logs: /xiaomeng/logs/es7
network.host: 0.0.0.0
discovery.seed_hosts: ["10.0.0.91","10.0.0.92","10.0.0.93"]
cluster.initial_master_nodes: ["10.0.0.91","10.0.0.92","10.0.0.93"]4.4 停止 elk91 的服务
kill `ps -ef | grep 'elasticsearch' | awk '$3==1 {print $2}'`4.5 编写启动脚本，使用systemctl 管理
cat > /usr/lib/systemd/system/es7.service <<EOF
[Unit]
Description=Oldboyedu linux87 ES7 server daemon
Documentation=www.oldboyedu.com
After=network.target
[Service]
User=oldboyedu
LimitNOFILE=131070
LimitNPROC=8192
ExecStart=/oldboyedu/softwares/elasticsearch/bin/elasticsearch
[Install]
WantedBy=multi-user.target
EOF
systemctl daemon-reload
systemctl enable --now es74.6 将 elk91 的软件包同步到其他节点
scp -rp /xiaomeng/softwares/elasticsearch 10.0.0.92:/xiaomeng/softwares/
scp -rp /xiaomeng/softwares/elasticsearch 10.0.0.93:/xaiomeng/softwares/
scp /usr/lib/systemd/system/es7.service 10.0.0.92:/usr/lib/systemd/es7.service
scp /usr/lib/systemd/system/es7.service 10.0.0.92:/usr/lib/systemd/es7.service
scp /etc/sysctl.d/es.conf 10.0.0.92:/etc/sysctl.d/es.conf
scp /etc/sysctl.d/es.conf 10.0.0.93:/etc/sysctl.d/es.conf
scp /etc/profile.d/elk.sh 10.0.0.92:/etc/profile.d/elk.sh
scp /etc/profile.d/elk.sh 10.0.0.93:/etc/profile.d/elk.sh4.7 其他节点修改权限
chown xiaomeng:xiaomeng -R /xaiomeng/softwares/elasticsearch/*4.8 所有节点启动服务
source /etc/profile.d/elk.sh
sysctl -f /etc/sysctl.d/es.conf
sys temctl daemon-reload
systemctl enable --now es74.9 检查集群是否可用
[root@elk91 ~]# curl 10.0.0.92:9200/_cat/nodes
10.0.0.91 17 94 1 0.26 0.23 0.18 cdfhilmrstw * elk91
10.0.0.92 7 96 1 0.80 0.31 0.15 cdfhilmrstw - elk92
10.0.0.93 8 83 1 0.60 0.25 0.13 cdfhilmrstw - elk93
[root@elk91 ~]#5.、Elasticsearch 多实例部署 【了解即可】
5.1 下载二进制的·elasticsearch 软件包
elasticsearch-7.17.5-linux-x86_64.tar.gz
elasticsearch-6.8.23.tar.gz5.2 解压软件包
tar xf elasticsearch-6.8.23.tar.gz -C /xiaomeng/sohtwares
5.3 创建数据和目录日志
install -d /xiaomeng/(data,logs)/es6 -o xiaomeng --g xiaomeng 5.4 修改配置文件
# yy /xiaomeng/softwares/elasticsearch-6.8.23/config/elasticsearch.yml
cluster.name: xiaomeng
node.name: elk91
path.data: /xiaomeng/data/es6
path.logs: /xiaomeng/logs/es6
network.host: 0.0.0.0
http.port: 19200
transport.tcp.port: 19300
discovery.zen.ping.unicast.hosts: ["10.0.0.91", "10.0.0.92","10.0.0.93"]
discovery.zen.minimum_master_nodes: 2
...
参数说明:
cluster.name: 集群名称
node.name: 节点的名称
path.data: 数据目录
path.logs: 日志目录
network.host: 监听地址
http.port: 监听的端口号，web页面，走的http/https协议。
transport.tcp.port: ES集群内部数据传输端口，走的是tcp协议。
discovery.zen.ping.unicast.hosts: 集群的数据广播节点。
discovery.zen.minimum_master_nodes: 参与master选举的投票数量，建议是集群的半数以上，以防止脑裂。5.5 所有节点配置JDk环境
jdk-8u291-linux-x64.tar.gz
tar xf jdk-8u291-linux-x64.tar.gz -C /oldboyedu/softwares/
# scp -r /oldboyedu/softwares/jdk1.8.0_291/ 10.0.0.92:/oldboyedu/softwares/
# scp -r /oldboyedu/softwares/jdk1.8.0_291/ 10.0.0.93:/oldboyedu/softwares/5.6 编写启动脚本，使用systemctl管理
cat > /usr/lib/systemd/system/es6.service <<EOF
[Unit]
Description=xiaomeng linux ES6 server daemon
Documentation=www.xiaomeng.com
After=network.target
[Service]
User=xiaomeng
LimitNOFILE=131070
LimitNPROC=8192
Environment=JAVA_HOME=/xiaomeng/softwares/jdk1.8.0_291
ExecStart=/xiaomeng/softwares/elasticsearch-6.8.23/bin/elasticsearch
[Install]
WantedBy=multi-user.target
EOF5.7 同步软件到其他节点
scp -rp /xiaomeng/softwares/elasticsearch-6.8.23/ 10.0.0.92:/xiaomeng/softwares/
scp -rp /xiaomeng/softwares/elasticsearch-6.8.23/ 10.0.0.93:/xiaomeng/softwares/
scp /usr/lib/systemd/system/es6.service 10.0.0.92:/usr/lib/systemd/system/es6.service
scp /usr/lib/systemd/system/es6.service 10.0.0.93:/usr/lib/systemd/system/es6.service5.8 所有节点修改主机名和权限
sed -ri "/^node.name:/s#(node.name:) elk91#\1 `hostname`#" /xiaomeng/softwares/elasticsearch-6.8.23/config/elasticsearch.yml
chown xiaomeng:xiaomeng -R /xiaomeng/softwares/elasticsearch-6.8.23/5.9 所有节点启动服务
systemctl daemon-reload
systemctl enable --now es65.10 访问 ES 6的WebUI
[root@elk91 ~]# curl 10.0.0.91:19200/_cat/nodes
10.0.0.91 20 97 26 0.62 0.21 0.10 mdi - elk91
10.0.0.92 18 97 24 0.54 0.27 0.17 mdi * elk92
10.0.0.93 14 96 21 0.31 0.14 0.08 mdi - elk93
[root@elk91 ~]#6、 Ela taicSearch 基于docker部署
6.1安装docker环境
docker-compose-binary.tar.gz
tar xf oldboyedu-docker-compose-binary.tar.gz
./install-docker.sh install6.2 基于 docker部署ES服务
docker run -p 29200:9200 \
-e "discovery.type=single-node" \
--name "xiaomeng-linux-es7" \
--restart always \
-d \
docker.elastic.co/elasticsearch/elasticsearch:7.17.5