日志分析集群安装部署（ELK） 保姆级教程

创建用户es不用root用户 创建的用户是elasticsearch密码:elasticsearch

useradd elasticsearch && echo elasticsearch|passwd --stdin elasticsearch

1.优化最打进程数、最大文件打开数、优化虚拟内存 、elastic.co vim /etc/security/limit.conf

* soft nofile 65536
* hard nofile 131072
* soft nproc 4096
* hard nproc 6553

vim /etc/sysctl.conf

vm.max_map_count=262144
sysctl -p

1.获取安装包官网 Elastic --- 搜索 AI 公司 | Elastic

tar  xf  elasticsearch-8.13.4-linux-x86_64.tar.gz -C /usr/local
mv  /usr/local/elasticsearch-8.13.4 /usr/local/es
chown -R elasticsearch.elasticsearch /usr/local/es

2.配置环境变量

vim  /etc/profile
JAVA_HOME=/usr/local/es/jdk
ES_HOME=/usr/local/es
PATH=$JAVA_HOME/bin:$ES_HOME/bin:$PATH
export JAVA_HOME  ES_HOME
#刷新环境变量
source  /etc/profile

4.创建用来存储数据和存放证书并赋予权限

mkdir -p /usr/local/es/data
mkdir -p /usr/local/es/config/certs
chown  -R  elasticsearch:elasticsearch  /usr/local/es

注意：所有节点的操作都是一样的，到目前为止 5.签发证书 证书如果说是购买的不用生成非法证书 可以只在第一台服务器上生成非法正书

su - elasticsearch
cd  /usr/local/es/bin
./elasticsearch-certutil  ca   #生成ca证书
直接回车2次
./elasticsearch-certutil  cert  --ca  elastic-stack-ca.p12
直接回车3次
cd /usr/local/es
证书转移到我们创建存放证书的目录
mv elastic-certificates.p12    config/certs
mv elastic-stack-ca.p12 config/certs/

6.设置集群多节点HTTP证书

cd  /usr/lolcal/es/bin
./elasticsearch-certutil  http
是否生成CSR，选择 N ，不需要
  Generate a CSR? [y/N]N
是否使用已经存在的CA证书，选择 y ，因为已经创建签发好了CA 
Use an existing CA? [y/N]y
指定CA证书的路径地址，CA Path:后写绝对路径   
CA Path: /usr/local/es/config/certs/elastic-stack-ca.p12
设置密钥库的密码，直接 回车 即可   
Password for elastic-stack-ca.p12:
设置证书的失效时间，这里的y表示年，5y则代表失效时间5年 
For how long should your certificate be valid? [5y] 5y
是否需要为每个节点都生成证书，选择 N 无需每个节点都配置证书
Generate a certificate per node? [y/N]N
输入需连接集群节点主机名信息，一行输入一个IP地址，空行回车结束
es1-flower.com
es2-flower.com
es-3.flower.com
确认以上是否为正确的配置，输入 Y 表示信息正确
Is this correct [Y/n]Y
输入需连接集群节点IP信息，一行输入一个IP地址，空行回车结束
192.168.72.100
192.168.72.101
192.168.72.102
确认以上是否为正确的配置，输入 Y 表示信息正确
Is this correct [Y/n]Y
是否要更改以上这些选项，选择 N ，不更改证书选项配置 
Do you wish to change any of these options? [y/N]N
是否要给证书加密，不需要加密，两次 回车 即可 
cd  /usr/local/es
unzip  elasticsearch-ssl-http.zip  #解压http证书的压缩包
mv ./elasticsearch/http.p12   config/certis
mv ./kibana/elasticsearch-ca.pem  config/certs
​
#再把证书分发到其他的节点上
scp   /usr/local/es/config/certs/*   192.168.72.101:/usr/local/es/config/certs
scp   /usr/local/es/config/certs/*   192.168.72.101:/usr/local/es/config/certs

7.修改配置

cd  /usr/local/es/config
vim   elasticsearch.yml
cluster.name: xingdian-es
node.name: es-1.xingdian.com
path.data: /usr/local/es/data
path.logs: /usr/local/es/logs
network.host: 0.0.0.0
http.port: 9200  # 种子主机，在选举时用于发现其他主机的，最好配置多个
discovery.seed_hosts: ["es-1.xingdian.com","es-2.xingdian.com","es-3.xingdian.com"]
cluster.initial_master_nodes: ["es-1.xingdian.com","es-2.xingdian.com","es-3.xingdian.com"]
xpack.security.enabled: true
xpack.security.enrollment.enabled: true
xpack.security.http.ssl:
 enabled: true
 keystore.path: /usr/local/es/config/certs/http.p12
 keystore.password: 123456  #如果生成证书时设置了密码则要添加密码配置
 truststore.path: /usr/local/es/config/certs/http.p12
 truststore.password: 123456 #如果生成证书时设置了密码则要添加密码配置
xpack.security.transport.ssl:
 enabled: true
 verification_mode: certificate
 keystore.path: /usr/local/es/config/certs/elastic-certificates.p12
 keystore.password: 123456  #如果生成证书时设置了密码则要添加密码配置
 truststore.path: /usr/local/es/config/certs/elastic-certificates.p12
 truststore.password: 123456 #如果生成证书时设置了密码则要添加密码配置
http.host: [_local_, _site_]
ingest.geoip.downloader.enabled: false
xpack.security.http.ssl.client_authentication: none
http.cors.enabled: true
http.cors.allow-origin: "*"

8.JVM参数调整

vim  /usr/local/es/config/jvm.options
内存大小
-Xms2g
-Xms2g
注意该值为实际内存的二分之一
#启动集群 普通用户

nohup  usr/local/es/bin/elasticsearch  &

9.设置的登入密码

/usr/local/es/bin/elasticsearch-reset-password -u elastic -i

10.浏览器访问 win上做解析 https://es1-flower.com 访问成功表示elasticsearch集群搭建成功

kibana的搭建 1.在官方获取安装包 官网[https://www.elastic.co] 2.安装部署

tar xf kibana-8.13.4-linux-x86_64.tar.gz  -C /usr/local/
mv /usr/local/kibana-8.13.4/ /usr/local/kibana
mkdir /usr/local/kibana/config/certs   #创建证书存储目录

11.配置修改

vim  /usr/local/kibana/config/kiabna.yml
server.port: 5601                   #kibana 的端口
server.host: "192.168.72.158"  #kibana 的主机ip
server.name: "Kibana"
elasticsearch.hosts: ["https://es-1.com:9200","https://es-2.com:9200","https://es-3.com:9200"]
elasticsearch.username: "kibana"
elasticsearch.password: "elastic"
elasticsearch.ssl.certificateAuthorities: [ "/usr/local/kibana/config/certs/elasticsearch-ca.pem" ]
i18n.locale: "zh-CN"

12.获取elasticsearch的ca证书

scp     192.168.72.100:/usr/local/es/config/certs/elasticsearch-ca.pem    /usr/local/kibana/config/certs

13.创建运行用户

useradd  kibana
echo "kibana" | passwd  --stdin  kibana
chown  kibana.kibana  /usr/local/kibana/  -R

14.用普通用户启动kibana

nohup  /usr/local/kibana/bin/kibana  &

15.如果报这样的错

Error: [config validation of [elasticsearch].username]: value of "elastic" is forbidden.
在es集群中设置为kibana账户设定密码
usr/local/es/bin/elasticsearch-reset-password  -u kibana  -i

16.浏览器进行访问 IP:5601

logstah的搭建

1.logstash的部署对应跟es集群是同一版本 官方获取安装包 <www.elastic.co>

2.解压安装

tar xf logstash-8.13.4-linux-x86_64.tar.gz  -C /usr/local/
mv /usr/local/logstash-8.13.4/   /usr/local/logstsh

3.配置jdk环境 logstash中自带的是Java17

vim /etc/profile
JAVA_HOME=/usr/local/logstash/jdk
PATH=$JAVA_HOME/bin:$PATH
export JAVA_HOME PATH
#做软连接的方式方便调用logstash
ln -s /usr/local/logstash/bin/logstash   /usr/bin/logstash

4.模拟运行是否可以使用

logstash -e 'input { stdin { } } output { stdout {} }'

5.创建logstash存储elasticsearch集群的ca证书

mkdir  /usr/local/logstash/config/certs

6.将证书存放在创建的目录下

scp 192.168.72.100:/usr/local/es/config/certs/elasticsearch-ca.pem   /usr/local/logstash/config/certs

7.创建logstash采集数据的配置文件 采集的是nginx的日志{在这里创建在opt目录下}

vim  /opt/stdin.comf
input {
    file {
        path => "/var/log/nginx/access.log"
        start_position => "beginning"
    }
}
output {
    elasticsearch {
        index => "nginx_access_logstash"
        hosts => [ "https://es1-flower.com:9200" ]
        cacert => "/usr/local.logstash/config/certs/elasticsearch-ca.pem"
        user => "elastic"
        password => "elastic"
    }
}