elk部署安装 - 技术栈

elk部署

前提准备
1、elasticsearch
2、kibana
3、logstash

前提准备

1、提前装好docker docker-compose相关命令

2、替换docker仓库地址国内镜像源

bash 复制代码

cd /etc/docker
vi daemon.json
# 替换内容
{"registry-mirrors": [
"https://docker.1panel.dev",
"https://docker.fxxk.dedyn.io",
"https://docker.xn--6oq72ry9d5zx.cn",
"https://docker.m.daocloud.io",
"https://a.ussh.net",
"https://docker.zhai.cm"]}

参考地址：https://blog.csdn.net/llc580231/article/details/139979603

3、写一个docker-compose.yml(新建一个网络、可通过容器名访问网络)

bash 复制代码

version: '2.1'
services:

 # 网络配置
networks:
  elastic_net:
    driver: bridge

4、提前下载好相关镜像

bash 复制代码

docker pull elasticsearch:7.10.1
docker pull kibana:7.10.1
docker pull logstash:7.10.1
docker images

1、elasticsearch

bash 复制代码

#(需要是root账户)
vim /etc/sysctl.conf 
#(文件最后添加一行)
vm.max_map_count=262144 
# 生效
sysctl -p

docker-compose.yml新增es相关配置

bash 复制代码

es:
    image: elasticsearch:7.10.1
    container_name: es
    environment:
      - "discovery.type=single-node"
      - "TZ=Asia/Shanghai"
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - /home/es:/usr/share/elasticsearch/data
    ports:
      - 9200:9200
    networks:
      - elastic_net

执行新建容器命令./docker-compose -f conf/docker-compose.yml up --build -d es

访问9200端口成功

2、kibana

docker-compose.yml新增kibana相关配置

bash 复制代码

kibana:
    container_name: kibana
    image: kibana:7.10.1
    restart: unless-stopped
    environment:
      - "TZ=Asia/Shanghai"
      - "I18N_LOCALE=zh-CN"
      - "ELASTICSEARCH_HOSTS=http://es:9200"
    ports:
      - 5601:5601
    volumes:
      - /home/kibana/config/kibana.yml:/usr/share/kibana/config/kibana.yml
    networks:
      - elastic_net

新建文件/home/kibana/config/kibana.yml

bash 复制代码

server.name: kibana
# kibana的主机地址 0.0.0.0可表示监听所有IP
server.host: "0.0.0.0"
# # kibana访问es的URL、es容器名称
elasticsearch.hosts: [ "http://es:9200" ]
# #这里是在elasticsearch设置密码时的值，没有密码可不配置
elasticsearch.username: "elastic"
elasticsearch.password: "\"123456\""
# # # 显示登陆页面
xpack.monitoring.ui.container.elasticsearch.enabled: true
# # 语言配置中文
i18n.locale: "zh-CN"

执行新建容器命令./docker-compose -f conf/docker-compose.yml up --build -d kibana

访问5601成功

3、logstash

docker-compose.yml新增logstash相关配置

bash 复制代码

logstash:
    container_name: logstash
    image: logstash:7.10.1
    ports:
      - 5044:5044
    volumes:
      - /home/logstash/pipeline:/usr/share/logstash/pipeline
      - /home/logstash/config:/usr/share/logstash/config
      - /home/nginx/log:/home/nginx/log
    networks:
      - elastic_net

新建/home/logstash/config和/home/logstash/pipeline文件夹

/home/logstash/config文件新建配置文件pipelines.yml、logstash.yml

bash 复制代码

#pipelines.yml
- pipeline.id: nginx
  path.config: "/usr/share/logstash/pipeline/nginx_log.conf"

#logstash.yml内容 es为容器名
http.host: "0.0.0.0"
xpack.monitoring.elasticsearch.hosts: [ "http://es:9200" ]

/home/logstash/pipeline文件新建 nginx_log.conf 文件

bash 复制代码

#配置
input {
  file {
    type => "messages_log"
    path => "/home/nginx/log/error.log"
  }
  file {
    type => "secure_log"
    path => "/home/nginx/log/access.log"
  }
}
output {
  if [type] == "messages_log" {
    elasticsearch {
      hosts => ["http://es:9200"]
      index => "messages_log_%{+YYYY-MM-dd}"
      user => "elastic"  # 如果需要认证，确保用户名正确  
      password => "123456"  # 如果需要认证，确保密码正确  
    }
  }
  if [type] == "secure_log" {
    elasticsearch {
      hosts => ["http://es:9200"]
      index => "secure_log_%{+YYYY-MM-dd}"
      user => "elastic"  # 如果需要认证，确保用户名正确  
      password => "123456"  # 如果需要认证，确保密码正确  
    }
  }
}

执行新建容器命令./docker-compose -f conf/docker-compose.yml up --build -d logstash

可在kibana上查看es有相关索引

部署起来不是很难、但是没有实际应用logstash的功能、重点在/logstash/pipeline下的管道配置文件*.conf 输入输出过滤等操作、要写好相关的处理逻辑才能将错乱的日志处理成标准日志、至于conf文件的编写感觉很难需要学习相关的语法。

参考官网https://www.elastic.co/guide/en/logstash/current/config-examples.html