graylog使用Sidecars方式收集springboot程序的日志

青春不流名2024-04-25 14:11

1、部署graylog后台服务

使用docker-compose启动三个服务程序，包括graylog、mongodb、opensearch。

docker-compose.yml内容如下

version: '3'

services:

MongoDB: https://hub.docker.com/_/mongo/

mongodb:

image: mongo:6.0.14

privileged: true

networks:

  • graylog

opensearch:

image: "opensearchproject/opensearch:2.12.0"

environment:

  • "OPENSEARCH_JAVA_OPTS=-Xms1g -Xmx1g"

  • "bootstrap.memory_lock=true"

  • "discovery.type=single-node"

  • "action.auto_create_index=false"

  • "plugins.security.ssl.http.enabled=false"

  • "plugins.security.disabled=true"

Can generate a password for `OPENSEARCH_INITIAL_ADMIN_PASSWORD` using a linux device via:

tr -dc A-Z-a-z-0-9_@#%^-_=+ < /dev/urandom | head -c${1:-32}

  • OPENSEARCH_INITIAL_ADMIN_PASSWORD=+_8r#wliY3Pv5-HMIf4qzXImYzZf-M=M

privileged: true

ulimits:

memlock:

hard: -1

soft: -1

nofile:

soft: 65536

hard: 65536

restart: "on-failure"

networks:

  • graylog

Graylog: https://hub.docker.com/r/graylog/graylog/

graylog:

image: graylog/graylog:5.2

environment:

  • GRAYLOG_NODE_ID_FILE=/usr/share/graylog/data/config/node-id

  • GRAYLOG_HTTP_BIND_ADDRESS=0.0.0.0:9000

  • GRAYLOG_ELASTICSEARCH_HOSTS=http://opensearch:9200

  • GRAYLOG_MONGODB_URI=mongodb://mongodb:27017/graylog

CHANGE ME (must be at least 16 characters)!

  • GRAYLOG_PASSWORD_SECRET=somepasswordpepper

Password: admin

  • GRAYLOG_ROOT_PASSWORD_SHA2=8c6976e5b5410415bde908bd4dee15dfb167a9c873fc4bb8a81f6f2ab448a918

  • GRAYLOG_HTTP_EXTERNAL_URI=http://127.0.0.1:9000/

privileged: true

entrypoint: /usr/bin/tini -- wait-for-it elasticsearch:9200 -- /docker-entrypoint.sh

networks:

  • graylog

restart: always

depends_on:

  • mongodb

  • opensearch

ports:

Graylog web interface and REST API

  • 9000:9000

Syslog TCP

  • 1514:1514

Syslog UDP

  • 1514:1514/udp

GELF TCP

  • 12201:12201

GELF UDP

  • 12201:12201/udp

  • 5044:5044

networks:

graylog:

driver: bridge

2、部署k8s上graylog的后台服务

graylog2-service.yaml

启动运行kubectl apply -f graylog2-service.yaml

apiVersion: v1

kind: Service

metadata:

labels:

app: graylog2

name: graylog2

namespace: example

spec:

type: ClusterIP

ports:

  • name: rest

port: 9000

protocol: TCP

  • name: tsyslog

port: 1514

protocol: TCP

  • name: usyslog

port: 1514

protocol: UDP

  • name: tgelf

port: 12201

protocol: TCP

  • name: ugelf

port: 12201

protocol: UDP

  • name: tbeat

port: 5044

protocol: TCP

  • name: ubeat

port: 5044

protocol: UDP

apiVersion: v1

kind: Endpoints

metadata:

labels:

app: graylog2

name: graylog2

namespace: example

subsets:

  • addresses:

  • ip: 上面执行docker-compose up -d的主机ip

ports:

  • name: rest

port: 9000

protocol: TCP

  • name: tsyslog

port: 1514

protocol: TCP

  • name: usyslog

port: 1514

protocol: UDP

  • name: tgelf

port: 12201

protocol: TCP

  • name: ugelf

port: 12201

protocol: UDP

  • name: tbeat

port: 5044

protocol: TCP

  • name: ubeat

port: 5044

protocol: UDP

3、登录graylog后台配置Sidecars信息

登录地址

http://ip:9000

登录用户名/密码

admin/admin

顶部菜单选择【System】中的之后一个Sidecars，点击【Create or reuse a token for the graylog-sidecar user】

其中Token Name自定义填写，点击Create Token按钮，会生成token信息。

生成的Token需要复制保留，下面file-service.yaml文件需要使用到

查看节点id信息，在System元素处，会看见节点id信息：Node ID

节点ID需要复制保留，下面file-service.yaml文件需要使用到

4、部署springboot程序在k8s上

file-service.yaml，启动运行kubectl apply -f file-service.yaml

apiVersion: v1

kind: Service

metadata:

name: file-service

namespace: example

labels:

service: file-service

spec:

ports:

  • name: http

port: 8080

protocol: TCP

targetPort: 8080

nodePort: 30250

selector:

app: file-service

type: NodePort

apiVersion: v1

kind: Service

metadata:

name: file-service-headless

namespace: example

labels:

app: file-service

spec:

ports:

  • name: http

port: 8080

targetPort: 8080

clusterIP: None

selector:

app: file-service

apiVersion: apps/v1

kind: StatefulSet

metadata:

name: file-service

namespace: example

spec:

replicas: 1

serviceName: file-service-headless

selector:

matchLabels:

app: file-service

template:

metadata:

labels:

app: file-service

release: default

spec:

restartPolicy: Always

containers:

  • name: file-service

resources:

requests:

ephemeral-storage: 2048Mi

limits:

ephemeral-storage: 2048Mi

imagePullPolicy: IfNotPresent

image: file-service:v1

ports:

  • name: http

containerPort: 8080

volumeMounts:

  • name: volume-localtime

mountPath: /etc/localtime

  • name: graylog2-logs

#springboot程序容器内产生日志的目录

mountPath: /opt/file-service/logs

  • name: sidecar-collector-logs

imagePullPolicy: IfNotPresent

image: graylog-log-sidecar-collector:latest

env:

  • name: GS_SERVER_URL

#定义graylog后台服务的地址

value: "http://graylog2:9000/api/"

#定义graylog服务的节点id，取值来自上一步复制的内容

  • name: GS_NODE_ID

value: "5861f0cb-e128-4f2e-a17b-3e42f8bff6af"

#节点名称自定义

  • name: GS_NODE_NAME

value: "sidecar-collector-logs-file-service"

#使用的token，取值来自上一步复制的内容

  • name: GS_SERVER_API_TOKEN

value: "17b7haug3bvflmtuj23e34eg9raen6bsmcppdo1aluls7s05juvn"

#采集器容器的目录,通过挂载的方式GS_LIST_LOG_FILES与运行springboot程序的graylog2-logs目录进行关联

  • name: GS_LIST_LOG_FILES

value: "/graylog2-logs"

volumeMounts:

  • name: graylog2-logs

mountPath: /graylog2-logs

volumes:

  • name: volume-localtime

hostPath:

path: /etc/localtime

type: ''

#同时映射了宿主机目录，如果产生的日志不够，可以在这个文件夹内手动添加*.log日志。

  • name: graylog2-logs

hostPath:

path: /home/volume

type: DirectoryOrCreate

5、配置Sidecars

springboot正常启动后，会自动注册到graylog上。

下面是两个容器，一个复制运行程序，另一个负责日志收集上报。

sidecars的运行状态已经是Running，但此时还需要配置file beat信息

点击名称链接，进入新页面，可以看到加载的日志目录信息

指派filebeat配置

编辑Configuration下的Log Collectors，选项下图的内容进行编辑，Executable Path要改成实际的目录，

graylog-log-sidecar-collector:latest 镜像我将filebeat放到了

/usr/share/filebeat/bin/filebeat，需要根据实际情况进行修改。点击Update Collectors进行保存。

Configurations中选择下面内容进行编辑

模板配置内容

Needed for Graylog

fields_under_root: true

fields.collector_node_id: ${sidecar.nodeName}

fields.gl2_source_collector: ${sidecar.nodeId}

output.logstash:

hosts: ["${user.graylog_host}:5044"]

#hosts: ["${user.graylog_host}:12201"]

path:

data: ${sidecar.spoolDir!"/var/lib/graylog-sidecar/collectors/filebeat"}/data

logs: ${sidecar.spoolDir!"/var/lib/graylog-sidecar/collectors/filebeat"}/log

filebeat.inputs:

  • type: log

id: sidecar-collector-logs-file-service

enabled: true

paths:

  • /graylog2-logs/*.log

fields_under_root: true

fields:

event_source_product: springboot

修改右侧的环境变量信息 ，k8s中使用域名访问服务。

重启Sidecar插件服务

观察容器内日志

6、配置Input

填写名称即可，选择Global全局的。

7、查看收集日志效果

8、Sidecar Dockerfile

FROM debian:buster-slim

LABEL maintainer 'Markus Gulden <mg@gulden.consulting>'

RUN apt-get update && apt-get install -y openssl libapr1 libdbi1 libexpat1 ca-certificates

ENV SIDECAR_BINARY_URL https://github.com/Graylog2/collector-sidecar/releases/download/1.5.0/graylog-sidecar_1.5.0-1_amd64.deb

ENV FILEBEAT_BINARY_URL https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-7.1.1-amd64.deb

RUN apt-get install -y --no-install-recommends curl && curl -Lo sidecar.deb {SIDECAR_BINARY_URL} \&\& dpkg -i sidecar.deb \&\& rm sidecar.deb \&\& curl -Lo filebeat.deb {FILEBEAT_BINARY_URL} && dpkg -i filebeat.deb && rm filebeat.deb && apt-get purge -y --auto-remove curl

#GS_LIST_LOG_FILES="[]"

ENV GS_UPDATE_INTERVAL=10 \

GS_TLS_SKIP_VERIFY="false" \

GS_SEND_STATUS="true" \

GS_CACHE_PATH="/var/cache/graylog-sidecar" \

GS_COLLECTOR_CONFIGURATION_DIRECTORY="/var/lib/graylog-sidecar/generated" \

GS_LOG_PATH="/var/log/graylog-sidecar" \

GS_LOG_ROTATE_MAX_FILE_SIZE="1MiB" \

GS_LOG_ROTATE_KEEP_FILES=100 \

GS_COLLECTOR_BINARIES_WHITELIST="["/usr/bin/filebeat", "/usr/bin/packetbeat", "/usr/bin/metricbeat", "/usr/bin/heartbeat", "/usr/bin/auditbeat", "/usr/bin/journalbeat", "/usr/share/filebeat/bin/filebeat", "/usr/share/packetbeat/bin/packetbeat", "/usr/share/metricbeat/bin/metricbeat", "/usr/share/heartbeat/bin/heartbeat", "/usr/share/auditbeat/bin/auditbeat", "/usr/share/journalbeat/bin/journalbeat", "/usr/bin/nxlog", "/opt/nxlog/bin/nxlog"]"

ADD ./data /data

CMD /usr/bin/graylog-sidecar -c /data/sidecar.yml

相关推荐
木鱼时刻21 小时前
容器与 Kubernetes 基本概念与架构
容器·架构·kubernetes
chuanauc1 天前
Kubernets K8s 学习
java·学习·kubernetes
庸子2 天前
基于Jenkins和Kubernetes构建DevOps自动化运维管理平台
运维·kubernetes·jenkins
李白你好2 天前
高级运维！Kubernetes（K8S）常用命令的整理集合
运维·容器·kubernetes
Connie14512 天前
k8s多集群管理中的联邦和舰队如何理解？
云原生·容器·kubernetes
伤不起bb2 天前
Kubernetes 服务发布基础
云原生·容器·kubernetes
别骂我h2 天前
Kubernetes服务发布基础
云原生·容器·kubernetes
weixin_399380692 天前
k8s一键部署tongweb企业版7049m6（by why+lqw）
java·linux·运维·服务器·云原生·容器·kubernetes
斯普信专业组3 天前
K8s环境下基于Nginx WebDAV与TLS/SSL的文件上传下载部署指南
nginx·kubernetes·ssl
&如歌的行板&3 天前
如何在postman中动态请求k8s中的pod ip（基于nacos）
云原生·容器·kubernetes
热门推荐
01集群聊天服务器---MySQL数据库的建立02GPU 进阶笔记（二）：华为昇腾 910B GPU03Coze扣子平台完整体验和实践（附国内和国际版对比）04使用Ruby接入实时行情API教程05Word粘贴时出现“运行时错误53,文件未找到：MathPage.WLL“的解决方案06扣子（coze）实战|我用扣子搭建了一个自动分析小红薯笔记内容的AI应用|详细步骤拆解07Java学习第十五部分——MyBatis08扣子空间的使用教程与大模型技术思考09DeepSeek各版本说明与优缺点分析10基于odoo17的设计模式详解---装饰模式