云原生链路观测平台 openobserve + fluent-bit,日志收集

grpc-opentracing
https://github.com/grpc-ecosystem/grpc-opentracing

openobserve + fluent-bit

为啥会选择这个组合 一个 rust 写的一个是c写的,性能和内存方面不用担心,比java 那套好太多了

openobserve 文档 :https://openobserve.ai/docs/user-guide/users/

fluent-bit 文档: https://docs.fluentbit.io/manual/installation/linux/ubuntu

fluent-bit linux 服务器red hat 8 安装,安装地址
https://docs.fluentbit.io/manual/installation/linux/redhat-centos

复制代码
[ec2-user@master yum.repos.d]$ cat fluent-bit.repo
[fluent-bit]
name=Fluent Bit
baseurl=https://packages.fluentbit.io/centos/8/$basearch/
gpgcheck=1
gpgkey=https://packages.fluentbit.io/fluentbit.key
repo_gpgcheck=1
enabled=1


[ec2-user@master yum.repos.d]$ pwd
/etc/yum.repos.d

sudo find / -name fluent-bit
/opt/fluent-bit/bin/fluent-bit -v

openobserve 和 fluent-bit 都是二进制文件启动很简单,fluent-bit 负责传输,目前给出fluent-bit 配置

配置文件

复制代码
# fluent-bit -c config_pro.conf

@SET MY_DIR=/home/ec2-user/data/docker/services/api/logs
@SET MY_APP=news_api
@SET MY_DIR1=/home/ec2-user/data/docker/services/yst/logs
@SET MY_APP1=news_yst
@SET MY_DIR2=/home/ec2-user/data/docker/services/task/logs
@SET MY_APP2=news_task

[SERVICE]
    Flush        1
    Daemon       Off
    Log_Level    info
    Parsers_File parsers.conf
[INPUT]
    Name        tail
    Path        /Users/admin/go/src/goBoss/my_colly/my_github/zlog/logs/*.log
    Tag         logs
    Parser      json_parser
    DB          ./flb_logs.db
    Mem_Buf_Limit 5MB
    Skip_Long_Lines On
    Rotate_Wait 10
    Path_Key    filename
    Exclude_Path /Users/admin/go/src/goBoss/my_colly/my_github/zlog/logs/*[0-9][0-9][0-9][0-9]-[0-9][0-9]-[0-9][0-9]*.log
# [FILTER]
#     Name        lua
#     Match       logs
#     Script      process_time.lua
#     Call        extract_time

[OUTPUT]
    Name        http
    Match       *
    Host        127.0.0.1
    Port        5080
    URI         /api/default/test4/_json
    tls         Off
    Format      json
#     Json_date_key    _timestamp
#     Json_date_format iso8601
    HTTP_User   [email protected]
    HTTP_Passwd 2rlpVuHNxjDkceZd
    compress    gzip

# [OUTPUT]
#     Name        stdout
#     Match       *
#     Format      json_lines

还可以直接脚本过滤数据,或者添加字段

process_time.lua

lua 复制代码
function extract_time(tag, timestamp, record)
   return 1, timestamp, record
--     if record.time then
--         -- Parse the time string into a Unix timestamp
--         local year, month, day, hour, min, sec = record.time:match("(%d+)-(%d+)-(%d+) (%d+):(%d+):(%d+)")
--         if year then
--             local time = os.time({year=year, month=month, day=day, hour=hour, min=min, sec=sec})
--             record.timestamp = os.date("!%Y-%m-%dT%H:%M:%SZ", time)
--         end
--     end
--     return 1, timestamp, record
end

启动命令:
fluent-bit -c config.conf

systemd 启动

复制代码
[Unit]
Description=Fluent Bit
After=network.target

[Service]
Type=simple
WorkingDirectory=/data/fluent_bit
#Environment="MY_DIR=/data/widgets/logs"
#Environment="MY_APP=widgets_app"
ExecStart=/opt/fluent-bit/bin/fluent-bit -c /data/fluent_bit/config_pro.conf
Restart=on-failure
ExecReload=/bin/kill -HUP $MAINPID

[Install]
WantedBy=multi-user.target

openobserve 启动更简单,直接设置一个启动或者登录直接可以启动了

例如我docker 启动

yaml 复制代码
version: '3.8'

services:
  otel-collector:
    image: otel/opentelemetry-collector-contrib:0.109.0
    command: ["--config=/etc/otel-collector-config.yaml"]
    volumes:
      - ./otel-collector-config.yaml:/etc/otel-collector-config.yaml
    ports:
      - "4317:4317"   # OTLP gRPC receiver
      - "4318:4318"   # OTLP HTTP receiver

    depends_on:
      - openobserve

  openobserve:
    image: openobserve/openobserve:latest
    container_name: openobserve
    ports:
      - "5080:5080"  # OpenObserve Web UI
      - "9200:9200"  # OpenObserve Ingest API
      - "5081:5081"  # OpenObserve gRPC API
    environment:
      ZO_ROOT_USER_EMAIL: "[email protected]"  # 设置 root 用户的电子邮件
      ZO_ROOT_USER_PASSWORD: "123456"    # 设置 root 用户的密码

parsers.conf 其他解析文件

https://github.com/fluent/fluent-bit/blob/master/conf/parsers.conf

具体其他文档

https://docs.fluentbit.io/manual/pipeline/inputs/collectd

https://blog.csdn.net/easylife206/article/details/141616347

https://github.com/open-telemetry/opentelemetry-collector

jaeger

https://github.com/jaegertracing/jaeger

https://github.com/jaegertracing/jaeger/tree/main/examples/hotrod

OpenTelemetry Collector 中文文档
https://opentelemetry.opendocs.io/docs/collector/deployment/agent/

openobserve 文档
https://openobserve.ai/docs/quickstart/

opentelemetry-go 案例文档
https://github.com/open-telemetry/opentelemetry-go/blob/main/example/otel-collector/otel-collector.yaml

官方文档
https://opentelemetry.io/zh/docs/collector/quick-start/

grpc 微服务文档
https://www.yuque.com/bobby-zpcyu/lp7nfv/ad544k

相关推荐
Leon-zy1 小时前
【云计算】云计算中IaaS、PaaS、SaaS介绍
云原生·云计算·paas
iangyu3 小时前
centos7部署k8s集群
云原生·容器·kubernetes
时迁2474 小时前
【k8s】k8s是怎么实现自动扩缩的
云原生·容器·kubernetes·k8s
编程一生5 小时前
微服务相比传统服务的优势
微服务·云原生·架构
matrixlzp9 小时前
K8S Service 原理、案例
云原生·容器·kubernetes
孔令飞11 小时前
Go:终于有了处理未定义字段的实用方案
人工智能·云原生·go
玄明Hanko11 小时前
Quarkus+Docker最全面完整教程:手把手搞定Java云原生
后端·docker·云原生
hoho不爱喝酒1 天前
微服务Nacos组件的介绍、安装、使用
微服务·云原生·架构
阿里云云原生1 天前
API 即 MCP|Higress 发布 MCP Marketplace,加速存量 API 跨入 MCP 时代
云原生
爱吃龙利鱼1 天前
rocky9.4部署k8s群集v1.28.2版本(containerd)(纯命令)
云原生·容器·kubernetes