云原生链路观测平台 openobserve + fluent-bit,日志收集

grpc-opentracing
https://github.com/grpc-ecosystem/grpc-opentracing

openobserve + fluent-bit

为啥会选择这个组合一个 rust 写的一个是c写的，性能和内存方面不用担心，比java 那套好太多了

openobserve 文档：https://openobserve.ai/docs/user-guide/users/

fluent-bit 文档: https://docs.fluentbit.io/manual/installation/linux/ubuntu

fluent-bit linux 服务器red hat 8 安装,安装地址
https://docs.fluentbit.io/manual/installation/linux/redhat-centos

复制代码

[ec2-user@master yum.repos.d]$ cat fluent-bit.repo
[fluent-bit]
name=Fluent Bit
baseurl=https://packages.fluentbit.io/centos/8/$basearch/
gpgcheck=1
gpgkey=https://packages.fluentbit.io/fluentbit.key
repo_gpgcheck=1
enabled=1


[ec2-user@master yum.repos.d]$ pwd
/etc/yum.repos.d

sudo find / -name fluent-bit
/opt/fluent-bit/bin/fluent-bit -v

openobserve 和 fluent-bit 都是二进制文件启动很简单，fluent-bit 负责传输，目前给出fluent-bit 配置

配置文件

复制代码

# fluent-bit -c config_pro.conf

@SET MY_DIR=/home/ec2-user/data/docker/services/api/logs
@SET MY_APP=news_api
@SET MY_DIR1=/home/ec2-user/data/docker/services/yst/logs
@SET MY_APP1=news_yst
@SET MY_DIR2=/home/ec2-user/data/docker/services/task/logs
@SET MY_APP2=news_task

[SERVICE]
    Flush        1
    Daemon       Off
    Log_Level    info
    Parsers_File parsers.conf
[INPUT]
    Name        tail
    Path        /Users/admin/go/src/goBoss/my_colly/my_github/zlog/logs/*.log
    Tag         logs
    Parser      json_parser
    DB          ./flb_logs.db
    Mem_Buf_Limit 5MB
    Skip_Long_Lines On
    Rotate_Wait 10
    Path_Key    filename
    Exclude_Path /Users/admin/go/src/goBoss/my_colly/my_github/zlog/logs/*[0-9][0-9][0-9][0-9]-[0-9][0-9]-[0-9][0-9]*.log
# [FILTER]
#     Name        lua
#     Match       logs
#     Script      process_time.lua
#     Call        extract_time

[OUTPUT]
    Name        http
    Match       *
    Host        127.0.0.1
    Port        5080
    URI         /api/default/test4/_json
    tls         Off
    Format      json
#     Json_date_key    _timestamp
#     Json_date_format iso8601
    HTTP_User   admin@example.com
    HTTP_Passwd 2rlpVuHNxjDkceZd
    compress    gzip

# [OUTPUT]
#     Name        stdout
#     Match       *
#     Format      json_lines

还可以直接脚本过滤数据，或者添加字段

process_time.lua

lua 复制代码

function extract_time(tag, timestamp, record)
   return 1, timestamp, record
--     if record.time then
--         -- Parse the time string into a Unix timestamp
--         local year, month, day, hour, min, sec = record.time:match("(%d+)-(%d+)-(%d+) (%d+):(%d+):(%d+)")
--         if year then
--             local time = os.time({year=year, month=month, day=day, hour=hour, min=min, sec=sec})
--             record.timestamp = os.date("!%Y-%m-%dT%H:%M:%SZ", time)
--         end
--     end
--     return 1, timestamp, record
end

启动命令：
fluent-bit -c config.conf

systemd 启动

复制代码

[Unit]
Description=Fluent Bit
After=network.target

[Service]
Type=simple
WorkingDirectory=/data/fluent_bit
#Environment="MY_DIR=/data/widgets/logs"
#Environment="MY_APP=widgets_app"
ExecStart=/opt/fluent-bit/bin/fluent-bit -c /data/fluent_bit/config_pro.conf
Restart=on-failure
ExecReload=/bin/kill -HUP $MAINPID

[Install]
WantedBy=multi-user.target

openobserve 启动更简单，直接设置一个启动或者登录直接可以启动了

例如我docker 启动

yaml 复制代码

version: '3.8'

services:
  otel-collector:
    image: otel/opentelemetry-collector-contrib:0.109.0
    command: ["--config=/etc/otel-collector-config.yaml"]
    volumes:
      - ./otel-collector-config.yaml:/etc/otel-collector-config.yaml
    ports:
      - "4317:4317"   # OTLP gRPC receiver
      - "4318:4318"   # OTLP HTTP receiver

    depends_on:
      - openobserve

  openobserve:
    image: openobserve/openobserve:latest
    container_name: openobserve
    ports:
      - "5080:5080"  # OpenObserve Web UI
      - "9200:9200"  # OpenObserve Ingest API
      - "5081:5081"  # OpenObserve gRPC API
    environment:
      ZO_ROOT_USER_EMAIL: "admin@example.com"  # 设置 root 用户的电子邮件
      ZO_ROOT_USER_PASSWORD: "123456"    # 设置 root 用户的密码

parsers.conf 其他解析文件

https://github.com/fluent/fluent-bit/blob/master/conf/parsers.conf

具体其他文档

https://docs.fluentbit.io/manual/pipeline/inputs/collectd

https://blog.csdn.net/easylife206/article/details/141616347

https://github.com/open-telemetry/opentelemetry-collector

jaeger

https://github.com/jaegertracing/jaeger

https://github.com/jaegertracing/jaeger/tree/main/examples/hotrod

OpenTelemetry Collector 中文文档
https://opentelemetry.opendocs.io/docs/collector/deployment/agent/

openobserve 文档
https://openobserve.ai/docs/quickstart/

opentelemetry-go 案例文档
https://github.com/open-telemetry/opentelemetry-go/blob/main/example/otel-collector/otel-collector.yaml

官方文档
https://opentelemetry.io/zh/docs/collector/quick-start/

grpc 微服务文档
https://www.yuque.com/bobby-zpcyu/lp7nfv/ad544k