opentelemetry全链路初探--日志跳转trace

前言

上一小节描述了metrics、traces，本小节来把log也加进去，并且做一个traces与log的联动

• 当查看日志的时候，可以同时跳转到对应的jaeger，查看分段trace情况

应用服务

• 本次要测试的应用服务架构为 a.py-->b.py
• 业务服务会往对应的目录打印日志，并且日志包含了trace_id

a.py

import tornado.httpserver as httpserver
import tornado.web
from tornado.ioloop import IOLoop
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.resources import SERVICE_NAME, Resource
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.trace import get_tracer
from opentelemetry.propagate import inject
import requests
import logging


trace.set_tracer_provider(
    TracerProvider(resource=Resource.create({SERVICE_NAME: "line-a"}))
)
span_processor = BatchSpanProcessor(OTLPSpanExporter(endpoint="http://127.0.0.1:4318/v1/traces"))
trace.get_tracer_provider().add_span_processor(span_processor)

logger = logging.getLogger('a.py')
logger.setLevel(logging.INFO)
formatter = logging.Formatter( '%(asctime)s - %(name)s - %(levelname)s - %(message)s', datefmt='%Y-%m-%d %H:%M:%S')
file_handler = logging.FileHandler('logs/access-a.log', mode='a', encoding='utf-8')
file_handler.setFormatter(formatter)
logger.addHandler(file_handler)

def traced(name):
    def decorator(func):
        def wrapper(*args, **kwargs):
            tracer = get_tracer(__name__)
            with tracer.start_as_current_span(name):
                trace_id = format(trace.get_current_span().get_span_context().trace_id, '032x')
                logger.info('{} {}'.format(name, trace_id))
                return func(*args, **kwargs)
        return wrapper
    return decorator

class TestFlow(tornado.web.RequestHandler):
    def get(self):
        views()
        self.finish('hello world')

@traced("phase-1")
def views():
    views_2()
    headers = {}
    inject(headers)
    requests.get("http://127.0.0.1:20000", headers=headers)

@traced("phase-2")
def views_2():
    pass

def applications():
    urls = []
    urls.append([r'/', TestFlow])
    return tornado.web.Application(urls)

def main():
    app = applications()
    server = httpserver.HTTPServer(app)
    server.bind(10000, '0.0.0.0')
    server.start(1)
    IOLoop.current().start()


if __name__ == "__main__":
    try:
        main()
    except KeyboardInterrupt as e:
        IOLoop.current().stop()
    finally:
        IOLoop.current().close()

同理配置b.py

...
trace.set_tracer_provider(
    TracerProvider(resource=Resource.create({SERVICE_NAME: "line-b"}))
)

logger = logging.getLogger('b.py')
logger.setLevel(logging.INFO)
formatter = logging.Formatter( '%(asctime)s - %(name)s - %(levelname)s - %(message)s', datefmt='%Y-%m-%d %H:%M:%S')
file_handler = logging.FileHandler('logs/access-b.log', mode='a', encoding='utf-8')

...
@traced("phase-3")
def views(headers):
    pass

...
def main():
    app = applications()
    server = httpserver.HTTPServer(app)
    server.bind(20000, '0.0.0.0')
    server.start(1)
    IOLoop.current().start()
...

访问curl 127.0.0.1:10000之后，就会往日志文件中写入日志

▶ cat access-a.log
2025-08-21 11:22:18 - a.py - INFO - phase-1 38f0e8cef3e331993eca1675fa8c956b
2025-08-21 11:22:18 - a.py - INFO - phase-2 38f0e8cef3e331993eca1675fa8c956b

▶ cat access-b.log
2025-08-21 11:22:18 - b.py - INFO - phase-3 38f0e8cef3e331993eca1675fa8c956b

日志系统loki

1）安装loki

loki.yaml

auth_enabled: false

server:
  http_listen_port: 3100

common:
  instance_addr: 127.0.0.1
  path_prefix: /loki
  storage:
    filesystem:
      chunks_directory: /loki/chunks
      rules_directory: /loki/rules
  replication_factor: 1
  ring:
    kvstore:
      store: inmemory

schema_config:
  configs:
    - from: 2020-10-24
      store: tsdb
      object_store: filesystem
      schema: v13
      index:
        prefix: index_
        period: 24h

ruler:
  alertmanager_url: http://localhost:9093

docker run -d --name grafana-loki \
  -p 3100:3100 \
  -v ./loki.yaml:/etc/loki/local-config.yaml \
  grafana/loki:v3.5.3

2）安装promtail

promtail.yaml

server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://10.22.12.178:3100/loki/api/v1/push

scrape_configs:
- job_name: system
  static_configs:
  - labels:
      job: access_log
      __path__: /var/log/access*.log

docker run -d --name grafana-promtail \
  -v ./promtail.yaml:/etc/promtail/config.yml \
  -v ./logs:/var/log:ro \
  grafana/promtail:3.5

3）安装grafana

docker run -d --name grafana-grafana \
  -p 3000:3000 \
  grafana/grafana:12.1.1

4）在grafana上配置loki

登录grafana --> connections --> data sources --> 选择loki --> 开始配置

最关键的步骤就是把connection改成loki的地址10.22.12.178:3100

5）查看下刚才写入的日志

日志已经进来了

logs跳转至traces

之前已经将traces采集进jaeger，现在又拥有了日志，将他们通过traceid联动起来，在grafana找到日志，并且通过traceid字段跳转去jaeger查询详细信息

登录grafana --> connections --> data sources --> 选择刚才配置好的loki --> Derived fields

• Name：随便取一个
• Type：Regex in log line
• Regex：([0-9a-f]{32,})
• URL：配置jaeger的URL，http://10.22.12.178:16686/trace/${__value.raw}

配置完成之后查看日志，有会个跳转按钮，点击之后就来带jaeger-UI的查询页面

联系我

• 联系我，做深入的交流

---

至此，本文结束

在下才疏学浅，有撒汤漏水的，请各位不吝赐教...