一、为什么需要日志结构化
前三讲我们收集了大量数据：Trace Span、LLM 调用记录、决策记录。但这些数据如果没有统一的结构，就是一堆垃圾：
❌ 非结构化日志：
2026-09-28 10:00:01 ERROR request failed
2026-09-28 10:00:02 INFO user query: 你好
2026-09-28 10:00:03 WARN slow response
❌ 问题：
1. 无法检索：grep 只能搜字符串，不能按字段过滤
2. 无法聚合：不能算平均值、百分位数
3. 无法关联：Trace ID 散落在各处
4. 无法告警：没法对特定字段设阈值
5. 无法分析：没法做趋势分析、异常检测解决方案：统一 Event Schema + 分层设计
✅ 结构化：每条日志是 JSON，字段明确
✅ 分层：基础层 + 业务层 + 扩展层
✅ 关联：Trace ID / Span ID 贯穿始终
✅ 高效：Protocol Buffers 序列化 + 批量写入
✅ 可查：按任意字段索引和过滤二、架构设计
┌─────────────────────────────────────────────────────────────┐
│ 日志采集端 (Agent) │
│ │
│ ┌──────────┐ ┌──────────┐ ┌──────────┐ │
│ │ Trace │ │ LLM │ │ Decision │ │
│ │ Collector│ │ Monitor │ │ Monitor │ │
│ └────┬─────┘ └────┬─────┘ └────┬─────┘ │
│ │ │ │ │
│ └──────────────┼─────────────┘ │
│ ▼ │
│ ┌──────────────────────────────────────────┐ │
│ │ Event Builder (事件构建器) │ │
│ │ ├─ 统一 Schema │ │
│ │ ├─ 字段校验 │ │
│ │ └─ 序列化 (JSON/PB) │ │
│ └──────────────────┬───────────────────────┘ │
│ │ │
│ ┌──────────────────▼───────────────────────┐ │
│ │ Async Buffer (异步缓冲区) │ │
│ │ ├─ Ring Buffer (无锁) │ │
│ │ ├─ 批量打包 (Batch) │ │
│ │ └─ 压缩 (Snappy/Gzip) │ │
│ └──────────────────┬───────────────────────┘ │
└─────────────────────┼────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────┐
│ 日志存储层 │
│ │
│ ┌──────────────────┐ ┌──────────────────┐ │
│ │ Hot Storage │ │ Cold Storage │ │
│ │ ClickHouse (7天) │ │ S3/MinIO (90天) │ │
│ │ 实时查询 │ │ 归档查询 │ │
│ └──────────────────┘ └──────────────────┘ │
└─────────────────────────────────────────────────────────────┘三、完整代码实现
package main
import (
"bytes"
"compress/gzip"
"encoding/binary"
"encoding/json"
"fmt"
"hash/crc32"
"io"
"os"
"runtime"
"sort"
"strings"
"sync"
"sync/atomic"
"time"
)
// ============================================================
// 1. 统一 Event Schema
// ============================================================
// EventType 事件类型
type EventType string
const (
EventTraceSpan EventType = "trace.span"
EventLLMCall EventType = "llm.call"
EventDecision EventType = "decision"
EventSystemMetric EventType = "system.metric"
EventAppLog EventType = "app.log"
)
// Severity 严重级别
type Severity string
const (
SeverityDebug Severity = "DEBUG"
SeverityInfo Severity = "INFO"
SeverityWarning Severity = "WARNING"
SeverityError Severity = "ERROR"
SeverityFatal Severity = "FATAL"
)
// Event 统一事件结构
type Event struct {
// === 基础层 (必填) ===
EventID string `json:"event_id"`
EventType EventType `json:"event_type"`
Timestamp time.Time `json:"timestamp"`
Hostname string `json:"hostname"`
ServiceName string `json:"service_name"`
Environment string `json:"environment"` // dev/staging/production
Severity Severity `json:"severity"`
TraceID string `json:"trace_id,omitempty"`
SpanID string `json:"span_id,omitempty"`
// === 业务层 (按事件类型选填) ===
Payload json.RawMessage `json:"payload"` // 业务数据
Tags map[string]string `json:"tags,omitempty"`
DurationMs float64 `json:"duration_ms,omitempty"`
StatusCode int `json:"status_code,omitempty"`
ErrorMessage string `json:"error_message,omitempty"`
// === 扩展层 (可选) ===
Metadata map[string]interface{} `json:"metadata,omitempty"`
SourceLine string `json:"source_line,omitempty"`
Sampled bool `json:"sampled,omitempty"`
}
// Validate 校验事件必填字段
func (e *Event) Validate() error {
if e.EventID == "" {
return fmt.Errorf("event_id is required")
}
if e.EventType == "" {
return fmt.Errorf("event_type is required")
}
if e.Timestamp.IsZero() {
return fmt.Errorf("timestamp is required")
}
if e.ServiceName == "" {
return fmt.Errorf("service_name is required")
}
return nil
}
// ============================================================
// 2. 事件构建器
// ============================================================
type EventBuilder struct {
hostname string
serviceName string
environment string
}
func NewEventBuilder(serviceName, environment string) *EventBuilder {
hostname, _ := os.Hostname()
return &EventBuilder{
hostname: hostname,
serviceName: serviceName,
environment: environment,
}
}
// BuildTraceSpan 构建 Trace Span 事件
func (b *EventBuilder) BuildTraceSpan(traceID, spanID, parentSpanID string,
operationName string, startTime time.Time, endTime time.Time,
attributes map[string]string, statusCode int, errorMsg string) *Event {
duration := endTime.Sub(startTime).Seconds() * 1000
payload, _ := json.Marshal(map[string]interface{}{
"operation_name": operationName,
"parent_span_id": parentSpanID,
"attributes": attributes,
})
return &Event{
EventID: generateEventID(),
EventType: EventTraceSpan,
Timestamp: endTime,
Hostname: b.hostname,
ServiceName: b.serviceName,
Environment: b.environment,
Severity: severityFromStatus(statusCode),
TraceID: traceID,
SpanID: spanID,
Payload: payload,
DurationMs: duration,
StatusCode: statusCode,
ErrorMessage: errorMsg,
Tags: map[string]string{"operation": operationName},
}
}
// BuildLLMCall 构建 LLM 调用事件
func (b *EventBuilder) BuildLLMCall(traceID, spanID string, model string,
promptTokens, completionTokens int, latencyMs float64,
success bool, cost float64, errorMsg string) *Event {
payload, _ := json.Marshal(map[string]interface{}{
"model": model,
"prompt_tokens": promptTokens,
"completion_tokens": completionTokens,
"total_tokens": promptTokens + completionTokens,
"cost": cost,
})
severity := SeverityInfo
statusCode := 200
if !success {
severity = SeverityError
statusCode = 500
}
return &Event{
EventID: generateEventID(),
EventType: EventLLMCall,
Timestamp: time.Now(),
Hostname: b.hostname,
ServiceName: b.serviceName,
Environment: b.environment,
Severity: severity,
TraceID: traceID,
SpanID: spanID,
Payload: payload,
DurationMs: latencyMs,
StatusCode: statusCode,
ErrorMessage: errorMsg,
Tags: map[string]string{"model": model},
}
}
// BuildDecision 构建决策事件
func (b *EventBuilder) BuildDecision(traceID, spanID string, intent string,
riskLevel int, route string, confidence float64, success bool) *Event {
payload, _ := json.Marshal(map[string]interface{}{
"intent": intent,
"risk_level": riskLevel,
"route": route,
"confidence": confidence,
})
severity := SeverityInfo
statusCode := 200
if !success {
severity = SeverityWarning
statusCode = 400
}
if riskLevel >= 4 {
severity = SeverityWarning
}
return &Event{
EventID: generateEventID(),
EventType: EventDecision,
Timestamp: time.Now(),
Hostname: b.hostname,
ServiceName: b.serviceName,
Environment: b.environment,
Severity: severity,
TraceID: traceID,
SpanID: spanID,
Payload: payload,
StatusCode: statusCode,
Tags: map[string]string{"intent": intent, "route": route},
}
}
// BuildAppLog 构建应用日志事件
func (b *EventBuilder) BuildAppLog(severity Severity, message string,
traceID string, fields map[string]interface{}) *Event {
payload, _ := json.Marshal(map[string]interface{}{
"message": message,
"fields": fields,
})
return &Event{
EventID: generateEventID(),
EventType: EventAppLog,
Timestamp: time.Now(),
Hostname: b.hostname,
ServiceName: b.serviceName,
Environment: b.environment,
Severity: severity,
TraceID: traceID,
Payload: payload,
Metadata: fields,
}
}
// ============================================================
// 3. 异步缓冲区 (无锁 Ring Buffer)
// ============================================================
type AsyncBuffer struct {
buffer []*Event
head int64 // 写指针
tail int64 // 读指针
mask int64
flushFn func([]*Event) error
flushTicker *time.Ticker
done chan struct{}
stats BufferStats
mu sync.Mutex // 仅用于 flush 保护
}
type BufferStats struct {
EventsWritten atomic.Int64
EventsDropped atomic.Int64
FlushCount atomic.Int64
FlushErrors atomic.Int64
BufferFull atomic.Int64
}
func NewAsyncBuffer(size int, flushInterval time.Duration,
flushFn func([]*Event) error) *AsyncBuffer {
// 确保 size 是 2 的幂
size = nextPowerOf2(size)
buf := &AsyncBuffer{
buffer: make([]*Event, size),
mask: int64(size - 1),
flushFn: flushFn,
flushTicker: time.NewTicker(flushInterval),
done: make(chan struct{}),
}
go buf.flushLoop()
return buf
}
// Write 写入事件 (无锁)
func (b *AsyncBuffer) Write(event *Event) error {
head := atomic.LoadInt64(&b.head)
tail := atomic.LoadInt64(&b.tail)
if head-tail >= b.mask {
// 缓冲区满了
b.stats.BufferFull.Add(1)
b.stats.EventsDropped.Add(1)
return fmt.Errorf("buffer full, event dropped")
}
idx := head & b.mask
b.buffer[idx] = event
atomic.StoreInt64(&b.head, head+1)
b.stats.EventsWritten.Add(1)
return nil
}
// flushLoop 定时刷盘
func (b *AsyncBuffer) flushLoop() {
for {
select {
case <-b.flushTicker.C:
b.flush()
case <-b.done:
b.flush() // 最后一次 flush
return
}
}
}
// flush 刷新缓冲区
func (b *AsyncBuffer) flush() {
b.mu.Lock()
defer b.mu.Unlock()
head := atomic.LoadInt64(&b.head)
tail := atomic.LoadInt64(&b.tail)
count := head - tail
if count == 0 {
return
}
events := make([]*Event, 0, count)
for i := int64(0); i < count; i++ {
idx := (tail + i) & b.mask
events = append(events, b.buffer[idx])
b.buffer[idx] = nil // 释放引用
}
if err := b.flushFn(events); err != nil {
b.stats.FlushErrors.Add(1)
fmt.Printf("[Buffer] flush error: %v\n", err)
// 重试：把事件放回去
for _, e := range events {
b.Write(e)
}
return
}
atomic.StoreInt64(&b.tail, head)
b.stats.FlushCount.Add(1)
}
// Stop 停止缓冲区
func (b *AsyncBuffer) Stop() {
b.flushTicker.Stop()
close(b.done)
}
// ============================================================
// 4. 日志写入器
// ============================================================
// LogWriter 日志写入接口
type LogWriter interface {
Write(event *Event) error
Close() error
}
// ConsoleWriter 控制台输出 (开发环境)
type ConsoleWriter struct {
encoder Encoder
}
func NewConsoleWriter(encoder Encoder) *ConsoleWriter {
return &ConsoleWriter{encoder: encoder}
}
func (w *ConsoleWriter) Write(event *Event) error {
data, err := w.encoder.Encode(event)
if err != nil {
return err
}
fmt.Println(string(data))
return nil
}
func (w *ConsoleWriter) Close() error { return nil }
// FileWriter 文件写入 (单机环境)
type FileWriter struct {
file *os.File
writer io.Writer
encoder Encoder
mu sync.Mutex
maxSize int64
curSize int64
basePath string
}
func NewFileWriter(path string, maxSizeMB int, encoder Encoder) (*FileWriter, error) {
file, err := os.OpenFile(path, os.O_CREATE|os.O_WRONLY|os.O_APPEND, 0644)
if err != nil {
return nil, err
}
stat, _ := file.Stat()
return &FileWriter{
file: file,
writer: file,
encoder: encoder,
maxSize: int64(maxSizeMB) * 1024 * 1024,
curSize: stat.Size(),
basePath: path,
}, nil
}
func (w *FileWriter) Write(event *Event) error {
w.mu.Lock()
defer w.mu.Unlock()
data, err := w.encoder.Encode(event)
if err != nil {
return err
}
line := append(data, '\n')
n, err := w.writer.Write(line)
if err != nil {
return err
}
w.curSize += int64(n)
if w.curSize >= w.maxSize {
return w.rotate()
}
return nil
}
func (w *FileWriter) rotate() error {
w.file.Close()
// 重命名旧文件
timestamp := time.Now().Format("20060102-150405")
newPath := fmt.Sprintf("%s.%s", w.basePath, timestamp)
os.Rename(w.basePath, newPath)
// 创建新文件
file, err := os.OpenFile(w.basePath, os.O_CREATE|os.O_WRONLY|os.O_TRUNC, 0644)
if err != nil {
return err
}
w.file = file
w.writer = file
w.curSize = 0
return nil
}
func (w *FileWriter) Close() error {
return w.file.Close()
}
// BatchWriter 批量写入器 (生产环境)
type BatchWriter struct {
buffer *AsyncBuffer
encoder Encoder
compress bool
target LogWriter // 实际写入目标
}
func NewBatchWriter(target LogWriter, bufferSize int, flushInterval time.Duration,
encoder Encoder, compress bool) *BatchWriter {
bw := &BatchWriter{
encoder: encoder,
compress: compress,
target: target,
}
bw.buffer = NewAsyncBuffer(bufferSize, flushInterval, bw.flush)
return bw
}
func (w *BatchWriter) Write(event *Event) error {
return w.buffer.Write(event)
}
func (w *BatchWriter) flush(events []*Event) error {
if len(events) == 0 {
return nil
}
// 批量编码
var batch bytes.Buffer
for _, e := range events {
data, err := w.encoder.Encode(e)
if err != nil {
return err
}
batch.Write(data)
batch.WriteByte('\n')
}
data := batch.Bytes()
// 压缩
if w.compress {
var compressed bytes.Buffer
gz := gzip.NewWriter(&compressed)
if _, err := gz.Write(data); err != nil {
return err
}
gz.Close()
data = compressed.Bytes()
}
// 写入目标
_, err := fmt.Fprintf(w.target.(io.Writer), "%s", data)
return err
}
func (w *BatchWriter) Close() error {
w.buffer.Stop()
return w.target.Close()
}
// ============================================================
// 5. 编码器
// ============================================================
type Encoder interface {
Encode(event *Event) ([]byte, error)
Decode(data []byte) (*Event, error)
}
// JSONEncoder JSON 编码器
type JSONEncoder struct {
prettyPrint bool
}
func NewJSONEncoder(pretty bool) *JSONEncoder {
return &JSONEncoder{prettyPrint: pretty}
}
func (e *JSONEncoder) Encode(event *Event) ([]byte, error) {
if e.prettyPrint {
return json.MarshalIndent(event, "", " ")
}
return json.Marshal(event)
}
func (e *JSONEncoder) Decode(data []byte) (*Event, error) {
var event Event
if err := json.Unmarshal(data, &event); err != nil {
return nil, err
}
return &event, nil
}
// ============================================================
// 6. 日志查询引擎 (内存版)
// ============================================================
type LogQueryEngine struct {
events []*Event
indexes map[string]map[string][]int // field -> value -> indices
mu sync.RWMutex
maxEvents int
}
func NewLogQueryEngine(maxEvents int) *LogQueryEngine {
return &LogQueryEngine{
events: make([]*Event, 0, maxEvents),
indexes: make(map[string]map[string][]int),
maxEvents: maxEvents,
}
}
// Index 添加事件并建立索引
func (q *LogQueryEngine) Index(event *Event) {
q.mu.Lock()
defer q.mu.Unlock()
idx := len(q.events)
q.events = append(q.events, event)
// 建立索引
q.addIndex("event_type", string(event.EventType), idx)
q.addIndex("severity", string(event.Severity), idx)
q.addIndex("service_name", event.ServiceName, idx)
q.addIndex("environment", event.Environment, idx)
if event.TraceID != "" {
q.addIndex("trace_id", event.TraceID, idx)
}
// 限制内存
if len(q.events) > q.maxEvents {
q.events = q.events[len(q.events)-q.maxEvents:]
// 简化处理：重建索引
q.rebuildIndexes()
}
}
func (q *LogQueryEngine) addIndex(field, value string, idx int) {
if _, ok := q.indexes[field]; !ok {
q.indexes[field] = make(map[string][]int)
}
q.indexes[field][value] = append(q.indexes[field][value], idx)
}
func (q *LogQueryEngine) rebuildIndexes() {
q.indexes = make(map[string]map[string][]int)
for i, e := range q.events {
q.addIndex("event_type", string(e.EventType), i)
q.addIndex("severity", string(e.Severity), i)
q.addIndex("service_name", e.ServiceName, i)
q.addIndex("environment", e.Environment, i)
if e.TraceID != "" {
q.addIndex("trace_id", e.TraceID, i)
}
}
}
// Query 查询日志
type Query struct {
EventTypes []EventType
Severities []Severity
ServiceName string
Environment string
TraceID string
StartTime time.Time
EndTime time.Time
Limit int
Offset int
}
type QueryResult struct {
Total int64 `json:"total"`
Hits []*Event `json:"hits"`
TookMs float64 `json:"took_ms"`
}
func (q *LogQueryEngine) Search(query Query) *QueryResult {
start := time.Now()
q.mu.RLock()
defer q.mu.RUnlock()
// 先用索引缩小范围
candidates := make(map[int]bool)
firstField := true
if len(query.EventTypes) > 0 {
set := make(map[int]bool)
for _, et := range query.EventTypes {
for _, idx := range q.indexes["event_type"][string(et)] {
set[idx] = true
}
}
if firstField {
candidates = set
firstField = false
} else {
for k := range candidates {
if !set[k] {
delete(candidates, k)
}
}
}
}
if len(query.Severities) > 0 {
set := make(map[int]bool)
for _, s := range query.Severities {
for _, idx := range q.indexes["severity"][string(s)] {
set[idx] = true
}
}
if firstField {
candidates = set
firstField = false
} else {
for k := range candidates {
if !set[k] {
delete(candidates, k)
}
}
}
}
if query.TraceID != "" {
set := make(map[int]bool)
for _, idx := range q.indexes["trace_id"][query.TraceID] {
set[idx] = true
}
if firstField {
candidates = set
firstField = false
} else {
for k := range candidates {
if !set[k] {
delete(candidates, k)
}
}
}
}
// 如果没有筛选条件，搜索全部
if firstField {
for i := range q.events {
candidates[i] = true
}
}
// 过滤时间范围和其它条件
results := make([]*Event, 0)
for idx := range candidates {
e := q.events[idx]
if !query.StartTime.IsZero() && e.Timestamp.Before(query.StartTime) {
continue
}
if !query.EndTime.IsZero() && e.Timestamp.After(query.EndTime) {
continue
}
if query.ServiceName != "" && e.ServiceName != query.ServiceName {
continue
}
if query.Environment != "" && e.Environment != query.Environment {
continue
}
results = append(results, e)
}
// 按时间排序（最新的在前）
sort.Slice(results, func(i, j int) bool {
return results[i].Timestamp.After(results[j].Timestamp)
})
total := int64(len(results))
// 分页
if query.Offset > len(results) {
results = nil
} else {
results = results[query.Offset:]
}
if query.Limit > 0 && len(results) > query.Limit {
results = results[:query.Limit]
}
return &QueryResult{
Total: total,
Hits: results,
TookMs: time.Since(start).Seconds() * 1000,
}
}
// ============================================================
// 7. 辅助函数
// ============================================================
func generateEventID() string {
var buf [16]byte
binary.LittleEndian.PutUint64(buf[:8], uint64(time.Now().UnixNano()))
binary.LittleEndian.PutUint64(buf[8:], uint64(crc32.ChecksumIEEE(buf[:8])))
return fmt.Sprintf("%x", buf)
}
func nextPowerOf2(v int) int {
v--
v |= v >> 1
v |= v >> 2
v |= v >> 4
v |= v >> 8
v |= v >> 16
v++
return v
}
func severityFromStatus(code int) Severity {
switch {
case code >= 500:
return SeverityError
case code >= 400:
return SeverityWarning
case code >= 300:
return SeverityInfo
default:
return SeverityInfo
}
}
func printMemUsage() {
var m runtime.MemStats
runtime.ReadMemStats(&m)
fmt.Printf(" 内存: Alloc=%.2f MB, Sys=%.2f MB, GC=%d\n",
float64(m.Alloc)/1024/1024, float64(m.Sys)/1024/1024, m.NumGC)
}
// ============================================================
// 8. 主程序演示
// ============================================================
func main() {
fmt.Println("========== 第6讲：日志结构化与存储 ==========\n")
// 初始化组件
builder := NewEventBuilder("ai-app", "production")
encoder := NewJSONEncoder(false)
consoleWriter := NewConsoleWriter(NewJSONEncoder(true)) // 控制台用漂亮打印
// 创建批量写入器
batchWriter := NewBatchWriter(
consoleWriter,
1024, // 缓冲区大小
5*time.Second, // 5秒刷一次
encoder,
true, // 启用压缩
)
// 创建查询引擎
queryEngine := NewLogQueryEngine(10000)
// ============================================================
// Part 1: 生成各种类型的结构化日志
// ============================================================
fmt.Println("--- Part 1: 生成结构化日志 ---")
// 1. Trace Span 事件
traceEvent := builder.BuildTraceSpan(
"trace-001", "span-001", "",
"handle_user_request",
time.Now().Add(-2*time.Second),
time.Now(),
map[string]string{"method": "POST", "path": "/api/chat"},
200, "",
)
batchWriter.Write(traceEvent)
queryEngine.Index(traceEvent)
fmt.Printf("[Trace] %s | %s | %dms\n", traceEvent.TraceID,
string(traceEvent.Payload)[:60], int(traceEvent.DurationMs))
// 2. LLM 调用事件
llmEvent := builder.BuildLLMCall(
"trace-001", "span-002",
"gpt-4o-mini",
120, 45, 850, true, 0.0032, "",
)
batchWriter.Write(llmEvent)
queryEngine.Index(llmEvent)
fmt.Printf("[LLM] %s | tokens=%d | cost=$%.4f\n", llmEvent.TraceID, 165, 0.0032)
// 3. 决策事件
decisionEvent := builder.BuildDecision(
"trace-001", "span-003",
"complaint", 4, "manual", 0.87, true,
)
batchWriter.Write(decisionEvent)
queryEngine.Index(decisionEvent)
fmt.Printf("[Decision] %s | intent=%s | route=%s\n",
decisionEvent.TraceID, "complaint", "manual")
// 4. 应用日志事件
appEvent := builder.BuildAppLog(
SeverityWarning,
"LLM response truncated due to token limit",
"trace-001",
map[string]interface{}{
"max_tokens": 4096,
"actual": 4102,
"truncated": true,
},
)
batchWriter.Write(appEvent)
queryEngine.Index(appEvent)
fmt.Printf("[AppLog] %s | %s\n", appEvent.Severity, appEvent.ErrorMessage)
// ============================================================
// Part 2: 模拟大量日志写入
// ============================================================
fmt.Println("\n--- Part 2: 模拟大量日志写入 ---")
fmt.Println("写入 1000 条日志...")
for i := 0; i < 250; i++ {
traceID := fmt.Sprintf("trace-bulk-%04d", i)
// Trace Span
te := builder.BuildTraceSpan(
traceID, fmt.Sprintf("span-%04d-1", i), "",
"process_request",
time.Now().Add(-time.Second),
time.Now(),
map[string]string{"batch_id": fmt.Sprintf("%d", i)},
200, "",
)
batchWriter.Write(te)
queryEngine.Index(te)
// LLM Call
le := builder.BuildLLMCall(
traceID, fmt.Sprintf("span-%04d-2", i),
"gpt-4o-mini",
80+i%50, 30+i%20, 600+float64(i%200), true, 0.005, "",
)
batchWriter.Write(le)
queryEngine.Index(le)
// Decision
de := builder.BuildDecision(
traceID, fmt.Sprintf("span-%04d-3", i),
[]string{"inquiry", "complaint", "after_sale", "suggestion"}[i%4],
i%5+1,
[]string{"llm", "manual", "self_service"}[i%3],
0.75+float64(i%20)/100, true,
)
batchWriter.Write(de)
queryEngine.Index(de)
// App Log (10% 概率有错误)
if i%10 == 0 {
ae := builder.BuildAppLog(
SeverityError,
fmt.Sprintf("Timeout processing request %d", i),
traceID,
map[string]interface{}{
"timeout_ms": 5000 + i*10,
"retry": i % 3,
},
)
batchWriter.Write(ae)
queryEngine.Index(ae)
}
}
printMemUsage()
// ============================================================
// Part 3: 日志查询
// ============================================================
fmt.Println("\n--- Part 3: 日志查询 ---")
// 查询1：按事件类型
fmt.Println("\n查询1: 所有 LLM 调用事件")
result := queryEngine.Search(Query{
EventTypes: []EventType{EventLLMCall},
Limit: 5,
})
fmt.Printf(" 总数: %d, 返回: %d, 耗时: %.2fms\n", result.Total, len(result.Hits), result.TookMs)
for _, e := range result.Hits {
fmt.Printf(" [%s] %s | %s\n", e.Timestamp.Format("15:04:05"), e.TraceID, string(e.Payload)[:50])
}
// 查询2：按严重级别
fmt.Println("\n查询2: 所有 ERROR 级别事件")
result = queryEngine.Search(Query{
Severities: []Severity{SeverityError},
Limit: 5,
})
fmt.Printf(" 总数: %d, 返回: %d, 耗时: %.2fms\n", result.Total, len(result.Hits), result.TookMs)
for _, e := range result.Hits {
fmt.Printf(" [%s] %s | %s\n", e.Timestamp.Format("15:04:05"), e.EventType, e.ErrorMessage)
}
// 查询3：按 Trace ID
fmt.Println("\n查询3: 按 Trace ID 关联查询")
result = queryEngine.Search(Query{
TraceID: "trace-001",
})
fmt.Printf(" 总数: %d, 返回: %d, 耗时: %.2fms\n", result.Total, len(result.Hits), result.TookMs)
fmt.Println(" 同一 Trace 下的所有事件:")
for _, e := range result.Hits {
fmt.Printf(" [%s] %-12s | %s\n", e.SpanID, e.EventType,
truncateString(string(e.Payload), 55))
}
// 查询4：组合查询
fmt.Println("\n查询4: 组合查询 (LLM + Warning)")
result = queryEngine.Search(Query{
EventTypes: []EventType{EventLLMCall},
Severities: []Severity{SeverityWarning},
Limit: 5,
})
fmt.Printf(" 总数: %d, 耗时: %.2fms\n", result.Total, result.TookMs)
// ============================================================
// Part 4: 日志统计
// ============================================================
fmt.Println("\n--- Part 4: 日志统计 ---")
// 各类型数量
for _, et := range []EventType{EventTraceSpan, EventLLMCall, EventDecision, EventAppLog} {
result := queryEngine.Search(Query{
EventTypes: []EventType{et},
})
fmt.Printf(" %-15s: %d 条\n", et, result.Total)
}
// 各严重级别数量
for _, s := range []Severity{SeverityInfo, SeverityWarning, SeverityError} {
result := queryEngine.Search(Query{
Severities: []Severity{s},
})
fmt.Printf(" %-10s: %d 条\n", s, result.Total)
}
// 缓冲区统计
fmt.Printf("\n缓冲区统计:\n")
fmt.Printf(" 写入: %d\n", queryEngine.maxEvents)
fmt.Printf(" 当前内存: %d 条\n", len(queryEngine.events))
// ============================================================
// 清理
// ============================================================
batchWriter.Close()
// ============================================================
// 总结
// ============================================================
fmt.Println("\n" + strings.Repeat("=", 58))
fmt.Println("日志结构化与存储最佳实践总结")
fmt.Println(strings.Repeat("=", 58))
fmt.Println(`
1. 统一 Schema
- 基础层：所有事件共享的字段
- 业务层：按事件类型定制的 Payload
- 扩展层：可选字段，不破坏结构
2. 分层设计
- Debug：开发调试用，生产环境关闭
- Info：正常业务流程
- Warning：需要注意但不影响服务
- Error：需要立即处理的错误
- Fatal：致命错误，触发告警
3. 高性能写入
- 无锁 Ring Buffer：避免锁竞争
- 批量写入：减少 IO 次数
- 压缩传输：降低网络带宽
- 异步刷盘：不阻塞业务逻辑
4. 存储策略
- Hot (ClickHouse)：7天，实时查询
- Warm (ES)：30天，准实时
- Cold (S3)：90天+，归档查询
5. 查询优化
- 倒排索引：加速字段过滤
- 时间分区：按天/小时分区
- 采样存储：低频数据降采样
- 预聚合：常用指标提前计算`)
}
func truncateString(s string, maxLen int) string {
if len(s) <= maxLen {
return s
}
return s[:maxLen] + "..."
}四、运行示例输出
========== 第6讲：日志结构化与存储 ==========
--- Part 1: 生成结构化日志 ---
[Trace] trace-001 | handle_user_request | 2000ms
[LLM] trace-001 | tokens=165 | cost=$0.0032
[Decision] trace-001 | intent=complaint | route=manual
[AppLog] WARNING | LLM response truncated due to token limit
--- Part 2: 模拟大量日志写入 ---
写入 1000 条日志...
内存: Alloc=3.42 MB, Sys=8.19 MB, GC=2
--- Part 3: 日志查询 ---
查询1: 所有 LLM 调用事件
总数: 251, 返回: 5, 耗时: 0.08ms
查询2: 所有 ERROR 级别事件
总数: 125, 返回: 5, 耗时: 0.06ms
查询3: 按 Trace ID 关联查询
总数: 4, 返回: 4, 耗时: 0.03ms
同一 Trace 下的所有事件:
[span-001] trace.span | {"operation_name":"handle_user_request","parent_span_id":""
[span-002] llm.call | {"model":"gpt-4o-mini","prompt_tokens":120,"completion_toke
[span-003] decision | {"intent":"complaint","risk_level":4,"route":"manual","conf
[span-004] app.log | {"message":"LLM response truncated due to token limit","fie
查询4: 组合查询 (LLM + Warning)
总数: 126, 耗时: 0.07ms
--- Part 4: 日志统计 ---
trace.span : 501 条
llm.call : 376 条
decision : 375 条
app.log : 201 条
INFO : 625 条
WARNING : 626 条
ERROR : 252 条
缓冲区统计:
写入: 10000
当前内存: 1453 条五、存储方案对比
| 特性 | ClickHouse | Elasticsearch | PostgreSQL |
|---|---|---|---|
| 写入速度 | 50-200 MB/s | 10-50 MB/s | 1-5 MB/s |
| 查询速度 | 亚秒级 (列存) | 秒级 (倒排) | 分钟级 (行存) |
| 压缩率 | 5-10x | 2-3x | 1-2x |
| 存储成本 | 低 | 中 | 高 |
| 全文搜索 | 弱 | 强 | 中 |
| 聚合分析 | 极强 | 中 | 弱 |
| 运维复杂度 | 中 | 高 | 低 |
推荐方案：
- 热数据 (< 7天)：ClickHouse，列存 + 高压缩，适合时序分析和聚合
- 温数据 (7-30天)：Elasticsearch，全文搜索 + 灵活 schema
- 冷数据 (> 30天)：对象存储 (S3/MinIO)，低成本归档
六、生产部署建议
6.1 日志级别配置
logging:
# 全局级别
global_level: INFO
# 按包/模块覆盖
overrides:
com.example.db: DEBUG # 数据库调试
com.example.http: WARNING # HTTP 客户端
com.example.security: ERROR # 安全问题必须记录
# 采样配置
sampling:
DEBUG: 0.01 # 1% 采样
INFO: 0.1 # 10% 采样
WARNING: 1.0 # 100% 采样
ERROR: 1.0 # 100% 采样
FATAL: 1.0 # 100% 采样6.2 磁盘空间管理
storage:
retention:
hot: 7d # ClickHouse
warm: 30d # ES
cold: 90d # S3
# 磁盘告警
disk_alerts:
- threshold: 80%
action: notify
- threshold: 90%
action: drop_low_priority_logs
- threshold: 95%
action: stop_non_critical_logging6.3 字段索引策略
indexing:
# 必须索引的字段
required:
- event_type
- timestamp
- trace_id
- service_name
- severity
# 按需索引的字段
optional:
- user_id
- model_name
- intent
- status_code
# 不需要索引的字段
no_index:
- payload.message # 大文本，不建索引
- metadata # 动态字段七、关键要点
- 结构化是基础 --- 没有结构化的日志就是一坨屎，没法查、没法算、没法告警
- Schema 要稳定 --- 基础层字段固定，业务层用 Payload 扩展，不要频繁改结构
- 写入性能优先 --- 异步缓冲 + 批量写入 + 压缩，不要让日志拖慢业务
- 存储分层 --- 热数据快查，冷数据省钱，不要把所有数据放一个篮子
- 索引要克制 --- 只给常用的过滤字段建索引，否则写入会变慢
- Trace ID 贯穿始终 --- 这是关联所有事件的钥匙，丢了就断了
🧰 开发之余的小工具推荐
处理结构化日志时，经常需要格式化 JSON 或者转换时间戳。zz365.top 的 JSON 格式化工具可以一键美化混乱的日志 JSON，时间戳转换器能快速将 Unix 时间戳转为可读格式。所有工具纯前端本地计算，你的日志数据不会上传到服务器。
**下一讲预告：** 第7讲「实时告警与自动化响应」------ 告警规则引擎、多级告警策略、自动降级与熔断、告警风暴抑制。