第8讲：记忆系统的监控与运维——Prometheus 指标、Grafana 面板与告警体系

一、监控架构

┌──────────────────────────────────────────────────────────────────────────────┐
│                        记忆系统监控架构                                         │
├──────────────────────────────────────────────────────────────────────────────┤
│                                                                                │
│  ┌────────────────────────────────────────────────────────────────────────┐   │
│  │                          应用层 (Application)                             │   │
│  │  ┌─────────────┐  ┌─────────────┐  ┌─────────────┐  ┌─────────────┐  │   │
│  │  │ Memory      │  │ Security    │  │ HA          │  │ Compress    │  │   │
│  │  │ Manager     │  │ Manager     │  │ Manager     │  │ Manager     │  │   │
│  │  └──────┬──────┘  └──────┬──────┘  └──────┬──────┘  └──────┬──────┘  │   │
│  └─────────┼───────────────┼───────────────┼───────────────┼──────────┘   │
│            │               │               │               │               │
│            ▼               ▼               ▼               ▼               │
│  ┌────────────────────────────────────────────────────────────────────────┐   │
│  │                      指标收集层 (Metrics Collector)                      │   │
│  │                                                                        │   │
│  │  ┌──────────────────────────────────────────────────────────────────┐  │   │
│  │  │                  Prometheus HTTP Handler (/metrics)              │  │   │
│  │  └──────────────────────────────────────────────────────────────────┘  │   │
│  │                                                                        │   │
│  │  ┌─────────────┐  ┌─────────────┐  ┌─────────────┐  ┌─────────────┐  │   │
│  │  │ Counter     │  │ Gauge       │  │ Histogram   │  │ Summary     │  │   │
│  │  │ (计数)      │  │ (瞬时值)    │  │ (分布)      │  │ (分位数)    │  │   │
│  │  └─────────────┘  └─────────────┘  └─────────────┘  └─────────────┘  │   │
│  └────────────────────────────────┬───────────────────────────────────────┘   │
│                                   │                                             │
│                                   ▼                                             │
│  ┌────────────────────────────────────────────────────────────────────────┐   │
│  │                     指标暴露层 (Exporter)                                │   │
│  │                                                                        │   │
│  │  ┌────────────────────┐  ┌────────────────────┐  ┌──────────────────┐  │   │
│  │  │ 内置 Metrics       │  │ 自定义 Collector   │  │ 健康检查端点     │  │   │
│  │  │ Go runtime         │  │ 业务指标           │  │ /healthz         │  │   │
│  │  │ 进程指标           │  │ 层级别指标         │  │ /readyz          │  │   │
│  │  └────────────────────┘  └────────────────────┘  └──────────────────┘  │   │
│  └────────────────────────────────┬───────────────────────────────────────┘   │
│                                   │                                             │
│                                   ▼                                             │
│  ┌────────────────────────────────────────────────────────────────────────┐   │
│  │                       数据存储与可视化层                                  │   │
│  │                                                                        │   │
│  │  ┌────────────────────┐  ┌────────────────────┐  ┌──────────────────┐  │   │
│  │  │ Prometheus Server  │  │ Grafana            │  │ Alertmanager     │  │   │
│  │  │ 时序数据库         │  │ 仪表盘             │  │ 告警管理         │  │   │
│  │  │ 抓取 / 查询        │  │ 图表 / 报警        │  │ 通知 / 静默      │  │   │
│  │  └────────────────────┘  └────────────────────┘  └──────────────────┘  │   │
│  └────────────────────────────────────────────────────────────────────────┘   │
│                                                                                │
│  ┌────────────────────────────────────────────────────────────────────────┐   │
│  │                       告警通知渠道                                       │   │
│  │  ┌──────────┐  ┌──────────┐  ┌──────────┐  ┌──────────┐  ┌────────┐ │   │
│  │  │ Email    │  │ Slack    │  │ WeChat   │  │ PagerDuty│  │ Webhook│ │   │
│  │  └──────────┘  └──────────┘  └──────────┘  └──────────┘  └────────┘ │   │
│  └────────────────────────────────────────────────────────────────────────┘   │
│                                                                                │
└──────────────────────────────────────────────────────────────────────────────┘

---

二、核心数据结构

// memory/monitoring/types.go
package monitoring

import (
    "context"
    "sync"
    "time"
)

// MetricType 指标类型
type MetricType string

const (
    MetricCounter   MetricType = "counter"
    MetricGauge     MetricType = "gauge"
    MetricHistogram MetricType = "histogram"
    MetricSummary   MetricType = "summary"
)

// MetricLabel 指标标签
type MetricLabel struct {
    Name  string `json:"name"`
    Value string `json:"value"`
}

// MetricSample 指标样本
type MetricSample struct {
    Name      string         `json:"name"`
    Type      MetricType     `json:"type"`
    Value     float64        `json:"value"`
    Labels    []MetricLabel  `json:"labels,omitempty"`
    Timestamp time.Time      `json:"timestamp"`
    Help      string         `json:"help"`
}

// MemoryMetrics 记忆系统整体指标
type MemoryMetrics struct {
    // 操作计数
    TotalOps       int64            `json:"total_ops"`
    SuccessOps     int64            `json:"success_ops"`
    FailedOps      int64            `json:"failed_ops"`
    OpsByType      map[string]int64 `json:"ops_by_type"`      // save, get, search, delete
    OpsByLayer     map[string]int64 `json:"ops_by_layer"`     // working, episodic, semantic
    OpsByResult    map[string]int64 `json:"ops_by_result"`    // success, denied, error

    // 延迟
    LatencyP50     float64          `json:"latency_p50_ms"`
    LatencyP90     float64          `json:"latency_p90_ms"`
    LatencyP99     float64          `json:"latency_p99_ms"`
    AvgLatency     float64          `json:"avg_latency_ms"`

    // 存储
    TotalEntries   int64            `json:"total_entries"`
    EntriesByLayer map[string]int64 `json:"entries_by_layer"`
    StorageSize    int64            `json:"storage_size_bytes"`

    // 安全
    AuthCheckCount int64            `json:"auth_check_count"`
    AuthDeniedCount int64           `json:"auth_denied_count"`
    EncryptCount   int64            `json:"encrypt_count"`
    DecryptCount   int64            `json:"decrypt_count"`

    // 复制与高可用
    ReplicationLag map[string]int64 `json:"replication_lag"`
    FailoverCount  int64            `json:"failover_count"`
    IsLeader       bool             `json:"is_leader"`

    // 压缩
    CompressedCount int64           `json:"compressed_count"`
    ForgottenCount  int64           `json:"forgotten_count"`
    MergedCount     int64           `json:"merged_count"`

    UpdatedAt time.Time `json:"updated_at"`
}

// HealthStatus 健康检查状态
type HealthStatus struct {
    Status      string            `json:"status"`       // healthy, degraded, unhealthy
    Components  map[string]ComponentHealth `json:"components"`
    Uptime      string            `json:"uptime"`
    Version     string            `json:"version"`
    LastCheck   time.Time         `json:"last_check"`
}

type ComponentHealth struct {
    Status  string `json:"status"`  // up, down, degraded
    Message string `json:"message,omitempty"`
    Latency string `json:"latency,omitempty"`
}

// AlertRule 告警规则
type AlertRule struct {
    ID          string            `json:"id"`
    Name        string            `json:"name"`
    Metric      string            `json:"metric"`
    Condition   AlertCondition    `json:"condition"`
    Threshold   float64           `json:"threshold"`
    Duration    time.Duration     `json:"duration"`     // 持续多久才触发
    Severity    AlertSeverity     `json:"severity"`
    Labels      map[string]string `json:"labels"`
    Annotations map[string]string `json:"annotations"`
    Enabled     bool              `json:"enabled"`
}

type AlertCondition string

const (
    CondGreaterThan     AlertCondition = "gt"
    CondLessThan        AlertCondition = "lt"
    CondEqual           AlertCondition = "eq"
    CondNotEqual        AlertCondition = "ne"
    CondGreaterOrEqual  AlertCondition = "gte"
    CondLessOrEqual     AlertCondition = "lte"
)

type AlertSeverity string

const (
    SevCritical AlertSeverity = "critical"
    SevWarning  AlertSeverity = "warning"
    SevInfo     AlertSeverity = "info"
)

// MonitoringConfig 监控配置
type MonitoringConfig struct {
    Enabled          bool              `yaml:"enabled"`
    ListenAddr       string            `yaml:"listen_addr"`       // :9090
    MetricsPath      string            `yaml:"metrics_path"`      // /metrics
    HealthPath       string            `yaml:"health_path"`       // /healthz
    CollectInterval  string            `yaml:"collect_interval"`  // 15s
    ExportGoMetrics  bool              `yaml:"export_go_metrics"`
    ExportProcessMetrics bool          `yaml:"export_process_metrics"`
    HistogramBuckets []float64         `yaml:"histogram_buckets"`
    AlertRules       []AlertRule       `yaml:"alert_rules"`
}

// MetricsCollector 指标收集器接口
type MetricsCollector interface {
    Collect(ctx context.Context) ([]MetricSample, error)
    Name() string
}

---

三、指标收集器

// memory/monitoring/collector.go
package monitoring

import (
    "context"
    "fmt"
    "log"
    "runtime"
    "sync"
    "time"

    "github.com/prometheus/client_golang/prometheus"
    "github.com/shirou/gopsutil/v3/cpu"
    "github.com/shirou/gopsutil/v3/disk"
    "github.com/shirou/gopsutil/v3/mem"
    "github.com/shirou/gopsutil/v3/net"
)

// MemoryCollector 记忆系统指标收集器
type MemoryCollector struct {
    metrics *MemoryMetrics
    mu      sync.RWMutex

    // Prometheus 指标
    totalOps        *prometheus.CounterVec
    latencyHist     *prometheus.HistogramVec
    activeEntries   prometheus.Gauge
    storageSize     prometheus.Gauge
    replicationLag  *prometheus.GaugeVec
    authDenied      *prometheus.CounterVec
    encryptOps      *prometheus.CounterVec
    compressOps     *prometheus.CounterVec

    collectors []MetricsCollector
    interval   time.Duration
    stopCh     chan struct{}
}

func NewMemoryCollector(config MonitoringConfig) *MemoryCollector {
    mc := &MemoryCollector{
        metrics: &MemoryMetrics{
            OpsByType:     make(map[string]int64),
            OpsByLayer:    make(map[string]int64),
            OpsByResult:   make(map[string]int64),
            EntriesByLayer: make(map[string]int64),
            ReplicationLag: make(map[string]int64),
        },
        interval: parseDuration(config.CollectInterval, 15*time.Second),
        stopCh:   make(chan struct{}),
    }

    // 初始化 Prometheus 指标
    mc.totalOps = prometheus.NewCounterVec(
        prometheus.CounterOpts{
            Name: "memory_total_operations",
            Help: "Total number of memory operations",
        },
        []string{"type", "layer", "result"},
    )

    mc.latencyHist = prometheus.NewHistogramVec(
        prometheus.HistogramOpts{
            Name:    "memory_operation_latency_seconds",
            Help:    "Operation latency distribution",
            Buckets: config.HistogramBuckets,
        },
        []string{"type"},
    )

    mc.activeEntries = prometheus.NewGauge(
        prometheus.GaugeOpts{
            Name: "memory_active_entries",
            Help: "Number of active memory entries",
        },
    )

    mc.storageSize = prometheus.NewGauge(
        prometheus.GaugeOpts{
            Name: "memory_storage_size_bytes",
            Help: "Total storage size in bytes",
        },
    )

    mc.replicationLag = prometheus.NewGaugeVec(
        prometheus.GaugeOpts{
            Name: "memory_replication_lag",
            Help: "Replication lag in entries",
        },
        []string{"node_id"},
    )

    mc.authDenied = prometheus.NewCounterVec(
        prometheus.CounterOpts{
            Name: "memory_auth_denied_total",
            Help: "Total authentication denials",
        },
        []string{"reason"},
    )

    mc.encryptOps = prometheus.NewCounterVec(
        prometheus.CounterOpts{
            Name: "memory_encryption_operations_total",
            Help: "Total encryption/decryption operations",
        },
        []string{"operation"}, // encrypt, decrypt
    )

    mc.compressOps = prometheus.NewCounterVec(
        prometheus.CounterOpts{
            Name: "memory_compress_operations_total",
            Help: "Total compression operations",
        },
        []string{"operation"}, // compress, forget, merge
    )

    // 注册指标
    prometheus.MustRegister(
        mc.totalOps,
        mc.latencyHist,
        mc.activeEntries,
        mc.storageSize,
        mc.replicationLag,
        mc.authDenied,
        mc.encryptOps,
        mc.compressOps,
    )

    return mc
}

// Start 启动自动收集
func (mc *MemoryCollector) Start() {
    ticker := time.NewTicker(mc.interval)
    go func() {
        for {
            select {
            case <-ticker.C:
                mc.collectSystemMetrics()
                mc.collectCustomMetrics()
            case <-mc.stopCh:
                ticker.Stop()
                return
            }
        }
    }()
    log.Printf("[MONITOR] Metrics collector started, interval=%v", mc.interval)
}

// Stop 停止收集
func (mc *MemoryCollector) Stop() {
    close(mc.stopCh)
}

// RecordOperation 记录操作
func (mc *MemoryCollector) RecordOperation(opType, layer, result string, latency time.Duration) {
    mc.mu.Lock()
    mc.metrics.TotalOps++
    if result == "success" {
        mc.metrics.SuccessOps++
    } else {
        mc.metrics.FailedOps++
    }
    mc.metrics.OpsByType[opType]++
    mc.metrics.OpsByLayer[layer]++
    mc.metrics.OpsByResult[result]++
    mc.metrics.AvgLatency = (mc.metrics.AvgLatency*float64(mc.metrics.TotalOps-1) +
        float64(latency.Milliseconds())) / float64(mc.metrics.TotalOps)
    mc.mu.Unlock()

    // Prometheus 指标
    mc.totalOps.WithLabelValues(opType, layer, result).Inc()
    mc.latencyHist.WithLabelValues(opType).Observe(latency.Seconds())
}

// RecordAuthDenial 记录认证拒绝
func (mc *MemoryCollector) RecordAuthDenial(reason string) {
    mc.mu.Lock()
    mc.metrics.AuthDeniedCount++
    mc.mu.Unlock()

    mc.authDenied.WithLabelValues(reason).Inc()
}

// RecordEncryption 记录加密操作
func (mc *MemoryCollector) RecordEncryption(operation string) {
    mc.mu.Lock()
    if operation == "encrypt" {
        mc.metrics.EncryptCount++
    } else {
        mc.metrics.DecryptCount++
    }
    mc.mu.Unlock()

    mc.encryptOps.WithLabelValues(operation).Inc()
}

// RecordCompression 记录压缩操作
func (mc *MemoryCollector) RecordCompression(operation string) {
    mc.mu.Lock()
    switch operation {
    case "compress":
        mc.metrics.CompressedCount++
    case "forget":
        mc.metrics.ForgottenCount++
    case "merge":
        mc.metrics.MergedCount++
    }
    mc.mu.Unlock()

    mc.compressOps.WithLabelValues(operation).Inc()
}

// UpdateEntriesCount 更新条目计数
func (mc *MemoryCollector) UpdateEntriesCount(layer string, count int64) {
    mc.mu.Lock()
    mc.metrics.EntriesByLayer[layer] = count
    total := int64(0)
    for _, c := range mc.metrics.EntriesByLayer {
        total += c
    }
    mc.metrics.TotalEntries = total
    mc.mu.Unlock()

    mc.activeEntries.Set(float64(total))
}

// UpdateStorageSize 更新存储大小
func (mc *MemoryCollector) UpdateStorageSize(size int64) {
    mc.mu.Lock()
    mc.metrics.StorageSize = size
    mc.mu.Unlock()

    mc.storageSize.Set(float64(size))
}

// UpdateReplicationLag 更新复制延迟
func (mc *MemoryCollector) UpdateReplicationLag(nodeID string, lag int64) {
    mc.mu.Lock()
    mc.metrics.ReplicationLag[nodeID] = lag
    mc.mu.Unlock()

    mc.replicationLag.WithLabelValues(nodeID).Set(float64(lag))
}

// GetMetrics 获取当前指标快照
func (mc *MemoryCollector) GetMetrics() *MemoryMetrics {
    mc.mu.RLock()
    defer mc.mu.RUnlock()

    snapshot := *mc.metrics
    snapshot.UpdatedAt = time.Now()
    return &snapshot
}

// collectSystemMetrics 收集系统指标
func (mc *MemoryCollector) collectSystemMetrics() {
    // CPU 使用率
    cpuPercent, err := cpu.Percent(0, false)
    if err == nil && len(cpuPercent) > 0 {
        prometheus.MustRegister(prometheus.NewGaugeFunc(
            prometheus.GaugeOpts{
                Name: "system_cpu_usage_percent",
                Help: "System CPU usage percentage",
            },
            func() float64 { return cpuPercent[0] },
        ))
    }

    // 内存使用
    memInfo, err := mem.VirtualMemory()
    if err == nil {
        prometheus.MustRegister(prometheus.NewGaugeFunc(
            prometheus.GaugeOpts{
                Name: "system_memory_usage_bytes",
                Help: "System memory usage in bytes",
            },
            func() float64 { return float64(memInfo.Used) },
        ))
    }

    // Go runtime 指标
    if mc.config.ExportGoMetrics {
        var m runtime.MemStats
        runtime.ReadMemStats(&m)
        // goroutines
        prometheus.MustRegister(prometheus.NewGaugeFunc(
            prometheus.GaugeOpts{
                Name: "go_goroutines",
                Help: "Number of goroutines",
            },
            func() float64 { return float64(runtime.NumGoroutine()) },
        ))
        // GC 次数
        prometheus.MustRegister(prometheus.NewGaugeFunc(
            prometheus.GaugeOpts{
                Name: "go_gc_total",
                Help: "Total GC cycles",
            },
            func() float64 { return float64(m.NumGC) },
        ))
    }
}

// collectCustomMetrics 收集自定义指标
func (mc *MemoryCollector) collectCustomMetrics() {
    for _, collector := range mc.collectors {
        samples, err := collector.Collect(context.Background())
        if err != nil {
            log.Printf("[MONITOR] Collector %s failed: %v", collector.Name(), err)
            continue
        }
        for _, sample := range samples {
            mc.processSample(sample)
        }
    }
}

// processSample 处理指标样本
func (mc *MemoryCollector) processSample(sample MetricSample) {
    labels := make([]string, len(sample.Labels)*2)
    for i, label := range sample.Labels {
        labels[i*2] = label.Name
        labels[i*2+1] = label.Value
    }

    switch sample.Type {
    case MetricCounter:
        counter := prometheus.NewCounterVec(
            prometheus.CounterOpts{
                Name: sample.Name,
                Help: sample.Help,
            },
            getLabelNames(sample.Labels),
        )
        counter.WithLabelValues(labels...).Add(sample.Value)
        prometheus.MustRegister(counter)

    case MetricGauge:
        gauge := prometheus.NewGaugeVec(
            prometheus.GaugeOpts{
                Name: sample.Name,
                Help: sample.Help,
            },
            getLabelNames(sample.Labels),
        )
        gauge.WithLabelValues(labels...).Set(sample.Value)
        prometheus.MustRegister(gauge)
    }
}

// RegisterCollector 注册自定义收集器
func (mc *MemoryCollector) RegisterCollector(collector MetricsCollector) {
    mc.collectors = append(mc.collectors, collector)
}

func getLabelNames(labels []MetricLabel) []string {
    names := make([]string, len(labels))
    for i, l := range labels {
        names[i] = l.Name
    }
    return names
}

func parseDuration(s string, defaultVal time.Duration) time.Duration {
    d, err := time.ParseDuration(s)
    if err != nil {
        return defaultVal
    }
    return d
}

---

四、健康检查

// memory/monitoring/health.go
package monitoring

import (
    "context"
    "encoding/json"
    "fmt"
    "log"
    "net/http"
    "sync"
    "time"
)

// HealthChecker 健康检查器
type HealthChecker struct {
    components map[string]HealthCheckFunc
    status     *HealthStatus
    mu         sync.RWMutex
    startTime  time.Time
    version    string
}

type HealthCheckFunc func(ctx context.Context) ComponentHealth

func NewHealthChecker(version string) *HealthChecker {
    return &HealthChecker{
        components: make(map[string]HealthCheckFunc),
        status: &HealthStatus{
            Status:     "healthy",
            Components: make(map[string]ComponentHealth),
            Version:    version,
        },
        startTime: time.Now(),
        version:   version,
    }
}

// RegisterCheck 注册健康检查函数
func (hc *HealthChecker) RegisterCheck(name string, fn HealthCheckFunc) {
    hc.mu.Lock()
    defer hc.mu.Unlock()
    hc.components[name] = fn
}

// Check 执行所有健康检查
func (hc *HealthChecker) Check(ctx context.Context) *HealthStatus {
    hc.mu.Lock()
    defer hc.mu.Unlock()

    status := &HealthStatus{
        Status:     "healthy",
        Components: make(map[string]ComponentHealth),
        Uptime:     time.Since(hc.startTime).String(),
        Version:    hc.version,
        LastCheck:  time.Now(),
    }

    for name, fn := range hc.components {
        componentStatus := fn(ctx)
        status.Components[name] = componentStatus

        if componentStatus.Status == "down" {
            status.Status = "unhealthy"
        } else if componentStatus.Status == "degraded" && status.Status == "healthy" {
            status.Status = "degraded"
        }
    }

    hc.status = status
    return status
}

// HTTPHandler HTTP 健康检查处理器
func (hc *HealthChecker) HTTPHandler(w http.ResponseWriter, r *http.Request) {
    ctx := r.Context()
    status := hc.Check(ctx)

    w.Header().Set("Content-Type", "application/json")

    switch r.URL.Path {
    case "/healthz":
        // 存活检查
        if status.Status == "unhealthy" {
            w.WriteHeader(http.StatusServiceUnavailable)
        } else {
            w.WriteHeader(http.StatusOK)
        }

    case "/readyz":
        // 就绪检查
        if status.Status == "unhealthy" || status.Status == "degraded" {
            w.WriteHeader(http.StatusServiceUnavailable)
        } else {
            w.WriteHeader(http.StatusOK)
        }

    default:
        w.WriteHeader(http.StatusOK)
    }

    json.NewEncoder(w).Encode(status)
}

// PredefinedChecks 预定义的健康检查

// DatabaseHealthCheck 数据库健康检查
func DatabaseHealthCheck(db interface{ Ping() error }) HealthCheckFunc {
    return func(ctx context.Context) ComponentHealth {
        start := time.Now()
        err := db.Ping()
        latency := time.Since(start)

        if err != nil {
            return ComponentHealth{
                Status:  "down",
                Message: fmt.Sprintf("database unreachable: %v", err),
                Latency: latency.String(),
            }
        }

        return ComponentHealth{
            Status:  "up",
            Message: "database connected",
            Latency: latency.String(),
        }
    }
}

// MemoryStoreHealthCheck 记忆存储健康检查
func MemoryStoreHealthCheck(store interface{ Count(ctx context.Context) (int64, error) }) HealthCheckFunc {
    return func(ctx context.Context) ComponentHealth {
        start := time.Now()
        count, err := store.Count(ctx)
        latency := time.Since(start)

        if err != nil {
            return ComponentHealth{
                Status:  "degraded",
                Message: fmt.Sprintf("store query failed: %v", err),
                Latency: latency.String(),
            }
        }

        return ComponentHealth{
            Status:  "up",
            Message: fmt.Sprintf("store healthy, %d entries", count),
            Latency: latency.String(),
        }
    }
}

// ClusterHealthCheck 集群健康检查
func ClusterHealthCheck(cluster interface{ GetLeader() string }) HealthCheckFunc {
    return func(ctx context.Context) ComponentHealth {
        leader := cluster.GetLeader()
        if leader == "" {
            return ComponentHealth{
                Status:  "down",
                Message: "no leader elected",
            }
        }

        return ComponentHealth{
            Status:  "up",
            Message: fmt.Sprintf("leader: %s", leader),
        }
    }
}

// EncryptionHealthCheck 加密模块健康检查
func EncryptionHealthCheck(encryptor interface{ Healthy() bool }) HealthCheckFunc {
    return func(ctx context.Context) ComponentHealth {
        if encryptor.Healthy() {
            return ComponentHealth{
                Status:  "up",
                Message: "encryption module healthy",
            }
        }

        return ComponentHealth{
            Status:  "degraded",
            Message: "encryption module degraded",
        }
    }
}

// DiskSpaceHealthCheck 磁盘空间健康检查
func DiskSpaceHealthCheck(path string, thresholdPercent float64) HealthCheckFunc {
    return func(ctx context.Context) ComponentHealth {
        diskInfo, err := disk.Usage(path)
        if err != nil {
            return ComponentHealth{
                Status:  "degraded",
                Message: fmt.Sprintf("cannot get disk info: %v", err),
            }
        }

        usedPercent := diskInfo.UsedPercent
        if usedPercent > thresholdPercent {
            return ComponentHealth{
                Status:  "degraded",
                Message: fmt.Sprintf("disk usage %.1f%% exceeds threshold %.1f%%", usedPercent, thresholdPercent),
            }
        }

        return ComponentHealth{
            Status:  "up",
            Message: fmt.Sprintf("disk usage %.1f%%", usedPercent),
        }
    }
}

---

五、告警管理器

// memory/monitoring/alert.go
package monitoring

import (
    "context"
    "fmt"
    "log"
    "sync"
    "time"
)

// AlertManager 告警管理器
type AlertManager struct {
    rules      []AlertRule
    alerts     []*Alert
    evaluator  AlertEvaluator
    notifiers  []AlertNotifier
    silenceMap map[string]time.Time // rule_id -> until
    mu         sync.RWMutex
    evalInterval time.Duration
    stopCh     chan struct{}
}

type Alert struct {
    ID          string            `json:"id"`
    RuleID      string            `json:"rule_id"`
    Name        string            `json:"name"`
    Severity    AlertSeverity     `json:"severity"`
    Status      AlertStatus       `json:"status"` // firing, resolved
    Value       float64           `json:"value"`
    Threshold   float64           `json:"threshold"`
    StartedAt   time.Time         `json:"started_at"`
    ResolvedAt  *time.Time        `json:"resolved_at,omitempty"`
    Labels      map[string]string `json:"labels"`
    Annotations map[string]string `json:"annotations"`
}

type AlertStatus string

const (
    AlertFiring   AlertStatus = "firing"
    AlertResolved AlertStatus = "resolved"
)

type AlertEvaluator interface {
    Evaluate(rule AlertRule, metrics *MemoryMetrics) (bool, float64, error)
}

type AlertNotifier interface {
    Send(ctx context.Context, alert *Alert) error
    Name() string
}

func NewAlertManager(rules []AlertRule, evaluator AlertEvaluator, interval time.Duration) *AlertManager {
    am := &AlertManager{
        rules:        rules,
        alerts:       make([]*Alert, 0),
        evaluator:    evaluator,
        notifiers:    make([]AlertNotifier, 0),
        silenceMap:   make(map[string]time.Time),
        evalInterval: interval,
        stopCh:       make(chan struct{}),
    }
    return am
}

// Start 启动告警评估循环
func (am *AlertManager) Start(ctx context.Context, metricsProvider func() *MemoryMetrics) {
    ticker := time.NewTicker(am.evalInterval)
    go func() {
        for {
            select {
            case <-ticker.C:
                metrics := metricsProvider()
                am.evaluateAll(ctx, metrics)
            case <-am.stopCh:
                ticker.Stop()
                return
            }
        }
    }()
    log.Printf("[ALERT] Alert manager started, interval=%v, rules=%d", am.evalInterval, len(am.rules))
}

// Stop 停止告警评估
func (am *AlertManager) Stop() {
    close(am.stopCh)
}

// RegisterNotifier 注册通知器
func (am *AlertManager) RegisterNotifier(notifier AlertNotifier) {
    am.mu.Lock()
    defer am.mu.Unlock()
    am.notifiers = append(am.notifiers, notifier)
}

// SilenceRule 静默规则一段时间
func (am *AlertManager) SilenceRule(ruleID string, duration time.Duration) {
    am.mu.Lock()
    defer am.mu.Unlock()
    am.silenceMap[ruleID] = time.Now().Add(duration)
    log.Printf("[ALERT] Rule %s silenced until %v", ruleID, time.Now().Add(duration))
}

// evaluateAll 评估所有规则
func (am *AlertManager) evaluateAll(ctx context.Context, metrics *MemoryMetrics) {
    am.mu.RLock()
    rules := am.rules
    am.mu.RUnlock()

    for _, rule := range rules {
        if !rule.Enabled {
            continue
        }

        // 检查是否被静默
        am.mu.RLock()
        silenceUntil, silenced := am.silenceMap[rule.ID]
        am.mu.RUnlock()
        if silenced && time.Now().Before(silenceUntil) {
            continue
        }

        // 评估规则
        triggered, value, err := am.evaluator.Evaluate(rule, metrics)
        if err != nil {
            log.Printf("[ALERT] Evaluate rule %s failed: %v", rule.ID, err)
            continue
        }

        if triggered {
            am.fireAlert(ctx, rule, value)
        } else {
            am.resolveAlert(ctx, rule)
        }
    }
}

// fireAlert 触发告警
func (am *AlertManager) fireAlert(ctx context.Context, rule AlertRule, value float64) {
    am.mu.Lock()
    defer am.mu.Unlock()

    // 检查是否已经触发了相同的告警
    for _, alert := range am.alerts {
        if alert.RuleID == rule.ID && alert.Status == AlertFiring {
            return // 已触发，不重复发送
        }
    }

    alert := &Alert{
        ID:        fmt.Sprintf("alert-%d", time.Now().UnixNano()),
        RuleID:    rule.ID,
        Name:      rule.Name,
        Severity:  rule.Severity,
        Status:    AlertFiring,
        Value:     value,
        Threshold: rule.Threshold,
        StartedAt: time.Now(),
        Labels:    rule.Labels,
        Annotations: rule.Annotations,
    }

    am.alerts = append(am.alerts, alert)

    // 发送通知
    for _, notifier := range am.notifiers {
        if err := notifier.Send(ctx, alert); err != nil {
            log.Printf("[ALERT] Notifier %s failed: %v", notifier.Name(), err)
        }
    }

    log.Printf("[ALERT] FIRING: %s (value=%.2f, threshold=%.2f)", rule.Name, value, rule.Threshold)
}

// resolveAlert 解决告警
func (am *AlertManager) resolveAlert(ctx context.Context, rule AlertRule) {
    am.mu.Lock()
    defer am.mu.Unlock()

    for i, alert := range am.alerts {
        if alert.RuleID == rule.ID && alert.Status == AlertFiring {
            now := time.Now()
            am.alerts[i].Status = AlertResolved
            am.alerts[i].ResolvedAt = &now

            log.Printf("[ALERT] RESOLVED: %s", rule.Name)
            break
        }
    }
}

// GetActiveAlerts 获取活跃告警
func (am *AlertManager) GetActiveAlerts() []*Alert {
    am.mu.RLock()
    defer am.mu.RUnlock()

    var active []*Alert
    for _, alert := range am.alerts {
        if alert.Status == AlertFiring {
            active = append(active, alert)
        }
    }
    return active
}

// DefaultAlertEvaluator 默认告警评估器
type DefaultAlertEvaluator struct {
    duration time.Duration
}

func NewDefaultAlertEvaluator(duration time.Duration) *DefaultAlertEvaluator {
    return &DefaultAlertEvaluator{duration: duration}
}

func (e *DefaultAlertEvaluator) Evaluate(rule AlertRule, metrics *MemoryMetrics) (bool, float64, error) {
    var value float64

    switch rule.Metric {
    case "memory_failover_count":
        value = float64(metrics.FailoverCount)
    case "memory_replication_lag_max":
        maxLag := int64(0)
        for _, lag := range metrics.ReplicationLag {
            if lag > maxLag {
                maxLag = lag
            }
        }
        value = float64(maxLag)
    case "memory_auth_denied_rate":
        total := float64(metrics.AuthCheckCount)
        if total == 0 {
            return false, 0, nil
        }
        value = float64(metrics.AuthDeniedCount) / total * 100
    case "memory_latency_p99":
        value = metrics.LatencyP99
    case "memory_storage_usage":
        value = float64(metrics.StorageSize)
    case "memory_total_entries":
        value = float64(metrics.TotalEntries)
    default:
        return false, 0, fmt.Errorf("unknown metric: %s", rule.Metric)
    }

    switch rule.Condition {
    case CondGreaterThan:
        return value > rule.Threshold, value, nil
    case CondLessThan:
        return value < rule.Threshold, value, nil
    case CondGreaterOrEqual:
        return value >= rule.Threshold, value, nil
    case CondLessOrEqual:
        return value <= rule.Threshold, value, nil
    default:
        return false, value, nil
    }
}

---

六、通知器实现

// memory/monitoring/notifiers.go
package monitoring

import (
    "bytes"
    "context"
    "encoding/json"
    "fmt"
    "log"
    "net/http"
    "time"
)

// SlackNotifier Slack 通知器
type SlackNotifier struct {
    webhookURL string
    channel    string
    username   string
    client     *http.Client
}

func NewSlackNotifier(webhookURL, channel, username string) *SlackNotifier {
    return &SlackNotifier{
        webhookURL: webhookURL,
        channel:    channel,
        username:   username,
        client:     &http.Client{Timeout: 10 * time.Second},
    }
}

func (sn *SlackNotifier) Name() string {
    return "slack"
}

func (sn *SlackNotifier) Send(ctx context.Context, alert *Alert) error {
    color := "#36a64f"
    if alert.Severity == SevCritical {
        color = "#ff0000"
    } else if alert.Severity == SevWarning {
        color = "#ffa500"
    }

    payload := map[string]interface{}{
        "channel": sn.channel,
        "username": sn.username,
        "attachments": []map[string]interface{}{
            {
                "color": color,
                "title": fmt.Sprintf("[%s] %s", alert.Severity, alert.Name),
                "text": fmt.Sprintf(
                    "Value: %.2f\nThreshold: %.2f\nStarted: %s",
                    alert.Value, alert.Threshold, alert.StartedAt.Format(time.RFC3339),
                ),
                "fields": []map[string]interface{}{
                    {"title": "Rule ID", "value": alert.RuleID, "short": true},
                    {"title": "Status", "value": string(alert.Status), "short": true},
                },
                "footer": "Memory System Monitor",
                "ts": time.Now().Unix(),
            },
        },
    }

    data, _ := json.Marshal(payload)
    resp, err := sn.client.Post(sn.webhookURL, "application/json", bytes.NewReader(data))
    if err != nil {
        return fmt.Errorf("send slack: %w", err)
    }
    defer resp.Body.Close()

    if resp.StatusCode != http.StatusOK {
        return fmt.Errorf("slack returned %d", resp.StatusCode)
    }

    return nil
}

// EmailNotifier 邮件通知器
type EmailNotifier struct {
    smtpHost string
    smtpPort int
    from     string
    to       []string
}

func NewEmailNotifier(smtpHost string, smtpPort int, from string, to []string) *EmailNotifier {
    return &EmailNotifier{
        smtpHost: smtpHost,
        smtpPort: smtpPort,
        from:     from,
        to:       to,
    }
}

func (en *EmailNotifier) Name() string {
    return "email"
}

func (en *EmailNotifier) Send(ctx context.Context, alert *Alert) error {
    subject := fmt.Sprintf("[%s] %s - Memory System Alert", alert.Severity, alert.Name)
    body := fmt.Sprintf(`
Alert Details:
==============
Name:      %s
Severity:  %s
Rule ID:   %s
Value:     %.2f
Threshold: %.2f
Started:   %s
Status:    %s

Labels:
%s
Annotations:
%s
`,
        alert.Name, alert.Severity, alert.RuleID,
        alert.Value, alert.Threshold,
        alert.StartedAt.Format(time.RFC3339),
        alert.Status,
        formatMap(alert.Labels),
        formatMap(alert.Annotations),
    )

    msg := fmt.Sprintf("From: %s\r\nTo: %s\r\nSubject: %s\r\n\r\n%s",
        en.from, en.to[0], subject, body)

    // 实际发送需要使用 net/smtp
    log.Printf("[EMAIL NOTIFIER] Would send email:\n%s", msg)
    return nil
}

// WebhookNotifier Webhook 通知器
type WebhookNotifier struct {
    url    string
    secret string
    client *http.Client
}

func NewWebhookNotifier(url, secret string) *WebhookNotifier {
    return &WebhookNotifier{
        url:    url,
        secret: secret,
        client: &http.Client{Timeout: 10 * time.Second},
    }
}

func (wn *WebhookNotifier) Name() string {
    return "webhook"
}

func (wn *WebhookNotifier) Send(ctx context.Context, alert *Alert) error {
    payload := map[string]interface{}{
        "event":     "alert",
        "timestamp": time.Now().Unix(),
        "alert":     alert,
    }

    data, _ := json.Marshal(payload)
    req, err := http.NewRequestWithContext(ctx, "POST", wn.url, bytes.NewReader(data))
    if err != nil {
        return fmt.Errorf("create request: %w", err)
    }

    req.Header.Set("Content-Type", "application/json")
    req.Header.Set("X-Secret", wn.secret)

    resp, err := wn.client.Do(req)
    if err != nil {
        return fmt.Errorf("send webhook: %w", err)
    }
    defer resp.Body.Close()

    if resp.StatusCode >= 300 {
        return fmt.Errorf("webhook returned %d", resp.StatusCode)
    }

    return nil
}

func formatMap(m map[string]string) string {
    if m == nil {
        return "(none)"
    }
    var result string
    for k, v := range m {
        result += fmt.Sprintf("  %s: %s\n", k, v)
    }
    return result
}

---

七、Grafana 仪表盘配置

// grafana/dashboard.json
{
  "dashboard": {
    "title": "Memory System Dashboard",
    "tags": ["memory", "monitoring"],
    "timezone": "browser",
    "panels": [
      {
        "title": "Operations Overview",
        "type": "stat",
        "datasource": "Prometheus",
        "targets": [
          {
            "expr": "rate(memory_total_operations[5m])",
            "legendFormat": "Total Ops/s"
          }
        ],
        "gridPos": {"h": 4, "w": 4, "x": 0, "y": 0}
      },
      {
        "title": "Active Entries",
        "type": "stat",
        "datasource": "Prometheus",
        "targets": [
          {
            "expr": "memory_active_entries",
            "legendFormat": "Entries"
          }
        ],
        "gridPos": {"h": 4, "w": 4, "x": 4, "y": 0}
      },
      {
        "title": "Storage Size",
        "type": "stat",
        "datasource": "Prometheus",
        "targets": [
          {
            "expr": "memory_storage_size_bytes",
            "legendFormat": "Bytes"
          }
        ],
        "gridPos": {"h": 4, "w": 4, "x": 8, "y": 0}
      },
      {
        "title": "Operations by Type",
        "type": "piechart",
        "datasource": "Prometheus",
        "targets": [
          {
            "expr": "sum(rate(memory_total_operations[5m])) by (type)",
            "legendFormat": "{{type}}"
          }
        ],
// grafana/dashboard.json (续)
        "gridPos": {"h": 8, "w": 6, "x": 0, "y": 4}
      },
      {
        "title": "Operation Latency (P99)",
        "type": "graph",
        "datasource": "Prometheus",
        "targets": [
          {
            "expr": "histogram_quantile(0.99, sum(rate(memory_operation_latency_seconds_bucket[5m])) by (le))",
            "legendFormat": "P99"
          },
          {
            "expr": "histogram_quantile(0.90, sum(rate(memory_operation_latency_seconds_bucket[5m])) by (le))",
            "legendFormat": "P90"
          },
          {
            "expr": "histogram_quantile(0.50, sum(rate(memory_operation_latency_seconds_bucket[5m])) by (le))",
            "legendFormat": "P50"
          }
        ],
        "gridPos": {"h": 8, "w": 6, "x": 6, "y": 4}
      },
      {
        "title": "Replication Lag",
        "type": "graph",
        "datasource": "Prometheus",
        "targets": [
          {
            "expr": "memory_replication_lag",
            "legendFormat": "{{node_id}}"
          }
        ],
        "gridPos": {"h": 6, "w": 6, "x": 0, "y": 12}
      },
      {
        "title": "Auth Denied Rate",
        "type": "graph",
        "datasource": "Prometheus",
        "targets": [
          {
            "expr": "rate(memory_auth_denied_total[5m])",
            "legendFormat": "Denied/s"
          }
        ],
        "gridPos": {"h": 6, "w": 6, "x": 6, "y": 12}
      },
      {
        "title": "Encryption Operations",
        "type": "graph",
        "datasource": "Prometheus",
        "targets": [
          {
            "expr": "rate(memory_encryption_operations_total[5m])",
            "legendFormat": "{{operation}}"
          }
        ],
        "gridPos": {"h": 6, "w": 6, "x": 0, "y": 18}
      },
      {
        "title": "System Resources",
        "type": "graph",
        "datasource": "Prometheus",
        "targets": [
          {
            "expr": "system_cpu_usage_percent",
            "legendFormat": "CPU %"
          },
          {
            "expr": "system_memory_usage_bytes",
            "legendFormat": "Memory"
          },
          {
            "expr": "go_goroutines",
            "legendFormat": "Goroutines"
          }
        ],
        "gridPos": {"h": 6, "w": 6, "x": 6, "y": 18}
      }
    ]
  }
}

---

八、Prometheus 告警规则

# prometheus/alerts.yml
groups:
  - name: memory_system_alerts
    interval: 30s
    rules:

      # 高可用告警
      - alert: HighReplicationLag
        expr: memory_replication_lag > 1000
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Replication lag is high on {{ $labels.node_id }}"
          description: "Node {{ $labels.node_id }} has {{ $value }} entries behind the leader"

      - alert: FailoverTriggered
        expr: rate(memory_failover_count[5m]) > 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "Failover has been triggered"
          description: "A failover event occurred in the last 5 minutes"

      - alert: NoLeader
        expr: absent(memory_is_leader)
        for: 10s
        labels:
          severity: critical
        annotations:
          summary: "No cluster leader"
          description: "The cluster has no elected leader"

      # 性能告警
      - alert: HighLatency
        expr: histogram_quantile(0.99, rate(memory_operation_latency_seconds_bucket[5m])) > 0.5
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "High operation latency detected"
          description: "P99 latency is {{ $value }}s (threshold: 0.5s)"

      - alert: OperationErrors
        expr: rate(memory_total_operations{result="error"}[5m]) / rate(memory_total_operations[5m]) > 0.01
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "High error rate"
          description: "Error rate is {{ $value | humanizePercentage }}"

      # 安全告警
      - alert: BruteForceAttack
        expr: rate(memory_auth_denied_total[1m]) > 10
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "Possible brute force attack"
          description: "Auth denied rate is {{ $value }}/s in the last minute"

      - alert: EncryptionKeyExpiring
        expr: memory_encryption_key_expiry_days < 7
        for: 1h
        labels:
          severity: warning
        annotations:
          summary: "Encryption key expiring soon"
          description: "Key {{ $labels.key_id }} expires in {{ $value }} days"

      # 存储告警
      - alert: StorageAlmostFull
        expr: system_disk_usage_percent > 85
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "Disk space running low"
          description: "Disk usage is {{ $value }}%"

      - alert: TooManyEntries
        expr: memory_active_entries > 10000000
        for: 15m
        labels:
          severity: warning
        annotations:
          summary: "Too many memory entries"
          description: "Active entries ({{ $value }}) exceed 10 million, consider compression"

      # 系统告警
      - alert: HighGoroutineCount
        expr: go_goroutines > 10000
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "High goroutine count"
          description: "{{ $value }} goroutines running"

      - alert: ProcessDown
        expr: up{job="memory-system"} == 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "Memory system process is down"
          description: "Instance {{ $labels.instance }} has been down for more than 1 minute"

---

九、生产坑位

#	坑位	现象	解决方案
1	指标基数爆炸​	高基数标签（如 user_id）导致 Prometheus 内存耗尽	限制标签基数；使用聚合指标替代原始标签；分层聚合
2	收集间隔过短​	频繁采集影响业务性能	合理设置收集间隔（≥15s）；使用批处理；异步采集
3	告警风暴​	一个根因触发多条关联告警	使用告警分组；依赖关系分析；抑制规则
4	健康检查误报​	短暂网络抖动导致健康检查失败	增加重试机制；滑动窗口检测；区分存活/就绪检查
5	监控数据丢失​	监控系统自身宕机导致数据断层	本地缓存 + 远程写入；多副本 Prometheus；Thanos 联邦
6	Grafana 面板过多​	数百个面板难以维护	使用模板变量；动态面板；按团队/服务分类
7	告警疲劳​	频繁收到低优先级告警导致忽视	严格分级（Critical/Warning/Info）；静默维护窗口；自动降噪

---

十、评测指标

指标	说明	目标值	测量方式
指标采集延迟	从产生到可查询的延迟	< 30秒	Prometheus 抓取间隔 + 处理时间
告警响应时间	从触发到通知送达	< 1分钟	端到端测试
监控系统可用性	监控组件自身的可用性	>= 99.99%	健康检查
指标存储容量	每天产生的时序数据量	< 100GB/天	实际测量
查询响应时间	Grafana 面板加载时间	< 5秒	实际测量
告警准确率	有效告警占总告警比例	>= 95%	人工审核抽样
误报率	错误触发的告警比例	< 5%	告警回放分析

---

十一、使用示例

// examples/monitoring_demo.go
package main

import (
    "context"
    "fmt"
    "log"
    "math/rand"
    "net/http"
    "time"

    "memory/monitoring"
)

func main() {
    ctx := context.Background()

    fmt.Println("=== 记忆系统监控演示 ===\n")

    // 1. 初始化监控配置
    config := monitoring.MonitoringConfig{
        Enabled:         true,
        ListenAddr:      ":9090",
        MetricsPath:     "/metrics",
        HealthPath:      "/healthz",
        CollectInterval: "15s",
        ExportGoMetrics: true,
        HistogramBuckets: []float64{.005, .01, .025, .05, .1, .25, .5, 1, 2.5, 5},
    }

    // 2. 初始化指标收集器
    collector := monitoring.NewMemoryCollector(config)
    collector.Start()
    defer collector.Stop()
    fmt.Println("✓ 指标收集器已启动")

    // 3. 初始化健康检查器
    healthChecker := monitoring.NewHealthChecker("1.0.0")

    // 注册健康检查
    healthChecker.RegisterCheck("memory_store", func(ctx context.Context) monitoring.ComponentHealth {
        return monitoring.ComponentHealth{
            Status:  "up",
            Message: "in-memory store healthy",
            Latency: "2ms",
        }
    })

    healthChecker.RegisterCheck("encryption", func(ctx context.Context) monitoring.ComponentHealth {
        return monitoring.ComponentHealth{
            Status:  "up",
            Message: "AES-256-GCM encryption healthy",
        }
    })

    healthChecker.RegisterCheck("cluster", func(ctx context.Context) monitoring.ComponentHealth {
        return monitoring.ComponentHealth{
            Status:  "up",
            Message: "leader: node-1",
        }
    })

    fmt.Println("✓ 健康检查器已初始化")

    // 4. 初始化告警管理器
    alertRules := []monitoring.AlertRule{
        {
            ID:        "rule-001",
            Name:      "High Replication Lag",
            Metric:    "memory_replication_lag_max",
            Condition: monitoring.CondGreaterThan,
            Threshold: 1000,
            Duration:  5 * time.Minute,
            Severity:  monitoring.SevWarning,
            Enabled:   true,
        },
        {
            ID:        "rule-002",
            Name:      "High Auth Denied Rate",
            Metric:    "memory_auth_denied_rate",
            Condition: monitoring.CondGreaterThan,
            Threshold: 10,
            Duration:  1 * time.Minute,
            Severity:  monitoring.SevCritical,
            Enabled:   true,
        },
        {
            ID:        "rule-003",
            Name:      "High Latency P99",
            Metric:    "memory_latency_p99",
            Condition: monitoring.CondGreaterThan,
            Threshold: 500,
            Duration:  5 * time.Minute,
            Severity:  monitoring.SevWarning,
            Enabled:   true,
        },
    }

    evaluator := monitoring.NewDefaultAlertEvaluator(5 * time.Minute)
    alertManager := monitoring.NewAlertManager(alertRules, evaluator, 30*time.Second)

    // 注册通知器
    slackNotifier := monitoring.NewSlackNotifier(
        "https://hooks.slack.com/services/xxx",
        "#monitoring",
        "Memory Bot",
    )
    alertManager.RegisterNotifier(slackNotifier)

    webhookNotifier := monitoring.NewWebhookNotifier(
        "https://api.example.com/webhook/alert",
        "shared-secret",
    )
    alertManager.RegisterNotifier(webhookNotifier)

    alertManager.Start(ctx, collector.GetMetrics)
    defer alertManager.Stop()
    fmt.Println("✓ 告警管理器已启动")

    // 5. 启动 HTTP 服务暴露指标和健康检查
    http.Handle(config.MetricsPath, promhttp.Handler())
    http.HandleFunc(config.HealthPath, healthChecker.HTTPHandler)
    http.HandleFunc("/readyz", healthChecker.HTTPHandler)

    go func() {
        addr := config.ListenAddr
        log.Printf("[HTTP] Metrics server listening on %s", addr)
        if err := http.ListenAndServe(addr, nil); err != nil {
            log.Fatalf("http server: %v", err)
        }
    }()

    // 6. 模拟业务操作并记录指标
    fmt.Println("\n=== 模拟业务操作 ===")

    opTypes := []string{"save", "get", "search", "delete"}
    layers := []string{"working", "episodic", "semantic"}
    results := []string{"success", "success", "success", "success", "error"} // 80% success

    for i := 0; i < 20; i++ {
        opType := opTypes[rand.Intn(len(opTypes))]
        layer := layers[rand.Intn(len(layers))]
        result := results[rand.Intn(len(results))]
        latency := time.Duration(rand.Intn(200)+1) * time.Millisecond

        collector.RecordOperation(opType, layer, result, latency)

        // 偶尔记录认证拒绝
        if rand.Float64() < 0.1 {
            collector.RecordAuthDenial("insufficient_permissions")
        }

        // 偶尔记录加密操作
        if rand.Float64() < 0.3 {
            collector.RecordEncryption("encrypt")
        }

        time.Sleep(500 * time.Millisecond)
    }

    // 7. 更新存储指标
    collector.UpdateEntriesCount("working", 1250)
    collector.UpdateEntriesCount("episodic", 8500)
    collector.UpdateEntriesCount("semantic", 3200)
    collector.UpdateStorageSize(157286400) // 150MB

    // 8. 更新复制延迟
    collector.UpdateReplicationLag("node-2", 42)
    collector.UpdateReplicationLag("node-3", 156)

    // 9. 获取当前指标
    fmt.Println("\n=== 当前指标快照 ===")
    metrics := collector.GetMetrics()
    fmt.Printf("总操作数: %d\n", metrics.TotalOps)
    fmt.Printf("成功: %d, 失败: %d\n", metrics.SuccessOps, metrics.FailedOps)
    fmt.Printf("平均延迟: %.2f ms\n", metrics.AvgLatency)
    fmt.Printf("总条目数: %d\n", metrics.TotalEntries)
    fmt.Printf("存储大小: %.2f MB\n", float64(metrics.StorageSize)/1024/1024)
    fmt.Printf("认证拒绝: %d\n", metrics.AuthDeniedCount)
    fmt.Printf("加密操作: %d 加密 / %d 解密\n", metrics.EncryptCount, metrics.DecryptCount)
    fmt.Printf("复制延迟: ")
    for node, lag := range metrics.ReplicationLag {
        fmt.Printf("%s=%d ", node, lag)
    }
    fmt.Println()

    // 10. 健康检查
    fmt.Println("\n=== 健康检查 ===")
    health := healthChecker.Check(ctx)
    fmt.Printf("整体状态: %s\n", health.Status)
    fmt.Printf("运行时间: %s\n", health.Uptime)
    for name, component := range health.Components {
        fmt.Printf("  %s: %s (%s)\n", name, component.Status, component.Message)
    }

    // 11. 查看活跃告警
    fmt.Println("\n=== 活跃告警 ===")
    activeAlerts := alertManager.GetActiveAlerts()
    if len(activeAlerts) == 0 {
        fmt.Println("暂无活跃告警")
    } else {
        for _, alert := range activeAlerts {
            fmt.Printf("  [%s] %s (value=%.2f)\n", alert.Severity, alert.Name, alert.Value)
        }
    }

    fmt.Println("\n=== 访问指标 ===")
    fmt.Printf("Prometheus 指标: http://localhost%s%s\n", config.ListenAddr, config.MetricsPath)
    fmt.Printf("健康检查: http://localhost%s%s\n", config.ListenAddr, config.HealthPath)
    fmt.Printf("就绪检查: http://localhost%s/readyz\n", config.ListenAddr)

    fmt.Println("\n演示完成！")
}

---

十二、本讲总结

┌──────────────────────────────────────────────────────────────────────────┐
│                    第8讲 核心收获                                          │
│                                                                           │
│  ✅ 实现了指标收集器：                                                     │
│     - Prometheus Counter/Gauge/Histogram                                  │
│     - 操作计数、延迟分布、存储量、复制延迟                                │
│     - 系统指标（CPU/内存/磁盘）                                           │
│     - Go Runtime 指标（goroutine/GC）                                    │
│                                                                           │
│  ✅ 实现了健康检查器：                                                     │
│     - 存活检查 (/healthz)                                                 │
│     - 就绪检查 (/readyz)                                                  │
│     - 组件级健康（数据库/存储/集群/加密）                                │
│     - 磁盘空间预警                                                        │
│                                                                           │
│  ✅ 实现了告警管理器：                                                     │
│     - 多维度告警规则（延迟/错误率/复制延迟/安全）                        │
│     - 告警分级（Critical/Warning/Info）                                  │
│     - 静默机制                                                            │
│     - 多通知渠道（Slack/Email/Webhook）                                  │
│                                                                           │
│  ✅ 提供了 Grafana 仪表盘配置                                              │
│  ✅ 提供了 Prometheus 告警规则配置                                         │
│  ✅ 覆盖了 7 个生产坑位与解决方案                                         │
│  ✅ 定义了监控系统的关键评测指标                                          │
│                                                                           │
└──────────────────────────────────────────────────────────────────────────┘

---

十三、下一讲预告

第9讲：《记忆系统的性能优化与基准测试------缓存策略、连接池、批量操作与压测方法论》

你将学到：

• 多级缓存策略（L1/L2/L3）
• 连接池优化（gRPC/数据库）
• 批量操作与管道化
• 读写分离与负载均衡
• 基准测试框架（Benchmark）
• 性能调优实战案例

---

🧰 开发之余的小工具推荐

在监控系统运维中，经常需要处理时间戳转换、持续时间计算和指标数值格式化。zz365.top 的 Unix 时间戳转换工具可以快速将监控日志中的 Unix 时间戳转为可读日期，帮助定位故障时间点。其 JSON 格式化工具则可以美化 Prometheus 告警规则的 JSON 配置，便于审查和调试。所有处理均在浏览器本地完成，不会泄露你的监控配置信息。