AI Agent 可观测性实战教程：用 Apache Doris 5 分钟搭建 Agent 监控系统

OpenClaw 暴露近 1000 个未认证实例、512 个漏洞。本教程手把手搭建 AI Observe Stack（OpenTelemetry + Apache Doris + Grafana），实现 Agent 安全审计、成本分析和行为追踪，5 分钟跑通完整链路。

关键词：Apache Doris · AI Agent · 可观测性 · OpenTelemetry · OpenClaw · Grafana · SelectDB · 实战教程

---

前置准备

• Docker 环境
• Apache Doris 2.1+ 或阿里云 SelectDB
• 任意 AI Agent（本教程以 OpenClaw 为例）

Step 1：5 分钟启动 AI Observe Stack

# 克隆项目
git clone https://github.com/velodb/ai-observe-stack.git
cd ai-observe-stack/docker
​
# 一键启动
docker compose up -d
​
# 等待 Doris 就绪（首次约 3 分钟）
docker compose ps
# 确认所有服务 STATUS 为 running，doris 显示 (healthy)

Step 2：创建可观测数据表

-- 连接 Doris
mysql -h 127.0.0.1 -P 9030 -uroot
​
-- Doris App 插件自动创建表，也可手动创建
-- 日志表（VARIANT 类型存储嵌套 JSON）
CREATE TABLE agent_logs (
  log_time DATETIME,
  trace_id VARCHAR(64),
  session_id VARCHAR(64),
   level VARCHAR(16),
  message TEXT,
  log_attributes VARIANT,  -- 嵌套 JSON 原生存储
   INDEX idx_msg(message) USING INVERTED PROPERTIES(
       "parser" = "unicode",
       "support_phrase" = "true"
   ),
   INDEX idx_level(level) USING INVERTED
) ENGINE=OLAP
DUPLICATE KEY(log_time)
DISTRIBUTED BY HASH(trace_id) BUCKETS 10;
​
-- Trace 表
CREATE TABLE agent_traces (
  start_time DATETIME,
  trace_id VARCHAR(64),
  span_id VARCHAR(64),
  parent_span_id VARCHAR(64),
  operation VARCHAR(128),
  service VARCHAR(64),
  duration_ms INT,
  tags VARIANT
) ENGINE=OLAP
DUPLICATE KEY(start_time)
DISTRIBUTED BY HASH(trace_id) BUCKETS 10;
​
-- Metrics 表
CREATE TABLE agent_metrics (
  ts DATETIME,
  metric_name VARCHAR(64),
  metric_value DOUBLE,
  labels VARIANT
) ENGINE=OLAP
DUPLICATE KEY(ts)
DISTRIBUTED BY HASH(metric_name) BUCKETS 5;

Step 3：对接 AI Agent

# 安装 OTel 插件
mkdir -p ~/.openclaw/plugins
cd ~/.openclaw/plugins
git clone https://github.com/henrikrexed/openclaw-observability-plugin otel-observability
cd otel-observability && npm install

配置 OpenTelemetry：

// ~/.openclaw/openclaw.json
{
 "plugins": {
   "load": { "paths": ["~/.openclaw/plugins/otel-observability"] },
   "entries": {
     "otel-observability": {
       "enabled": true,
       "config": {
         "endpoint": "http://127.0.0.1:4318",
         "protocol": "http",
         "serviceName": "openclaw",
         "traces": true,
         "metrics": true
      }
    }
  }
}
}

启动日志采集（filelog 方式）：

docker run -d \
 --name openclaw-log-collector \
 --network docker_aiobs-net \
 -v ~/.openclaw/logs:/openclaw-logs:ro \
 -v ~/.openclaw/agents:/openclaw-agents:ro \
 -v $(pwd)/../examples/openclaw/otel-collector-log-config.yaml:/etc/otelcol-contrib/config.yaml:ro \
otel/opentelemetry-collector-contrib:0.144.0 \
 --config=/etc/otelcol-contrib/config.yaml
​
# 重启 Agent
openclaw gateway restart

Step 4：安全审计 SQL 查询

4.1 检测危险命令

-- 查找所有危险 shell 命令
SELECT log_time, session_id,
      log_attributes['command'] AS command,
      log_attributes['risk_category'] AS risk_category
FROM agent_logs
WHERE search('message:"rm -rf" OR message:"sudo" OR message:"chmod 777"')
 AND log_time > NOW() - INTERVAL 1 HOUR
ORDER BY log_time DESC;
​
-- 风险评分排序
SELECT session_id,
      SUM(CASE WHEN log_attributes['tool'] = 'exec' THEN 3 ELSE 0 END) +
      SUM(CASE WHEN log_attributes['tool'] = 'web' THEN 2 ELSE 0 END) +
      SUM(CASE WHEN log_attributes['tool'] = 'outbound' THEN 5 ELSE 0 END) +
      SUM(CASE WHEN log_attributes['status'] = 'error' THEN 1 ELSE 0 END) +
      SUM(CASE WHEN log_attributes['file_type'] IN ('SSH_KEY','ENV_FILE','CREDENTIALS') THEN 10 ELSE 0 END)
      AS risk_score
FROM agent_logs
WHERE log_time > NOW() - INTERVAL 24 HOUR
GROUP BY session_id
ORDER BY risk_score DESC;

4.2 检测 Prompt Injection

-- 查找 prompt injection 模式
SELECT log_time, session_id,
      log_attributes['injection_type'] AS injection_type,
      log_attributes['source_tool'] AS source_tool,
      message
FROM agent_logs
WHERE search('message:"ignore previous instructions" OR message:"you are now" OR message:"DAN mode"')
ORDER BY log_time DESC;

Step 5：成本分析 SQL 查询

-- Token 消耗趋势（按模型）
SELECT DATE_TRUNC('minute', log_time) AS ts,
      log_attributes['model_name'] AS model,
      SUM(CAST(log_attributes['input_tokens'] AS BIGINT)) AS total_input,
      SUM(CAST(log_attributes['output_tokens'] AS BIGINT)) AS total_output
FROM agent_logs
WHERE log_time > NOW() - INTERVAL 1 HOUR
GROUP BY ts, model
ORDER BY ts;
​
-- Context Window 滚雪球效应
SELECT session_id,
      log_attributes['turn_number'] AS turn,
      CAST(log_attributes['input_tokens'] AS BIGINT) AS input_tokens
FROM agent_logs
WHERE log_attributes['type'] = 'llm_call'
ORDER BY session_id, turn;
-- 可看到每轮调用的 input tokens 持续增长
​
-- 每个问题的成本
SELECT session_id,
      MAX(CAST(log_attributes['turn_number'] AS INT)) AS ai_steps,
      SUM(CAST(log_attributes['input_tokens'] AS BIGINT)) AS total_input,
      SUM(CAST(log_attributes['input_tokens'] AS BIGINT)) * 0.00003 AS cost_usd
FROM agent_logs
WHERE log_attributes['type'] = 'llm_call'
GROUP BY session_id
ORDER BY total_input DESC
LIMIT 20;

Step 6：行为分析 SQL 查询

-- 工具调用分布
SELECT log_attributes['tool_name'] AS tool,
      COUNT(*) AS calls,
      SUM(CASE WHEN log_attributes['status'] = 'ERROR' THEN 1 ELSE 0 END) AS errors
FROM agent_logs
WHERE log_attributes['tool_name'] IS NOT NULL
GROUP BY tool
ORDER BY calls DESC;
​
-- P95 延迟分析
SELECT operation,
      AVG(duration_ms) AS avg_ms,
      PERCENTILE_APPROX(duration_ms, 0.95) AS p95_ms
FROM agent_traces
WHERE start_time > NOW() - INTERVAL 1 HOUR
GROUP BY operation
ORDER BY p95_ms DESC;
​
-- Trace 调用链还原
SELECT span_id, parent_span_id, operation, duration_ms
FROM agent_traces
WHERE trace_id = 'your_trace_id'
ORDER BY start_time;

Step 7：生产环境对接 SelectDB

# 配置连接信息
cp .env.example .env
# 编辑 .env
DORIS_FE_HTTP_ENDPOINT=http://<cluster>.selectdb.com:http_port
DORIS_FE_MYSQL_ENDPOINT=<cluster>.selectdb.com:mysql_port
DORIS_USERNAME=admin
DORIS_PASSWORD=<password>
​
# without-doris 模式（数据直写云端）
docker compose -f docker-compose-without-doris.yaml up -d

部署 Checklist

序号	步骤	验证方式
1	docker compose up -d	docker compose ps 全部 running
2	等待 Doris healthy	~3 分钟
3	安装 OTel 插件	npm install 成功
4	配置 endpoint	openclaw.json 中 endpoint:4318
5	启动日志采集	docker ps 有 log-collector
6	重启 Agent	openclaw gateway restart
7	打开 Grafana	http://localhost:3000
8	验证 Dashboard 有数据	触发一次 Agent 对话后检查

常见问题

Q1：支持哪些 AI Agent 框架？ A：任何支持 OpenTelemetry 的框架------LangChain、AutoGen、CrewAI 均可。

Q2：日志是嵌套 JSON 怎么存？ A：Doris VARIANT 类型原生存储嵌套 JSON，无需预定义 schema，通过 log_attributes['field'] 访问。

Q3：生产环境用本地 Doris 还是 SelectDB？ A：本地 Doris 适合开发测试。生产环境推荐阿里云 SelectDB，免运维、高可用、弹性扩缩容。

关于 Apache Doris ：Apache Doris 是高性能实时分析数据库，支持 PB 级数据亚秒级查询，广泛应用于报表分析、Ad-hoc 查询、统一数仓等场景。SelectDB 是 Apache Doris 的商业化公司，提供企业级支持和云服务。欢迎加入 Doris 社区 交流更多实践。