DolphinDB边缘计算架构：边缘节点数据预处理

七夜zippoe2026-06-30 8:20

目录

    • 摘要
    • 一、边缘计算概述
      • [1.1 什么是边缘计算](#1.1 什么是边缘计算)
      • [1.2 边缘计算优势](#1.2 边缘计算优势)
      • [1.3 适用场景](#1.3 适用场景)
    • 二、边缘节点部署
      • [2.1 边缘节点架构](#2.1 边缘节点架构)
      • [2.2 边缘节点启动](#2.2 边缘节点启动)
      • [2.3 边缘节点管理](#2.3 边缘节点管理)
    • 三、边缘数据预处理
      • [3.1 数据采集](#3.1 数据采集)
      • [3.2 本地预处理](#3.2 本地预处理)
      • [3.3 本地聚合](#3.3 本地聚合)
    • 四、边缘云协同
      • [4.1 数据同步策略](#4.1 数据同步策略)
      • [4.2 断点续传](#4.2 断点续传)
      • [4.3 双向同步](#4.3 双向同步)
    • 五、边缘智能分析
      • [5.1 本地告警](#5.1 本地告警)
      • [5.2 本地统计](#5.2 本地统计)
      • [5.3 边缘ML推理](#5.3 边缘ML推理)
    • 六、离线运行
      • [6.1 本地缓存](#6.1 本地缓存)
      • [6.2 离线处理](#6.2 离线处理)
      • [6.3 重连恢复](#6.3 重连恢复)
    • 七、实战案例
      • [7.1 边缘计算节点部署](#7.1 边缘计算节点部署)
    • 八、总结
    • 参考资料

摘要

本文深入讲解DolphinDB边缘计算架构。从边缘计算原理到节点部署，从数据预处理到本地分析，从边缘云协同到数据同步，全面介绍边缘计算架构的核心方法。通过丰富的代码示例，帮助读者掌握边缘节点数据预处理的核心技能。

一、边缘计算概述

1.1 什么是边缘计算

边缘计算是在数据源附近进行计算的模式：
#mermaid-svg-qEEZhDatOLD7k9MH{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-qEEZhDatOLD7k9MH .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-qEEZhDatOLD7k9MH .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-qEEZhDatOLD7k9MH .error-icon{fill:#552222;}#mermaid-svg-qEEZhDatOLD7k9MH .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-qEEZhDatOLD7k9MH .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-qEEZhDatOLD7k9MH .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-qEEZhDatOLD7k9MH .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-qEEZhDatOLD7k9MH .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-qEEZhDatOLD7k9MH .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-qEEZhDatOLD7k9MH .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-qEEZhDatOLD7k9MH .marker{fill:#333333;stroke:#333333;}#mermaid-svg-qEEZhDatOLD7k9MH .marker.cross{stroke:#333333;}#mermaid-svg-qEEZhDatOLD7k9MH svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-qEEZhDatOLD7k9MH p{margin:0;}#mermaid-svg-qEEZhDatOLD7k9MH .label{font-family:"trebuchet ms",verdana,arial,sans-serif;color:#333;}#mermaid-svg-qEEZhDatOLD7k9MH .cluster-label text{fill:#333;}#mermaid-svg-qEEZhDatOLD7k9MH .cluster-label span{color:#333;}#mermaid-svg-qEEZhDatOLD7k9MH .cluster-label span p{background-color:transparent;}#mermaid-svg-qEEZhDatOLD7k9MH .label text,#mermaid-svg-qEEZhDatOLD7k9MH span{fill:#333;color:#333;}#mermaid-svg-qEEZhDatOLD7k9MH .node rect,#mermaid-svg-qEEZhDatOLD7k9MH .node circle,#mermaid-svg-qEEZhDatOLD7k9MH .node ellipse,#mermaid-svg-qEEZhDatOLD7k9MH .node polygon,#mermaid-svg-qEEZhDatOLD7k9MH .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-qEEZhDatOLD7k9MH .rough-node .label text,#mermaid-svg-qEEZhDatOLD7k9MH .node .label text,#mermaid-svg-qEEZhDatOLD7k9MH .image-shape .label,#mermaid-svg-qEEZhDatOLD7k9MH .icon-shape .label{text-anchor:middle;}#mermaid-svg-qEEZhDatOLD7k9MH .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-qEEZhDatOLD7k9MH .rough-node .label,#mermaid-svg-qEEZhDatOLD7k9MH .node .label,#mermaid-svg-qEEZhDatOLD7k9MH .image-shape .label,#mermaid-svg-qEEZhDatOLD7k9MH .icon-shape .label{text-align:center;}#mermaid-svg-qEEZhDatOLD7k9MH .node.clickable{cursor:pointer;}#mermaid-svg-qEEZhDatOLD7k9MH .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-qEEZhDatOLD7k9MH .arrowheadPath{fill:#333333;}#mermaid-svg-qEEZhDatOLD7k9MH .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-qEEZhDatOLD7k9MH .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-qEEZhDatOLD7k9MH .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-qEEZhDatOLD7k9MH .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-qEEZhDatOLD7k9MH .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-qEEZhDatOLD7k9MH .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-qEEZhDatOLD7k9MH .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-qEEZhDatOLD7k9MH .cluster text{fill:#333;}#mermaid-svg-qEEZhDatOLD7k9MH .cluster span{color:#333;}#mermaid-svg-qEEZhDatOLD7k9MH div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-qEEZhDatOLD7k9MH .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-qEEZhDatOLD7k9MH rect.text{fill:none;stroke-width:0;}#mermaid-svg-qEEZhDatOLD7k9MH .icon-shape,#mermaid-svg-qEEZhDatOLD7k9MH .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-qEEZhDatOLD7k9MH .icon-shape p,#mermaid-svg-qEEZhDatOLD7k9MH .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-qEEZhDatOLD7k9MH .icon-shape .label rect,#mermaid-svg-qEEZhDatOLD7k9MH .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-qEEZhDatOLD7k9MH .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-qEEZhDatOLD7k9MH .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-qEEZhDatOLD7k9MH :root{--mermaid-font-family:"trebuchet ms",verdana,arial,sans-serif;} 边缘计算架构
设备层
边缘节点
边缘计算
本地处理
云端
中心分析

1.2 边缘计算优势

优势 说明
低延迟 本地处理，响应快
节省带宽 本地预处理，减少传输
数据安全 敏感数据本地处理
离线运行 断网时仍可工作

1.3 适用场景

场景 说明
实时控制 毫秒级响应
数据预处理 本地清洗聚合
本地分析 边缘智能分析
断网运行 离线数据处理

二、边缘节点部署

2.1 边缘节点架构

python 复制代码 
// 边缘节点配置
edgeConfig = dict(STRING, ANY, [
    ["nodeId", "edge_001"],
    ["location", "车间A"],
    ["dataDir", "/data/dolphindb"],
    ["maxMemory", "4G"],
    ["syncInterval", 60000]  // 同步间隔60秒
])

2.2 边缘节点启动

python 复制代码 
// 边缘节点启动脚本
// 配置文件: edge.cfg

// 边缘节点配置
localSite = "edge_001:8848:edge01"
mode = "edge"
maxMemSize = 4
dataSync = 1

2.3 边缘节点管理

python 复制代码 
// 查看边缘节点状态
getClusterPerf()

// 查看节点配置
getConfig()

// 查看资源使用
getMemoryStatus()

三、边缘数据预处理

3.1 数据采集

python 复制代码 
// 边缘数据采集
// 创建本地流表
share streamTable(100000:0, 
    `device_id`timestamp`temperature`humidity`pressure,
    [SYMBOL, TIMESTAMP, DOUBLE, DOUBLE, DOUBLE]) as edge_stream

// 启用持久化
enableTablePersistence(edge_stream, true, true, 1000000)

3.2 本地预处理

python 复制代码 
// 边缘预处理：数据清洗
def edgePreprocess(data) {
    // 缺失值处理
    cleaned = select device_id, timestamp,
                     iif(temperature is null, avg(temperature), temperature) as temperature,
                     iif(humidity is null, avg(humidity), humidity) as humidity,
                     iif(pressure is null, avg(pressure), pressure) as pressure
              from data
    
    // 异常值过滤
    cleaned = select * from cleaned
              where temperature between -40 and 100
              and humidity between 0 and 100
    
    return cleaned
}

// 订阅预处理
subscribeTable(, "edge_stream", "preprocess", -1,
    def(msg) {
        cleaned = edgePreprocess(msg)
        edge_cleaned.append!(cleaned)
    }, true)

3.3 本地聚合

python 复制代码 
// 边缘聚合：降低数据量
share table(1:0, 
    `device_id`time_window`avg_temp`max_temp`min_temp`cnt,
    [SYMBOL, TIMESTAMP, DOUBLE, DOUBLE, DOUBLE, LONG]) as edge_agg

// 时间序列引擎
agg = createTimeSeriesEngine("edge_agg_engine", 60000,
    <[avg(temperature) as avg_temp,
      max(temperature) as max_temp,
      min(temperature) as min_temp,
      count(*) as cnt]>,
    edge_agg, `timestamp, `device_id)

// 订阅
subscribeTable(, "edge_stream", "agg", -1, agg, true)

四、边缘云协同

4.1 数据同步策略

python 复制代码 
// 边缘到云端数据同步
def syncToCloud(edgeData, cloudConn) {
    // 获取未同步数据
    unsynced = select * from edgeData where synced = false
    
    if (unsynced.rows() > 0) {
        // 同步到云端
        cloudTable = loadTable(cloudConn, "sensor_data")
        cloudTable.append!(unsynced)
        
        // 标记已同步
        update edgeData set synced = true where synced = false
    }
    
    return unsynced.rows()
}

4.2 断点续传

python 复制代码 
// 断点续传机制
def resumeSync(syncTable, cloudConn) {
    // 获取最后同步位置
    lastSync = exec max(timestamp) from syncTable where synced = true
    
    if (isNull(lastSync)) {
        lastSync = 1970.01.01
    }
    
    // 获取待同步数据
    pending = select * from syncTable where timestamp > lastSync
    
    // 同步
    if (pending.rows() > 0) {
        loadTable(cloudConn, "sensor_data").append!(pending)
    }
}

4.3 双向同步

python 复制代码 
// 双向同步：云端配置下发
def syncFromCloud(cloudConn, localConfig) {
    // 从云端获取配置
    cloudConfig = select * from loadTable(cloudConn, "edge_config")
                  where edge_id = localConfig.nodeId
    
    // 更新本地配置
    if (cloudConfig.rows() > 0) {
        // 应用配置
        updateLocalConfig(cloudConfig)
    }
}

五、边缘智能分析

5.1 本地告警

python 复制代码 
// 边缘告警检测
share table(1:0, 
    `device_id`timestamp`alert_type`value,
    [SYMBOL, TIMESTAMP, SYMBOL, DOUBLE]) as edge_alerts

// 订阅告警检测
subscribeTable(, "edge_stream", "alert", -1,
    def(msg) {
        alerts = select device_id, timestamp,
                        `temperature_high as alert_type,
                        temperature as value
                 from msg
                 where temperature > 30
        
        if (alerts.rows() > 0) {
            edge_alerts.append!(alerts)
            // 本地告警通知
            notifyAlert(alerts)
        }
    }, true)

5.2 本地统计

python 复制代码 
// 边缘统计分析
def edgeStatistics(data) {
    return select device_id,
                  count(*) as cnt,
                  avg(temperature) as avg_temp,
                  max(temperature) as max_temp,
                  min(temperature) as min_temp,
                  std(temperature) as std_temp
           from data
           group by device_id
}

5.3 边缘ML推理

python 复制代码 
// 边缘ML推理
def edgeInference(data, model) {
    // 使用预训练模型进行推理
    predictions = predict(model, data)
    return predictions
}

六、离线运行

6.1 本地缓存

python 复制代码 
// 本地数据缓存
share table(100000:0, 
    `device_id`timestamp`temperature`humidity`synced,
    [SYMBOL, TIMESTAMP, DOUBLE, DOUBLE, BOOL]) as local_cache

// 写入本地缓存
def writeToLocalCache(data) {
    data[`synced] = false
    local_cache.append!(data)
}

6.2 离线处理

python 复制代码 
// 离线处理队列
share table(1:0, 
    `task_id`task_type`params`status`create_time,
    [STRING, STRING, STRING, STRING, TIMESTAMP]) as offline_queue

// 添加离线任务
def addOfflineTask(taskType, params) {
    insert into offline_queue values (
        "task_" + string(now()),
        taskType,
        params,
        "pending",
        now()
    )
}

6.3 重连恢复

python 复制代码 
// 重连恢复机制
def reconnectAndSync(cloudConn) {
    // 检查连接
    if (not isConnected(cloudConn)) {
        // 重连
        cloudConn = reconnect(cloudConn)
    }
    
    // 同步离线数据
    resumeSync(local_cache, cloudConn)
}

七、实战案例

7.1 边缘计算节点部署

python 复制代码 
// ========== 边缘计算节点部署 ==========

// 1. 创建边缘数据表
share streamTable(100000:0, 
    `device_id`timestamp`temperature`humidity`pressure,
    [SYMBOL, TIMESTAMP, DOUBLE, DOUBLE, DOUBLE]) as edge_stream

// 2. 创建本地存储
db = database("dfs://edge_db", VALUE, 1..100)
schema = table(1:0, `device_id`timestamp`temperature`humidity`pressure,
               [SYMBOL, TIMESTAMP, DOUBLE, DOUBLE, DOUBLE])
db.createPartitionedTable(schema, `sensor_data, `device_id)

// 3. 启用持久化
enableTablePersistence(edge_stream, true, true, 1000000)

// 4. 本地预处理
share streamTable(100000:0, 
    `device_id`timestamp`temperature`humidity`pressure,
    [SYMBOL, TIMESTAMP, DOUBLE, DOUBLE, DOUBLE]) as edge_cleaned

subscribeTable(, "edge_stream", "preprocess", -1,
    def(msg) {
        cleaned = select device_id, timestamp,
                         iif(temperature is null, avg(temperature), temperature) as temperature,
                         iif(humidity is null, avg(humidity), humidity) as humidity,
                         iif(pressure is null, avg(pressure), pressure) as pressure
                  from msg
                  where temperature between -40 and 100
        edge_cleaned.append!(cleaned)
    }, true)

// 5. 本地聚合
share table(1:0, 
    `device_id`time_window`avg_temp`cnt,
    [SYMBOL, TIMESTAMP, DOUBLE, LONG]) as edge_agg

agg = createTimeSeriesEngine("edge_agg", 60000,
    <[avg(temperature) as avg_temp, count(*) as cnt]>,
    edge_agg, `timestamp, `device_id)

subscribeTable(, "edge_cleaned", "agg", -1, agg, true)

// 6. 本地存储
subscribeTable(, "edge_cleaned", "store", -1,
    def(msg) {
        loadTable("dfs://edge_db", "sensor_data").append!(msg)
    }, 10000, 5000)

// 7. 云端同步（定时）
def syncToCloudTask() {
    // 同步聚合数据到云端
    print("同步数据到云端...")
}

scheduleJob("sync_cloud", "云端同步", syncToCloudTask,
            00:01, 2024.01.01, 2030.12.31, 'D')

print("边缘计算节点部署完成")

八、总结

本文详细介绍了DolphinDB边缘计算架构：

  1. 边缘计算原理：低延迟、节省带宽、数据安全
  2. 边缘节点部署：配置、启动、管理
  3. 数据预处理：采集、清洗、聚合
  4. 边缘云协同：数据同步、断点续传、双向同步
  5. 边缘智能：本地告警、统计分析、ML推理
  6. 离线运行：本地缓存、离线处理、重连恢复

思考题：

  1. 边缘计算和云计算如何协同？
  2. 如何设计边缘节点的数据同步策略？
  3. 如何保证边缘节点的可靠性？

参考资料

热门推荐
012026年6月AI大模型全景报告：GPT-5.6、Claude Opus 4.8、Gemini 3.5，中美AI三足鼎立谁主沉浮？022026年6月AI行业全景：从百模大战到Agent元年，这30天发生了什么？032026 年 AI 编程工具终极横评：Cursor vs Claude Code vs Copilot vs Windsurf04【AI】2026 年具身智能模型和世界模型总结05Trae国际版与国内版深度测评：AI原生IDE的双生花06飞书长连接_事件订阅（接收消息，审批任务状态变更）07Claude Code、Codex、Cursor三分天下：2026年AI编程Agent生态全景剖析082026 AI 编程工具终极实战指南：Cursor vs Claude Code vs Copilot，开发者该怎么选？09GitHub 镜像站点102026年AI架构实战：彻底解决OpenAI接口超时与封号，Python调用GPT-5.2/Sora2企业级架构详解（附源码+压测报告）