DolphinDB实时聚合计算：多维度聚合

七夜zippoe2026-07-18 8:13

目录

    • 摘要
    • 一、聚合计算概述
      • [1.1 聚合类型](#1.1 聚合类型)
      • [1.2 聚合函数](#1.2 聚合函数)
      • [1.3 聚合维度](#1.3 聚合维度)
    • 二、基础聚合
      • [2.1 单表聚合](#2.1 单表聚合)
      • [2.2 分组聚合](#2.2 分组聚合)
      • [2.3 条件聚合](#2.3 条件聚合)
    • 三、多维度聚合
      • [3.1 多列分组](#3.1 多列分组)
      • [3.2 Cube聚合](#3.2 Cube聚合)
      • [3.3 Rollup聚合](#3.3 Rollup聚合)
    • 四、层级聚合
      • [4.1 组织层级](#4.1 组织层级)
      • [4.2 时间层级](#4.2 时间层级)
      • [4.3 上卷下钻](#4.3 上卷下钻)
    • 五、实时聚合引擎
      • [5.1 时间序列聚合](#5.1 时间序列聚合)
      • [5.2 多度量聚合](#5.2 多度量聚合)
      • [5.3 自定义聚合](#5.3 自定义聚合)
    • 六、聚合优化
      • [6.1 增量聚合](#6.1 增量聚合)
      • [6.2 并行聚合](#6.2 并行聚合)
      • [6.3 预聚合](#6.3 预聚合)
    • 七、实战案例
      • [7.1 完整实时聚合系统](#7.1 完整实时聚合系统)
    • 八、总结
    • 参考资料

摘要

本文深入讲解DolphinDB实时聚合计算技术。从聚合函数到多维度聚合，从层级聚合到实时汇总，从分组统计到聚合优化，全面介绍实时聚合计算的核心方法。通过丰富的代码示例，帮助读者掌握多维度聚合的核心技能。

一、聚合计算概述

1.1 聚合类型

#mermaid-svg-b98mfiomieni9Lg0{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-b98mfiomieni9Lg0 .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-b98mfiomieni9Lg0 .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-b98mfiomieni9Lg0 .error-icon{fill:#552222;}#mermaid-svg-b98mfiomieni9Lg0 .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-b98mfiomieni9Lg0 .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-b98mfiomieni9Lg0 .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-b98mfiomieni9Lg0 .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-b98mfiomieni9Lg0 .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-b98mfiomieni9Lg0 .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-b98mfiomieni9Lg0 .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-b98mfiomieni9Lg0 .marker{fill:#333333;stroke:#333333;}#mermaid-svg-b98mfiomieni9Lg0 .marker.cross{stroke:#333333;}#mermaid-svg-b98mfiomieni9Lg0 svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-b98mfiomieni9Lg0 p{margin:0;}#mermaid-svg-b98mfiomieni9Lg0 .label{font-family:"trebuchet ms",verdana,arial,sans-serif;color:#333;}#mermaid-svg-b98mfiomieni9Lg0 .cluster-label text{fill:#333;}#mermaid-svg-b98mfiomieni9Lg0 .cluster-label span{color:#333;}#mermaid-svg-b98mfiomieni9Lg0 .cluster-label span p{background-color:transparent;}#mermaid-svg-b98mfiomieni9Lg0 .label text,#mermaid-svg-b98mfiomieni9Lg0 span{fill:#333;color:#333;}#mermaid-svg-b98mfiomieni9Lg0 .node rect,#mermaid-svg-b98mfiomieni9Lg0 .node circle,#mermaid-svg-b98mfiomieni9Lg0 .node ellipse,#mermaid-svg-b98mfiomieni9Lg0 .node polygon,#mermaid-svg-b98mfiomieni9Lg0 .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-b98mfiomieni9Lg0 .rough-node .label text,#mermaid-svg-b98mfiomieni9Lg0 .node .label text,#mermaid-svg-b98mfiomieni9Lg0 .image-shape .label,#mermaid-svg-b98mfiomieni9Lg0 .icon-shape .label{text-anchor:middle;}#mermaid-svg-b98mfiomieni9Lg0 .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-b98mfiomieni9Lg0 .rough-node .label,#mermaid-svg-b98mfiomieni9Lg0 .node .label,#mermaid-svg-b98mfiomieni9Lg0 .image-shape .label,#mermaid-svg-b98mfiomieni9Lg0 .icon-shape .label{text-align:center;}#mermaid-svg-b98mfiomieni9Lg0 .node.clickable{cursor:pointer;}#mermaid-svg-b98mfiomieni9Lg0 .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-b98mfiomieni9Lg0 .arrowheadPath{fill:#333333;}#mermaid-svg-b98mfiomieni9Lg0 .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-b98mfiomieni9Lg0 .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-b98mfiomieni9Lg0 .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-b98mfiomieni9Lg0 .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-b98mfiomieni9Lg0 .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-b98mfiomieni9Lg0 .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-b98mfiomieni9Lg0 .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-b98mfiomieni9Lg0 .cluster text{fill:#333;}#mermaid-svg-b98mfiomieni9Lg0 .cluster span{color:#333;}#mermaid-svg-b98mfiomieni9Lg0 div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-b98mfiomieni9Lg0 .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-b98mfiomieni9Lg0 rect.text{fill:none;stroke-width:0;}#mermaid-svg-b98mfiomieni9Lg0 .icon-shape,#mermaid-svg-b98mfiomieni9Lg0 .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-b98mfiomieni9Lg0 .icon-shape p,#mermaid-svg-b98mfiomieni9Lg0 .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-b98mfiomieni9Lg0 .icon-shape .label rect,#mermaid-svg-b98mfiomieni9Lg0 .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-b98mfiomieni9Lg0 .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-b98mfiomieni9Lg0 .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-b98mfiomieni9Lg0 :root{--mermaid-font-family:"trebuchet ms",verdana,arial,sans-serif;} 聚合计算
单维度聚合
聚合结果
多维度聚合
层级聚合

1.2 聚合函数

函数 说明
sum 求和
avg 平均值
max 最大值
min 最小值
count 计数
std 标准差

1.3 聚合维度

维度 说明
时间维度 按时间聚合
设备维度 按设备聚合
产品维度 按产品聚合
区域维度 按区域聚合

二、基础聚合

2.1 单表聚合

python 复制代码 
// 单表聚合
def basicAggregation(data) {
    return select sum(temperature) as total,
                  avg(temperature) as mean,
                  max(temperature) as max_val,
                  min(temperature) as min_val,
                  count(*) as count,
                  std(temperature) as std_val
           from data
}

2.2 分组聚合

python 复制代码 
// 分组聚合
def groupAggregation(data, groupCol) {
    return select eval(groupCol) as group_key,
                  sum(temperature) as total,
                  avg(temperature) as mean,
                  count(*) as count
           from data
           group by eval(groupCol)
}

2.3 条件聚合

python 复制代码 
// 条件聚合
def conditionalAggregation(data) {
    return select sum(iif(temperature > 25, temperature, 0)) as high_temp_sum,
                  sum(iif(temperature <= 25, temperature, 0)) as low_temp_sum,
                  count(iif(temperature > 25, 1, 0)) as high_count,
                  count(iif(temperature <= 25, 1, 0)) as low_count
           from data
}

三、多维度聚合

3.1 多列分组

python 复制代码 
// 多列分组聚合
def multiDimAggregation(data) {
    return select device_id,
                  bar(timestamp, 1h) as hour,
                  sum(temperature) as total,
                  avg(temperature) as mean,
                  max(temperature) as max_val,
                  min(temperature) as min_val,
                  count(*) as count
           from data
           group by device_id, bar(timestamp, 1h)
}

3.2 Cube聚合

python 复制代码 
// Cube聚合（多维度组合）
def cubeAggregation(data) {
    // 按设备聚合
    byDevice = select device_id, "all" as hour,
                      sum(temperature) as total, avg(temperature) as mean
               from data
               group by device_id
    
    // 按时间聚合
    byHour = select "all" as device_id, bar(timestamp, 1h) as hour,
                    sum(temperature) as total, avg(temperature) as mean
             from data
             group by bar(timestamp, 1h)
    
    // 按设备和时间聚合
    byBoth = select device_id, bar(timestamp, 1h) as hour,
                    sum(temperature) as total, avg(temperature) as mean
             from data
             group by device_id, bar(timestamp, 1h)
    
    // 合并
    return byDevice.union(byHour).union(byBoth)
}

3.3 Rollup聚合

python 复制代码 
// Rollup聚合（层级聚合）
def rollupAggregation(data) {
    // 层级：设备 -> 车间 -> 工厂
    
    // 设备级别
    deviceLevel = select device_id, workshop, factory,
                         sum(temperature) as total
                  from data
                  group by device_id, workshop, factory
    
    // 车间级别
    workshopLevel = select "all" as device_id, workshop, factory,
                           sum(temperature) as total
                    from data
                    group by workshop, factory
    
    // 工厂级别
    factoryLevel = select "all" as device_id, "all" as workshop, factory,
                          sum(temperature) as total
                   from data
                   group by factory
    
    return deviceLevel.union(workshopLevel).union(factoryLevel)
}

四、层级聚合

4.1 组织层级

python 复制代码 
// 组织层级聚合
def hierarchyAggregation(data, hierarchy) {
    results = array(ANY, 0)
    
    for (level in hierarchy) {
        agg = select eval(level) as level_key,
                    sum(temperature) as total,
                    avg(temperature) as mean
             from data
             group by eval(level)
        
        results.append!(agg)
    }
    
    return results
}

4.2 时间层级

python 复制代码 
// 时间层级聚合
def timeHierarchyAggregation(data) {
    // 分钟级
    minute = select bar(timestamp, 1m) as time, avg(temperature) as mean
             from data group by bar(timestamp, 1m)
    
    // 小时级
    hour = select bar(timestamp, 1h) as time, avg(temperature) as mean
           from data group by bar(timestamp, 1h)
    
    // 天级
    day = select date(timestamp) as time, avg(temperature) as mean
          from data group by date(timestamp)
    
    return dict(STRING, ANY, [
        ["minute", minute],
        ["hour", hour],
        ["day", day]
    ])
}

4.3 上卷下钻

python 复制代码 
// 上卷（聚合到更高层级）
def rollup(data, fromLevel, toLevel) {
    return select eval(toLevel) as level,
                  sum(temperature) as total,
                  avg(temperature) as mean
           from data
           group by eval(toLevel)
}

// 下钻（展开到更低层级）
def drilldown(data, fromLevel, toLevel, filter = "") {
    filtered = select * from data where eval(filter)
    
    return select eval(toLevel) as level,
                  sum(temperature) as total,
                  avg(temperature) as mean
           from filtered
           group by eval(toLevel)
}

五、实时聚合引擎

5.1 时间序列聚合

python 复制代码 
// 创建流表
share streamTable(100000:0, 
    `device_id`timestamp`temperature`humidity,
    [SYMBOL, TIMESTAMP, DOUBLE, DOUBLE]) as sensor_stream

// 创建聚合结果表
share table(1:0, 
    `time_window`device_id`avg_temp`max_temp`min_temp`count,
    [TIMESTAMP, SYMBOL, DOUBLE, DOUBLE, DOUBLE, LONG]) as agg_result

// 创建聚合引擎
aggEngine = createTimeSeriesEngine("sensor_agg", 60000,
    <[avg(temperature) as avg_temp,
      max(temperature) as max_temp,
      min(temperature) as min_temp,
      count(*) as count]>,
    agg_result, `timestamp, `device_id)

// 订阅
subscribeTable(, "sensor_stream", "agg", -1, aggEngine, true)

5.2 多度量聚合

python 复制代码 
// 多度量聚合
share table(1:0, 
    `time_window`device_id`avg_temp`avg_humid`max_temp`min_temp,
    [TIMESTAMP, SYMBOL, DOUBLE, DOUBLE, DOUBLE, DOUBLE]) as multi_agg

multiAggEngine = createTimeSeriesEngine("multi_agg", 60000,
    <[avg(temperature) as avg_temp,
      avg(humidity) as avg_humid,
      max(temperature) as max_temp,
      min(temperature) as min_temp]>,
    multi_agg, `timestamp, `device_id)

subscribeTable(, "sensor_stream", "multi_agg", -1, multiAggEngine, true)

5.3 自定义聚合

python 复制代码 
// 自定义聚合函数
def customAgg(data) {
    return dict(STRING, ANY, [
        ["mean", avg(data)],
        ["median", med(data)],
        ["mode", mode(data)],
        ["range", max(data) - min(data)],
        ["iqr", percentile(data, 75) - percentile(data, 25)]
    ])
}

六、聚合优化

6.1 增量聚合

python 复制代码 
// 增量聚合
share dict(STRING, ANY) as aggState

def incrementalAgg(newData) {
    for (row in newData) {
        key = row.device_id
        
        if (not aggState.has(key)) {
            aggState[key] = dict(STRING, ANY, [
                ["sum", 0.0],
                ["count", 0],
                ["max", -infinity],
                ["min", infinity]
            ])
        }
        
        state = aggState[key]
        state["sum"] += row.temperature
        state["count"] += 1
        state["max"] = max(state["max"], row.temperature)
        state["min"] = min(state["min"], row.temperature)
    }
}

6.2 并行聚合

python 复制代码 
// 并行聚合
def parallelAgg(data, numWorkers = 4) {
    results = array(ANY, 0)
    
    // 分区处理
    for (i in 0..numWorkers) {
        partition = select * from data where device_id % numWorkers = i
        results.append!(aggPartition(partition))
    }
    
    // 合并结果
    return mergeAggResults(results)
}

def mergeAggResults(results) {
    totalSum = sum(each(def(r) { r.sum }, results))
    totalCount = sum(each(def(r) { r.count }, results))
    
    return dict(STRING, ANY, [
        ["sum", totalSum],
        ["count", totalCount],
        ["avg", totalSum / totalCount]
    ])
}

6.3 预聚合

python 复制代码 
// 预聚合表
share table(1:0, 
    `device_id`hour`pre_sum`pre_count`pre_max`pre_min,
    [SYMBOL, TIMESTAMP, DOUBLE, LONG, DOUBLE, DOUBLE]) as pre_agg

// 定时预聚合
def preAggregationTask() {
    while (true) {
        now = now()
        hourStart = bar(now, 1h)
        
        // 聚合最近一小时数据
        agg = select device_id,
                    sum(temperature) as pre_sum,
                    count(*) as pre_count,
                    max(temperature) as pre_max,
                    min(temperature) as pre_min
             from sensor_stream
             where timestamp >= hourStart
             group by device_id
        
        pre_agg.append!(agg)
        sleep(3600000)
    }
}

七、实战案例

7.1 完整实时聚合系统

python 复制代码 
// ========== 实时聚合计算系统 ==========

// 1. 创建数据流
share streamTable(100000:0, 
    `device_id`timestamp`temperature`humidity`pressure,
    [SYMBOL, TIMESTAMP, DOUBLE, DOUBLE, DOUBLE]) as sensor_stream

enableTablePersistence(sensor_stream, true, true, 1000000)

// 2. 创建聚合结果表
share table(1:0, 
    `time_window`device_id`avg_temp`avg_humid`max_temp`min_temp`count,
    [TIMESTAMP, SYMBOL, DOUBLE, DOUBLE, DOUBLE, DOUBLE, LONG]) as agg_result

// 3. 创建聚合引擎
aggEngine = createTimeSeriesEngine("sensor_agg", 60000,
    <[avg(temperature) as avg_temp,
      avg(humidity) as avg_humid,
      max(temperature) as max_temp,
      min(temperature) as min_temp,
      count(*) as count]>,
    agg_result, `timestamp, `device_id)

subscribeTable(, "sensor_stream", "agg", -1, aggEngine, true)

// 4. 多维度聚合接口
def getMultiDimAgg(startTime, endTime) {
    t = loadTable("dfs://sensor_db", "sensor_data")
    
    return select device_id,
                  date(timestamp) as date,
                  bar(timestamp, 1h) as hour,
                  avg(temperature) as avg_temp,
                  max(temperature) as max_temp,
                  min(temperature) as min_temp,
                  count(*) as count
           from t
           where timestamp between startTime and endTime
           group by device_id, date(timestamp), bar(timestamp, 1h)
}

addFunctionView(getMultiDimAgg)

// 5. 模拟数据
def generateMockData() {
    while (true) {
        data = table(
            take(1..10, 10) as device_id,
            take(now(), 10) as timestamp,
            rand(20.0..30.0, 10) as temperature,
            rand(40.0..60.0, 10) as humidity,
            rand(1000.0..1020.0, 10) as pressure
        )
        sensor_stream.append!(data)
        sleep(5000)
    }
}

submitJob("mock_data", "模拟数据", generateMockData)

print("实时聚合计算系统启动完成")

八、总结

本文详细介绍了DolphinDB实时聚合计算：

  1. 基础聚合：单表聚合、分组聚合、条件聚合
  2. 多维度聚合：多列分组、Cube聚合、Rollup聚合
  3. 层级聚合：组织层级、时间层级、上卷下钻
  4. 实时聚合引擎：时间序列聚合、多度量聚合、自定义聚合
  5. 聚合优化：增量聚合、并行聚合、预聚合

思考题：

  1. 如何设计高效的多维度聚合？
  2. 如何优化实时聚合性能？
  3. 如何处理聚合中的数据倾斜？

参考资料

相关推荐
踏月的造梦星球11 小时前
DMDPC 学习：架构、部署、运维与调优
运维·数据库·学习·架构
韩楚风11 小时前
【参天引擎】事务生命周期 / MVCC / Undo / ACID / 分布式事务 功能域整体解析
数据库·分布式·mysql·架构·cantian
renhongxia112 小时前
世界模型，是“空中楼阁”还是AGI的“最后一块拼图”？
运维·服务器·数据库·人工智能·算法·agi
程序员无隅12 小时前
Coding Agent 为什么压缩上下文后还能继续工作？上下文模块设计拆解
java·开发语言·数据库
愿做无知一猿12 小时前
Nacos连接MySQL异常？DataGrip竟成救星
数据库·mysql
G.O.G.O.G12 小时前
LeetCode SQL 从入门到精通（MySQL）06（上）
数据库·sql·mysql·leetcode
研究员子楚13 小时前
GEO行业发展标准体系白皮书V2.0-第10卷 · 全球篇：跨国标准协同与全球品牌语义治理框架
数据库·人工智能·microsoft·架构·geo
Omics Pro13 小时前
深度学习多组学互作：组内+组间
数据库·人工智能·深度学习·mysql·搜索引擎·自然语言处理
残*影15 小时前
如何优雅地保存MySQL数据变更历史？
数据库·mysql
热门推荐
012026年7月AI圈大地震：GPT-5.6被政府限制、Claude入驻Slack、Anthropic自研芯片02GitHub 镜像站点03AI科技热点日报 | 2026年07月01日04幻兽帕鲁 - 服务器管理员权限与 GM 命令完全指南05AI 编程 IDE 全景解析 2026：Agent 全面接管开发链路06国内可直接用、免费额度/永久免费的大模型API清单（含 SiliconFlow、火山、阿里、智谱、百度、Kimi、DeepSeek、DMXAPI 等）072026 国产 AI 大模型横评：DeepSeek、通义千问、Kimi、文心一言、星火、豆包谁更能打？082026年AI技术突破与产业落地全景：从GPT-5到多模态智能体的新纪元09Codex 下载安装指南：Windows 和 macOS 官方版下载102026 年 AI 大模型 & AI 编程工具实战全总结