DolphinDB实时聚合计算：多维度聚合

目录

• • 摘要
  • 一、聚合计算概述
  • • [1.1 聚合类型](#1.1 聚合类型)
    • [1.2 聚合函数](#1.2 聚合函数)
    • [1.3 聚合维度](#1.3 聚合维度)
  • 二、基础聚合
  • • [2.1 单表聚合](#2.1 单表聚合)
    • [2.2 分组聚合](#2.2 分组聚合)
    • [2.3 条件聚合](#2.3 条件聚合)
  • 三、多维度聚合
  • • [3.1 多列分组](#3.1 多列分组)
    • [3.2 Cube聚合](#3.2 Cube聚合)
    • [3.3 Rollup聚合](#3.3 Rollup聚合)
  • 四、层级聚合
  • • [4.1 组织层级](#4.1 组织层级)
    • [4.2 时间层级](#4.2 时间层级)
    • [4.3 上卷下钻](#4.3 上卷下钻)
  • 五、实时聚合引擎
  • • [5.1 时间序列聚合](#5.1 时间序列聚合)
    • [5.2 多度量聚合](#5.2 多度量聚合)
    • [5.3 自定义聚合](#5.3 自定义聚合)
  • 六、聚合优化
  • • [6.1 增量聚合](#6.1 增量聚合)
    • [6.2 并行聚合](#6.2 并行聚合)
    • [6.3 预聚合](#6.3 预聚合)
  • 七、实战案例
  • • [7.1 完整实时聚合系统](#7.1 完整实时聚合系统)
  • 八、总结
  • 参考资料

摘要

本文深入讲解DolphinDB实时聚合计算技术。从聚合函数到多维度聚合，从层级聚合到实时汇总，从分组统计到聚合优化，全面介绍实时聚合计算的核心方法。通过丰富的代码示例，帮助读者掌握多维度聚合的核心技能。

---

一、聚合计算概述

1.1 聚合类型

#mermaid-svg-b98mfiomieni9Lg0{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-b98mfiomieni9Lg0 .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-b98mfiomieni9Lg0 .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-b98mfiomieni9Lg0 .error-icon{fill:#552222;}#mermaid-svg-b98mfiomieni9Lg0 .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-b98mfiomieni9Lg0 .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-b98mfiomieni9Lg0 .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-b98mfiomieni9Lg0 .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-b98mfiomieni9Lg0 .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-b98mfiomieni9Lg0 .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-b98mfiomieni9Lg0 .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-b98mfiomieni9Lg0 .marker{fill:#333333;stroke:#333333;}#mermaid-svg-b98mfiomieni9Lg0 .marker.cross{stroke:#333333;}#mermaid-svg-b98mfiomieni9Lg0 svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-b98mfiomieni9Lg0 p{margin:0;}#mermaid-svg-b98mfiomieni9Lg0 .label{font-family:"trebuchet ms",verdana,arial,sans-serif;color:#333;}#mermaid-svg-b98mfiomieni9Lg0 .cluster-label text{fill:#333;}#mermaid-svg-b98mfiomieni9Lg0 .cluster-label span{color:#333;}#mermaid-svg-b98mfiomieni9Lg0 .cluster-label span p{background-color:transparent;}#mermaid-svg-b98mfiomieni9Lg0 .label text,#mermaid-svg-b98mfiomieni9Lg0 span{fill:#333;color:#333;}#mermaid-svg-b98mfiomieni9Lg0 .node rect,#mermaid-svg-b98mfiomieni9Lg0 .node circle,#mermaid-svg-b98mfiomieni9Lg0 .node ellipse,#mermaid-svg-b98mfiomieni9Lg0 .node polygon,#mermaid-svg-b98mfiomieni9Lg0 .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-b98mfiomieni9Lg0 .rough-node .label text,#mermaid-svg-b98mfiomieni9Lg0 .node .label text,#mermaid-svg-b98mfiomieni9Lg0 .image-shape .label,#mermaid-svg-b98mfiomieni9Lg0 .icon-shape .label{text-anchor:middle;}#mermaid-svg-b98mfiomieni9Lg0 .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-b98mfiomieni9Lg0 .rough-node .label,#mermaid-svg-b98mfiomieni9Lg0 .node .label,#mermaid-svg-b98mfiomieni9Lg0 .image-shape .label,#mermaid-svg-b98mfiomieni9Lg0 .icon-shape .label{text-align:center;}#mermaid-svg-b98mfiomieni9Lg0 .node.clickable{cursor:pointer;}#mermaid-svg-b98mfiomieni9Lg0 .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-b98mfiomieni9Lg0 .arrowheadPath{fill:#333333;}#mermaid-svg-b98mfiomieni9Lg0 .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-b98mfiomieni9Lg0 .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-b98mfiomieni9Lg0 .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-b98mfiomieni9Lg0 .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-b98mfiomieni9Lg0 .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-b98mfiomieni9Lg0 .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-b98mfiomieni9Lg0 .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-b98mfiomieni9Lg0 .cluster text{fill:#333;}#mermaid-svg-b98mfiomieni9Lg0 .cluster span{color:#333;}#mermaid-svg-b98mfiomieni9Lg0 div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-b98mfiomieni9Lg0 .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-b98mfiomieni9Lg0 rect.text{fill:none;stroke-width:0;}#mermaid-svg-b98mfiomieni9Lg0 .icon-shape,#mermaid-svg-b98mfiomieni9Lg0 .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-b98mfiomieni9Lg0 .icon-shape p,#mermaid-svg-b98mfiomieni9Lg0 .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-b98mfiomieni9Lg0 .icon-shape .label rect,#mermaid-svg-b98mfiomieni9Lg0 .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-b98mfiomieni9Lg0 .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-b98mfiomieni9Lg0 .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-b98mfiomieni9Lg0 :root{--mermaid-font-family:"trebuchet ms",verdana,arial,sans-serif;} 聚合计算
单维度聚合
聚合结果
多维度聚合
层级聚合

1.2 聚合函数

函数	说明
sum	求和
avg	平均值
max	最大值
min	最小值
count	计数
std	标准差

1.3 聚合维度

维度	说明
时间维度	按时间聚合
设备维度	按设备聚合
产品维度	按产品聚合
区域维度	按区域聚合

---

二、基础聚合

2.1 单表聚合

// 单表聚合
def basicAggregation(data) {
    return select sum(temperature) as total,
                  avg(temperature) as mean,
                  max(temperature) as max_val,
                  min(temperature) as min_val,
                  count(*) as count,
                  std(temperature) as std_val
           from data
}

2.2 分组聚合

// 分组聚合
def groupAggregation(data, groupCol) {
    return select eval(groupCol) as group_key,
                  sum(temperature) as total,
                  avg(temperature) as mean,
                  count(*) as count
           from data
           group by eval(groupCol)
}

2.3 条件聚合

// 条件聚合
def conditionalAggregation(data) {
    return select sum(iif(temperature > 25, temperature, 0)) as high_temp_sum,
                  sum(iif(temperature <= 25, temperature, 0)) as low_temp_sum,
                  count(iif(temperature > 25, 1, 0)) as high_count,
                  count(iif(temperature <= 25, 1, 0)) as low_count
           from data
}

---

三、多维度聚合

3.1 多列分组

// 多列分组聚合
def multiDimAggregation(data) {
    return select device_id,
                  bar(timestamp, 1h) as hour,
                  sum(temperature) as total,
                  avg(temperature) as mean,
                  max(temperature) as max_val,
                  min(temperature) as min_val,
                  count(*) as count
           from data
           group by device_id, bar(timestamp, 1h)
}

3.2 Cube聚合

// Cube聚合（多维度组合）
def cubeAggregation(data) {
    // 按设备聚合
    byDevice = select device_id, "all" as hour,
                      sum(temperature) as total, avg(temperature) as mean
               from data
               group by device_id
    
    // 按时间聚合
    byHour = select "all" as device_id, bar(timestamp, 1h) as hour,
                    sum(temperature) as total, avg(temperature) as mean
             from data
             group by bar(timestamp, 1h)
    
    // 按设备和时间聚合
    byBoth = select device_id, bar(timestamp, 1h) as hour,
                    sum(temperature) as total, avg(temperature) as mean
             from data
             group by device_id, bar(timestamp, 1h)
    
    // 合并
    return byDevice.union(byHour).union(byBoth)
}

3.3 Rollup聚合

// Rollup聚合（层级聚合）
def rollupAggregation(data) {
    // 层级：设备 -> 车间 -> 工厂
    
    // 设备级别
    deviceLevel = select device_id, workshop, factory,
                         sum(temperature) as total
                  from data
                  group by device_id, workshop, factory
    
    // 车间级别
    workshopLevel = select "all" as device_id, workshop, factory,
                           sum(temperature) as total
                    from data
                    group by workshop, factory
    
    // 工厂级别
    factoryLevel = select "all" as device_id, "all" as workshop, factory,
                          sum(temperature) as total
                   from data
                   group by factory
    
    return deviceLevel.union(workshopLevel).union(factoryLevel)
}

---

四、层级聚合

4.1 组织层级

// 组织层级聚合
def hierarchyAggregation(data, hierarchy) {
    results = array(ANY, 0)
    
    for (level in hierarchy) {
        agg = select eval(level) as level_key,
                    sum(temperature) as total,
                    avg(temperature) as mean
             from data
             group by eval(level)
        
        results.append!(agg)
    }
    
    return results
}

4.2 时间层级

// 时间层级聚合
def timeHierarchyAggregation(data) {
    // 分钟级
    minute = select bar(timestamp, 1m) as time, avg(temperature) as mean
             from data group by bar(timestamp, 1m)
    
    // 小时级
    hour = select bar(timestamp, 1h) as time, avg(temperature) as mean
           from data group by bar(timestamp, 1h)
    
    // 天级
    day = select date(timestamp) as time, avg(temperature) as mean
          from data group by date(timestamp)
    
    return dict(STRING, ANY, [
        ["minute", minute],
        ["hour", hour],
        ["day", day]
    ])
}

4.3 上卷下钻

// 上卷（聚合到更高层级）
def rollup(data, fromLevel, toLevel) {
    return select eval(toLevel) as level,
                  sum(temperature) as total,
                  avg(temperature) as mean
           from data
           group by eval(toLevel)
}

// 下钻（展开到更低层级）
def drilldown(data, fromLevel, toLevel, filter = "") {
    filtered = select * from data where eval(filter)
    
    return select eval(toLevel) as level,
                  sum(temperature) as total,
                  avg(temperature) as mean
           from filtered
           group by eval(toLevel)
}

---

五、实时聚合引擎

5.1 时间序列聚合

// 创建流表
share streamTable(100000:0, 
    `device_id`timestamp`temperature`humidity,
    [SYMBOL, TIMESTAMP, DOUBLE, DOUBLE]) as sensor_stream

// 创建聚合结果表
share table(1:0, 
    `time_window`device_id`avg_temp`max_temp`min_temp`count,
    [TIMESTAMP, SYMBOL, DOUBLE, DOUBLE, DOUBLE, LONG]) as agg_result

// 创建聚合引擎
aggEngine = createTimeSeriesEngine("sensor_agg", 60000,
    <[avg(temperature) as avg_temp,
      max(temperature) as max_temp,
      min(temperature) as min_temp,
      count(*) as count]>,
    agg_result, `timestamp, `device_id)

// 订阅
subscribeTable(, "sensor_stream", "agg", -1, aggEngine, true)

5.2 多度量聚合

// 多度量聚合
share table(1:0, 
    `time_window`device_id`avg_temp`avg_humid`max_temp`min_temp,
    [TIMESTAMP, SYMBOL, DOUBLE, DOUBLE, DOUBLE, DOUBLE]) as multi_agg

multiAggEngine = createTimeSeriesEngine("multi_agg", 60000,
    <[avg(temperature) as avg_temp,
      avg(humidity) as avg_humid,
      max(temperature) as max_temp,
      min(temperature) as min_temp]>,
    multi_agg, `timestamp, `device_id)

subscribeTable(, "sensor_stream", "multi_agg", -1, multiAggEngine, true)

5.3 自定义聚合

// 自定义聚合函数
def customAgg(data) {
    return dict(STRING, ANY, [
        ["mean", avg(data)],
        ["median", med(data)],
        ["mode", mode(data)],
        ["range", max(data) - min(data)],
        ["iqr", percentile(data, 75) - percentile(data, 25)]
    ])
}

---

六、聚合优化

6.1 增量聚合

// 增量聚合
share dict(STRING, ANY) as aggState

def incrementalAgg(newData) {
    for (row in newData) {
        key = row.device_id
        
        if (not aggState.has(key)) {
            aggState[key] = dict(STRING, ANY, [
                ["sum", 0.0],
                ["count", 0],
                ["max", -infinity],
                ["min", infinity]
            ])
        }
        
        state = aggState[key]
        state["sum"] += row.temperature
        state["count"] += 1
        state["max"] = max(state["max"], row.temperature)
        state["min"] = min(state["min"], row.temperature)
    }
}

6.2 并行聚合

// 并行聚合
def parallelAgg(data, numWorkers = 4) {
    results = array(ANY, 0)
    
    // 分区处理
    for (i in 0..numWorkers) {
        partition = select * from data where device_id % numWorkers = i
        results.append!(aggPartition(partition))
    }
    
    // 合并结果
    return mergeAggResults(results)
}

def mergeAggResults(results) {
    totalSum = sum(each(def(r) { r.sum }, results))
    totalCount = sum(each(def(r) { r.count }, results))
    
    return dict(STRING, ANY, [
        ["sum", totalSum],
        ["count", totalCount],
        ["avg", totalSum / totalCount]
    ])
}

6.3 预聚合

// 预聚合表
share table(1:0, 
    `device_id`hour`pre_sum`pre_count`pre_max`pre_min,
    [SYMBOL, TIMESTAMP, DOUBLE, LONG, DOUBLE, DOUBLE]) as pre_agg

// 定时预聚合
def preAggregationTask() {
    while (true) {
        now = now()
        hourStart = bar(now, 1h)
        
        // 聚合最近一小时数据
        agg = select device_id,
                    sum(temperature) as pre_sum,
                    count(*) as pre_count,
                    max(temperature) as pre_max,
                    min(temperature) as pre_min
             from sensor_stream
             where timestamp >= hourStart
             group by device_id
        
        pre_agg.append!(agg)
        sleep(3600000)
    }
}

---

七、实战案例

7.1 完整实时聚合系统

// ========== 实时聚合计算系统 ==========

// 1. 创建数据流
share streamTable(100000:0, 
    `device_id`timestamp`temperature`humidity`pressure,
    [SYMBOL, TIMESTAMP, DOUBLE, DOUBLE, DOUBLE]) as sensor_stream

enableTablePersistence(sensor_stream, true, true, 1000000)

// 2. 创建聚合结果表
share table(1:0, 
    `time_window`device_id`avg_temp`avg_humid`max_temp`min_temp`count,
    [TIMESTAMP, SYMBOL, DOUBLE, DOUBLE, DOUBLE, DOUBLE, LONG]) as agg_result

// 3. 创建聚合引擎
aggEngine = createTimeSeriesEngine("sensor_agg", 60000,
    <[avg(temperature) as avg_temp,
      avg(humidity) as avg_humid,
      max(temperature) as max_temp,
      min(temperature) as min_temp,
      count(*) as count]>,
    agg_result, `timestamp, `device_id)

subscribeTable(, "sensor_stream", "agg", -1, aggEngine, true)

// 4. 多维度聚合接口
def getMultiDimAgg(startTime, endTime) {
    t = loadTable("dfs://sensor_db", "sensor_data")
    
    return select device_id,
                  date(timestamp) as date,
                  bar(timestamp, 1h) as hour,
                  avg(temperature) as avg_temp,
                  max(temperature) as max_temp,
                  min(temperature) as min_temp,
                  count(*) as count
           from t
           where timestamp between startTime and endTime
           group by device_id, date(timestamp), bar(timestamp, 1h)
}

addFunctionView(getMultiDimAgg)

// 5. 模拟数据
def generateMockData() {
    while (true) {
        data = table(
            take(1..10, 10) as device_id,
            take(now(), 10) as timestamp,
            rand(20.0..30.0, 10) as temperature,
            rand(40.0..60.0, 10) as humidity,
            rand(1000.0..1020.0, 10) as pressure
        )
        sensor_stream.append!(data)
        sleep(5000)
    }
}

submitJob("mock_data", "模拟数据", generateMockData)

print("实时聚合计算系统启动完成")

---

八、总结

本文详细介绍了DolphinDB实时聚合计算：

1. 基础聚合：单表聚合、分组聚合、条件聚合
2. 多维度聚合：多列分组、Cube聚合、Rollup聚合
3. 层级聚合：组织层级、时间层级、上卷下钻
4. 实时聚合引擎：时间序列聚合、多度量聚合、自定义聚合
5. 聚合优化：增量聚合、并行聚合、预聚合

思考题：

1. 如何设计高效的多维度聚合？
2. 如何优化实时聚合性能？
3. 如何处理聚合中的数据倾斜？

---

参考资料

• DolphinDB聚合函数
• DolphinDB时间序列引擎

---