目录
-
- 摘要
- 一、告警系统概述
-
- [1.1 告警系统架构](#1.1 告警系统架构)
- [1.2 告警类型](#1.2 告警类型)
- [1.3 告警级别](#1.3 告警级别)
- 二、告警规则定义
-
- [2.1 告警规则表](#2.1 告警规则表)
- [2.2 规则配置接口](#2.2 规则配置接口)
- 三、阈值检测
-
- [3.1 单阈值检测](#3.1 单阈值检测)
- [3.2 区间阈值检测](#3.2 区间阈值检测)
- [3.3 动态阈值检测](#3.3 动态阈值检测)
- 四、告警触发
-
- [4.1 告警记录表](#4.1 告警记录表)
- [4.2 告警触发函数](#4.2 告警触发函数)
- [4.3 实时告警检测](#4.3 实时告警检测)
- 五、告警抑制
-
- [5.1 时间窗口抑制](#5.1 时间窗口抑制)
- [5.2 告警聚合](#5.2 告警聚合)
- [5.3 告警降噪](#5.3 告警降噪)
- 六、告警通知
-
- [6.1 通知渠道](#6.1 通知渠道)
- [6.2 通知发送](#6.2 通知发送)
- [6.3 告警升级](#6.3 告警升级)
- 七、告警闭环
-
- [7.1 告警确认](#7.1 告警确认)
- [7.2 告警处理](#7.2 告警处理)
- [7.3 告警统计](#7.3 告警统计)
- 八、实战案例
-
- [8.1 完整告警系统](#8.1 完整告警系统)
- 九、总结
- 参考资料
摘要
本文深入讲解DolphinDB实时告警系统技术。从告警规则定义到阈值检测,从告警触发到告警通知,从告警抑制到告警闭环,全面介绍实时告警系统的核心方法。通过丰富的代码示例,帮助读者掌握阈值告警设计的核心技能。
一、告警系统概述
1.1 告警系统架构
#mermaid-svg-2UkTWu9WbE4tnw7I{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-2UkTWu9WbE4tnw7I .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-2UkTWu9WbE4tnw7I .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-2UkTWu9WbE4tnw7I .error-icon{fill:#552222;}#mermaid-svg-2UkTWu9WbE4tnw7I .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-2UkTWu9WbE4tnw7I .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-2UkTWu9WbE4tnw7I .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-2UkTWu9WbE4tnw7I .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-2UkTWu9WbE4tnw7I .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-2UkTWu9WbE4tnw7I .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-2UkTWu9WbE4tnw7I .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-2UkTWu9WbE4tnw7I .marker{fill:#333333;stroke:#333333;}#mermaid-svg-2UkTWu9WbE4tnw7I .marker.cross{stroke:#333333;}#mermaid-svg-2UkTWu9WbE4tnw7I svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-2UkTWu9WbE4tnw7I p{margin:0;}#mermaid-svg-2UkTWu9WbE4tnw7I .label{font-family:"trebuchet ms",verdana,arial,sans-serif;color:#333;}#mermaid-svg-2UkTWu9WbE4tnw7I .cluster-label text{fill:#333;}#mermaid-svg-2UkTWu9WbE4tnw7I .cluster-label span{color:#333;}#mermaid-svg-2UkTWu9WbE4tnw7I .cluster-label span p{background-color:transparent;}#mermaid-svg-2UkTWu9WbE4tnw7I .label text,#mermaid-svg-2UkTWu9WbE4tnw7I span{fill:#333;color:#333;}#mermaid-svg-2UkTWu9WbE4tnw7I .node rect,#mermaid-svg-2UkTWu9WbE4tnw7I .node circle,#mermaid-svg-2UkTWu9WbE4tnw7I .node ellipse,#mermaid-svg-2UkTWu9WbE4tnw7I .node polygon,#mermaid-svg-2UkTWu9WbE4tnw7I .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-2UkTWu9WbE4tnw7I .rough-node .label text,#mermaid-svg-2UkTWu9WbE4tnw7I .node .label text,#mermaid-svg-2UkTWu9WbE4tnw7I .image-shape .label,#mermaid-svg-2UkTWu9WbE4tnw7I .icon-shape .label{text-anchor:middle;}#mermaid-svg-2UkTWu9WbE4tnw7I .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-2UkTWu9WbE4tnw7I .rough-node .label,#mermaid-svg-2UkTWu9WbE4tnw7I .node .label,#mermaid-svg-2UkTWu9WbE4tnw7I .image-shape .label,#mermaid-svg-2UkTWu9WbE4tnw7I .icon-shape .label{text-align:center;}#mermaid-svg-2UkTWu9WbE4tnw7I .node.clickable{cursor:pointer;}#mermaid-svg-2UkTWu9WbE4tnw7I .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-2UkTWu9WbE4tnw7I .arrowheadPath{fill:#333333;}#mermaid-svg-2UkTWu9WbE4tnw7I .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-2UkTWu9WbE4tnw7I .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-2UkTWu9WbE4tnw7I .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-2UkTWu9WbE4tnw7I .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-2UkTWu9WbE4tnw7I .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-2UkTWu9WbE4tnw7I .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-2UkTWu9WbE4tnw7I .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-2UkTWu9WbE4tnw7I .cluster text{fill:#333;}#mermaid-svg-2UkTWu9WbE4tnw7I .cluster span{color:#333;}#mermaid-svg-2UkTWu9WbE4tnw7I div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-2UkTWu9WbE4tnw7I .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-2UkTWu9WbE4tnw7I rect.text{fill:none;stroke-width:0;}#mermaid-svg-2UkTWu9WbE4tnw7I .icon-shape,#mermaid-svg-2UkTWu9WbE4tnw7I .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-2UkTWu9WbE4tnw7I .icon-shape p,#mermaid-svg-2UkTWu9WbE4tnw7I .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-2UkTWu9WbE4tnw7I .icon-shape .label rect,#mermaid-svg-2UkTWu9WbE4tnw7I .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-2UkTWu9WbE4tnw7I .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-2UkTWu9WbE4tnw7I .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-2UkTWu9WbE4tnw7I :root{--mermaid-font-family:"trebuchet ms",verdana,arial,sans-serif;} 告警系统架构
数据流
规则引擎
告警触发
告警处理
告警通知
1.2 告警类型
| 类型 | 说明 |
|---|---|
| 阈值告警 | 超过设定阈值 |
| 趋势告警 | 数据趋势异常 |
| 状态告警 | 状态变更告警 |
| 复合告警 | 多条件组合 |
1.3 告警级别
| 级别 | 说明 | 响应时间 |
|---|---|---|
| 紧急 | 严重故障 | 立即 |
| 严重 | 重要异常 | 15分钟 |
| 警告 | 一般异常 | 1小时 |
| 提示 | 轻微异常 | 4小时 |
二、告警规则定义
2.1 告警规则表
python
// 告警规则表
share table(1:0,
`rule_id`rule_name`device_id`metric`operator`threshold`level`enabled,
[STRING, STRING, SYMBOL, STRING, STRING, DOUBLE, INT, BOOL]) as alert_rules
// 初始化规则
insert into alert_rules values (
"temp_high", "温度过高", `all, "temperature", ">", 80, 1, true
)
insert into alert_rules values (
"temp_low", "温度过低", `all, "temperature", "<", 10, 2, true
)
insert into alert_rules values (
"humidity_high", "湿度过高", `all, "humidity", ">", 90, 2, true
)
insert into alert_rules values (
"pressure_high", "压力过高", `all, "pressure", ">", 1100, 1, true
)
2.2 规则配置接口
python
// 添加告警规则
def addAlertRule(ruleId, ruleName, deviceId, metric, operator, threshold, level) {
insert into alert_rules values (
ruleId, ruleName, deviceId, metric, operator, threshold, level, true
)
}
// 更新告警规则
def updateAlertRule(ruleId, threshold, level) {
update alert_rules
set threshold = threshold, level = level
where rule_id = ruleId
}
// 禁用告警规则
def disableAlertRule(ruleId) {
update alert_rules set enabled = false where rule_id = ruleId
}
// 启用告警规则
def enableAlertRule(ruleId) {
update alert_rules set enabled = true where rule_id = ruleId
}
三、阈值检测
3.1 单阈值检测
python
// 单阈值检测
def checkThreshold(value, operator, threshold) {
if (operator == ">") {
return value > threshold
} else if (operator == ">=") {
return value >= threshold
} else if (operator == "<") {
return value < threshold
} else if (operator == "<=") {
return value <= threshold
} else if (operator == "==") {
return value == threshold
} else if (operator == "!=") {
return value != threshold
}
return false
}
3.2 区间阈值检测
python
// 区间阈值检测
def checkRange(value, lower, upper) {
return value < lower or value > upper
}
// 带警告区间的检测
def checkRangeWithWarning(value, normalLower, normalUpper, warnLower, warnUpper) {
if (value < warnLower or value > warnUpper) {
return "critical"
} else if (value < normalLower or value > normalUpper) {
return "warning"
}
return "normal"
}
3.3 动态阈值检测
python
// 动态阈值(基于历史数据)
def calculateDynamicThreshold(deviceId, metric, multiplier = 3) {
// 获取历史数据
data = select eval(metric) as value
from sensor_data
where device_id = deviceId
and timestamp > now() - 86400000 * 7 // 最近7天
// 计算统计量
mean = avg(data.value)
std = std(data.value)
return dict(STRING, ANY, [
["upper", mean + multiplier * std],
["lower", mean - multiplier * std],
["mean", mean],
["std", std]
])
}
四、告警触发
4.1 告警记录表
python
// 告警记录表
share table(1:0,
`alert_id`rule_id`device_id`alert_time`metric`value`threshold`level`status,
[STRING, STRING, SYMBOL, TIMESTAMP, STRING, DOUBLE, DOUBLE, INT, STRING]) as alert_log
// 告警计数器
alertCounter = 0
def generateAlertId() {
alertCounter += 1
return "ALT" + format(now(), "yyyyMMddHHmmss") + string(alertCounter)
}
4.2 告警触发函数
python
// 触发告警
def triggerAlert(ruleId, deviceId, metric, value, threshold, level) {
alertId = generateAlertId()
insert into alert_log values (
alertId, ruleId, deviceId, now(), metric, value, threshold, level, "new"
)
// 发送通知
sendAlertNotification(alertId, deviceId, metric, value, level)
return alertId
}
4.3 实时告警检测
python
// 实时告警检测
def detectAlerts(data) {
for (row in data) {
// 获取适用规则
rules = select * from alert_rules
where (device_id = row.device_id or device_id = `all)
and enabled = true
for (rule in rules) {
value = row[rule.metric]
if (checkThreshold(value, rule.operator, rule.threshold)) {
triggerAlert(rule.rule_id, row.device_id, rule.metric,
value, rule.threshold, rule.level)
}
}
}
}
// 订阅检测
subscribeTable(, "sensor_stream", "alert_detect", -1,
def(msg) { detectAlerts(msg) }, true)
五、告警抑制
5.1 时间窗口抑制
python
// 告警抑制配置
share table(1:0,
`rule_id`device_id`suppress_seconds,
[STRING, SYMBOL, INT]) as alert_suppress
// 检查是否抑制
def shouldSuppress(ruleId, deviceId) {
// 查找最近告警
recent = select * from alert_log
where rule_id = ruleId
and device_id = deviceId
and alert_time > now() - 300000 // 5分钟内
order by alert_time desc
limit 1
if (recent.rows() > 0) {
// 获取抑制时间
suppress = select suppress_seconds from alert_suppress
where rule_id = ruleId and device_id = deviceId
if (suppress.rows() > 0) {
suppressSeconds = suppress.suppress_seconds[0]
lastTime = recent.alert_time[0]
if (now() - lastTime < suppressSeconds * 1000) {
return true
}
}
}
return false
}
5.2 告警聚合
python
// 告警聚合
def aggregateAlerts(timeWindow = 300000) {
return select rule_id, device_id,
count(*) as alert_count,
min(alert_time) as first_alert,
max(alert_time) as last_alert,
avg(value) as avg_value
from alert_log
where alert_time > now() - timeWindow
and status = "new"
group by rule_id, device_id
}
5.3 告警降噪
python
// 告警降噪规则
def filterNoise(alerts) {
// 过滤短时间内重复告警
filtered = select * from alerts
where not shouldSuppress(rule_id, device_id)
return filtered
}
六、告警通知
6.1 通知渠道
python
// 通知渠道配置
share table(1:0,
`channel_id`channel_type`config`enabled,
[STRING, STRING, STRING, BOOL]) as notification_channel
// 初始化渠道
insert into notification_channel values (
"email", "email", "smtp://smtp.company.com", true
)
insert into notification_channel values (
"sms", "sms", "api://sms.company.com", true
)
insert into notification_channel values (
"wechat", "wechat", "api://wechat.company.com", true
)
6.2 通知发送
python
// 发送告警通知
def sendAlertNotification(alertId, deviceId, metric, value, level) {
// 构建消息
message = "告警: 设备" + deviceId + " " + metric + " = " + string(value) +
" (级别: " + string(level) + ")"
// 根据级别选择通知渠道
if (level == 1) {
// 紧急:所有渠道
sendEmail(message)
sendSms(message)
sendWechat(message)
} else if (level == 2) {
// 严重:邮件+微信
sendEmail(message)
sendWechat(message)
} else {
// 其他:仅邮件
sendEmail(message)
}
}
def sendEmail(message) {
print("发送邮件: " + message)
}
def sendSms(message) {
print("发送短信: " + message)
}
def sendWechat(message) {
print("发送微信: " + message)
}
6.3 告警升级
python
// 告警升级规则
def escalateAlert(alertId, escalateMinutes = 30) {
alert = select * from alert_log where alert_id = alertId
if (alert.rows() > 0 and alert.status[0] == "new") {
alertTime = alert.alert_time[0]
if (now() - alertTime > escalateMinutes * 60000) {
// 升级告警
update alert_log set level = level - 1 where alert_id = alertId
// 重新发送通知
sendAlertNotification(alertId, alert.device_id[0],
alert.metric[0], alert.value[0], alert.level[0] - 1)
}
}
}
七、告警闭环
7.1 告警确认
python
// 确认告警
def acknowledgeAlert(alertId, operator, note) {
update alert_log
set status = "acknowledged"
where alert_id = alertId
// 记录操作
insert into alert_operation values (
alertId, "acknowledge", operator, now(), note
)
}
7.2 告警处理
python
// 处理告警
def resolveAlert(alertId, operator, solution) {
update alert_log
set status = "resolved"
where alert_id = alertId
// 记录处理
insert into alert_operation values (
alertId, "resolve", operator, now(), solution
)
}
7.3 告警统计
python
// 告警统计
def getAlertStats(startTime, endTime) {
return select level, status, count(*) as count
from alert_log
where alert_time between startTime and endTime
group by level, status
}
八、实战案例
8.1 完整告警系统
python
// ========== 实时告警系统 ==========
// 1. 创建数据表
share table(1:0,
`rule_id`rule_name`device_id`metric`operator`threshold`level`enabled,
[STRING, STRING, SYMBOL, STRING, STRING, DOUBLE, INT, BOOL]) as alert_rules
share table(1:0,
`alert_id`rule_id`device_id`alert_time`metric`value`threshold`level`status,
[STRING, STRING, SYMBOL, TIMESTAMP, STRING, DOUBLE, DOUBLE, INT, STRING]) as alert_log
// 2. 初始化规则
insert into alert_rules values ("temp_high", "温度过高", `all, "temperature", ">", 80, 1, true)
insert into alert_rules values ("temp_low", "温度过低", `all, "temperature", "<", 10, 2, true)
insert into alert_rules values ("humidity_high", "湿度过高", `all, "humidity", ">", 90, 2, true)
// 3. 创建传感器数据流
share streamTable(100000:0,
`device_id`timestamp`temperature`humidity`pressure,
[SYMBOL, TIMESTAMP, DOUBLE, DOUBLE, DOUBLE]) as sensor_stream
// 4. 实时告警检测
def detectAlerts(data) {
for (row in data) {
rules = select * from alert_rules where enabled = true
for (rule in rules) {
value = row[rule.metric]
if (checkThreshold(value, rule.operator, rule.threshold)) {
alertId = "ALT" + format(now(), "yyyyMMddHHmmss")
insert into alert_log values (
alertId, rule.rule_id, row.device_id, now(),
rule.metric, value, rule.threshold, rule.level, "new"
)
print("告警: " + rule.rule_name + " 设备:" + row.device_id + " 值:" + string(value))
}
}
}
}
subscribeTable(, "sensor_stream", "alert_detect", -1,
def(msg) { detectAlerts(msg) }, true)
// 5. 模拟数据
def generateMockSensor() {
while (true) {
data = table(
take(1..10, 10) as device_id,
take(now(), 10) as timestamp,
rand(20.0..100.0, 10) as temperature,
rand(40.0..100.0, 10) as humidity,
rand(1000.0..1100.0, 10) as pressure
)
sensor_stream.append!(data)
sleep(5000)
}
}
submitJob("mock_sensor", "模拟传感器数据", generateMockSensor)
print("实时告警系统启动完成")
九、总结
本文详细介绍了DolphinDB实时告警系统:
- 告警规则:规则定义、规则配置
- 阈值检测:单阈值、区间阈值、动态阈值
- 告警触发:告警记录、实时检测
- 告警抑制:时间窗口、告警聚合、告警降噪
- 告警通知:通知渠道、通知发送、告警升级
- 告警闭环:告警确认、告警处理、告警统计
思考题:
- 如何设计高效的告警规则引擎?
- 如何平衡告警的敏感度和误报率?
- 如何实现告警的智能降噪?