1. 前言
最近我看到Rust语言的热度还是蛮高的，对Rust的应用进行监控，还是需要使用IT监控的工业标准OpenTelemetry (简称OTel)。OpenTelemetry很早就对Rust语言有支持(当然支持的程度跟很多语言比还比较初级)，这次就写篇文章介绍一下。
很多企业应用的偏好是：监控代码不要侵入应用代码之中。因此，在这篇文章里，我就把使用OpenTelemetry Rust SDK的内容排除在外了。压缩一下监控的需求，我们有办法做到零侵入，因为OpenTelemetry有个杀手锏叫做OBI (OpenTelemetry eBPF Instrumentation)，利用利用 Linux 内核的 eBPF(extended Berkeley Packet Filter)机制来监控系统调用，这个方法不受编程语言的限制。我曾专门写过一篇文章介绍OBI。也许Rust程序员不了解这方面的内容，我就在这篇文章里再专门介绍一下如果在Rust应用实施这种监控。
2. 做一个用于演示的Rust应用
我的这个应用是个小的Web服务器，只用于演示如何被OpenTelemetry监控。为了简化，我在这个应用里没有使用任何Web应用的框架(Axum、Actix Web 或 Hyper之类)。以下是建立的过程：
第一步：先建项目：
bash
cargo new rust-obi-demo第二步：用下面的代码更新 main.rs：
rust
use std::io::{Read, Write};
use std::net::{TcpListener, TcpStream};
use std::thread;
use std::time::Duration;
fn main() -> std::io::Result<()> {
let listener = TcpListener::bind("0.0.0.0:8080")?;
println!("Rust HTTP server listening on 0.0.0.0:8080");
println!("Endpoints:");
println!(" GET / - HTTP 200");
println!(" GET /hello - HTTP 200");
println!(" GET /slow - HTTP 200, delayed response");
println!(" GET /error - HTTP 500");
println!(" GET /not-found - HTTP 404");
for stream in listener.incoming() {
match stream {
Ok(stream) => {
// 每个请求使用一个线程处理，便于模拟并发请求。
thread::spawn(move || {
if let Err(err) = handle_request(stream) {
eprintln!("Request handling error: {err}");
}
});
}
Err(err) => {
eprintln!("Failed to accept connection: {err}");
}
}
}
Ok(())
}
fn handle_request(mut stream: TcpStream) -> std::io::Result<()> {
let mut buffer = [0_u8; 4096];
let n = stream.read(&mut buffer)?;
if n == 0 {
return Ok(());
}
let request = String::from_utf8_lossy(&buffer[..n]);
let request_line = request.lines().next().unwrap_or("");
let mut parts = request_line.split_whitespace();
let method = parts.next().unwrap_or("");
let path = parts.next().unwrap_or("/");
// 忽略查询字符串，例如 /hello?id=1 被识别为 /hello。
let route = path.split('?').next().unwrap_or("/");
println!("{method} {path}");
let (status, reason, body) = match (method, route) {
("GET", "/") => (200, "OK", r#"{"message":"Welcome to Rust OBI demo"}"#),
("GET", "/hello") => (200, "OK", r#"{"message":"Hello, world!"}"#),
("GET", "/slow") => {
// 模拟一个耗时较长的业务操作。
thread::sleep(Duration::from_millis(800));
(200, "OK", r#"{"message":"Slow operation completed"}"#)
}
("GET", "/error") => (
500,
"Internal Server Error",
r#"{"error":"Simulated internal server error"}"#,
),
("GET", "/not-found") => (404, "Not Found", r#"{"error":"Resource not found"}"#),
(_, _) => (404, "Not Found", r#"{"error":"Unknown endpoint"}"#),
};
let response = format!(
"HTTP/1.1 {status} {reason}\r\n\
Content-Type: application/json\r\n\
Content-Length: {}\r\n\
Connection: close\r\n\
\r\n\
{}",
body.len(),
body
);
stream.write_all(response.as_bytes())?;
stream.flush()?;
println!("Response: {status} {reason}");
Ok(())
} 第三步：编译：
bash
cargo build --release于是生成了二进制执行文件用于演示： ./target/release/rust-obi-demo
3. 搭建测试环境
如果要用OpenTelemetry监控您的IT系统，肯定需要以下几个部分：
- 应用 + OpenTelemetry Agent
- OpenTelemetry Collector (用于采集、调整、转发或路由监控数据)。
- OBI 工具(将来会内嵌到OpenTelemetry Collector内)
- OpenTelemetry 后端 (用于存储、展现和分析监控数据)
我这篇文章没有使用任何针对Rust应用的OpenTelemetry Agent。对于演示系统或者超小型用户，我们可以使用一个既免费又好用的工具，基于Docker的Grafana LGTM，它包含了OpenTelemetry Collector和一个小型的OpenTelemetry 后端。它的安装极其简单，只需要一个支持Docker的Linux环境。下面是安装的步骤(我把LGTM安装到/opt/lgtm目录 (其实任何目录均可))：
bash
docker pull grafana/otel-lgtm
mkdir /opt/lgtm
cd /opt/lgtm
wget https://raw.githubusercontent.com/grafana/docker-otel-lgtm/main/run-lgtm.sh
chmod +x run-lgtm.sh
sed -i 's/3000:3000/3100:3000/' run-lgtm.sh注意最后一行命令，因为LGTM的Grafana的默认对外端口是3000，这个端口经常和一些应用程序冲突，我就改成了3100.
下面是启动LGTM的方法，就一个命令：
bash
cd /opt/lgtm
./run-lgtm.shLGTM要监听以下的端口：
- 4317/4318 是OTLP端口，用来接收metrics/traces/logs数据
- 3100 是Grafana UI的端口
- 9090 是Prometheus的端口，用于调试
- 4040 是Pyroscope接收Profiles的端口，将来会整合进入4317/4318
至少在OBI工具被完全整合进入OpenTelemetry Collector之前，我们还需要下载这个工具。到 OBI 的发行网页(Releases · open-telemetry/opentelemetry-ebpf-instrumentation · GitHub)，根据您的平台选择最新的发行版来下载。比如最常见的平台(amd64) 和当前最新的版本号v1.0.0-rc.1，可下载：obi-v1.0.0-rc.1-linux-amd64.tar.gz。
配置OBI的方式非常多。可以采用环境变量，也可以用配置文件。对于这个简单的测试，我使用了以下的配置文件：obi.yaml
XML
discovery:
instrument:
- open_ports: 8080,9090,28080
log_level: DEBUG
ebpf:
context_propagation: all
otel_traces_export:
endpoint: http://localhost:4318
otel_metrics_export:
endpoint: http://localhost:4318注意：
- 这个配置文件说明我要监控8080、9090和28080三个端口的应用。
- localhost是Grafana LGTM Stack所在的宿主机地址，如果是其它IP地址(比如192.168.108.128)，直接修改即可。
以下是运行OBI的命令(在应用所在的Linux主机上运行，注意应用的端口必须在 obi.yaml 设置的范围之内)：
bash
sudo ./obi --config=./obi.yaml至于配置OBI的详细说明，请参照官方文档：Configure OBI | OpenTelemetry
现在就差启动应用了。为了让OpenTelemetry很好的区分不同的应用，请在启动应用之前，设置OTEL_SERVICE_NAME环境变量。用以下命令启动编译好的Rust应用：
bash
./target/release/rust-obi-demo4. 设置Grafana
使用"http://localhost:3100"就可以访问LGTM的Grafana UI界面(如果是远程的话，用主机名或者IP替换localhost)，使用admin/admin登录。虽然LGTM支持各种类型的数据：metrics/traces/logs/profiles，但是对于Rust应用来说，OBI目前只收集traces，基于traces可以生成一些RED metrics (RED指request rate, error rate, duration)。所以我们的Grafana Dashboard只展现这两种数据。
以下是我的主要OBI Dashboard的界面：
下面是Dashboard的源代码，可以导入。
bash
{
"apiVersion": "dashboard.grafana.app/v2",
"kind": "Dashboard",
"metadata": {
"name": "ad8krnh",
"namespace": "default",
"uid": "bKBcTrEPw0WklhXRlSyjfYEHb4onRYDhnAwDeRF28zcX",
"resourceVersion": "1774931703763993",
"generation": 18,
"creationTimestamp": "2026-03-02T03:52:01Z",
"labels": {
"grafana.app/deprecatedInternalID": "4"
},
"annotations": {
"grafana.app/createdBy": "user:eferfp17srfnkb",
"grafana.app/saved-from-ui": "Grafana v12.4.1 (46a02dc12a)",
"grafana.app/updatedBy": "user:eferfp17srfnkb",
"grafana.app/updatedTimestamp": "2026-03-31T04:35:03Z"
}
},
"spec": {
"annotations": [
{
"kind": "AnnotationQuery",
"spec": {
"query": {
"kind": "DataQuery",
"group": "grafana",
"version": "v0",
"datasource": {
"name": "-- Grafana --"
},
"spec": {}
},
"enable": true,
"hide": true,
"iconColor": "rgba(0, 211, 255, 1)",
"name": "Annotations & Alerts",
"builtIn": true,
"legacyOptions": {
"type": "dashboard"
}
}
}
],
"cursorSync": "Off",
"editable": true,
"elements": {
"panel-1": {
"kind": "Panel",
"spec": {
"id": 1,
"title": "延迟时间",
"links": [],
"data": {
"kind": "QueryGroup",
"spec": {
"queries": [
{
"kind": "PanelQuery",
"spec": {
"query": {
"kind": "DataQuery",
"group": "prometheus",
"version": "v0",
"datasource": {
"name": "prometheus"
},
"spec": {
"editorMode": "code",
"exemplar": false,
"expr": "histogram_quantile(0.90, sum by (le, http_route, service_name) (rate(http_server_request_duration_seconds_bucket{service_name!~\"lgtm|\"}[2m])))\r\n",
"instant": true,
"legendFormat": "__auto",
"range": true
}
},
"refId": "A",
"hidden": false
}
}
],
"transformations": [],
"queryOptions": {}
}
},
"vizConfig": {
"kind": "VizConfig",
"group": "timeseries",
"version": "12.4.1",
"spec": {
"options": {
"legend": {
"calcs": [],
"displayMode": "list",
"placement": "bottom",
"showLegend": true
},
"tooltip": {
"hideZeros": false,
"mode": "single",
"sort": "none"
}
},
"fieldConfig": {
"defaults": {
"unit": "s",
"thresholds": {
"mode": "absolute",
"steps": [
{
"value": 0,
"color": "green"
},
{
"value": 80,
"color": "red"
}
]
},
"color": {
"mode": "palette-classic"
},
"custom": {
"axisBorderShow": false,
"axisCenteredZero": false,
"axisColorMode": "text",
"axisLabel": "",
"axisPlacement": "auto",
"barAlignment": 0,
"barWidthFactor": 0.6,
"drawStyle": "line",
"fillOpacity": 0,
"gradientMode": "none",
"hideFrom": {
"legend": false,
"tooltip": false,
"viz": false
},
"insertNulls": false,
"lineInterpolation": "linear",
"lineWidth": 1,
"pointSize": 5,
"scaleDistribution": {
"type": "linear"
},
"showPoints": "never",
"showValues": false,
"spanNulls": false,
"stacking": {
"group": "A",
"mode": "none"
},
"thresholdsStyle": {
"mode": "off"
}
}
},
"overrides": []
}
}
}
}
},
"panel-2": {
"kind": "Panel",
"spec": {
"id": 2,
"title": "吞吐量",
"links": [],
"data": {
"kind": "QueryGroup",
"spec": {
"queries": [
{
"kind": "PanelQuery",
"spec": {
"query": {
"kind": "DataQuery",
"group": "prometheus",
"version": "v0",
"datasource": {
"name": "prometheus"
},
"spec": {
"editorMode": "code",
"expr": "sum by (http_route, http_response_status_code, service_name) (\r\n rate(http_server_request_duration_seconds_count{service_name!~\"lgtm|\"}[2m])\r\n)\r\n",
"instant": true,
"legendFormat": "__auto",
"range": true
}
},
"refId": "A",
"hidden": false
}
}
],
"transformations": [],
"queryOptions": {}
}
},
"vizConfig": {
"kind": "VizConfig",
"group": "timeseries",
"version": "12.4.1",
"spec": {
"options": {
"legend": {
"calcs": [],
"displayMode": "list",
"placement": "bottom",
"showLegend": true
},
"tooltip": {
"hideZeros": false,
"mode": "single",
"sort": "none"
}
},
"fieldConfig": {
"defaults": {
"unit": "reqps",
"thresholds": {
"mode": "absolute",
"steps": [
{
"value": 0,
"color": "green"
},
{
"value": 80,
"color": "red"
}
]
},
"color": {
"mode": "palette-classic"
},
"custom": {
"axisBorderShow": false,
"axisCenteredZero": false,
"axisColorMode": "text",
"axisLabel": "",
"axisPlacement": "auto",
"barAlignment": 0,
"barWidthFactor": 0.6,
"drawStyle": "line",
"fillOpacity": 0,
"gradientMode": "none",
"hideFrom": {
"legend": false,
"tooltip": false,
"viz": false
},
"insertNulls": false,
"lineInterpolation": "linear",
"lineWidth": 1,
"pointSize": 5,
"scaleDistribution": {
"type": "linear"
},
"showPoints": "never",
"showValues": false,
"spanNulls": false,
"stacking": {
"group": "A",
"mode": "none"
},
"thresholdsStyle": {
"mode": "off"
}
}
},
"overrides": []
}
}
}
}
},
"panel-4": {
"kind": "Panel",
"spec": {
"id": 4,
"title": "Traces",
"links": [],
"data": {
"kind": "QueryGroup",
"spec": {
"queries": [
{
"kind": "PanelQuery",
"spec": {
"query": {
"kind": "DataQuery",
"group": "tempo",
"version": "v0",
"datasource": {
"name": "tempo"
},
"spec": {
"filters": [
{
"id": "3aaf9885",
"operator": "=",
"scope": "span"
},
{
"id": "service-name",
"isCustomValue": false,
"operator": "!=",
"scope": "resource",
"tag": "service.name",
"value": [
"lgtm"
],
"valueType": "string"
}
],
"key": "Q-a3d09f2b-d713-4e41-964b-de64d135e60a-0",
"limit": 20,
"metricsQueryType": "range",
"queryType": "traceqlSearch",
"serviceMapUseNativeHistograms": false,
"tableType": "traces"
}
},
"refId": "A",
"hidden": false
}
}
],
"transformations": [],
"queryOptions": {}
}
},
"vizConfig": {
"kind": "VizConfig",
"group": "table",
"version": "12.4.1",
"spec": {
"options": {
"cellHeight": "sm",
"showHeader": true
},
"fieldConfig": {
"defaults": {
"thresholds": {
"mode": "absolute",
"steps": [
{
"value": 0,
"color": "green"
},
{
"value": 80,
"color": "red"
}
]
},
"custom": {
"align": "auto",
"cellOptions": {
"type": "auto"
},
"footer": {
"reducers": []
},
"inspect": false
}
},
"overrides": []
}
}
}
}
},
"panel-5": {
"kind": "Panel",
"spec": {
"id": 5,
"title": "请求包大小",
"links": [],
"data": {
"kind": "QueryGroup",
"spec": {
"queries": [
{
"kind": "PanelQuery",
"spec": {
"query": {
"kind": "DataQuery",
"group": "prometheus",
"version": "v0",
"datasource": {
"name": "prometheus"
},
"spec": {
"editorMode": "code",
"expr": "histogram_quantile(0.90, sum by (le, http_route, service_name) (rate(http_server_request_body_size_bytes_bucket{service_name!~\"lgtm|\"}[2m])))",
"instant": true,
"key": "Q-33194457-04d2-4472-b32e-942fd5e718af-0",
"legendFormat": "__auto",
"range": true
}
},
"refId": "A",
"hidden": false
}
}
],
"transformations": [],
"queryOptions": {}
}
},
"vizConfig": {
"kind": "VizConfig",
"group": "timeseries",
"version": "12.4.1",
"spec": {
"options": {
"legend": {
"calcs": [],
"displayMode": "list",
"placement": "bottom",
"showLegend": true
},
"tooltip": {
"hideZeros": false,
"mode": "single",
"sort": "none"
}
},
"fieldConfig": {
"defaults": {
"unit": "decbytes",
"thresholds": {
"mode": "absolute",
"steps": [
{
"value": 0,
"color": "green"
},
{
"value": 80,
"color": "red"
}
]
},
"color": {
"mode": "palette-classic"
},
"custom": {
"axisBorderShow": false,
"axisCenteredZero": false,
"axisColorMode": "text",
"axisLabel": "",
"axisPlacement": "auto",
"barAlignment": 0,
"barWidthFactor": 0.6,
"drawStyle": "line",
"fillOpacity": 0,
"gradientMode": "none",
"hideFrom": {
"legend": false,
"tooltip": false,
"viz": false
},
"insertNulls": false,
"lineInterpolation": "linear",
"lineWidth": 1,
"pointSize": 5,
"scaleDistribution": {
"type": "linear"
},
"showPoints": "never",
"showValues": false,
"spanNulls": false,
"stacking": {
"group": "A",
"mode": "none"
},
"thresholdsStyle": {
"mode": "off"
}
}
},
"overrides": []
}
}
}
}
},
"panel-6": {
"kind": "Panel",
"spec": {
"id": 6,
"title": "返回包大小",
"links": [],
"data": {
"kind": "QueryGroup",
"spec": {
"queries": [
{
"kind": "PanelQuery",
"spec": {
"query": {
"kind": "DataQuery",
"group": "prometheus",
"version": "v0",
"datasource": {
"name": "prometheus"
},
"spec": {
"editorMode": "code",
"expr": "histogram_quantile(0.90, sum by (le, http_route, service_name) (rate(http_server_response_body_size_bytes_bucket{service_name!~\"lgtm|\"}[2m])))\r\n",
"instant": true,
"legendFormat": "__auto",
"range": true
}
},
"refId": "A",
"hidden": false
}
}
],
"transformations": [],
"queryOptions": {}
}
},
"vizConfig": {
"kind": "VizConfig",
"group": "timeseries",
"version": "12.4.1",
"spec": {
"options": {
"legend": {
"calcs": [],
"displayMode": "list",
"placement": "bottom",
"showLegend": true
},
"tooltip": {
"hideZeros": false,
"mode": "single",
"sort": "none"
}
},
"fieldConfig": {
"defaults": {
"unit": "decbytes",
"thresholds": {
"mode": "absolute",
"steps": [
{
"value": 0,
"color": "green"
},
{
"value": 80,
"color": "red"
}
]
},
"color": {
"mode": "palette-classic"
},
"custom": {
"axisBorderShow": false,
"axisCenteredZero": false,
"axisColorMode": "text",
"axisLabel": "",
"axisPlacement": "auto",
"barAlignment": 0,
"barWidthFactor": 0.6,
"drawStyle": "line",
"fillOpacity": 0,
"gradientMode": "none",
"hideFrom": {
"legend": false,
"tooltip": false,
"viz": false
},
"insertNulls": false,
"lineInterpolation": "linear",
"lineWidth": 1,
"pointSize": 5,
"scaleDistribution": {
"type": "linear"
},
"showPoints": "never",
"showValues": false,
"spanNulls": false,
"stacking": {
"group": "A",
"mode": "none"
},
"thresholdsStyle": {
"mode": "off"
}
}
},
"overrides": []
}
}
}
}
},
"panel-7": {
"kind": "Panel",
"spec": {
"id": 7,
"title": "失败次数/秒",
"links": [],
"data": {
"kind": "QueryGroup",
"spec": {
"queries": [
{
"kind": "PanelQuery",
"spec": {
"query": {
"kind": "DataQuery",
"group": "prometheus",
"version": "v0",
"datasource": {
"name": "prometheus"
},
"spec": {
"editorMode": "code",
"expr": "sum by (client, server) (rate(traces_service_graph_request_failed_total[2m]))",
"instant": true,
"legendFormat": "__auto",
"range": true
}
},
"refId": "A",
"hidden": false
}
}
],
"transformations": [],
"queryOptions": {}
}
},
"vizConfig": {
"kind": "VizConfig",
"group": "timeseries",
"version": "12.4.1",
"spec": {
"options": {
"legend": {
"calcs": [],
"displayMode": "list",
"placement": "bottom",
"showLegend": true
},
"tooltip": {
"hideZeros": false,
"mode": "single",
"sort": "none"
}
},
"fieldConfig": {
"defaults": {
"unit": "reqps",
"thresholds": {
"mode": "absolute",
"steps": [
{
"value": 0,
"color": "green"
},
{
"value": 80,
"color": "red"
}
]
},
"color": {
"mode": "palette-classic"
},
"custom": {
"axisBorderShow": false,
"axisCenteredZero": false,
"axisColorMode": "text",
"axisLabel": "",
"axisPlacement": "auto",
"barAlignment": 0,
"barWidthFactor": 0.6,
"drawStyle": "line",
"fillOpacity": 0,
"gradientMode": "none",
"hideFrom": {
"legend": false,
"tooltip": false,
"viz": false
},
"insertNulls": false,
"lineInterpolation": "linear",
"lineWidth": 1,
"pointSize": 5,
"scaleDistribution": {
"type": "linear"
},
"showPoints": "never",
"showValues": false,
"spanNulls": false,
"stacking": {
"group": "A",
"mode": "none"
},
"thresholdsStyle": {
"mode": "off"
}
}
},
"overrides": []
}
}
}
}
},
"panel-8": {
"kind": "Panel",
"spec": {
"id": 8,
"title": "总流量",
"links": [],
"data": {
"kind": "QueryGroup",
"spec": {
"queries": [
{
"kind": "PanelQuery",
"spec": {
"query": {
"kind": "DataQuery",
"group": "prometheus",
"version": "v0",
"datasource": {
"name": "prometheus"
},
"spec": {
"editorMode": "code",
"expr": "sum by (service) (rate(traces_spanmetrics_size_total[2m]))",
"instant": true,
"legendFormat": "__auto",
"range": true
}
},
"refId": "A",
"hidden": false
}
}
],
"transformations": [],
"queryOptions": {}
}
},
"vizConfig": {
"kind": "VizConfig",
"group": "timeseries",
"version": "12.4.1",
"spec": {
"options": {
"legend": {
"calcs": [],
"displayMode": "list",
"placement": "bottom",
"showLegend": true
},
"tooltip": {
"hideZeros": false,
"mode": "single",
"sort": "none"
}
},
"fieldConfig": {
"defaults": {
"unit": "Bps",
"thresholds": {
"mode": "absolute",
"steps": [
{
"value": 0,
"color": "green"
},
{
"value": 80,
"color": "red"
}
]
},
"color": {
"mode": "palette-classic"
},
"custom": {
"axisBorderShow": false,
"axisCenteredZero": false,
"axisColorMode": "text",
"axisLabel": "",
"axisPlacement": "auto",
"barAlignment": 0,
"barWidthFactor": 0.6,
"drawStyle": "line",
"fillOpacity": 0,
"gradientMode": "none",
"hideFrom": {
"legend": false,
"tooltip": false,
"viz": false
},
"insertNulls": false,
"lineInterpolation": "linear",
"lineWidth": 1,
"pointSize": 5,
"scaleDistribution": {
"type": "linear"
},
"showPoints": "never",
"showValues": false,
"spanNulls": false,
"stacking": {
"group": "A",
"mode": "none"
},
"thresholdsStyle": {
"mode": "off"
}
}
},
"overrides": []
}
}
}
}
},
"panel-9": {
"kind": "Panel",
"spec": {
"id": 9,
"title": "服务调用图",
"links": [],
"data": {
"kind": "QueryGroup",
"spec": {
"queries": [
{
"kind": "PanelQuery",
"spec": {
"query": {
"kind": "DataQuery",
"group": "tempo",
"version": "v0",
"datasource": {
"name": "tempo"
},
"spec": {
"key": "Q-b5c0b572-5c72-48f7-a7fe-eefba712e204-0",
"limit": 20,
"metricsQueryType": "range",
"queryType": "serviceMap",
"serviceMapUseNativeHistograms": false,
"tableType": "traces"
}
},
"refId": "A",
"hidden": false
}
}
],
"transformations": [],
"queryOptions": {}
}
},
"vizConfig": {
"kind": "VizConfig",
"group": "nodeGraph",
"version": "12.4.1",
"spec": {
"options": {
"edges": {},
"layoutAlgorithm": "layered",
"nodes": {},
"zoomMode": "cooperative"
},
"fieldConfig": {
"defaults": {},
"overrides": []
}
}
}
}
}
},
"layout": {
"kind": "GridLayout",
"spec": {
"items": [
{
"kind": "GridLayoutItem",
"spec": {
"x": 0,
"y": 0,
"width": 12,
"height": 8,
"element": {
"kind": "ElementReference",
"name": "panel-2"
}
}
},
{
"kind": "GridLayoutItem",
"spec": {
"x": 12,
"y": 0,
"width": 12,
"height": 8,
"element": {
"kind": "ElementReference",
"name": "panel-1"
}
}
},
{
"kind": "GridLayoutItem",
"spec": {
"x": 0,
"y": 8,
"width": 12,
"height": 8,
"element": {
"kind": "ElementReference",
"name": "panel-5"
}
}
},
{
"kind": "GridLayoutItem",
"spec": {
"x": 12,
"y": 8,
"width": 12,
"height": 8,
"element": {
"kind": "ElementReference",
"name": "panel-6"
}
}
},
{
"kind": "GridLayoutItem",
"spec": {
"x": 0,
"y": 16,
"width": 12,
"height": 8,
"element": {
"kind": "ElementReference",
"name": "panel-7"
}
}
},
{
"kind": "GridLayoutItem",
"spec": {
"x": 12,
"y": 16,
"width": 12,
"height": 8,
"element": {
"kind": "ElementReference",
"name": "panel-8"
}
}
},
{
"kind": "GridLayoutItem",
"spec": {
"x": 0,
"y": 24,
"width": 12,
"height": 8,
"element": {
"kind": "ElementReference",
"name": "panel-9"
}
}
},
{
"kind": "GridLayoutItem",
"spec": {
"x": 12,
"y": 24,
"width": 12,
"height": 8,
"element": {
"kind": "ElementReference",
"name": "panel-4"
}
}
}
]
}
},
"links": [],
"liveNow": false,
"preload": false,
"tags": [],
"timeSettings": {
"timezone": "browser",
"from": "now-1h",
"to": "now",
"autoRefresh": "",
"autoRefreshIntervals": [
"5s",
"10s",
"30s",
"1m",
"5m",
"15m",
"30m",
"1h",
"2h",
"1d"
],
"hideTimepicker": false,
"fiscalYearStartMonth": 0
},
"title": "OBI dashboard",
"variables": []
}
}下面是我的RED metrics (RED指request rate, error rate, duration) dashboard:
下面是它的源码(可导入)：
bash
{
"apiVersion": "dashboard.grafana.app/v2",
"kind": "Dashboard",
"metadata": {
"name": "adsm9xr",
"namespace": "default",
"uid": "8dabee4a-9319-4e98-bad6-1a68d3cd6f01",
"resourceVersion": "1791645887005010",
"generation": 1,
"creationTimestamp": "2026-10-10T15:24:46Z",
"labels": {
"grafana.app/deprecatedInternalID": "1254655419084800"
},
"annotations": {
"grafana.app/createdBy": "user:eferfp17srfnkb",
"grafana.app/saved-from-ui": "Grafana v13.2.3 (90ffed056f)"
}
},
"spec": {
"annotations": [
{
"kind": "AnnotationQuery",
"spec": {
"query": {
"kind": "DataQuery",
"group": "grafana",
"version": "v0",
"datasource": {
"name": "-- Grafana --"
},
"spec": {}
},
"enable": true,
"hide": true,
"iconColor": "rgba(0, 211, 255, 1)",
"name": "Annotations & Alerts",
"builtIn": true
}
}
],
"cursorSync": "Off",
"description": "request rate, error rate, duration",
"editable": true,
"elements": {
"panel-1": {
"kind": "Panel",
"spec": {
"id": 1,
"title": "Request Rate",
"description": "",
"links": [],
"data": {
"kind": "QueryGroup",
"spec": {
"queries": [
{
"kind": "PanelQuery",
"spec": {
"query": {
"kind": "DataQuery",
"group": "prometheus",
"version": "v0",
"datasource": {
"name": "prometheus"
},
"spec": {
"editorMode": "code",
"expr": "sum(rate({__name__=~\"http_server_request_duration_seconds_count|http_server_request_duration_count|http_server_duration_milliseconds_count|http_server_duration_seconds_count|http_server_duration_count\", job=~\"$job\", instance=~\"$instance\"}[5m]))",
"legendFormat": "__auto",
"range": true
}
},
"refId": "A",
"hidden": false
}
}
],
"transformations": [],
"queryOptions": {}
}
},
"vizConfig": {
"kind": "VizConfig",
"group": "timeseries",
"version": "13.2.3",
"spec": {
"options": {
"legend": {
"calcs": [],
"displayMode": "list",
"placement": "bottom",
"showLegend": true
},
"tooltip": {
"hideZeros": false,
"mode": "single",
"sort": "none"
}
},
"fieldConfig": {
"defaults": {
"unit": "reqps",
"min": 0,
"thresholds": {
"mode": "absolute",
"steps": [
{
"value": 0,
"color": "green"
},
{
"value": 80,
"color": "red"
}
]
},
"color": {
"mode": "palette-classic"
},
"custom": {
"axisBorderShow": false,
"axisCenteredZero": false,
"axisColorMode": "text",
"axisLabel": "",
"axisPlacement": "auto",
"barAlignment": 0,
"barWidthFactor": 0.6,
"drawStyle": "line",
"fillOpacity": 0,
"gradientMode": "none",
"hideFrom": {
"legend": false,
"tooltip": false,
"viz": false
},
"insertNulls": false,
"lineInterpolation": "linear",
"lineWidth": 1,
"pointSize": 5,
"scaleDistribution": {
"type": "linear"
},
"showPoints": "auto",
"showValues": false,
"spanNulls": false,
"stacking": {
"group": "A",
"mode": "none"
},
"thresholdsStyle": {
"mode": "off"
}
}
},
"overrides": []
}
}
}
}
},
"panel-2": {
"kind": "Panel",
"spec": {
"id": 2,
"title": "Error Rate",
"description": "",
"links": [],
"data": {
"kind": "QueryGroup",
"spec": {
"queries": [
{
"kind": "PanelQuery",
"spec": {
"query": {
"kind": "DataQuery",
"group": "prometheus",
"version": "v0",
"datasource": {
"name": "prometheus"
},
"spec": {
"editorMode": "code",
"expr": "sum(rate({__name__=~\"http_server_request_duration_seconds_count|http_server_request_duration_count|http_server_duration_milliseconds_count|http_server_duration_seconds_count|http_server_duration_count\", job=~\"$job\", instance=~\"$instance\", http_response_status_code=~\"5..\"}[5m])) / sum(rate({__name__=~\"http_server_request_duration_seconds_count|http_server_request_duration_count|http_server_duration_milliseconds_count|http_server_duration_seconds_count|http_server_duration_count\", job=~\"$job\", instance=~\"$instance\"}[5m]))",
"legendFormat": "__auto",
"range": true
}
},
"refId": "A",
"hidden": false
}
}
],
"transformations": [],
"queryOptions": {}
}
},
"vizConfig": {
"kind": "VizConfig",
"group": "timeseries",
"version": "13.2.3",
"spec": {
"options": {
"legend": {
"calcs": [],
"displayMode": "list",
"placement": "bottom",
"showLegend": true
},
"tooltip": {
"hideZeros": false,
"mode": "single",
"sort": "none"
}
},
"fieldConfig": {
"defaults": {
"unit": "percentunit",
"min": 0,
"thresholds": {
"mode": "absolute",
"steps": [
{
"value": 0,
"color": "green"
},
{
"value": 80,
"color": "red"
}
]
},
"color": {
"mode": "palette-classic"
},
"custom": {
"axisBorderShow": false,
"axisCenteredZero": false,
"axisColorMode": "text",
"axisLabel": "",
"axisPlacement": "auto",
"barAlignment": 0,
"barWidthFactor": 0.6,
"drawStyle": "line",
"fillOpacity": 0,
"gradientMode": "none",
"hideFrom": {
"legend": false,
"tooltip": false,
"viz": false
},
"insertNulls": false,
"lineInterpolation": "linear",
"lineWidth": 1,
"pointSize": 5,
"scaleDistribution": {
"type": "linear"
},
"showPoints": "auto",
"showValues": false,
"spanNulls": false,
"stacking": {
"group": "A",
"mode": "none"
},
"thresholdsStyle": {
"mode": "off"
}
}
},
"overrides": []
}
}
}
}
},
"panel-3": {
"kind": "Panel",
"spec": {
"id": 3,
"title": "Duration histogram (s)",
"description": "",
"links": [],
"data": {
"kind": "QueryGroup",
"spec": {
"queries": [
{
"kind": "PanelQuery",
"spec": {
"query": {
"kind": "DataQuery",
"group": "prometheus",
"version": "v0",
"datasource": {
"name": "prometheus"
},
"spec": {
"editorMode": "code",
"exemplar": true,
"expr": "sum by (le) (rate({__name__=~\"http_server_request_duration_seconds_bucket|http_server_request_duration_bucket|http_server_duration_milliseconds_bucket|http_server_duration_seconds_bucket|http_server_duration_bucket\", job=~\"$job\", instance=~\"$instance\"}[5m]))",
"format": "heatmap",
"instant": false,
"legendFormat": "{{le}}",
"range": true
}
},
"refId": "A",
"hidden": false
}
}
],
"transformations": [],
"queryOptions": {}
}
},
"vizConfig": {
"kind": "VizConfig",
"group": "bargauge",
"version": "13.2.3",
"spec": {
"options": {
"displayMode": "gradient",
"legend": {
"calcs": [],
"displayMode": "list",
"placement": "bottom",
"showLegend": false
},
"maxVizHeight": 300,
"minVizHeight": 10,
"minVizWidth": 0,
"namePlacement": "auto",
"orientation": "auto",
"reduceOptions": {
"calcs": [
"lastNotNull"
],
"fields": "",
"values": false
},
"showUnfilled": true,
"sizing": "auto",
"valueMode": "color"
},
"fieldConfig": {
"defaults": {
"unit": "reqps",
"thresholds": {
"mode": "absolute",
"steps": [
{
"value": 0,
"color": "green"
},
{
"value": 80,
"color": "red"
}
]
},
"color": {
"mode": "thresholds"
}
},
"overrides": []
}
}
}
}
},
"panel-4": {
"kind": "Panel",
"spec": {
"id": 4,
"title": "Duration percentiles",
"description": "",
"links": [],
"data": {
"kind": "QueryGroup",
"spec": {
"queries": [
{
"kind": "PanelQuery",
"spec": {
"query": {
"kind": "DataQuery",
"group": "prometheus",
"version": "v0",
"datasource": {
"name": "prometheus"
},
"spec": {
"editorMode": "code",
"exemplar": true,
"expr": "histogram_quantile(0.95, sum by (le) (rate({__name__=~\"http_server_request_duration_seconds_bucket|http_server_request_duration_bucket|http_server_duration_milliseconds_bucket|http_server_duration_seconds_bucket|http_server_duration_bucket\", job=~\"$job\", instance=~\"$instance\"}[5m])))",
"format": "time_series",
"instant": false,
"legendFormat": "95th",
"range": true
}
},
"refId": "A",
"hidden": false
}
},
{
"kind": "PanelQuery",
"spec": {
"query": {
"kind": "DataQuery",
"group": "prometheus",
"version": "v0",
"datasource": {
"name": "prometheus"
},
"spec": {
"editorMode": "code",
"exemplar": true,
"expr": "histogram_quantile(0.5, sum by (le) (rate({__name__=~\"http_server_request_duration_seconds_bucket|http_server_request_duration_bucket|http_server_duration_milliseconds_bucket|http_server_duration_seconds_bucket|http_server_duration_bucket\", job=~\"$job\", instance=~\"$instance\"}[5m])))",
"legendFormat": "50th",
"range": true
}
},
"refId": "B",
"hidden": false
}
}
],
"transformations": [],
"queryOptions": {}
}
},
"vizConfig": {
"kind": "VizConfig",
"group": "timeseries",
"version": "13.2.3",
"spec": {
"options": {
"legend": {
"calcs": [],
"displayMode": "list",
"placement": "bottom",
"showLegend": true
},
"tooltip": {
"hideZeros": false,
"mode": "single",
"sort": "none"
}
},
"fieldConfig": {
"defaults": {
"unit": "s",
"min": 0,
"thresholds": {
"mode": "absolute",
"steps": [
{
"value": 0,
"color": "green"
},
{
"value": 80,
"color": "red"
}
]
},
"color": {
"mode": "palette-classic"
},
"custom": {
"axisBorderShow": false,
"axisCenteredZero": false,
"axisColorMode": "text",
"axisLabel": "",
"axisPlacement": "auto",
"barAlignment": 0,
"barWidthFactor": 0.6,
"drawStyle": "line",
"fillOpacity": 0,
"gradientMode": "none",
"hideFrom": {
"legend": false,
"tooltip": false,
"viz": false
},
"insertNulls": false,
"lineInterpolation": "linear",
"lineWidth": 1,
"pointSize": 5,
"scaleDistribution": {
"type": "linear"
},
"showPoints": "auto",
"showValues": false,
"spanNulls": false,
"stacking": {
"group": "A",
"mode": "none"
},
"thresholdsStyle": {
"mode": "off"
}
}
},
"overrides": []
}
}
}
}
},
"panel-5": {
"kind": "Panel",
"spec": {
"id": 5,
"title": "Duration Heatmap",
"description": "",
"links": [],
"data": {
"kind": "QueryGroup",
"spec": {
"queries": [
{
"kind": "PanelQuery",
"spec": {
"query": {
"kind": "DataQuery",
"group": "prometheus",
"version": "v0",
"datasource": {
"name": "prometheus"
},
"spec": {
"editorMode": "code",
"exemplar": false,
"expr": "sum by (le) (rate({__name__=~\"http_server_request_duration_seconds_bucket|http_server_request_duration_bucket|http_server_duration_milliseconds_bucket|http_server_duration_seconds_bucket|http_server_duration_bucket\", job=~\"$job\", instance=~\"$instance\"}[5m]))",
"format": "heatmap",
"instant": false,
"range": true
}
},
"refId": "A",
"hidden": false
}
}
],
"transformations": [],
"queryOptions": {}
}
},
"vizConfig": {
"kind": "VizConfig",
"group": "heatmap",
"version": "13.2.3",
"spec": {
"options": {
"calculate": false,
"cellGap": 1,
"color": {
"exponent": 0.5,
"fill": "dark-orange",
"mode": "scheme",
"reverse": false,
"scale": "exponential",
"scheme": "Oranges",
"steps": 64
},
"exemplars": {
"color": "rgba(255,0,255,0.7)"
},
"filterValues": {
"le": 1e-9
},
"legend": {
"placement": "bottom",
"show": true
},
"rowsFrame": {
"layout": "auto"
},
"tooltip": {
"mode": "single",
"yHistogram": false
},
"yAxis": {
"axisPlacement": "left",
"reverse": false,
"unit": "s"
}
},
"fieldConfig": {
"defaults": {
"custom": {
"hideFrom": {
"legend": false,
"tooltip": false,
"viz": false
},
"scaleDistribution": {
"type": "linear"
}
}
},
"overrides": []
}
}
}
}
},
"panel-6": {
"kind": "Panel",
"spec": {
"id": 6,
"title": "",
"description": "",
"links": [],
"data": {
"kind": "QueryGroup",
"spec": {
"queries": [],
"transformations": [],
"queryOptions": {}
}
},
"vizConfig": {
"kind": "VizConfig",
"group": "text",
"version": "13.2.3",
"spec": {
"options": {
"code": {
"language": "plaintext",
"showLineNumbers": false,
"showMiniMap": false
},
"content": "# RED Metrics: (r)equest rate, (e)rror rate, (d)uration\n\nThis dashboard uses explicit bucket histograms and the stable [OpenTelemetry metrics semantic conventions](https://opentelemetry.io/docs/specs/semconv/general/metrics/).\nTo enable this for the [OpenTelemetry Java instrumentation agent](https://github.com/open-telemetry/opentelemetry-java-instrumentation/), set the following environment variables:\n\n```\nexport OTEL_EXPORTER_OTLP_METRICS_DEFAULT_HISTOGRAM_AGGREGATION=explicit_bucket_histogram\n```",
"mode": "markdown"
},
"fieldConfig": {
"defaults": {},
"overrides": []
}
}
}
}
}
},
"layout": {
"kind": "GridLayout",
"spec": {
"items": [
{
"kind": "GridLayoutItem",
"spec": {
"x": 0,
"y": 0,
"width": 12,
"height": 5,
"element": {
"kind": "ElementReference",
"name": "panel-6"
}
}
},
{
"kind": "GridLayoutItem",
"spec": {
"x": 12,
"y": 0,
"width": 12,
"height": 8,
"element": {
"kind": "ElementReference",
"name": "panel-2"
}
}
},
{
"kind": "GridLayoutItem",
"spec": {
"x": 0,
"y": 5,
"width": 12,
"height": 8,
"element": {
"kind": "ElementReference",
"name": "panel-1"
}
}
},
{
"kind": "GridLayoutItem",
"spec": {
"x": 12,
"y": 8,
"width": 12,
"height": 8,
"element": {
"kind": "ElementReference",
"name": "panel-3"
}
}
},
{
"kind": "GridLayoutItem",
"spec": {
"x": 0,
"y": 13,
"width": 12,
"height": 8,
"element": {
"kind": "ElementReference",
"name": "panel-5"
}
}
},
{
"kind": "GridLayoutItem",
"spec": {
"x": 12,
"y": 16,
"width": 12,
"height": 8,
"element": {
"kind": "ElementReference",
"name": "panel-4"
}
}
}
]
}
},
"links": [],
"liveNow": false,
"preload": false,
"tags": [],
"timeSettings": {
"timezone": "browser",
"from": "now-30m",
"to": "now",
"autoRefresh": "",
"autoRefreshIntervals": [
"5s",
"10s",
"30s",
"1m",
"5m",
"15m",
"30m",
"1h",
"2h",
"1d"
],
"hideTimepicker": false,
"fiscalYearStartMonth": 0
},
"title": "RED Metrics Dashboard",
"variables": [
{
"kind": "QueryVariable",
"spec": {
"name": "job",
"current": {
"text": "All",
"value": [
"$__all"
]
},
"hide": "dontHide",
"refresh": "onDashboardLoad",
"skipUrlSync": false,
"query": {
"kind": "DataQuery",
"group": "prometheus",
"version": "v0",
"datasource": {
"name": "prometheus"
},
"spec": {
"query": "label_values(job)",
"refId": "PrometheusVariableQueryEditor-VariableQuery"
}
},
"regex": "",
"regexApplyTo": "value",
"sort": "disabled",
"definition": "label_values(job)",
"options": [],
"multi": true,
"includeAll": true,
"allValue": ".+",
"allowCustomValue": true
}
},
{
"kind": "QueryVariable",
"spec": {
"name": "instance",
"current": {
"text": "All",
"value": [
"$__all"
]
},
"label": "instance",
"hide": "dontHide",
"refresh": "onDashboardLoad",
"skipUrlSync": false,
"query": {
"kind": "DataQuery",
"group": "prometheus",
"version": "v0",
"datasource": {
"name": "prometheus"
},
"spec": {
"query": "label_values({job=~\"$job\"},instance)",
"refId": "PrometheusVariableQueryEditor-VariableQuery"
}
},
"regex": "",
"regexApplyTo": "value",
"sort": "disabled",
"definition": "label_values({job=~\"$job\"},instance)",
"options": [],
"multi": true,
"includeAll": true,
"allValue": ".+",
"allowCustomValue": true
}
}
]
}
}5. 总结
本文介绍如何使用OpenTelemetry的OBI（eBPF Instrumentation）对Rust应用实现零侵入式监控。通过搭建基于Grafana LGTM的监控栈，结合OBI工具自动采集Rust Web服务的调用链数据，无需修改代码即可生成延迟、吞吐量、错误率等RED指标，并在Grafana中可视化展示，验证了eBPF在跨语言监控中的高效性与实用性。