MySQL 数据库性能监控:基于 mysqld_exporter 监控连接数、QPS、慢查询及主从复制延迟

MySQL 数据库性能监控:基于 mysqld_exporter 监控连接数、QPS、慢查询及主从复制延迟

业务方投诉"数据库慢",你登上服务器一看------CPU 20%、内存 30%、磁盘没满。一切正常,但应用就是卡。因为你看的是"服务器指标",不是"数据库指标"。mysqld_exporter 告诉你 MySQL 到底在经历什么。


一、一次"服务器正常"的数据库故障

"数据库又超时了。"

老李打开监控:CPU 20%,内存 3GB/16GB,磁盘 IO 5%,网络流量正常。服务器看起来"健康得不能再健康"。

他登录 MySQL 执行 SHOW PROCESSLIST------结果被震住了:300 多个活跃连接,80% 处于 Waiting for table metadata lock 状态。再查慢查询日志,一条原本 20ms 的 UPDATE 语句,现在跑了 15 秒还没结束。
#mermaid-svg-HCMGTsvd16CpZ9i8{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-HCMGTsvd16CpZ9i8 .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-HCMGTsvd16CpZ9i8 .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-HCMGTsvd16CpZ9i8 .error-icon{fill:#552222;}#mermaid-svg-HCMGTsvd16CpZ9i8 .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-HCMGTsvd16CpZ9i8 .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-HCMGTsvd16CpZ9i8 .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-HCMGTsvd16CpZ9i8 .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-HCMGTsvd16CpZ9i8 .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-HCMGTsvd16CpZ9i8 .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-HCMGTsvd16CpZ9i8 .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-HCMGTsvd16CpZ9i8 .marker{fill:#333333;stroke:#333333;}#mermaid-svg-HCMGTsvd16CpZ9i8 .marker.cross{stroke:#333333;}#mermaid-svg-HCMGTsvd16CpZ9i8 svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-HCMGTsvd16CpZ9i8 p{margin:0;}#mermaid-svg-HCMGTsvd16CpZ9i8 .label{font-family:"trebuchet ms",verdana,arial,sans-serif;color:#333;}#mermaid-svg-HCMGTsvd16CpZ9i8 .cluster-label text{fill:#333;}#mermaid-svg-HCMGTsvd16CpZ9i8 .cluster-label span{color:#333;}#mermaid-svg-HCMGTsvd16CpZ9i8 .cluster-label span p{background-color:transparent;}#mermaid-svg-HCMGTsvd16CpZ9i8 .label text,#mermaid-svg-HCMGTsvd16CpZ9i8 span{fill:#333;color:#333;}#mermaid-svg-HCMGTsvd16CpZ9i8 .node rect,#mermaid-svg-HCMGTsvd16CpZ9i8 .node circle,#mermaid-svg-HCMGTsvd16CpZ9i8 .node ellipse,#mermaid-svg-HCMGTsvd16CpZ9i8 .node polygon,#mermaid-svg-HCMGTsvd16CpZ9i8 .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-HCMGTsvd16CpZ9i8 .rough-node .label text,#mermaid-svg-HCMGTsvd16CpZ9i8 .node .label text,#mermaid-svg-HCMGTsvd16CpZ9i8 .image-shape .label,#mermaid-svg-HCMGTsvd16CpZ9i8 .icon-shape .label{text-anchor:middle;}#mermaid-svg-HCMGTsvd16CpZ9i8 .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-HCMGTsvd16CpZ9i8 .rough-node .label,#mermaid-svg-HCMGTsvd16CpZ9i8 .node .label,#mermaid-svg-HCMGTsvd16CpZ9i8 .image-shape .label,#mermaid-svg-HCMGTsvd16CpZ9i8 .icon-shape .label{text-align:center;}#mermaid-svg-HCMGTsvd16CpZ9i8 .node.clickable{cursor:pointer;}#mermaid-svg-HCMGTsvd16CpZ9i8 .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-HCMGTsvd16CpZ9i8 .arrowheadPath{fill:#333333;}#mermaid-svg-HCMGTsvd16CpZ9i8 .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-HCMGTsvd16CpZ9i8 .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-HCMGTsvd16CpZ9i8 .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-HCMGTsvd16CpZ9i8 .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-HCMGTsvd16CpZ9i8 .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-HCMGTsvd16CpZ9i8 .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-HCMGTsvd16CpZ9i8 .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-HCMGTsvd16CpZ9i8 .cluster text{fill:#333;}#mermaid-svg-HCMGTsvd16CpZ9i8 .cluster span{color:#333;}#mermaid-svg-HCMGTsvd16CpZ9i8 div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-HCMGTsvd16CpZ9i8 .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-HCMGTsvd16CpZ9i8 rect.text{fill:none;stroke-width:0;}#mermaid-svg-HCMGTsvd16CpZ9i8 .icon-shape,#mermaid-svg-HCMGTsvd16CpZ9i8 .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-HCMGTsvd16CpZ9i8 .icon-shape p,#mermaid-svg-HCMGTsvd16CpZ9i8 .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-HCMGTsvd16CpZ9i8 .icon-shape .label rect,#mermaid-svg-HCMGTsvd16CpZ9i8 .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-HCMGTsvd16CpZ9i8 .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-HCMGTsvd16CpZ9i8 .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-HCMGTsvd16CpZ9i8 :root{--mermaid-font-family:"trebuchet ms",verdana,arial,sans-serif;} MySQL 内部(实际已崩)
服务器监控(看起来正常)
假象
假象
假象
CPU 20% ✅
内存 3GB/16GB ✅
磁盘 IO 5% ✅
300+ 活跃连接
80% 等待 metadata lock
一条 UPDATE 跑了 15 秒

问题找到了,但更扎心的是------为什么这些数据库指标没有人提前监控?

服务器监控显示"正常",是因为它在看操作系统,而不是 MySQL。CPU 20% 说明 MySQL 没跑满?错------是因为大量连接被阻塞等待,线程根本没在工作。

这就需要 mysqld_exporter


二、mysqld_exporter 是什么?

mysqld_exporter 是 Prometheus 官方出品的 MySQL 监控工具,通过执行 SHOW STATUSSHOW VARIABLESSHOW SLAVE STATUS 等 SQL 语句,把 MySQL 的内部状态转换成 Prometheus 指标。

它能回答的四个核心问题

问题 mysqld_exporter 指标 为什么重要
数据库有多少人在用? mysql_global_status_threads_connected 连接数超限会导致拒绝服务
每秒处理多少请求? mysql_global_status_queries 计算 rate() QPS 突降说明应用挂了,突升说明被攻击
SQL 写得有多烂? mysql_global_status_slow_queries 慢查询侵蚀整体性能
主从同步还活着吗? mysql_slave_status_seconds_behind_master 主从延迟=读到旧数据

一句话价值:让 MySQL 从"黑盒"变成"透明盒"。


三、整体架构图


#mermaid-svg-8DgL39RCxrhiHv4M{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-8DgL39RCxrhiHv4M .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-8DgL39RCxrhiHv4M .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-8DgL39RCxrhiHv4M .error-icon{fill:#552222;}#mermaid-svg-8DgL39RCxrhiHv4M .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-8DgL39RCxrhiHv4M .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-8DgL39RCxrhiHv4M .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-8DgL39RCxrhiHv4M .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-8DgL39RCxrhiHv4M .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-8DgL39RCxrhiHv4M .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-8DgL39RCxrhiHv4M .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-8DgL39RCxrhiHv4M .marker{fill:#333333;stroke:#333333;}#mermaid-svg-8DgL39RCxrhiHv4M .marker.cross{stroke:#333333;}#mermaid-svg-8DgL39RCxrhiHv4M svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-8DgL39RCxrhiHv4M p{margin:0;}#mermaid-svg-8DgL39RCxrhiHv4M .label{font-family:"trebuchet ms",verdana,arial,sans-serif;color:#333;}#mermaid-svg-8DgL39RCxrhiHv4M .cluster-label text{fill:#333;}#mermaid-svg-8DgL39RCxrhiHv4M .cluster-label span{color:#333;}#mermaid-svg-8DgL39RCxrhiHv4M .cluster-label span p{background-color:transparent;}#mermaid-svg-8DgL39RCxrhiHv4M .label text,#mermaid-svg-8DgL39RCxrhiHv4M span{fill:#333;color:#333;}#mermaid-svg-8DgL39RCxrhiHv4M .node rect,#mermaid-svg-8DgL39RCxrhiHv4M .node circle,#mermaid-svg-8DgL39RCxrhiHv4M .node ellipse,#mermaid-svg-8DgL39RCxrhiHv4M .node polygon,#mermaid-svg-8DgL39RCxrhiHv4M .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-8DgL39RCxrhiHv4M .rough-node .label text,#mermaid-svg-8DgL39RCxrhiHv4M .node .label text,#mermaid-svg-8DgL39RCxrhiHv4M .image-shape .label,#mermaid-svg-8DgL39RCxrhiHv4M .icon-shape .label{text-anchor:middle;}#mermaid-svg-8DgL39RCxrhiHv4M .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-8DgL39RCxrhiHv4M .rough-node .label,#mermaid-svg-8DgL39RCxrhiHv4M .node .label,#mermaid-svg-8DgL39RCxrhiHv4M .image-shape .label,#mermaid-svg-8DgL39RCxrhiHv4M .icon-shape .label{text-align:center;}#mermaid-svg-8DgL39RCxrhiHv4M .node.clickable{cursor:pointer;}#mermaid-svg-8DgL39RCxrhiHv4M .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-8DgL39RCxrhiHv4M .arrowheadPath{fill:#333333;}#mermaid-svg-8DgL39RCxrhiHv4M .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-8DgL39RCxrhiHv4M .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-8DgL39RCxrhiHv4M .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-8DgL39RCxrhiHv4M .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-8DgL39RCxrhiHv4M .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-8DgL39RCxrhiHv4M .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-8DgL39RCxrhiHv4M .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-8DgL39RCxrhiHv4M .cluster text{fill:#333;}#mermaid-svg-8DgL39RCxrhiHv4M .cluster span{color:#333;}#mermaid-svg-8DgL39RCxrhiHv4M div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-8DgL39RCxrhiHv4M .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-8DgL39RCxrhiHv4M rect.text{fill:none;stroke-width:0;}#mermaid-svg-8DgL39RCxrhiHv4M .icon-shape,#mermaid-svg-8DgL39RCxrhiHv4M .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-8DgL39RCxrhiHv4M .icon-shape p,#mermaid-svg-8DgL39RCxrhiHv4M .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-8DgL39RCxrhiHv4M .icon-shape .label rect,#mermaid-svg-8DgL39RCxrhiHv4M .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-8DgL39RCxrhiHv4M .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-8DgL39RCxrhiHv4M .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-8DgL39RCxrhiHv4M :root{--mermaid-font-family:"trebuchet ms",verdana,arial,sans-serif;} 可视化
存储与告警
采集层
MySQL 实例
MySQL Master

主库
MySQL Slave

从库
mysqld_exporter

主库采集
mysqld_exporter

从库采集
Prometheus

指标存储 + 告警规则
Grafana 仪表盘

Dashboard ID: 7362
告警通知

  • 蓝色 :MySQL 实例和 mysqld_exporter------每个数据库实例配一个 Exporter,负责采集内部指标
  • 橙色:Prometheus 采集层
  • 绿色:展示和告警层

注意:主从实例需要分别部署 Exporter,它们的指标各有侧重。


四、核心指标详解

4.1 连接数监控

连接数是最直接的"压力表"------连接数高不一定有问题,但连接数耗尽一定是灾难。
#mermaid-svg-oxdHWlm0FobX0NCc{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-oxdHWlm0FobX0NCc .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-oxdHWlm0FobX0NCc .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-oxdHWlm0FobX0NCc .error-icon{fill:#552222;}#mermaid-svg-oxdHWlm0FobX0NCc .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-oxdHWlm0FobX0NCc .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-oxdHWlm0FobX0NCc .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-oxdHWlm0FobX0NCc .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-oxdHWlm0FobX0NCc .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-oxdHWlm0FobX0NCc .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-oxdHWlm0FobX0NCc .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-oxdHWlm0FobX0NCc .marker{fill:#333333;stroke:#333333;}#mermaid-svg-oxdHWlm0FobX0NCc .marker.cross{stroke:#333333;}#mermaid-svg-oxdHWlm0FobX0NCc svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-oxdHWlm0FobX0NCc p{margin:0;}#mermaid-svg-oxdHWlm0FobX0NCc .label{font-family:"trebuchet ms",verdana,arial,sans-serif;color:#333;}#mermaid-svg-oxdHWlm0FobX0NCc .cluster-label text{fill:#333;}#mermaid-svg-oxdHWlm0FobX0NCc .cluster-label span{color:#333;}#mermaid-svg-oxdHWlm0FobX0NCc .cluster-label span p{background-color:transparent;}#mermaid-svg-oxdHWlm0FobX0NCc .label text,#mermaid-svg-oxdHWlm0FobX0NCc span{fill:#333;color:#333;}#mermaid-svg-oxdHWlm0FobX0NCc .node rect,#mermaid-svg-oxdHWlm0FobX0NCc .node circle,#mermaid-svg-oxdHWlm0FobX0NCc .node ellipse,#mermaid-svg-oxdHWlm0FobX0NCc .node polygon,#mermaid-svg-oxdHWlm0FobX0NCc .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-oxdHWlm0FobX0NCc .rough-node .label text,#mermaid-svg-oxdHWlm0FobX0NCc .node .label text,#mermaid-svg-oxdHWlm0FobX0NCc .image-shape .label,#mermaid-svg-oxdHWlm0FobX0NCc .icon-shape .label{text-anchor:middle;}#mermaid-svg-oxdHWlm0FobX0NCc .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-oxdHWlm0FobX0NCc .rough-node .label,#mermaid-svg-oxdHWlm0FobX0NCc .node .label,#mermaid-svg-oxdHWlm0FobX0NCc .image-shape .label,#mermaid-svg-oxdHWlm0FobX0NCc .icon-shape .label{text-align:center;}#mermaid-svg-oxdHWlm0FobX0NCc .node.clickable{cursor:pointer;}#mermaid-svg-oxdHWlm0FobX0NCc .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-oxdHWlm0FobX0NCc .arrowheadPath{fill:#333333;}#mermaid-svg-oxdHWlm0FobX0NCc .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-oxdHWlm0FobX0NCc .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-oxdHWlm0FobX0NCc .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-oxdHWlm0FobX0NCc .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-oxdHWlm0FobX0NCc .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-oxdHWlm0FobX0NCc .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-oxdHWlm0FobX0NCc .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-oxdHWlm0FobX0NCc .cluster text{fill:#333;}#mermaid-svg-oxdHWlm0FobX0NCc .cluster span{color:#333;}#mermaid-svg-oxdHWlm0FobX0NCc div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-oxdHWlm0FobX0NCc .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-oxdHWlm0FobX0NCc rect.text{fill:none;stroke-width:0;}#mermaid-svg-oxdHWlm0FobX0NCc .icon-shape,#mermaid-svg-oxdHWlm0FobX0NCc .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-oxdHWlm0FobX0NCc .icon-shape p,#mermaid-svg-oxdHWlm0FobX0NCc .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-oxdHWlm0FobX0NCc .icon-shape .label rect,#mermaid-svg-oxdHWlm0FobX0NCc .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-oxdHWlm0FobX0NCc .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-oxdHWlm0FobX0NCc .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-oxdHWlm0FobX0NCc :root{--mermaid-font-family:"trebuchet ms",verdana,arial,sans-serif;} 连接池状态


max_connections

最大连接数
threads_connected

已用连接数
threads_running

正在执行的连接
使用率 = connected / max_connections
使用率 > 80%?
✅ 正常
⚠️ 连接数告警

需扩容或排查泄漏

指标名 含义 正常值 告警阈值
mysql_global_status_threads_connected 当前连接数 低于 max_connections 的 80% > max_connections × 0.8
mysql_global_status_threads_running 正在执行的连接 越低越好,高说明阻塞 > 50% 的 connected
mysql_global_variables_max_connections 最大允许连接数 根据规格配置 配置不足时需要调大

PromQL 示例:连接数使用率

promql 复制代码
mysql_global_status_threads_connected / mysql_global_variables_max_connections

4.2 QPS/TPS 监控

QPS 是数据库吞吐量的核心指标。
#mermaid-svg-ixpWApBvyPZQCiwf{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-ixpWApBvyPZQCiwf .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-ixpWApBvyPZQCiwf .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-ixpWApBvyPZQCiwf .error-icon{fill:#552222;}#mermaid-svg-ixpWApBvyPZQCiwf .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-ixpWApBvyPZQCiwf .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-ixpWApBvyPZQCiwf .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-ixpWApBvyPZQCiwf .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-ixpWApBvyPZQCiwf .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-ixpWApBvyPZQCiwf .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-ixpWApBvyPZQCiwf .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-ixpWApBvyPZQCiwf .marker{fill:#333333;stroke:#333333;}#mermaid-svg-ixpWApBvyPZQCiwf .marker.cross{stroke:#333333;}#mermaid-svg-ixpWApBvyPZQCiwf svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-ixpWApBvyPZQCiwf p{margin:0;}#mermaid-svg-ixpWApBvyPZQCiwf .label{font-family:"trebuchet ms",verdana,arial,sans-serif;color:#333;}#mermaid-svg-ixpWApBvyPZQCiwf .cluster-label text{fill:#333;}#mermaid-svg-ixpWApBvyPZQCiwf .cluster-label span{color:#333;}#mermaid-svg-ixpWApBvyPZQCiwf .cluster-label span p{background-color:transparent;}#mermaid-svg-ixpWApBvyPZQCiwf .label text,#mermaid-svg-ixpWApBvyPZQCiwf span{fill:#333;color:#333;}#mermaid-svg-ixpWApBvyPZQCiwf .node rect,#mermaid-svg-ixpWApBvyPZQCiwf .node circle,#mermaid-svg-ixpWApBvyPZQCiwf .node ellipse,#mermaid-svg-ixpWApBvyPZQCiwf .node polygon,#mermaid-svg-ixpWApBvyPZQCiwf .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-ixpWApBvyPZQCiwf .rough-node .label text,#mermaid-svg-ixpWApBvyPZQCiwf .node .label text,#mermaid-svg-ixpWApBvyPZQCiwf .image-shape .label,#mermaid-svg-ixpWApBvyPZQCiwf .icon-shape .label{text-anchor:middle;}#mermaid-svg-ixpWApBvyPZQCiwf .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-ixpWApBvyPZQCiwf .rough-node .label,#mermaid-svg-ixpWApBvyPZQCiwf .node .label,#mermaid-svg-ixpWApBvyPZQCiwf .image-shape .label,#mermaid-svg-ixpWApBvyPZQCiwf .icon-shape .label{text-align:center;}#mermaid-svg-ixpWApBvyPZQCiwf .node.clickable{cursor:pointer;}#mermaid-svg-ixpWApBvyPZQCiwf .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-ixpWApBvyPZQCiwf .arrowheadPath{fill:#333333;}#mermaid-svg-ixpWApBvyPZQCiwf .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-ixpWApBvyPZQCiwf .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-ixpWApBvyPZQCiwf .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-ixpWApBvyPZQCiwf .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-ixpWApBvyPZQCiwf .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-ixpWApBvyPZQCiwf .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-ixpWApBvyPZQCiwf .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-ixpWApBvyPZQCiwf .cluster text{fill:#333;}#mermaid-svg-ixpWApBvyPZQCiwf .cluster span{color:#333;}#mermaid-svg-ixpWApBvyPZQCiwf div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-ixpWApBvyPZQCiwf .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-ixpWApBvyPZQCiwf rect.text{fill:none;stroke-width:0;}#mermaid-svg-ixpWApBvyPZQCiwf .icon-shape,#mermaid-svg-ixpWApBvyPZQCiwf .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-ixpWApBvyPZQCiwf .icon-shape p,#mermaid-svg-ixpWApBvyPZQCiwf .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-ixpWApBvyPZQCiwf .icon-shape .label rect,#mermaid-svg-ixpWApBvyPZQCiwf .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-ixpWApBvyPZQCiwf .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-ixpWApBvyPZQCiwf .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-ixpWApBvyPZQCiwf :root{--mermaid-font-family:"trebuchet ms",verdana,arial,sans-serif;} 是





QPS 变化
QPS 突降?
检查上游应用是否挂了
QPS 突升?
检查是否被攻击或代码变更
QPS 正常但业务慢?
检查慢查询
✅ 正常

promql 复制代码
# QPS(每秒查询数)
rate(mysql_global_status_queries[1m])

# TPS(每秒事务数,针对 InnoDB)
rate(mysql_global_status_innodb_row_lock_waits[1m])

趋势解读:

现象 可能原因
QPS 突降 上游应用可能挂了
QPS 突升 被攻击,或代码逻辑变化产生大量查询
QPS 正常但业务慢 单条查询变慢了(看慢查询)

4.3 慢查询监控

promql 复制代码
# 慢查询速率(过去5分钟)
rate(mysql_global_status_slow_queries[5m])

# 慢查询占总查询的比例
mysql_global_status_slow_queries / mysql_global_status_queries

经验值:慢查询占比超过 0.1%(千分之一)就需要关注,超过 1% 就是严重问题。

4.4 InnoDB 行锁与缓冲池

行锁等待是"数据库慢"的常见原因。
#mermaid-svg-8CGS51nR5uDcKWW4{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-8CGS51nR5uDcKWW4 .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-8CGS51nR5uDcKWW4 .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-8CGS51nR5uDcKWW4 .error-icon{fill:#552222;}#mermaid-svg-8CGS51nR5uDcKWW4 .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-8CGS51nR5uDcKWW4 .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-8CGS51nR5uDcKWW4 .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-8CGS51nR5uDcKWW4 .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-8CGS51nR5uDcKWW4 .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-8CGS51nR5uDcKWW4 .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-8CGS51nR5uDcKWW4 .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-8CGS51nR5uDcKWW4 .marker{fill:#333333;stroke:#333333;}#mermaid-svg-8CGS51nR5uDcKWW4 .marker.cross{stroke:#333333;}#mermaid-svg-8CGS51nR5uDcKWW4 svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-8CGS51nR5uDcKWW4 p{margin:0;}#mermaid-svg-8CGS51nR5uDcKWW4 .label{font-family:"trebuchet ms",verdana,arial,sans-serif;color:#333;}#mermaid-svg-8CGS51nR5uDcKWW4 .cluster-label text{fill:#333;}#mermaid-svg-8CGS51nR5uDcKWW4 .cluster-label span{color:#333;}#mermaid-svg-8CGS51nR5uDcKWW4 .cluster-label span p{background-color:transparent;}#mermaid-svg-8CGS51nR5uDcKWW4 .label text,#mermaid-svg-8CGS51nR5uDcKWW4 span{fill:#333;color:#333;}#mermaid-svg-8CGS51nR5uDcKWW4 .node rect,#mermaid-svg-8CGS51nR5uDcKWW4 .node circle,#mermaid-svg-8CGS51nR5uDcKWW4 .node ellipse,#mermaid-svg-8CGS51nR5uDcKWW4 .node polygon,#mermaid-svg-8CGS51nR5uDcKWW4 .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-8CGS51nR5uDcKWW4 .rough-node .label text,#mermaid-svg-8CGS51nR5uDcKWW4 .node .label text,#mermaid-svg-8CGS51nR5uDcKWW4 .image-shape .label,#mermaid-svg-8CGS51nR5uDcKWW4 .icon-shape .label{text-anchor:middle;}#mermaid-svg-8CGS51nR5uDcKWW4 .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-8CGS51nR5uDcKWW4 .rough-node .label,#mermaid-svg-8CGS51nR5uDcKWW4 .node .label,#mermaid-svg-8CGS51nR5uDcKWW4 .image-shape .label,#mermaid-svg-8CGS51nR5uDcKWW4 .icon-shape .label{text-align:center;}#mermaid-svg-8CGS51nR5uDcKWW4 .node.clickable{cursor:pointer;}#mermaid-svg-8CGS51nR5uDcKWW4 .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-8CGS51nR5uDcKWW4 .arrowheadPath{fill:#333333;}#mermaid-svg-8CGS51nR5uDcKWW4 .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-8CGS51nR5uDcKWW4 .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-8CGS51nR5uDcKWW4 .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-8CGS51nR5uDcKWW4 .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-8CGS51nR5uDcKWW4 .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-8CGS51nR5uDcKWW4 .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-8CGS51nR5uDcKWW4 .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-8CGS51nR5uDcKWW4 .cluster text{fill:#333;}#mermaid-svg-8CGS51nR5uDcKWW4 .cluster span{color:#333;}#mermaid-svg-8CGS51nR5uDcKWW4 div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-8CGS51nR5uDcKWW4 .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-8CGS51nR5uDcKWW4 rect.text{fill:none;stroke-width:0;}#mermaid-svg-8CGS51nR5uDcKWW4 .icon-shape,#mermaid-svg-8CGS51nR5uDcKWW4 .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-8CGS51nR5uDcKWW4 .icon-shape p,#mermaid-svg-8CGS51nR5uDcKWW4 .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-8CGS51nR5uDcKWW4 .icon-shape .label rect,#mermaid-svg-8CGS51nR5uDcKWW4 .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-8CGS51nR5uDcKWW4 .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-8CGS51nR5uDcKWW4 .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-8CGS51nR5uDcKWW4 :root{--mermaid-font-family:"trebuchet ms",verdana,arial,sans-serif;} 缓冲池命中率


read_requests

总读请求
buffer_pool_reads

从磁盘读取
命中率 = (1 - reads/requests) × 100
命中率 < 99%?
✅ 命中率正常
⬆️ 增加 buffer pool 大小

指标 含义 高值说明
mysql_global_status_innodb_row_lock_waits 行锁等待次数 事务设计有问题,或同一行被频繁争用
mysql_global_status_innodb_row_lock_time 行锁总等待时间 单位微秒,除以等待次数得平均耗时
mysql_global_status_innodb_buffer_pool_reads 从磁盘读的页数 高说明 buffer pool 太小
mysql_global_status_innodb_buffer_pool_read_requests 总读请求 用 reads/requests 得命中率

PromQL 示例:InnoDB 缓冲池命中率

promql 复制代码
(1 - (mysql_global_status_innodb_buffer_pool_reads / mysql_global_status_innodb_buffer_pool_read_requests)) * 100

命中率低于 99% 就该考虑增加 buffer pool 大小了。

4.5 主从复制延迟------最容易被忽略

主从延迟会导致"刚写入的数据读不到",是微服务架构里的隐形杀手。
#mermaid-svg-NdJdLsMn4w5Yupgx{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-NdJdLsMn4w5Yupgx .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-NdJdLsMn4w5Yupgx .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-NdJdLsMn4w5Yupgx .error-icon{fill:#552222;}#mermaid-svg-NdJdLsMn4w5Yupgx .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-NdJdLsMn4w5Yupgx .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-NdJdLsMn4w5Yupgx .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-NdJdLsMn4w5Yupgx .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-NdJdLsMn4w5Yupgx .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-NdJdLsMn4w5Yupgx .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-NdJdLsMn4w5Yupgx .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-NdJdLsMn4w5Yupgx .marker{fill:#333333;stroke:#333333;}#mermaid-svg-NdJdLsMn4w5Yupgx .marker.cross{stroke:#333333;}#mermaid-svg-NdJdLsMn4w5Yupgx svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-NdJdLsMn4w5Yupgx p{margin:0;}#mermaid-svg-NdJdLsMn4w5Yupgx .label{font-family:"trebuchet ms",verdana,arial,sans-serif;color:#333;}#mermaid-svg-NdJdLsMn4w5Yupgx .cluster-label text{fill:#333;}#mermaid-svg-NdJdLsMn4w5Yupgx .cluster-label span{color:#333;}#mermaid-svg-NdJdLsMn4w5Yupgx .cluster-label span p{background-color:transparent;}#mermaid-svg-NdJdLsMn4w5Yupgx .label text,#mermaid-svg-NdJdLsMn4w5Yupgx span{fill:#333;color:#333;}#mermaid-svg-NdJdLsMn4w5Yupgx .node rect,#mermaid-svg-NdJdLsMn4w5Yupgx .node circle,#mermaid-svg-NdJdLsMn4w5Yupgx .node ellipse,#mermaid-svg-NdJdLsMn4w5Yupgx .node polygon,#mermaid-svg-NdJdLsMn4w5Yupgx .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-NdJdLsMn4w5Yupgx .rough-node .label text,#mermaid-svg-NdJdLsMn4w5Yupgx .node .label text,#mermaid-svg-NdJdLsMn4w5Yupgx .image-shape .label,#mermaid-svg-NdJdLsMn4w5Yupgx .icon-shape .label{text-anchor:middle;}#mermaid-svg-NdJdLsMn4w5Yupgx .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-NdJdLsMn4w5Yupgx .rough-node .label,#mermaid-svg-NdJdLsMn4w5Yupgx .node .label,#mermaid-svg-NdJdLsMn4w5Yupgx .image-shape .label,#mermaid-svg-NdJdLsMn4w5Yupgx .icon-shape .label{text-align:center;}#mermaid-svg-NdJdLsMn4w5Yupgx .node.clickable{cursor:pointer;}#mermaid-svg-NdJdLsMn4w5Yupgx .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-NdJdLsMn4w5Yupgx .arrowheadPath{fill:#333333;}#mermaid-svg-NdJdLsMn4w5Yupgx .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-NdJdLsMn4w5Yupgx .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-NdJdLsMn4w5Yupgx .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-NdJdLsMn4w5Yupgx .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-NdJdLsMn4w5Yupgx .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-NdJdLsMn4w5Yupgx .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-NdJdLsMn4w5Yupgx .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-NdJdLsMn4w5Yupgx .cluster text{fill:#333;}#mermaid-svg-NdJdLsMn4w5Yupgx .cluster span{color:#333;}#mermaid-svg-NdJdLsMn4w5Yupgx div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-NdJdLsMn4w5Yupgx .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-NdJdLsMn4w5Yupgx rect.text{fill:none;stroke-width:0;}#mermaid-svg-NdJdLsMn4w5Yupgx .icon-shape,#mermaid-svg-NdJdLsMn4w5Yupgx .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-NdJdLsMn4w5Yupgx .icon-shape p,#mermaid-svg-NdJdLsMn4w5Yupgx .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-NdJdLsMn4w5Yupgx .icon-shape .label rect,#mermaid-svg-NdJdLsMn4w5Yupgx .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-NdJdLsMn4w5Yupgx .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-NdJdLsMn4w5Yupgx .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-NdJdLsMn4w5Yupgx :root{--mermaid-font-family:"trebuchet ms",verdana,arial,sans-serif;} 监控指标
复制链路
主从复制架构
MySQL Master

主库
MySQL Slave

从库
binlog

写入
IO 线程

拉取 binlog
relay log

中继日志
SQL 线程

回放
seconds_behind_master

延迟秒数
slave_io_running

IO 线程状态
slave_sql_running

SQL 线程状态

指标 含义 正常值 告警阈值
mysql_slave_status_seconds_behind_master 延迟秒数 0 > 10 秒,持续 5 分钟
mysql_slave_status_slave_sql_running SQL 线程状态 1 不为 1 说明同步中断
mysql_slave_status_slave_io_running IO 线程状态 1 不为 1 说明拉取 binlog 失败

注意:seconds_behind_master=0 不代表没有延迟------如果主库没写入,延迟就是 0,但 IO 线程可能已经断了。所以必须同时监控两个 Running 状态。


五、实战:部署与配置

5.1 创建 MySQL 监控用户

sql 复制代码
CREATE USER 'mysqld_exporter'@'%' IDENTIFIED BY 'strong_password';
GRANT PROCESS, REPLICATION CLIENT, SELECT ON *.* TO 'mysqld_exporter'@'%';
FLUSH PRIVILEGES;

5.2 部署 mysqld_exporter(Docker 方式)

bash 复制代码
docker run -d \
  --name mysqld_exporter \
  -p 9104:9104 \
  -e DATA_SOURCE_NAME="mysqld_exporter:strong_password@(mysql-master:3306)/" \
  prom/mysqld-exporter:latest

5.3 K8s 部署(Deployment + Service)

yaml 复制代码
apiVersion: apps/v1
kind: Deployment
metadata:
  name: mysqld-exporter-master
  namespace: monitoring
spec:
  replicas: 1
  selector:
    matchLabels:
      app: mysqld-exporter
      instance: master
  template:
    metadata:
      labels:
        app: mysqld-exporter
        instance: master
    spec:
      containers:
      - name: exporter
        image: prom/mysqld-exporter:latest
        args:
        - --collect.info_schema.processlist
        - --collect.info_schema.innodb_tablespaces
        - --collect.perf_schema.table_lock_waits
        env:
        - name: DATA_SOURCE_NAME
          value: "mysqld_exporter:password@(mysql-master:3306)/"
        ports:
        - containerPort: 9104

5.4 Prometheus 采集配置(ServiceMonitor)

yaml 复制代码
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: mysql-monitor
spec:
  selector:
    matchLabels:
      app: mysqld-exporter
  endpoints:
  - port: metrics
    interval: 15s
    namespaceSelector:
    matchNames:
    - monitoring

5.5 Grafana 导入仪表盘

Grafana 官方 MySQL 仪表盘 ID:7362 (最流行)或 6239

复制代码
登录 Grafana → + → Import
输入 Dashboard ID:7362
选择 Prometheus 数据源

六、告警规则配置

#mermaid-svg-WBf68pCDlKsg3eey{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-WBf68pCDlKsg3eey .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-WBf68pCDlKsg3eey .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-WBf68pCDlKsg3eey .error-icon{fill:#552222;}#mermaid-svg-WBf68pCDlKsg3eey .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-WBf68pCDlKsg3eey .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-WBf68pCDlKsg3eey .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-WBf68pCDlKsg3eey .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-WBf68pCDlKsg3eey .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-WBf68pCDlKsg3eey .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-WBf68pCDlKsg3eey .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-WBf68pCDlKsg3eey .marker{fill:#333333;stroke:#333333;}#mermaid-svg-WBf68pCDlKsg3eey .marker.cross{stroke:#333333;}#mermaid-svg-WBf68pCDlKsg3eey svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-WBf68pCDlKsg3eey p{margin:0;}#mermaid-svg-WBf68pCDlKsg3eey .label{font-family:"trebuchet ms",verdana,arial,sans-serif;color:#333;}#mermaid-svg-WBf68pCDlKsg3eey .cluster-label text{fill:#333;}#mermaid-svg-WBf68pCDlKsg3eey .cluster-label span{color:#333;}#mermaid-svg-WBf68pCDlKsg3eey .cluster-label span p{background-color:transparent;}#mermaid-svg-WBf68pCDlKsg3eey .label text,#mermaid-svg-WBf68pCDlKsg3eey span{fill:#333;color:#333;}#mermaid-svg-WBf68pCDlKsg3eey .node rect,#mermaid-svg-WBf68pCDlKsg3eey .node circle,#mermaid-svg-WBf68pCDlKsg3eey .node ellipse,#mermaid-svg-WBf68pCDlKsg3eey .node polygon,#mermaid-svg-WBf68pCDlKsg3eey .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-WBf68pCDlKsg3eey .rough-node .label text,#mermaid-svg-WBf68pCDlKsg3eey .node .label text,#mermaid-svg-WBf68pCDlKsg3eey .image-shape .label,#mermaid-svg-WBf68pCDlKsg3eey .icon-shape .label{text-anchor:middle;}#mermaid-svg-WBf68pCDlKsg3eey .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-WBf68pCDlKsg3eey .rough-node .label,#mermaid-svg-WBf68pCDlKsg3eey .node .label,#mermaid-svg-WBf68pCDlKsg3eey .image-shape .label,#mermaid-svg-WBf68pCDlKsg3eey .icon-shape .label{text-align:center;}#mermaid-svg-WBf68pCDlKsg3eey .node.clickable{cursor:pointer;}#mermaid-svg-WBf68pCDlKsg3eey .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-WBf68pCDlKsg3eey .arrowheadPath{fill:#333333;}#mermaid-svg-WBf68pCDlKsg3eey .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-WBf68pCDlKsg3eey .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-WBf68pCDlKsg3eey .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-WBf68pCDlKsg3eey .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-WBf68pCDlKsg3eey .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-WBf68pCDlKsg3eey .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-WBf68pCDlKsg3eey .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-WBf68pCDlKsg3eey .cluster text{fill:#333;}#mermaid-svg-WBf68pCDlKsg3eey .cluster span{color:#333;}#mermaid-svg-WBf68pCDlKsg3eey div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-WBf68pCDlKsg3eey .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-WBf68pCDlKsg3eey rect.text{fill:none;stroke-width:0;}#mermaid-svg-WBf68pCDlKsg3eey .icon-shape,#mermaid-svg-WBf68pCDlKsg3eey .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-WBf68pCDlKsg3eey .icon-shape p,#mermaid-svg-WBf68pCDlKsg3eey .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-WBf68pCDlKsg3eey .icon-shape .label rect,#mermaid-svg-WBf68pCDlKsg3eey .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-WBf68pCDlKsg3eey .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-WBf68pCDlKsg3eey .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-WBf68pCDlKsg3eey :root{--mermaid-font-family:"trebuchet ms",verdana,arial,sans-serif;} 是持续5分钟

是持续2分钟

是持续5分钟

是持续1分钟

是持续10分钟

是持续5分钟

Prometheus 告警评估
连接数使用率 > 80%?
🔔 MySQL_High_Connection_Count
慢查询速率 > 10/s?
🔔 MySQL_High_Slow_Queries
主从延迟 > 10秒?
🔔 MySQL_Replication_Delay
同步线程异常?
🔔 MySQL_Slave_Not_Running
缓冲池命中率 < 99%?
🔔 MySQL_Low_Buffer_Pool_Hit
行锁等待 > 10/s?
🔔 MySQL_High_Row_Lock_Waits
✅ 数据库健康

yaml 复制代码
groups:
- name: mysql-alerts
  rules:
  # 连接数超过最大连接的80%
  - alert: MySQL_High_Connection_Count
    expr: mysql_global_status_threads_connected / mysql_global_variables_max_connections > 0.8
    for: 5m
    annotations:
      summary: "连接数过高 {{ $value | humanizePercentage }}"

  # 慢查询速率过高
  - alert: MySQL_High_Slow_Queries
    expr: rate(mysql_global_status_slow_queries[5m]) > 10
    for: 2m
    annotations:
      summary: "慢查询过多,当前速率 {{ $value }}/s"

  # 主从延迟超过10秒
  - alert: MySQL_Replication_Delay
    expr: mysql_slave_status_seconds_behind_master > 10
    for: 5m
    annotations:
      summary: "主从延迟 {{ $value }}秒"

  # 主从同步线程中断
  - alert: MySQL_Slave_Not_Running
    expr: mysql_slave_status_slave_sql_running != 1 or mysql_slave_status_slave_io_running != 1
    for: 1m
    annotations:
      summary: "主从同步线程异常"

  # InnoDB缓冲池命中率低于99%
  - alert: MySQL_Low_Buffer_Pool_Hit
    expr: |
      (1 - (mysql_global_status_innodb_buffer_pool_reads /
            mysql_global_status_innodb_buffer_pool_read_requests)) * 100 < 99
    for: 10m
    annotations:
      summary: "缓冲池命中率 {{ $value }}%,考虑增加内存"

  # 行锁等待过多
  - alert: MySQL_High_Row_Lock_Waits
    expr: rate(mysql_global_status_innodb_row_lock_waits[5m]) > 10
    for: 5m
    annotations:
      summary: "行锁等待过多,检查事务设计"

七、总结

MySQL 监控和服务器监控是两回事。

服务器 CPU 20%,MySQL 可能已经挂了------因为连接都在等锁,线程不工作,CPU 自然低。

mysqld_exporter 的价值:把 MySQL 的"内出血"指标暴露出来。
#mermaid-svg-7Le2zkDqjGkb76Ua{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-7Le2zkDqjGkb76Ua .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-7Le2zkDqjGkb76Ua .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-7Le2zkDqjGkb76Ua .error-icon{fill:#552222;}#mermaid-svg-7Le2zkDqjGkb76Ua .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-7Le2zkDqjGkb76Ua .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-7Le2zkDqjGkb76Ua .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-7Le2zkDqjGkb76Ua .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-7Le2zkDqjGkb76Ua .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-7Le2zkDqjGkb76Ua .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-7Le2zkDqjGkb76Ua .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-7Le2zkDqjGkb76Ua .marker{fill:#333333;stroke:#333333;}#mermaid-svg-7Le2zkDqjGkb76Ua .marker.cross{stroke:#333333;}#mermaid-svg-7Le2zkDqjGkb76Ua svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-7Le2zkDqjGkb76Ua p{margin:0;}#mermaid-svg-7Le2zkDqjGkb76Ua .label{font-family:"trebuchet ms",verdana,arial,sans-serif;color:#333;}#mermaid-svg-7Le2zkDqjGkb76Ua .cluster-label text{fill:#333;}#mermaid-svg-7Le2zkDqjGkb76Ua .cluster-label span{color:#333;}#mermaid-svg-7Le2zkDqjGkb76Ua .cluster-label span p{background-color:transparent;}#mermaid-svg-7Le2zkDqjGkb76Ua .label text,#mermaid-svg-7Le2zkDqjGkb76Ua span{fill:#333;color:#333;}#mermaid-svg-7Le2zkDqjGkb76Ua .node rect,#mermaid-svg-7Le2zkDqjGkb76Ua .node circle,#mermaid-svg-7Le2zkDqjGkb76Ua .node ellipse,#mermaid-svg-7Le2zkDqjGkb76Ua .node polygon,#mermaid-svg-7Le2zkDqjGkb76Ua .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-7Le2zkDqjGkb76Ua .rough-node .label text,#mermaid-svg-7Le2zkDqjGkb76Ua .node .label text,#mermaid-svg-7Le2zkDqjGkb76Ua .image-shape .label,#mermaid-svg-7Le2zkDqjGkb76Ua .icon-shape .label{text-anchor:middle;}#mermaid-svg-7Le2zkDqjGkb76Ua .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-7Le2zkDqjGkb76Ua .rough-node .label,#mermaid-svg-7Le2zkDqjGkb76Ua .node .label,#mermaid-svg-7Le2zkDqjGkb76Ua .image-shape .label,#mermaid-svg-7Le2zkDqjGkb76Ua .icon-shape .label{text-align:center;}#mermaid-svg-7Le2zkDqjGkb76Ua .node.clickable{cursor:pointer;}#mermaid-svg-7Le2zkDqjGkb76Ua .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-7Le2zkDqjGkb76Ua .arrowheadPath{fill:#333333;}#mermaid-svg-7Le2zkDqjGkb76Ua .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-7Le2zkDqjGkb76Ua .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-7Le2zkDqjGkb76Ua .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-7Le2zkDqjGkb76Ua .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-7Le2zkDqjGkb76Ua .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-7Le2zkDqjGkb76Ua .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-7Le2zkDqjGkb76Ua .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-7Le2zkDqjGkb76Ua .cluster text{fill:#333;}#mermaid-svg-7Le2zkDqjGkb76Ua .cluster span{color:#333;}#mermaid-svg-7Le2zkDqjGkb76Ua div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-7Le2zkDqjGkb76Ua .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-7Le2zkDqjGkb76Ua rect.text{fill:none;stroke-width:0;}#mermaid-svg-7Le2zkDqjGkb76Ua .icon-shape,#mermaid-svg-7Le2zkDqjGkb76Ua .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-7Le2zkDqjGkb76Ua .icon-shape p,#mermaid-svg-7Le2zkDqjGkb76Ua .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-7Le2zkDqjGkb76Ua .icon-shape .label rect,#mermaid-svg-7Le2zkDqjGkb76Ua .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-7Le2zkDqjGkb76Ua .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-7Le2zkDqjGkb76Ua .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-7Le2zkDqjGkb76Ua :root{--mermaid-font-family:"trebuchet ms",verdana,arial,sans-serif;} 四个最该监控的信号
① 连接数使用率 > 80%

快要拒绝服务
② 慢查询速率 > 10/s

SQL 需要优化
③ 主从延迟 > 10秒

业务读到旧数据
④ 缓冲池命中率 < 99%

内存不足
扩容 / 排查连接泄漏
优化索引 / 重写 SQL
检查网络 / 从库性能
增加 buffer pool 内存

指标 告警条件 含义
线程连接数 / max_connections > 0.8 连接数快满了,能赶在拒绝服务前扩容
rate(mysql_global_status_slow_queries[5m]) > 10 SQL 写得有问题,需要优化索引或重写
mysql_slave_status_seconds_behind_master > 10 主从延迟,业务可能读到旧数据
1 - innodb_buffer_pool_reads/read_requests < 0.99 内存不足,需要给 MySQL 加内存

记住:数据库慢,别只看服务器------问 mysqld_exporter,它会告诉你真相。