目录
- 健康检查与就绪探针
-
- 引言
- [1. 健康检查基础概念](#1. 健康检查基础概念)
-
- [1.1 健康检查的重要性](#1.1 健康检查的重要性)
- [1.2 健康检查的类型](#1.2 健康检查的类型)
- [1.3 健康检查的演进](#1.3 健康检查的演进)
- [2. 探针设计与原理](#2. 探针设计与原理)
-
- [2.1 存活探针（Liveness Probe）](#2.1 存活探针（Liveness Probe）)
- [2.2 就绪探针（Readiness Probe）](#2.2 就绪探针（Readiness Probe）)
- [2.3 启动探针（Startup Probe）](#2.3 启动探针（Startup Probe）)
- [2.4 探针状态转换](#2.4 探针状态转换)
- [3. 健康检查架构设计](#3. 健康检查架构设计)
-
- [3.1 健康检查系统架构](#3.1 健康检查系统架构)
- [3.2 探针检查流程](#3.2 探针检查流程)
- [3.3 容错与降级策略](#3.3 容错与降级策略)
- [4. Python健康检查系统实现](#4. Python健康检查系统实现)
-
- [4.1 基础健康检查框架](#4.1 基础健康检查框架)
- [4.2 内置健康检查器实现](#4.2 内置健康检查器实现)
- [4.3 健康检查管理器](#4.3 健康检查管理器)
- [4.4 Web端点集成](#4.4 Web端点集成)
- [5. 高级特性实现](#5. 高级特性实现)
-
- [5.1 智能健康检查](#5.1 智能健康检查)
- [5.2 健康检查状态机](#5.2 健康检查状态机)
- [5.3 健康检查告警系统](#5.3 健康检查告警系统)
- [6. 配置与使用示例](#6. 配置与使用示例)
-
- [6.1 配置管理系统](#6.1 配置管理系统)
- [6.2 使用示例](#6.2 使用示例)
- [7. 测试与验证](#7. 测试与验证)
-
- [7.1 单元测试](#7.1 单元测试)
- [7.2 集成测试](#7.2 集成测试)
- [8. 生产环境部署](#8. 生产环境部署)
-
- [8.1 Kubernetes部署配置](#8.1 Kubernetes部署配置)
- [8.2 监控与告警配置](#8.2 监控与告警配置)
- [8.3 部署清单检查](#8.3 部署清单检查)
- [9. 总结与展望](#9. 总结与展望)
-
- [9.1 关键收获](#9.1 关键收获)
- [9.2 性能数据总结](#9.2 性能数据总结)
- [9.3 未来发展方向](#9.3 未来发展方向)
- 附录
-
- [A. 健康检查最佳实践](#A. 健康检查最佳实践)
- [B. 常见问题解答](#B. 常见问题解答)
- [C. 性能优化建议](#C. 性能优化建议)
『宝藏代码胶囊开张啦！』------ 我的 CodeCapsule 来咯！✨写代码不再头疼！我的新站点 CodeCapsule 主打一个 "白菜价"+"量身定制 "！无论是卡脖子的毕设/课设/文献复现 ，需要灵光一现的算法改进 ，还是想给项目加个"外挂"，这里都有便宜又好用的代码方案等你发现！低成本，高适配，助你轻松通关！速来围观 👉 CodeCapsule官网
健康检查与就绪探针
引言
在现代分布式系统和云原生架构中，健康检查和就绪探针是确保系统可靠性和弹性的关键组件。随着微服务、容器化和Kubernetes的普及，服务实例的动态性和故障恢复能力变得至关重要。据统计，合理配置的健康检查可以将系统可用性提升40%以上，并减少80%的级联故障。
本文深入探讨健康检查和就绪探针的设计原理、实现方法和最佳实践，提供完整的Python实现方案，帮助构建高可用的现代应用系统。
1. 健康检查基础概念
1.1 健康检查的重要性
健康检查是系统监控和自我修复的基础机制，主要价值体现在：
- 故障检测：快速发现故障实例
- 负载均衡：避免将流量路由到不健康的实例
- 自动恢复：触发自动重启或替换故障实例
- 优雅部署：确保新版本完全就绪后再接收流量
- 系统自愈：减少人工干预，提高系统韧性
1.2 健康检查的类型
健康检查体系 存活探针 Liveness 就绪探针 Readiness 启动探针 Startup 业务健康检查 检查应用是否运行 失败时重启容器 检查应用是否就绪 失败时停止流量 检查应用启动状态 保护慢启动应用 检查业务功能 检查外部依赖
1.3 健康检查的演进
健康检查技术经历了多个阶段的演进：
- 简单端口检查（2000s）：检查端口是否开放
- HTTP端点检查（2010s）：返回200状态码
- 依赖关系检查（2015s）：检查数据库、缓存等
- 业务逻辑检查（2018s）：检查核心业务流程
- 智能健康检查（2020s）：AI驱动，自适应阈值
2. 探针设计与原理
2.1 存活探针（Liveness Probe）
存活探针用于确定应用程序是否正在运行。如果存活探针失败，容器编排器（如Kubernetes）会杀死容器并重新启动它。
设计原则：
- 检查应用程序内部状态
- 失败时采取激进措施（重启）
- 避免过于敏感，防止频繁重启
数学表示 ：
设应用状态为 S S S，存活探针函数为 L ( S ) L(S) L(S)，则：
L ( S ) = { 1 if S ∈ HealthyStates 0 otherwise L(S) = \begin{cases} 1 & \text{if } S \in \text{HealthyStates} \\ 0 & \text{otherwise} \end{cases} L(S)={10if S∈HealthyStatesotherwise
连续失败次数阈值： F l i v e n e s s = max ( 1 , ⌊ T t i m e o u t T i n t e r v a l ⌋ ) F_{liveness} = \max(1, \lfloor \frac{T_{timeout}}{T_{interval}} \rfloor) Fliveness=max(1,⌊TintervalTtimeout⌋)
2.2 就绪探针（Readiness Probe）
就绪探针用于确定应用程序是否准备好接收流量。如果就绪探针失败，容器编排器会从服务负载均衡器中移除该实例。
设计原则：
- 检查外部依赖和初始化状态
- 失败时采取保守措施（停止流量）
- 比存活探针更严格
数学表示 ：
设依赖状态集合为 D = { d 1 , d 2 , . . . , d n } D = \{d_1, d_2, ..., d_n\} D={d1,d2,...,dn}，就绪探针函数为 R ( D ) R(D) R(D)，则：
R ( D ) = ∏ i = 1 n r i ( d i ) R(D) = \prod_{i=1}^{n} r_i(d_i) R(D)=i=1∏nri(di)
其中 r i ( d i ) r_i(d_i) ri(di) 是单个依赖的就绪状态：
r i ( d i ) = { 1 if d i is ready 0 otherwise r_i(d_i) = \begin{cases} 1 & \text{if } d_i \text{ is ready} \\ 0 & \text{otherwise} \end{cases} ri(di)={10if di is readyotherwise
2.3 启动探针（Startup Probe）
启动探针用于保护慢启动的应用程序。在启动探针成功之前，不会运行其他探针。
设计原则：
- 专门用于应用启动阶段
- 允许更长的检查间隔和超时时间
- 成功后移交控制权给其他探针
2.4 探针状态转换
容器启动 启动探针失败 容器重启 启动探针成功 Starting Healthy 存活探针成功 就绪探针失败 就绪探针恢复 存活探针失败 Running NotReady Failed style #ff9 #9f9
#ccf
#f99
3. 健康检查架构设计
3.1 健康检查系统架构
编排层 监控层 依赖层 检查层 应用层 Kubernetes Docker 负载均衡器 监控系统 告警系统 仪表盘 日志系统 数据库 缓存 消息队列 外部API 文件系统 存活检查器 就绪检查器 启动检查器 依赖检查器 业务检查器 健康检查端点 探针管理器 状态收集器 指标暴露器
3.2 探针检查流程
健康检查的完整流程包括以下阶段：
- 初始化阶段：加载配置，注册检查器
- 执行阶段：并行执行各项检查
- 聚合阶段：合并检查结果，应用逻辑
- 决策阶段：根据策略决定最终状态
- 响应阶段：返回适当的状态码和消息
- 监控阶段：记录指标，触发告警
3.3 容错与降级策略
在设计健康检查系统时，需要考虑以下容错策略：
- 超时控制：每个检查设置合理的超时时间
- 重试机制：对临时失败进行检查重试
- 降级策略：部分依赖失败时降级运行
- 缓存结果：对稳定依赖缓存检查结果
- 熔断机制：对频繁失败的依赖启用熔断
4. Python健康检查系统实现
4.1 基础健康检查框架
python
"""
健康检查与就绪探针系统
设计原则：
1. 模块化设计：支持多种类型的检查器
2. 异步执行：支持并发检查，提高性能
3. 状态管理：清晰的状态转换和生命周期
4. 配置驱动：支持动态配置和热更新
5. 监控集成：与监控系统无缝集成
"""
import asyncio
import logging
import time
import json
from typing import Dict, List, Optional, Any, Callable, Set
from enum import Enum, auto
from dataclasses import dataclass, field, asdict
from abc import ABC, abstractmethod
from datetime import datetime, timedelta
from contextlib import asynccontextmanager
import inspect
import functools
import hashlib
from concurrent.futures import ThreadPoolExecutor, TimeoutError
import threading
import socket
import ssl
import urllib.parse
# 配置日志
logging.basicConfig(
level=logging.INFO,
format='%(asctime)s - %(name)s - %(levelname)s - %(message)s'
)
logger = logging.getLogger(__name__)
class HealthStatus(Enum):
"""健康状态枚举"""
HEALTHY = "healthy" # 健康
UNHEALTHY = "unhealthy" # 不健康
DEGRADED = "degraded" # 降级运行
UNKNOWN = "unknown" # 未知状态
STARTING = "starting" # 启动中
@classmethod
def from_bool(cls, is_healthy: bool) -> 'HealthStatus':
"""从布尔值转换"""
return cls.HEALTHY if is_healthy else cls.UNHEALTHY
def is_healthy(self) -> bool:
"""判断是否健康"""
return self in [self.HEALTHY, self.DEGRADED]
class ProbeType(Enum):
"""探针类型枚举"""
LIVENESS = "liveness" # 存活探针
READINESS = "readiness" # 就绪探针
STARTUP = "startup" # 启动探针
CUSTOM = "custom" # 自定义探针
@property
def default_path(self) -> str:
"""默认端点路径"""
return f"/health/{self.value}"
class CheckSeverity(Enum):
"""检查严重性枚举"""
CRITICAL = "critical" # 关键检查 - 失败则整体失败
HIGH = "high" # 高优先级检查
MEDIUM = "medium" # 中优先级检查
LOW = "low" # 低优先级检查
@property
def weight(self) -> int:
"""权重值"""
weights = {
self.CRITICAL: 100,
self.HIGH: 70,
self.MEDIUM: 40,
self.LOW: 10
}
return weights[self]
@dataclass
class CheckResult:
"""检查结果"""
# 基础信息
check_name: str
severity: CheckSeverity
status: HealthStatus
timestamp: datetime = field(default_factory=datetime.now)
# 详细信息
message: str = ""
details: Dict[str, Any] = field(default_factory=dict)
error: Optional[str] = None
duration_ms: float = 0.0
# 性能指标
execution_time: Optional[datetime] = None
response_time: Optional[float] = None
def to_dict(self) -> Dict[str, Any]:
"""转换为字典"""
result = asdict(self)
result['timestamp'] = self.timestamp.isoformat()
if self.execution_time:
result['execution_time'] = self.execution_time.isoformat()
# 转换为基本类型
for key, value in result.items():
if isinstance(value, Enum):
result[key] = value.value
return result
@property
def is_successful(self) -> bool:
"""检查是否成功"""
return self.status in [HealthStatus.HEALTHY, HealthStatus.DEGRADED]
@property
def weight(self) -> int:
"""获取权重"""
return self.severity.weight if self.is_successful else 0
@dataclass
class HealthResponse:
"""健康检查响应"""
# 总体状态
status: HealthStatus
timestamp: datetime = field(default_factory=datetime.now)
# 检查结果
checks: Dict[str, CheckResult] = field(default_factory=dict)
# 聚合信息
overall_score: float = 0.0
total_checks: int = 0
successful_checks: int = 0
failed_checks: int = 0
degraded_checks: int = 0
# 元数据
service_name: str = "unknown"
service_version: str = "unknown"
instance_id: str = "unknown"
def __post_init__(self):
"""后初始化处理"""
self._aggregate_results()
def _aggregate_results(self):
"""聚合检查结果"""
self.total_checks = len(self.checks)
if self.total_checks == 0:
self.overall_score = 100.0 if self.status.is_healthy() else 0.0
return
# 统计各类检查
self.successful_checks = sum(
1 for r in self.checks.values()
if r.status == HealthStatus.HEALTHY
)
self.failed_checks = sum(
1 for r in self.checks.values()
if r.status == HealthStatus.UNHEALTHY
)
self.degraded_checks = sum(
1 for r in self.checks.values()
if r.status == HealthStatus.DEGRADED
)
# 计算加权分数
total_weight = sum(r.severity.weight for r in self.checks.values())
successful_weight = sum(r.weight for r in self.checks.values())
if total_weight > 0:
self.overall_score = (successful_weight / total_weight) * 100
else:
self.overall_score = 0.0
# 如果有关键检查失败，整体状态应为不健康
critical_failed = any(
r.severity == CheckSeverity.CRITICAL and r.status == HealthStatus.UNHEALTHY
for r in self.checks.values()
)
if critical_failed:
self.status = HealthStatus.UNHEALTHY
def to_dict(self) -> Dict[str, Any]:
"""转换为字典"""
result = asdict(self)
result['status'] = self.status.value
result['timestamp'] = self.timestamp.isoformat()
# 转换检查结果
result['checks'] = {
name: check.to_dict() for name, check in self.checks.items()
}
# 添加摘要信息
result['summary'] = {
'total_checks': self.total_checks,
'successful': self.successful_checks,
'failed': self.failed_checks,
'degraded': self.degraded_checks,
'score': round(self.overall_score, 2),
'is_healthy': self.status.is_healthy()
}
return result
def to_json(self, indent: Optional[int] = None) -> str:
"""转换为JSON"""
return json.dumps(self.to_dict(), indent=indent, ensure_ascii=False)
@property
def http_status_code(self) -> int:
"""获取HTTP状态码"""
if self.status == HealthStatus.HEALTHY:
return 200
elif self.status == HealthStatus.DEGRADED:
return 200 # 仍返回200，但状态为degraded
elif self.status == HealthStatus.UNHEALTHY:
return 503 # 服务不可用
else:
return 500 # 服务器错误
class HealthChecker(ABC):
"""健康检查器抽象基类"""
def __init__(
self,
name: str,
severity: CheckSeverity = CheckSeverity.MEDIUM,
timeout_seconds: float = 5.0,
enabled: bool = True
):
self.name = name
self.severity = severity
self.timeout_seconds = timeout_seconds
self.enabled = enabled
# 统计信息
self.execution_count = 0
self.success_count = 0
self.failure_count = 0
self.total_duration_ms = 0.0
self.last_execution: Optional[datetime] = None
self.last_status: Optional[HealthStatus] = None
@abstractmethod
async def _check(self) -> CheckResult:
"""执行检查的具体实现"""
pass
async def check(self) -> CheckResult:
"""执行健康检查（带超时和异常处理）"""
if not self.enabled:
return CheckResult(
check_name=self.name,
severity=self.severity,
status=HealthStatus.UNKNOWN,
message="检查器已禁用"
)
start_time = time.time()
self.execution_count += 1
self.last_execution = datetime.now()
try:
# 执行检查（带超时）
result = await asyncio.wait_for(
self._check(),
timeout=self.timeout_seconds
)
# 更新统计
duration_ms = (time.time() - start_time) * 1000
self.total_duration_ms += duration_ms
result.duration_ms = duration_ms
if result.is_successful:
self.success_count += 1
else:
self.failure_count += 1
self.last_status = result.status
return result
except asyncio.TimeoutError:
duration_ms = (time.time() - start_time) * 1000
self.failure_count += 1
result = CheckResult(
check_name=self.name,
severity=self.severity,
status=HealthStatus.UNHEALTHY,
message=f"检查超时（{self.timeout_seconds}s）",
duration_ms=duration_ms
)
self.last_status = result.status
return result
except Exception as e:
duration_ms = (time.time() - start_time) * 1000
self.failure_count += 1
result = CheckResult(
check_name=self.name,
severity=self.severity,
status=HealthStatus.UNHEALTHY,
message=f"检查异常: {str(e)}",
error=str(e),
duration_ms=duration_ms
)
self.last_status = result.status
return result
def get_stats(self) -> Dict[str, Any]:
"""获取统计信息"""
total = self.execution_count
return {
'name': self.name,
'enabled': self.enabled,
'execution_count': total,
'success_count': self.success_count,
'failure_count': self.failure_count,
'success_rate': (self.success_count / total * 100) if total > 0 else 0,
'avg_duration_ms': (self.total_duration_ms / total) if total > 0 else 0,
'last_execution': self.last_execution.isoformat() if self.last_execution else None,
'last_status': self.last_status.value if self.last_status else None
}
class HealthCheckRegistry:
"""健康检查注册表"""
def __init__(self):
self._checkers: Dict[str, HealthChecker] = {}
self._checker_groups: Dict[str, List[str]] = {}
self._default_groups = {
'critical': [],
'dependencies': [],
'infrastructure': [],
'business': []
}
def register(
self,
checker: HealthChecker,
groups: Optional[List[str]] = None
):
"""注册健康检查器"""
if checker.name in self._checkers:
logger.warning(f"健康检查器已存在: {checker.name}")
return
self._checkers[checker.name] = checker
# 添加到分组
if groups:
for group in groups:
if group not in self._checker_groups:
self._checker_groups[group] = []
self._checker_groups[group].append(checker.name)
def unregister(self, name: str):
"""取消注册健康检查器"""
if name in self._checkers:
del self._checkers[name]
# 从所有分组中移除
for group in self._checker_groups.values():
if name in group:
group.remove(name)
def get_checker(self, name: str) -> Optional[HealthChecker]:
"""获取检查器"""
return self._checkers.get(name)
def get_checkers(self, group: Optional[str] = None) -> List[HealthChecker]:
"""获取检查器列表"""
if group:
checker_names = self._checker_groups.get(group, [])
return [self._checkers[name] for name in checker_names if name in self._checkers]
else:
return list(self._checkers.values())
def get_all_checkers(self) -> Dict[str, HealthChecker]:
"""获取所有检查器"""
return self._checkers.copy()
async def run_checks(
self,
group: Optional[str] = None,
parallel: bool = True
) -> Dict[str, CheckResult]:
"""运行健康检查"""
checkers = self.get_checkers(group)
if not checkers:
return {}
if parallel:
# 并行执行
tasks = [checker.check() for checker in checkers]
results = await asyncio.gather(*tasks, return_exceptions=True)
# 处理结果
check_results = {}
for checker, result in zip(checkers, results):
if isinstance(result, Exception):
check_results[checker.name] = CheckResult(
check_name=checker.name,
severity=checker.severity,
status=HealthStatus.UNHEALTHY,
message=f"检查执行异常: {str(result)}",
error=str(result)
)
else:
check_results[checker.name] = result
else:
# 串行执行
check_results = {}
for checker in checkers:
result = await checker.check()
check_results[checker.name] = result
return check_results
def get_stats(self) -> Dict[str, Any]:
"""获取所有检查器的统计信息"""
stats = {}
for name, checker in self._checkers.items():
stats[name] = checker.get_stats()
return {
'total_checkers': len(self._checkers),
'checker_stats': stats,
'groups': {
group: len(checkers)
for group, checkers in self._checker_groups.items()
}
}4.2 内置健康检查器实现
python
"""
内置健康检查器实现
包含常见的健康检查类型
"""
import psutil
import os
import sys
import redis
import pymongo
import pymysql
import sqlite3
from sqlalchemy import create_engine, text
from kafka import KafkaProducer, KafkaConsumer
import aiohttp
import asyncpg
import ssl as ssl_module
class SystemHealthChecker(HealthChecker):
"""系统健康检查器"""
def __init__(
self,
name: str = "system",
severity: CheckSeverity = CheckSeverity.HIGH,
cpu_threshold: float = 90.0, # CPU使用率阈值
memory_threshold: float = 90.0, # 内存使用率阈值
disk_threshold: float = 90.0, # 磁盘使用率阈值
check_disk: bool = True,
**kwargs
):
super().__init__(name, severity, **kwargs)
self.cpu_threshold = cpu_threshold
self.memory_threshold = memory_threshold
self.disk_threshold = disk_threshold
self.check_disk = check_disk
async def _check(self) -> CheckResult:
"""检查系统资源"""
details = {}
warnings = []
try:
# 检查CPU使用率
cpu_percent = psutil.cpu_percent(interval=0.5)
details['cpu_percent'] = cpu_percent
if cpu_percent > self.cpu_threshold:
warnings.append(f"CPU使用率高: {cpu_percent:.1f}%")
# 检查内存使用率
memory = psutil.virtual_memory()
details['memory_percent'] = memory.percent
details['memory_available_gb'] = memory.available / (1024**3)
if memory.percent > self.memory_threshold:
warnings.append(f"内存使用率高: {memory.percent:.1f}%")
# 检查磁盘使用率
if self.check_disk:
disk_usage = psutil.disk_usage('/')
details['disk_percent'] = disk_usage.percent
details['disk_free_gb'] = disk_usage.free / (1024**3)
if disk_usage.percent > self.disk_threshold:
warnings.append(f"磁盘使用率高: {disk_usage.percent:.1f}%")
# 检查系统负载
load_avg = os.getloadavg()
details['load_avg_1min'] = load_avg[0]
details['load_avg_5min'] = load_avg[1]
details['load_avg_15min'] = load_avg[2]
# 确定状态
if warnings:
status = HealthStatus.DEGRADED
message = f"系统资源警告: {', '.join(warnings)}"
else:
status = HealthStatus.HEALTHY
message = "系统资源正常"
return CheckResult(
check_name=self.name,
severity=self.severity,
status=status,
message=message,
details=details
)
except Exception as e:
return CheckResult(
check_name=self.name,
severity=self.severity,
status=HealthStatus.UNHEALTHY,
message=f"系统检查失败: {str(e)}",
error=str(e),
details={'error': str(e)}
)
class DatabaseHealthChecker(HealthChecker):
"""数据库健康检查器"""
def __init__(
self,
name: str,
connection_url: str,
check_query: str = "SELECT 1",
severity: CheckSeverity = CheckSeverity.CRITICAL,
**kwargs
):
super().__init__(name, severity, **kwargs)
self.connection_url = connection_url
self.check_query = check_query
# 解析数据库类型
self.db_type = self._parse_db_type(connection_url)
def _parse_db_type(self, url: str) -> str:
"""解析数据库类型"""
url_lower = url.lower()
if url_lower.startswith('postgresql://') or url_lower.startswith('postgres://'):
return 'postgresql'
elif url_lower.startswith('mysql://') or url_lower.startswith('mariadb://'):
return 'mysql'
elif url_lower.startswith('sqlite://'):
return 'sqlite'
elif url_lower.startswith('mongodb://'):
return 'mongodb'
elif url_lower.startswith('redis://'):
return 'redis'
else:
return 'unknown'
async def _check(self) -> CheckResult:
"""检查数据库连接和查询"""
details = {
'db_type': self.db_type,
'connection_url': self._mask_credentials(self.connection_url)
}
try:
if self.db_type == 'postgresql':
result = await self._check_postgresql()
elif self.db_type == 'mysql':
result = await self._check_mysql()
elif self.db_type == 'sqlite':
result = await self._check_sqlite()
elif self.db_type == 'mongodb':
result = await self._check_mongodb()
elif self.db_type == 'redis':
result = await self._check_redis()
else:
# 通用SQLAlchemy检查
result = await self._check_sqlalchemy()
details.update(result.get('details', {}))
return CheckResult(
check_name=self.name,
severity=self.severity,
status=result['status'],
message=result['message'],
details=details
)
except Exception as e:
return CheckResult(
check_name=self.name,
severity=self.severity,
status=HealthStatus.UNHEALTHY,
message=f"数据库检查失败: {str(e)}",
error=str(e),
details=details
)
async def _check_postgresql(self) -> Dict[str, Any]:
"""检查PostgreSQL数据库"""
import asyncpg
try:
# 解析连接参数
parsed_url = urllib.parse.urlparse(self.connection_url)
# 创建连接
conn = await asyncpg.connect(
host=parsed_url.hostname or 'localhost',
port=parsed_url.port or 5432,
user=parsed_url.username,
password=parsed_url.password,
database=parsed_url.path.lstrip('/') if parsed_url.path else None,
ssl='require' if parsed_url.scheme == 'postgresql+ssl' else None
)
# 执行检查查询
start_time = time.time()
result = await conn.fetchval(self.check_query)
query_time = time.time() - start_time
# 获取数据库信息
version = await conn.fetchval('SELECT version()')
db_size = await conn.fetchval(
"SELECT pg_database_size(current_database())"
)
await conn.close()
return {
'status': HealthStatus.HEALTHY,
'message': f"PostgreSQL数据库正常 (版本: {version.split()[0]})",
'details': {
'version': version,
'database_size_bytes': db_size,
'query_time_seconds': query_time,
'check_result': result
}
}
except Exception as e:
raise Exception(f"PostgreSQL检查失败: {str(e)}")
async def _check_mysql(self) -> Dict[str, Any]:
"""检查MySQL数据库"""
# 使用线程池执行同步IO操作
loop = asyncio.get_event_loop()
def sync_check():
import pymysql
parsed_url = urllib.parse.urlparse(self.connection_url)
conn = pymysql.connect(
host=parsed_url.hostname or 'localhost',
port=parsed_url.port or 3306,
user=parsed_url.username,
password=parsed_url.password,
database=parsed_url.path.lstrip('/') if parsed_url.path else None,
charset='utf8mb4',
cursorclass=pymysql.cursors.DictCursor
)
try:
start_time = time.time()
with conn.cursor() as cursor:
cursor.execute(self.check_query)
result = cursor.fetchone()
query_time = time.time() - start_time
# 获取数据库信息
cursor.execute("SELECT VERSION() as version")
version_info = cursor.fetchone()
version = version_info['version']
cursor.execute(
"SELECT SUM(data_length + index_length) as size "
"FROM information_schema.TABLES "
"WHERE table_schema = DATABASE()"
)
size_info = cursor.fetchone()
db_size = size_info['size'] or 0
return {
'status': HealthStatus.HEALTHY,
'message': f"MySQL数据库正常 (版本: {version})",
'details': {
'version': version,
'database_size_bytes': db_size,
'query_time_seconds': query_time,
'check_result': result
}
}
finally:
conn.close()
try:
return await loop.run_in_executor(None, sync_check)
except Exception as e:
raise Exception(f"MySQL检查失败: {str(e)}")
async def _check_sqlite(self) -> Dict[str, Any]:
"""检查SQLite数据库"""
def sync_check():
parsed_url = urllib.parse.urlparse(self.connection_url)
db_path = parsed_url.path.lstrip('/')
if db_path == ':memory:' or not db_path:
db_path = ':memory:'
conn = sqlite3.connect(db_path)
try:
start_time = time.time()
cursor = conn.cursor()
cursor.execute(self.check_query)
result = cursor.fetchone()
query_time = time.time() - start_time
# 获取数据库信息
cursor.execute("SELECT sqlite_version()")
version = cursor.fetchone()[0]
# 获取数据库大小
if db_path != ':memory:':
import os
db_size = os.path.getsize(db_path)
else:
db_size = 0
return {
'status': HealthStatus.HEALTHY,
'message': f"SQLite数据库正常 (版本: {version})",
'details': {
'version': version,
'database_size_bytes': db_size,
'query_time_seconds': query_time,
'check_result': result
}
}
finally:
conn.close()
loop = asyncio.get_event_loop()
try:
return await loop.run_in_executor(None, sync_check)
except Exception as e:
raise Exception(f"SQLite检查失败: {str(e)}")
async def _check_mongodb(self) -> Dict[str, Any]:
"""检查MongoDB数据库"""
def sync_check():
import pymongo
from pymongo.errors import ConnectionFailure
client = pymongo.MongoClient(
self.connection_url,
serverSelectionTimeoutMS=5000
)
try:
start_time = time.time()
# 执行ping命令
client.admin.command('ping')
query_time = time.time() - start_time
# 获取服务器信息
server_info = client.server_info()
version = server_info.get('version', 'unknown')
# 获取数据库统计
db = client.get_database()
db_stats = db.command('dbStats')
return {
'status': HealthStatus.HEALTHY,
'message': f"MongoDB数据库正常 (版本: {version})",
'details': {
'version': version,
'database_size_bytes': db_stats.get('dataSize', 0),
'query_time_seconds': query_time,
'storage_engine': server_info.get('storageEngine', {})
}
}
except ConnectionFailure as e:
raise Exception(f"MongoDB连接失败: {str(e)}")
finally:
client.close()
loop = asyncio.get_event_loop()
try:
return await loop.run_in_executor(None, sync_check)
except Exception as e:
raise Exception(f"MongoDB检查失败: {str(e)}")
async def _check_redis(self) -> Dict[str, Any]:
"""检查Redis数据库"""
def sync_check():
import redis
parsed_url = urllib.parse.urlparse(self.connection_url)
# 解析Redis连接参数
kwargs = {
'host': parsed_url.hostname or 'localhost',
'port': parsed_url.port or 6379,
'db': 0
}
if parsed_url.path:
# 从路径解析数据库编号，例如 /1
try:
db_num = int(parsed_url.path.lstrip('/'))
kwargs['db'] = db_num
except ValueError:
pass
if parsed_url.password:
kwargs['password'] = parsed_url.password
# 创建Redis连接
client = redis.Redis(**kwargs, socket_connect_timeout=5)
try:
start_time = time.time()
# 执行ping命令
result = client.ping()
query_time = time.time() - start_time
if not result:
raise Exception("Redis ping命令失败")
# 获取Redis信息
info = client.info()
return {
'status': HealthStatus.HEALTHY,
'message': f"Redis正常 (版本: {info.get('redis_version', 'unknown')})",
'details': {
'version': info.get('redis_version'),
'used_memory_bytes': info.get('used_memory'),
'connected_clients': info.get('connected_clients'),
'query_time_seconds': query_time,
'check_result': result
}
}
except redis.ConnectionError as e:
raise Exception(f"Redis连接失败: {str(e)}")
finally:
client.close()
loop = asyncio.get_event_loop()
try:
return await loop.run_in_executor(None, sync_check)
except Exception as e:
raise Exception(f"Redis检查失败: {str(e)}")
async def _check_sqlalchemy(self) -> Dict[str, Any]:
"""使用SQLAlchemy进行通用检查"""
def sync_check():
from sqlalchemy import create_engine, text
from sqlalchemy.exc import SQLAlchemyError
engine = create_engine(self.connection_url, pool_pre_ping=True)
try:
start_time = time.time()
with engine.connect() as conn:
result = conn.execute(text(self.check_query))
row = result.fetchone()
query_time = time.time() - start_time
# 获取数据库信息
dialect = engine.dialect.name
driver = engine.dialect.driver
return {
'status': HealthStatus.HEALTHY,
'message': f"数据库正常 ({dialect} via {driver})",
'details': {
'dialect': dialect,
'driver': driver,
'query_time_seconds': query_time,
'check_result': str(row) if row else None
}
}
except SQLAlchemyError as e:
raise Exception(f"数据库检查失败: {str(e)}")
finally:
engine.dispose()
loop = asyncio.get_event_loop()
try:
return await loop.run_in_executor(None, sync_check)
except Exception as e:
raise Exception(f"SQLAlchemy检查失败: {str(e)}")
def _mask_credentials(self, url: str) -> str:
"""隐藏连接字符串中的凭据"""
try:
parsed = urllib.parse.urlparse(url)
if parsed.username or parsed.password:
# 替换密码为***
masked_netloc = parsed.hostname or ''
if parsed.port:
masked_netloc += f':{parsed.port}'
return urllib.parse.urlunparse((
parsed.scheme,
masked_netloc,
parsed.path,
parsed.params,
parsed.query,
parsed.fragment
))
return url
except:
return '***masked***'
class HTTPHealthChecker(HealthChecker):
"""HTTP服务健康检查器"""
def __init__(
self,
name: str,
url: str,
method: str = 'GET',
expected_status: int = 200,
timeout_seconds: float = 10.0,
verify_ssl: bool = True,
headers: Optional[Dict[str, str]] = None,
**kwargs
):
super().__init__(name, timeout_seconds=timeout_seconds, **kwargs)
self.url = url
self.method = method.upper()
self.expected_status = expected_status
self.verify_ssl = verify_ssl
self.headers = headers or {}
async def _check(self) -> CheckResult:
"""检查HTTP服务"""
details = {
'url': self.url,
'method': self.method,
'expected_status': self.expected_status
}
try:
timeout = aiohttp.ClientTimeout(total=self.timeout_seconds)
ssl_context = None if self.verify_ssl else ssl_module.SSLContext()
async with aiohttp.ClientSession(
timeout=timeout,
headers=self.headers
) as session:
start_time = time.time()
async with session.request(
self.method,
self.url,
ssl=ssl_context
) as response:
response_time = time.time() - start_time
# 读取响应体
response_body = await response.text()
details.update({
'actual_status': response.status,
'response_time_seconds': response_time,
'response_size_bytes': len(response_body),
'headers': dict(response.headers)
})
if response.status == self.expected_status:
status = HealthStatus.HEALTHY
message = f"HTTP服务正常 (状态码: {response.status})"
else:
status = HealthStatus.UNHEALTHY
message = (
f"HTTP服务异常: "
f"期望状态码 {self.expected_status}, "
f"实际状态码 {response.status}"
)
# 检查响应时间
if response_time > self.timeout_seconds * 0.8:
status = HealthStatus.DEGRADED
message = f"HTTP服务响应慢: {response_time:.2f}s"
return CheckResult(
check_name=self.name,
severity=self.severity,
status=status,
message=message,
details=details
)
except aiohttp.ClientError as e:
return CheckResult(
check_name=self.name,
severity=self.severity,
status=HealthStatus.UNHEALTHY,
message=f"HTTP连接失败: {str(e)}",
error=str(e),
details=details
)
except asyncio.TimeoutError:
return CheckResult(
check_name=self.name,
severity=self.severity,
status=HealthStatus.UNHEALTHY,
message=f"HTTP请求超时 ({self.timeout_seconds}s)",
details=details
)
except Exception as e:
return CheckResult(
check_name=self.name,
severity=self.severity,
status=HealthStatus.UNHEALTHY,
message=f"HTTP检查异常: {str(e)}",
error=str(e),
details=details
)
class PortHealthChecker(HealthChecker):
"""端口健康检查器"""
def __init__(
self,
name: str,
host: str = 'localhost',
port: int = 80,
timeout_seconds: float = 5.0,
**kwargs
):
super().__init__(name, timeout_seconds=timeout_seconds, **kwargs)
self.host = host
self.port = port
async def _check(self) -> CheckResult:
"""检查端口是否开放"""
details = {
'host': self.host,
'port': self.port
}
loop = asyncio.get_event_loop()
try:
start_time = time.time()
# 异步套接字连接
try:
reader, writer = await asyncio.wait_for(
asyncio.open_connection(self.host, self.port),
timeout=self.timeout_seconds
)
writer.close()
await writer.wait_closed()
response_time = time.time() - start_time
details['response_time_seconds'] = response_time
return CheckResult(
check_name=self.name,
severity=self.severity,
status=HealthStatus.HEALTHY,
message=f"端口 {self.port} 正常开放",
details=details
)
except (ConnectionRefusedError, OSError) as e:
response_time = time.time() - start_time
details['response_time_seconds'] = response_time
return CheckResult(
check_name=self.name,
severity=self.severity,
status=HealthStatus.UNHEALTHY,
message=f"端口 {self.port} 连接被拒绝: {str(e)}",
error=str(e),
details=details
)
except asyncio.TimeoutError:
return CheckResult(
check_name=self.name,
severity=self.severity,
status=HealthStatus.UNHEALTHY,
message=f"端口 {self.port} 连接超时 ({self.timeout_seconds}s)",
details=details
)
except Exception as e:
return CheckResult(
check_name=self.name,
severity=self.severity,
status=HealthStatus.UNHEALTHY,
message=f"端口检查异常: {str(e)}",
error=str(e),
details=details
)
class FileSystemHealthChecker(HealthChecker):
"""文件系统健康检查器"""
def __init__(
self,
name: str,
path: str,
check_type: str = 'exists', # 'exists', 'writable', 'size'
min_free_gb: float = 1.0,
**kwargs
):
super().__init__(name, **kwargs)
self.path = path
self.check_type = check_type
self.min_free_gb = min_free_gb
async def _check(self) -> CheckResult:
"""检查文件系统"""
details = {
'path': self.path,
'check_type': self.check_type
}
loop = asyncio.get_event_loop()
def sync_check():
import os
import stat
result = {
'status': HealthStatus.HEALTHY,
'message': '',
'details': details
}
try:
# 检查文件/目录是否存在
if not os.path.exists(self.path):
result['status'] = HealthStatus.UNHEALTHY
result['message'] = f"路径不存在: {self.path}"
return result
# 根据检查类型执行检查
if self.check_type == 'exists':
result['message'] = f"路径存在: {self.path}"
# 获取详细信息
stat_info = os.stat(self.path)
details.update({
'size_bytes': stat_info.st_size,
'modification_time': stat_info.st_mtime,
'is_file': os.path.isfile(self.path),
'is_directory': os.path.isdir(self.path)
})
elif self.check_type == 'writable':
# 检查是否可写
if not os.access(self.path, os.W_OK):
result['status'] = HealthStatus.UNHEALTHY
result['message'] = f"路径不可写: {self.path}"
else:
result['message'] = f"路径可写: {self.path}"
# 检查权限
stat_info = os.stat(self.path)
details['permissions'] = oct(stat_info.st_mode)[-3:]
elif self.check_type == 'size':
# 检查磁盘空间
if os.path.isfile(self.path):
# 文件大小检查
size_bytes = os.path.getsize(self.path)
details['size_bytes'] = size_bytes
result['message'] = f"文件大小: {size_bytes} 字节"
else:
# 目录磁盘空间检查
statvfs = os.statvfs(self.path)
free_bytes = statvfs.f_bavail * statvfs.f_frsize
free_gb = free_bytes / (1024**3)
details.update({
'free_bytes': free_bytes,
'free_gb': free_gb,
'total_bytes': statvfs.f_blocks * statvfs.f_frsize,
'used_percent': (1 - statvfs.f_bavail / statvfs.f_blocks) * 100
})
if free_gb < self.min_free_gb:
result['status'] = HealthStatus.UNHEALTHY
result['message'] = (
f"磁盘空间不足: {free_gb:.2f}GB 可用, "
f"需要至少 {self.min_free_gb}GB"
)
else:
result['message'] = f"磁盘空间充足: {free_gb:.2f}GB 可用"
return result
except Exception as e:
result['status'] = HealthStatus.UNHEALTHY
result['message'] = f"文件系统检查失败: {str(e)}"
result['error'] = str(e)
return result
try:
result = await loop.run_in_executor(None, sync_check)
return CheckResult(
check_name=self.name,
severity=self.severity,
status=result['status'],
message=result['message'],
error=result.get('error'),
details=result['details']
)
except Exception as e:
return CheckResult(
check_name=self.name,
severity=self.severity,
status=HealthStatus.UNHEALTHY,
message=f"文件系统检查异常: {str(e)}",
error=str(e),
details=details
)
class KafkaHealthChecker(HealthChecker):
"""Kafka健康检查器"""
def __init__(
self,
name: str,
bootstrap_servers: List[str],
topic: Optional[str] = None,
timeout_seconds: float = 10.0,
**kwargs
):
super().__init__(name, timeout_seconds=timeout_seconds, **kwargs)
self.bootstrap_servers = bootstrap_servers
self.topic = topic
async def _check(self) -> CheckResult:
"""检查Kafka集群"""
details = {
'bootstrap_servers': self.bootstrap_servers,
'topic': self.topic
}
def sync_check():
from kafka import KafkaAdminClient
from kafka.errors import KafkaError
try:
start_time = time.time()
# 创建管理客户端
admin_client = KafkaAdminClient(
bootstrap_servers=self.bootstrap_servers,
request_timeout_ms=int(self.timeout_seconds * 1000)
)
try:
# 列出主题
topics = admin_client.list_topics()
details['topics_count'] = len(topics)
details['topics'] = list(topics)
# 检查特定主题
if self.topic:
if self.topic not in topics:
return {
'status': HealthStatus.UNHEALTHY,
'message': f"主题不存在: {self.topic}",
'details': details
}
# 获取主题详情
from kafka.admin import ConfigResource, ConfigResourceType
config_resource = ConfigResource(
ConfigResourceType.TOPIC,
self.topic
)
configs = admin_client.describe_configs([config_resource])
for config in configs.values():
details['topic_config'] = {
k: v.value for k, v in config.items()
}
break
response_time = time.time() - start_time
details['response_time_seconds'] = response_time
return {
'status': HealthStatus.HEALTHY,
'message': f"Kafka集群正常 (主题数: {len(topics)})",
'details': details
}
finally:
admin_client.close()
except KafkaError as e:
raise Exception(f"Kafka检查失败: {str(e)}")
loop = asyncio.get_event_loop()
try:
result = await loop.run_in_executor(None, sync_check)
return CheckResult(
check_name=self.name,
severity=self.severity,
status=result['status'],
message=result['message'],
details=result['details']
)
except Exception as e:
return CheckResult(
check_name=self.name,
severity=self.severity,
status=HealthStatus.UNHEALTHY,
message=f"Kafka检查异常: {str(e)}",
error=str(e),
details=details
)
class CustomHealthChecker(HealthChecker):
"""自定义健康检查器"""
def __init__(
self,
name: str,
check_func: Callable[[], Any],
severity: CheckSeverity = CheckSeverity.MEDIUM,
**kwargs
):
super().__init__(name, severity, **kwargs)
self.check_func = check_func
# 分析函数签名
self.is_async = inspect.iscoroutinefunction(check_func)
async def _check(self) -> CheckResult:
"""执行自定义检查"""
details = {
'check_type': 'custom',
'is_async': self.is_async
}
try:
start_time = time.time()
if self.is_async:
# 异步函数
result = await self.check_func()
else:
# 同步函数 - 在线程池中执行
loop = asyncio.get_event_loop()
result = await loop.run_in_executor(None, self.check_func)
duration_ms = (time.time() - start_time) * 1000
details['execution_time_ms'] = duration_ms
# 根据返回类型判断结果
if isinstance(result, CheckResult):
# 直接返回CheckResult
result.duration_ms = duration_ms
result.details.update(details)
return result
elif isinstance(result, bool):
# 布尔值
status = HealthStatus.HEALTHY if result else HealthStatus.UNHEALTHY
message = "自定义检查通过" if result else "自定义检查失败"
return CheckResult(
check_name=self.name,
severity=self.severity,
status=status,
message=message,
details=details
)
elif isinstance(result, dict):
# 字典结果
status_str = result.get('status', 'healthy')
status = HealthStatus(status_str) if status_str in HealthStatus._value2member_map_ else HealthStatus.HEALTHY
details.update(result.get('details', {}))
return CheckResult(
check_name=self.name,
severity=self.severity,
status=status,
message=result.get('message', '自定义检查完成'),
details=details
)
else:
# 其他类型 - 尝试转换为字符串
details['raw_result'] = str(result)
return CheckResult(
check_name=self.name,
severity=self.severity,
status=HealthStatus.HEALTHY,
message="自定义检查完成",
details=details
)
except Exception as e:
return CheckResult(
check_name=self.name,
severity=self.severity,
status=HealthStatus.UNHEALTHY,
message=f"自定义检查失败: {str(e)}",
error=str(e),
details=details
)4.3 健康检查管理器
python
class HealthCheckManager:
"""健康检查管理器"""
def __init__(
self,
service_name: str = "unknown",
service_version: str = "unknown",
instance_id: Optional[str] = None
):
self.service_name = service_name
self.service_version = service_version
self.instance_id = instance_id or self._generate_instance_id()
# 注册表
self.registry = HealthCheckRegistry()
# 状态缓存
self._cache: Dict[str, HealthResponse] = {}
self._cache_ttl = 30 # 缓存30秒
self._cache_lock = threading.Lock()
# 历史记录
self._history: List[HealthResponse] = []
self._max_history = 100
# 探针配置
self._probe_configs: Dict[ProbeType, Dict[str, Any]] = {}
# 初始化默认检查器
self._init_default_checkers()
def _generate_instance_id(self) -> str:
"""生成实例ID"""
import socket
import os
hostname = socket.gethostname()
pid = os.getpid()
timestamp = int(time.time() * 1000)
return f"{hostname}-{pid}-{timestamp}"
def _init_default_checkers(self):
"""初始化默认检查器"""
# 系统检查器
system_checker = SystemHealthChecker(
name="system",
severity=CheckSeverity.HIGH
)
self.registry.register(system_checker, groups=['infrastructure'])
# 进程检查器
process_checker = CustomHealthChecker(
name="process",
severity=CheckSeverity.HIGH,
check_func=lambda: True # 进程存在检查
)
self.registry.register(process_checker, groups=['critical'])
def register_checker(
self,
checker: HealthChecker,
groups: Optional[List[str]] = None
):
"""注册健康检查器"""
self.registry.register(checker, groups)
def unregister_checker(self, name: str):
"""取消注册健康检查器"""
self.registry.unregister(name)
def configure_probe(
self,
probe_type: ProbeType,
check_groups: Optional[List[str]] = None,
check_names: Optional[List[str]] = None,
cache_ttl: int = 30,
parallel: bool = True
):
"""配置探针"""
self._probe_configs[probe_type] = {
'check_groups': check_groups or [],
'check_names': check_names or [],
'cache_ttl': cache_ttl,
'parallel': parallel
}
async def run_health_check(
self,
probe_type: ProbeType = ProbeType.LIVENESS,
force_refresh: bool = False
) -> HealthResponse:
"""运行健康检查"""
# 检查缓存
cache_key = f"{probe_type.value}_{self.instance_id}"
if not force_refresh:
with self._cache_lock:
cached = self._cache.get(cache_key)
if cached:
cache_age = (datetime.now() - cached.timestamp).total_seconds()
if cache_age < self._cache_ttl:
return cached
# 获取检查配置
probe_config = self._probe_configs.get(probe_type, {})
check_groups = probe_config.get('check_groups', [])
check_names = probe_config.get('check_names', [])
parallel = probe_config.get('parallel', True)
# 确定要运行的检查器
checkers_to_run = []
if check_names:
# 按名称指定检查器
for name in check_names:
checker = self.registry.get_checker(name)
if checker:
checkers_to_run.append(checker)
elif check_groups:
# 按分组指定检查器
for group in check_groups:
group_checkers = self.registry.get_checkers(group)
checkers_to_run.extend(group_checkers)
else:
# 运行所有检查器
checkers_to_run = self.registry.get_checkers()
# 去除重复
seen = set()
unique_checkers = []
for checker in checkers_to_run:
if checker.name not in seen:
seen.add(checker.name)
unique_checkers.append(checker)
# 运行检查
check_results = {}
if unique_checkers:
# 临时创建只包含指定检查器的注册表
temp_registry = HealthCheckRegistry()
for checker in unique_checkers:
temp_registry.register(checker)
# 运行检查
check_results = await temp_registry.run_checks(parallel=parallel)
# 构建响应
health_response = HealthResponse(
status=HealthStatus.HEALTHY, # 默认状态
checks=check_results,
service_name=self.service_name,
service_version=self.service_version,
instance_id=self.instance_id
)
# 更新缓存
with self._cache_lock:
self._cache[cache_key] = health_response
# 添加到历史记录
self._history.append(health_response)
if len(self._history) > self._max_history:
self._history.pop(0)
return health_response
async def get_liveness(self) -> HealthResponse:
"""获取存活状态"""
return await self.run_health_check(ProbeType.LIVENESS)
async def get_readiness(self) -> HealthResponse:
"""获取就绪状态"""
return await self.run_health_check(ProbeType.READINESS)
async def get_startup(self) -> HealthResponse:
"""获取启动状态"""
return await self.run_health_check(ProbeType.STARTUP)
async def get_detailed_health(self) -> HealthResponse:
"""获取详细健康状态"""
return await self.run_health_check(ProbeType.CUSTOM, force_refresh=True)
def get_history(self, limit: int = 10) -> List[HealthResponse]:
"""获取历史记录"""
return self._history[-limit:] if self._history else []
def get_checker_stats(self) -> Dict[str, Any]:
"""获取检查器统计信息"""
return self.registry.get_stats()
def get_overall_stats(self) -> Dict[str, Any]:
"""获取总体统计信息"""
history = self.get_history()
if not history:
return {
'total_checks': 0,
'average_score': 0,
'availability_percent': 0,
'last_status': 'unknown'
}
total_checks = sum(len(h.checks) for h in history)
total_score = sum(h.overall_score for h in history)
healthy_count = sum(1 for h in history if h.status.is_healthy())
return {
'total_checks': total_checks,
'average_score': total_score / len(history) if history else 0,
'availability_percent': (healthy_count / len(history)) * 100 if history else 0,
'last_status': history[-1].status.value if history else 'unknown',
'history_size': len(history)
}4.4 Web端点集成
python
from fastapi import FastAPI, HTTPException, Depends
from fastapi.responses import JSONResponse
import uvicorn
from typing import Optional
class HealthCheckAPI:
"""健康检查API"""
def __init__(
self,
manager: HealthCheckManager,
app: Optional[FastAPI] = None,
prefix: str = "/health"
):
self.manager = manager
self.app = app or FastAPI(title="健康检查API")
self.prefix = prefix.rstrip('/')
# 注册路由
self._register_routes()
def _register_routes(self):
"""注册API路由"""
@self.app.get(f"{self.prefix}/liveness")
async def liveness_probe():
"""存活探针"""
try:
health_response = await self.manager.get_liveness()
return JSONResponse(
content=health_response.to_dict(),
status_code=health_response.http_status_code
)
except Exception as e:
logger.error(f"存活探针错误: {e}")
return JSONResponse(
content={
"status": HealthStatus.UNHEALTHY.value,
"message": f"内部错误: {str(e)}",
"timestamp": datetime.now().isoformat()
},
status_code=500
)
@self.app.get(f"{self.prefix}/readiness")
async def readiness_probe():
"""就绪探针"""
try:
health_response = await self.manager.get_readiness()
return JSONResponse(
content=health_response.to_dict(),
status_code=health_response.http_status_code
)
except Exception as e:
logger.error(f"就绪探针错误: {e}")
return JSONResponse(
content={
"status": HealthStatus.UNHEALTHY.value,
"message": f"内部错误: {str(e)}",
"timestamp": datetime.now().isoformat()
},
status_code=500
)
@self.app.get(f"{self.prefix}/startup")
async def startup_probe():
"""启动探针"""
try:
health_response = await self.manager.get_startup()
return JSONResponse(
content=health_response.to_dict(),
status_code=health_response.http_status_code
)
except Exception as e:
logger.error(f"启动探针错误: {e}")
return JSONResponse(
content={
"status": HealthStatus.UNHEALTHY.value,
"message": f"内部错误: {str(e)}",
"timestamp": datetime.now().isoformat()
},
status_code=500
)
@self.app.get(f"{self.prefix}/detailed")
async def detailed_health():
"""详细健康检查"""
try:
health_response = await self.manager.get_detailed_health()
return JSONResponse(
content=health_response.to_dict(),
status_code=health_response.http_status_code
)
except Exception as e:
logger.error(f"详细健康检查错误: {e}")
return JSONResponse(
content={
"status": HealthStatus.UNHEALTHY.value,
"message": f"内部错误: {str(e)}",
"timestamp": datetime.now().isoformat()
},
status_code=500
)
@self.app.get(f"{self.prefix}/stats")
async def health_stats():
"""健康检查统计"""
try:
checker_stats = self.manager.get_checker_stats()
overall_stats = self.manager.get_overall_stats()
return {
"checker_stats": checker_stats,
"overall_stats": overall_stats,
"timestamp": datetime.now().isoformat()
}
except Exception as e:
logger.error(f"健康统计错误: {e}")
raise HTTPException(status_code=500, detail=str(e))
@self.app.get(f"{self.prefix}/history")
async def health_history(limit: int = 10):
"""健康检查历史"""
try:
history = self.manager.get_history(limit)
return {
"history": [h.to_dict() for h in history],
"limit": limit,
"total": len(history)
}
except Exception as e:
logger.error(f"健康历史错误: {e}")
raise HTTPException(status_code=500, detail=str(e))
@self.app.get(f"{self.prefix}")
async def health_root():
"""健康检查根路径"""
return {
"service": self.manager.service_name,
"version": self.manager.service_version,
"instance_id": self.manager.instance_id,
"endpoints": {
"liveness": f"{self.prefix}/liveness",
"readiness": f"{self.prefix}/readiness",
"startup": f"{self.prefix}/startup",
"detailed": f"{self.prefix}/detailed",
"stats": f"{self.prefix}/stats",
"history": f"{self.prefix}/history"
},
"timestamp": datetime.now().isoformat()
}
def run(
self,
host: str = "0.0.0.0",
port: int = 8080,
**kwargs
):
"""运行健康检查API服务器"""
uvicorn.run(
self.app,
host=host,
port=port,
**kwargs
)5. 高级特性实现
5.1 智能健康检查
python
class AdaptiveHealthChecker(HealthChecker):
"""自适应健康检查器"""
def __init__(
self,
name: str,
base_checker: HealthChecker,
adaptation_window: int = 10, # 观察窗口大小
failure_threshold: float = 0.7, # 失败率阈值
recovery_time: float = 60.0, # 恢复时间（秒）
**kwargs
):
super().__init__(name, **kwargs)
self.base_checker = base_checker
self.adaptation_window = adaptation_window
self.failure_threshold = failure_threshold
self.recovery_time = recovery_time
# 历史记录
self.check_history: List[bool] = []
self.last_failure_time: Optional[datetime] = None
self.degraded_mode = False
async def _check(self) -> CheckResult:
"""执行自适应检查"""
# 如果处于降级模式且未过恢复时间，跳过检查
if self.degraded_mode and self.last_failure_time:
time_since_failure = (datetime.now() - self.last_failure_time).total_seconds()
if time_since_failure < self.recovery_time:
return CheckResult(
check_name=self.name,
severity=self.base_checker.severity,
status=HealthStatus.DEGRADED,
message=f"检查器处于降级模式，跳过检查 (恢复时间剩余: {self.recovery_time - time_since_failure:.1f}s)",
details={
'degraded_mode': True,
'time_since_failure': time_since_failure,
'recovery_time': self.recovery_time
}
)
else:
# 恢复时间已过，退出降级模式
self.degraded_mode = False
self.last_failure_time = None
# 执行基础检查
base_result = await self.base_checker.check()
# 更新历史记录
self.check_history.append(base_result.is_successful)
if len(self.check_history) > self.adaptation_window:
self.check_history.pop(0)
# 计算失败率
if len(self.check_history) >= self.adaptation_window:
failure_rate = 1 - (sum(self.check_history) / len(self.check_history))
if failure_rate > self.failure_threshold:
# 进入降级模式
self.degraded_mode = True
self.last_failure_time = datetime.now()
# 返回降级结果（如果基础检查成功）
if base_result.is_successful:
return CheckResult(
check_name=self.name,
severity=self.base_checker.severity,
status=HealthStatus.DEGRADED,
message=f"检查器进入降级模式 (失败率: {failure_rate:.1%})",
details={
**base_result.details,
'degraded_mode': True,
'failure_rate': failure_rate,
'adaptation_window': self.adaptation_window
}
)
return base_result
def get_adaptation_stats(self) -> Dict[str, Any]:
"""获取自适应统计信息"""
if not self.check_history:
return {
'adaptation_window': self.adaptation_window,
'history_size': 0,
'failure_rate': 0.0,
'degraded_mode': self.degraded_mode
}
failure_rate = 1 - (sum(self.check_history) / len(self.check_history))
return {
'adaptation_window': self.adaptation_window,
'history_size': len(self.check_history),
'failure_rate': failure_rate,
'failure_threshold': self.failure_threshold,
'degraded_mode': self.degraded_mode,
'last_failure_time': self.last_failure_time.isoformat() if self.last_failure_time else None,
'time_in_degraded_mode': (
(datetime.now() - self.last_failure_time).total_seconds()
if self.last_failure_time and self.degraded_mode else 0
)
}
class CompositeHealthChecker(HealthChecker):
"""组合健康检查器"""
def __init__(
self,
name: str,
checkers: List[HealthChecker],
aggregation_strategy: str = 'worst_of', # 'worst_of', 'best_of', 'weighted'
weights: Optional[Dict[str, float]] = None,
**kwargs
):
super().__init__(name, **kwargs)
self.checkers = checkers
self.aggregation_strategy = aggregation_strategy
self.weights = weights or {}
# 确保所有检查器都有权重
for checker in self.checkers:
if checker.name not in self.weights:
self.weights[checker.name] = 1.0
async def _check(self) -> CheckResult:
"""执行组合检查"""
# 并行执行所有检查
tasks = [checker.check() for checker in self.checkers]
results = await asyncio.gather(*tasks, return_exceptions=True)
# 处理结果
check_results = {}
for checker, result in zip(self.checkers, results):
if isinstance(result, Exception):
check_results[checker.name] = CheckResult(
check_name=checker.name,
severity=checker.severity,
status=HealthStatus.UNHEALTHY,
message=f"检查执行异常: {str(result)}",
error=str(result)
)
else:
check_results[checker.name] = result
# 根据策略聚合结果
if self.aggregation_strategy == 'worst_of':
# 取最差结果
aggregated_result = self._aggregate_worst_of(check_results)
elif self.aggregation_strategy == 'best_of':
# 取最好结果
aggregated_result = self._aggregate_best_of(check_results)
elif self.aggregation_strategy == 'weighted':
# 加权聚合
aggregated_result = self._aggregate_weighted(check_results)
else:
# 默认使用最差结果
aggregated_result = self._aggregate_worst_of(check_results)
# 设置详细信息
aggregated_result.details['sub_checks'] = {
name: result.to_dict() for name, result in check_results.items()
}
aggregated_result.details['aggregation_strategy'] = self.aggregation_strategy
return aggregated_result
def _aggregate_worst_of(self, results: Dict[str, CheckResult]) -> CheckResult:
"""最差结果聚合"""
# 状态优先级: UNHEALTHY > DEGRADED > HEALTHY > UNKNOWN
status_priority = {
HealthStatus.UNHEALTHY: 4,
HealthStatus.DEGRADED: 3,
HealthStatus.HEALTHY: 2,
HealthStatus.UNKNOWN: 1
}
# 找到最差状态
worst_result = max(
results.values(),
key=lambda r: status_priority.get(r.status, 0)
)
# 收集所有失败消息
failed_messages = [
f"{name}: {r.message}"
for name, r in results.items()
if not r.is_successful
]
message = worst_result.message
if len(failed_messages) > 1:
message = f"多个检查失败: {'; '.join(failed_messages)}"
return CheckResult(
check_name=self.name,
severity=self.severity,
status=worst_result.status,
message=message,
details={'worst_check': worst_result.check_name}
)
def _aggregate_best_of(self, results: Dict[str, CheckResult]) -> CheckResult:
"""最好结果聚合"""
# 状态优先级: HEALTHY > DEGRADED > UNKNOWN > UNHEALTHY
status_priority = {
HealthStatus.HEALTHY: 4,
HealthStatus.DEGRADED: 3,
HealthStatus.UNKNOWN: 2,
HealthStatus.UNHEALTHY: 1
}
# 找到最好状态
best_result = max(
results.values(),
key=lambda r: status_priority.get(r.status, 0)
)
return CheckResult(
check_name=self.name,
severity=self.severity,
status=best_result.status,
message=f"最佳检查结果: {best_result.message}",
details={'best_check': best_result.check_name}
)
def _aggregate_weighted(self, results: Dict[str, CheckResult]) -> CheckResult:
"""加权聚合"""
# 状态数值
status_values = {
HealthStatus.HEALTHY: 100,
HealthStatus.DEGRADED: 50,
HealthStatus.UNKNOWN: 25,
HealthStatus.UNHEALTHY: 0
}
# 计算加权分数
total_weight = sum(self.weights.get(name, 1.0) for name in results.keys())
weighted_score = 0
for name, result in results.items():
weight = self.weights.get(name, 1.0)
status_value = status_values.get(result.status, 0)
weighted_score += (status_value * weight)
# 归一化到0-100
if total_weight > 0:
normalized_score = weighted_score / total_weight
else:
normalized_score = 0
# 确定最终状态
if normalized_score >= 90:
status = HealthStatus.HEALTHY
message = f"加权健康分数: {normalized_score:.1f}"
elif normalized_score >= 50:
status = HealthStatus.DEGRADED
message = f"加权健康分数较低: {normalized_score:.1f}"
else:
status = HealthStatus.UNHEALTHY
message = f"加权健康分数过低: {normalized_score:.1f}"
return CheckResult(
check_name=self.name,
severity=self.severity,
status=status,
message=message,
details={
'weighted_score': normalized_score,
'total_weight': total_weight,
'weights': self.weights
}
)5.2 健康检查状态机
python
class HealthStateMachine:
"""健康状态机"""
def __init__(
self,
manager: HealthCheckManager,
state_change_callback: Optional[Callable] = None
):
self.manager = manager
self.state_change_callback = state_change_callback
# 状态定义
self.states = {
'INITIALIZING': {
'transitions': ['STARTING', 'FAILED']
},
'STARTING': {
'transitions': ['READY', 'FAILED'],
'probe': ProbeType.STARTUP
},
'READY': {
'transitions': ['RUNNING', 'DEGRADED', 'FAILED'],
'probe': ProbeType.READINESS
},
'RUNNING': {
'transitions': ['DEGRADED', 'STOPPING', 'FAILED'],
'probe': ProbeType.LIVENESS
},
'DEGRADED': {
'transitions': ['RUNNING', 'STOPPING', 'FAILED'],
'probe': ProbeType.LIVENESS
},
'STOPPING': {
'transitions': ['STOPPED']
},
'STOPPED': {
'transitions': []
},
'FAILED': {
'transitions': ['RECOVERING', 'STOPPED']
},
'RECOVERING': {
'transitions': ['STARTING', 'FAILED']
}
}
# 当前状态
self.current_state = 'INITIALIZING'
self.previous_state = None
self.state_entry_time = datetime.now()
# 状态历史
self.state_history = []
self.max_history = 1000
# 状态统计
self.state_stats = {state: {'count': 0, 'total_time': 0} for state in self.states}
# 故障统计
self.failure_stats = {
'total_failures': 0,
'consecutive_failures': 0,
'last_failure_time': None,
'recovery_count': 0
}
# 监控线程
self.monitoring = False
self.monitor_thread = None
self.check_interval = 10 # 检查间隔（秒）
def can_transition(self, from_state: str, to_state: str) -> bool:
"""检查状态转换是否允许"""
if from_state not in self.states:
return False
return to_state in self.states[from_state]['transitions']
def transition(self, new_state: str) -> bool:
"""执行状态转换"""
if not self.can_transition(self.current_state, new_state):
logger.warning(
f"不允许的状态转换: {self.current_state} -> {new_state}"
)
return False
# 记录状态停留时间
if self.current_state in self.state_stats:
time_in_state = (datetime.now() - self.state_entry_time).total_seconds()
self.state_stats[self.current_state]['total_time'] += time_in_state
# 更新状态
self.previous_state = self.current_state
self.current_state = new_state
self.state_entry_time = datetime.now()
# 更新统计
if new_state in self.state_stats:
self.state_stats[new_state]['count'] += 1
# 记录状态历史
state_record = {
'state': new_state,
'previous_state': self.previous_state,
'timestamp': self.state_entry_time,
'metadata': {}
}
self.state_history.append(state_record)
if len(self.state_history) > self.max_history:
self.state_history.pop(0)
# 更新故障统计
if new_state == 'FAILED':
self.failure_stats['total_failures'] += 1
self.failure_stats['consecutive_failures'] += 1
self.failure_stats['last_failure_time'] = datetime.now()
elif new_state == 'RUNNING' and self.previous_state in ['FAILED', 'DEGRADED']:
self.failure_stats['consecutive_failures'] = 0
if self.previous_state == 'FAILED':
self.failure_stats['recovery_count'] += 1
# 调用回调函数
if self.state_change_callback:
try:
self.state_change_callback(
old_state=self.previous_state,
new_state=new_state,
transition_time=self.state_entry_time
)
except Exception as e:
logger.error(f"状态转换回调失败: {e}")
logger.info(f"状态转换: {self.previous_state} -> {new_state}")
return True
async def evaluate_state(self):
"""评估当前状态"""
# 获取健康检查结果
probe_type = self.states.get(self.current_state, {}).get('probe')
if not probe_type:
return
try:
health_response = await self.manager.run_health_check(probe_type)
# 根据健康检查结果决定状态转换
if health_response.status == HealthStatus.HEALTHY:
if self.current_state == 'DEGRADED':
self.transition('RUNNING')
elif self.current_state == 'FAILED':
self.transition('RECOVERING')
elif health_response.status == HealthStatus.DEGRADED:
if self.current_state == 'RUNNING':
self.transition('DEGRADED')
elif self.current_state == 'STARTING':
# 启动时降级，仍然进入READY状态
self.transition('READY')
elif health_response.status == HealthStatus.UNHEALTHY:
if self.current_state in ['RUNNING', 'DEGRADED', 'READY']:
self.transition('FAILED')
elif self.current_state == 'STARTING':
self.transition('FAILED')
elif health_response.status == HealthStatus.STARTING:
if self.current_state == 'INITIALIZING':
self.transition('STARTING')
except Exception as e:
logger.error(f"状态评估失败: {e}")
# 健康检查失败，转换为失败状态
if self.current_state in ['RUNNING', 'DEGRADED', 'READY', 'STARTING']:
self.transition('FAILED')
def start_monitoring(self):
"""启动状态监控"""
if self.monitoring:
return
self.monitoring = True
async def monitor_loop():
while self.monitoring:
try:
await self.evaluate_state()
await asyncio.sleep(self.check_interval)
except Exception as e:
logger.error(f"监控循环错误: {e}")
await asyncio.sleep(self.check_interval)
# 创建新事件循环
loop = asyncio.new_event_loop()
asyncio.set_event_loop(loop)
self.monitor_thread = threading.Thread(
target=lambda: loop.run_until_complete(monitor_loop()),
daemon=True
)
self.monitor_thread.start()
def stop_monitoring(self):
"""停止状态监控"""
self.monitoring = False
if self.monitor_thread and self.monitor_thread.is_alive():
self.monitor_thread.join(timeout=5)
def get_state_info(self) -> Dict[str, Any]:
"""获取状态信息"""
current_time = datetime.now()
time_in_current_state = (current_time - self.state_entry_time).total_seconds()
return {
'current_state': self.current_state,
'previous_state': self.previous_state,
'state_entry_time': self.state_entry_time.isoformat(),
'time_in_current_state': time_in_current_state,
'state_stats': {
state: {
'count': stats['count'],
'total_time': stats['total_time'],
'avg_time': stats['total_time'] / stats['count'] if stats['count'] > 0 else 0
}
for state, stats in self.state_stats.items()
},
'failure_stats': {
**self.failure_stats,
'last_failure_time': (
self.failure_stats['last_failure_time'].isoformat()
if self.failure_stats['last_failure_time'] else None
)
},
'history_size': len(self.state_history),
'monitoring_active': self.monitoring
}
def is_ready(self) -> bool:
"""是否就绪"""
return self.current_state in ['READY', 'RUNNING', 'DEGRADED']
def is_alive(self) -> bool:
"""是否存活"""
return self.current_state not in ['FAILED', 'STOPPED', 'STOPPING']
def get_recommended_action(self) -> str:
"""获取推荐操作"""
recommendations = {
'FAILED': '检查日志并重启服务',
'DEGRADED': '监控系统状态，检查资源使用',
'STARTING': '等待服务启动完成',
'RECOVERING': '监控恢复过程',
'STOPPING': '等待服务停止',
'STOPPED': '服务已停止，可安全关闭'
}
return recommendations.get(self.current_state, '无特殊操作')5.3 健康检查告警系统
python
class HealthAlertSystem:
"""健康检查告警系统"""
def __init__(
self,
manager: HealthCheckManager,
alert_rules: Optional[Dict[str, Dict]] = None
):
self.manager = manager
self.alert_rules = alert_rules or self._get_default_rules()
# 告警状态
self.active_alerts: Dict[str, Dict] = {}
self.alert_history: List[Dict] = []
self.max_history = 1000
# 告警抑制
self.suppressed_alerts: Set[str] = set()
self.suppression_rules: Dict[str, Dict] = {}
# 告警通知器
self.notifiers: List[Callable] = []
def _get_default_rules(self) -> Dict[str, Dict]:
"""获取默认告警规则"""
return {
'health_score_low': {
'description': '健康分数过低',
'condition': lambda stats: stats.get('average_score', 100) < 80,
'severity': 'warning',
'cooldown': 300, # 5分钟冷却
'check_interval': 60
},
'availability_low': {
'description': '可用性过低',
'condition': lambda stats: stats.get('availability_percent', 100) < 95,
'severity': 'critical',
'cooldown': 600,
'check_interval': 60
},
'consecutive_failures': {
'description': '连续健康检查失败',
'condition': lambda stats: stats.get('consecutive_failures', 0) >= 3,
'severity': 'critical',
'cooldown': 300,
'check_interval': 30
},
'system_resources_high': {
'description': '系统资源使用率高',
'condition': lambda checker_stats: any(
'cpu_percent' in stats.get('details', {}) and
stats['details']['cpu_percent'] > 90
for stats in checker_stats.values()
),
'severity': 'warning',
'cooldown': 300,
'check_interval': 60
}
}
def add_notifier(self, notifier: Callable):
"""添加告警通知器"""
self.notifiers.append(notifier)
def suppress_alert(self, alert_id: str, duration_seconds: int = 3600):
"""抑制告警"""
self.suppressed_alerts.add(alert_id)
# 设置定时取消抑制
def remove_suppression():
time.sleep(duration_seconds)
if alert_id in self.suppressed_alerts:
self.suppressed_alerts.remove(alert_id)
threading.Thread(target=remove_suppression, daemon=True).start()
async def check_alerts(self):
"""检查告警条件"""
# 获取健康检查统计
checker_stats = self.manager.get_checker_stats()
overall_stats = self.manager.get_overall_stats()
# 合并统计信息
all_stats = {
**overall_stats,
'checker_stats': checker_stats.get('checker_stats', {})
}
# 检查每个告警规则
current_time = time.time()
for rule_id, rule in self.alert_rules.items():
# 检查冷却时间
if rule_id in self.active_alerts:
last_triggered = self.active_alerts[rule_id].get('last_triggered', 0)
if current_time - last_triggered < rule.get('cooldown', 0):
continue
# 检查抑制状态
if rule_id in self.suppressed_alerts:
continue
# 评估告警条件
try:
condition_met = rule['condition'](all_stats)
if condition_met:
# 触发告警
await self._trigger_alert(rule_id, rule, all_stats)
elif rule_id in self.active_alerts:
# 条件不再满足，清除告警
await self._clear_alert(rule_id, rule, all_stats)
except Exception as e:
logger.error(f"告警规则评估失败 {rule_id}: {e}")
async def _trigger_alert(self, rule_id: str, rule: Dict, stats: Dict):
"""触发告警"""
current_time = time.time()
alert_data = {
'id': rule_id,
'rule': rule,
'severity': rule['severity'],
'description': rule['description'],
'triggered_at': current_time,
'last_triggered': current_time,
'stats': stats,
'status': 'active'
}
# 更新活跃告警
self.active_alerts[rule_id] = alert_data
# 添加到历史记录
self.alert_history.append(alert_data.copy())
if len(self.alert_history) > self.max_history:
self.alert_history.pop(0)
# 发送通知
await self._send_notifications(alert_data)
logger.warning(
f"告警触发: {rule_id} - {rule['description']} "
f"(严重性: {rule['severity']})"
)
async def _clear_alert(self, rule_id: str, rule: Dict, stats: Dict):
"""清除告警"""
if rule_id not in self.active_alerts:
return
current_time = time.time()
alert_data = self.active_alerts[rule_id]
# 更新告警状态
alert_data.update({
'cleared_at': current_time,
'duration_seconds': current_time - alert_data['triggered_at'],
'status': 'cleared'
})
# 从活跃告警中移除
del self.active_alerts[rule_id]
# 发送恢复通知
recovery_data = alert_data.copy()
recovery_data['description'] = f"告警恢复: {rule['description']}"
await self._send_notifications(recovery_data)
logger.info(
f"告警恢复: {rule_id} - {rule['description']} "
f"(持续时间: {alert_data['duration_seconds']:.1f}秒)"
)
async def _send_notifications(self, alert_data: Dict):
"""发送告警通知"""
for notifier in self.notifiers:
try:
await notifier(alert_data)
except Exception as e:
logger.error(f"告警通知发送失败: {e}")
def get_active_alerts(self) -> List[Dict]:
"""获取活跃告警"""
return list(self.active_alerts.values())
def get_alert_history(
self,
limit: int = 100,
severity: Optional[str] = None
) -> List[Dict]:
"""获取告警历史"""
history = self.alert_history
if severity:
history = [h for h in history if h.get('severity') == severity]
return history[-limit:] if history else []
def get_alert_stats(self) -> Dict[str, Any]:
"""获取告警统计"""
history_last_24h = [
h for h in self.alert_history
if time.time() - h.get('triggered_at', 0) <= 86400
]
return {
'active_alerts': len(self.active_alerts),
'total_alerts_24h': len(history_last_24h),
'suppressed_alerts': len(self.suppressed_alerts),
'alert_history_size': len(self.alert_history),
'alert_rules': len(self.alert_rules),
'notifiers': len(self.notifiers)
}6. 配置与使用示例
6.1 配置管理系统
python
import yaml
import toml
from pathlib import Path
class HealthCheckConfig:
"""健康检查配置管理器"""
CONFIG_SCHEMA = {
'type': 'object',
'properties': {
'version': {'type': 'string'},
'service': {
'type': 'object',
'properties': {
'name': {'type': 'string'},
'version': {'type': 'string'},
'instance_id': {'type': 'string'}
},
'required': ['name']
},
'probes': {
'type': 'object',
'properties': {
'liveness': {'$ref': '#/definitions/probe'},
'readiness': {'$ref': '#/definitions/probe'},
'startup': {'$ref': '#/definitions/probe'},
'detailed': {'$ref': '#/definitions/probe'}
}
},
'checkers': {
'type': 'array',
'items': {'$ref': '#/definitions/checker'}
},
'alerting': {
'type': 'object',
'properties': {
'enabled': {'type': 'boolean'},
'rules': {'type': 'object'}
}
}
},
'required': ['version', 'service'],
'definitions': {
'probe': {
'type': 'object',
'properties': {
'check_groups': {
'type': 'array',
'items': {'type': 'string'}
},
'check_names': {
'type': 'array',
'items': {'type': 'string'}
},
'cache_ttl': {'type': 'integer', 'minimum': 0},
'parallel': {'type': 'boolean'}
}
},
'checker': {
'type': 'object',
'properties': {
'name': {'type': 'string'},
'type': {'type': 'string'},
'enabled': {'type': 'boolean'},
'severity': {
'type': 'string',
'enum': ['critical', 'high', 'medium', 'low']
},
'timeout_seconds': {'type': 'number', 'minimum': 0.1},
'groups': {
'type': 'array',
'items': {'type': 'string'}
},
'config': {'type': 'object'}
},
'required': ['name', 'type']
}
}
}
def __init__(self, config_path: Optional[Union[str, Path]] = None):
self.config = {}
self.config_path = Path(config_path) if config_path else None
if config_path and Path(config_path).exists():
self.load_config(config_path)
else:
self._load_default_config()
def _load_default_config(self):
"""加载默认配置"""
self.config = {
'version': '1.0',
'service': {
'name': 'health-check-service',
'version': '1.0.0'
},
'probes': {
'liveness': {
'check_groups': ['critical'],
'cache_ttl': 10,
'parallel': True
},
'readiness': {
'check_groups': ['critical', 'dependencies'],
'cache_ttl': 30,
'parallel': True
},
'startup': {
'check_groups': ['critical'],
'cache_ttl': 5,
'parallel': False
},
'detailed': {
'check_groups': ['critical', 'dependencies', 'infrastructure', 'business'],
'cache_ttl': 60,
'parallel': True
}
},
'checkers': [
{
'name': 'system',
'type': 'system',
'enabled': True,
'severity': 'high',
'timeout_seconds': 5,
'groups': ['infrastructure'],
'config': {
'cpu_threshold': 90,
'memory_threshold': 90,
'disk_threshold': 90
}
},
{
'name': 'process',
'type': 'custom',
'enabled': True,
'severity': 'critical',
'timeout_seconds': 1,
'groups': ['critical'],
'config': {
'check_func': 'lambda: True'
}
}
]
}
def load_config(self, config_path: Union[str, Path]):
"""加载配置文件"""
config_path = Path(config_path)
if not config_path.exists():
raise FileNotFoundError(f"配置文件不存在: {config_path}")
# 根据文件扩展名确定格式
suffix = config_path.suffix.lower()
try:
with open(config_path, 'r', encoding='utf-8') as f:
content = f.read()
if suffix == '.json':
config = json.loads(content)
elif suffix in ['.yaml', '.yml']:
config = yaml.safe_load(content)
elif suffix == '.toml':
config = toml.loads(content)
else:
raise ValueError(f"不支持的配置文件格式: {suffix}")
# 验证配置（简化版）
if self._validate_config(config):
self.config = config
self.config_path = config_path
logger.info(f"配置文件加载成功: {config_path}")
else:
raise ValueError("配置文件验证失败")
except Exception as e:
logger.error(f"配置文件加载失败: {e}")
raise
def _validate_config(self, config: Dict) -> bool:
"""验证配置（简化实现）"""
required_keys = ['version', 'service']
for key in required_keys:
if key not in config:
logger.error(f"配置缺少必需键: {key}")
return False
# 检查服务配置
service_config = config.get('service', {})
if 'name' not in service_config:
logger.error("服务配置缺少name字段")
return False
return True
def create_manager_from_config(self) -> HealthCheckManager:
"""从配置创建健康检查管理器"""
service_config = self.config.get('service', {})
manager = HealthCheckManager(
service_name=service_config.get('name', 'unknown'),
service_version=service_config.get('version', 'unknown'),
instance_id=service_config.get('instance_id')
)
# 配置探针
probe_configs = self.config.get('probes', {})
for probe_name, probe_config in probe_configs.items():
try:
probe_type = ProbeType(probe_name)
manager.configure_probe(probe_type, **probe_config)
except ValueError:
logger.warning(f"未知的探针类型: {probe_name}")
# 创建和注册检查器
checkers_config = self.config.get('checkers', [])
for checker_config in checkers_config:
try:
checker = self._create_checker_from_config(checker_config)
if checker:
groups = checker_config.get('groups', [])
manager.register_checker(checker, groups)
except Exception as e:
logger.error(f"创建检查器失败 {checker_config.get('name')}: {e}")
return manager
def _create_checker_from_config(self, config: Dict) -> Optional[HealthChecker]:
"""从配置创建检查器"""
checker_type = config.get('type', '').lower()
checker_name = config.get('name', 'unnamed')
severity = CheckSeverity(config.get('severity', 'medium'))
timeout = config.get('timeout_seconds', 5.0)
enabled = config.get('enabled', True)
checker_config = config.get('config', {})
if not enabled:
logger.info(f"检查器已禁用: {checker_name}")
return None
try:
if checker_type == 'system':
return SystemHealthChecker(
name=checker_name,
severity=severity,
timeout_seconds=timeout,
**checker_config
)
elif checker_type == 'database':
return DatabaseHealthChecker(
name=checker_name,
severity=severity,
timeout_seconds=timeout,
**checker_config
)
elif checker_type == 'http':
return HTTPHealthChecker(
name=checker_name,
severity=severity,
timeout_seconds=timeout,
**checker_config
)
elif checker_type == 'port':
return PortHealthChecker(
name=checker_name,
severity=severity,
timeout_seconds=timeout,
**checker_config
)
elif checker_type == 'filesystem':
return FileSystemHealthChecker(
name=checker_name,
severity=severity,
timeout_seconds=timeout,
**checker_config
)
elif checker_type == 'kafka':
return KafkaHealthChecker(
name=checker_name,
severity=severity,
timeout_seconds=timeout,
**checker_config
)
elif checker_type == 'custom':
# 自定义检查器需要特殊处理check_func
check_func_str = checker_config.get('check_func')
if check_func_str:
# 安全地评估函数（生产环境应使用更安全的方式）
import ast
try:
# 解析为AST
tree = ast.parse(check_func_str, mode='eval')
# 限制允许的节点类型
allowed_nodes = {
ast.Expression, ast.Lambda, ast.Call,
ast.Name, ast.Constant, ast.Attribute,
ast.BinOp, ast.UnaryOp, ast.Compare,
ast.BoolOp, ast.Subscript, ast.Index
}
for node in ast.walk(tree):
if type(node) not in allowed_nodes:
raise ValueError(f"不允许的AST节点: {type(node).__name__}")
# 编译和执行
code = compile(tree, '<string>', 'eval')
check_func = eval(code, {'__builtins__': {}})
return CustomHealthChecker(
name=checker_name,
severity=severity,
timeout_seconds=timeout,
check_func=check_func
)
except Exception as e:
logger.error(f"解析自定义检查函数失败 {checker_name}: {e}")
return None
else:
logger.error(f"自定义检查器缺少check_func: {checker_name}")
return None
else:
logger.warning(f"未知的检查器类型: {checker_type}")
return None
except Exception as e:
logger.error(f"创建检查器失败 {checker_name} ({checker_type}): {e}")
return None
def save_config(self, config_path: Optional[Union[str, Path]] = None):
"""保存配置"""
save_path = Path(config_path) if config_path else self.config_path
if not save_path:
raise ValueError("未指定配置保存路径")
# 确保目录存在
save_path.parent.mkdir(parents=True, exist_ok=True)
# 根据文件扩展名确定格式
suffix = save_path.suffix.lower()
try:
with open(save_path, 'w', encoding='utf-8') as f:
if suffix == '.json':
json.dump(self.config, f, indent=2, ensure_ascii=False)
elif suffix in ['.yaml', '.yml']:
yaml.dump(self.config, f, default_flow_style=False, allow_unicode=True)
elif suffix == '.toml':
toml.dump(self.config, f)
else:
# 默认使用JSON
json.dump(self.config, f, indent=2, ensure_ascii=False)
logger.info(f"配置文件保存成功: {save_path}")
except Exception as e:
logger.error(f"配置文件保存失败: {e}")
raise6.2 使用示例
python
def health_check_system_demo():
"""健康检查系统演示"""
print("=" * 60)
print("健康检查与就绪探针系统演示")
print("=" * 60)
# 1. 基础使用
print("\n1. 基础使用")
print("-" * 40)
# 创建健康检查管理器
manager = HealthCheckManager(
service_name="demo-service",
service_version="1.0.0"
)
# 添加系统检查器
system_checker = SystemHealthChecker(
name="system_resources",
severity=CheckSeverity.HIGH,
cpu_threshold=95,
memory_threshold=95
)
manager.register_checker(system_checker, groups=['infrastructure'])
# 添加HTTP检查器（模拟）
http_checker = CustomHealthChecker(
name="api_health",
severity=CheckSeverity.CRITICAL,
check_func=lambda: {
'status': 'healthy',
'message': 'API服务正常',
'response_time': 0.1
}
)
manager.register_checker(http_checker, groups=['dependencies'])
# 配置探针
manager.configure_probe(
ProbeType.LIVENESS,
check_groups=['infrastructure'],
cache_ttl=10
)
manager.configure_probe(
ProbeType.READINESS,
check_groups=['infrastructure', 'dependencies'],
cache_ttl=30
)
# 运行健康检查
import asyncio
async def run_checks():
print("运行存活探针:")
liveness = await manager.get_liveness()
print(f" 状态: {liveness.status.value}")
print(f" 分数: {liveness.overall_score:.1f}")
print("\n运行就绪探针:")
readiness = await manager.get_readiness()
print(f" 状态: {readiness.status.value}")
print(f" 检查数: {readiness.total_checks}")
print("\n运行详细检查:")
detailed = await manager.get_detailed_health()
print(f" 状态: {detailed.status.value}")
print(f" 成功检查: {detailed.successful_checks}/{detailed.total_checks}")
return liveness, readiness, detailed
liveness, readiness, detailed = asyncio.run(run_checks())
# 2. Web API集成
print("\n2. Web API集成")
print("-" * 40)
# 创建API
api = HealthCheckAPI(manager, prefix="/api/health")
print(f"健康检查端点:")
print(f" 存活探针: GET {api.prefix}/liveness")
print(f" 就绪探针: GET {api.prefix}/readiness")
print(f" 启动探针: GET {api.prefix}/startup")
print(f" 详细检查: GET {api.prefix}/detailed")
print(f" 统计信息: GET {api.prefix}/stats")
# 3. 状态机演示
print("\n3. 状态机演示")
print("-" * 40)
state_machine = HealthStateMachine(manager)
print("初始状态:", state_machine.current_state)
# 模拟状态转换
state_machine.transition('STARTING')
print("启动状态:", state_machine.current_state)
state_machine.transition('READY')
print("就绪状态:", state_machine.current_state)
state_machine.transition('RUNNING')
print("运行状态:", state_machine.current_state)
state_info = state_machine.get_state_info()
print(f"状态统计: 总状态数 = {len(state_info['state_stats'])}")
# 4. 告警系统演示
print("\n4. 告警系统演示")
print("-" * 40)
alert_system = HealthAlertSystem(manager)
# 添加简单的控制台通知器
async def console_notifier(alert_data):
print(f"[告警] {alert_data['description']} (严重性: {alert_data['severity']})")
alert_system.add_notifier(console_notifier)
# 运行告警检查
async def check_alerts():
await alert_system.check_alerts()
active_alerts = alert_system.get_active_alerts()
print(f"活跃告警: {len(active_alerts)} 个")
if active_alerts:
for alert in active_alerts:
print(f" - {alert['description']}")
asyncio.run(check_alerts())
# 5. 配置管理演示
print("\n5. 配置管理演示")
print("-" * 40)
# 创建配置
config = HealthCheckConfig()
# 添加更多检查器配置
config.config['checkers'].extend([
{
'name': 'database_main',
'type': 'database',
'enabled': True,
'severity': 'critical',
'timeout_seconds': 5,
'groups': ['dependencies'],
'config': {
'connection_url': 'postgresql://user:pass@localhost:5432/main',
'check_query': 'SELECT 1'
}
},
{
'name': 'redis_cache',
'type': 'database',
'enabled': True,
'severity': 'high',
'timeout_seconds': 3,
'groups': ['dependencies'],
'config': {
'connection_url': 'redis://localhost:6379/0'
}
}
])
# 保存配置
config.save_config("health_check_config.yaml")
print("配置已保存到 health_check_config.yaml")
# 从配置创建管理器
config_manager = config.create_manager_from_config()
print(f"从配置创建的服务: {config_manager.service_name}")
# 6. 统计信息
print("\n6. 系统统计")
print("-" * 40)
checker_stats = manager.get_checker_stats()
overall_stats = manager.get_overall_stats()
print(f"检查器总数: {checker_stats['total_checkers']}")
print(f"总体可用性: {overall_stats['availability_percent']:.1f}%")
print(f"平均健康分数: {overall_stats['average_score']:.1f}")
print("\n演示完成!")
return manager, api, state_machine, alert_system
def production_health_check_setup():
"""生产环境健康检查设置"""
# 从配置文件加载
config = HealthCheckConfig("config/health_check.yaml")
# 创建管理器
manager = config.create_manager_from_config()
# 创建状态机
state_machine = HealthStateMachine(manager)
def state_change_callback(old_state, new_state, transition_time):
"""状态变化回调"""
logger.info(
f"服务状态变化: {old_state} -> {new_state} "
f"({transition_time.isoformat()})"
)
# 这里可以添加状态变化处理逻辑
# 例如：发送通知、更新监控指标等
state_machine.state_change_callback = state_change_callback
# 启动状态监控
state_machine.start_monitoring()
# 创建告警系统
alert_system = HealthAlertSystem(manager)
# 添加告警通知器（示例：发送到Slack）
async def slack_notifier(alert_data):
"""Slack告警通知器"""
# 这里实现Slack通知逻辑
pass
# 添加告警通知器（示例：发送邮件）
async def email_notifier(alert_data):
"""邮件告警通知器"""
# 这里实现邮件通知逻辑
pass
alert_system.add_notifier(slack_notifier)
alert_system.add_notifier(email_notifier)
# 创建API
api = HealthCheckAPI(manager, prefix="/health")
return {
'manager': manager,
'state_machine': state_machine,
'alert_system': alert_system,
'api': api
}
if __name__ == "__main__":
# 运行演示
demo_result = health_check_system_demo()
# 可以在这里启动API服务器
# demo_result[1].run(host="0.0.0.0", port=8080)7. 测试与验证
7.1 单元测试
python
import pytest
import asyncio
from unittest.mock import Mock, patch, AsyncMock
class TestHealthCheckSystem:
"""健康检查系统测试"""
@pytest.fixture
def mock_checker(self):
"""创建模拟检查器"""
checker = Mock(spec=HealthChecker)
checker.name = "test_checker"
checker.severity = CheckSeverity.MEDIUM
checker.timeout_seconds = 5.0
checker.enabled = True
# 模拟检查结果
check_result = CheckResult(
check_name="test_checker",
severity=CheckSeverity.MEDIUM,
status=HealthStatus.HEALTHY,
message="测试检查通过"
)
checker.check = AsyncMock(return_value=check_result)
return checker
@pytest.fixture
def health_manager(self):
"""创建健康检查管理器"""
return HealthCheckManager(
service_name="test-service",
service_version="1.0.0"
)
@pytest.mark.asyncio
async def test_health_check_registry(self, mock_checker):
"""测试健康检查注册表"""
registry = HealthCheckRegistry()
# 注册检查器
registry.register(mock_checker, groups=['test'])
# 获取检查器
checker = registry.get_checker("test_checker")
assert checker is not None
assert checker.name == "test_checker"
# 获取分组检查器
group_checkers = registry.get_checkers("test")
assert len(group_checkers) == 1
assert group_checkers[0].name == "test_checker"
# 运行检查
results = await registry.run_checks()
assert "test_checker" in results
assert results["test_checker"].status == HealthStatus.HEALTHY
# 取消注册
registry.unregister("test_checker")
assert registry.get_checker("test_checker") is None
@pytest.mark.asyncio
async def test_health_check_manager(self, health_manager, mock_checker):
"""测试健康检查管理器"""
# 注册检查器
health_manager.register_checker(mock_checker, groups=['critical'])
# 配置探针
health_manager.configure_probe(
ProbeType.LIVENESS,
check_groups=['critical']
)
# 运行存活探针
response = await health_manager.get_liveness()
assert response.status == HealthStatus.HEALTHY
assert len(response.checks) == 2 # 系统检查器 + 测试检查器
assert response.service_name == "test-service"
assert response.instance_id is not None
@pytest.mark.asyncio
async def test_database_health_checker(self):
"""测试数据库健康检查器"""
# 使用模拟数据库连接
with patch('asyncpg.connect') as mock_connect:
mock_conn = AsyncMock()
mock_conn.fetchval = AsyncMock(side_effect=[1, "PostgreSQL 14.0", 1024])
mock_conn.close = AsyncMock()
mock_connect.return_value = mock_conn
checker = DatabaseHealthChecker(
name="test_db",
connection_url="postgresql://test:test@localhost/test",
severity=CheckSeverity.CRITICAL
)
result = await checker.check()
assert result.status == HealthStatus.HEALTHY
assert "PostgreSQL" in result.message
assert result.details['db_type'] == 'postgresql'
@pytest.mark.asyncio
async def test_http_health_checker(self):
"""测试HTTP健康检查器"""
with patch('aiohttp.ClientSession') as mock_session:
mock_response = Mock()
mock_response.status = 200
mock_response.text = AsyncMock(return_value="OK")
mock_response.headers = {'Content-Type': 'application/json'}
mock_session_instance = AsyncMock()
mock_session_instance.__aenter__.return_value.request = AsyncMock(
return_value=mock_response
)
mock_session_instance.__aexit__ = AsyncMock()
mock_session.return_value = mock_session_instance
checker = HTTPHealthChecker(
name="test_api",
url="http://example.com/health",
expected_status=200
)
result = await checker.check()
assert result.status == HealthStatus.HEALTHY
assert result.details['actual_status'] == 200
def test_health_state_machine(self):
"""测试健康状态机"""
manager = Mock(spec=HealthCheckManager)
state_machine = HealthStateMachine(manager)
# 测试状态转换
assert state_machine.current_state == 'INITIALIZING'
# 有效转换
assert state_machine.transition('STARTING') is True
assert state_machine.current_state == 'STARTING'
# 无效转换
assert state_machine.transition('RUNNING') is False
assert state_machine.current_state == 'STARTING'
# 有效转换
assert state_machine.transition('READY') is True
assert state_machine.current_state == 'READY'
# 获取状态信息
state_info = state_machine.get_state_info()
assert state_info['current_state'] == 'READY'
assert 'state_stats' in state_info
@pytest.mark.asyncio
async def test_adaptive_health_checker(self):
"""测试自适应健康检查器"""
# 创建模拟基础检查器
base_checker = Mock(spec=HealthChecker)
base_checker.name = "base_checker"
base_checker.severity = CheckSeverity.MEDIUM
# 第一次检查成功，第二次失败
success_result = CheckResult(
check_name="base_checker",
severity=CheckSeverity.MEDIUM,
status=HealthStatus.HEALTHY,
message="成功"
)
failure_result = CheckResult(
check_name="base_checker",
severity=CheckSeverity.MEDIUM,
status=HealthStatus.UNHEALTHY,
message="失败"
)
base_checker.check = AsyncMock(side_effect=[
success_result, failure_result, failure_result, failure_result
])
# 创建自适应检查器
adaptive_checker = AdaptiveHealthChecker(
name="adaptive_checker",
base_checker=base_checker,
adaptation_window=3,
failure_threshold=0.6 # 60%失败率触发降级
)
# 第一次检查
result1 = await adaptive_checker.check()
assert result1.status == HealthStatus.HEALTHY
assert not adaptive_checker.degraded_mode
# 第二次检查
result2 = await adaptive_checker.check()
assert result2.status == HealthStatus.UNHEALTHY
assert not adaptive_checker.degraded_mode # 尚未达到阈值
# 第三次检查
result3 = await adaptive_checker.check()
assert result3.status == HealthStatus.UNHEALTHY
assert adaptive_checker.degraded_mode # 达到阈值，进入降级模式
# 第四次检查（降级模式）
result4 = await adaptive_checker.check()
assert result4.status == HealthStatus.DEGRADED # 降级模式返回DEGRADED
# 获取自适应统计
stats = adaptive_checker.get_adaptation_stats()
assert stats['failure_rate'] == 1.0 # 全部失败
assert stats['degraded_mode'] is True
class TestHealthAlertSystem:
"""健康检查告警系统测试"""
@pytest.fixture
def alert_system(self):
"""创建告警系统"""
manager = Mock(spec=HealthCheckManager)
# 模拟统计信息
manager.get_checker_stats = Mock(return_value={
'total_checkers': 5,
'checker_stats': {}
})
manager.get_overall_stats = Mock(return_value={
'average_score': 75,
'availability_percent': 90,
'consecutive_failures': 2
})
return HealthAlertSystem(manager)
@pytest.mark.asyncio
async def test_alert_triggering(self, alert_system):
"""测试告警触发"""
# 添加控制台通知器
notifications = []
async def test_notifier(alert_data):
notifications.append(alert_data)
alert_system.add_notifier(test_notifier)
# 运行告警检查
await alert_system.check_alerts()
# 检查告警是否触发
active_alerts = alert_system.get_active_alerts()
# 根据规则，average_score < 80 应该触发告警
assert len(active_alerts) >= 1
# 检查通知是否发送
assert len(notifications) >= 1
# 检查告警内容
alert = active_alerts[0]
assert alert['severity'] in ['warning', 'critical']
assert 'description' in alert
def test_alert_suppression(self, alert_system):
"""测试告警抑制"""
alert_id = 'test_alert'
# 抑制告警
alert_system.suppress_alert(alert_id, duration_seconds=1)
# 检查是否被抑制
assert alert_id in alert_system.suppressed_alerts
# 等待抑制过期
import time
time.sleep(1.1)
# 检查是否自动取消抑制
assert alert_id not in alert_system.suppressed_alerts
def test_alert_stats(self, alert_system):
"""测试告警统计"""
stats = alert_system.get_alert_stats()
assert 'active_alerts' in stats
assert 'total_alerts_24h' in stats
assert 'suppressed_alerts' in stats
# 初始状态应该没有活跃告警
assert stats['active_alerts'] == 0
if __name__ == "__main__":
# 运行测试
pytest.main([__file__, '-v', '--tb=short'])7.2 集成测试
python
class IntegrationTestSuite:
"""集成测试套件"""
@staticmethod
async def test_complete_health_check_system():
"""测试完整的健康检查系统"""
print("=" * 60)
print("健康检查系统集成测试")
print("=" * 60)
test_results = {
'total': 0,
'passed': 0,
'failed': 0,
'tests': []
}
# 测试1：基本管理器功能
print("\n1. 测试基本管理器功能...")
try:
manager = HealthCheckManager(
service_name="integration-test",
service_version="1.0.0"
)
# 添加自定义检查器
def simple_check():
return True
custom_checker = CustomHealthChecker(
name="simple_check",
severity=CheckSeverity.MEDIUM,
check_func=simple_check
)
manager.register_checker(custom_checker)
# 运行健康检查
response = await manager.get_detailed_health()
if response.status.is_healthy():
print(" ✓ 基本管理器功能测试通过")
test_results['passed'] += 1
else:
print(f" ✗ 基本管理器功能测试失败: {response.status}")
test_results['failed'] += 1
test_results['total'] += 1
except Exception as e:
print(f" ✗ 基本管理器功能测试异常: {e}")
test_results['failed'] += 1
test_results['total'] += 1
# 测试2：Web API集成
print("\n2. 测试Web API集成...")
try:
manager = HealthCheckManager(
service_name="api-test",
service_version="1.0.0"
)
api = HealthCheckAPI(manager)
# 测试端点响应
import json
from fastapi.testclient import TestClient
client = TestClient(api.app)
# 测试存活探针
response = client.get("/health/liveness")
assert response.status_code in [200, 503]
# 测试就绪探针
response = client.get("/health/readiness")
assert response.status_code in [200, 503]
# 测试根路径
response = client.get("/health")
assert response.status_code == 200
data = response.json()
assert 'service' in data
assert 'endpoints' in data
print(" ✓ Web API集成测试通过")
test_results['passed'] += 1
test_results['total'] += 1
except Exception as e:
print(f" ✗ Web API集成测试异常: {e}")
test_results['failed'] += 1
test_results['total'] += 1
# 测试3：状态机功能
print("\n3. 测试状态机功能...")
try:
manager = HealthCheckManager(
service_name="state-machine-test",
service_version="1.0.0"
)
state_machine = HealthStateMachine(manager)
# 测试状态转换
assert state_machine.current_state == 'INITIALIZING'
# 有效转换
assert state_machine.transition('STARTING') is True
assert state_machine.current_state == 'STARTING'
# 无效转换
assert state_machine.transition('RUNNING') is False
assert state_machine.current_state == 'STARTING'
# 获取状态信息
state_info = state_machine.get_state_info()
assert 'current_state' in state_info
assert 'state_stats' in state_info
print(" ✓ 状态机功能测试通过")
test_results['passed'] += 1
test_results['total'] += 1
except Exception as e:
print(f" ✗ 状态机功能测试异常: {e}")
test_results['failed'] += 1
test_results['total'] += 1
# 测试4：配置管理
print("\n4. 测试配置管理...")
try:
import tempfile
import os
# 创建临时配置文件
config_content = {
'version': '1.0',
'service': {
'name': 'config-test',
'version': '1.0.0'
},
'probes': {
'liveness': {
'check_groups': ['critical'],
'cache_ttl': 10
}
},
'checkers': [
{
'name': 'test_checker',
'type': 'custom',
'enabled': True,
'severity': 'medium',
'config': {
'check_func': 'lambda: True'
}
}
]
}
with tempfile.NamedTemporaryFile(mode='w', suffix='.yaml', delete=False) as f:
import yaml
yaml.dump(config_content, f)
temp_file = f.name
try:
# 加载配置
config = HealthCheckConfig(temp_file)
# 从配置创建管理器
manager = config.create_manager_from_config()
assert manager.service_name == 'config-test'
assert len(manager.registry.get_all_checkers()) >= 1
print(" ✓ 配置管理测试通过")
test_results['passed'] += 1
finally:
# 清理临时文件
os.unlink(temp_file)
test_results['total'] += 1
except Exception as e:
print(f" ✗ 配置管理测试异常: {e}")
test_results['failed'] += 1
test_results['total'] += 1
# 测试5：告警系统
print("\n5. 测试告警系统...")
try:
manager = HealthCheckManager(
service_name="alert-test",
service_version="1.0.0"
)
alert_system = HealthAlertSystem(manager)
# 添加测试通知器
alerts_received = []
async def test_notifier(alert_data):
alerts_received.append(alert_data)
alert_system.add_notifier(test_notifier)
# 运行告警检查
await alert_system.check_alerts()
# 获取告警统计
stats = alert_system.get_alert_stats()
assert 'active_alerts' in stats
print(" ✓ 告警系统测试通过")
test_results['passed'] += 1
test_results['total'] += 1
except Exception as e:
print(f" ✗ 告警系统测试异常: {e}")
test_results['failed'] += 1
test_results['total'] += 1
# 输出测试结果
print("\n" + "=" * 60)
print("集成测试结果汇总:")
print("=" * 60)
print(f"总测试数: {test_results['total']}")
print(f"通过: {test_results['passed']}")
print(f"失败: {test_results['failed']}")
success_rate = (test_results['passed'] / test_results['total'] * 100
if test_results['total'] > 0 else 0)
print(f"成功率: {success_rate:.1f}%")
if test_results['failed'] == 0:
print("\n所有集成测试通过! ✓")
else:
print("\n有集成测试失败，请检查! ✗")
return test_results
if __name__ == "__main__":
# 运行集成测试
import asyncio
asyncio.run(IntegrationTestSuite.test_complete_health_check_system())8. 生产环境部署
8.1 Kubernetes部署配置
yaml
# kubernetes/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: health-check-service
namespace: production
labels:
app: health-check-service
version: v1.0.0
spec:
replicas: 3
selector:
matchLabels:
app: health-check-service
template:
metadata:
labels:
app: health-check-service
version: v1.0.0
spec:
containers:
- name: health-check-service
image: health-check-service:1.0.0
ports:
- containerPort: 8080
name: http
env:
- name: APP_ENV
value: "production"
- name: SERVICE_NAME
value: "health-check-service"
- name: SERVICE_VERSION
value: "1.0.0"
# 资源限制
resources:
requests:
memory: "128Mi"
cpu: "100m"
limits:
memory: "256Mi"
cpu: "200m"
# 健康检查配置
livenessProbe:
httpGet:
path: /health/liveness
port: 8080
initialDelaySeconds: 30
periodSeconds: 10
timeoutSeconds: 5
successThreshold: 1
failureThreshold: 3
readinessProbe:
httpGet:
path: /health/readiness
port: 8080
initialDelaySeconds: 5
periodSeconds: 5
timeoutSeconds: 3
successThreshold: 1
failureThreshold: 3
startupProbe:
httpGet:
path: /health/startup
port: 8080
initialDelaySeconds: 0
periodSeconds: 10
timeoutSeconds: 5
successThreshold: 1
failureThreshold: 30
# 安全上下文
securityContext:
runAsNonRoot: true
runAsUser: 1000
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
capabilities:
drop:
- ALL
# 生命周期钩子
lifecycle:
preStop:
exec:
command: ["/bin/sh", "-c", "sleep 10"]
# 卷挂载（用于配置）
volumeMounts:
- name: config-volume
mountPath: /app/config
readOnly: true
volumes:
- name: config-volume
configMap:
name: health-check-config
# 节点亲和性
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100
podAffinityTerm:
labelSelector:
matchExpressions:
- key: app
operator: In
values:
- health-check-service
topologyKey: kubernetes.io/hostname
# 安全设置
securityContext:
fsGroup: 1000
runAsNonRoot: true
---
# kubernetes/service.yaml
apiVersion: v1
kind: Service
metadata:
name: health-check-service
namespace: production
labels:
app: health-check-service
spec:
selector:
app: health-check-service
ports:
- name: http
port: 80
targetPort: 8080
protocol: TCP
type: ClusterIP
---
# kubernetes/configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: health-check-config
namespace: production
data:
health_check.yaml: |
version: "1.0"
service:
name: "health-check-service"
version: "1.0.0"
probes:
liveness:
check_groups: ["critical"]
cache_ttl: 10
parallel: true
readiness:
check_groups: ["critical", "dependencies"]
cache_ttl: 30
parallel: true
startup:
check_groups: ["critical"]
cache_ttl: 5
parallel: false
checkers:
- name: "system"
type: "system"
enabled: true
severity: "high"
timeout_seconds: 5
groups: ["infrastructure"]
config:
cpu_threshold: 90
memory_threshold: 90
disk_threshold: 90
- name: "database_main"
type: "database"
enabled: true
severity: "critical"
timeout_seconds: 10
groups: ["dependencies"]
config:
connection_url: "postgresql://${DB_USER}:${DB_PASSWORD}@${DB_HOST}:${DB_PORT}/${DB_NAME}"
check_query: "SELECT 1"8.2 监控与告警配置
yaml
# monitoring/prometheus-rules.yaml
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
name: health-check-rules
namespace: monitoring
spec:
groups:
- name: health-check
rules:
- alert: HealthCheckFailing
expr: |
health_check_status{probe="liveness"} != 1
for: 1m
labels:
severity: critical
service: "{{ $labels.service }}"
annotations:
summary: "健康检查失败 (服务: {{ $labels.service }})"
description: |
服务 {{ $labels.service }} 的健康检查已连续失败1分钟。
实例: {{ $labels.instance }}
探针: {{ $labels.probe }}
- alert: HealthCheckDegraded
expr: |
health_check_score < 80
for: 5m
labels:
severity: warning
service: "{{ $labels.service }}"
annotations:
summary: "健康分数过低 (服务: {{ $labels.service }})"
description: |
服务 {{ $labels.service }} 的健康分数低于80已持续5分钟。
当前分数: {{ $value }}
阈值: 80
- alert: HighErrorRate
expr: |
rate(health_check_errors_total[5m]) > 0.1
for: 2m
labels:
severity: warning
service: "{{ $labels.service }}"
annotations:
summary: "健康检查错误率过高 (服务: {{ $labels.service }})"
description: |
服务 {{ $labels.service }} 的健康检查错误率超过10%。
当前错误率: {{ $value | humanizePercentage }}
- alert: SlowHealthCheck
expr: |
health_check_duration_seconds > 5
for: 3m
labels:
severity: warning
service: "{{ $labels.service }}"
annotations:
summary: "健康检查响应慢 (服务: {{ $labels.service }})"
description: |
服务 {{ $labels.service }} 的健康检查响应时间超过5秒。
当前响应时间: {{ $value }}s8.3 部署清单检查
python
class DeploymentChecklist:
"""部署清单检查"""
CHECKLIST_ITEMS = [
{
'id': 'HC-001',
'category': '健康检查',
'description': '存活探针已配置',
'required': True,
'severity': 'critical'
},
{
'id': 'HC-002',
'category': '健康检查',
'description': '就绪探针已配置',
'required': True,
'severity': 'critical'
},
{
'id': 'HC-003',
'category': '健康检查',
'description': '启动探针已配置（慢启动应用）',
'required': False,
'severity': 'high'
},
{
'id': 'HC-004',
'category': '健康检查',
'description': '健康检查端点受保护',
'required': True,
'severity': 'high'
},
{
'id': 'HC-005',
'category': '健康检查',
'description': '健康检查配置了合理的超时时间',
'required': True,
'severity': 'medium'
},
{
'id': 'HC-006',
'category': '健康检查',
'description': '健康检查包含外部依赖检查',
'required': True,
'severity': 'high'
},
{
'id': 'HC-007',
'category': '监控',
'description': '健康检查指标已暴露',
'required': True,
'severity': 'high'
},
{
'id': 'HC-008',
'category': '监控',
'description': '配置了健康检查告警',
'required': True,
'severity': 'high'
},
{
'id': 'HC-009',
'category': '安全',
'description': '健康检查端点有速率限制',
'required': False,
'severity': 'medium'
},
{
'id': 'HC-010',
'category': '安全',
'description': '敏感信息已从健康检查响应中移除',
'required': True,
'severity': 'critical'
}
]
@classmethod
def verify_deployment(cls, deployment_config: Dict) -> Dict:
"""验证部署配置"""
results = []
for item in cls.CHECKLIST_ITEMS:
result = {
'id': item['id'],
'category': item['category'],
'description': item['description'],
'required': item['required'],
'severity': item['severity'],
'status': 'pending',
'details': ''
}
try:
# 执行检查
check_method = getattr(cls, f"_check_{item['id'].replace('-', '_')}")
passed, details = check_method(deployment_config)
result['status'] = 'passed' if passed else 'failed'
result['details'] = details
except AttributeError:
result['status'] = 'not_implemented'
result['details'] = '检查方法未实现'
except Exception as e:
result['status'] = 'error'
result['details'] = str(e)
results.append(result)
# 汇总结果
total_checks = len(results)
passed_checks = len([r for r in results if r['status'] == 'passed'])
failed_required = [
r for r in results
if r['status'] != 'passed' and r['required']
]
deployment_approved = len(failed_required) == 0
return {
'timestamp': datetime.now().isoformat(),
'results': results,
'summary': {
'total_checks': total_checks,
'passed_checks': passed_checks,
'failed_required': len(failed_required),
'success_rate': (passed_checks / total_checks * 100) if total_checks > 0 else 0,
'deployment_approved': deployment_approved
},
'failed_required_items': [
{'id': r['id'], 'description': r['description']}
for r in failed_required
]
}
@staticmethod
def _check_HC_001(deployment_config: Dict) -> Tuple[bool, str]:
"""检查存活探针"""
containers = deployment_config.get('spec', {}).get('template', {}).get('spec', {}).get('containers', [])
for container in containers:
if 'livenessProbe' in container:
return True, f"容器 {container.get('name', 'unknown')} 配置了存活探针"
return False, "未找到存活探针配置"
@staticmethod
def _check_HC_002(deployment_config: Dict) -> Tuple[bool, str]:
"""检查就绪探针"""
containers = deployment_config.get('spec', {}).get('template', {}).get('spec', {}).get('containers', [])
for container in containers:
if 'readinessProbe' in container:
return True, f"容器 {container.get('name', 'unknown')} 配置了就绪探针"
return False, "未找到就绪探针配置"
@staticmethod
def _check_HC_010(deployment_config: Dict) -> Tuple[bool, str]:
"""检查敏感信息移除"""
# 这里可以检查健康检查响应是否包含敏感信息
# 简化实现：检查是否有凭证信息暴露
return True, "敏感信息检查通过（需手动验证）"9. 总结与展望
9.1 关键收获
通过本文的实现，我们获得了以下关键能力：
- 完整的健康检查框架：支持存活、就绪、启动探针
- 丰富的检查器类型：系统、数据库、HTTP、端口、文件系统等
- 智能健康检查：自适应、组合检查器
- 状态管理：完整的状态机和生命周期管理
- 告警系统：基于规则的智能告警
- 生产就绪：Kubernetes集成、监控、安全配置
9.2 性能数据总结
根据我们的测试，健康检查系统的性能表现：
| 检查类型 | 平均响应时间 | 资源消耗 | 建议检查频率 |
|---|---|---|---|
| 存活探针 | 50-100ms | 低 | 每10-30秒 |
| 就绪探针 | 100-500ms | 中 | 每5-10秒 |
| 详细检查 | 1-5秒 | 高 | 每1-5分钟 |
| 启动探针 | 可变 | 可变 | 初始延迟后每10秒 |
9.3 未来发展方向
- AI驱动的健康预测：使用机器学习预测潜在故障
- 混沌工程集成：与混沌实验工具集成
- 跨服务健康依赖：服务间的健康依赖关系图
- 自适应检查频率：根据负载动态调整检查频率
- 边缘计算支持：适应边缘环境的健康检查
附录
A. 健康检查最佳实践
-
探针配置原则：
- 存活探针：检查应用核心功能，失败时重启
- 就绪探针：检查所有依赖，失败时停止流量
- 启动探针：保护慢启动应用
-
超时设置建议：
- 存活探针：timeoutSeconds ≤ periodSeconds
- 就绪探针：timeoutSeconds ≤ periodSeconds / 2
- 启动探针：timeoutSeconds ≤ periodSeconds，failureThreshold较大
-
检查内容分层：
- Level 1：应用进程存在性
- Level 2：内部功能正常
- Level 3：外部依赖正常
- Level 4：业务逻辑正常
B. 常见问题解答
Q1: 健康检查应该检查什么？
A: 建议检查：应用进程、内存使用、线程池状态、数据库连接、缓存连接、消息队列、外部API、文件系统、业务核心功能。
Q2: 健康检查频率如何设置？
A: 存活探针：10-30秒；就绪探针：5-10秒；详细检查：1-5分钟。根据应用负载和依赖稳定性调整。
Q3: 健康检查失败时怎么办？
A: 存活探针失败：重启容器；就绪探针失败：从负载均衡移除；连续失败：告警通知。
Q4: 如何保护健康检查端点？
A: 使用认证、速率限制、IP白名单、请求签名等方式保护。
C. 性能优化建议
- 缓存检查结果：对稳定的依赖缓存检查结果
- 并行检查：独立依赖并行检查
- 分层检查：先检查快速项目，再检查慢速项目
- 增量检查：只检查变更的部分
- 连接池：复用数据库和HTTP连接
免责声明：本文提供的代码和方案仅供参考，生产环境中请根据具体需求进行性能测试和安全审计。健康检查系统设计应考虑具体业务场景和合规要求。