第14章 Jenkins故障排除

zzu123zsw2025-09-05 20:11

14.1 故障排除概述

常见故障类型

系统级故障：

复制代码 
Jenkins系统故障分类：

1. 启动和连接问题
   - Jenkins无法启动
   - 端口冲突
   - 网络连接问题
   - SSL/TLS证书问题

2. 性能和资源问题
   - 内存不足或泄漏
   - CPU使用率过高
   - 磁盘空间不足
   - 数据库连接问题

3. 配置和权限问题
   - 配置文件损坏
   - 权限设置错误
   - 插件冲突
   - 环境变量问题

4. 构建和部署问题
   - 构建失败
   - 代码检出问题
   - 依赖下载失败
   - 部署环境问题

故障诊断流程：

复制代码 
故障排除标准流程：

1. 问题识别
   - 收集故障现象
   - 确定影响范围
   - 记录错误信息
   - 分析故障时间

2. 信息收集
   - 查看系统日志
   - 检查配置文件
   - 分析性能指标
   - 收集环境信息

3. 问题分析
   - 确定根本原因
   - 分析关联因素
   - 评估修复方案
   - 制定恢复计划

4. 问题解决
   - 实施修复措施
   - 验证修复效果
   - 监控系统状态
   - 记录解决过程

14.2 日志分析

Jenkins日志系统

日志配置：

xml 复制代码 
<!-- logback.xml -->
<configuration>
    <!-- 控制台输出 -->
    <appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
        <encoder>
            <pattern>%d{HH:mm:ss.SSS} [%thread] %-5level %logger{36} - %msg%n</pattern>
        </encoder>
    </appender>
    
    <!-- 文件输出 -->
    <appender name="FILE" class="ch.qos.logback.core.rolling.RollingFileAppender">
        <file>${JENKINS_HOME}/logs/jenkins.log</file>
        <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
            <fileNamePattern>${JENKINS_HOME}/logs/jenkins.%d{yyyy-MM-dd}.%i.log</fileNamePattern>
            <maxFileSize>100MB</maxFileSize>
            <maxHistory>30</maxHistory>
            <totalSizeCap>10GB</totalSizeCap>
        </rollingPolicy>
        <encoder>
            <pattern>%d{yyyy-MM-dd HH:mm:ss.SSS} [%thread] %-5level %logger{36} - %msg%n</pattern>
        </encoder>
    </appender>
    
    <!-- 错误日志单独记录 -->
    <appender name="ERROR_FILE" class="ch.qos.logback.core.rolling.RollingFileAppender">
        <file>${JENKINS_HOME}/logs/error.log</file>
        <filter class="ch.qos.logback.classic.filter.LevelFilter">
            <level>ERROR</level>
            <onMatch>ACCEPT</onMatch>
            <onMismatch>DENY</onMismatch>
        </filter>
        <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
            <fileNamePattern>${JENKINS_HOME}/logs/error.%d{yyyy-MM-dd}.log</fileNamePattern>
            <maxHistory>90</maxHistory>
        </rollingPolicy>
        <encoder>
            <pattern>%d{yyyy-MM-dd HH:mm:ss.SSS} [%thread] %-5level %logger{36} - %msg%n</pattern>
        </encoder>
    </appender>
    
    <!-- 审计日志 -->
    <appender name="AUDIT_FILE" class="ch.qos.logback.core.rolling.RollingFileAppender">
        <file>${JENKINS_HOME}/logs/audit.log</file>
        <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
            <fileNamePattern>${JENKINS_HOME}/logs/audit.%d{yyyy-MM-dd}.log</fileNamePattern>
            <maxHistory>365</maxHistory>
        </rollingPolicy>
        <encoder>
            <pattern>%d{yyyy-MM-dd HH:mm:ss.SSS} [%thread] %logger{36} - %msg%n</pattern>
        </encoder>
    </appender>
    
    <!-- 性能日志 -->
    <appender name="PERF_FILE" class="ch.qos.logback.core.rolling.RollingFileAppender">
        <file>${JENKINS_HOME}/logs/performance.log</file>
        <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
            <fileNamePattern>${JENKINS_HOME}/logs/performance.%d{yyyy-MM-dd}.log</fileNamePattern>
            <maxHistory>7</maxHistory>
        </rollingPolicy>
        <encoder>
            <pattern>%d{yyyy-MM-dd HH:mm:ss.SSS} - %msg%n</pattern>
        </encoder>
    </appender>
    
    <!-- 根日志级别 -->
    <root level="INFO">
        <appender-ref ref="STDOUT" />
        <appender-ref ref="FILE" />
        <appender-ref ref="ERROR_FILE" />
    </root>
    
    <!-- 特定包的日志级别 -->
    <logger name="hudson.security" level="DEBUG" additivity="false">
        <appender-ref ref="AUDIT_FILE" />
    </logger>
    
    <logger name="jenkins.security" level="DEBUG" additivity="false">
        <appender-ref ref="AUDIT_FILE" />
    </logger>
    
    <logger name="hudson.model.Run" level="DEBUG" additivity="false">
        <appender-ref ref="PERF_FILE" />
    </logger>
    
    <!-- 插件日志 -->
    <logger name="org.jenkinsci.plugins" level="INFO" />
    <logger name="hudson.plugins" level="INFO" />
    
    <!-- Git插件详细日志 -->
    <logger name="hudson.plugins.git" level="DEBUG" />
    
    <!-- Pipeline日志 -->
    <logger name="org.jenkinsci.plugins.workflow" level="DEBUG" />
    
    <!-- 减少噪音日志 -->
    <logger name="org.eclipse.jetty" level="WARN" />
    <logger name="org.apache.sshd" level="WARN" />
</configuration>

日志分析脚本：

bash 复制代码 
#!/bin/bash
# jenkins_log_analyzer.sh

JENKINS_HOME="/var/lib/jenkins"
LOG_DIR="$JENKINS_HOME/logs"
REPORT_DIR="/tmp/jenkins_analysis"
DATE=$(date +%Y%m%d_%H%M%S)

mkdir -p $REPORT_DIR

echo "=== Jenkins日志分析报告 ===" > "$REPORT_DIR/analysis_$DATE.txt"
echo "生成时间: $(date)" >> "$REPORT_DIR/analysis_$DATE.txt"
echo "" >> "$REPORT_DIR/analysis_$DATE.txt"

# 1. 错误统计
echo "=== 错误统计 ===" >> "$REPORT_DIR/analysis_$DATE.txt"
if [ -f "$LOG_DIR/error.log" ]; then
    echo "今日错误总数: $(grep "$(date +%Y-%m-%d)" $LOG_DIR/error.log | wc -l)" >> "$REPORT_DIR/analysis_$DATE.txt"
    echo "" >> "$REPORT_DIR/analysis_$DATE.txt"
    
    echo "错误类型分布:" >> "$REPORT_DIR/analysis_$DATE.txt"
    grep "$(date +%Y-%m-%d)" $LOG_DIR/error.log | \
        sed -n 's/.*ERROR.*- \(.*\)/\1/p' | \
        cut -d' ' -f1-3 | \
        sort | uniq -c | sort -nr | head -10 >> "$REPORT_DIR/analysis_$DATE.txt"
    echo "" >> "$REPORT_DIR/analysis_$DATE.txt"
fi

# 2. 性能分析
echo "=== 性能分析 ===" >> "$REPORT_DIR/analysis_$DATE.txt"
if [ -f "$LOG_DIR/jenkins.log" ]; then
    # 慢查询分析
    echo "慢操作分析 (>5秒):" >> "$REPORT_DIR/analysis_$DATE.txt"
    grep -E "took [5-9][0-9]{3,}ms|took [0-9]{5,}ms" $LOG_DIR/jenkins.log | \
        tail -20 >> "$REPORT_DIR/analysis_$DATE.txt"
    echo "" >> "$REPORT_DIR/analysis_$DATE.txt"
    
    # 内存警告
    echo "内存警告:" >> "$REPORT_DIR/analysis_$DATE.txt"
    grep -i "OutOfMemoryError\|memory" $LOG_DIR/jenkins.log | \
        grep "$(date +%Y-%m-%d)" | tail -10 >> "$REPORT_DIR/analysis_$DATE.txt"
    echo "" >> "$REPORT_DIR/analysis_$DATE.txt"
fi

# 3. 安全事件
echo "=== 安全事件 ===" >> "$REPORT_DIR/analysis_$DATE.txt"
if [ -f "$LOG_DIR/audit.log" ]; then
    echo "今日登录失败次数: $(grep "$(date +%Y-%m-%d)" $LOG_DIR/audit.log | grep -i "login.*failed" | wc -l)" >> "$REPORT_DIR/analysis_$DATE.txt"
    echo "" >> "$REPORT_DIR/analysis_$DATE.txt"
    
    echo "异常IP访问:" >> "$REPORT_DIR/analysis_$DATE.txt"
    grep "$(date +%Y-%m-%d)" $LOG_DIR/audit.log | \
        grep -oE "[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}" | \
        sort | uniq -c | sort -nr | head -10 >> "$REPORT_DIR/analysis_$DATE.txt"
    echo "" >> "$REPORT_DIR/analysis_$DATE.txt"
fi

# 4. 构建分析
echo "=== 构建分析 ===" >> "$REPORT_DIR/analysis_$DATE.txt"
if [ -f "$LOG_DIR/jenkins.log" ]; then
    echo "今日构建总数: $(grep "$(date +%Y-%m-%d)" $LOG_DIR/jenkins.log | grep -c "Started by")" >> "$REPORT_DIR/analysis_$DATE.txt"
    echo "构建失败数: $(grep "$(date +%Y-%m-%d)" $LOG_DIR/jenkins.log | grep -c "Build failed")" >> "$REPORT_DIR/analysis_$DATE.txt"
    echo "" >> "$REPORT_DIR/analysis_$DATE.txt"
    
    echo "失败构建详情:" >> "$REPORT_DIR/analysis_$DATE.txt"
    grep "$(date +%Y-%m-%d)" $LOG_DIR/jenkins.log | \
        grep "Build failed" | tail -10 >> "$REPORT_DIR/analysis_$DATE.txt"
    echo "" >> "$REPORT_DIR/analysis_$DATE.txt"
fi

# 5. 插件问题
echo "=== 插件问题 ===" >> "$REPORT_DIR/analysis_$DATE.txt"
if [ -f "$LOG_DIR/jenkins.log" ]; then
    echo "插件错误:" >> "$REPORT_DIR/analysis_$DATE.txt"
    grep "$(date +%Y-%m-%d)" $LOG_DIR/jenkins.log | \
        grep -i "plugin.*error\|plugin.*exception" | \
        tail -10 >> "$REPORT_DIR/analysis_$DATE.txt"
    echo "" >> "$REPORT_DIR/analysis_$DATE.txt"
fi

# 6. 系统资源
echo "=== 系统资源 ===" >> "$REPORT_DIR/analysis_$DATE.txt"
echo "当前系统状态:" >> "$REPORT_DIR/analysis_$DATE.txt"
echo "CPU使用率: $(top -bn1 | grep "Cpu(s)" | awk '{print $2}' | cut -d'%' -f1)%" >> "$REPORT_DIR/analysis_$DATE.txt"
echo "内存使用: $(free | grep Mem | awk '{printf "%.1f%%", $3/$2 * 100.0}')" >> "$REPORT_DIR/analysis_$DATE.txt"
echo "磁盘使用: $(df -h $JENKINS_HOME | tail -1 | awk '{print $5}')" >> "$REPORT_DIR/analysis_$DATE.txt"
echo "" >> "$REPORT_DIR/analysis_$DATE.txt"

# 7. 网络连接
echo "=== 网络连接 ===" >> "$REPORT_DIR/analysis_$DATE.txt"
echo "Jenkins端口连接数: $(netstat -an | grep :8080 | wc -l)" >> "$REPORT_DIR/analysis_$DATE.txt"
echo "Agent连接数: $(netstat -an | grep :50000 | wc -l)" >> "$REPORT_DIR/analysis_$DATE.txt"
echo "" >> "$REPORT_DIR/analysis_$DATE.txt"

# 生成HTML报告
cat > "$REPORT_DIR/analysis_$DATE.html" << EOF
<!DOCTYPE html>
<html>
<head>
    <title>Jenkins日志分析报告</title>
    <meta charset="UTF-8">
    <style>
        body { font-family: Arial, sans-serif; margin: 20px; }
        .header { background-color: #f0f0f0; padding: 10px; border-radius: 5px; }
        .section { margin: 20px 0; padding: 15px; border: 1px solid #ddd; border-radius: 5px; }
        .error { background-color: #ffe6e6; }
        .warning { background-color: #fff3cd; }
        .info { background-color: #e6f3ff; }
        pre { background-color: #f8f9fa; padding: 10px; border-radius: 3px; overflow-x: auto; }
        .metric { display: inline-block; margin: 10px; padding: 10px; background-color: #f8f9fa; border-radius: 5px; }
    </style>
</head>
<body>
    <div class="header">
        <h1>Jenkins日志分析报告</h1>
        <p>生成时间: $(date)</p>
    </div>
EOF

# 转换文本报告为HTML
awk '
BEGIN { in_section = 0 }
/^=== .* ===$/ {
    if (in_section) print "    </pre>\n</div>"
    section = $0
    gsub(/^=== | ===$/, "", section)
    class = "info"
    if (section ~ /错误|Error/) class = "error"
    if (section ~ /警告|Warning/) class = "warning"
    print "<div class=\"section " class "\">\n    <h2>" section "</h2>\n    <pre>"
    in_section = 1
    next
}
{
    if (in_section) {
        gsub(/&/, "\\&amp;"); gsub(/</, "\\&lt;"); gsub(/>/, "\\&gt;")
        print
    }
}
END { if (in_section) print "    </pre>\n</div>" }
' "$REPORT_DIR/analysis_$DATE.txt" >> "$REPORT_DIR/analysis_$DATE.html"

echo "</body></html>" >> "$REPORT_DIR/analysis_$DATE.html"

echo "日志分析完成，报告保存在: $REPORT_DIR/analysis_$DATE.html"
echo "文本报告: $REPORT_DIR/analysis_$DATE.txt"

实时日志监控

日志监控脚本：

python 复制代码 
#!/usr/bin/env python3
# jenkins_log_monitor.py

import os
import re
import time
import json
import smtplib
from datetime import datetime
from email.mime.text import MIMEText
from email.mime.multipart import MIMEMultipart
from watchdog.observers import Observer
from watchdog.events import FileSystemEventHandler

class JenkinsLogMonitor(FileSystemEventHandler):
    def __init__(self, config_file='monitor_config.json'):
        self.config = self.load_config(config_file)
        self.patterns = self.compile_patterns()
        self.alert_counts = {}
        self.last_alert_time = {}
        
    def load_config(self, config_file):
        """加载监控配置"""
        default_config = {
            'log_files': [
                '/var/lib/jenkins/logs/jenkins.log',
                '/var/lib/jenkins/logs/error.log'
            ],
            'alert_patterns': {
                'critical': [
                    r'OutOfMemoryError',
                    r'java\.lang\.OutOfMemoryError',
                    r'Jenkins is going to shut down',
                    r'SEVERE.*Exception'
                ],
                'warning': [
                    r'WARNING.*',
                    r'Failed to.*',
                    r'Connection.*refused',
                    r'Timeout.*'
                ],
                'security': [
                    r'Authentication failed',
                    r'Access denied',
                    r'Invalid.*credentials',
                    r'Unauthorized.*access'
                ]
            },
            'alert_thresholds': {
                'critical': 1,
                'warning': 5,
                'security': 3
            },
            'alert_intervals': {
                'critical': 300,  # 5分钟
                'warning': 900,   # 15分钟
                'security': 600   # 10分钟
            },
            'email': {
                'enabled': True,
                'smtp_server': 'localhost',
                'smtp_port': 587,
                'username': '',
                'password': '',
                'from_addr': 'jenkins@company.com',
                'to_addrs': ['admin@company.com']
            },
            'webhook': {
                'enabled': False,
                'url': 'https://hooks.slack.com/services/xxx',
                'channel': '#jenkins-alerts'
            }
        }
        
        try:
            with open(config_file, 'r') as f:
                config = json.load(f)
                # 合并默认配置
                for key, value in default_config.items():
                    if key not in config:
                        config[key] = value
                return config
        except FileNotFoundError:
            # 创建默认配置文件
            with open(config_file, 'w') as f:
                json.dump(default_config, f, indent=2)
            return default_config
    
    def compile_patterns(self):
        """编译正则表达式模式"""
        compiled_patterns = {}
        for level, patterns in self.config['alert_patterns'].items():
            compiled_patterns[level] = [re.compile(pattern, re.IGNORECASE) for pattern in patterns]
        return compiled_patterns
    
    def on_modified(self, event):
        """文件修改事件处理"""
        if event.is_directory:
            return
            
        if event.src_path in self.config['log_files']:
            self.process_log_file(event.src_path)
    
    def process_log_file(self, file_path):
        """处理日志文件"""
        try:
            with open(file_path, 'r', encoding='utf-8', errors='ignore') as f:
                # 移动到文件末尾
                f.seek(0, 2)
                file_size = f.tell()
                
                # 读取最后1KB的内容
                read_size = min(1024, file_size)
                f.seek(file_size - read_size)
                
                lines = f.readlines()
                
                # 处理新增的日志行
                for line in lines[-10:]:  # 只处理最后10行
                    self.analyze_log_line(line.strip(), file_path)
                    
        except Exception as e:
            print(f"处理日志文件错误: {e}")
    
    def analyze_log_line(self, line, file_path):
        """分析日志行"""
        if not line:
            return
            
        timestamp = datetime.now()
        
        for level, patterns in self.patterns.items():
            for pattern in patterns:
                if pattern.search(line):
                    self.handle_alert(level, line, file_path, timestamp)
                    break
    
    def handle_alert(self, level, message, file_path, timestamp):
        """处理告警"""
        alert_key = f"{level}_{hash(message) % 1000}"
        
        # 计数器
        if alert_key not in self.alert_counts:
            self.alert_counts[alert_key] = 0
        self.alert_counts[alert_key] += 1
        
        # 检查是否达到告警阈值
        threshold = self.config['alert_thresholds'].get(level, 1)
        if self.alert_counts[alert_key] < threshold:
            return
        
        # 检查告警间隔
        interval = self.config['alert_intervals'].get(level, 300)
        last_time = self.last_alert_time.get(alert_key, 0)
        current_time = timestamp.timestamp()
        
        if current_time - last_time < interval:
            return
        
        # 发送告警
        self.send_alert(level, message, file_path, timestamp, self.alert_counts[alert_key])
        
        # 更新最后告警时间
        self.last_alert_time[alert_key] = current_time
        
        # 重置计数器
        self.alert_counts[alert_key] = 0
    
    def send_alert(self, level, message, file_path, timestamp, count):
        """发送告警"""
        alert_data = {
            'level': level,
            'message': message,
            'file_path': file_path,
            'timestamp': timestamp.isoformat(),
            'count': count,
            'hostname': os.uname().nodename
        }
        
        print(f"[{timestamp}] {level.upper()} ALERT: {message}")
        
        # 发送邮件告警
        if self.config['email']['enabled']:
            self.send_email_alert(alert_data)
        
        # 发送Webhook告警
        if self.config['webhook']['enabled']:
            self.send_webhook_alert(alert_data)
    
    def send_email_alert(self, alert_data):
        """发送邮件告警"""
        try:
            email_config = self.config['email']
            
            msg = MIMEMultipart()
            msg['From'] = email_config['from_addr']
            msg['To'] = ', '.join(email_config['to_addrs'])
            msg['Subject'] = f"Jenkins {alert_data['level'].upper()} Alert - {alert_data['hostname']}"
            
            body = f"""
Jenkins告警通知

告警级别: {alert_data['level'].upper()}
发生时间: {alert_data['timestamp']}
主机名称: {alert_data['hostname']}
日志文件: {alert_data['file_path']}
出现次数: {alert_data['count']}

告警内容:
{alert_data['message']}

请及时处理相关问题。

-- Jenkins监控系统
"""
            
            msg.attach(MIMEText(body, 'plain', 'utf-8'))
            
            server = smtplib.SMTP(email_config['smtp_server'], email_config['smtp_port'])
            if email_config['username']:
                server.starttls()
                server.login(email_config['username'], email_config['password'])
            
            server.send_message(msg)
            server.quit()
            
            print(f"邮件告警已发送: {email_config['to_addrs']}")
            
        except Exception as e:
            print(f"发送邮件告警失败: {e}")
    
    def send_webhook_alert(self, alert_data):
        """发送Webhook告警"""
        try:
            import requests
            
            webhook_config = self.config['webhook']
            
            # Slack格式的消息
            payload = {
                'channel': webhook_config['channel'],
                'username': 'Jenkins Monitor',
                'icon_emoji': ':warning:' if alert_data['level'] == 'warning' else ':rotating_light:',
                'attachments': [{
                    'color': 'danger' if alert_data['level'] == 'critical' else 'warning',
                    'title': f"Jenkins {alert_data['level'].upper()} Alert",
                    'fields': [
                        {'title': '主机', 'value': alert_data['hostname'], 'short': True},
                        {'title': '时间', 'value': alert_data['timestamp'], 'short': True},
                        {'title': '文件', 'value': alert_data['file_path'], 'short': True},
                        {'title': '次数', 'value': str(alert_data['count']), 'short': True},
                        {'title': '消息', 'value': f"```{alert_data['message']}```", 'short': False}
                    ],
                    'footer': 'Jenkins Monitor',
                    'ts': int(datetime.fromisoformat(alert_data['timestamp']).timestamp())
                }]
            }
            
            response = requests.post(webhook_config['url'], json=payload, timeout=10)
            response.raise_for_status()
            
            print(f"Webhook告警已发送: {webhook_config['channel']}")
            
        except Exception as e:
            print(f"发送Webhook告警失败: {e}")
    
    def start_monitoring(self):
        """开始监控"""
        print("启动Jenkins日志监控...")
        print(f"监控文件: {self.config['log_files']}")
        
        observer = Observer()
        
        # 为每个日志文件添加监控
        for log_file in self.config['log_files']:
            if os.path.exists(log_file):
                log_dir = os.path.dirname(log_file)
                observer.schedule(self, log_dir, recursive=False)
                print(f"已添加监控: {log_file}")
            else:
                print(f"警告: 日志文件不存在: {log_file}")
        
        observer.start()
        
        try:
            while True:
                time.sleep(1)
        except KeyboardInterrupt:
            print("\n停止监控...")
            observer.stop()
        
        observer.join()

def main():
    monitor = JenkinsLogMonitor()
    monitor.start_monitoring()

if __name__ == '__main__':
    main()

14.3 常见问题诊断

启动问题

Jenkins无法启动诊断：

bash 复制代码 
#!/bin/bash
# jenkins_startup_diagnosis.sh

echo "=== Jenkins启动问题诊断 ==="

# 1. 检查Java环境
echo "1. 检查Java环境"
if command -v java &> /dev/null; then
    echo "✓ Java已安装: $(java -version 2>&1 | head -1)"
    
    # 检查Java版本
    JAVA_VERSION=$(java -version 2>&1 | head -1 | cut -d'"' -f2 | cut -d'.' -f1-2)
    if [[ "$JAVA_VERSION" < "1.8" ]]; then
        echo "✗ Java版本过低，需要Java 8或更高版本"
    else
        echo "✓ Java版本符合要求"
    fi
else
    echo "✗ Java未安装或不在PATH中"
fi
echo ""

# 2. 检查Jenkins进程
echo "2. 检查Jenkins进程"
JENKINS_PID=$(pgrep -f jenkins.war)
if [ -n "$JENKINS_PID" ]; then
    echo "✓ Jenkins进程正在运行 (PID: $JENKINS_PID)"
    echo "进程信息: $(ps -p $JENKINS_PID -o pid,ppid,cmd,%mem,%cpu)"
else
    echo "✗ Jenkins进程未运行"
fi
echo ""

# 3. 检查端口占用
echo "3. 检查端口占用"
PORT_8080=$(netstat -tlnp 2>/dev/null | grep :8080 || ss -tlnp 2>/dev/null | grep :8080)
if [ -n "$PORT_8080" ]; then
    echo "✓ 端口8080已被占用:"
    echo "$PORT_8080"
else
    echo "✗ 端口8080未被占用"
fi

PORT_50000=$(netstat -tlnp 2>/dev/null | grep :50000 || ss -tlnp 2>/dev/null | grep :50000)
if [ -n "$PORT_50000" ]; then
    echo "✓ Agent端口50000已被占用:"
    echo "$PORT_50000"
else
    echo "✗ Agent端口50000未被占用"
fi
echo ""

# 4. 检查Jenkins目录权限
echo "4. 检查Jenkins目录权限"
JENKINS_HOME="/var/lib/jenkins"
if [ -d "$JENKINS_HOME" ]; then
    echo "✓ Jenkins主目录存在: $JENKINS_HOME"
    
    JENKINS_USER="jenkins"
    OWNER=$(stat -c '%U' $JENKINS_HOME)
    if [ "$OWNER" = "$JENKINS_USER" ]; then
        echo "✓ 目录所有者正确: $OWNER"
    else
        echo "✗ 目录所有者错误: $OWNER (应该是: $JENKINS_USER)"
    fi
    
    PERMS=$(stat -c '%a' $JENKINS_HOME)
    echo "目录权限: $PERMS"
else
    echo "✗ Jenkins主目录不存在: $JENKINS_HOME"
fi
echo ""

# 5. 检查磁盘空间
echo "5. 检查磁盘空间"
DISK_USAGE=$(df -h $JENKINS_HOME 2>/dev/null | tail -1 | awk '{print $5}' | sed 's/%//')
if [ -n "$DISK_USAGE" ]; then
    if [ "$DISK_USAGE" -gt 90 ]; then
        echo "✗ 磁盘空间不足: ${DISK_USAGE}%"
    elif [ "$DISK_USAGE" -gt 80 ]; then
        echo "⚠ 磁盘空间紧张: ${DISK_USAGE}%"
    else
        echo "✓ 磁盘空间充足: ${DISK_USAGE}%"
    fi
else
    echo "✗ 无法检查磁盘空间"
fi
echo ""

# 6. 检查内存
echo "6. 检查系统内存"
MEM_TOTAL=$(free -m | grep Mem | awk '{print $2}')
MEM_USED=$(free -m | grep Mem | awk '{print $3}')
MEM_USAGE=$((MEM_USED * 100 / MEM_TOTAL))

echo "总内存: ${MEM_TOTAL}MB"
echo "已用内存: ${MEM_USED}MB (${MEM_USAGE}%)"

if [ "$MEM_USAGE" -gt 90 ]; then
    echo "✗ 内存使用率过高: ${MEM_USAGE}%"
elif [ "$MEM_USAGE" -gt 80 ]; then
    echo "⚠ 内存使用率较高: ${MEM_USAGE}%"
else
    echo "✓ 内存使用率正常: ${MEM_USAGE}%"
fi
echo ""

# 7. 检查配置文件
echo "7. 检查配置文件"
CONFIG_FILE="$JENKINS_HOME/config.xml"
if [ -f "$CONFIG_FILE" ]; then
    echo "✓ 主配置文件存在: $CONFIG_FILE"
    
    # 检查XML格式
    if xmllint --noout "$CONFIG_FILE" 2>/dev/null; then
        echo "✓ 配置文件XML格式正确"
    else
        echo "✗ 配置文件XML格式错误"
    fi
else
    echo "✗ 主配置文件不存在: $CONFIG_FILE"
fi
echo ""

# 8. 检查日志文件
echo "8. 检查启动日志"
LOG_FILES=(
    "/var/log/jenkins/jenkins.log"
    "$JENKINS_HOME/logs/jenkins.log"
    "/var/lib/jenkins/jenkins.log"
)

for LOG_FILE in "${LOG_FILES[@]}"; do
    if [ -f "$LOG_FILE" ]; then
        echo "✓ 找到日志文件: $LOG_FILE"
        echo "最近的错误信息:"
        tail -20 "$LOG_FILE" | grep -i "error\|exception\|failed" | tail -5
        break
    fi
done
echo ""

# 9. 检查系统服务
echo "9. 检查系统服务"
if systemctl is-active jenkins &>/dev/null; then
    echo "✓ Jenkins服务正在运行"
    echo "服务状态: $(systemctl is-active jenkins)"
else
    echo "✗ Jenkins服务未运行"
    echo "服务状态: $(systemctl is-active jenkins 2>/dev/null || echo 'unknown')"
fi

if systemctl is-enabled jenkins &>/dev/null; then
    echo "✓ Jenkins服务已启用自启动"
else
    echo "⚠ Jenkins服务未启用自启动"
fi
echo ""

# 10. 生成修复建议
echo "=== 修复建议 ==="
if [ -z "$JENKINS_PID" ]; then
    echo "Jenkins未运行，建议检查:"
    echo "1. 使用 'systemctl start jenkins' 启动服务"
    echo "2. 检查 'systemctl status jenkins' 查看详细状态"
    echo "3. 查看 'journalctl -u jenkins -f' 实时日志"
fi

if [ "$DISK_USAGE" -gt 90 ] 2>/dev/null; then
    echo "磁盘空间不足，建议:"
    echo "1. 清理旧的构建日志和工作空间"
    echo "2. 使用 'jenkins-cli.jar' 清理旧构建"
    echo "3. 配置日志轮转策略"
fi

if [ "$MEM_USAGE" -gt 90 ] 2>/dev/null; then
    echo "内存不足，建议:"
    echo "1. 增加系统内存"
    echo "2. 调整JVM堆大小参数"
    echo "3. 减少并发构建数量"
fi

echo "诊断完成。"

性能问题

性能问题诊断脚本：

bash 复制代码 
#!/bin/bash
# jenkins_performance_diagnosis.sh

echo "=== Jenkins性能问题诊断 ==="

JENKINS_PID=$(pgrep -f jenkins.war)
if [ -z "$JENKINS_PID" ]; then
    echo "✗ Jenkins进程未运行"
    exit 1
fi

echo "Jenkins PID: $JENKINS_PID"
echo ""

# 1. CPU使用率分析
echo "1. CPU使用率分析"
CPU_USAGE=$(ps -p $JENKINS_PID -o %cpu --no-headers)
echo "Jenkins CPU使用率: ${CPU_USAGE}%"

# 系统整体CPU
SYS_CPU=$(top -bn1 | grep "Cpu(s)" | awk '{print $2}' | cut -d'%' -f1)
echo "系统CPU使用率: ${SYS_CPU}%"

# CPU核心数
CPU_CORES=$(nproc)
echo "CPU核心数: $CPU_CORES"
echo ""

# 2. 内存使用分析
echo "2. 内存使用分析"
# 进程内存
MEM_INFO=$(ps -p $JENKINS_PID -o %mem,vsz,rss --no-headers)
echo "Jenkins内存使用: $MEM_INFO"

# JVM堆内存
if command -v jstat &> /dev/null; then
    echo "JVM堆内存信息:"
    jstat -gc $JENKINS_PID | tail -1 | awk '{
        printf "Eden: %.1fMB, Survivor: %.1fMB, Old: %.1fMB\n", $3/1024, ($4+$5)/1024, $7/1024
        printf "Metaspace: %.1fMB, Total Heap: %.1fMB\n", $9/1024, ($3+$4+$5+$7)/1024
    }'
    
    # GC统计
    echo "GC统计信息:"
    jstat -gc $JENKINS_PID | tail -1 | awk '{
        printf "Young GC次数: %d, Full GC次数: %d\n", $12, $13
        printf "GC总时间: %.2f秒\n", $14
    }'
fi
echo ""

# 3. 线程分析
echo "3. 线程分析"
if command -v jstack &> /dev/null; then
    THREAD_COUNT=$(jstack $JENKINS_PID 2>/dev/null | grep "^\"" | wc -l)
    echo "总线程数: $THREAD_COUNT"
    
    # 线程状态统计
    echo "线程状态分布:"
    jstack $JENKINS_PID 2>/dev/null | grep "java.lang.Thread.State:" | \
        sort | uniq -c | sort -nr
    
    # 查找阻塞线程
    BLOCKED_THREADS=$(jstack $JENKINS_PID 2>/dev/null | grep -A 1 "BLOCKED" | wc -l)
    if [ "$BLOCKED_THREADS" -gt 0 ]; then
        echo "⚠ 发现 $BLOCKED_THREADS 个阻塞线程"
    fi
fi
echo ""

# 4. 磁盘I/O分析
echo "4. 磁盘I/O分析"
JENKINS_HOME="/var/lib/jenkins"
if command -v iotop &> /dev/null; then
    echo "磁盘I/O使用情况:"
    iotop -a -o -d 1 -n 3 | grep jenkins || echo "未发现Jenkins相关I/O"
fi

# 磁盘使用情况
echo "磁盘使用情况:"
df -h $JENKINS_HOME

# 大文件查找
echo "查找大文件 (>100MB):"
find $JENKINS_HOME -type f -size +100M -exec ls -lh {} \; 2>/dev/null | head -10
echo ""

# 5. 网络连接分析
echo "5. 网络连接分析"
echo "Jenkins端口连接数:"
netstat -an | grep :8080 | awk '{print $6}' | sort | uniq -c

echo "Agent连接数:"
netstat -an | grep :50000 | awk '{print $6}' | sort | uniq -c

# 检查网络延迟
echo "网络连接延迟测试:"
ping -c 3 localhost | tail -1
echo ""

# 6. 数据库连接分析
echo "6. 数据库连接分析"
if [ -f "$JENKINS_HOME/config.xml" ]; then
    DB_CONFIG=$(grep -i "database\|jdbc" $JENKINS_HOME/config.xml)
    if [ -n "$DB_CONFIG" ]; then
        echo "发现数据库配置"
        # 这里可以添加数据库连接测试
    else
        echo "使用默认H2数据库"
    fi
fi
echo ""

# 7. 插件性能分析
echo "7. 插件性能分析"
PLUGIN_DIR="$JENKINS_HOME/plugins"
if [ -d "$PLUGIN_DIR" ]; then
    PLUGIN_COUNT=$(ls $PLUGIN_DIR/*.jpi 2>/dev/null | wc -l)
    echo "已安装插件数量: $PLUGIN_COUNT"
    
    # 查找大插件
    echo "大插件文件 (>10MB):"
    find $PLUGIN_DIR -name "*.jpi" -size +10M -exec ls -lh {} \; | head -5
fi
echo ""

# 8. 构建队列分析
echo "8. 构建队列分析"
if command -v curl &> /dev/null; then
    QUEUE_INFO=$(curl -s http://localhost:8080/queue/api/json 2>/dev/null)
    if [ $? -eq 0 ]; then
        QUEUE_LENGTH=$(echo $QUEUE_INFO | grep -o '"items":\[' | wc -l)
        echo "当前队列长度: $QUEUE_LENGTH"
    else
        echo "无法获取队列信息"
    fi
fi
echo ""

# 9. 系统负载分析
echo "9. 系统负载分析"
echo "系统负载:"
uptime

echo "内存使用详情:"
free -h

echo "交换空间使用:"
swapon --show 2>/dev/null || echo "无交换空间配置"
echo ""

# 10. 性能建议
echo "=== 性能优化建议 ==="

# CPU建议
if (( $(echo "$CPU_USAGE > 80" | bc -l) )); then
    echo "CPU使用率过高建议:"
    echo "1. 减少并发构建数量"
    echo "2. 优化构建脚本"
    echo "3. 增加构建节点"
fi

# 内存建议
MEM_PERCENT=$(ps -p $JENKINS_PID -o %mem --no-headers | cut -d. -f1)
if [ "$MEM_PERCENT" -gt 80 ]; then
    echo "内存使用率过高建议:"
    echo "1. 增加JVM堆内存大小"
    echo "2. 优化插件配置"
    echo "3. 清理旧的构建数据"
fi

# 线程建议
if [ "$THREAD_COUNT" -gt 500 ]; then
    echo "线程数过多建议:"
    echo "1. 检查是否有线程泄漏"
    echo "2. 优化并发配置"
    echo "3. 重启Jenkins服务"
fi

echo "诊断完成。"

14.4 恢复策略

备份恢复

Jenkins备份脚本：

bash 复制代码 
#!/bin/bash
# jenkins_backup.sh

JENKINS_HOME="/var/lib/jenkins"
BACKUP_DIR="/backup/jenkins"
DATE=$(date +%Y%m%d_%H%M%S)
BACKUP_NAME="jenkins_backup_$DATE"
RETENTION_DAYS=30

# 创建备份目录
mkdir -p $BACKUP_DIR

echo "=== Jenkins备份开始 ==="
echo "备份时间: $(date)"
echo "Jenkins主目录: $JENKINS_HOME"
echo "备份目录: $BACKUP_DIR"
echo "备份名称: $BACKUP_NAME"
echo ""

# 1. 停止Jenkins服务（可选）
read -p "是否停止Jenkins服务进行完整备份？(y/N): " -n 1 -r
echo
if [[ $REPLY =~ ^[Yy]$ ]]; then
    echo "停止Jenkins服务..."
    systemctl stop jenkins
    JENKINS_STOPPED=true
else
    echo "在线备份模式（可能存在数据不一致）"
    JENKINS_STOPPED=false
fi

# 2. 创建备份
echo "开始备份..."
cd $(dirname $JENKINS_HOME)

# 排除不需要备份的目录和文件
EXCLUDE_PATTERNS=(
    "--exclude=workspace/*"
    "--exclude=builds/*/workspace"
    "--exclude=.m2/repository"
    "--exclude=caches"
    "--exclude=logs/*.log"
    "--exclude=*.tmp"
    "--exclude=*.log"
)

# 创建tar备份
tar czf "$BACKUP_DIR/${BACKUP_NAME}.tar.gz" \
    "${EXCLUDE_PATTERNS[@]}" \
    $(basename $JENKINS_HOME)

if [ $? -eq 0 ]; then
    echo "✓ 备份创建成功: $BACKUP_DIR/${BACKUP_NAME}.tar.gz"
else
    echo "✗ 备份创建失败"
    exit 1
fi

# 3. 备份数据库（如果使用外部数据库）
if [ -f "$JENKINS_HOME/config.xml" ]; then
    DB_CONFIG=$(grep -i "jdbc" $JENKINS_HOME/config.xml)
    if [ -n "$DB_CONFIG" ]; then
        echo "检测到外部数据库配置，请手动备份数据库"
    fi
fi

# 4. 创建备份清单
cat > "$BACKUP_DIR/${BACKUP_NAME}_manifest.txt" << EOF
Jenkins备份清单
================

备份时间: $(date)
备份版本: $BACKUP_NAME
Jenkins版本: $(cat $JENKINS_HOME/jenkins.install.InstallUtil.lastExecVersion 2>/dev/null || echo "未知")
系统信息: $(uname -a)

备份内容:
- 配置文件
- 作业配置
- 插件
- 用户数据
- 凭据
- 系统配置

排除内容:
- 工作空间
- 构建日志
- 缓存文件
- 临时文件

备份大小: $(du -h $BACKUP_DIR/${BACKUP_NAME}.tar.gz | cut -f1)
EOF

# 5. 验证备份
echo "验证备份完整性..."
tar tzf "$BACKUP_DIR/${BACKUP_NAME}.tar.gz" > /dev/null
if [ $? -eq 0 ]; then
    echo "✓ 备份文件完整性验证通过"
else
    echo "✗ 备份文件损坏"
fi

# 6. 重启Jenkins服务
if [ "$JENKINS_STOPPED" = true ]; then
    echo "重启Jenkins服务..."
    systemctl start jenkins
    
    # 等待服务启动
    echo "等待Jenkins启动..."
    for i in {1..30}; do
        if curl -s http://localhost:8080 > /dev/null; then
            echo "✓ Jenkins已成功启动"
            break
        fi
        sleep 2
    done
fi

# 7. 清理旧备份
echo "清理超过 $RETENTION_DAYS 天的旧备份..."
find $BACKUP_DIR -name "jenkins_backup_*.tar.gz" -mtime +$RETENTION_DAYS -delete
find $BACKUP_DIR -name "jenkins_backup_*_manifest.txt" -mtime +$RETENTION_DAYS -delete

# 8. 备份统计
echo ""
echo "=== 备份完成 ==="
echo "备份文件: $BACKUP_DIR/${BACKUP_NAME}.tar.gz"
echo "备份大小: $(du -h $BACKUP_DIR/${BACKUP_NAME}.tar.gz | cut -f1)"
echo "清单文件: $BACKUP_DIR/${BACKUP_NAME}_manifest.txt"
echo "当前备份总数: $(ls $BACKUP_DIR/jenkins_backup_*.tar.gz 2>/dev/null | wc -l)"
echo "备份目录使用: $(du -sh $BACKUP_DIR | cut -f1)"
echo "完成时间: $(date)"

Jenkins恢复脚本：

bash 复制代码 
#!/bin/bash
# jenkins_restore.sh

JENKINS_HOME="/var/lib/jenkins"
BACKUP_DIR="/backup/jenkins"

if [ $# -ne 1 ]; then
    echo "用法: $0 <备份文件名>"
    echo "可用备份:"
    ls -la $BACKUP_DIR/jenkins_backup_*.tar.gz 2>/dev/null | awk '{print $9}' | xargs -n1 basename
    exit 1
fi

BACKUP_FILE="$1"
BACKUP_PATH="$BACKUP_DIR/$BACKUP_FILE"

if [ ! -f "$BACKUP_PATH" ]; then
    echo "✗ 备份文件不存在: $BACKUP_PATH"
    exit 1
fi

echo "=== Jenkins恢复开始 ==="
echo "恢复时间: $(date)"
echo "备份文件: $BACKUP_PATH"
echo "Jenkins主目录: $JENKINS_HOME"
echo ""

# 1. 确认恢复操作
echo "⚠ 警告: 此操作将覆盖当前Jenkins配置和数据"
read -p "确认继续恢复操作？(yes/no): " -r
if [[ ! $REPLY =~ ^yes$ ]]; then
    echo "恢复操作已取消"
    exit 0
fi

# 2. 停止Jenkins服务
echo "停止Jenkins服务..."
systemctl stop jenkins
if [ $? -eq 0 ]; then
    echo "✓ Jenkins服务已停止"
else
    echo "✗ 停止Jenkins服务失败"
    exit 1
fi

# 3. 备份当前配置
echo "备份当前配置..."
CURRENT_BACKUP="$BACKUP_DIR/current_backup_$(date +%Y%m%d_%H%M%S).tar.gz"
if [ -d "$JENKINS_HOME" ]; then
    tar czf "$CURRENT_BACKUP" -C $(dirname $JENKINS_HOME) $(basename $JENKINS_HOME)
    echo "✓ 当前配置已备份到: $CURRENT_BACKUP"
fi

# 4. 清理现有目录
echo "清理现有Jenkins目录..."
rm -rf "$JENKINS_HOME"
mkdir -p "$JENKINS_HOME"

# 5. 恢复备份
echo "恢复备份数据..."
tar xzf "$BACKUP_PATH" -C $(dirname $JENKINS_HOME)
if [ $? -eq 0 ]; then
    echo "✓ 备份数据恢复成功"
else
    echo "✗ 备份数据恢复失败"
    
    # 尝试恢复当前备份
    if [ -f "$CURRENT_BACKUP" ]; then
        echo "尝试恢复原始配置..."
        tar xzf "$CURRENT_BACKUP" -C $(dirname $JENKINS_HOME)
    fi
    exit 1
fi

# 6. 修复权限
echo "修复文件权限..."
chown -R jenkins:jenkins "$JENKINS_HOME"
chmod -R 755 "$JENKINS_HOME"
chmod 600 "$JENKINS_HOME"/secrets/* 2>/dev/null

# 7. 验证关键文件
echo "验证关键文件..."
KEY_FILES=(
    "$JENKINS_HOME/config.xml"
    "$JENKINS_HOME/secrets/master.key"
    "$JENKINS_HOME/secrets/hudson.util.Secret"
)

for file in "${KEY_FILES[@]}"; do
    if [ -f "$file" ]; then
        echo "✓ $file 存在"
    else
        echo "⚠ $file 不存在"
    fi
done

# 8. 启动Jenkins服务
echo "启动Jenkins服务..."
systemctl start jenkins

# 9. 等待服务启动
echo "等待Jenkins启动..."
for i in {1..60}; do
    if curl -s http://localhost:8080 > /dev/null 2>&1; then
        echo "✓ Jenkins已成功启动"
        break
    fi
    
    if [ $i -eq 60 ]; then
        echo "✗ Jenkins启动超时"
        echo "请检查日志: journalctl -u jenkins -f"
        exit 1
    fi
    
    echo "等待中... ($i/60)"
    sleep 2
done

# 10. 验证恢复
echo "验证恢复结果..."
if curl -s http://localhost:8080/api/json > /dev/null 2>&1; then
    echo "✓ Jenkins API响应正常"
else
    echo "⚠ Jenkins API无响应"
fi

# 11. 显示恢复信息
echo ""
echo "=== 恢复完成 ==="
echo "恢复文件: $BACKUP_PATH"
echo "当前备份: $CURRENT_BACKUP"
echo "完成时间: $(date)"
echo "Jenkins访问地址: http://localhost:8080"

# 12. 恢复后检查清单
echo ""
echo "=== 恢复后检查清单 ==="
echo "□ 检查Jenkins是否正常启动"
echo "□ 验证用户登录功能"
echo "□ 检查作业配置是否完整"
echo "□ 验证插件是否正常工作"
echo "□ 测试构建功能"
echo "□ 检查Agent连接状态"
echo "□ 验证凭据是否可用"
echo "□ 检查系统配置"

灾难恢复

灾难恢复计划：

bash 复制代码 
#!/bin/bash
# jenkins_disaster_recovery.sh

DR_CONFIG="/etc/jenkins/disaster_recovery.conf"
LOG_FILE="/var/log/jenkins_dr.log"

# 加载配置
if [ -f "$DR_CONFIG" ]; then
    source "$DR_CONFIG"
else
    echo "灾难恢复配置文件不存在: $DR_CONFIG"
    exit 1
fi

log_message() {
    echo "$(date '+%Y-%m-%d %H:%M:%S') - $1" | tee -a "$LOG_FILE"
}

log_message "=== Jenkins灾难恢复开始 ==="

# 1. 评估损坏程度
assess_damage() {
    log_message "评估系统损坏程度..."
    
    local damage_level=0
    
    # 检查Jenkins主目录
    if [ ! -d "$JENKINS_HOME" ]; then
        log_message "✗ Jenkins主目录丢失"
        damage_level=$((damage_level + 3))
    elif [ ! -f "$JENKINS_HOME/config.xml" ]; then
        log_message "✗ 主配置文件丢失"
        damage_level=$((damage_level + 2))
    else
        log_message "✓ Jenkins主目录存在"
    fi
    
    # 检查作业配置
    if [ -d "$JENKINS_HOME/jobs" ]; then
        job_count=$(find "$JENKINS_HOME/jobs" -name "config.xml" | wc -l)
        log_message "✓ 发现 $job_count 个作业配置"
    else
        log_message "✗ 作业配置目录丢失"
        damage_level=$((damage_level + 2))
    fi
    
    # 检查插件
    if [ -d "$JENKINS_HOME/plugins" ]; then
        plugin_count=$(ls "$JENKINS_HOME/plugins"/*.jpi 2>/dev/null | wc -l)
        log_message "✓ 发现 $plugin_count 个插件"
    else
        log_message "✗ 插件目录丢失"
        damage_level=$((damage_level + 1))
    fi
    
    # 检查凭据
    if [ -f "$JENKINS_HOME/secrets/master.key" ]; then
        log_message "✓ 主密钥存在"
    else
        log_message "✗ 主密钥丢失"
        damage_level=$((damage_level + 2))
    fi
    
    log_message "损坏评估完成，损坏级别: $damage_level"
    echo $damage_level
}

# 2. 选择恢复策略
select_recovery_strategy() {
    local damage_level=$1
    
    if [ $damage_level -ge 6 ]; then
        echo "full_restore"
    elif [ $damage_level -ge 3 ]; then
        echo "partial_restore"
    else
        echo "repair"
    fi
}

# 3. 完全恢复
full_restore() {
    log_message "执行完全恢复..."
    
    # 停止服务
    systemctl stop jenkins
    
    # 查找最新备份
    latest_backup=$(ls -t "$BACKUP_DIR"/jenkins_backup_*.tar.gz 2>/dev/null | head -1)
    if [ -z "$latest_backup" ]; then
        log_message "✗ 未找到可用备份"
        return 1
    fi
    
    log_message "使用备份: $latest_backup"
    
    # 清理现有目录
    rm -rf "$JENKINS_HOME"
    mkdir -p "$JENKINS_HOME"
    
    # 恢复备份
    tar xzf "$latest_backup" -C $(dirname "$JENKINS_HOME")
    
    # 修复权限
    chown -R jenkins:jenkins "$JENKINS_HOME"
    
    # 启动服务
    systemctl start jenkins
    
    log_message "完全恢复完成"
}

# 4. 部分恢复
partial_restore() {
    log_message "执行部分恢复..."
    
    # 备份当前状态
    current_backup="$BACKUP_DIR/emergency_backup_$(date +%Y%m%d_%H%M%S).tar.gz"
    if [ -d "$JENKINS_HOME" ]; then
        tar czf "$current_backup" -C $(dirname "$JENKINS_HOME") $(basename "$JENKINS_HOME")
        log_message "当前状态已备份: $current_backup"
    fi
    
    # 查找最新备份
    latest_backup=$(ls -t "$BACKUP_DIR"/jenkins_backup_*.tar.gz 2>/dev/null | head -1)
    if [ -z "$latest_backup" ]; then
        log_message "✗ 未找到可用备份"
        return 1
    fi
    
    # 创建临时目录
    temp_dir="/tmp/jenkins_restore_$$"
    mkdir -p "$temp_dir"
    
    # 解压备份到临时目录
    tar xzf "$latest_backup" -C "$temp_dir"
    
    # 选择性恢复
    backup_jenkins_home="$temp_dir/$(basename "$JENKINS_HOME")"
    
    # 恢复配置文件
    if [ ! -f "$JENKINS_HOME/config.xml" ] && [ -f "$backup_jenkins_home/config.xml" ]; then
        cp "$backup_jenkins_home/config.xml" "$JENKINS_HOME/"
        log_message "✓ 恢复主配置文件"
    fi
    
    # 恢复作业配置
    if [ ! -d "$JENKINS_HOME/jobs" ] && [ -d "$backup_jenkins_home/jobs" ]; then
        cp -r "$backup_jenkins_home/jobs" "$JENKINS_HOME/"
        log_message "✓ 恢复作业配置"
    fi
    
    # 恢复插件
    if [ ! -d "$JENKINS_HOME/plugins" ] && [ -d "$backup_jenkins_home/plugins" ]; then
        cp -r "$backup_jenkins_home/plugins" "$JENKINS_HOME/"
        log_message "✓ 恢复插件"
    fi
    
    # 恢复凭据
    if [ ! -d "$JENKINS_HOME/secrets" ] && [ -d "$backup_jenkins_home/secrets" ]; then
        cp -r "$backup_jenkins_home/secrets" "$JENKINS_HOME/"
        log_message "✓ 恢复凭据"
    fi
    
    # 清理临时目录
    rm -rf "$temp_dir"
    
    # 修复权限
    chown -R jenkins:jenkins "$JENKINS_HOME"
    
    log_message "部分恢复完成"
}

# 5. 修复模式
repair_mode() {
    log_message "执行修复模式..."
    
    # 修复权限
    chown -R jenkins:jenkins "$JENKINS_HOME"
    chmod -R 755 "$JENKINS_HOME"
    chmod 600 "$JENKINS_HOME"/secrets/* 2>/dev/null
    
    # 检查配置文件
    if [ -f "$JENKINS_HOME/config.xml" ]; then
        if ! xmllint --noout "$JENKINS_HOME/config.xml" 2>/dev/null; then
            log_message "⚠ 主配置文件XML格式错误，尝试修复"
            # 这里可以添加XML修复逻辑
        fi
    fi
    
    # 重建索引
    if [ -d "$JENKINS_HOME/jobs" ]; then
        find "$JENKINS_HOME/jobs" -name "builds" -type d -exec chmod 755 {} \;
        log_message "✓ 重建作业索引"
    fi
    
    log_message "修复完成"
}

# 6. 验证恢复结果
verify_recovery() {
    log_message "验证恢复结果..."
    
    # 启动Jenkins（如果未运行）
    if ! systemctl is-active jenkins &>/dev/null; then
        systemctl start jenkins
        sleep 10
    fi
    
    # 等待服务启动
    for i in {1..30}; do
        if curl -s http://localhost:8080 > /dev/null 2>&1; then
            log_message "✓ Jenkins服务响应正常"
            break
        fi
        sleep 2
    done
    
    # 检查API
    if curl -s http://localhost:8080/api/json > /dev/null 2>&1; then
        log_message "✓ Jenkins API正常"
    else
        log_message "✗ Jenkins API异常"
        return 1
    fi
    
    # 检查作业
    job_count=$(find "$JENKINS_HOME/jobs" -name "config.xml" 2>/dev/null | wc -l)
    log_message "✓ 发现 $job_count 个作业"
    
    # 检查插件
    plugin_count=$(ls "$JENKINS_HOME/plugins"/*.jpi 2>/dev/null | wc -l)
    log_message "✓ 发现 $plugin_count 个插件"
    
    log_message "恢复验证完成"
}

# 7. 生成恢复报告
generate_recovery_report() {
    local strategy=$1
    local start_time=$2
    local end_time=$(date)
    
    cat > "/tmp/jenkins_recovery_report_$(date +%Y%m%d_%H%M%S).txt" << EOF
Jenkins灾难恢复报告
==================

恢复策略: $strategy
开始时间: $start_time
结束时间: $end_time

系统信息:
- 主机名: $(hostname)
- 操作系统: $(uname -a)
- Jenkins版本: $(cat "$JENKINS_HOME/jenkins.install.InstallUtil.lastExecVersion" 2>/dev/null || echo "未知")

恢复结果:
- Jenkins服务状态: $(systemctl is-active jenkins)
- 作业数量: $(find "$JENKINS_HOME/jobs" -name "config.xml" 2>/dev/null | wc -l)
- 插件数量: $(ls "$JENKINS_HOME/plugins"/*.jpi 2>/dev/null | wc -l)
- 用户数量: $(find "$JENKINS_HOME/users" -name "config.xml" 2>/dev/null | wc -l)

详细日志: $LOG_FILE

恢复后检查清单:
□ 验证用户登录
□ 测试作业执行
□ 检查插件功能
□ 验证Agent连接
□ 测试构建功能
□ 检查权限设置
□ 验证凭据可用性
□ 测试通知功能
EOF
    
    log_message "恢复报告已生成: /tmp/jenkins_recovery_report_$(date +%Y%m%d_%H%M%S).txt"
}

# 主流程
main() {
    local start_time=$(date)
    
    # 评估损坏
    damage_level=$(assess_damage)
    
    # 选择策略
    strategy=$(select_recovery_strategy $damage_level)
    log_message "选择恢复策略: $strategy"
    
    # 执行恢复
    case $strategy in
        "full_restore")
            full_restore
            ;;
        "partial_restore")
            partial_restore
            ;;
        "repair")
            repair_mode
            ;;
    esac
    
    # 验证结果
    verify_recovery
    
    # 生成报告
    generate_recovery_report "$strategy" "$start_time"
    
    log_message "=== Jenkins灾难恢复完成 ==="
}

# 执行主流程
main "$@"

灾难恢复配置文件：

bash 复制代码 
# /etc/jenkins/disaster_recovery.conf

# Jenkins配置
JENKINS_HOME="/var/lib/jenkins"
JENKINS_USER="jenkins"
JENKINS_SERVICE="jenkins"

# 备份配置
BACKUP_DIR="/backup/jenkins"
REMOTE_BACKUP_HOST="backup.company.com"
REMOTE_BACKUP_USER="backup"
REMOTE_BACKUP_PATH="/backup/jenkins"

# 恢复配置
MAX_RECOVERY_TIME="3600"  # 最大恢复时间（秒）
VERIFICATION_TIMEOUT="300"  # 验证超时时间（秒）

# 通知配置
NOTIFY_EMAIL="admin@company.com"
NOTIFY_SLACK_WEBHOOK="https://hooks.slack.com/services/xxx"

# 监控配置
HEALTH_CHECK_URL="http://localhost:8080/api/json"
HEALTH_CHECK_INTERVAL="30"  # 健康检查间隔（秒）

14.5 监控和预警

健康检查

Jenkins健康检查脚本：

python 复制代码 
#!/usr/bin/env python3
# jenkins_health_check.py

import requests
import json
import time
import sys
import subprocess
from datetime import datetime, timedelta

class JenkinsHealthChecker:
    def __init__(self, config_file='health_check_config.json'):
        self.config = self.load_config(config_file)
        self.results = []
        
    def load_config(self, config_file):
        """加载健康检查配置"""
        default_config = {
            'jenkins_url': 'http://localhost:8080',
            'username': '',
            'api_token': '',
            'timeout': 30,
            'checks': {
                'service_status': True,
                'api_response': True,
                'disk_space': True,
                'memory_usage': True,
                'queue_length': True,
                'node_status': True,
                'plugin_status': True,
                'build_health': True
            },
            'thresholds': {
                'disk_usage_warning': 80,
                'disk_usage_critical': 90,
                'memory_usage_warning': 80,
                'memory_usage_critical': 90,
                'queue_length_warning': 10,
                'queue_length_critical': 20,
                'response_time_warning': 5,
                'response_time_critical': 10
            },
            'notification': {
                'email_enabled': False,
                'slack_enabled': False,
                'webhook_url': ''
            }
        }
        
        try:
            with open(config_file, 'r') as f:
                config = json.load(f)
                # 合并默认配置
                for key, value in default_config.items():
                    if key not in config:
                        config[key] = value
                return config
        except FileNotFoundError:
            with open(config_file, 'w') as f:
                json.dump(default_config, f, indent=2)
            return default_config
    
    def check_service_status(self):
        """检查Jenkins服务状态"""
        try:
            result = subprocess.run(['systemctl', 'is-active', 'jenkins'], 
                                  capture_output=True, text=True)
            status = result.stdout.strip()
            
            if status == 'active':
                return {'status': 'OK', 'message': 'Jenkins服务正在运行'}
            else:
                return {'status': 'CRITICAL', 'message': f'Jenkins服务状态: {status}'}
        except Exception as e:
            return {'status': 'UNKNOWN', 'message': f'无法检查服务状态: {e}'}
    
    def check_api_response(self):
        """检查API响应"""
        try:
            start_time = time.time()
            
            auth = None
            if self.config['username'] and self.config['api_token']:
                auth = (self.config['username'], self.config['api_token'])
            
            response = requests.get(
                f"{self.config['jenkins_url']}/api/json",
                auth=auth,
                timeout=self.config['timeout']
            )
            
            response_time = time.time() - start_time
            
            if response.status_code == 200:
                if response_time > self.config['thresholds']['response_time_critical']:
                    status = 'CRITICAL'
                elif response_time > self.config['thresholds']['response_time_warning']:
                    status = 'WARNING'
                else:
                    status = 'OK'
                
                return {
                    'status': status,
                    'message': f'API响应正常，响应时间: {response_time:.2f}秒',
                    'response_time': response_time
                }
            else:
                return {
                    'status': 'CRITICAL',
                    'message': f'API响应异常，状态码: {response.status_code}'
                }
        except requests.exceptions.Timeout:
            return {'status': 'CRITICAL', 'message': 'API请求超时'}
        except Exception as e:
            return {'status': 'CRITICAL', 'message': f'API请求失败: {e}'}
    
    def check_disk_space(self):
        """检查磁盘空间"""
        try:
            result = subprocess.run(['df', '-h', '/var/lib/jenkins'], 
                                  capture_output=True, text=True)
            lines = result.stdout.strip().split('\n')
            if len(lines) >= 2:
                usage_str = lines[1].split()[4].rstrip('%')
                usage = int(usage_str)
                
                if usage >= self.config['thresholds']['disk_usage_critical']:
                    status = 'CRITICAL'
                elif usage >= self.config['thresholds']['disk_usage_warning']:
                    status = 'WARNING'
                else:
                    status = 'OK'
                
                return {
                    'status': status,
                    'message': f'磁盘使用率: {usage}%',
                    'usage': usage
                }
            else:
                return {'status': 'UNKNOWN', 'message': '无法获取磁盘使用率'}
        except Exception as e:
            return {'status': 'UNKNOWN', 'message': f'检查磁盘空间失败: {e}'}
    
    def check_memory_usage(self):
        """检查内存使用"""
        try:
            # 获取Jenkins进程内存使用
            result = subprocess.run(['pgrep', '-f', 'jenkins.war'], 
                                  capture_output=True, text=True)
            if result.stdout.strip():
                pid = result.stdout.strip().split()[0]
                
                mem_result = subprocess.run(['ps', '-p', pid, '-o', '%mem'], 
                                          capture_output=True, text=True)
                if len(mem_result.stdout.strip().split('\n')) >= 2:
                    mem_usage_str = mem_result.stdout.strip().split('\n')[1].strip()
                    mem_usage = float(mem_usage_str)
                    
                    if mem_usage >= self.config['thresholds']['memory_usage_critical']:
                        status = 'CRITICAL'
                    elif mem_usage >= self.config['thresholds']['memory_usage_warning']:
                        status = 'WARNING'
                    else:
                        status = 'OK'
                    
                    return {
                        'status': status,
                        'message': f'内存使用率: {mem_usage:.1f}%',
                        'usage': mem_usage
                    }
            
            return {'status': 'UNKNOWN', 'message': '无法获取内存使用率'}
        except Exception as e:
            return {'status': 'UNKNOWN', 'message': f'检查内存使用失败: {e}'}
    
    def check_queue_length(self):
        """检查构建队列长度"""
        try:
            auth = None
            if self.config['username'] and self.config['api_token']:
                auth = (self.config['username'], self.config['api_token'])
            
            response = requests.get(
                f"{self.config['jenkins_url']}/queue/api/json",
                auth=auth,
                timeout=self.config['timeout']
            )
            
            if response.status_code == 200:
                queue_data = response.json()
                queue_length = len(queue_data.get('items', []))
                
                if queue_length >= self.config['thresholds']['queue_length_critical']:
                    status = 'CRITICAL'
                elif queue_length >= self.config['thresholds']['queue_length_warning']:
                    status = 'WARNING'
                else:
                    status = 'OK'
                
                return {
                    'status': status,
                    'message': f'构建队列长度: {queue_length}',
                    'queue_length': queue_length
                }
            else:
                return {'status': 'UNKNOWN', 'message': '无法获取队列信息'}
        except Exception as e:
            return {'status': 'UNKNOWN', 'message': f'检查队列长度失败: {e}'}
    
    def check_node_status(self):
        """检查节点状态"""
        try:
            auth = None
            if self.config['username'] and self.config['api_token']:
                auth = (self.config['username'], self.config['api_token'])
            
            response = requests.get(
                f"{self.config['jenkins_url']}/computer/api/json",
                auth=auth,
                timeout=self.config['timeout']
            )
            
            if response.status_code == 200:
                computer_data = response.json()
                computers = computer_data.get('computer', [])
                
                total_nodes = len(computers)
                offline_nodes = []
                
                for computer in computers:
                    if computer.get('offline', False):
                        offline_nodes.append(computer.get('displayName', 'Unknown'))
                
                offline_count = len(offline_nodes)
                
                if offline_count > 0:
                    status = 'WARNING' if offline_count < total_nodes else 'CRITICAL'
                    message = f'离线节点: {offline_count}/{total_nodes} ({offline_nodes})'
                else:
                    status = 'OK'
                    message = f'所有节点在线: {total_nodes}/{total_nodes}'
                
                return {
                    'status': status,
                    'message': message,
                    'total_nodes': total_nodes,
                    'offline_nodes': offline_count
                }
            else:
                return {'status': 'UNKNOWN', 'message': '无法获取节点信息'}
        except Exception as e:
            return {'status': 'UNKNOWN', 'message': f'检查节点状态失败: {e}'}
    
    def run_all_checks(self):
        """运行所有健康检查"""
        print(f"开始Jenkins健康检查 - {datetime.now()}")
        print("=" * 50)
        
        checks = {
            'service_status': self.check_service_status,
            'api_response': self.check_api_response,
            'disk_space': self.check_disk_space,
            'memory_usage': self.check_memory_usage,
            'queue_length': self.check_queue_length,
            'node_status': self.check_node_status
        }
        
        overall_status = 'OK'
        
        for check_name, check_func in checks.items():
            if self.config['checks'].get(check_name, True):
                result = check_func()
                result['check_name'] = check_name
                result['timestamp'] = datetime.now().isoformat()
                
                self.results.append(result)
                
                # 更新整体状态
                if result['status'] == 'CRITICAL':
                    overall_status = 'CRITICAL'
                elif result['status'] == 'WARNING' and overall_status != 'CRITICAL':
                    overall_status = 'WARNING'
                
                # 输出结果
                status_icon = {
                    'OK': '✓',
                    'WARNING': '⚠',
                    'CRITICAL': '✗',
                    'UNKNOWN': '?'
                }.get(result['status'], '?')
                
                print(f"{status_icon} {check_name}: {result['message']}")
        
        print("=" * 50)
        print(f"整体状态: {overall_status}")
        
        # 生成报告
        self.generate_report(overall_status)
        
        return overall_status
    
    def generate_report(self, overall_status):
        """生成健康检查报告"""
        report = {
            'timestamp': datetime.now().isoformat(),
            'overall_status': overall_status,
            'checks': self.results,
            'summary': {
                'total_checks': len(self.results),
                'ok_count': len([r for r in self.results if r['status'] == 'OK']),
                'warning_count': len([r for r in self.results if r['status'] == 'WARNING']),
                'critical_count': len([r for r in self.results if r['status'] == 'CRITICAL']),
                'unknown_count': len([r for r in self.results if r['status'] == 'UNKNOWN'])
            }
        }
        
        # 保存报告
        report_file = f"/tmp/jenkins_health_report_{datetime.now().strftime('%Y%m%d_%H%M%S')}.json"
        with open(report_file, 'w') as f:
            json.dump(report, f, indent=2)
        
        print(f"健康检查报告已保存: {report_file}")
        
        # 发送通知
        if overall_status in ['WARNING', 'CRITICAL']:
            self.send_notification(report)
    
    def send_notification(self, report):
        """发送通知"""
        if self.config['notification']['slack_enabled']:
            self.send_slack_notification(report)
    
    def send_slack_notification(self, report):
        """发送Slack通知"""
        try:
            webhook_url = self.config['notification']['webhook_url']
            if not webhook_url:
                return
            
            color = {
                'OK': 'good',
                'WARNING': 'warning',
                'CRITICAL': 'danger'
            }.get(report['overall_status'], 'warning')
            
            fields = []
            for check in report['checks']:
                if check['status'] != 'OK':
                    fields.append({
                        'title': check['check_name'],
                        'value': check['message'],
                        'short': True
                    })
            
            payload = {
                'attachments': [{
                    'color': color,
                    'title': f"Jenkins健康检查 - {report['overall_status']}",
                    'fields': fields,
                    'footer': 'Jenkins Health Check',
                    'ts': int(datetime.now().timestamp())
                }]
            }
            
            response = requests.post(webhook_url, json=payload, timeout=10)
            response.raise_for_status()
            
            print("Slack通知已发送")
        except Exception as e:
            print(f"发送Slack通知失败: {e}")

def main():
    checker = JenkinsHealthChecker()
    status = checker.run_all_checks()
    
    # 设置退出码
    exit_codes = {
        'OK': 0,
        'WARNING': 1,
        'CRITICAL': 2,
        'UNKNOWN': 3
    }
    
    sys.exit(exit_codes.get(status, 3))

if __name__ == '__main__':
    main()

14.6 本章小结

本章详细介绍了Jenkins故障排除的各个方面：

故障排除概述：

  • 常见故障类型分类
  • 标准诊断流程
  • 问题分析方法

日志分析：

  • 日志系统配置
  • 自动化日志分析
  • 实时监控机制

常见问题诊断：

  • 启动问题诊断
  • 性能问题分析
  • 系统资源监控

恢复策略：

  • 备份恢复流程
  • 灾难恢复计划
  • 数据完整性验证

监控和预警：

  • 健康检查机制
  • 自动化监控
  • 通知和报警

通过本章的学习，您应该能够：

  • 快速诊断Jenkins常见问题
  • 实施有效的恢复策略
  • 建立完善的监控体系
  • 制定灾难恢复计划

下一章预告

下一章我们将学习Jenkins最佳实践，包括：

  • 企业级部署架构
  • 安全最佳实践
  • 性能优化策略
  • 运维管理规范
  • 团队协作模式

练习题

理论练习

  1. 故障分类

    • 列举Jenkins常见的5种故障类型
    • 说明每种故障的典型症状
    • 描述相应的诊断方法

  2. 日志分析

    • 解释Jenkins日志级别的含义
    • 设计日志轮转策略
    • 制定日志监控规则

  3. 恢复策略

    • 比较完全恢复和部分恢复的适用场景
    • 设计备份验证流程
    • 制定RTO和RPO指标

实践练习

  1. 故障模拟

    bash 复制代码 
    # 模拟磁盘空间不足
dd if=/dev/zero of=/var/lib/jenkins/large_file bs=1M count=1000

# 模拟配置文件损坏
echo "invalid xml" > /var/lib/jenkins/config.xml

# 模拟内存不足
stress --vm 1 --vm-bytes 2G --timeout 60s

  2. 监控脚本

    • 编写Jenkins健康检查脚本
    • 实现自动化告警机制
    • 配置监控仪表板

  3. 恢复演练

    • 执行完整的备份恢复流程
    • 验证数据完整性
    • 测试业务连续性

思考题

  1. 如何设计一个高可用的Jenkins架构来最小化故障影响？

  2. 在云环境中，Jenkins故障排除有哪些特殊考虑？

  3. 如何平衡监控的全面性和性能开销？

  4. 制定Jenkins灾难恢复计划时需要考虑哪些关键因素？

  5. 如何建立有效的故障预防机制？

相关推荐
INFINI Labs4 小时前
Easysearch 国产替代 Elasticsearch：8 大核心问题解读
大数据·elasticsearch·搜索引擎·jenkins·easysearch
wodongx1236 小时前
从零开始部署Android环境的Jenkins CI/CD流水线（docker环境，Win系统）
android·docker·jenkins
oLingXi126 小时前
Unity开发CI/CD工具Jenkins的安装(Windows10)
运维·unity·ci/cd·jenkins
ThreeAu.19 小时前
Jenkins+Tomcat持续集成教程
ci/cd·tomcat·jenkins
BatyTao21 小时前
第十三章 Ant与Jenkins的集成
jmeter·jenkins·ant
qq_5693841221 小时前
Jenkins(速通版)
java·kubernetes·jenkins
haogexiaole2 天前
es的java调用
java·elasticsearch·jenkins
这样の我2 天前
elasticsearch更换为opensearch
大数据·elasticsearch·jenkins
会飞的小蛮猪2 天前
Jenkins运维之路(共享库集成流水线发布)
ci/cd·jenkins·自动化运维
会飞的小蛮猪2 天前
Jenkins运维之路(初次调试共享库)
运维·经验分享·docker·容器·jenkins
热门推荐
01Multisim使用教程详尽版--(2025最新版)02UV 工具安装与国内镜像源配置指南03GitHub 镜像站点04Spec-Kit 使用指南0546个Nano-banana 精选提示词，持续更新中06UV安装并设置国内源07保姆级教程：手把手教你用Dify实现完美多轮对话（附Chatflow和提示词）082025年华为杯研赛数学建模竞赛C题完整参考论文 （含模型、MATLAB和Python代码）09VsCode远程Copilot无法使用Claude Agent问题10Linux下V2Ray安装配置指南