升级手册
# ArgoCD 升级操作手册
> 适用范围：EKS 生产集群 ArgoCD `v3.0.6 → v3.1.x → v3.2.x` 两跳升级
> 生产命名空间：`argocd` | 演练命名空间：`argocd-test`
---
## 目录
1. [目录结构](#目录结构)
2. [前置条件](#前置条件)
3. [升级 YAML 准备](#升级-yaml-准备)
4. [生产升级流程](#生产升级流程)
5. [回滚流程](#回滚流程)
6. [从备份完整恢复](#从备份完整恢复)
7. [升级后巡检](#升级后巡检)
8. [跨集群 RBAC 修复](#跨集群-rbac-修复)
9. [演练流程（argocd-test）](#演练流程argocd-test)
10. [关键配置说明](#关键配置说明)
11. [常见问题](#常见问题)
---
## 目录结构
```
argocd-jenkins/
├── README.md # 本手册
├── argocd-check-v32.sh # 升级后只读巡检（生产/演练通用）
├── argocd-upgrade-post-v31-prod.sh # 生产第一跳升级后修复（NS=argocd）
├── argocd-upgrade-post-v32-prod.sh # 生产第二跳升级后修复（NS=argocd）
├── argocd-backup-prod.sh # 生产备份（NS=argocd，升级前必须执行）
├── argocd-restore-prod.sh # 生产从备份完整恢复（NS=argocd）
├── argocd-fix-manager-rbac-prod.sh # 修复目标集群 argocd-manager RBAC（在目标集群执行）
└── argocd-test/
├── argocd-backup-test.sh # 演练备份（NS=argocd-test）
├── argocd-restore-test.sh # 演练从备份完整恢复（NS=argocd-test）
├── argocd-delete-test.sh # 演练清理命名空间（NS=argocd-test）
├── argocd-fix-manager-rbac-test.sh # 修复目标集群 argocd-manager RBAC
├── argocd-patch-manifest-test.py # 将 install.yaml 的命名空间替换为 argocd-test（演练用）
├── argocd-setup-test.sh # 演练环境初始化（仅 argocd-test NS）
├── argocd-upgrade-post-v31-test.sh # 演练第一跳升级后修复（NS=argocd-test）
└── argocd-upgrade-post-v32-test.sh # 演练第二跳升级后修复（NS=argocd-test）
```
---
## 前置条件
### 工具
```bash
# kubectl 已配置并指向目标集群
kubectl version --client
kubectl config current-context
# python3 + PyYAML（演练 patch 脚本依赖）
python3 --version
pip3 install pyyaml
```
### 权限确认
```bash
# 确认对 argocd 命名空间有 apply 权限
kubectl auth can-i create deployment -n argocd
kubectl auth can-i patch configmap -n argocd
# 确认 ClusterRole 操作权限（install.yaml 含集群级 RBAC）
kubectl auth can-i create clusterrole
```
### 升级前状态快照
```bash
# 确认当前版本
kubectl -n argocd get deployment argocd-server \
-o jsonpath='{.spec.template.spec.containers[0].image}'; echo
# 确认所有 Pod 健康
kubectl -n argocd get pods
# 确认 Redis HA 正常
kubectl -n argocd get pods -l app=argocd-redis-ha
kubectl -n argocd get service argocd-redis-ha-haproxy
# 确认 replicas（升级后需恢复为 2）
kubectl -n argocd get deployment argocd-server argocd-repo-server \
-o=custom-columns='NAME:.metadata.name,REPLICAS:.spec.replicas'
# 确认 argocd-cmd-params-cm 当前值（升级前记录，apply 后会被重置）
kubectl -n argocd get configmap argocd-cmd-params-cm \
-o jsonpath='{.data}' | python3 -m json.tool
```
---
## 升级 YAML 准备
> **说明**：install.yaml 必须使用 `ha/install.yaml`（含 Redis HA StatefulSet），
> 不能用默认的 `install.yaml`。生产直接使用原始 HA YAML，无需 patch 命名空间。
### 回滚备用 YAML（v3.0.6，升级前提前准备）
```bash
# 升级前提前下载，回滚第一跳时需要
curl -fsSL "https://raw.githubusercontent.com/argoproj/argo-cd/v3.0.6/manifests/ha/install.yaml" \
-o argocd-install-v3.0.6-ha-prod.yaml
```
### 第一跳 YAML（v3.1.x）
```bash
# 确认最新 v3.1.x patch 版本：https://github.com/argoproj/argo-cd/releases
# 示例使用 v3.1.16，实际替换为当前最新版本
V31="v3.1.16"
curl -fsSL "https://raw.githubusercontent.com/argoproj/argo-cd/${V31}/manifests/ha/install.yaml" \
-o argocd-install-v31-ha-prod.yaml
echo "下载完成: argocd-install-v31-ha-prod.yaml"
```
### 第二跳 YAML（v3.2.x）
```bash
# 确认最新 v3.2.x patch 版本：https://github.com/argoproj/argo-cd/releases
V32="v3.2.12" # 替换为实际版本，如 v3.2.3
curl -fsSL "https://raw.githubusercontent.com/argoproj/argo-cd/${V32}/manifests/ha/install.yaml" \
-o argocd-install-v32-ha-prod.yaml
echo "下载完成: argocd-install-v32-ha-prod.yaml"
```
### dry-run 预检（apply 前必做）
```bash
# 第一跳预检
kubectl apply --dry-run=server -f argocd-install-v31-ha-prod.yaml -n argocd 2>&1 \
| grep -E "error|Error|Warning" || echo "dry-run 通过"
# 第二跳预检
kubectl apply --dry-run=server -f argocd-install-v32-ha-prod.yaml -n argocd 2>&1 \
| grep -E "error|Error|Warning" || echo "dry-run 通过"
```
---
## 生产升级流程
### Step 0：升级前备份
```bash
# 在 argocd-jenkins/ 根目录执行
./argocd-backup-prod.sh
# 输出：./argocd-backup-YYYYMMDD-HHMMSS/
# 确认末尾出现 "备份完整性验证通过" 再继续
```
备份内容：Application / AppProject / ApplicationSet CR、ConfigMap、Secret、
ClusterRole/CRB、CRD、Deployment/StatefulSet/Service，并生成 `manifest.txt` 记录资源数量。
---
### Step 1：第一跳升级（v3.0.6 → v3.1.x）
```bash
cd argocd-jenkins/ # 回到根目录
# 1. 应用新版本 YAML
kubectl apply -f argocd-install-v31-ha-prod.yaml -n argocd
# 2. 等待 rollout 完成（按顺序等，确保依赖链就绪）
kubectl -n argocd rollout status statefulset/argocd-redis-ha-server --timeout=300s
kubectl -n argocd rollout status deployment/argocd-redis-ha-haproxy --timeout=300s
kubectl -n argocd rollout status deployment/argocd-server --timeout=300s
kubectl -n argocd rollout status deployment/argocd-repo-server --timeout=300s
kubectl -n argocd rollout status deployment/argocd-applicationset-controller --timeout=300s
kubectl -n argocd rollout status deployment/argocd-notifications-controller --timeout=300s
kubectl -n argocd rollout status statefulset/argocd-application-controller --timeout=300s
# 3. 立即执行修复脚本（apply 会通过三路合并重置 argocd-cmd-params-cm，必须立即恢复）
./argocd-upgrade-post-v31-prod.sh
```
修复脚本自动处理：
- 恢复 `redis.server=argocd-redis-ha-haproxy:6379`
- 恢复 `server.insecure=true`
- 强制恢复 `replicas=2`（argocd-server、argocd-repo-server）
- 校验 `argocd-manager-role` 保护状态
- Redis HA 连通性验证
---
### Step 2：第二跳升级（v3.1.x → v3.2.x）
```bash
# 1. 应用新版本 YAML
kubectl apply -f argocd-install-v32-ha-prod.yaml -n argocd
# 2. 等待 rollout 完成
kubectl -n argocd rollout status statefulset/argocd-redis-ha-server --timeout=300s
kubectl -n argocd rollout status deployment/argocd-redis-ha-haproxy --timeout=300s
kubectl -n argocd rollout status deployment/argocd-server --timeout=300s
kubectl -n argocd rollout status deployment/argocd-repo-server --timeout=300s
kubectl -n argocd rollout status deployment/argocd-applicationset-controller --timeout=300s
kubectl -n argocd rollout status deployment/argocd-notifications-controller --timeout=300s
kubectl -n argocd rollout status statefulset/argocd-application-controller --timeout=300s
# 3. 立即执行修复脚本
./argocd-upgrade-post-v32-prod.sh
```
v3.2.x 额外检查（修复脚本自动执行）：
- `applicationset-controller` RBAC 新增 `coordination.k8s.io/leases` 权限
- 手动验证：`kubectl get clusterrole argocd-applicationset-controller -o yaml | grep coordination`
---
### Step 3：升级后巡检
```bash
./argocd-check-v32.sh
# 输出目录：/tmp/argocd-check-YYYYMMDD-HHMMSS/
# 关注末尾 PASS / WARN / FAIL 统计，FAIL=0 视为通过
```
---
## 回滚流程
### 触发条件
- `CrashLoopBackOff` 持续 > 5 分钟
- Degraded Application 比例 > 30%
- Redis HA 连通失败且无法自愈
### 第二跳回滚（v3.2.x → v3.1.x）
```bash
# 1. 回滚镜像
kubectl apply -f argocd-install-v31-ha-prod.yaml -n argocd
# 2. 立即修复 CM（apply 会重置，必须立即 patch，不能等待）
kubectl -n argocd patch configmap argocd-cmd-params-cm --type merge \
-p '{"data":{"redis.server":"argocd-redis-ha-haproxy:6379","server.insecure":"true"}}'
# 3. 恢复 replicas=2
kubectl -n argocd scale deployment argocd-server argocd-repo-server --replicas=2
# 4. 重启使 CM 生效
kubectl -n argocd rollout restart deployment/argocd-server deployment/argocd-repo-server
kubectl -n argocd rollout restart statefulset/argocd-application-controller
kubectl -n argocd rollout status deployment/argocd-server --timeout=180s
```
### 第一跳回滚（v3.1.x → v3.0.6）
```bash
kubectl apply -f argocd-install-v3.0.6-ha-prod.yaml -n argocd
kubectl -n argocd patch configmap argocd-cmd-params-cm --type merge \
-p '{"data":{"redis.server":"argocd-redis-ha-haproxy:6379","server.insecure":"true"}}'
kubectl -n argocd scale deployment argocd-server argocd-repo-server --replicas=2
kubectl -n argocd rollout restart deployment/argocd-server deployment/argocd-repo-server
kubectl -n argocd rollout restart statefulset/argocd-application-controller
kubectl -n argocd rollout status deployment/argocd-server --timeout=180s
```
---
## 从备份完整恢复
适用场景：回滚后仍异常、CR/ConfigMap/Secret 数据损坏。
```bash
# 在 argocd-jenkins/ 根目录执行
# 脚本启动时会显示备份 manifest.txt 内容，并要求输入 yes 确认 kubectl context
./argocd-restore-prod.sh argocd-backup-<YYYYMMDD-HHMMSS>/
```
恢复顺序（任一步骤失败立即中止）：
| 步骤 | 内容 |
|---|---|
| 1 | CRD apply + 等待 `Established` |
| 2 | 命名空间 |
| 3 | ClusterRole / ClusterRoleBinding |
| 4 | ServiceAccount / Role / RoleBinding / ConfigMap / Secret |
| 4a | ConfigMap 恢复后强制校验 redis.server + server.insecure |
| 5 | Service / Deployment / StatefulSet + 等待 rollout 就绪 |
| 6 | Application / AppProject / ApplicationSet CR |
| 验证 | 与 manifest.txt 核对资源数量，不匹配则 exit 1 |
---
## 升级后巡检
```bash
./argocd-check-v32.sh
```
检查项：
- argocd-server 镜像版本符合 v3.2.x
- 所有 Pod Running，无 CrashLoop
- Pod 重启次数（>0 需关注）
- argocd-cmd-params-cm 中 redis.server / server.insecure 值
- replicas=2（argocd-server、argocd-repo-server）
- Redis HA HAProxy ping 连通性
- applicationset-controller RBAC（coordination.k8s.io/leases）
---
## 跨集群 RBAC 修复
**症状**：ArgoCD UI 中目标集群 Application 显示 `SyncFailed`
**日志**：`User "system:serviceaccount:kube-system:argocd-manager" cannot get resource`
```bash
# 1. 切换到目标集群（不是 ArgoCD 所在集群）
kubectl config use-context <target-cluster-context>
# 2. 执行修复
./argocd-fix-manager-rbac-prod.sh
# 3. 切回 ArgoCD 集群，触发重新同步
kubectl config use-context <argocd-cluster-context>
argocd app sync <app-name>
# 或在 ArgoCD UI 点击 Sync / Refresh
```
---
## 演练流程（argocd-test）
演练在独立测试集群的 `argocd-test` 命名空间执行，与生产完全隔离。
### 环境初始化
```bash
cd argocd-test/
# 1. 下载 v3.0.6 HA YAML（如已存在跳过）
curl -fsSL https://raw.githubusercontent.com/argoproj/argo-cd/v3.0.6/manifests/ha/install.yaml \
-o argocd-install-v3.0.6-ha.yaml
# 2. Patch 命名空间（argocd → argocd-test）
python3 argocd-patch-manifest-test.py argocd-install-v3.0.6-ha.yaml argocd-install-v3.0.6-ha-test.yaml
# 3. 部署基线（或使用 setup 脚本自动完成以上步骤）
./argocd-setup-test.sh
```
### 第一跳演练（v3.0.6 → v3.1.x）
```bash
# 1. 下载并 patch v3.1.x YAML
V31="v3.1.16" # 替换为当前最新版本
curl -fsSL "https://raw.githubusercontent.com/argoproj/argo-cd/${V31}/manifests/ha/install.yaml" \
-o argocd-install-${V31}-ha.yaml
python3 argocd-patch-manifest-test.py argocd-install-${V31}-ha.yaml argocd-install-v31-ha-test.yaml
# 2. dry-run 预检
kubectl apply --dry-run=server -f argocd-install-v31-ha-test.yaml -n argocd-test 2>&1 \
| grep -E "error|Error|Warning" || echo "dry-run 通过"
# 3. 应用
kubectl apply -f argocd-install-v31-ha-test.yaml -n argocd-test
# 4. 等待 rollout
kubectl -n argocd-test rollout status statefulset/argocd-redis-ha-server --timeout=300s
kubectl -n argocd-test rollout status deployment/argocd-redis-ha-haproxy --timeout=300s
kubectl -n argocd-test rollout status deployment/argocd-server --timeout=300s
kubectl -n argocd-test rollout status deployment/argocd-repo-server --timeout=300s
kubectl -n argocd-test rollout status deployment/argocd-applicationset-controller --timeout=300s
kubectl -n argocd-test rollout status deployment/argocd-notifications-controller --timeout=300s
kubectl -n argocd-test rollout status statefulset/argocd-application-controller --timeout=300s
# 5. 执行修复脚本
./argocd-upgrade-post-v31-test.sh
```
### 第二跳演练（v3.1.x → v3.2.x）
```bash
V32="v3.2.x" # 替换为实际版本
curl -fsSL "https://raw.githubusercontent.com/argoproj/argo-cd/${V32}/manifests/ha/install.yaml" \
-o argocd-install-${V32}-ha.yaml
python3 argocd-patch-manifest-test.py argocd-install-${V32}-ha.yaml argocd-install-v32-ha-test.yaml
kubectl apply --dry-run=server -f argocd-install-v32-ha-test.yaml -n argocd-test 2>&1 \
| grep -E "error|Error|Warning" || echo "dry-run 通过"
kubectl apply -f argocd-install-v32-ha-test.yaml -n argocd-test
kubectl -n argocd-test rollout status statefulset/argocd-redis-ha-server --timeout=300s
kubectl -n argocd-test rollout status deployment/argocd-redis-ha-haproxy --timeout=300s
kubectl -n argocd-test rollout status deployment/argocd-server --timeout=300s
kubectl -n argocd-test rollout status deployment/argocd-repo-server --timeout=300s
kubectl -n argocd-test rollout status deployment/argocd-applicationset-controller --timeout=300s
kubectl -n argocd-test rollout status deployment/argocd-notifications-controller --timeout=300s
kubectl -n argocd-test rollout status statefulset/argocd-application-controller --timeout=300s
./argocd-upgrade-post-v32-test.sh
```
### 演练验证
```bash
# Pod 状态
kubectl -n argocd-test get pods -o wide
# CM 关键字段
kubectl -n argocd-test get configmap argocd-cmd-params-cm \
-o jsonpath='{.data.redis\.server}{"\n"}{.data.server\.insecure}{"\n"}'
# 期望：argocd-redis-ha-haproxy:6379 / true
# 镜像版本
kubectl -n argocd-test get deployment argocd-server \
-o jsonpath='{.spec.template.spec.containers[?(@.name=="argocd-server")].image}'; echo
# 详细巡检参考 argocd-test/README.md 阶段五
```
### 清理演练环境
```bash
cd argocd-test/
# 传入备份目录名作为确认凭证，脚本会二次确认后删除
./argocd-delete-test.sh ./argocd-backup-<YYYYMMDD-HHMMSS>/
```
---
## 关键配置说明
| 配置项 | 值 | 原因 |
|---|---|---|
| `redis.server` | `argocd-redis-ha-haproxy:6379` | `kubectl apply -f install.yaml` 通过三路合并重置此字段，**每次 apply 后必须立即 patch** |
| `server.insecure` | `true` | TLS 在 Ingress 层终止，同上，apply 后会被重置 |
| `replicas` | 生产=2 / 演练=1 | install.yaml 可能覆盖为 1，升级后升级脚本自动强制恢复 |
| `ha/install.yaml` | 必须用 HA 版本 | 包含 Redis HA StatefulSet，普通 install.yaml 无 Redis HA |
### argocd-patch-manifest-test.py 说明
仅演练环境需要。作用：将 `install.yaml` 中所有 `namespace: argocd` 深度替换为 `argocd-test`，
覆盖 metadata、subjects、webhooks 等所有位置，并保留 CRD。
```bash
# 用法
python3 argocd-test/argocd-patch-manifest-test.py <input-ha.yaml> <output-test.yaml>
```
生产环境直接使用原始 `ha/install.yaml`，不需要 patch。
---
## 常见问题
**Q: apply 后 argocd-server CrashLoop，日志显示无法连接 Redis**
A: `install.yaml` 三路合并重置了 `argocd-cmd-params-cm`。立即执行：
```bash
kubectl -n argocd patch configmap argocd-cmd-params-cm --type merge \
-p '{"data":{"redis.server":"argocd-redis-ha-haproxy:6379","server.insecure":"true"}}'
kubectl -n argocd rollout restart deployment/argocd-server
```
**Q: applicationset-controller 报 Forbidden（v3.2.x）**
A: v3.2.x 新增了 `coordination.k8s.io/leases` 权限。检查 ClusterRole 是否包含：
```bash
kubectl get clusterrole argocd-applicationset-controller -o yaml | grep -A3 coordination
```
若缺失，重新 apply `argocd-install-v32-ha-prod.yaml` 并执行 `./argocd-upgrade-post-v32-prod.sh`。
**Q: 恢复后 Application 数量少于备份记录**
A: `argocd-restore-prod.sh` 末尾会对比 `manifest.txt` 并 exit 1。先检查 CRD 是否 Established：
```bash
kubectl get crd | grep argoproj.io
kubectl wait crd/applications.argoproj.io --for=condition=Established --timeout=60s
```
CRD 就绪后重新执行 `./argocd-restore-prod.sh <backup-dir>`。
**Q: dry-run 报 unknown field 或 CRD 不存在**
A: 目标集群 CRD 版本与 install.yaml 不匹配。先 apply CRD 部分：
```bash
kubectl apply -f argocd-install-v31-ha-prod.yaml -n argocd \
--selector="app.kubernetes.io/part-of=argocd" --dry-run=server
```
**Q: argocd-manager-role 在目标集群丢失**
A: 切换到目标集群执行修复，再回 ArgoCD 集群触发同步：
```bash
kubectl config use-context <target-cluster-context>
./argocd-fix-manager-rbac-prod.sh
kubectl config use-context <argocd-cluster-context>
argocd app sync <app-name>
```
**Q: Redis HA HAProxy ping 失败（需要密码）**
A: 生产 Redis 配置了 AUTH，redis-cli 需要 `-a <password>` 参数，或通过 Secret 获取：
```bash
REDIS_PASS=$(kubectl -n argocd get secret argocd-redis-ha -o jsonpath='{.data.auth}' | base64 -d)
kubectl -n argocd exec <redis-ha-pod> -c redis -- \
redis-cli -h argocd-redis-ha-haproxy -p 6379 -a "${REDIS_PASS}" ping
```备份 恢复 升级 巡检 补丁等配置文件
#!/usr/bin/env bash
# argocd-backup-prod.sh
# 备份 argocd 命名空间所有资源 + 集群级 RBAC + CRD
#
# 使用方法：
# ./argocd-backup-prod.sh
# 备份目录：./argocd-backup-YYYYMMDD-HHMMSS/
set -euo pipefail
NS="argocd"
BACKUP_DIR="./argocd-backup-$(date +%Y%m%d-%H%M%S)"
mkdir -p "${BACKUP_DIR}"
echo "====================================================================="
echo " ArgoCD 资源备份"
echo " 命名空间: ${NS}"
echo " 备份目录: ${BACKUP_DIR}"
echo " 时间: $(date '+%Y-%m-%d %H:%M:%S')"
echo "====================================================================="
echo ""
# ─────────────────────────────────────────────────────────────────────────────
# Step 1: ArgoCD CR（最重要：Application / AppProject / ApplicationSet）
# ─────────────────────────────────────────────────────────────────────────────
echo "[1/5] 备份 ArgoCD CR（Application / AppProject / ApplicationSet）..."
for cr in applications appprojects applicationsets; do
count=$(kubectl -n "${NS}" get "${cr}.argoproj.io" --no-headers 2>/dev/null | wc -l | tr -d ' ' || echo "0")
if [[ "${count}" -gt 0 ]]; then
kubectl -n "${NS}" get "${cr}.argoproj.io" -o yaml > "${BACKUP_DIR}/${cr}.yaml"
echo " ✅ ${cr}: ${count} 个 → ${cr}.yaml"
else
echo " ℹ️ ${cr}: 无资源，跳过"
fi
done
echo ""
# ─────────────────────────────────────────────────────────────────────────────
# Step 2: ConfigMap（argocd-cm / argocd-cmd-params-cm / argocd-rbac-cm 等）
# ─────────────────────────────────────────────────────────────────────────────
echo "[2/5] 备份 ConfigMap..."
kubectl -n "${NS}" get configmap -o yaml > "${BACKUP_DIR}/configmaps.yaml"
count=$(kubectl -n "${NS}" get configmap --no-headers | wc -l | tr -d ' ')
echo " ✅ ConfigMap: ${count} 个 → configmaps.yaml"
echo ""
# ─────────────────────────────────────────────────────────────────────────────
# Step 3: Secret（repo 凭证、argocd-secret、TLS 等）
# ─────────────────────────────────────────────────────────────────────────────
echo "[3/5] 备份 Secret..."
kubectl -n "${NS}" get secret -o yaml > "${BACKUP_DIR}/secrets.yaml"
count=$(kubectl -n "${NS}" get secret --no-headers | wc -l | tr -d ' ')
echo " ✅ Secret: ${count} 个 → secrets.yaml（含敏感数据，请妥善保管）"
echo ""
# ─────────────────────────────────────────────────────────────────────────────
# Step 4: 命名空间内其余资源（Deployment / Service / SA / Role 等）
# ─────────────────────────────────────────────────────────────────────────────
echo "[4/5] 备份命名空间内工作负载资源..."
for resource in deployment statefulset service serviceaccount role rolebinding; do
kubectl -n "${NS}" get "${resource}" -o yaml > "${BACKUP_DIR}/${resource}s.yaml" 2>/dev/null
done
echo " ✅ deployment / statefulset / service / serviceaccount / role / rolebinding → 各独立文件"
echo ""
# ─────────────────────────────────────────────────────────────────────────────
# Step 5: 集群级资源（ClusterRole / CRB / CRD）
# ─────────────────────────────────────────────────────────────────────────────
echo "[5/5] 备份集群级 RBAC 和 CRD..."
# ClusterRole
kubectl get clusterrole -o yaml 2>/dev/null | python3 - <<'PYEOF' > "${BACKUP_DIR}/clusterroles.yaml" || true
import sys, yaml
docs = [d for d in yaml.safe_load_all(sys.stdin)
if d and 'argocd' in d.get('metadata', {}).get('name', '')]
print(yaml.dump_all(docs, default_flow_style=False, allow_unicode=True))
PYEOF
cr_count=$(grep -c "^kind: ClusterRole$" "${BACKUP_DIR}/clusterroles.yaml" 2>/dev/null || true)
if [[ "${cr_count:-0}" -gt 0 ]]; then
echo " ✅ ClusterRole: ${cr_count} 个 → clusterroles.yaml"
else
echo " ⚠️ ClusterRole: 0 个（权限不足或无 argocd ClusterRole，升级时由 install.yaml 重建）"
fi
# ClusterRoleBinding
kubectl get clusterrolebinding -o yaml 2>/dev/null | python3 - <<'PYEOF' > "${BACKUP_DIR}/clusterrolebindings.yaml" || true
import sys, yaml
docs = [d for d in yaml.safe_load_all(sys.stdin)
if d and 'argocd' in d.get('metadata', {}).get('name', '')]
print(yaml.dump_all(docs, default_flow_style=False, allow_unicode=True))
PYEOF
crb_count=$(grep -c "^kind: ClusterRoleBinding$" "${BACKUP_DIR}/clusterrolebindings.yaml" 2>/dev/null || true)
if [[ "${crb_count:-0}" -gt 0 ]]; then
echo " ✅ ClusterRoleBinding: ${crb_count} 个 → clusterrolebindings.yaml"
else
echo " ⚠️ ClusterRoleBinding: 0 个（权限不足或无 argocd CRB，升级时由 install.yaml 重建）"
fi
# CRD
kubectl get crd -o yaml 2>/dev/null | python3 - <<'PYEOF' > "${BACKUP_DIR}/crds.yaml" || true
import sys, yaml
docs = [d for d in yaml.safe_load_all(sys.stdin)
if d and 'argoproj.io' in d.get('metadata', {}).get('name', '')]
print(yaml.dump_all(docs, default_flow_style=False, allow_unicode=True))
PYEOF
crd_count=$(grep -c "^kind: CustomResourceDefinition$" "${BACKUP_DIR}/crds.yaml" 2>/dev/null || true)
if [[ "${crd_count:-0}" -gt 0 ]]; then
echo " ✅ CRD: ${crd_count} 个 → crds.yaml"
else
echo " ⚠️ CRD: 0 个（权限不足或无 argoproj.io CRD，升级时由 install.yaml 重建）"
fi
echo ""
# ─────────────────────────────────────────────────────────────────────────────
# 备份完整性验证 + manifest.txt 生成（供 restore 恢复核对用）
# ─────────────────────────────────────────────────────────────────────────────
verify_backup() {
local fail=0
echo "====================================================================="
echo " 备份完整性验证"
echo "====================================================================="
echo ""
echo "--- 文件检查 ---"
for f in configmaps.yaml secrets.yaml crds.yaml clusterroles.yaml clusterrolebindings.yaml \
deployments.yaml statefulsets.yaml services.yaml serviceaccounts.yaml \
roles.yaml rolebindings.yaml; do
local path="${BACKUP_DIR}/${f}"
if [[ -f "${path}" ]] && [[ -s "${path}" ]]; then
local lines
lines=$(wc -l < "${path}" | tr -d ' ')
echo " ✅ ${f} (${lines} 行)"
else
echo " ℹ️ ${f} 不存在或为空（如无对应资源则正常）"
fi
done
echo ""
echo "--- CRD 数量验证（ArgoCD 预期 >= 4 个）---"
local crd_count=0
if [[ -f "${BACKUP_DIR}/crds.yaml" ]]; then
crd_count=$(grep -c "^kind: CustomResourceDefinition$" "${BACKUP_DIR}/crds.yaml" 2>/dev/null || true)
fi
if [[ "${crd_count}" -lt 4 ]]; then
echo " ⚠️ CRD: ${crd_count} 个 < 4（权限不足无法备份，升级时由 install.yaml 重建，不阻断备份）"
else
echo " ✅ CRD: ${crd_count} 个（>= 4）"
fi
echo ""
echo "--- 关键 ConfigMap 验证 ---"
if grep -q "argocd-cmd-params-cm" "${BACKUP_DIR}/configmaps.yaml" 2>/dev/null; then
echo " ✅ argocd-cmd-params-cm 存在于备份"
else
echo " ⛔ argocd-cmd-params-cm 未在 configmaps.yaml 中找到"
fail=1
fi
echo ""
echo "--- 关键 Secret 验证 ---"
if grep -q "argocd-secret" "${BACKUP_DIR}/secrets.yaml" 2>/dev/null; then
echo " ✅ argocd-secret 存在于备份"
else
echo " ⚠️ argocd-secret 未在 secrets.yaml 中找到（如已外部管理则正常）"
fi
echo ""
echo "--- 生成 manifest.txt（restore 恢复时核对数量用）---"
{
echo "backup_time=$(date '+%Y-%m-%d %H:%M:%S')"
echo "namespace=${NS}"
echo "crd_count=${crd_count}"
for cr in applications appprojects applicationsets; do
local cnt
cnt=$(kubectl -n "${NS}" get "${cr}.argoproj.io" --no-headers 2>/dev/null \
| wc -l | tr -d ' ' || echo "0")
echo "${cr}_count=${cnt}"
done
echo "configmap_count=$(kubectl -n "${NS}" get configmap --no-headers 2>/dev/null \
| wc -l | tr -d ' ' || echo "0")"
echo "secret_count=$(kubectl -n "${NS}" get secret --no-headers 2>/dev/null \
| wc -l | tr -d ' ' || echo "0")"
echo "deployment_count=$(kubectl -n "${NS}" get deployment --no-headers 2>/dev/null \
| wc -l | tr -d ' ' || echo "0")"
} > "${BACKUP_DIR}/manifest.txt"
echo " ✅ manifest.txt 已生成:"
cat "${BACKUP_DIR}/manifest.txt" | sed 's/^/ /'
echo ""
if [[ "${fail}" -eq 1 ]]; then
echo " ⛔ 备份验证失败！请确认集群状态后重新备份"
echo " 不建议使用不完整的备份作为恢复依据"
exit 1
fi
echo " ✅ 备份完整性验证通过"
echo ""
}
verify_backup
# ─────────────────────────────────────────────────────────────────────────────
# 备份清单
# ─────────────────────────────────────────────────────────────────────────────
echo "====================================================================="
echo " 备份完成 ✅"
echo " 目录: ${BACKUP_DIR}"
echo ""
ls -lh "${BACKUP_DIR}"
echo ""
echo " 下一步:"
echo " 升级失败需回滚时: ./argocd-restore-prod.sh ${BACKUP_DIR}"
echo "====================================================================="#!/usr/bin/env bash
# argocd-restore-prod.sh
# 从备份恢复 argocd 命名空间资源 + 集群级 RBAC + CRD
#
# 使用方法：
# ./argocd-restore-prod.sh <backup-dir>
#
# 恢复顺序（每步均有错误检查，任何失败立即中止）：
# 1. 前置校验（context 确认 + 备份目录完整性）
# 2. CRD（apply + 等待 Established，再进行下一步）
# 3. 命名空间
# 4. ClusterRole / ClusterRoleBinding
# 5. ServiceAccount / Role / RoleBinding / ConfigMap / Secret（+ CM 强制校验）
# 6. Service / Deployment / StatefulSet（+ 等待 rollout 就绪）
# 7. Application / AppProject / ApplicationSet CR
# 8. 恢复后计数核对（与 manifest.txt 对比）
set -euo pipefail
NS="argocd"
REDIS_HA_ADDR="argocd-redis-ha-haproxy:6379"
if [[ $# -lt 1 ]]; then
echo "Usage: $0 <backup-dir>"
echo " 示例: $0 ./argocd-backup-20260803-120000"
exit 1
fi
BACKUP_DIR="$1"
if [[ ! -d "${BACKUP_DIR}" ]]; then
echo "⛔ 备份目录不存在: ${BACKUP_DIR}"
exit 1
fi
echo "====================================================================="
echo " ArgoCD 资源恢复"
echo " 命名空间: ${NS}"
echo " 备份目录: ${BACKUP_DIR}"
echo " 时间: $(date '+%Y-%m-%d %H:%M:%S')"
echo "====================================================================="
echo ""
# ─────────────────────────────────────────────────────────────────────────────
# 前置校验 1: kubectl context 确认（防止误操作到错误集群）
# ─────────────────────────────────────────────────────────────────────────────
CURRENT_CTX=$(kubectl config current-context 2>/dev/null || echo "unknown")
echo " ⚠️ 当前 kubectl context: ${CURRENT_CTX}"
echo " 恢复目标命名空间: ${NS}"
if [[ -f "${BACKUP_DIR}/manifest.txt" ]]; then
echo ""
echo " 备份时记录 (manifest.txt):"
cat "${BACKUP_DIR}/manifest.txt" | sed 's/^/ /'
fi
echo ""
read -rp " 确认恢复到此集群？输入 yes 继续: " CTX_CONFIRM
if [[ "${CTX_CONFIRM}" != "yes" ]]; then
echo " 已取消。请先切换到正确的 kubectl context 后重试"
exit 0
fi
echo ""
# ─────────────────────────────────────────────────────────────────────────────
# 前置校验 2: 关键备份文件存在性检查
# ─────────────────────────────────────────────────────────────────────────────
echo "[PRE] 备份目录完整性检查..."
PRE_FAIL=0
for f in configmaps.yaml secrets.yaml crds.yaml; do
if [[ ! -f "${BACKUP_DIR}/${f}" ]] || [[ ! -s "${BACKUP_DIR}/${f}" ]]; then
echo " ⛔ 关键备份文件缺失或为空: ${f}"
PRE_FAIL=1
else
echo " ✅ ${f}"
fi
done
if [[ "${PRE_FAIL}" -eq 1 ]]; then
echo " ⛔ 备份目录不完整，中止恢复（请重新执行 ./argocd-backup-prod.sh）"
exit 1
fi
echo " ✅ 备份目录检查通过"
echo ""
# ─────────────────────────────────────────────────────────────────────────────
# 工具：清理 kubectl export YAML 中的运行时字段，apply 前必须去除
# ─────────────────────────────────────────────────────────────────────────────
clean_yaml() {
python3 - "$1" <<'PYEOF'
import sys, yaml
def clean(obj):
if not isinstance(obj, dict):
return obj
# 移除运行时字段
for key in ('resourceVersion', 'uid', 'generation',
'creationTimestamp', 'selfLink', 'managedFields'):
obj.pop(key, None)
metadata = obj.get('metadata', {})
for key in ('resourceVersion', 'uid', 'generation',
'creationTimestamp', 'selfLink', 'managedFields'):
metadata.pop(key, None)
# 移除 last-applied-configuration（可能超大，引发 apply 失败或三路合并异常覆盖 redis.server）
annotations = metadata.get('annotations', {})
if annotations:
annotations.pop('kubectl.kubernetes.io/last-applied-configuration', None)
if not annotations:
metadata.pop('annotations', None)
# 移除 status（避免 apply 冲突）
obj.pop('status', None)
# 递归清理子字段
for v in obj.values():
if isinstance(v, dict):
clean(v)
elif isinstance(v, list):
for item in v:
if isinstance(item, dict):
clean(item)
return obj
input_file = sys.argv[1]
with open(input_file) as f:
docs = list(yaml.safe_load_all(f))
cleaned = [clean(d) for d in docs if d]
print(yaml.dump_all(cleaned, default_flow_style=False, allow_unicode=True))
PYEOF
}
# ─────────────────────────────────────────────────────────────────────────────
# 工具：apply 并严格检查错误（任何失败立即 exit 1，不静默忽略）
# ─────────────────────────────────────────────────────────────────────────────
apply_if_exists() {
local file="$1"
local label="$2"
if [[ -f "${file}" ]] && [[ -s "${file}" ]]; then
local tmp
tmp=$(mktemp /tmp/argocd-restore-XXXXXX.yaml)
clean_yaml "${file}" > "${tmp}"
local out
if ! out=$(kubectl apply -f "${tmp}" 2>&1); then
echo " ⛔ ${label} apply 失败（中止恢复）:"
echo "${out}" | sed 's/^/ /'
rm -f "${tmp}"
exit 1
fi
echo "${out}" | grep -v "^$" | sed 's/^/ /'
rm -f "${tmp}"
echo " ✅ ${label}"
else
echo " ℹ️ ${label}: 备份文件不存在或为空，跳过"
fi
}
# ─────────────────────────────────────────────────────────────────────────────
# Step 1: CRD（apply + 等待每个 CRD Established，再继续）
# ─────────────────────────────────────────────────────────────────────────────
echo "[1/6] 恢复 CRD..."
apply_if_exists "${BACKUP_DIR}/crds.yaml" "CRD (argoproj.io)"
echo " 等待 CRD Established（最多 60s/个，未就绪则后续 CR apply 必定失败）..."
for crd in applications.argoproj.io appprojects.argoproj.io applicationsets.argoproj.io; do
if kubectl get crd "${crd}" &>/dev/null 2>&1; then
if kubectl wait crd/"${crd}" --for=condition=Established --timeout=60s 2>/dev/null; then
echo " ✅ ${crd} Established"
else
echo " ⛔ ${crd} 等待 Established 超时，中止恢复"
exit 1
fi
fi
done
echo ""
# ─────────────────────────────────────────────────────────────────────────────
# Step 2: 命名空间
# ─────────────────────────────────────────────────────────────────────────────
echo "[2/6] 恢复命名空间 ${NS}..."
kubectl create namespace "${NS}" --dry-run=client -o yaml | kubectl apply -f -
echo " ✅ namespace/${NS} 就绪"
echo ""
# ─────────────────────────────────────────────────────────────────────────────
# Step 3: 集群级 RBAC
# ─────────────────────────────────────────────────────────────────────────────
echo "[3/6] 恢复 ClusterRole / ClusterRoleBinding..."
apply_if_exists "${BACKUP_DIR}/clusterroles.yaml" "ClusterRole"
apply_if_exists "${BACKUP_DIR}/clusterrolebindings.yaml" "ClusterRoleBinding"
echo ""
# ─────────────────────────────────────────────────────────────────────────────
# Step 4: 基础资源（ServiceAccount / Role / RoleBinding / ConfigMap / Secret）
# ─────────────────────────────────────────────────────────────────────────────
echo "[4/6] 恢复基础资源..."
apply_if_exists "${BACKUP_DIR}/serviceaccounts.yaml" "ServiceAccount"
apply_if_exists "${BACKUP_DIR}/roles.yaml" "Role"
apply_if_exists "${BACKUP_DIR}/rolebindings.yaml" "RoleBinding"
apply_if_exists "${BACKUP_DIR}/configmaps.yaml" "ConfigMap"
apply_if_exists "${BACKUP_DIR}/secrets.yaml" "Secret"
# ConfigMap 恢复后强制校验并修复 redis.server（防止备份遗漏或三路合并异常覆盖）
echo " [CM 强制校验] 确认 redis.server + server.insecure..."
CURRENT_REDIS=$(kubectl -n "${NS}" get configmap argocd-cmd-params-cm \
-o jsonpath='{.data.redis\.server}' 2>/dev/null || echo "")
CURRENT_INSECURE=$(kubectl -n "${NS}" get configmap argocd-cmd-params-cm \
-o jsonpath='{.data.server\.insecure}' 2>/dev/null || echo "")
if [[ "${CURRENT_REDIS}" != "${REDIS_HA_ADDR}" || "${CURRENT_INSECURE}" != "true" ]]; then
echo " ⚠️ CM 值不正确（redis.server=[${CURRENT_REDIS}] insecure=[${CURRENT_INSECURE}]），强制修复..."
kubectl -n "${NS}" patch configmap argocd-cmd-params-cm \
--type merge \
-p "{\"data\":{\"redis.server\":\"${REDIS_HA_ADDR}\",\"server.insecure\":\"true\"}}"
echo " ✅ redis.server + server.insecure 已强制修复"
else
echo " ✅ redis.server=${CURRENT_REDIS} server.insecure=${CURRENT_INSECURE}"
fi
echo ""
# ─────────────────────────────────────────────────────────────────────────────
# Step 5: 工作负载（Service / Deployment / StatefulSet + 等待 rollout 就绪）
# ─────────────────────────────────────────────────────────────────────────────
echo "[5/6] 恢复工作负载..."
apply_if_exists "${BACKUP_DIR}/services.yaml" "Service"
apply_if_exists "${BACKUP_DIR}/deployments.yaml" "Deployment"
apply_if_exists "${BACKUP_DIR}/statefulsets.yaml" "StatefulSet"
echo " 等待工作负载 rollout 就绪（最多 5 分钟）..."
ROLLOUT_FAIL=0
for deploy in argocd-server argocd-repo-server argocd-applicationset-controller \
argocd-notifications-controller argocd-redis-ha-haproxy; do
if kubectl -n "${NS}" get deployment "${deploy}" &>/dev/null 2>&1; then
replicas=$(kubectl -n "${NS}" get deployment "${deploy}" \
-o jsonpath='{.spec.replicas}' 2>/dev/null || echo "0")
if [[ "${replicas:-0}" -gt 0 ]]; then
if ! kubectl -n "${NS}" rollout status deployment/"${deploy}" --timeout=300s; then
echo " ❌ ${deploy} rollout 超时/失败"
ROLLOUT_FAIL=$((ROLLOUT_FAIL + 1))
fi
fi
fi
done
for sts in argocd-application-controller argocd-redis-ha-server; do
if kubectl -n "${NS}" get statefulset "${sts}" &>/dev/null 2>&1; then
replicas=$(kubectl -n "${NS}" get statefulset "${sts}" \
-o jsonpath='{.spec.replicas}' 2>/dev/null || echo "0")
if [[ "${replicas:-0}" -gt 0 ]]; then
if ! kubectl -n "${NS}" rollout status statefulset/"${sts}" --timeout=300s; then
echo " ❌ ${sts} rollout 超时/失败"
ROLLOUT_FAIL=$((ROLLOUT_FAIL + 1))
fi
fi
fi
done
if [[ "${ROLLOUT_FAIL}" -gt 0 ]]; then
echo " ⛔ ${ROLLOUT_FAIL} 个工作负载 rollout 失败，请检查上方详情后手动处理"
exit 1
fi
echo " ✅ 所有工作负载就绪"
echo ""
# ─────────────────────────────────────────────────────────────────────────────
# Step 6: ArgoCD CR（Application / AppProject / ApplicationSet）
# ─────────────────────────────────────────────────────────────────────────────
echo "[6/6] 恢复 ArgoCD CR..."
apply_if_exists "${BACKUP_DIR}/appprojects.yaml" "AppProject"
apply_if_exists "${BACKUP_DIR}/applications.yaml" "Application"
apply_if_exists "${BACKUP_DIR}/applicationsets.yaml" "ApplicationSet"
echo ""
# ─────────────────────────────────────────────────────────────────────────────
# 恢复后验证（与 manifest.txt 核对，最终确认无丢失）
# ─────────────────────────────────────────────────────────────────────────────
echo "====================================================================="
echo " 恢复后验证"
echo "====================================================================="
echo ""
VERIFY_FAIL=0
echo "--- ArgoCD CR 数量核对 ---"
for cr in applications appprojects applicationsets; do
actual=$(kubectl -n "${NS}" get "${cr}.argoproj.io" --no-headers 2>/dev/null \
| wc -l | tr -d ' ' || echo "0")
if [[ -f "${BACKUP_DIR}/manifest.txt" ]]; then
expected=$(grep "^${cr}_count=" "${BACKUP_DIR}/manifest.txt" 2>/dev/null \
| cut -d= -f2 || echo "")
if [[ -n "${expected}" ]] && [[ "${actual}" -ne "${expected}" ]]; then
echo " ⚠️ ${cr}: 恢复 ${actual} 个，备份记录 ${expected} 个（数量不匹配）"
VERIFY_FAIL=$((VERIFY_FAIL + 1))
else
echo " ✅ ${cr}: ${actual} 个"
fi
else
echo " ✅ ${cr}: ${actual} 个（无 manifest.txt，跳过数量核对）"
fi
done
echo ""
echo "--- argocd-cmd-params-cm 关键字段 ---"
FINAL_REDIS=$(kubectl -n "${NS}" get configmap argocd-cmd-params-cm \
-o jsonpath='{.data.redis\.server}' 2>/dev/null || echo "")
FINAL_INSECURE=$(kubectl -n "${NS}" get configmap argocd-cmd-params-cm \
-o jsonpath='{.data.server\.insecure}' 2>/dev/null || echo "")
if [[ "${FINAL_REDIS}" == "${REDIS_HA_ADDR}" ]]; then
echo " ✅ redis.server: ${FINAL_REDIS}"
else
echo " ⛔ redis.server 最终值错误: [${FINAL_REDIS}]（预期 ${REDIS_HA_ADDR}）"
VERIFY_FAIL=$((VERIFY_FAIL + 1))
fi
if [[ "${FINAL_INSECURE}" == "true" ]]; then
echo " ✅ server.insecure: true"
else
echo " ⚠️ server.insecure: [${FINAL_INSECURE}]（预期 true）"
fi
echo ""
echo "--- CRD 验证 ---"
crd_count=$(kubectl get crd 2>/dev/null | grep -c "argoproj.io" || true)
if [[ "${crd_count}" -lt 4 ]]; then
echo " ⚠️ argoproj.io CRD 数量 ${crd_count} < 4"
else
echo " ✅ argoproj.io CRD: ${crd_count} 个"
fi
echo ""
echo "--- Pod 状态 ---"
kubectl -n "${NS}" get pods -o wide
echo ""
if [[ "${VERIFY_FAIL}" -gt 0 ]]; then
echo "====================================================================="
echo " ⛔ 恢复验证有 ${VERIFY_FAIL} 项失败，请检查上方详情"
echo "====================================================================="
exit 1
fi
echo "====================================================================="
echo " ✅ 恢复完成并验证通过"
echo "====================================================================="#!/usr/bin/env bash
# argocd-fix-manager-rbac-prod.sh
# 修复 argocd-manager ClusterRole/CRB 被误删导致的跨集群 SyncFailed
#
# 症状：
# User "system:serviceaccount:kube-system:argocd-manager" cannot get resource
# "serviceaccounts/services/..." in namespace "xxx" → SyncFailed
#
# 执行集群：生产目标集群（被外部 ArgoCD 管理的那个集群，不是源 ArgoCD 集群）
#
# 使用方法：
# kubectl config use-context <target-cluster-context>
# ./argocd-fix-manager-rbac-prod.sh
set -euo pipefail
echo "====================================================================="
echo " 修复 argocd-manager RBAC"
echo " 集群: $(kubectl config current-context 2>/dev/null || echo 'unknown')"
echo " 时间: $(date '+%Y-%m-%d %H:%M:%S')"
echo "====================================================================="
echo ""
# ─────────────────────────────────────────────────────────────────────────────
# Step 1: 确认现状
# ─────────────────────────────────────────────────────────────────────────────
echo "[1/3] 检查当前状态..."
echo -n " ServiceAccount argocd-manager (kube-system): "
kubectl get sa argocd-manager -n kube-system &>/dev/null && echo "存在 ✅" || echo "不存在 ⚠️"
echo -n " ClusterRole argocd-manager-role: "
kubectl get clusterrole argocd-manager-role &>/dev/null && echo "存在 ✅" || echo "不存在（将重建）"
echo -n " ClusterRoleBinding argocd-manager-role-binding: "
kubectl get clusterrolebinding argocd-manager-role-binding &>/dev/null && echo "存在 ✅" || echo "不存在（将重建）"
echo ""
# ─────────────────────────────────────────────────────────────────────────────
# Step 2: 重建 ClusterRole + ClusterRoleBinding
# ─────────────────────────────────────────────────────────────────────────────
echo "[2/3] 重建 ClusterRole / ClusterRoleBinding..."
kubectl apply -f - <<'EOF'
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: argocd-manager-role
rules:
- apiGroups: ['*']
resources: ['*']
verbs: ['*']
- nonResourceURLs: ['*']
verbs: ['*']
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: argocd-manager-role-binding
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: argocd-manager-role
subjects:
- kind: ServiceAccount
name: argocd-manager
namespace: kube-system
EOF
echo " ✅ ClusterRole + ClusterRoleBinding 已重建"
echo ""
# ─────────────────────────────────────────────────────────────────────────────
# Step 3: 验证权限
# ─────────────────────────────────────────────────────────────────────────────
echo "[3/3] 验证权限恢复..."
for resource in serviceaccounts services deployments; do
result=$(kubectl auth can-i get "${resource}" \
--as=system:serviceaccount:kube-system:argocd-manager \
--all-namespaces 2>/dev/null || echo "no")
if [[ "${result}" == "yes" ]]; then
echo " ✅ get ${resource}: yes"
else
echo " ❌ get ${resource}: ${result}"
fi
done
echo ""
echo "====================================================================="
echo " 修复完成 ✅"
echo ""
echo " 下一步：在源 ArgoCD 触发重新同步"
echo " argocd app sync <app-name>"
echo " 或在 ArgoCD UI 点击 Sync / Refresh"
echo "====================================================================="argocd-fix-manager-rbac-prod.sh
#!/usr/bin/env bash
# argocd-upgrade-post-v31-prod.sh
# ArgoCD v3.0.x → v3.1.x 升级后修复脚本（argocd 生产环境，第一跳）
#
# 生产集群说明：
# - NS="argocd"（生产命名空间）
# - replicas=2（升级后强制恢复，install.yaml 覆盖时可能降为 1）
# - Redis HA：argocd-redis-ha-haproxy:6379
# - argocd-manager ClusterRole 受保护（kubectl apply 不删除，此处仅校验现状）
#
# 使用方法：
# 1. kubectl apply -f argocd-install-v31-ha-prod.yaml -n argocd
# 2. 等待所有 Pod Running
# 3. 立即执行本脚本
set -euo pipefail
NS="argocd"
REDIS_HA_ADDR="argocd-redis-ha-haproxy:6379"
echo "====================================================================="
echo " ArgoCD v3.0.x → v3.1.x 升级后修复（argocd 生产环境 --- 第一跳）"
echo " 命名空间: ${NS}"
echo " 时间: $(date '+%Y-%m-%d %H:%M:%S')"
echo "====================================================================="
echo ""
# ─────────────────────────────────────────────────────────────────────────────
# 工具函数
# ─────────────────────────────────────────────────────────────────────────────
has_node_taint() {
kubectl get nodes -o jsonpath='{.items[*].spec.taints[*].key}' 2>/dev/null \
| tr ' ' '\n' | grep -q "^${1}$"
}
check_pods_healthy() {
local bad
# 排除 Terminating：rolling upgrade 过渡期旧 Pod 正常处于此状态
bad=$(kubectl -n "${NS}" get pods --no-headers 2>/dev/null \
| grep -v "Running\|Completed\|Terminating" | grep -c "." || true)
if [[ "${bad}" -gt 0 ]]; then
echo " ⚠️ 有 ${bad} 个 Pod 未处于 Running 状态："
kubectl -n "${NS}" get pods --no-headers | grep -v "Running\|Completed\|Terminating" || true
echo ""
echo " ⛔ 回滚触发条件：CrashLoopBackOff > 5min / Degraded App > 30%"
echo " 回滚命令：kubectl apply -f argocd-install-v3.0.6-ha-prod.yaml -n ${NS}"
return 1
fi
}
# ─────────────────────────────────────────────────────────────────────────────
# Step 0: 前置校验
# ─────────────────────────────────────────────────────────────────────────────
echo "[0/6] 前置校验..."
CURRENT_IMG=$(kubectl -n "${NS}" get deployment argocd-server \
-o jsonpath='{.spec.template.spec.containers[?(@.name=="argocd-server")].image}' \
2>/dev/null || echo "unknown")
echo " 当前 argocd-server 镜像: ${CURRENT_IMG}"
if echo "${CURRENT_IMG}" | grep -qE "v3\.0\."; then
echo " ⚠️ 镜像仍是 v3.0.x，apply 未完成，请等待 rollout 后重试"
exit 1
fi
# 前置 Pod 状态仅作诊断快照，不阻断脚本（redis config 尚未恢复，Pod 可能 CrashLoop）
check_pods_healthy || echo " ⚠️ 存在异常 Pod，将继续尝试恢复配置..."
# argocd-manager ClusterRole 保护校验（kubectl apply 不会删除此资源，此处仅记录现状）
echo " [保护校验] argocd-manager ClusterRole..."
if kubectl get clusterrole argocd-manager-role &>/dev/null; then
echo " ✅ ClusterRole argocd-manager-role 存在，apply 不影响"
else
echo " ⚠️ ClusterRole argocd-manager-role 不存在！请在目标集群检查："
echo " kubectl config use-context <target-cluster-context>"
echo " ./argocd-test/argocd-fix-manager-rbac-test.sh"
fi
# 确认 Redis HA HAProxy 服务存在
if ! kubectl -n "${NS}" get service argocd-redis-ha-haproxy &>/dev/null; then
echo " ⛔ argocd-redis-ha-haproxy Service 不存在！Redis HA 异常，中止修复"
echo " 请先确认 Redis HA 状态: kubectl -n ${NS} get pods -l app=argocd-redis-ha"
exit 1
fi
echo " ✅ 前置校验通过"
echo ""
# ─────────────────────────────────────────────────────────────────────────────
# Steps 1+2: 原子恢复 redis.server + server.insecure
# ─────────────────────────────────────────────────────────────────────────────
echo "[1-2/6] 检查并原子恢复 redis.server + server.insecure..."
CURRENT_REDIS=$(kubectl -n "${NS}" get configmap argocd-cmd-params-cm \
-o jsonpath='{.data.redis\.server}' 2>/dev/null || echo "")
INSECURE_VAL=$(kubectl -n "${NS}" get configmap argocd-cmd-params-cm \
-o jsonpath='{.data.server\.insecure}' 2>/dev/null || echo "")
CONFIG_CHANGED=0
if [[ "${CURRENT_REDIS}" != "${REDIS_HA_ADDR}" || "${INSECURE_VAL}" != "true" ]]; then
echo " 当前: redis.server=[${CURRENT_REDIS}] server.insecure=[${INSECURE_VAL}]"
kubectl -n "${NS}" patch configmap argocd-cmd-params-cm \
--type merge \
-p "{\"data\":{\"redis.server\":\"${REDIS_HA_ADDR}\",\"server.insecure\":\"true\"}}"
echo " ✅ 已原子写入: redis.server=${REDIS_HA_ADDR} server.insecure=true"
CONFIG_CHANGED=1
else
echo " ✅ redis.server + server.insecure 均已正确，无需修改"
fi
echo ""
# ─────────────────────────────────────────────────────────────────────────────
# Step 3: 生产环境强制恢复 replicas=2
# ─────────────────────────────────────────────────────────────────────────────
echo "[3/6] 检查并恢复 replicas=2（生产必须为 2）..."
REPLICAS_CHANGED=0
for deploy in argocd-server argocd-repo-server; do
cur=$(kubectl -n "${NS}" get deployment "${deploy}" \
-o jsonpath='{.spec.replicas}' 2>/dev/null || echo "0")
if [[ "${cur}" -lt 2 ]]; then
kubectl -n "${NS}" scale deployment "${deploy}" --replicas=2
echo " ✅ ${deploy}: replicas ${cur} → 2"
REPLICAS_CHANGED=1
else
echo " ✅ ${deploy}: replicas=${cur}（已满足 ≥ 2）"
fi
done
echo ""
# ─────────────────────────────────────────────────────────────────────────────
# Step 4: tolerations
# ─────────────────────────────────────────────────────────────────────────────
echo "[4/6] 检查 tolerations..."
if has_node_taint "sretool"; then
echo " 检测到节点存在 sretool:NoSchedule taint，恢复 tolerations..."
TOLERATION_PATCH='{"spec":{"template":{"spec":{"tolerations":[{"key":"sretool","operator":"Equal","value":"true","effect":"NoSchedule"}]}}}}'
kubectl -n "${NS}" patch deployment argocd-server -p "${TOLERATION_PATCH}"
kubectl -n "${NS}" patch deployment argocd-repo-server -p "${TOLERATION_PATCH}"
echo " ✅ tolerations 已恢复"
else
echo " ✅ 节点无 sretool taint，跳过"
fi
echo ""
# ─────────────────────────────────────────────────────────────────────────────
# Step 5: 重启所有组件，使 redis.server + insecure 配置生效
# ─────────────────────────────────────────────────────────────────────────────
echo "[5/6] 重启组件使配置生效..."
if [[ "${CONFIG_CHANGED}" -eq 0 && "${REPLICAS_CHANGED}" -eq 0 ]]; then
echo " ✅ CM 配置和 replicas 均已正确，跳过重启节省时间"
echo " 如需强制重启: kubectl -n ${NS} rollout restart deployment/argocd-server deployment/argocd-repo-server"
else
kubectl -n "${NS}" rollout restart deployment/argocd-server
kubectl -n "${NS}" rollout restart deployment/argocd-repo-server
kubectl -n "${NS}" rollout restart deployment/argocd-notifications-controller
kubectl -n "${NS}" rollout restart statefulset/argocd-application-controller
# applicationset-controller 可能未读 redis config，但一并重启保证 CM 变更生效
(kubectl -n "${NS}" rollout restart deployment/argocd-applicationset-controller 2>/dev/null || true)
ROLLOUT_PIDS=(); ROLLOUT_NAMES=()
kubectl -n "${NS}" rollout status deployment/argocd-server --timeout=180s &
ROLLOUT_PIDS+=($!); ROLLOUT_NAMES+=("argocd-server")
kubectl -n "${NS}" rollout status deployment/argocd-repo-server --timeout=180s &
ROLLOUT_PIDS+=($!); ROLLOUT_NAMES+=("argocd-repo-server")
kubectl -n "${NS}" rollout status deployment/argocd-notifications-controller --timeout=180s &
ROLLOUT_PIDS+=($!); ROLLOUT_NAMES+=("argocd-notifications-controller")
kubectl -n "${NS}" rollout status statefulset/argocd-application-controller --timeout=300s &
ROLLOUT_PIDS+=($!); ROLLOUT_NAMES+=("argocd-application-controller")
(kubectl -n "${NS}" rollout status deployment/argocd-applicationset-controller --timeout=180s || true) &
APPSET_PID=$!
ROLLOUT_FAIL=0
for i in "${!ROLLOUT_PIDS[@]}"; do
if ! wait "${ROLLOUT_PIDS[$i]}"; then
echo " ❌ ${ROLLOUT_NAMES[$i]} rollout 超时/失败"
ROLLOUT_FAIL=$((ROLLOUT_FAIL + 1))
fi
done
wait "${APPSET_PID}" || true
[[ "${ROLLOUT_FAIL}" -eq 0 ]] || { echo " ⛔ ${ROLLOUT_FAIL} 个组件 rollout 失败，请检查上方详情"; exit 1; }
fi
check_pods_healthy
echo " ✅ 所有组件重启完成，连接 Redis HA: ${REDIS_HA_ADDR}"
echo ""
# ─────────────────────────────────────────────────────────────────────────────
# Step 6: Redis HA 连通性验证
# ─────────────────────────────────────────────────────────────────────────────
echo "[6/6] Redis HA 连通性验证..."
HA_POD=$(kubectl -n "${NS}" get pod -l app.kubernetes.io/name=argocd-redis-ha \
-o jsonpath='{.items[0].metadata.name}' 2>/dev/null || echo "")
# fallback: 兼容不同版本 label
[[ -z "${HA_POD}" ]] && HA_POD=$(kubectl -n "${NS}" get pod \
--field-selector=status.phase=Running \
-o name 2>/dev/null | grep "argocd-redis-ha-server" | head -1 | sed 's|pod/||' || echo "")
if [[ -n "${HA_POD}" ]]; then
PING=$(kubectl -n "${NS}" exec "${HA_POD}" -c redis -- \
redis-cli -h argocd-redis-ha-haproxy -p 6379 ping 2>/dev/null || echo "FAIL")
if [[ "${PING}" == "PONG" ]]; then
echo " ✅ Redis HA HAProxy 连通 [${HA_POD} → argocd-redis-ha-haproxy:6379]: PONG"
else
echo " ⚠️ Redis HA HAProxy ping 失败，请检查:"
echo " kubectl -n ${NS} get pods -l app=argocd-redis-ha-haproxy"
echo " kubectl -n ${NS} logs -l app=argocd-redis-ha-haproxy --tail=50"
echo " kubectl -n ${NS} logs ${HA_POD} -c redis --tail=50"
fi
else
echo " ⚠️ 未找到 Redis HA Pod（label: app=argocd-redis-ha），请手动验证"
fi
echo ""
# ─────────────────────────────────────────────────────────────────────────────
# 最终验证
# ─────────────────────────────────────────────────────────────────────────────
echo "====================================================================="
echo " 最终验证"
echo "====================================================================="
echo ""
echo "--- Pod 状态 ---"
kubectl -n "${NS}" get pods -o wide
echo ""
echo "--- argocd-cmd-params-cm 关键字段 ---"
echo -n " redis.server: "; kubectl -n "${NS}" get configmap argocd-cmd-params-cm -o jsonpath='{.data.redis\.server}'; echo ""
echo -n " server.insecure: "; kubectl -n "${NS}" get configmap argocd-cmd-params-cm -o jsonpath='{.data.server\.insecure}'; echo ""
echo ""
echo "--- replicas ---"
for deploy in argocd-server argocd-repo-server; do
echo -n " ${deploy}: "
kubectl -n "${NS}" get deployment "${deploy}" -o jsonpath='{.spec.replicas}' 2>/dev/null; echo ""
done
echo ""
echo "====================================================================="
echo " 修复完成 ✅ (argocd 生产环境 --- 第一跳)"
echo ""
echo " [验证重点]"
echo " ✅ install.yaml apply 后 redis.server 是否被重置 → 脚本已恢复"
echo " ✅ Redis HA HAProxy 连通性"
echo " ✅ server.insecure 已恢复"
echo " ✅ replicas=2 已强制恢复"
echo ""
echo " 下一步: 第二跳升级"
echo " kubectl apply -f argocd-install-v32-ha-prod.yaml -n ${NS}"
echo " ./argocd-upgrade-post-v32-prod.sh"
echo ""
echo " [回滚操作] 如升级后出现 CrashLoopBackOff > 5min 或 Degraded App > 30%："
echo " # 1. 回滚镜像："
echo " kubectl apply -f argocd-install-v3.0.6-ha-prod.yaml -n ${NS}"
echo " # 2. 强制恢复 CM（install.yaml apply 会重置 argocd-cmd-params-cm，必须立即 patch）："
echo " kubectl -n ${NS} patch configmap argocd-cmd-params-cm --type merge \\"
echo " -p '{\"data\":{\"redis.server\":\"argocd-redis-ha-haproxy:6379\",\"server.insecure\":\"true\"}}'"
echo " # 3. 恢复 replicas=2："
echo " kubectl -n ${NS} scale deployment argocd-server argocd-repo-server --replicas=2"
echo " # 4. 重启使 CM 生效："
echo " kubectl -n ${NS} rollout restart deployment/argocd-server deployment/argocd-repo-server"
echo " kubectl -n ${NS} rollout restart statefulset/argocd-application-controller"
echo " kubectl -n ${NS} rollout status deployment/argocd-server --timeout=180s"
echo " # 5. 若 argocd-manager RBAC 丢失（目标集群执行）："
echo " # kubectl config use-context <target-cluster-context>"
echo " # ./argocd-test/argocd-fix-manager-rbac-test.sh"
echo " # 6. 若 CR/ConfigMap 数据损坏，从备份完整恢复："
echo " # ./argocd-test/argocd-restore-test.sh argocd-backup-<YYYYMMDD-HHMMSS>/"
echo "====================================================================="argocd-upgrade-post-v31-prod.sh
#!/usr/bin/env bash
# argocd-upgrade-post-v32-prod.sh
# ArgoCD v3.1.x → v3.2.x 升级后修复脚本（argocd 生产环境，第二跳）
#
# 生产集群说明：
# - NS="argocd"（生产命名空间）
# - replicas=2（升级后强制恢复，install.yaml 覆盖时可能降为 1）
# - Redis HA：argocd-redis-ha-haproxy:6379
# - argocd-manager ClusterRole 受保护（kubectl apply 不删除，此处仅校验现状）
#
# 使用方法：
# 1. kubectl apply -f argocd-install-v32-ha-prod.yaml -n argocd
# 2. 等待所有 Pod Running
# 3. 立即执行本脚本
set -euo pipefail
NS="argocd"
REDIS_HA_ADDR="argocd-redis-ha-haproxy:6379"
echo "====================================================================="
echo " ArgoCD v3.1.x → v3.2.x 升级后修复（argocd 生产环境 --- 第二跳）"
echo " 命名空间: ${NS}"
echo " 时间: $(date '+%Y-%m-%d %H:%M:%S')"
echo "====================================================================="
echo ""
# ─────────────────────────────────────────────────────────────────────────────
# 工具函数
# ─────────────────────────────────────────────────────────────────────────────
has_node_taint() {
kubectl get nodes -o jsonpath='{.items[*].spec.taints[*].key}' 2>/dev/null \
| tr ' ' '\n' | grep -q "^${1}$"
}
check_pods_healthy() {
local bad
# 排除 Terminating：rolling upgrade 过渡期旧 Pod 正常处于此状态
bad=$(kubectl -n "${NS}" get pods --no-headers 2>/dev/null \
| grep -v "Running\|Completed\|Terminating" | grep -c "." || true)
if [[ "${bad}" -gt 0 ]]; then
echo " ⚠️ 有 ${bad} 个 Pod 未处于 Running 状态："
kubectl -n "${NS}" get pods --no-headers | grep -v "Running\|Completed\|Terminating" || true
echo ""
echo " ⛔ 回滚触发条件：CrashLoopBackOff > 5min / Degraded App > 30%"
echo " 回滚命令：kubectl apply -f argocd-install-v31-ha-prod.yaml -n ${NS}"
return 1
fi
}
# ─────────────────────────────────────────────────────────────────────────────
# Step 0: 前置校验
# ─────────────────────────────────────────────────────────────────────────────
echo "[0/6] 前置校验..."
CURRENT_IMG=$(kubectl -n "${NS}" get deployment argocd-server \
-o jsonpath='{.spec.template.spec.containers[?(@.name=="argocd-server")].image}' \
2>/dev/null || echo "unknown")
echo " 当前 argocd-server 镜像: ${CURRENT_IMG}"
if echo "${CURRENT_IMG}" | grep -qE "v3\.[01]\."; then
echo " ⚠️ 镜像仍是 v3.0.x/v3.1.x，apply 未完成，请等待 rollout 后重试"
exit 1
fi
# 前置 Pod 状态仅作诊断快照，不阻断脚本（redis config 尚未恢复，Pod 可能 CrashLoop）
check_pods_healthy || echo " ⚠️ 存在异常 Pod，将继续尝试恢复配置..."
# argocd-manager ClusterRole 保护校验（kubectl apply 不会删除此资源，此处仅记录现状）
echo " [保护校验] argocd-manager ClusterRole..."
if kubectl get clusterrole argocd-manager-role &>/dev/null; then
echo " ✅ ClusterRole argocd-manager-role 存在，apply 不影响"
else
echo " ⚠️ ClusterRole argocd-manager-role 不存在！请在目标集群检查："
echo " kubectl config use-context <target-cluster-context>"
echo " ./argocd-test/argocd-fix-manager-rbac-test.sh"
fi
# 确认 Redis HA HAProxy 服务存在
if ! kubectl -n "${NS}" get service argocd-redis-ha-haproxy &>/dev/null; then
echo " ⛔ argocd-redis-ha-haproxy Service 不存在！Redis HA 异常，中止修复"
echo " 请先确认 Redis HA 状态: kubectl -n ${NS} get pods -l app=argocd-redis-ha"
exit 1
fi
echo " ✅ 前置校验通过"
echo ""
# ─────────────────────────────────────────────────────────────────────────────
# Steps 1+2: 原子恢复 redis.server + server.insecure
# ─────────────────────────────────────────────────────────────────────────────
echo "[1-2/6] 检查并原子恢复 redis.server + server.insecure..."
CURRENT_REDIS=$(kubectl -n "${NS}" get configmap argocd-cmd-params-cm \
-o jsonpath='{.data.redis\.server}' 2>/dev/null || echo "")
INSECURE_VAL=$(kubectl -n "${NS}" get configmap argocd-cmd-params-cm \
-o jsonpath='{.data.server\.insecure}' 2>/dev/null || echo "")
CONFIG_CHANGED=0
if [[ "${CURRENT_REDIS}" != "${REDIS_HA_ADDR}" || "${INSECURE_VAL}" != "true" ]]; then
echo " 当前: redis.server=[${CURRENT_REDIS}] server.insecure=[${INSECURE_VAL}]"
kubectl -n "${NS}" patch configmap argocd-cmd-params-cm \
--type merge \
-p "{\"data\":{\"redis.server\":\"${REDIS_HA_ADDR}\",\"server.insecure\":\"true\"}}"
echo " ✅ 已原子写入: redis.server=${REDIS_HA_ADDR} server.insecure=true"
CONFIG_CHANGED=1
else
echo " ✅ redis.server + server.insecure 均已正确，无需修改"
fi
echo ""
# ─────────────────────────────────────────────────────────────────────────────
# Step 3: 生产环境强制恢复 replicas=2
# ─────────────────────────────────────────────────────────────────────────────
echo "[3/6] 检查并恢复 replicas=2（生产必须为 2）..."
REPLICAS_CHANGED=0
for deploy in argocd-server argocd-repo-server; do
cur=$(kubectl -n "${NS}" get deployment "${deploy}" \
-o jsonpath='{.spec.replicas}' 2>/dev/null || echo "0")
if [[ "${cur}" -lt 2 ]]; then
kubectl -n "${NS}" scale deployment "${deploy}" --replicas=2
echo " ✅ ${deploy}: replicas ${cur} → 2"
REPLICAS_CHANGED=1
else
echo " ✅ ${deploy}: replicas=${cur}（已满足 ≥ 2）"
fi
done
echo ""
# ─────────────────────────────────────────────────────────────────────────────
# Step 4: tolerations
# ─────────────────────────────────────────────────────────────────────────────
echo "[4/6] 检查 tolerations..."
if has_node_taint "sretool"; then
echo " 检测到节点存在 sretool:NoSchedule taint，恢复 tolerations..."
TOLERATION_PATCH='{"spec":{"template":{"spec":{"tolerations":[{"key":"sretool","operator":"Equal","value":"true","effect":"NoSchedule"}]}}}}'
kubectl -n "${NS}" patch deployment argocd-server -p "${TOLERATION_PATCH}"
kubectl -n "${NS}" patch deployment argocd-repo-server -p "${TOLERATION_PATCH}"
echo " ✅ tolerations 已恢复"
else
echo " ✅ 节点无 sretool taint，跳过"
fi
echo ""
# ─────────────────────────────────────────────────────────────────────────────
# Step 5: 重启所有组件（v3.2.x 额外检查 applicationset-controller RBAC）
# ─────────────────────────────────────────────────────────────────────────────
echo "[5/6] 重启组件使配置生效..."
if [[ "${CONFIG_CHANGED}" -eq 0 && "${REPLICAS_CHANGED}" -eq 0 ]]; then
echo " ✅ CM 配置和 replicas 均已正确，跳过重启节省时间"
echo " 如需强制重启: kubectl -n ${NS} rollout restart deployment/argocd-server deployment/argocd-repo-server"
else
kubectl -n "${NS}" rollout restart deployment/argocd-server
kubectl -n "${NS}" rollout restart deployment/argocd-repo-server
kubectl -n "${NS}" rollout restart deployment/argocd-notifications-controller
kubectl -n "${NS}" rollout restart statefulset/argocd-application-controller
# v3.2.x 额外：applicationset-controller RBAC 变更（新增 coordination.k8s.io/leases）
ROLLOUT_PIDS=(); ROLLOUT_NAMES=()
kubectl -n "${NS}" rollout status deployment/argocd-server --timeout=180s &
ROLLOUT_PIDS+=($!); ROLLOUT_NAMES+=("argocd-server")
kubectl -n "${NS}" rollout status deployment/argocd-repo-server --timeout=180s &
ROLLOUT_PIDS+=($!); ROLLOUT_NAMES+=("argocd-repo-server")
kubectl -n "${NS}" rollout status deployment/argocd-notifications-controller --timeout=180s &
ROLLOUT_PIDS+=($!); ROLLOUT_NAMES+=("argocd-notifications-controller")
kubectl -n "${NS}" rollout status statefulset/argocd-application-controller --timeout=300s &
ROLLOUT_PIDS+=($!); ROLLOUT_NAMES+=("argocd-application-controller")
# v3.2 中 applicationset-controller 可能被重命名，单独追踪，失败不阻断必要组件
(kubectl -n "${NS}" rollout status deployment/argocd-applicationset-controller --timeout=180s || true) &
APPSET_PID=$!
ROLLOUT_FAIL=0
for i in "${!ROLLOUT_PIDS[@]}"; do
if ! wait "${ROLLOUT_PIDS[$i]}"; then
echo " ❌ ${ROLLOUT_NAMES[$i]} rollout 超时/失败"
ROLLOUT_FAIL=$((ROLLOUT_FAIL + 1))
fi
done
wait "${APPSET_PID}" || true # applicationset-controller 可选，失败不阻断
[[ "${ROLLOUT_FAIL}" -eq 0 ]] || { echo " ⛔ ${ROLLOUT_FAIL} 个必要组件 rollout 失败"; exit 1; }
fi
# v3.2 特有：检查 applicationset-controller RBAC 错误
APPSET_FORBIDDEN=$(kubectl -n "${NS}" logs \
-l app.kubernetes.io/name=argocd-applicationset-controller \
--tail=200 --since=30m 2>/dev/null | grep -c "Forbidden\|RBAC\|cannot" || true)
if [[ "${APPSET_FORBIDDEN}" -gt 0 ]]; then
echo " ⚠️ applicationset-controller 发现 ${APPSET_FORBIDDEN} 条 RBAC/Forbidden 日志："
kubectl -n "${NS}" logs \
-l app.kubernetes.io/name=argocd-applicationset-controller \
--tail=200 --since=30m | grep "Forbidden\|RBAC\|cannot"
else
echo " ✅ applicationset-controller 无 RBAC 错误"
fi
check_pods_healthy
echo " ✅ 所有组件重启完成，连接 Redis HA: ${REDIS_HA_ADDR}"
echo ""
# ─────────────────────────────────────────────────────────────────────────────
# Step 6: Redis HA 连通性验证
# ─────────────────────────────────────────────────────────────────────────────
echo "[6/6] Redis HA 连通性验证..."
HA_POD=$(kubectl -n "${NS}" get pod -l app.kubernetes.io/name=argocd-redis-ha \
-o jsonpath='{.items[0].metadata.name}' 2>/dev/null || echo "")
# fallback: 兼容不同版本 label
[[ -z "${HA_POD}" ]] && HA_POD=$(kubectl -n "${NS}" get pod \
--field-selector=status.phase=Running \
-o name 2>/dev/null | grep "argocd-redis-ha-server" | head -1 | sed 's|pod/||' || echo "")
if [[ -n "${HA_POD}" ]]; then
PING=$(kubectl -n "${NS}" exec "${HA_POD}" -c redis -- \
redis-cli -h argocd-redis-ha-haproxy -p 6379 ping 2>/dev/null || echo "FAIL")
if [[ "${PING}" == "PONG" ]]; then
echo " ✅ Redis HA HAProxy 连通 [${HA_POD} → argocd-redis-ha-haproxy:6379]: PONG"
else
echo " ⚠️ Redis HA HAProxy ping 失败，请检查:"
echo " kubectl -n ${NS} get pods -l app=argocd-redis-ha-haproxy"
echo " kubectl -n ${NS} logs -l app=argocd-redis-ha-haproxy --tail=50"
echo " kubectl -n ${NS} logs ${HA_POD} -c redis --tail=50"
fi
else
echo " ⚠️ 未找到 Redis HA Pod（label: app=argocd-redis-ha），请手动验证"
fi
echo ""
# ─────────────────────────────────────────────────────────────────────────────
# 最终验证
# ─────────────────────────────────────────────────────────────────────────────
echo "====================================================================="
echo " 最终验证"
echo "====================================================================="
echo ""
echo "--- Pod 状态 ---"
kubectl -n "${NS}" get pods -o wide
echo ""
echo "--- argocd-cmd-params-cm 关键字段 ---"
echo -n " redis.server: "; kubectl -n "${NS}" get configmap argocd-cmd-params-cm -o jsonpath='{.data.redis\.server}'; echo ""
echo -n " server.insecure: "; kubectl -n "${NS}" get configmap argocd-cmd-params-cm -o jsonpath='{.data.server\.insecure}'; echo ""
echo ""
echo "--- replicas ---"
for deploy in argocd-server argocd-repo-server; do
echo -n " ${deploy}: "
kubectl -n "${NS}" get deployment "${deploy}" -o jsonpath='{.spec.replicas}' 2>/dev/null; echo ""
done
echo ""
echo "====================================================================="
echo " 修复完成 ✅ (argocd 生产环境 --- 第二跳)"
echo ""
echo " [v3.2.x 行为变更验证要点]"
echo " 1. applicationset-controller RBAC: coordination.k8s.io/leases 权限"
echo " 检查: kubectl get clusterrole argocd-applicationset-controller -o yaml | grep coordination"
echo " 2. install.yaml apply 后 redis.server 被重置 → 脚本已恢复"
echo " 3. argocd-cmd-params-cm 中 server.insecure 被重置 → 脚本已恢复"
echo " ✅ replicas=2 已强制恢复"
echo ""
echo " [回滚操作] 如升级后出现 CrashLoopBackOff > 5min 或 Degraded App > 30%："
echo " # 1. 回滚至 v3.1.x："
echo " kubectl apply -f argocd-install-v31-ha-prod.yaml -n ${NS}"
echo " # 2. 强制恢复 CM（install.yaml apply 会重置 argocd-cmd-params-cm，必须立即 patch）："
echo " kubectl -n ${NS} patch configmap argocd-cmd-params-cm --type merge \\"
echo " -p '{\"data\":{\"redis.server\":\"argocd-redis-ha-haproxy:6379\",\"server.insecure\":\"true\"}}'"
echo " # 3. 恢复 replicas=2："
echo " kubectl -n ${NS} scale deployment argocd-server argocd-repo-server --replicas=2"
echo " # 4. 重启使 CM 生效："
echo " kubectl -n ${NS} rollout restart deployment/argocd-server deployment/argocd-repo-server"
echo " kubectl -n ${NS} rollout restart statefulset/argocd-application-controller"
echo " kubectl -n ${NS} rollout status deployment/argocd-server --timeout=180s"
echo " # 5. 若需完整回滚至 v3.0.6（重复步骤 1-4，使用 v3.0.6 yaml）："
echo " # kubectl apply -f argocd-install-v3.0.6-ha-prod.yaml -n ${NS}"
echo " # kubectl -n ${NS} patch configmap argocd-cmd-params-cm --type merge \\"
echo " # -p '{\"data\":{\"redis.server\":\"argocd-redis-ha-haproxy:6379\",\"server.insecure\":\"true\"}}'"
echo " # kubectl -n ${NS} scale deployment argocd-server argocd-repo-server --replicas=2"
echo " # kubectl -n ${NS} rollout restart deployment/argocd-server deployment/argocd-repo-server"
echo " # kubectl -n ${NS} rollout restart statefulset/argocd-application-controller"
echo " # 6. 若 argocd-manager RBAC 丢失（目标集群执行）："
echo " # kubectl config use-context <target-cluster-context>"
echo " # ./argocd-test/argocd-fix-manager-rbac-test.sh"
echo " # 7. 若 CR/ConfigMap 数据损坏，从备份完整恢复："
echo " # ./argocd-test/argocd-restore-test.sh argocd-backup-<YYYYMMDD-HHMMSS>/"
echo "====================================================================="