RuoYi Cloud:Gateway、Auth、System 接入 Prometheus JVM 监控,并练习多副本、滚动升级和故障恢复
本文基于前面已经部署完成的环境继续操作,不重新安装 MySQL、Redis、Nacos、UI 和监控组件。
本版已修正:
- 删除有问题的"批量
scp + ssh + sudo ctr import"命令,改成逐台 Worker 发送、逐台导入、逐台验证。- 删除会报
RuntimeError: 未正确修改 auth.replicas的跨区块正则脚本,改成按 YAML 顶层服务区块逐行修改并自动校验。- JAR 文件采用自动查找,避免固定文件名与实际构建结果不一致。
- ServiceMonitor 标签不再写死,自动读取当前 Prometheus 的选择器标签。
一、当前环境约定
text
命名空间:ruoyi
监控命名空间:monitoring
源码:~/ruoyi-k8s-lab/source/RuoYi-Cloud
Helm Chart:~/ruoyi-k8s-lab/charts/ruoyi-app
镜像构建目录:~/ruoyi-k8s-lab/images
镜像导出目录:~/ruoyi-k8s-lab/artifacts
Worker01:192.168.89.104
Worker02:192.168.89.105
Worker03:192.168.89.106
本阶段只修改:
text
ruoyi-gateway
ruoyi-auth
ruoyi-system
不会重新安装 MySQL、Redis、Nacos、UI、Prometheus、Grafana 和 Alertmanager。
二、部署前检查
在 k8s-master01 执行:
bash
cd "$HOME/ruoyi-k8s-lab"
echo '===== Helm Release ====='
helm list -n ruoyi
echo
echo '===== Deployment ====='
kubectl get deployment -n ruoyi
echo
echo '===== Pod ====='
kubectl get pods -n ruoyi -o wide
echo
echo '===== ServiceMonitor CRD ====='
kubectl get crd servicemonitors.monitoring.coreos.com
echo
echo '===== Prometheus ====='
kubectl get prometheus -n monitoring
必须确认:
text
ruoyi-mysql、ruoyi-redis、ruoyi-nacos、ruoyi-app 均为 deployed
Gateway、Auth、System、UI Pod 均为 Running
ServiceMonitor CRD 存在
Prometheus 资源存在
三、备份当前 Helm Chart
bash
cd "$HOME/ruoyi-k8s-lab"
BACKUP_TIME="$(date +%Y%m%d-%H%M%S)"
cp -a charts/ruoyi-app \
"charts/ruoyi-app.before-monitoring-${BACKUP_TIME}"
echo "备份完成:charts/ruoyi-app.before-monitoring-${BACKUP_TIME}"
四、确认三个服务已有 Actuator
bash
cd "$HOME/ruoyi-k8s-lab/source/RuoYi-Cloud"
grep -nH \
'spring-boot-starter-actuator' \
ruoyi-gateway/pom.xml \
ruoyi-auth/pom.xml \
ruoyi-modules/ruoyi-system/pom.xml
三个 POM 都应有输出。
检查 Prometheus Registry 是否已经存在:
bash
grep -nH \
'micrometer-registry-prometheus' \
ruoyi-gateway/pom.xml \
ruoyi-auth/pom.xml \
ruoyi-modules/ruoyi-system/pom.xml \
|| true
五、添加 Prometheus Registry
bash
cd "$HOME/ruoyi-k8s-lab/source/RuoYi-Cloud"
python3 <<'PY'
from pathlib import Path
pom_files = [
Path("ruoyi-gateway/pom.xml"),
Path("ruoyi-auth/pom.xml"),
Path("ruoyi-modules/ruoyi-system/pom.xml"),
]
dependency = """
<!-- Micrometer Prometheus 指标注册器 -->
<dependency>
<groupId>io.micrometer</groupId>
<artifactId>micrometer-registry-prometheus</artifactId>
<scope>runtime</scope>
</dependency>
"""
for pom in pom_files:
text = pom.read_text(encoding="utf-8")
if "micrometer-registry-prometheus" in text:
print(f"已存在,跳过:{pom}")
continue
marker = " </dependencies>"
if marker not in text:
raise RuntimeError(f"未找到 dependencies 结束标签:{pom}")
text = text.replace(marker, dependency + "\n" + marker, 1)
pom.write_text(text, encoding="utf-8")
print(f"添加成功:{pom}")
PY
检查:
bash
grep -n -A 5 -B 2 \
'micrometer-registry-prometheus' \
ruoyi-gateway/pom.xml \
ruoyi-auth/pom.xml \
ruoyi-modules/ruoyi-system/pom.xml
六、重新编译三个服务
bash
cd "$HOME/ruoyi-k8s-lab/source/RuoYi-Cloud"
export MAVEN_OPTS="-Xms128m -Xmx768m -XX:+UseG1GC"
mvn \
-DskipTests \
-T 1 \
clean package \
-pl ruoyi-gateway,ruoyi-auth,ruoyi-modules/ruoyi-system \
-am
成功时最后应出现:
text
BUILD SUCCESS
七、自动定位 JAR 并更新构建目录
bash
cd "$HOME/ruoyi-k8s-lab/source/RuoYi-Cloud"
GATEWAY_JAR="$(
find ruoyi-gateway/target \
-maxdepth 1 -type f \
-name 'ruoyi-gateway*.jar' \
! -name '*.original' \
| sort | head -n 1
)"
AUTH_JAR="$(
find ruoyi-auth/target \
-maxdepth 1 -type f \
-name 'ruoyi-auth*.jar' \
! -name '*.original' \
| sort | head -n 1
)"
SYSTEM_JAR="$(
find ruoyi-modules/ruoyi-system/target \
-maxdepth 1 -type f \
-name 'ruoyi-modules-system*.jar' \
! -name '*.original' \
| sort | head -n 1
)"
for jar in "$GATEWAY_JAR" "$AUTH_JAR" "$SYSTEM_JAR"; do
if [ -z "$jar" ] || [ ! -s "$jar" ]; then
echo "JAR 不存在或为空:$jar"
exit 1
fi
done
echo "Gateway:$GATEWAY_JAR"
echo "Auth: $AUTH_JAR"
echo "System: $SYSTEM_JAR"
复制:
bash
mkdir -p \
"$HOME/ruoyi-k8s-lab/images/gateway" \
"$HOME/ruoyi-k8s-lab/images/auth" \
"$HOME/ruoyi-k8s-lab/images/system" \
"$HOME/ruoyi-k8s-lab/artifacts"
install -m 0644 "$GATEWAY_JAR" \
"$HOME/ruoyi-k8s-lab/images/gateway/app.jar"
install -m 0644 "$AUTH_JAR" \
"$HOME/ruoyi-k8s-lab/images/auth/app.jar"
install -m 0644 "$SYSTEM_JAR" \
"$HOME/ruoyi-k8s-lab/images/system/app.jar"
ls -lh \
"$HOME/ruoyi-k8s-lab/images/gateway/app.jar" \
"$HOME/ruoyi-k8s-lab/images/auth/app.jar" \
"$HOME/ruoyi-k8s-lab/images/system/app.jar"
八、构建 monitoring-v1 镜像
bash
export IMAGE_TAG="3.6.8-monitoring-v1"
bash
sudo buildah bud \
--format docker --layers \
-t "localhost/ruoyi-gateway:${IMAGE_TAG}" \
"$HOME/ruoyi-k8s-lab/images/gateway"
bash
sudo buildah bud \
--format docker --layers \
-t "localhost/ruoyi-auth:${IMAGE_TAG}" \
"$HOME/ruoyi-k8s-lab/images/auth"
bash
sudo buildah bud \
--format docker --layers \
-t "localhost/ruoyi-system:${IMAGE_TAG}" \
"$HOME/ruoyi-k8s-lab/images/system"
检查:
bash
sudo buildah images \
| grep -E 'ruoyi-(gateway|auth|system)' \
| grep "$IMAGE_TAG"
九、导出 monitoring-v1 镜像
bash
export IMAGE_TAG="3.6.8-monitoring-v1"
export ARTIFACT_DIR="$HOME/ruoyi-k8s-lab/artifacts"
mkdir -p "$ARTIFACT_DIR"
for service in gateway auth system; do
rm -f "${ARTIFACT_DIR}/ruoyi-${service}-${IMAGE_TAG}.tar"
sudo buildah push \
"localhost/ruoyi-${service}:${IMAGE_TAG}" \
"docker-archive:${ARTIFACT_DIR}/ruoyi-${service}-${IMAGE_TAG}.tar:localhost/ruoyi-${service}:${IMAGE_TAG}"
done
sudo chown leecurry:leecurry \
"${ARTIFACT_DIR}/ruoyi-gateway-${IMAGE_TAG}.tar" \
"${ARTIFACT_DIR}/ruoyi-auth-${IMAGE_TAG}.tar" \
"${ARTIFACT_DIR}/ruoyi-system-${IMAGE_TAG}.tar"
ls -lh "${ARTIFACT_DIR}"/*"${IMAGE_TAG}".tar
十、修正版:逐台发送并导入 Worker
不再使用原来的批量远程导入命令。必须逐台发送、逐台导入、逐台验证。
10.1 master01 检查文件
bash
export IMAGE_TAG="3.6.8-monitoring-v1"
export ARTIFACT_DIR="$HOME/ruoyi-k8s-lab/artifacts"
for file in \
"${ARTIFACT_DIR}/ruoyi-gateway-${IMAGE_TAG}.tar" \
"${ARTIFACT_DIR}/ruoyi-auth-${IMAGE_TAG}.tar" \
"${ARTIFACT_DIR}/ruoyi-system-${IMAGE_TAG}.tar"
do
[ -s "$file" ] || {
echo "镜像文件不存在或为空:$file"
exit 1
}
done
10.2 发送到 worker01
bash
scp \
"${ARTIFACT_DIR}/ruoyi-gateway-${IMAGE_TAG}.tar" \
"${ARTIFACT_DIR}/ruoyi-auth-${IMAGE_TAG}.tar" \
"${ARTIFACT_DIR}/ruoyi-system-${IMAGE_TAG}.tar" \
leecurry@192.168.89.104:/tmp/
10.3 发送到 worker02
bash
scp \
"${ARTIFACT_DIR}/ruoyi-gateway-${IMAGE_TAG}.tar" \
"${ARTIFACT_DIR}/ruoyi-auth-${IMAGE_TAG}.tar" \
"${ARTIFACT_DIR}/ruoyi-system-${IMAGE_TAG}.tar" \
leecurry@192.168.89.105:/tmp/
10.4 发送到 worker03
bash
scp \
"${ARTIFACT_DIR}/ruoyi-gateway-${IMAGE_TAG}.tar" \
"${ARTIFACT_DIR}/ruoyi-auth-${IMAGE_TAG}.tar" \
"${ARTIFACT_DIR}/ruoyi-system-${IMAGE_TAG}.tar" \
leecurry@192.168.89.106:/tmp/
10.5 在每台 Worker 分别执行导入
分别切换到 worker01、worker02、worker03 的 Xshell 标签页,每台都执行:
bash
export IMAGE_TAG="3.6.8-monitoring-v1"
for image_file in \
"/tmp/ruoyi-gateway-${IMAGE_TAG}.tar" \
"/tmp/ruoyi-auth-${IMAGE_TAG}.tar" \
"/tmp/ruoyi-system-${IMAGE_TAG}.tar"
do
[ -s "$image_file" ] || {
echo "文件不存在或为空:$image_file"
exit 1
}
echo "正在导入:$image_file"
sudo ctr -n k8s.io images import "$image_file" || exit 1
done
sudo crictl images \
| grep -E 'ruoyi-(gateway|auth|system)' \
| grep "$IMAGE_TAG"
每台必须看到三个镜像。三个 Worker 全部验证成功后再继续。
十一、修正版:修改副本数和镜像标签
bash
cd "$HOME/ruoyi-k8s-lab/charts/ruoyi-app"
cp values-lab.yaml \
"values-lab.yaml.before-monitoring-$(date +%Y%m%d-%H%M%S)"
执行可靠的逐行区块处理脚本:
bash
python3 <<'PY'
from pathlib import Path
path = Path("values-lab.yaml")
lines = path.read_text(encoding="utf-8").splitlines()
services = {"gateway", "auth", "system"}
new_tag = "3.6.8-monitoring-v1"
current_section = None
inside_image = False
replicas_changed = {name: False for name in services}
tags_changed = {name: False for name in services}
for index, line in enumerate(lines):
stripped = line.strip()
if line and not line.startswith((" ", "\t")) and stripped.endswith(":"):
current_section = stripped[:-1]
inside_image = False
continue
if current_section not in services:
continue
if line.startswith(" replicas:"):
lines[index] = " replicas: 2"
replicas_changed[current_section] = True
continue
if line.startswith(" image:"):
inside_image = True
continue
if inside_image and line.startswith(" ") and not line.startswith(" "):
inside_image = False
if inside_image and line.startswith(" tag:"):
lines[index] = f' tag: "{new_tag}"'
tags_changed[current_section] = True
missing_replicas = sorted(
name for name, changed in replicas_changed.items() if not changed
)
missing_tags = sorted(
name for name, changed in tags_changed.items() if not changed
)
if missing_replicas:
raise RuntimeError(
"以下服务未找到 replicas:" + ", ".join(missing_replicas)
)
if missing_tags:
raise RuntimeError(
"以下服务未找到 image.tag:" + ", ".join(missing_tags)
)
path.write_text("\n".join(lines) + "\n", encoding="utf-8")
print("values-lab.yaml 修改成功")
for service in ("gateway", "auth", "system"):
print(f"{service}: replicas=2, tag={new_tag}")
PY
强制校验:
bash
python3 <<'PY'
from pathlib import Path
lines = Path("values-lab.yaml").read_text(
encoding="utf-8"
).splitlines()
services = {"gateway", "auth", "system"}
expected_tag = "3.6.8-monitoring-v1"
current = None
inside_image = False
result = {
name: {"replicas": None, "tag": None}
for name in services
}
for line in lines:
stripped = line.strip()
if line and not line.startswith((" ", "\t")) and stripped.endswith(":"):
current = stripped[:-1]
inside_image = False
continue
if current not in services:
continue
if line.startswith(" replicas:"):
result[current]["replicas"] = stripped.split(":", 1)[1].strip()
continue
if line.startswith(" image:"):
inside_image = True
continue
if inside_image and line.startswith(" ") and not line.startswith(" "):
inside_image = False
if inside_image and line.startswith(" tag:"):
result[current]["tag"] = (
stripped.split(":", 1)[1].strip().strip('"').strip("'")
)
errors = []
for service in ("gateway", "auth", "system"):
if result[service]["replicas"] != "2":
errors.append(f"{service}.replicas={result[service]['replicas']}")
if result[service]["tag"] != expected_tag:
errors.append(f"{service}.image.tag={result[service]['tag']}")
if errors:
raise SystemExit("校验失败:\n- " + "\n- ".join(errors))
print("校验通过:")
for service in ("gateway", "auth", "system"):
print(f"- {service}: replicas=2, tag={expected_tag}")
PY
查看:
bash
grep -nE \
'^(gateway|auth|system|ui):|^ replicas:|^ image:|^ repository:|^ tag:' \
values-lab.yaml
Gateway/Auth/System 应为 replicas: 2 和 monitoring-v1;UI 保持原值。
十二、更新 Gateway Deployment
bash
cd "$HOME/ruoyi-k8s-lab/charts/ruoyi-app"
cat <<'EOF' > templates/gateway-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: {{ .Values.gateway.name }}
labels:
app.kubernetes.io/name: {{ .Values.gateway.name }}
app.kubernetes.io/instance: {{ .Release.Name }}
spec:
replicas: {{ .Values.gateway.replicas }}
revisionHistoryLimit: 5
strategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 1
maxSurge: 0
selector:
matchLabels:
app.kubernetes.io/name: {{ .Values.gateway.name }}
app.kubernetes.io/instance: {{ .Release.Name }}
template:
metadata:
labels:
app.kubernetes.io/name: {{ .Values.gateway.name }}
app.kubernetes.io/instance: {{ .Release.Name }}
spec:
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100
podAffinityTerm:
topologyKey: kubernetes.io/hostname
labelSelector:
matchLabels:
app.kubernetes.io/name: {{ .Values.gateway.name }}
app.kubernetes.io/instance: {{ .Release.Name }}
initContainers:
- name: wait-for-nacos
image: "{{ .Values.global.busyboxImage.repository }}:{{ .Values.global.busyboxImage.tag }}"
imagePullPolicy: IfNotPresent
command:
- sh
- -c
- |
until nc -z ruoyi-nacos 8848; do
echo "等待 Nacos"
sleep 3
done
- name: wait-for-redis
image: "{{ .Values.global.busyboxImage.repository }}:{{ .Values.global.busyboxImage.tag }}"
imagePullPolicy: IfNotPresent
command:
- sh
- -c
- |
until nc -z ruoyi-redis 6379; do
echo "等待 Redis"
sleep 3
done
containers:
- name: gateway
image: "{{ .Values.gateway.image.repository }}:{{ .Values.gateway.image.tag }}"
imagePullPolicy: {{ .Values.global.imagePullPolicy }}
env:
- name: JAVA_OPTS
value: {{ .Values.gateway.javaOpts | quote }}
- name: SPRING_PROFILES_ACTIVE
value: {{ .Values.global.profile | quote }}
- name: MANAGEMENT_ENDPOINTS_WEB_EXPOSURE_INCLUDE
value: "health,info,prometheus"
- name: MANAGEMENT_ENDPOINT_PROMETHEUS_ENABLED
value: "true"
- name: MANAGEMENT_PROMETHEUS_METRICS_EXPORT_ENABLED
value: "true"
- name: MANAGEMENT_METRICS_TAGS_APPLICATION
value: {{ .Values.gateway.name | quote }}
- name: SPRING_CLOUD_NACOS_SERVER_ADDR
value: {{ .Values.global.nacos.serverAddr | quote }}
- name: SPRING_CLOUD_NACOS_CONFIG_SERVER_ADDR
value: {{ .Values.global.nacos.serverAddr | quote }}
- name: SPRING_CLOUD_NACOS_DISCOVERY_SERVER_ADDR
value: {{ .Values.global.nacos.serverAddr | quote }}
- name: SPRING_CLOUD_NACOS_USERNAME
valueFrom:
secretKeyRef:
name: ruoyi-app-nacos
key: username
- name: SPRING_CLOUD_NACOS_PASSWORD
valueFrom:
secretKeyRef:
name: ruoyi-app-nacos
key: password
- name: SPRING_CLOUD_NACOS_CONFIG_USERNAME
valueFrom:
secretKeyRef:
name: ruoyi-app-nacos
key: username
- name: SPRING_CLOUD_NACOS_CONFIG_PASSWORD
valueFrom:
secretKeyRef:
name: ruoyi-app-nacos
key: password
- name: SPRING_CLOUD_NACOS_DISCOVERY_USERNAME
valueFrom:
secretKeyRef:
name: ruoyi-app-nacos
key: username
- name: SPRING_CLOUD_NACOS_DISCOVERY_PASSWORD
valueFrom:
secretKeyRef:
name: ruoyi-app-nacos
key: password
ports:
- name: http
containerPort: {{ .Values.gateway.port }}
startupProbe:
tcpSocket:
port: http
periodSeconds: 5
failureThreshold: 60
readinessProbe:
tcpSocket:
port: http
periodSeconds: 10
failureThreshold: 6
livenessProbe:
tcpSocket:
port: http
periodSeconds: 20
failureThreshold: 6
resources:
{{ toYaml .Values.gateway.resources | indent 12 }}
EOF
十三、更新 Auth Deployment
bash
cd "$HOME/ruoyi-k8s-lab/charts/ruoyi-app"
cat <<'EOF' > templates/auth-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: {{ .Values.auth.name }}
labels:
app.kubernetes.io/name: {{ .Values.auth.name }}
app.kubernetes.io/instance: {{ .Release.Name }}
spec:
replicas: {{ .Values.auth.replicas }}
revisionHistoryLimit: 5
strategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 1
maxSurge: 0
selector:
matchLabels:
app.kubernetes.io/name: {{ .Values.auth.name }}
app.kubernetes.io/instance: {{ .Release.Name }}
template:
metadata:
labels:
app.kubernetes.io/name: {{ .Values.auth.name }}
app.kubernetes.io/instance: {{ .Release.Name }}
spec:
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100
podAffinityTerm:
topologyKey: kubernetes.io/hostname
labelSelector:
matchLabels:
app.kubernetes.io/name: {{ .Values.auth.name }}
app.kubernetes.io/instance: {{ .Release.Name }}
initContainers:
- name: wait-for-nacos
image: "{{ .Values.global.busyboxImage.repository }}:{{ .Values.global.busyboxImage.tag }}"
imagePullPolicy: IfNotPresent
command:
- sh
- -c
- |
until nc -z ruoyi-nacos 8848; do
echo "等待 Nacos"
sleep 3
done
- name: wait-for-redis
image: "{{ .Values.global.busyboxImage.repository }}:{{ .Values.global.busyboxImage.tag }}"
imagePullPolicy: IfNotPresent
command:
- sh
- -c
- |
until nc -z ruoyi-redis 6379; do
echo "等待 Redis"
sleep 3
done
containers:
- name: auth
image: "{{ .Values.auth.image.repository }}:{{ .Values.auth.image.tag }}"
imagePullPolicy: {{ .Values.global.imagePullPolicy }}
env:
- name: JAVA_OPTS
value: {{ .Values.auth.javaOpts | quote }}
- name: SPRING_PROFILES_ACTIVE
value: {{ .Values.global.profile | quote }}
- name: MANAGEMENT_ENDPOINTS_WEB_EXPOSURE_INCLUDE
value: "health,info,prometheus"
- name: MANAGEMENT_ENDPOINT_PROMETHEUS_ENABLED
value: "true"
- name: MANAGEMENT_PROMETHEUS_METRICS_EXPORT_ENABLED
value: "true"
- name: MANAGEMENT_METRICS_TAGS_APPLICATION
value: {{ .Values.auth.name | quote }}
- name: SPRING_CLOUD_NACOS_SERVER_ADDR
value: {{ .Values.global.nacos.serverAddr | quote }}
- name: SPRING_CLOUD_NACOS_CONFIG_SERVER_ADDR
value: {{ .Values.global.nacos.serverAddr | quote }}
- name: SPRING_CLOUD_NACOS_DISCOVERY_SERVER_ADDR
value: {{ .Values.global.nacos.serverAddr | quote }}
- name: SPRING_CLOUD_NACOS_USERNAME
valueFrom:
secretKeyRef:
name: ruoyi-app-nacos
key: username
- name: SPRING_CLOUD_NACOS_PASSWORD
valueFrom:
secretKeyRef:
name: ruoyi-app-nacos
key: password
- name: SPRING_CLOUD_NACOS_CONFIG_USERNAME
valueFrom:
secretKeyRef:
name: ruoyi-app-nacos
key: username
- name: SPRING_CLOUD_NACOS_CONFIG_PASSWORD
valueFrom:
secretKeyRef:
name: ruoyi-app-nacos
key: password
- name: SPRING_CLOUD_NACOS_DISCOVERY_USERNAME
valueFrom:
secretKeyRef:
name: ruoyi-app-nacos
key: username
- name: SPRING_CLOUD_NACOS_DISCOVERY_PASSWORD
valueFrom:
secretKeyRef:
name: ruoyi-app-nacos
key: password
ports:
- name: http
containerPort: {{ .Values.auth.port }}
startupProbe:
tcpSocket:
port: http
periodSeconds: 5
failureThreshold: 60
readinessProbe:
tcpSocket:
port: http
periodSeconds: 10
failureThreshold: 6
livenessProbe:
tcpSocket:
port: http
periodSeconds: 20
failureThreshold: 6
resources:
{{ toYaml .Values.auth.resources | indent 12 }}
EOF
十四、更新 System Deployment
bash
cd "$HOME/ruoyi-k8s-lab/charts/ruoyi-app"
cat <<'EOF' > templates/system-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: {{ .Values.system.name }}
labels:
app.kubernetes.io/name: {{ .Values.system.name }}
app.kubernetes.io/instance: {{ .Release.Name }}
spec:
replicas: {{ .Values.system.replicas }}
revisionHistoryLimit: 5
strategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 1
maxSurge: 0
selector:
matchLabels:
app.kubernetes.io/name: {{ .Values.system.name }}
app.kubernetes.io/instance: {{ .Release.Name }}
template:
metadata:
labels:
app.kubernetes.io/name: {{ .Values.system.name }}
app.kubernetes.io/instance: {{ .Release.Name }}
spec:
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100
podAffinityTerm:
topologyKey: kubernetes.io/hostname
labelSelector:
matchLabels:
app.kubernetes.io/name: {{ .Values.system.name }}
app.kubernetes.io/instance: {{ .Release.Name }}
initContainers:
- name: wait-for-nacos
image: "{{ .Values.global.busyboxImage.repository }}:{{ .Values.global.busyboxImage.tag }}"
imagePullPolicy: IfNotPresent
command:
- sh
- -c
- |
until nc -z ruoyi-nacos 8848; do
echo "等待 Nacos"
sleep 3
done
- name: wait-for-redis
image: "{{ .Values.global.busyboxImage.repository }}:{{ .Values.global.busyboxImage.tag }}"
imagePullPolicy: IfNotPresent
command:
- sh
- -c
- |
until nc -z ruoyi-redis 6379; do
echo "等待 Redis"
sleep 3
done
- name: wait-for-mysql
image: "{{ .Values.global.busyboxImage.repository }}:{{ .Values.global.busyboxImage.tag }}"
imagePullPolicy: IfNotPresent
command:
- sh
- -c
- |
until nc -z ruoyi-mysql 3306; do
echo "等待 Mysql"
sleep 3
done
containers:
- name: system
image: "{{ .Values.system.image.repository }}:{{ .Values.system.image.tag }}"
imagePullPolicy: {{ .Values.global.imagePullPolicy }}
env:
- name: JAVA_OPTS
value: {{ .Values.system.javaOpts | quote }}
- name: SPRING_PROFILES_ACTIVE
value: {{ .Values.global.profile | quote }}
- name: MANAGEMENT_ENDPOINTS_WEB_EXPOSURE_INCLUDE
value: "health,info,prometheus"
- name: MANAGEMENT_ENDPOINT_PROMETHEUS_ENABLED
value: "true"
- name: MANAGEMENT_PROMETHEUS_METRICS_EXPORT_ENABLED
value: "true"
- name: MANAGEMENT_METRICS_TAGS_APPLICATION
value: {{ .Values.system.name | quote }}
- name: SPRING_CLOUD_NACOS_SERVER_ADDR
value: {{ .Values.global.nacos.serverAddr | quote }}
- name: SPRING_CLOUD_NACOS_CONFIG_SERVER_ADDR
value: {{ .Values.global.nacos.serverAddr | quote }}
- name: SPRING_CLOUD_NACOS_DISCOVERY_SERVER_ADDR
value: {{ .Values.global.nacos.serverAddr | quote }}
- name: SPRING_CLOUD_NACOS_USERNAME
valueFrom:
secretKeyRef:
name: ruoyi-app-nacos
key: username
- name: SPRING_CLOUD_NACOS_PASSWORD
valueFrom:
secretKeyRef:
name: ruoyi-app-nacos
key: password
- name: SPRING_CLOUD_NACOS_CONFIG_USERNAME
valueFrom:
secretKeyRef:
name: ruoyi-app-nacos
key: username
- name: SPRING_CLOUD_NACOS_CONFIG_PASSWORD
valueFrom:
secretKeyRef:
name: ruoyi-app-nacos
key: password
- name: SPRING_CLOUD_NACOS_DISCOVERY_USERNAME
valueFrom:
secretKeyRef:
name: ruoyi-app-nacos
key: username
- name: SPRING_CLOUD_NACOS_DISCOVERY_PASSWORD
valueFrom:
secretKeyRef:
name: ruoyi-app-nacos
key: password
ports:
- name: http
containerPort: {{ .Values.system.port }}
startupProbe:
tcpSocket:
port: http
periodSeconds: 5
failureThreshold: 60
readinessProbe:
tcpSocket:
port: http
periodSeconds: 10
failureThreshold: 6
livenessProbe:
tcpSocket:
port: http
periodSeconds: 20
failureThreshold: 6
resources:
{{ toYaml .Values.system.resources | indent 12 }}
EOF
十五、给三个 Service 添加监控标签
bash
cd "$HOME/ruoyi-k8s-lab/charts/ruoyi-app"
cat <<'EOF' > templates/services.yaml
apiVersion: v1
kind: Service
metadata:
name: {{ .Values.gateway.name }}
labels:
app.kubernetes.io/name: {{ .Values.gateway.name }}
app.kubernetes.io/instance: {{ .Release.Name }}
monitoring: ruoyi-jvm
spec:
type: ClusterIP
selector:
app.kubernetes.io/name: {{ .Values.gateway.name }}
app.kubernetes.io/instance: {{ .Release.Name }}
ports:
- name: http
port: {{ .Values.gateway.port }}
targetPort: http
---
apiVersion: v1
kind: Service
metadata:
name: {{ .Values.auth.name }}
labels:
app.kubernetes.io/name: {{ .Values.auth.name }}
app.kubernetes.io/instance: {{ .Release.Name }}
monitoring: ruoyi-jvm
spec:
type: ClusterIP
selector:
app.kubernetes.io/name: {{ .Values.auth.name }}
app.kubernetes.io/instance: {{ .Release.Name }}
ports:
- name: http
port: {{ .Values.auth.port }}
targetPort: http
---
apiVersion: v1
kind: Service
metadata:
name: {{ .Values.system.name }}
labels:
app.kubernetes.io/name: {{ .Values.system.name }}
app.kubernetes.io/instance: {{ .Release.Name }}
monitoring: ruoyi-jvm
spec:
type: ClusterIP
selector:
app.kubernetes.io/name: {{ .Values.system.name }}
app.kubernetes.io/instance: {{ .Release.Name }}
ports:
- name: http
port: {{ .Values.system.port }}
targetPort: http
EOF
sed -i 's/^version:.*/version: 0.3.0/' Chart.yaml
十六、检查 Helm Chart
bash
cd "$HOME/ruoyi-k8s-lab"
helm lint \
./charts/ruoyi-app \
-f ./charts/ruoyi-app/values-lab.yaml
必须显示:
text
1 chart(s) linted, 0 chart(s) failed
渲染:
bash
helm template ruoyi-app \
./charts/ruoyi-app \
--namespace ruoyi \
-f ./charts/ruoyi-app/values-lab.yaml \
> /tmp/ruoyi-app-monitoring.yaml
检查:
bash
echo '===== 镜像 ====='
grep -n '3.6.8-monitoring-v1' /tmp/ruoyi-app-monitoring.yaml
echo
echo '===== 副本 ====='
grep -n 'replicas: 2' /tmp/ruoyi-app-monitoring.yaml
echo
echo '===== 监控环境变量 ====='
grep -nE \
'MANAGEMENT_ENDPOINTS_WEB_EXPOSURE_INCLUDE|MANAGEMENT_ENDPOINT_PROMETHEUS_ENABLED|MANAGEMENT_PROMETHEUS_METRICS_EXPORT_ENABLED' \
/tmp/ruoyi-app-monitoring.yaml
echo
echo '===== Service 标签 ====='
grep -n 'monitoring: ruoyi-jvm' /tmp/ruoyi-app-monitoring.yaml
echo
echo '===== 滚动策略与反亲和 ====='
grep -nE 'maxUnavailable|maxSurge|podAntiAffinity' \
/tmp/ruoyi-app-monitoring.yaml
预期:
text
monitoring-v1 镜像出现 3 次
replicas: 2 出现 3 次
monitoring: ruoyi-jvm 出现 3 次
三个 Deployment 都有 Prometheus 环境变量、滚动策略和反亲和
十七、升级 ruoyi-app
bash
cd "$HOME/ruoyi-k8s-lab"
helm upgrade --install ruoyi-app \
./charts/ruoyi-app \
--namespace ruoyi \
-f ./charts/ruoyi-app/values-lab.yaml \
--atomic \
--wait \
--timeout 20m
检查:
bash
helm list -n ruoyi
helm history ruoyi-app -n ruoyi
kubectl get deployment \
ruoyi-gateway ruoyi-auth ruoyi-system \
-n ruoyi
kubectl get pods \
-n ruoyi \
-l app.kubernetes.io/instance=ruoyi-app \
-o wide
三个 Deployment 都应为 2/2。
十八、验证 Prometheus 指标端点
bash
kubectl run ruoyi-metrics-check \
-n ruoyi \
--rm -i \
--restart=Never \
--image=m.daocloud.io/docker.io/library/busybox:1.36.1 \
-- sh -c '
set -e
for target in \
ruoyi-gateway:8080 \
ruoyi-auth:9200 \
ruoyi-system:9201
do
echo
echo "===== ${target} ====="
wget -qO- "http://${target}/actuator/prometheus" | head -n 12
done
'
每个服务都应输出 JVM 指标。
检查 application 标签:
bash
kubectl run ruoyi-metrics-tag-check \
-n ruoyi \
--rm -i \
--restart=Never \
--image=m.daocloud.io/docker.io/library/busybox:1.36.1 \
-- sh -c '
wget -qO- \
http://ruoyi-system:9201/actuator/prometheus \
| grep "application=\"ruoyi-system\"" \
| head
'
十九、创建 ServiceMonitor
bash
mkdir -p "$HOME/ruoyi-k8s-lab/monitoring"
cat <<'EOF' > \
"$HOME/ruoyi-k8s-lab/monitoring/ruoyi-servicemonitor.yaml"
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: ruoyi-jvm
namespace: monitoring
labels:
app.kubernetes.io/part-of: ruoyi
spec:
namespaceSelector:
matchNames:
- ruoyi
selector:
matchLabels:
monitoring: ruoyi-jvm
endpoints:
- port: http
path: /actuator/prometheus
interval: 15s
scrapeTimeout: 10s
scheme: http
EOF
kubectl apply -f \
"$HOME/ruoyi-k8s-lab/monitoring/ruoyi-servicemonitor.yaml"
自动匹配当前 Prometheus 的 ServiceMonitor 选择标签:
bash
PROMETHEUS_NAME="$(
kubectl get prometheus \
-n monitoring \
-o jsonpath='{.items[0].metadata.name}'
)"
if [ -z "$PROMETHEUS_NAME" ]; then
echo "没有找到 Prometheus 资源"
exit 1
fi
echo "Prometheus:$PROMETHEUS_NAME"
SELECTOR_LABELS="$(
kubectl get prometheus \
"$PROMETHEUS_NAME" \
-n monitoring \
-o go-template='{{range $key, $value := .spec.serviceMonitorSelector.matchLabels}}{{printf "%s=%s\n" $key $value}}{{end}}'
)"
if [ -n "$SELECTOR_LABELS" ]; then
printf '%s\n' "$SELECTOR_LABELS"
while IFS='=' read -r key value; do
[ -n "$key" ] || continue
kubectl label servicemonitor \
ruoyi-jvm \
-n monitoring \
"${key}=${value}" \
--overwrite
done <<< "$SELECTOR_LABELS"
else
echo "Prometheus 未配置 matchLabels,无需添加额外标签"
fi
检查:
bash
kubectl get servicemonitor \
ruoyi-jvm \
-n monitoring \
--show-labels
kubectl get svc \
-n ruoyi \
-l monitoring=ruoyi-jvm \
--show-labels
二十、检查 Prometheus Target
bash
sleep 30
浏览器打开:
text
http://192.168.89.104:30031/targets
搜索 ruoyi,应看到:
text
Gateway 2 个 UP
Auth 2 个 UP
System 2 个 UP
API 查询:
bash
curl -sG \
--data-urlencode \
'query=count by (service) (up{namespace="ruoyi",service=~"ruoyi-(gateway|auth|system)"} == 1)' \
http://192.168.89.104:30031/api/v1/query \
| python3 -m json.tool
二十一、Grafana 查询
打开:
text
http://192.168.89.104:30030
进入 Explore → Prometheus。
Target 存活
promql
up{namespace="ruoyi",service=~"ruoyi-(gateway|auth|system)"}
JVM 堆内存
promql
sum by (application) (
jvm_memory_used_bytes{namespace="ruoyi",area="heap"}
)
JVM 堆使用率
promql
100 *
sum by (application) (
jvm_memory_used_bytes{namespace="ruoyi",area="heap"}
)
/
sum by (application) (
jvm_memory_max_bytes{namespace="ruoyi",area="heap"}
)
JVM 活跃线程
promql
sum by (application) (
jvm_threads_live_threads{namespace="ruoyi"}
)
进程 CPU
promql
avg by (application) (
process_cpu_usage{namespace="ruoyi"}
) * 100
GC 暂停速率
promql
sum by (application) (
rate(jvm_gc_pause_seconds_count{namespace="ruoyi"}[5m])
)
HTTP 请求速率
promql
sum by (application) (
rate(http_server_requests_seconds_count{namespace="ruoyi"}[5m])
)
二十二、练习滚动升级到 monitoring-v2
22.1 创建 v2 标签并导出
bash
export OLD_TAG="3.6.8-monitoring-v1"
export NEW_TAG="3.6.8-monitoring-v2"
export ARTIFACT_DIR="$HOME/ruoyi-k8s-lab/artifacts"
for service in gateway auth system; do
sudo buildah tag \
"localhost/ruoyi-${service}:${OLD_TAG}" \
"localhost/ruoyi-${service}:${NEW_TAG}"
rm -f "${ARTIFACT_DIR}/ruoyi-${service}-${NEW_TAG}.tar"
sudo buildah push \
"localhost/ruoyi-${service}:${NEW_TAG}" \
"docker-archive:${ARTIFACT_DIR}/ruoyi-${service}-${NEW_TAG}.tar:localhost/ruoyi-${service}:${NEW_TAG}"
done
sudo chown leecurry:leecurry \
"${ARTIFACT_DIR}/ruoyi-gateway-${NEW_TAG}.tar" \
"${ARTIFACT_DIR}/ruoyi-auth-${NEW_TAG}.tar" \
"${ARTIFACT_DIR}/ruoyi-system-${NEW_TAG}.tar"
22.2 逐台发送
bash
scp "${ARTIFACT_DIR}"/*"${NEW_TAG}".tar \
leecurry@192.168.89.104:/tmp/
bash
scp "${ARTIFACT_DIR}"/*"${NEW_TAG}".tar \
leecurry@192.168.89.105:/tmp/
bash
scp "${ARTIFACT_DIR}"/*"${NEW_TAG}".tar \
leecurry@192.168.89.106:/tmp/
22.3 三台 Worker 分别导入
每台 Worker 都执行:
bash
export NEW_TAG="3.6.8-monitoring-v2"
for image_file in \
"/tmp/ruoyi-gateway-${NEW_TAG}.tar" \
"/tmp/ruoyi-auth-${NEW_TAG}.tar" \
"/tmp/ruoyi-system-${NEW_TAG}.tar"
do
[ -s "$image_file" ] || {
echo "文件不存在或为空:$image_file"
exit 1
}
sudo ctr -n k8s.io images import "$image_file" || exit 1
done
sudo crictl images \
| grep -E 'ruoyi-(gateway|auth|system)' \
| grep "$NEW_TAG"
22.4 修改 Helm 标签并升级
bash
cd "$HOME/ruoyi-k8s-lab/charts/ruoyi-app"
sed -i \
's/3\.6\.8-monitoring-v1/3.6.8-monitoring-v2/g' \
values-lab.yaml
grep -n '3.6.8-monitoring-v2' values-lab.yaml
另开窗口观察:
bash
watch -n 1 \
kubectl get pods \
-n ruoyi \
-l app.kubernetes.io/instance=ruoyi-app \
-o wide
原窗口升级:
bash
cd "$HOME/ruoyi-k8s-lab"
helm upgrade ruoyi-app \
./charts/ruoyi-app \
--namespace ruoyi \
-f ./charts/ruoyi-app/values-lab.yaml \
--atomic \
--wait \
--timeout 20m
检查:
bash
kubectl rollout status deployment/ruoyi-gateway \
-n ruoyi --timeout=10m
kubectl rollout status deployment/ruoyi-auth \
-n ruoyi --timeout=10m
kubectl rollout status deployment/ruoyi-system \
-n ruoyi --timeout=10m
kubectl get deployment \
-n ruoyi \
-o custom-columns='NAME:.metadata.name,IMAGES:.spec.template.spec.containers[*].image' \
| grep -E 'ruoyi-(gateway|auth|system)'
二十三、练习删除 Pod 后自动恢复
bash
GATEWAY_POD="$(
kubectl get pods \
-n ruoyi \
-l app.kubernetes.io/name=ruoyi-gateway \
-o jsonpath='{.items[0].metadata.name}'
)"
echo "$GATEWAY_POD"
kubectl delete pod "$GATEWAY_POD" -n ruoyi
kubectl get pods \
-n ruoyi \
-l app.kubernetes.io/name=ruoyi-gateway \
-o wide \
-w
重新恢复两个 1/1 Running 后按 Ctrl + C。
二十四、练习 Java 进程崩溃后的容器重启
bash
AUTH_POD="$(
kubectl get pods \
-n ruoyi \
-l app.kubernetes.io/name=ruoyi-auth \
-o jsonpath='{.items[0].metadata.name}'
)"
echo "$AUTH_POD"
kubectl get pod "$AUTH_POD" -n ruoyi \
-o custom-columns='NAME:.metadata.name,RESTARTS:.status.containerStatuses[0].restartCount'
kubectl exec -n ruoyi "$AUTH_POD" \
-- sh -c 'kill -9 1' || true
sleep 20
kubectl get pod "$AUTH_POD" -n ruoyi \
-o custom-columns='NAME:.metadata.name,STATUS:.status.phase,READY:.status.containerStatuses[0].ready,RESTARTS:.status.containerStatuses[0].restartCount'
RESTARTS 应增加,Pod 最终重新 Ready。
二十五、安全练习节点不可调度后的迁移
MySQL、Redis、Nacos 使用 local-path,不要对 Worker 直接执行
drain,也不要关闭承载本地 PVC 的节点。
bash
GATEWAY_POD="$(
kubectl get pods \
-n ruoyi \
-l app.kubernetes.io/name=ruoyi-gateway \
-o jsonpath='{.items[0].metadata.name}'
)"
GATEWAY_NODE="$(
kubectl get pod "$GATEWAY_POD" \
-n ruoyi \
-o jsonpath='{.spec.nodeName}'
)"
echo "Pod:${GATEWAY_POD}"
echo "节点:${GATEWAY_NODE}"
kubectl cordon "$GATEWAY_NODE"
kubectl delete pod "$GATEWAY_POD" -n ruoyi
kubectl get pods \
-n ruoyi \
-l app.kubernetes.io/name=ruoyi-gateway \
-o wide \
-w
新 Pod 在其他 Worker 启动后按 Ctrl + C,然后恢复节点:
bash
kubectl uncordon "$GATEWAY_NODE"
kubectl get nodes
二十六、清理 Worker 临时 tar
确认 monitoring-v2 正常运行后,在三个 Worker 分别执行:
bash
sudo rm -f \
/tmp/ruoyi-gateway-3.6.8-monitoring-v1.tar \
/tmp/ruoyi-auth-3.6.8-monitoring-v1.tar \
/tmp/ruoyi-system-3.6.8-monitoring-v1.tar \
/tmp/ruoyi-gateway-3.6.8-monitoring-v2.tar \
/tmp/ruoyi-auth-3.6.8-monitoring-v2.tar \
/tmp/ruoyi-system-3.6.8-monitoring-v2.tar
二十七、最终检查
bash
echo '===== Helm ====='
helm list -n ruoyi
echo
echo '===== Deployment ====='
kubectl get deployment \
ruoyi-gateway ruoyi-auth ruoyi-system \
-n ruoyi
echo
echo '===== Pod ====='
kubectl get pods -n ruoyi -o wide
echo
echo '===== ServiceMonitor ====='
kubectl get servicemonitor \
ruoyi-jvm \
-n monitoring \
--show-labels
echo
echo '===== 监控 Service ====='
kubectl get svc \
-n ruoyi \
-l monitoring=ruoyi-jvm
echo
echo '===== 节点 ====='
kubectl get nodes
最终标准:
text
Gateway:2/2
Auth:2/2
System:2/2
Gateway/Auth/System 镜像:3.6.8-monitoring-v2
Prometheus 中每个服务有 2 个 UP Target
Grafana 可以查询 JVM 堆、线程、CPU、GC、HTTP 指标
删除 Pod 后自动补回
Java 进程退出后容器自动重启
cordon 后无状态 Pod 可以迁移到其他 Worker
所有节点最终恢复 Ready,没有 SchedulingDisabled
二十八、回滚命令
查看 Helm 历史:
bash
helm history ruoyi-app -n ruoyi
回滚:
bash
helm rollback ruoyi-app <上一版本Revision> \
-n ruoyi \
--wait \
--timeout 20m
查看 Deployment 历史:
bash
kubectl rollout history deployment/ruoyi-gateway -n ruoyi
kubectl rollout history deployment/ruoyi-auth -n ruoyi
kubectl rollout history deployment/ruoyi-system -n ruoyi