K8s 运维体系搭建:全组件监控与日志收集实战(victoria-metrics-k8s-stack 0.87.0)

文章目录

- 环境
- 部署过程
- - 一、把k8s组件默认的127.0.0.1监听改为0.0.0.0，使监控能访问到
  - 二、部署victoria-metrics-k8s-stack
  - - 1.添加仓库
    - 2.下载grafana插件
    - [3.在/tmp目录上起个临时 http 服务分发](#3.在/tmp目录上起个临时 http 服务分发)
    - 4.验证
    - 5.编写value.yaml文件
    - 6.部署
  - 三、验证
  - - [1. 把grafana的svc改成nodeport模式](#1. 把grafana的svc改成nodeport模式)
    - 2.浏览器访问

环境

Ip	主机名	cpu	内存
192.168.10.11	master01	4c	6G
192.168.10.12	node1	4c	6G
192.168.10.13	node2	4c	6G
192.168.10.100	nfs	2c	2g

组件	版本
Ubuntu	Ubuntu 26.04 server
containerd	v2 2.2.2
Kubernetes	v1.36.1
victoria-metrics-k8s-stack	0.87.0
victoriametrics-logs-datasource	0.30.1

部署过程

一、把k8s组件默认的127.0.0.1监听改为0.0.0.0，使监控能访问到

1.备份原文件

powershell 复制代码

mkdir -p /root/k8s-bak
cp -r /etc/kubernetes/manifests/* /root/k8s-bak/
ls /root/k8s-bak/

2.用sed修改其配置文件

powershell 复制代码

sed -i 's/--bind-address=127.0.0.1/--bind-address=0.0.0.0/' /etc/kubernetes/manifests/kube-controller-manager.yaml
sed -i 's/--bind-address=127.0.0.1/--bind-address=0.0.0.0/' /etc/kubernetes/manifests/kube-scheduler.yaml
sed -i 's#--listen-metrics-urls=http://127.0.0.1:2381#--listen-metrics-urls=http://0.0.0.0:2381#' /etc/kubernetes/manifests/etcd.yaml
# 等待半分钟
kubectl -n kube-system get pod -l tier=control-plane -o wide

静态pod全起来即可

注:如果为多master群集，每个带有etcd的节点都需要改，且需要等第一个etcd起来再改第二个,以此类推

3.修改kube-proxy

powershell 复制代码

kubectl -n kube-system get cm kube-proxy -o yaml \
  | sed 's/metricsBindAddress: ""/metricsBindAddress: "0.0.0.0:10249"/' \
  | kubectl apply -f -

kubectl -n kube-system rollout restart daemonset kube-proxy

4.验证配置

powershell 复制代码

for x in "https://192.168.10.11:10257" "https://192.168.10.11:10259"; do
  echo -n "$x -> "; curl -k -o /dev/null -s -w "%{http_code}\n" $x/metrics
done
curl http://192.168.10.11:2381/metrics  -o /dev/null -s -w "etcd -> %{http_code}\n"
curl http://192.168.10.11:10249/metrics -o /dev/null -s -w "kube-proxy -> %{http_code}\n"

返回结果是200或者403就行，都代表通了

二、部署victoria-metrics-k8s-stack

1.添加仓库

powershell 复制代码

helm repo add vm https://victoriametrics.github.io/helm-charts/
helm repo update

2.下载grafana插件

注:grafana想要查看收集到的日志需要安装victoriametrics-logs-datasource插件，但是服务器代理/contrainerd代理/容器内环境变量设置代理均拉不下来，所以只能上传到服务器内然后让grafana拉取本地服务器内的插件包

浏览器访问

复制代码

https://github.com/VictoriaMetrics/victorialogs-datasource/releases/latest

目前最新版本是v0.30.1，下载zip包后传到服务器(192.168.10.11)上的/tmp目录下

3.在/tmp目录上起个临时 http 服务分发

这个页面不要关，后面起个新的连接，pod起来后就可以关闭了

powershell 复制代码

cd /tmp/
chmod 777 victoriametrics-logs-datasource-v0.30.1.zip
python3 -m http.server 8000

4.验证

在其他机器上看能不能访问到

200就代表通了

powershell 复制代码

 curl -I http://192.168.10.11:8000/victoriametrics-logs-datasource-v0.30.1.zip

5.编写value.yaml文件

vim value.yaml

powershell 复制代码

# =============================================================================
# victoria-metrics-k8s-stack ------ 生产配置(全集群版 HA)
# 集群标识: cluster=sit    存储类: nfs-client
# 单 master + 单 etcd(192.168.10.11)
# 【重点】所有数据存储组件均为集群版多副本,无单点:
#   - 指标: vmcluster (vmstorage×2 / vmselect×2 / vminsert×2, replicationFactor=2)
#   - 日志: vlcluster (vlstorage×2 / vlselect×2 / vlinsert×2)
# =============================================================================

# ======================== 指标存储:集群版 HA(vmcluster)=====================
vmsingle:
  enabled: false                         # ★单机版强制关闭,杜绝单点

vmcluster:
  enabled: true
  spec:
    retentionPeriod: "15d"
    replicationFactor: 2                  # 每份数据写2副本,挂1个storage不丢数据

    # --- 存储层:分片存储,每副本独立PVC(StatefulSet)---
    vmstorage:
      replicaCount: 2                     # ★集群:2副本,各自20Gi独立PVC
      storageDataPath: /vm-data
      storage:
        volumeClaimTemplate:
          spec:
            storageClassName: nfs-client
            accessModes: ["ReadWriteOnce"]
            resources:
              requests:
                storage: 20Gi
      resources:
        requests: { cpu: "250m", memory: 512Mi }
        limits:   { cpu: "1",    memory: 1Gi }

    # --- 查询层:多副本 ---
    vmselect:
      replicaCount: 2                     # ★集群:2副本
      cacheMountPath: /select-cache
      storage:
        volumeClaimTemplate:
          spec:
            storageClassName: nfs-client
            accessModes: ["ReadWriteOnce"]
            resources:
              requests:
                storage: 2Gi
      resources:
        requests: { cpu: "100m", memory: 256Mi }
        limits:   { cpu: "500m", memory: 512Mi }

    # --- 写入层:多副本 ---
    vminsert:
      replicaCount: 2                     # ★集群:2副本
      resources:
        requests: { cpu: "100m", memory: 128Mi }
        limits:   { cpu: "500m", memory: 256Mi }

# ============================ 采集器 vmagent ================================
# externalLabels: 给本集群采集的所有指标统一打 cluster=sit 标识
vmagent:
  enabled: true
  spec:
    externalLabels:
      cluster: sit                        # ★集群数据标识
    resources:
      requests: { cpu: "100m", memory: 256Mi }
      limits:   { cpu: "500m", memory: 512Mi }

# ------------------- 控制面组件抓取(endpoints 填具体 IP)-------------------
kubeApiServer:
  enabled: true
kubelet:
  enabled: true

kubeControllerManager:
  enabled: true
  endpoints:
    - 192.168.10.11
  service:
    enabled: true
    port: 10257
    targetPort: 10257
  vmScrape:
    spec:
      endpoints:
        - port: http-metrics
          scheme: https
          bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
          tlsConfig:
            insecureSkipVerify: true

kubeScheduler:
  enabled: true
  endpoints:
    - 192.168.10.11
  service:
    enabled: true
    port: 10259
    targetPort: 10259
  vmScrape:
    spec:
      endpoints:
        - port: http-metrics
          scheme: https
          bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
          tlsConfig:
            insecureSkipVerify: true

kubeEtcd:
  enabled: true
  endpoints:
    - 192.168.10.11
  service:
    enabled: true
    port: 2381
    targetPort: 2381
  vmScrape:
    spec:
      endpoints:
        - port: http-metrics
          scheme: http

kubeProxy:
  enabled: true
coreDns:
  enabled: true

# ---------------------------- 附属采集器 ------------------------------------
kube-state-metrics:
  enabled: true
prometheus-node-exporter:
  enabled: true

# ---------------------------- 告警 -----------------------------------------
vmalert:
  enabled: true
alertmanager:
  enabled: true

# ---------------------------- Grafana --------------------------------------
grafana:
  enabled: true
  adminPassword: "Admin@123456"
  env: #这里就是之前起的http服务分发
    GF_INSTALL_PLUGINS: "http://192.168.10.11:8000/victoriametrics-logs-datasource-v0.30.1.zip;victoriametrics-logs-datasource"
    GF_PLUGINS_ALLOW_LOADING_UNSIGNED_PLUGINS: "victoriametrics-logs-datasource"
  persistence:
    enabled: true
    storageClassName: nfs-client
    size: 5Gi

# ======================== 日志存储:集群版 HA(vlcluster)=====================
vlsingle:
  enabled: false                         # ★单机版强制关闭,杜绝单点

vlcluster:
  enabled: true
  spec:
    retentionPeriod: "15d"
    # --- 日志存储层:多副本 + 独立PVC ---
    vlstorage:
      replicaCount: 2                     # ★集群:2副本
      storage:
        volumeClaimTemplate:
          spec:
            storageClassName: nfs-client
            accessModes: ["ReadWriteOnce"]
            resources:
              requests:
                storage: 20Gi
      resources:
        requests: { cpu: "100m", memory: 256Mi }
        limits:   { cpu: "500m", memory: 512Mi }
    # --- 日志查询层:多副本 ---
    vlselect:
      replicaCount: 2                     # ★集群:2副本
    # --- 日志写入层:多副本 ---
    vlinsert:
      replicaCount: 2                     # ★集群:2副本

# ---------------------------- 日志采集 vlagent ------------------------------
vlagent:
  enabled: true
  spec:
    k8sCollector:
      enabled: true

6.部署

powershell 复制代码

helm install vmks vm/victoria-metrics-k8s-stack -f value.yaml   -n monitor --create-namespace

三、验证

1. 把grafana的svc改成nodeport模式

powershell 复制代码

kubectl get svc -A | grep grafana
kubectl edit svc -n monitor       vmks-grafana

把ClusterIP改成NodePort，注意大小写

查看随机的端口

powershell 复制代码

kubectl get svc -A | grep grafana

这里是30723

2.浏览器访问

k8s节点+端口

我这里是192.168.10.11:30723

账号admin，密码是自己设置的那个

这里可以改成中文

随便查看几个仪表盘

etcd

以及日志