k8s部署es和skywalking

张哲溪2023-12-06 21:58

使用k8s部署es和skywalking

skywalking介绍

skywalking架构

整个架构，分成上、下、左、右四部分：

  • 上部分
    Agent ：负责从应用中，收集链路信息，发送给 SkyWalking OAP 服务器。目前支持 SkyWalking、Zikpin、Jaeger 等提供的 Tracing 数据信息。而我们目前采用的是，SkyWalking Agent 收集 SkyWalking Tracing 数据，传递给服务器
  • 下部分
    SkyWalking OAP ：负责接收 Agent 发送的 Tracing 数据信息，然后进行分析(Analysis Core) ，存储到外部存储器( Storage )，最终提供查询( Query )功能
  • 右部分
    Storage ：Tracing 数据存储。目前支持 ES、MySQL、Sharding Sphere、TiDB、H2 多种存储器。而我们目前采用的是 ES ，主要考虑是 SkyWalking 开发团队自己的生产环境采用 ES 为主
  • 左部分
    SkyWalking UI ：负责提供控台，查看链路等等
    部署elasticsearch
    skywalking的持久化选择的是elasticsearch，请大家根据自己自己的情况来选择持久化的方式。

部署storageclass

我不太喜欢使用helm一键部署的方式来进行部署服务，helm的方式有点像黑盒。我喜欢使用yaml文件一个服务一个服务来部署。下面是部署的yaml 文件。

storageclass.yaml

yaml 复制代码 
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: nfs-storage
  namespace: skywalking 
provisioner: fuseim.pri/ifs # or choose another name, must match deployment's env PROVISIONER_NAME'
parameters:
  archiveOnDelete: "false"
reclaimPolicy: Retain

部署nfs-provisioner

由于在k8s 1.20以上版本弃用了selfLink，我这里就不用nfs-client-provisioner镜像了，使用dyrnq/nfs-subdir-external-provisioner。

yaml 复制代码 
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nfs-client-provisioner
spec:
  replicas: 1
  strategy:
    type: Recreate
  selector:
    matchLabels:
      app: nfs-client-provisioner
  template:
    metadata:
      labels:
        app: nfs-client-provisioner
    spec:
      serviceAccountName: nfs-client-provisioner
      containers:
        - name: nfs-client-provisioner
          image: dyrnq/nfs-subdir-external-provisioner:v4.0.2
          imagePullPolicy: IfNotPresent
          volumeMounts:
            - name: nfs-client-root
              mountPath: /persistentvolumes
          env:
            - name: PROVISIONER_NAME
              value: fuseim.pri/ifs
            - name: NFS_SERVER
              value: 192.168.5.166
            - name: NFS_PATH
              value: /mnt/jf/skywalking
      volumes:
        - name: nfs-client-root
          nfs:
            server: 192.168.5.166
            path: /mnt/jf/skywalking
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: nfs-client-provisioner
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: nfs-client-provisioner-runner
rules:
  - apiGroups: [""]
    resources: ["persistentvolumes"]
    verbs: ["get", "list", "watch", "create", "delete"]
  - apiGroups: [""]
    resources: ["persistentvolumeclaims"]
    verbs: ["get", "list", "watch", "update"]
  - apiGroups: ["storage.k8s.io"]
    resources: ["storageclasses"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["events"]
    verbs: ["create", "update", "patch"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: run-nfs-client-provisioner
subjects:
  - kind: ServiceAccount
    name: nfs-client-provisioner
    namespace: skywalking
roleRef:
  kind: ClusterRole
  name: nfs-client-provisioner-runner
  apiGroup: rbac.authorization.k8s.io
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: leader-locking-nfs-client-provisioner
rules:
  - apiGroups: [""]
    resources: ["endpoints"]
    verbs: ["get", "list", "watch", "create", "update", "patch"]
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: leader-locking-nfs-client-provisioner
subjects:
  - kind: ServiceAccount
    name: nfs-client-provisioner
roleRef:
  kind: Role
  name: leader-locking-nfs-client-provisioner
  apiGroup: rbac.authorization.k8s.io

部署elasticsearch

由于es属于有状态的服务，我这里选择使用statefulset方式来部署

es-statefulset.yaml

yaml 复制代码 
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: es
  namespace: skywalking
spec:
  serviceName: elasticsearch
  replicas: 3
  selector:
    matchLabels:
      app: elasticsearch
  template:
    metadata:
      labels: 
        app: elasticsearch
    spec:
      imagePullSecrets:
        - name: harborsecret
      initContainers:
      - name: increase-vm-max-map
        image: busybox:latest
        command: ["sysctl", "-w", "vm.max_map_count=262144"]
        securityContext:
          privileged: true
      - name: increase-fd-ulimit
        image: busybox:latest
        command: ["sh", "-c", "ulimit -n 65536"]
        securityContext:
          privileged: true
      containers:
      - name: elasticsearch
        image: elasticsearch:7.17.14
        ports:
        - name: rest
          containerPort: 9200
        - name: inter
          containerPort: 9300
        resources:
          limits:
            cpu: 1000m
          requests:
            cpu: 1000m
        volumeMounts:
        - name: data
          mountPath: /usr/share/elasticsearch/data
        env:
        - name: cluster.name
          value: k8s-logs
        - name: node.name
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: cluster.initial_master_nodes
          value: "es-0,es-1,es-2"
        - name: discovery.seed_hosts
          value: "elasticsearch"
        - name: ES_JAVA_OPTS
          value: "-Xms512m -Xmx512m"
        - name: network.host
          value: "0.0.0.0"
  volumeClaimTemplates:
  - metadata:
      name: data
      labels:
        app: elasticsearch
    spec:
      accessModes: [ "ReadWriteOnce" ]
      storageClassName: nfs-storage
      resources:
        requests:
          storage: 300Gi 
          
service.yaml
kind: Service
apiVersion: v1
metadata:
  name: elasticsearch
  namespace: skywalking
  labels:
    app: elasticsearch
spec:
  selector:
    app: elasticsearch
  clusterIP: None
  ports:
    - port: 9200
      name: rest
    - port: 9300
      name: inter-node
---
kind: Service
apiVersion: v1
metadata:
  name: elasticsearch-client
  namespace: skywalking
  labels:
    app: elasticsearch
spec:
  selector:
    app: elasticsearch
  ports:
    - port: 9200
      name: rest
    - port: 9300
      name: inter-node

部署完es以后，通过访问IP:9200进行访问。测试es是否部署成功。

在浏览器访问出现如下界面，证书部署成功：

部署skywalking

部署RBAC

RBAC.yaml

yaml 复制代码 
apiVersion: v1
kind: ServiceAccount
metadata:
  labels:
    app: skywalking
  name: skywalking-oap
  namespace: skywalking
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: skywalking
  namespace: skywalking  
  labels:
    app: skywalking
rules:
  - apiGroups: [""]
    resources: ["pods", "endpoints", "services", "nodes"]
    verbs: ["get", "watch", "list"]
  - apiGroups: ["extensions"]
    resources: ["deployments", "replicasets"]
    verbs: ["get", "watch", "list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: skywalking
  namespace: skywalking  
  labels:
    app: skywalking
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: skywalking
subjects:
  - kind: ServiceAccount
    name: skywalking-oap
    namespace: skywalking

部署数据初始化Job

yaml 复制代码 
apiVersion: batch/v1
kind: Job
metadata:
  name: "skywalking-es-init"
  namespace: skywalking  
  labels:
    app: skywalking-job
spec:
  template:
    metadata:
      name: "skywalking-es-init"
      labels:
        app: skywalking-job
    spec:
      serviceAccountName: skywalking-oap
      restartPolicy: Never
      initContainers:
      - name: wait-for-elasticsearch
        image: busybox:1.30
        imagePullPolicy: IfNotPresent
        command: ['sh', '-c', 'for i in $(seq 1 60); do nc -z -w3 elasticsearch 9200 && exit 0 || sleep 5; done; exit 1']
      containers:
      - name: oap
        image: skywalking.docker.scarf.sh/apache/skywalking-oap-server:8.9.0
        imagePullPolicy: IfNotPresent
        env:
        - name: JAVA_OPTS
          value: "-Xmx2g -Xms2g -Dmode=init"
        - name: SW_STORAGE
          value: elasticsearch
        - name: SW_STORAGE_ES_CLUSTER_NODES
          value: "elasticsearch:9200"
        volumeMounts:
      volumes:

部署OAP

oap.yaml

yaml 复制代码 
apiVersion: v1
kind: Service
metadata:
  name: oap-svc
  namespace: skywalking  
  labels:
    app: oap
spec:
  type: ClusterIP
  ports:
  - port: 11800
    name: grpc
  - port: 12800
    name: rest
  selector:
    app: oap
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: oap
  name: oap
  namespace: skywalking  
spec:
  replicas: 1
  selector:
    matchLabels:
      app: oap
  template:
    metadata:
      labels:
        app: oap
    spec:
      serviceAccountName: skywalking-oap
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 1
            podAffinityTerm:
              topologyKey: kubernetes.io/hostname
              labelSelector:
                matchLabels:
                  app: "skywalking"
      initContainers:
      - name: wait-for-elasticsearch
        image: busybox:1.30
        imagePullPolicy: IfNotPresent
        command: ['sh', '-c', 'for i in $(seq 1 60); do nc -z -w3 elasticsearch 9200 && exit 0 || sleep 5; done; exit 1']
      containers:
      - name: oap
        image: skywalking.docker.scarf.sh/apache/skywalking-oap-server:8.9.0
        imagePullPolicy: IfNotPresent
        livenessProbe:
          tcpSocket:
            port: 12800
          initialDelaySeconds: 15
          periodSeconds: 20
        readinessProbe:
          tcpSocket:
            port: 12800
          initialDelaySeconds: 15
          periodSeconds: 20
        ports:
        - containerPort: 11800
          name: grpc
        - containerPort: 12800
          name: rest
        env:
        - name: JAVA_OPTS
          value: "-Dmode=no-init -Xmx2g -Xms2g"
        - name: SW_CLUSTER
          value: kubernetes
        - name: SW_CLUSTER_K8S_NAMESPACE
          value: "skywalking"
        - name: SW_CLUSTER_K8S_LABEL
          value: "app=skywalking,release=skywalking,component=oap"
        # 记录数据
        - name: SW_CORE_RECORD_DATA_TTL
          value: "2"
        # Metrics数据  
        - name: SW_CORE_METRICS_DATA_TTL
          value: "2"
        - name: SKYWALKING_COLLECTOR_UID
          valueFrom:
            fieldRef:
              fieldPath: metadata.uid
        - name: SW_STORAGE
          value: elasticsearch
        - name: SW_STORAGE_ES_CLUSTER_NODES
          value: "elasticsearch:9200"

部署UI

skywalking的UI界面

ui.yaml

yaml 复制代码 
apiVersion: v1
kind: Service
metadata:
  labels:
    app: ui
  name: ui-svc
  namespace: skywalking  
spec:
  type: ClusterIP
  ports:
    - port: 80
      targetPort: 8080
      protocol: TCP
  selector:
    app: ui
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: ui
  namespace: skywalking  
  labels:
    app: ui
spec:
  replicas: 1
  selector:
    matchLabels:
        app: ui
  template:
    metadata:
      labels:
        app: ui
    spec:
      affinity:
      containers:
      - name: ui
        image: skywalking.docker.scarf.sh/apache/skywalking-ui:8.9.0
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 8080
          name: page
        env:
        - name: SW_OAP_ADDRESS
          value: http://elasticsearch:12800  #根据oap的svc一致

查看服务

测试部署项目接入agent

编写entrypoint.sh

在jvm启动参数中添加skywalking相关的启动参数

java 复制代码 
java -javaagent:/skywalking-agent/skywalking-agent.jar \
-Dskywalking.agent.service_name=${appName} \
-Dskywalking.collector.backend_service=skywalking的service名称:11800

添加java的agent

根据你后端语言选择不同的agent进行下载，下载地址，我这边使用的是java agent

下载java-agnet

可以选择将agent下载到指定的服务器

bash 复制代码 
wget https://archive.apache.org/dist/skywalking/java-agent/8.10.0/apache-skywalking-java-agent-8.10.0.tgz
tar xf apache-skywalking-java-agent-8.10.0.tgz 
ls
apache-skywalking-java-agent-8.10.0.tgz  skywalking-agent

我这直接将java的agent添加到dockerfile中。

bash 复制代码 
# 添加以下指令将SkyWalking Agent文件拷贝到容器内
ADD https://archive.apache.org/dist/skywalking/java-agent/8.10.0/apache-skywalking-java-agent-8.10.0.tgz $PROJECT_ROOT/conf/skywalking-agent.tgz
# 解压Agent文件
RUN tar -xzf $PROJECT_ROOT/conf/skywalking-agent.tgz -C $PROJECT_ROOT/conf/

调用链忽略（取消跟踪）

bash 复制代码 
cd skywalking-agent
cp optional-plugins/apm-trace-ignore-plugin-8.10.0.jar plugins/
vim config/apm-trace-ignore-plugin.config 
#添加下面这段
trace.ignore_path=${SW_AGENT_TRACE_IGNORE_PATH:GET:/actuator/**,Redisson/**,Mysql/**,HikariCP/**,Lettuce/**,/xxl-job/**,UndertowDispatch/**}

将服务部署到k8s集群上

根据自己对应的服务进行编写yaml文件

验证

部署完自己的微服务以后，就可以进行验证了，登录到swkwalking的UI界面，查看是否有自己部署的服务。

相关推荐
小白跃升坊20 分钟前
基于1Panel的AI运维
linux·运维·人工智能·ai大模型·教学·ai agent
Gary董33 分钟前
高并发的微服务架构如何设计
微服务·云原生·架构
东哥爱编程35 分钟前
使用Runpod进行gpu serverless推理
云原生·serverless
杨江40 分钟前
seafile docker安装说明
运维
好好沉淀1 小时前
Docker开发笔记(详解)
运维·docker·容器
zmjjdank1ng1 小时前
Linux 输出重定向
linux·运维
路由侠内网穿透.1 小时前
本地部署智能家居集成解决方案 ESPHome 并实现外部访问( Linux 版本）
linux·运维·服务器·网络协议·智能家居
树℡独1 小时前
ns-3仿真之应用层（三）
运维·服务器·ns3
VekiSon1 小时前
Linux内核驱动——基础概念与开发环境搭建
linux·运维·服务器·c语言·arm开发
skywalk81632 小时前
尝试在openi启智社区的dcu环境安装ollama最新版0.15.2（失败）
linux·运维·服务器·ollama
热门推荐
01GitHub 镜像站点02Clawdbot 中文汉化版 接入微信、飞书032026美赛A题智能手机电池续航时间预测的连续时间数学模型04OpenCode 入门教程：介绍 · 安装 · 配置第三方 API (如 Claude)052025 年大语言模型发展回顾：关键突破、意外转折与 2026 年展望06【Milvus】向量数据库pymilvus使用教程072026数学建模美赛题目特点与选题建议，常用四大模型汇总08Claude Code Skills 实用使用手册09一种新的LCA算法10UV安装并设置国内源