【k8s部署elasticsearch】k8s环境下安装elasticsearch集群和kibana

汪碧康2024-07-16 15:29

文章目录

简介

k8s集群中搭建有elasticsearch服务一般都会用到pvc，但是考虑到有些自建k8s环境下，搭建的共享存储可能会存在稳定性及性能问题，所以这次是通过采用节点亲和性和hostpath来实现，目前的operator的基本都是采用共享存储的方法。本文将根据现有环境及不同需求将elasticsearch集群的搭建采用hostpath+亲和性的权重+多个副本分区的方式来实现数据持久化和高可用。

一.条件及环境说明

k8s版本k8s-1.29.4，环境搭建在电信机房,六个worker节点，每个节点有一块非系统盘的ssd盘挂载到/data/路径，不采用ceph或nas之类的共享存储，也未采用operator。

二.需求说明

  • 搭建三个节点的elasticsearch集群，并配置三个主分区，2个副本分区。
  • 高可用：有一个k8s节点长期故障后或重启pod之后也不影响使用。
  • 高性能：读写本地磁盘实现高性能的io，共享存储如果资源及硬件性能不够的话，io将会是性能瓶颈。
  • 数据安全：在发生切换,节点故障以后，数据要尽可能的保证完整
  • 安装简单、管理维护容易

三.实现原理及说明

四.详细步骤

4.1.规划节点标签
节点名称 角色 标签
k8s-worker-120-81 主 storage-selector=node-a-master
k8s-worker-120-82 主 storage-selector=node-b-master
k8s-worker-120-83 主 storage-selector=node-c-master
k8s-worker-120-84 备 storage-selector=node-a-slve
k8s-worker-120-85 备 storage-selector=node-a-slve
k8s-worker-120-86 备 storage-selector=node-a-slve

注：这里的角色划分是指：redis由于是一主两从，每个redis的pod分别落在三个主的节点上，当主节点发生故障是，就切换到备节点，例如：名称为autopocket-es-a-0的pod，默认落在标签为storage-selector=node-a-master的节点上，当该节点发生故障是，就会根据权重匹配罗在storage-selector=node-a-slve的节点上。

4.2.创建三个statefulset和service headless配置

1.elasticsearch_a.yaml 节点a的statuflset配置和无头服务配置

yaml 复制代码 
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: autopocket-es-a
  namespace: zx-app
  labels:
    app: autopocket-es-a 
    appname: pcauto-pocket
    appgroup: autopocket-es
spec:
  serviceName: "autopocket-es-a"
  replicas: 1 
  selector:
    matchLabels:
      app: autopocket-es-a
      appgroup: autopocket-es
  template:
    metadata:
      labels:
        app: autopocket-es-a
        appgroup: autopocket-es
    spec:
      affinity:
        nodeAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 80
            preference:
              matchExpressions:
              - key: storage-selector 
                operator: In
                values:
                - node-a-master
          - weight: 20
            preference:
              matchExpressions:
              - key: storage-selector 
                operator: In
                values:
                - node-a-slave
      containers:
      - name: elasticsearch
        image: "xxx.cn-shenzhen.cr.aliyuncs.com/public/elasticsearch:7.17.22"
        command: ["/bin/sh","-c"]
        args:
        - mkdir -p /home/elastic/${POD_NAME}/data /home/elastic/${POD_NAME}/logs;chown elasticsearch.elasticsearch -R /home/elastic;su elasticsearch -c "/usr/share/elasticsearch/bin/elasticsearch"
        env:
        - name: cluster.name
          value: autopocket-es 
        - name: K8S_SERVICE_NAME
          value: autopocket-es-a 
        - name: POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        - name: discovery.seed_hosts
          value: "autopocket-es-a-0.autopocket-es-a.$(POD_NAMESPACE).svc.cluster.local,autopocket-es-b-0.autopocket-es-b.$(POD_NAMESPACE).svc.cluster.local,autopocket-es-c-0.autopocket-es-c.$(POD_NAMESPACE).svc.cluster.local"
        - name: cluster.initial_master_nodes
          value: "autopocket-es-a-0.autopocket-es-a.$(POD_NAMESPACE).svc.cluster.local"
        - name: node.name
          value: $(POD_NAME).$(K8S_SERVICE_NAME).$(POD_NAMESPACE).svc.cluster.local 
        - name: path.data
          value: /home/elastic/$(POD_NAME)/data
        - name: path.logs
          value: /home/elastic/$(POD_NAME)/logs
        - name: xpack.security.enabled
          value: "false"
        ports:
        - name: rest
          containerPort: 9200
        - name: inter-node
          containerPort: 9300
        volumeMounts:
        - name: elasticsearch-data
          mountPath: /home/elastic
      restartPolicy: Always
      terminationGracePeriodSeconds: 30
      volumes:
        - name: elasticsearch-data 
          hostPath:
            path: /data/es_data
            type: ""
      tolerations:
      - key: node.kubernetes.io/not-ready
        operator: Exists
        effect: NoExecute
        tolerationSeconds: 3600
      - key: node.kubernetes.io/unreachable
        operator: Exists
        effect: NoExecute
        tolerationSeconds: 3600
---
apiVersion: v1
kind: Service
metadata:
  name: autopocket-es-a
  namespace: zx-app
  labels:
    appname: pcauto-pocket
    app: autopocket-es-a
spec:
  ports:
  - port: 9200
    name: rest
  - port: 9300
    name: inter-node
  clusterIP: None
  selector:
    app: autopocket-es-a

2.elasticsearch_b.yaml 节点b的statuflset配置和无头服务配置

yaml 复制代码 
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: autopocket-es-b
  namespace: zx-app
  labels:
    app: autopocket-es-b 
    appname: pcauto-pocket
    appgroup: autopocket-es
spec:
  serviceName: "autopocket-es-b"
  replicas: 1 
  selector:
    matchLabels:
      app: autopocket-es-b
      appgroup: autopocket-es
  template:
    metadata:
      labels:
        app: autopocket-es-b
        appgroup: autopocket-es
    spec:
      affinity:
        nodeAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 80
            preference:
              matchExpressions:
              - key: storage-selector 
                operator: In
                values:
                - node-b-master
          - weight: 20
            preference:
              matchExpressions:
              - key: storage-selector 
                operator: In
                values:
                - node-b-slave
      containers:
      - name: elasticsearch
        image: "xxx.cn-shenzhen.cr.aliyuncs.com/public/elasticsearch:7.17.22"
        command: ["/bin/sh","-c"]
        args:
        - mkdir -p /home/elastic/${POD_NAME}/data /home/elastic/${POD_NAME}/logs;chown elasticsearch.elasticsearch -R /home/elastic;su elasticsearch -c "/usr/share/elasticsearch/bin/elasticsearch"
        env:
        - name: cluster.name
          value: autopocket-es 
        - name: K8S_SERVICE_NAME
          value: autopocket-es-b 
        - name: POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        - name: discovery.seed_hosts
          value: "autopocket-es-a-0.autopocket-es-a.$(POD_NAMESPACE).svc.cluster.local,autopocket-es-b-0.autopocket-es-b.$(POD_NAMESPACE).svc.cluster.local,autopocket-es-c-0.autopocket-es-c.$(POD_NAMESPACE).svc.cluster.local"
        - name: cluster.initial_master_nodes
          value: "autopocket-es-a-0.autopocket-es-a.$(POD_NAMESPACE).svc.cluster.local"
        - name: node.name
          value: $(POD_NAME).$(K8S_SERVICE_NAME).$(POD_NAMESPACE).svc.cluster.local 
        - name: path.data
          value: /home/elastic/$(POD_NAME)/data
        - name: path.logs
          value: /home/elastic/$(POD_NAME)/logs
        - name: xpack.security.enabled
          value: "false"
        ports:
        - name: rest
          containerPort: 9200
        - name: inter-node
          containerPort: 9300
        volumeMounts:
        - name: elasticsearch-data
          mountPath: /home/elastic
      restartPolicy: Always
      terminationGracePeriodSeconds: 30
      volumes:
        - name: elasticsearch-data 
          hostPath:
            path: /data/es_data
            type: ""
      tolerations:
      - key: node.kubernetes.io/not-ready
        operator: Exists
        effect: NoExecute
        tolerationSeconds: 3600
      - key: node.kubernetes.io/unreachable
        operator: Exists
        effect: NoExecute
        tolerationSeconds: 3600
---
apiVersion: v1
kind: Service
metadata:
  name: autopocket-es-b
  namespace: zx-app
  labels:
    appname: pcauto-pocket
    app: autopocket-es-b
spec:
  ports:
  - port: 9200
    name: rest
  - port: 9300
    name: inter-node
  clusterIP: None
  selector:
    app: autopocket-es-b

3.elasticsearch_c.yaml 节点c的statuflset配置和无头服务配置

yaml 复制代码 
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: autopocket-es-c
  namespace: zx-app
  labels:
    app: autopocket-es-c 
    appname: pcauto-pocket
    appgroup: autopocket-es
spec:
  serviceName: "autopocket-es-c"
  replicas: 1 
  selector:
    matchLabels:
      app: autopocket-es-c
      appgroup: autopocket-es
  template:
    metadata:
      labels:
        app: autopocket-es-c
        appgroup: autopocket-es
    spec:
      affinity:
        nodeAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 80
            preference:
              matchExpressions:
              - key: storage-selector 
                operator: In
                values:
                - node-c-master
          - weight: 20
            preference:
              matchExpressions:
              - key: storage-selector 
                operator: In
                values:
                - node-c-slave
      containers:
      - name: elasticsearch
        image: "xxx.cn-shenzhen.cr.aliyuncs.com/public/elasticsearch:7.17.22"
        command: ["/bin/sh","-c"]
        args:
        - mkdir -p /home/elastic/${POD_NAME}/data /home/elastic/${POD_NAME}/logs;chown elasticsearch.elasticsearch -R /home/elastic;su elasticsearch -c "/usr/share/elasticsearch/bin/elasticsearch"
        env:
        - name: cluster.name
          value: autopocket-es 
        - name: K8S_SERVICE_NAME
          value: autopocket-es-c 
        - name: POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        - name: discovery.seed_hosts
          value: "autopocket-es-a-0.autopocket-es-a.$(POD_NAMESPACE).svc.cluster.local,autopocket-es-b-0.autopocket-es-b.$(POD_NAMESPACE).svc.cluster.local,autopocket-es-c-0.autopocket-es-c.$(POD_NAMESPACE).svc.cluster.local"
        - name: cluster.initial_master_nodes
          value: "autopocket-es-a-0.autopocket-es-a.$(POD_NAMESPACE).svc.cluster.local"
        - name: node.name
          value: $(POD_NAME).$(K8S_SERVICE_NAME).$(POD_NAMESPACE).svc.cluster.local 
        - name: path.data
          value: /home/elastic/$(POD_NAME)/data
        - name: path.logs
          value: /home/elastic/$(POD_NAME)/logs
        - name: xpack.security.enabled
          value: "false"
        ports:
        - name: rest
          containerPort: 9200
        - name: inter-node
          containerPort: 9300
        volumeMounts:
        - name: elasticsearch-data
          mountPath: /home/elastic
      restartPolicy: Always
      terminationGracePeriodSeconds: 30
      volumes:
        - name: elasticsearch-data 
          hostPath:
            path: /data/es_data
            type: ""
      tolerations:
      - key: node.kubernetes.io/not-ready
        operator: Exists
        effect: NoExecute
        tolerationSeconds: 3600
      - key: node.kubernetes.io/unreachable
        operator: Exists
        effect: NoExecute
        tolerationSeconds: 3600
---
apiVersion: v1
kind: Service
metadata:
  name: autopocket-es-c
  namespace: zx-app
  labels:
    appname: pcauto-pocket
    app: autopocket-es-c
spec:
  ports:
  - port: 9200
    name: rest
  - port: 9300
    name: inter-node
  clusterIP: None
  selector:
    app: autopocket-es-c
4.3.创建service配置

给9200 9300配置一个service的代理服务，注意其中的selector是指向到appgroup,这是将三个statefulset定义了一个appgroup的标签。

复制代码 
apiVersion: v1
kind: Service
metadata:
  name: autopocket-es
  namespace: zx-app
  labels:
    app: autopocket-es
    appname: pcauto-zx
spec:
  ports:
  - port: 9200
    name: rest
  - port: 9300
    name: inter-node
  selector:
    appgroup: autopocket-es
  type: LoadBalancer

五.安装kibana

复制代码 
apiVersion: apps/v1
kind: Deployment
metadata:
  name: autopocket-es-kibana
  namespace: zx-app
  labels:
    app: autopocket-es-kibana
    appname: pcauto-pocket
spec:
  replicas: 1
  selector:
    matchLabels:
      app: autopocket-es-kibana
  template:
    metadata:
      labels:
        app: autopocket-es-kibana
    spec:
      containers:
      - name: kibana
        image: "xxx.cn-shenzhen.cr.aliyuncs.com/public/kibana:7.17.22"
        ports:
        - containerPort: 5601
        env:
        - name: ELASTICSEARCH_HOSTS
          value: '["http://autopocket-es-a-0.autopocket-es-a:9200","http://autopocket-es-b-0.autopocket-es-b:9200","http://autopocket-es-c-0.autopocket-es-c:9200"]'
---
apiVersion: v1
kind: Service
metadata:
  name: autopocket-es-kibana
  namespace: zx-app
  labels:
    app: autopocket-es-kibana
    appname: pcauto-pocket
spec:
  selector:
    app: autopocket-es-kibana
  ports:
  - port: 5601
    targetPort: 5601
  type: LoadBalancer

查看集群健康情况：http://10.16.xx.xx:9200/_cat/health?v
elasticsearch集群状态查看：http://10.16.1xx.xx:9200/_cat/nodes?v

索引状态查看：http://10.16.1xx.xx:9200/_cat/indices?v

kibana地址登录：http://10.16.1xx.xx:5601/app/home#/

六.调整索引分区

调整已存在索引的副本分区：curl -XPUT -H "Content-Type:application/json" -d '{"settings":{"number_of_replicas":2}}' 'http://10.16.xx.xx:9200/testindex/_settings'
调整集群最大分片数：curl -XPUT -H "Content-Type:application/json" http://10.16.xx.xx:9200/_cluster/settings -d '{"transient":{"cluster":{"max_shards_per_node":10000}}}'
调整默认主分片和副本分片数: curl -X PUT -H "Content-Type:application/json" -d '{"index_patterns":["*"],"settings":{"number_of_shards":3,"number_of_replicas":2}}' http://10.16.xx.xx:9200/_template/template_http_request_record

七.安装说明

  • 需要先在k8s的节点创建/data/es_data的路径来存放数据。
  • 使用配置前将配置中的autopocket替换成自己需要的命名,配置的命名空间为：zx-app，修改成自己的命名空间，appname为pcauto-pocket，修改成自己的应用名称，该标签也可以删除。
  • 配置中的镜像地址采用了私有的镜像地址，镜像是dockerhub上下载的elasticsearch:7.17.22 镜像到私有镜像仓库的，如果能直通外网的k8s可以直接用elasticsearch:7.17.22,kibana的镜像是kibana:7.17.22。否则配置一个可以通的镜像地址。
  • 默认没有配置用户名和密码。
  • 节点标签的命名规则若有更改需要在每个statefulset中的affinity: 中的标签名称进行修改。
相关推荐
skyeeeeee9 小时前
kubeadm安装k8s集群
后端·kubernetes
虚伪的空想家10 小时前
记录次etcd故障，fatal error: bus error
服务器·数据库·k8s·etcd
星光一影11 小时前
基于SpringBoot智慧社区系统/乡村振兴系统/大数据与人工智能平台
大数据·spring boot·后端·mysql·elasticsearch·vue
高旭博11 小时前
7. kubernetes资源——service服务
云原生·容器·kubernetes
斯普信专业组12 小时前
Docker 常用命令与时区配置指南
docker·容器·eureka
天一生水water1 天前
ubuntu使用毫秒镜像方式安装docker mysql
ubuntu·阿里云·docker
Qayrup1 天前
docker 搭建私有仓库,推送并拉取
运维·docker·容器
郭庆汝1 天前
docker拉取英伟达官方cuda11.8镜像
docker·cuda11.8
天下不喵1 天前
Ubuntu24.04安装Docker过程记录
docker
热门推荐
01GitHub 镜像站点02UV安装并设置国内源03BongoCat - 跨平台键盘猫动画工具04综合整理：pdf预览显示：你尝试预览的文件可能对你的计算机有害。如果你信任此文件以及其来源，请打开此文件以看其内容，如何解决以正常预览文件05Linux下V2Ray安装配置指南06安娜的档案(Anna’s Archive) 镜像网站/国内最新可访问入口（持续更新）07jdk21下载、安装（Windows、Linux、macOS）08《大数据技术原理与应用》实验报告三 熟悉HBase常用操作09PyCharm 社区版全平台安装指南10npm使用国内淘宝镜像的方法