ARM64环境部署EFK8.15.3收集K8S集群容器日志

环境规划

主机IP	系统	部署方式	ES版本	CPU架构	用户名	密码
192.168.1.225	Ubuntu 22.04.4 LTS	docker	elasticsearch:8.15.3	ARM64	elastic	llodyi4TMmZD

ES集群部署

创建持久化目录(所有节点)

mkdir -p  /data/es/{data,certs,logs,plugins}
mkdir -p /data/es/certs/{ca,es01}

服务器优化/配置(所有节点)

#提高进程及资源使用限制上线
vim /etc/security/limits.conf

* soft nofile 65536
* hard nofile 65536
* soft nproc 32000
* hard nproc 32000
* hard memlock unlimited
* soft memlock unlimited

vim /etc/systemd/system.conf

DefaultLimitNOFILE=65536
DefaultLimitNPROC=32000
DefaultLimitMEMLOCK=infinity

#重载配置文件
systemctl daemon-reload
systemctl daemon-reexec 

#修改虚拟内存最大映射数
vim /etc/sysctl.conf

#添加参数
vm.max_map_count = 262144

#重新加载/etc/sysctl.conf配置
sysctl -p

创建证书elastic-certificates.p12文件

[root@llody-dev ~]#docker run -itd --name es01 swr.cn-southwest-2.myhuaweicloud.com/llody/elasticsearch:8.15.3  /bin/bash
c5ace1a6f5a6d1fa623cbebf96a75b9c9b6d71e3832d9973fff809848e8ae4ac
[root@llody-dev ~]#docker exec -it es01 /bin/bash
elasticsearch@c5ace1a6f5a6:~$ ./bin/elasticsearch-certutil ca
This tool assists you in the generation of X.509 certificates and certificate
signing requests for use with SSL/TLS in the Elastic stack.

The 'ca' mode generates a new 'certificate authority'
This will create a new X.509 certificate and private key that can be used
to sign certificate when running in 'cert' mode.

Use the 'ca-dn' option if you wish to configure the 'distinguished name'
of the certificate authority

By default the 'ca' mode produces a single PKCS#12 output file which holds:
    * The CA certificate
    * The CA's private key

If you elect to generate PEM format certificates (the -pem option), then the output will
be a zip file containing individual files for the CA certificate and private key

Please enter the desired output file [elastic-stack-ca.p12]: 
Enter password for elastic-stack-ca.p12 : 
elasticsearch@c5ace1a6f5a6:~$ ./bin/elasticsearch-certutil cert --ca elastic-stack-ca.p12 --dns es01 --ip 192.168.1.225  
This tool assists you in the generation of X.509 certificates and certificate
signing requests for use with SSL/TLS in the Elastic stack.

The 'cert' mode generates X.509 certificate and private keys.
    * By default, this generates a single certificate and key for use
       on a single instance.
    * The '-multiple' option will prompt you to enter details for multiple
       instances and will generate a certificate and key for each one
    * The '-in' option allows for the certificate generation to be automated by describing
       the details of each instance in a YAML file

    * An instance is any piece of the Elastic Stack that requires an SSL certificate.
      Depending on your configuration, Elasticsearch, Logstash, Kibana, and Beats
      may all require a certificate and private key.
    * The minimum required value for each instance is a name. This can simply be the
      hostname, which will be used as the Common Name of the certificate. A full
      distinguished name may also be used.
    * A filename value may be required for each instance. This is necessary when the
      name would result in an invalid file or directory name. The name provided here
      is used as the directory name (within the zip) and the prefix for the key and
      certificate files. The filename is required if you are prompted and the name
      is not displayed in the prompt.
    * IP addresses and DNS names are optional. Multiple values can be specified as a
      comma separated string. If no IP addresses or DNS names are provided, you may
      disable hostname verification in your SSL configuration.


    * All certificates generated by this tool will be signed by a certificate authority (CA)
      unless the --self-signed command line option is specified.
      The tool can automatically generate a new CA for you, or you can provide your own with
      the --ca or --ca-cert command line options.


By default the 'cert' mode produces a single PKCS#12 output file which holds:
    * The instance certificate
    * The private key for the instance certificate
    * The CA certificate

If you specify any of the following options:
    * -pem (PEM formatted output)
    * -multiple (generate multiple certificates)
    * -in (generate certificates from an input file)
then the output will be be a zip file containing individual certificate/key files

Enter password for CA (elastic-stack-ca.p12) : 
Please enter the desired output file [elastic-certificates.p12]: 
Enter password for elastic-certificates.p12 : 

Certificates written to /usr/share/elasticsearch/elastic-certificates.p12

This file should be properly secured as it contains the private key for 
your instance.
This file is a self contained file and can be copied and used 'as is'
For each Elastic product that you wish to configure, you should copy
this '.p12' file to the relevant configuration directory
and then follow the SSL configuration instructions in the product guide.

For client applications, you may only need to copy the CA certificate and
configure the client to trust this certificate.
elasticsearch@c5ace1a6f5a6:~$ ls
LICENSE.txt  NOTICE.txt  README.asciidoc  bin  config  data  elastic-certificates.p12  elastic-stack-ca.p12  jdk  lib  logs  modules  plugins
elasticsearch@c5ace1a6f5a6:~$ pwd
/usr/share/elasticsearch
elasticsearch@c5ace1a6f5a6:~$ exit
exit

创建证书时。必须指定节点固定IP，否则证书将不可用。

拷贝加密文件到conf下(分发到所有节点)

[root@llody-dev ~]#docker cp es01:/usr/share/elasticsearch/elastic-stack-ca.p12 /data/es/certs/ca/ca.p12
[root@llody-dev ~]#docker cp es01:/usr/share/elasticsearch/elastic-certificates.p12 /data/es/certs/es01/es01.p12
[root@llody-dev ~]#openssl pkcs12 -in /data/es/certs/ca/ca.p12 -clcerts -nokeys -out /data/es/certs/ca/ca.crt
Enter Import Password:
MAC verified OK
[root@llody-dev ~]#openssl pkcs12 -in /data/es/certs/ca/ca.p12 -nocerts -nodes -out /data/es/certs/ca/ca.key
Enter Import Password:
MAC verified OK
[root@llody-dev ~]#openssl pkcs12 -in /data/es/certs/es01/es01.p12 -clcerts -nokeys -out /data/es/certs/es01/es01.crt
Enter Import Password:
MAC verified OK
[root@llody-dev ~]#openssl pkcs12 -in /data/es/certs/es01/es01.p12 -nocerts -nodes -out /data/es/certs/es01/es01.key
Enter Import Password:
MAC verified OK

密码与elastic密码保持一致即可。

数据目录授权

[root@llody-dev ~]#chown -R 1000:1000 /data/es/*
[root@llody-dev ~]#chmod 600 /data/es/certs/ca/*
[root@llody-dev ~]#chmod 600 /data/es/certs/es01/*

创建网络命名空间

[root@llody-dev ~]#docker network create elk

启动ElasticSearch服务

es01.yaml

# 创建运维目录
mkdir /root/shell
# 创建es01.yaml文件
vim /root/shell/es01.yaml

version: '3'

services:
  es01:
    image: swr.cn-southwest-2.myhuaweicloud.com/llody/elasticsearch:8.15.3
    container_name: es01
    restart: always
    ports:
      - "9200:9200"
      - "9300:9300"
    environment:
      - node.name=es01
      - cluster.name=es-cluster
      - cluster.initial_master_nodes=es01
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
      - xpack.security.enabled=true
      - xpack.security.http.ssl.enabled=true
      - xpack.security.http.ssl.key=certs/es01/es01.key
      - xpack.security.http.ssl.certificate=certs/es01/es01.crt
      - xpack.security.http.ssl.certificate_authorities=certs/ca/ca.crt
      - xpack.security.transport.ssl.enabled=true
      - xpack.security.transport.ssl.verification_mode=certificate
      - xpack.security.transport.ssl.key=certs/es01/es01.key
      - xpack.security.transport.ssl.certificate=certs/es01/es01.crt
      - xpack.security.transport.ssl.certificate_authorities=certs/ca/ca.crt
    user: "1000:1000"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - /data/es/data/:/usr/share/elasticsearch/data
      - /data/es/logs/:/usr/share/elasticsearch/logs
      - /data/es/plugins/:/usr/share/elasticsearch/plugins
      - /data/es/certs:/usr/share/elasticsearch/config/certs
    networks:
      - elk

networks:
  elk:
    external: true
    name: elk

# es01节点启动
docker-compose -p es -f shell/es01.yml up -d

如果是集群，一定要使用固定IP，并在创建证书时进行指定。

设置密码

[root@llody-dev ~]#docker exec -it es01 /bin/bash
elasticsearch@437bd4edc09f:~$  /usr/share/elasticsearch/bin/elasticsearch-setup-passwords interactive -u https://192.168.1.225:9200
******************************************************************************
Note: The 'elasticsearch-setup-passwords' tool has been deprecated. This       command will be removed in a future release.
******************************************************************************

Initiating the setup of passwords for reserved users elastic,apm_system,kibana,kibana_system,logstash_system,beats_system,remote_monitoring_user.
You will be prompted to enter passwords as the process progresses.
Please confirm that you would like to continue [y/N]y


Enter password for [elastic]: 
Reenter password for [elastic]: 
Enter password for [apm_system]: 
Reenter password for [apm_system]: 
Enter password for [kibana_system]: 
Reenter password for [kibana_system]: 
Enter password for [logstash_system]: 
Reenter password for [logstash_system]: 
Enter password for [beats_system]: 
Reenter password for [beats_system]: 
Enter password for [remote_monitoring_user]: 
Reenter password for [remote_monitoring_user]: 
Changed password for user [apm_system]
Changed password for user [kibana_system]
Changed password for user [kibana]
Changed password for user [logstash_system]
Changed password for user [beats_system]
Changed password for user [remote_monitoring_user]
Changed password for user [elastic]
elasticsearch@437bd4edc09f:~$ exit
exit

查看状态

[root@llody-dev ~]#curl -k  -u elastic:llodyi4TMmZD https://192.168.1.225:9200/_cat/nodes?v
ip         heap.percent ram.percent cpu load_1m load_5m load_15m node.role   master name
172.18.0.2           32          56  10    0.22    0.42     0.43 cdfhilmrstw *      es01
[root@llody-dev ~]#curl --cacert /data/es/certs/ca/ca.crt -u elastic:llodyi4TMmZD -X GET "https://192.168.1.225:9200/_cluster/health?pretty"
{
  "cluster_name" : "es-cluster",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 1,
  "number_of_data_nodes" : 1,
  "active_primary_shards" : 1,
  "active_shards" : 1,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 100.0
}

部署Kibana

配置Kibana与ES认证 证书

1. 创建证书目录

   [root@llody-dev ~]#mkdir -p /data/kibana/certs

2. 拷贝ES的ca证书

   [root@llody-dev ~/shell]#cp /data/es/certs/ca/ca.crt /data/kibana/certs/ca.crt

3. 配置权限

   [root@llody-dev ~]#chown -R 1000:1000 /data/kibana
   [root@llody-dev ~]#chmod 600 /data/kibana/certs/*

配置kibana.yml文件

server.host: "0.0.0.0"
server.port: 5601
elasticsearch.requestTimeout: 60000
monitoring.ui.container.elasticsearch.enabled: true
elasticsearch.hosts: ["https://192.168.1.225:9200"]
elasticsearch.username: "kibana_system"
elasticsearch.password: "llodyi4TMmZD"
elasticsearch.ssl.certificateAuthorities: ["/usr/share/kibana/config/certs/ca.crt"]
i18n.locale: "zh-CN"

配置kibana.yaml

version: '3'

services:
  kibana:
    image: swr.cn-southwest-2.myhuaweicloud.com/llody/kibana:8.15.3 
    container_name: kibana
    restart: always
    user: "1000:1000"
    ports:
      - "5601:5601"
    volumes:
      - /data/kibana/certs:/usr/share/kibana/config/certs
      - /data/kibana/config/kibana.yml:/usr/share/kibana/config/kibana.yml
    networks:
      - elk

networks:
  elk:
    external: true
    name: elk

[root@llody-dev ~]#docker-compose -p kibana -f /root/shell/kibana.yaml up -d

查看容器状态

部署filebeat

Docker

创建数据目录

[root@dev-2 ~]# mkdir -p /data/fluentd/{certs,config}
[root@dev-2 ~]# tree /data/filebeat/
/data/filebeat/
├── certs
│   └── ca.crt
└── config
    └── filebeat.yml

2 directories, 2 files

上传ES的ca证书到指定目录下。

配置filebeat.yml

filebeat.inputs:
- type: container
  enabled: true
  stream: all
  format: docker
  paths:
    - '/var/lib/docker/containers/*/*.log'
  multiline:
    # 做多行日志聚合，此处是针对java的异常堆栈信息聚合（所有非年-月-日开头的日志会被聚合到最开始匹配到的那一行展示）
    type: pattern
    pattern: '^\d+.\d+.\d+|^\d+.\w{3}.\d+'
    negate: true
    match: after
  processors:
    - add_docker_metadata:
        host: "unix:///var/run/docker.sock"
    # 去掉不需要的字段
    - drop_fields:
        ignore_missing: true # 字段不存在时不报错
        fields: 
          - agent.type.keyword
          - agent.type
          - agent.name
          - agent.id.keyword
          - agent.id
          - host.name
          - host.name.keyword
    - script:
        lang: javascript
        timeout: 100
        source: > 
          function process(event) { 
              var s = event.Get('container.image.name');
              if (!s) {
                return event;
              }
              var sp = s.lastIndexOf('/');
              sp = sp < 0 ? 0 : sp + 1;
              var ep = s.indexOf(':');
              if (ep < 0) {
                  event.Put('service', s.substring(sp));
              }
              event.Put('service', s.substring(sp, ep));
          }
    - drop_event:
        when: 
          or:
            - contains:
                service: 'filebeat'

output.elasticsearch:
  hosts: ["https://192.168.1.225:9200"]
  username: "elastic"
  password: "llodyi4TMmZD"
  # 传输压缩级别
  compression_level: 4
  ssl.certificate_authorities: ["/usr/share/filebeat/certs/ca.crt"]
  indices:
  # service会作为上报的日志文件的名字参与进去
  - index: "host-log-%{[service]}-%{+yyyy.MM.dd}"
    when.has_fields:
      - service
  - index: "host-log-default-%{+yyyy.MM.dd}"

setup.template.enabled: false
setup.ilm.enabled: false

配置filebeat.yaml启动文件

version: '3'

services:
  filebeat:
    image: swr.cn-southwest-2.myhuaweicloud.com/llody/filebeat:8.15.3
    container_name: filebeat
    restart: always
    user: root
    environment:
      - ELASTICSEARCH_HOSTS=https://192.168.1.225:9200
      - ELASTICSEARCH_USERNAME=elastic
      - ELASTICSEARCH_PASSWORD=llodyi4TMmZD
      - ELASTICSEARCH_SSL_CERTIFICATEAUTHORITIES=/usr/share/filebeat/certs/ca/ca.crt
    volumes:
      - /var/lib/docker/containers:/var/lib/docker/containers
      - /var/run/docker.sock:/var/run/docker.sock
      - /etc/localtime:/etc/localtime:ro
      - /data/filebeat/config/filebeat.yml:/usr/share/filebeat/filebeat.yml
      - /data/filebeat/certs:/usr/share/filebeat/certs
    networks:
      - elk

networks:
  elk:
    external: true
    name: elk

部署

[root@dev-2 ~]# docker-compose -p filebeat -f /root/shell/filebeat.yaml up -d
WARN[0000] /root/shell/filebeat.yaml: `version` is obsolete 
[+] Running 1/1
 ✔ Container filebeat  Started  

K8S

创建filebeat-dm.yaml文件

apiVersion: v1
kind: ConfigMap
metadata:
  name: filebeat-config
  namespace: kube-system
data:
  filebeat.yml: |
    filebeat.inputs:
    - type: container
      enabled: true
      stream: all
      format: docker
      paths:
        - /var/log/containers/*_dev_*.log
      processors:
        - add_kubernetes_metadata:
            in_cluster: true
            matchers:
            - logs_path:
                logs_path: "/var/log/containers/"

    output.elasticsearch:
      hosts: ["https://192.168.1.225:9200"]
      username: "elastic"
      password: "llodyi4TMmZD"
      compression_level: 4
      ssl.certificate_authorities: ["/usr/share/filebeat/certs/ca.crt"]
      indices:
      - index: "k8s-log-%{[kubernetes.namespace]}-%{[kubernetes.pod.name]}-%{+yyyy.MM.dd}"
        when.has_fields:
          - kubernetes.namespace
          - kubernetes.pod.name
      - index: "k8s-log-default-%{+yyyy.MM.dd}"

    setup.template.enabled: false
    setup.ilm.enabled: false
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: filebeat-es-ca
  namespace: kube-system
data:
  ca.crt: |
    Bag Attributes
      friendlyName: ca
      localKeyID: 54 69 6D 65 20 31 37 33 31 34 38 34 39 36 33 35 39 39 
    subject=/CN=Elastic Certificate Tool Autogenerated CA
    issuer=/CN=Elastic Certificate Tool Autogenerated CA
    -----BEGIN CERTIFICATE-----
    MIIDSjCCAjKgAwIBAgIVAPrLZp0pnA1sbAZRSsG0l5E+O9d5MA0GCSqGSIb3DQEB
    CwUAMDQxMjAwBgNVBAMTKUVsYXN0aWMgQ2VydGlmaWNhdGUgVG9vbCBBdXRvZ2Vu
    ZXJhdGVkIENBMB4XDTI0MTExMzA4MDIzMFoXDTI3MTExMzA4MDIzMFowNDEyMDAG
    A1UEAxMpRWxhc3RpYyBDZXJ0aWZpY2F0ZSBUb29sIEF1dG9nZW5lcmF0ZWQgQ0Ew
    ggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDVDUcg4ABSs7M984pxvw1F
    g329j6FOxBk+NtmBWkMEl9aou0qMVcXGCBIp1MxGnhVSuxhDPG1hS3hX5PlfNGJM
    gzvymW6oVf6+SiTqbSTl+psEbSeZyK2nzrTEOfyB+lEZ11+Owo0XwTVSVqKUj5B1
    KaRBbBOo5O99eo97Rv8HkbVVlCD1R4TCJqQXh29c8RZuMFQ8DacqvH6Gwj1Yvntm
    RT/QudQja1vED85MiSFPY8LdOrDHqzEhTY3scqC5gPrc/C7XJSbE+cfkPk56OEEU
    BCSZBiY8DR4b+30SiyZ3Miu4xUU/HRdLn1CtpZ6c8xooJipddInPlloyy3K0F24f
    AgMBAAGjUzBRMB0GA1UdDgQWBBT3xLSl9WS6321QvOUbP1qnzC9j8TAfBgNVHSME
    GDAWgBT3xLSl9WS6321QvOUbP1qnzC9j8TAPBgNVHRMBAf8EBTADAQH/MA0GCSqG
    SIb3DQEBCwUAA4IBAQAfFkd4ocTbZhtb456E9p+eR25HX1CqhOXOUfKTCy48qXCv
    q3uXmCxUJY/4+oeMFtv2DLYUi9cZ3BMzjGpXVGlKPhTGat+aELUgCTvf6Yelnive
    AYktjcDmc6VPrIzIZ9PcH+6BhTykgO6cV0MRgo4R0K3u+QydhokzX3cpiAGHbgHP
    DM2NF2tNYVKsOK0x8XrRVHZa75ZLsClkePtqWmWKcQ5Qh8vt8dDv8GCHPi95Aiit
    IEXNEBfStwNV+xFmQdfbjO5KAilXJdYnbdN/t2xQt9xJBHQ3t+tAWNdXoY6reXui
    /EftuD/YRKSJT0GMtC98HlpHP/gVv5kMXkKwMPqI
    -----END CERTIFICATE-----
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: filebeat
  namespace: kube-system
  labels:
    k8s-app: filebeat
spec:
  selector:
    matchLabels:
      k8s-app: filebeat
  template:
    metadata:
      labels:
        k8s-app: filebeat
    spec:
      serviceAccountName: filebeat
      terminationGracePeriodSeconds: 30
      containers:
      - name: filebeat
        image: swr.cn-southwest-2.myhuaweicloud.com/llody/filebeat:8.15.3
        args: [
          "--strict.perms=false",  # 忽略权限问题
          "-c", "/usr/share/filebeat/filebeat.yml",
          "-e",
        ]
        env:
        - name: NODE_NAME
          valueFrom:
            fieldRef:
              fieldPath: spec.nodeName
        securityContext:
          runAsUser: 0
        resources:
          limits:
            memory: 200Mi
          requests:
            cpu: 100m
            memory: 100Mi
        volumeMounts:
        - name: config
          mountPath: /usr/share/filebeat/filebeat.yml
          subPath: filebeat.yml
        - name: varlibdockercontainers
          mountPath: /var/lib/docker/containers
          readOnly: true
        - name: varlog
          mountPath: /var/log
          readOnly: true
        - name: certs
          mountPath: /usr/share/filebeat/certs
          readOnly: true
      volumes:
      - name: config
        configMap:
          defaultMode: 0600
          name: filebeat-config
      - name: certs
        configMap:
          defaultMode: 0600
          name: filebeat-es-ca
      - name: varlibdockercontainers
        hostPath:
          path: /var/lib/docker/containers
      - name: varlog
        hostPath:
          path: /var/log

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: filebeat
  namespace: kube-system

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: filebeat
subjects:
- kind: ServiceAccount
  name: filebeat
  namespace: kube-system
roleRef:
  kind: ClusterRole
  name: filebeat
  apiGroup: rbac.authorization.k8s.io

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: filebeat
  labels:
    k8s-app: filebeat
rules:
- apiGroups: [""]
  resources:
  - namespaces
  - pods
  verbs:
  - get
  - watch
  - list
    

注意：

1、挂载了filebeat.yml文件和ES的ca证书文件。

2、只收集dev命名空间日志，如要收集所有日志修改paths：/var/log/containers/*.log

3、创建的K8S集群日志索引规则：k8s-log-命名空间-容器名称-时间

部署

[root@node1 ~]#kubectl apply -f es/filebeat-dm.yaml 
configmap/filebeat-config configured
configmap/filebeat-es-ca unchanged
daemonset.apps/filebeat unchanged
serviceaccount/filebeat unchanged
clusterrolebinding.rbac.authorization.k8s.io/filebeat unchanged
clusterrole.rbac.authorization.k8s.io/filebeat unchanged

服务状态

验证日志上传

索引管理

主机Docker索引规则：host-log-容器名称-时间

K8S集群索引规则：k8s-log-命名空间-容器名称-时间

创建视图

Discover查看创建的视图

java报错日志查询示例

总结

EFK可以非常方便的收集容器的日志，可以通过设置部分指标提供告警等功能，后续再出实践，如果有帮助欢迎一键三连。