麒麟v10 arm64 部署 kubesphere 3.4 修改记录

arm64环境，默认安装 kubesphere 3.4 ，需要修改几个地方的镜像，并且会出现日志无法显示

1 fluentbit:v1.9.4 报错

sh 复制代码

<jemalloc>: Unsupported system page size
Error in GnuTLS initialization: ASN1 parser: Element was not found.
<jemalloc>: Unsupported system page size
<jemalloc>: Unsupported system page size
<jemalloc>: Unsupported system page size
malloc: Cannot allocate memory
level=error msg="Fluent bit exited" error="exit status 1"
level=info msg=backoff delay=8s

解决

更换为 fluentbit:v2.0.6

sh 复制代码

kubectl patch fluentbit -n kubesphere-logging-system fluent-bit --type='json' -p='[{"op": "replace", "path": "/spec/image", "value":"10.1.7.220/aicore/arm64/kubesphere/fluent-bit:v2.0.6"}]'`

2 defaultbackend 架构错误

里面写死了 defaultbackend-amd64:1.4

解决

更换为 defaultbackend-arm64:1.4

sh 复制代码

kubectl set image -n kubesphere-controls-system deployment/default-http-backend default-http-backend=defaultbackend-arm64:1.4

3 elasticsearch-curator 架构错误

kubesphere/elasticsearch-curator:v5.7.6 只有amd-x64

解决

更换为 kubesphere/elasticsearch-curator:v5.7.6-arm64

sh 复制代码

kubectl patch job/elasticsearch-logging-curator-elasticsearch-curator-28826940 -n kubesphere-logging-system \
--type='json' -p='[{"op": "replace", "path": "/spec/template/spec/containers/elasticsearch-curator/image", "value":"kubesphere/elasticsearch-curator:v5.7.6-arm64"}]'

4 buysbox:laster 加载不到

目前 docker-io 访问不了，但是 opensearch 的 initContainers 使用 buysbox:laster，但 imagePullPolicy:always，需要修改

解决

sh 复制代码

kubectl patch StatefulSet/opensearch-cluster-data -n kubesphere-logging-system --type='json' -p='[{"op": "replace", "path": "/spec/template/spec/initContainers/0/imagePullPolicy", "value":"IfNotPresent"}]'
kubectl patch StatefulSet/opensearch-cluster-master -n kubesphere-logging-system --type='json' -p='[{"op": "replace", "path": "/spec/template/spec/initContainers/0/imagePullPolicy", "value":"IfNotPresent"}]'

5 日志无法显示

根据这个帖子解决了，=

https://ask.kubesphere.io/forum/d/23443-kubesphere341-jie-ru-wai-bu-elasticsearchhou-eszhong-ke-yi-shou-ji-dao-ri-zhi-ye-mian-cha-bu-dao-eszhong-ke-yi-jian-suo/9

原因：

经过检查发现是配置字典 kubesphere-config 中的 indexPrefix 和 output 中的 logstashPrefix 配置不一致导致的

解决方案：

如果在kubesphere 页面进行容器日志查询、资源事件查询、审计日志查询需要配置kubesphere-config 中的 indexPrefix 和 output 中的 logstashPrefix一致

kubesphere-config中的配置如下：

output 中的 logstashPrefix 配置如下：

fluent-bit 未启动

在反复安装卸载后，有时 fluent-bit 会没出现

解决

创建 fluent-bit.yaml

yaml 复制代码

apiVersion: logging.kubesphere.io/v1alpha2
kind: FluentBit
metadata:
  name: fluent-bit
  namespace: kubesphere-logging-system
  labels:
    app.kubernetes.io/name: fluent-bit
spec:
  image: kubesphere/fluent-bit:v2.0.6
  positionDB:
    emptyDir: {}
  resources:
    limits:
      cpu: 500m
      memory: 200Mi
    requests:
      cpu: 10m
      memory: 25Mi
  fluentBitConfigName: fluent-bit-config
  tolerations:
    - operator: Exists
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: node-role.kubernetes.io/edge
            operator: DoesNotExist

执行
kubectl create -f fluent-bit.yaml

整理安装脚本

下载源码

https://github.com/kubesphere/ks-installer

修改文件

roles\download\defaults\main.yml

diff 复制代码

#ks-logger
docker_elasticsearch_curator_repo: "{{ base_repo }}{{ namespace_override | default('kubesphere') }}/elasticsearch-curator"
- docker_elasticsearch_curator_tag: v5.7.6-amd64
+ docker_elasticsearch_curator_tag: v5.7.6-arm64

#ingress:
- defaultbackend_repo: "{{ base_repo }}{{ namespace_override | default('mirrorgooglecontainers') }}/defaultbackend-arm64"
+ defaultbackend_repo: "{{ base_repo }}{{ namespace_override | default('mirrorgooglecontainers') }}/defaultbackend-arm64"
defaultbackend_tag: 1.4

roles\common\templates\custom-values-opensearch-data.yaml.j2

diff 复制代码

global:
-  dockerRegistry: ""
+  dockerRegistry: "{{ base_repo }}"

roles\common\templates\custom-values-opensearch-master.yaml.j2

diff 复制代码

global:
-  dockerRegistry: ""
+  dockerRegistry: "{{ base_repo }}"

roles\ks-core\config\tasks\main.yaml

diff 复制代码

- block:
  - name: KubeSphere | Getting es index prefix
    shell: >
-      {{ bin_dir }}/kubectl get cm -n kubesphere-system kubesphere-config -o jsonpath='{.data.kubesphere\.yaml}' | grep "logging:" -A 2 | grep "indexPrefix" | awk '{print $2}'
+      {{ bin_dir }}/kubectl get cm -n kubesphere-system kubesphere-config -o jsonpath='{.data.kubesphere\.yaml}' | grep "logging:" -A 6 | grep "indexPrefix" | awk '{print $2}'
    register: es_indexPrefix

roles\ks-core\config\templates\kubesphere-config.yaml.j2

diff 复制代码

{% if esIndexPrefix is defined %}
      indexPrefix: {{ esIndexPrefix }}
- {% elif common.opensearch.opensearchPrefix is defined %}
+ {% elif common.opensearch.opensearchPrefix is defined and common.opensearch.enabled == true %}
      indexPrefix: ks-{{ common.opensearch.opensearchPrefix }}-logging
{% else %}
-      indexPrefix: ks-{{ common.es.elkPrefix }}-log
+      indexPrefix: ks-{{ common.es.elkPrefix }}-logging
{% endif %}

- {% if common.opensearch.opensearchPrefix is defined  %}
+ {% if common.opensearch.opensearchPrefix is defined and common.opensearch.enabled == true  %}
      indexPrefix: ks-{{ common.opensearch.opensearchPrefix }}-events
{% else %}
      indexPrefix: ks-{{ common.es.elkPrefix }}-events
{% endif %}

roles\ks-core\ks-core\files\ks-core\values.yaml

diff 复制代码

-  defaultbackend_repo: "mirrorgooglecontainers/defaultbackend-amd64"
+  defaultbackend_repo: "mirrorgooglecontainers/defaultbackend-arm64"

roles\ks-logging\templates\custom-output-elasticsearch-logging.yaml.j2

diff 复制代码

-    logstashPrefix: "ks-{{ common.es.elkPrefix }}-log"
+    logstashPrefix: "ks-{{ common.es.elkPrefix }}-logging"

Dockerfile-arm64

dockerfile 复制代码

FROM kubesphere/ks-installer:v3.4.1

COPY roles/download/defaults/main.yml                                       /kubesphere/installer/roles/download/defaults/main.yml
COPY roles/common/templates/custom-values-opensearch-data.yaml.j2           /kubesphere/installer/roles/common/templates/
COPY roles/common/templates/custom-values-opensearch-master.yaml.j2         /kubesphere/installer/roles/common/templates/
COPY roles/ks-core/ks-core/files/ks-core/values.yaml                        /kubesphere/installer/roles/ks-core/ks-core/files/ks-core/values.yaml
COPY roles/ks-core/config/templates/kubesphere-config.yaml.j2               /kubesphere/installer/roles/ks-core/config/templates/kubesphere-config.yaml.j2
COPY roles/ks-core/config/tasks/main.yaml                                   /kubesphere/installer/roles/ks-core/config/tasks/main.yaml
COPY roles/ks-logging/templates/custom-output-elasticsearch-logging.yaml.j2 /kubesphere/installer/roles/ks-logging/templates/custom-output-elasticsearch-logging.yaml.j2

构建

sh 复制代码

docker build -t kubesphere/ks-installer:v3.4.1-arm64 -f Dockerfile-arm64 .

这样下次安装就可以全部正常了