Centos7离线部署K8s集群V1.28.8

本篇文章主要介绍centos7使用kubekey离线部署k8s-v1.28.8

机器ip	角色
10.100.100.1	外网机，用于下载包
10.100.100.10	master-1
10.100.100.11	node-1

一、获取部署工具和部署包

1.1、在外网机执行：

bash 复制代码

curl -sfL https://get-kk.kubesphere.io | sh -

或手动下载：

bash 复制代码

https://github.com/kubesphere/kubekey/releases/download/v4.0.5/kubekey-v4.0.5-linux-amd64.tar.gz

1.2、解压kubekey-v4.0.5-linux-amd64.tar.gz 后获得 kk 可执行程序

查看kk支持的k8s版本列表

bash 复制代码

./kk version --show-supported-k8s

1.3、生成 Manifest 文件

bash 复制代码

./kk create manifest --with-kubernetes v1.28.8 --arch amd64

yaml 复制代码

apiVersion: kubekey.kubesphere.io/v1alpha2
kind: Manifest
metadata:
  name: sample
spec:
  arches:
  - amd64
  operatingSystems:
  - arch: amd64
    type: linux
    id: contos
    version: "7"
    osImage: CentOS Linux 7 (Core)
    repository:
      iso:
        localPath: 
        url:
  kubernetesDistributions:
  - type: kubernetes
    version: v1.28.8
  components:
    helm: 
      version: v3.14.3
    cni: 
      version: v1.2.0
    etcd: 
      version: v3.5.13
    containerRuntimes:
    - type: docker
      version: 24.0.9
    - type: containerd
      version: 1.7.13
    calicoctl:
      version: v3.27.3
    crictl: 
      version: v1.29.0
    
  images:
  - docker.io/kubesphere/pause:3.9
  - docker.io/kubesphere/kube-apiserver:v1.28.8
  - docker.io/kubesphere/kube-controller-manager:v1.28.8
  - docker.io/kubesphere/kube-scheduler:v1.28.8
  - docker.io/kubesphere/kube-proxy:v1.28.8
  - docker.io/coredns/coredns:1.9.3
  - docker.io/kubesphere/k8s-dns-node-cache:1.22.20
  - docker.io/calico/kube-controllers:v3.27.3
  - docker.io/calico/cni:v3.27.3
  - docker.io/calico/node:v3.27.3
  - docker.io/calico/pod2daemon-flexvol:v3.27.3
  - docker.io/calico/typha:v3.27.3
  - docker.io/flannel/flannel:v0.21.3
  - docker.io/flannel/flannel-cni-plugin:v1.1.2
  - docker.io/cilium/cilium:v1.15.3
  - docker.io/cilium/operator-generic:v1.15.3
  - docker.io/hybridnetdev/hybridnet:v0.8.6
  - docker.io/kubeovn/kube-ovn:v1.10.10
  - docker.io/kubesphere/multus-cni:v3.8
  - docker.io/openebs/provisioner-localpv:3.3.0
  - docker.io/openebs/linux-utils:3.3.0
  - docker.io/library/haproxy:2.9.6-alpine
  - docker.io/plndr/kube-vip:v0.7.2
  - docker.io/kubesphere/kata-deploy:stable
  - docker.io/kubesphere/node-feature-discovery:v0.10.0
  registry:
    auths: {}

1.4、根据清单文件生成离线包：

bash 复制代码

export KKZONE=cn
./kk artifact export -m manifest-sample.yaml -o artifact.tar.gz

1.5、生成部署配置文件：

bash 复制代码

./kk create config

yaml 复制代码

apiVersion: kubekey.kubesphere.io/v1alpha2
kind: Cluster
metadata:
  name: sample
spec:
  hosts:
  - {name: master-1, address: 10.100.100.10, internalAddress: 10.100.100.10, user: root, password: "Qcloud@123"}
  - {name: node-1, address: 10.100.100.11, internalAddress: 10.100.100.11, user: root, password: "Qcloud@123"}
  roleGroups:
    etcd:
    - master-1
    control-plane: 
    - master-1
    worker:
    - node-1
  controlPlaneEndpoint:
    ## Internal loadbalancer for apiservers 
    # internalLoadbalancer: haproxy

    domain: lb.kubesphere.local
    address: ""
    port: 6443
  kubernetes:
    version: v1.28.8
    clusterName: cluster.local
    autoRenewCerts: true
    containerManager: docker
  etcd:
    type: kubekey
  network:
    plugin: calico
    kubePodsCIDR: 10.233.64.0/18
    kubeServiceCIDR: 10.233.0.0/18
    ## multus support. https://github.com/k8snetworkplumbingwg/multus-cni
    multusCNI:
      enabled: false
  registry:
    privateRegistry: ""
    namespaceOverride: ""
    registryMirrors: []
    insecureRegistries: []
  addons: []

1.6、将整个kk的目录和离线包以及清单文件全部拷贝到要部署k8s集群的所有机器上

二、执行部署脚本

在master-1机器上进入kk目录执行：

bash 复制代码

./kk create cluster -f config-sample.yaml -a artifact.tar.gz

待脚本执行完成，k8s集群就部署好了

三、问题处理

3.1、部署完机器发现calico插件报错导致所有节点都notready

问题：

bash 复制代码

Error:mountvolume.setup failed for volume "bpffs":hostPath type check failed /sys/fs/bpf is not a directory

**原因：**这个问题的核心在于 /sys/fs/bpf 目录不存在，并且你手动创建时遇到了权限拒绝。根据你的日志和环境，这通常是由于内核版本过低，不支持 BPF 文件系统导致的。

解决：升级内核https://blog.csdn.net/weixin_44418421/article/details/161724962?spm=1001.2014.3001.5502最好升级为ml版本lt版本一般都不开启BPF

3.2、containerd无法访问自己的私有仓库

问题：

bash 复制代码

tls failed to verfify certifacates x509

原因： containerd未配置私有仓库跳过tls和用户密码

解决：

bash 复制代码

    [plugins."io.containerd.grpc.v1.cri".registry]
       # 新加
      [plugins."io.containerd.grpc.v1.cri".registry.configs]
       # 新加
        [plugins."io.containerd.grpc.v1.cri".registry.configs."harbor.k8s.test.com".tls]
          insecure_skip_verify = true
        # 新加
        [plugins."io.containerd.grpc.v1.cri".registry.configs."harbor.k8s.test.com".auth]
          username = "admin"
          password = "test"
        # 新加
        [plugins."io.containerd.grpc.v1.cri".registry.mirrors."harbor.k8s.test.com"]
          endpoint = ["https://harbor.k8s.test.com"]

3.3、manifest-sample.yaml中将镜像地址替换为了国内仓库，部署k8s集群时报错拉不到镜像

原因： kk创建集群时默认从国外的镜像网站拉取，如果将manifest-sample.yaml中配置为了国内源，即使将镜像导出到artifact.tar.gz中，在部署时仍会去拉外网的

解决： 手动将镜像从国内拉下来，重新tag为国外的仓库，然后load到目标k8s机器上，再重新执行kk create cluster -f config-sample.yaml -a artifact.tar.gz

3.4、执行一次部署失败后，再次执行发现报错连不上etcd

原因： etcd信息会在每次部署时重新生成

解决： 删除不成功的集群后重新部署

bash 复制代码

./kk delete cluster -f config-sample.yaml
./kk create cluster -f config-sample.yaml -a artifact.tar.gz