【二进制部署k8s-1.29.4】二、证书及配置文件启动脚步的准备

文章目录

• • • • 简介
    • 一.准备证书相关的配置文件
    • • 1.1.ca-config.json
      • 1.2.ca-csr.json
      • 1.3.etcd-csr.json
      • 1.4.kube-apiserver-csr.json
      • 1.5.kube-controller-manager-csr.json
      • 1.6.kube-scheduler-csr.json
      • 1.7.admin-csr.json
      • 1.8.proxy-client-csr.json
    • 二.安装客户端相关软件及命令
    • 三.生成证书
    • 四.准备k8s配置文件
    • • 3.1.etcd.conf
      • 3.2.kube-apiserver.conf
      • 3.3.kube-controller-manager.conf
      • 3.4.kube-scheduler.conf
      • 3.5.kubelet.yaml
      • 3.6.containerd配置文件
    • 五.生成kubeconfig配置文件
    • 六.准备启动脚本
    • • 6.1.etcd.service
      • 6.2.kube-apiserver.service
      • 6.3.kube-controller-manager.service
      • 6.4.kube-scheduler.service
      • 6.5.kubelet.service
      • 6.6.containerd启动脚本
    • 七.总结

简介

本章节主要准备二进制安装k8s的过程中所使用到的证书配置文件，怎样生成证书，以及etcd、master端组件、worker端组件所用到的配置文件和启动脚本，同时利用脚本生成证书、和生成kubecofig配置文件。

一.准备证书相关的配置文件

1.1.ca-config.json

定义ca证书的过期时间

{
    "signing": {
      "default": {
        "expiry": "175200h"
      },
      "profiles": {
        "kubernetes": {
           "expiry": "175200h",
           "usages": [
              "signing",
              "key encipherment",
              "server auth",
              "client auth"
          ]
        }
      }
    }
  }

1.2.ca-csr.json

定义ca证书的加密算法、地域及组织单位

{
    "CN": "kubernetes",
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "L": "Guangzhou",
            "ST": "Guangdong",
            "O": "k8s",
            "OU": "System"
        }
    ]
}

1.3.etcd-csr.json

定义etcd证书中的域名、IP、加密算法及组织单位,配置中的三个IP为安装etcd的IP，现在是将etcd安装在master的三个IP上，所以配置的是master的IP

{
    "CN": "etcd",
    "hosts": [
        "10.16.120.81",
        "10.16.120.82",
        "10.16.120.83",
        "127.0.0.1"
    ],
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "L": "Guangzhou",
            "ST": "Guangdong"
        }
    ]
}

1.4.kube-apiserver-csr.json

定义api-server证书中的域名、IP、加密算法及组织单位，配置中的IP主要是master的IP，以及配置api-server的vip，或调用api-server的域名

{
  "CN": "kubernetes",
  "hosts": [
    "127.0.0.1",
    "10.16.120.80",
    "10.16.120.81",
    "10.16.120.82",
    "10.16.120.83",
    "10.1.0.1",
    "yt-pcauto-k8s.pc.com.cn",
    "kubernetes",
    "kubernetes.default",
    "kubernetes.default.svc",
    "kubernetes.default.svc.cluster",
    "kubernetes.default.svc.cluster.local"
  ],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "L": "Guangzhou",
      "ST": "Guangdong",
      "O": "k8s",
      "OU": "system"
    }
  ]
}

1.5.kube-controller-manager-csr.json

定义kube-controller-manager 证书中的api证书地址、节点IP、加密算法及组织单位，配置中的IP是kube-apiserver的vip，域名或127.0.0.1，主要是controller-manager一般都是和apiserver安装在同样的机器上

{
  "CN": "system:kube-controller-manager",
  "hosts": [
    "127.0.0.1",
    "10.16.120.80",
    "yt-pcauto-k8s.pc.com.cn"
  ],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "Guangdong",
      "L": "Guangzhou",
      "O": "system:kube-controller-manager",
      "OU": "system"
    }
  ]
}

1.6.kube-scheduler-csr.json

定义kube-scheduler证书中的api证书地址、节点IP、加密算法及组织单位，配置中的IP是kube-apiserver的vip，域名或127.0.0.1，主要是controller-manager一般都是和apiserver安装在同样的机器上

{
  "CN": "system:kube-scheduler",
  "hosts": [
    "127.0.0.1",
    "10.16.120.80",
    "yt-pcauto-k8s.pc.com.cn"
  ],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "Guangdong",
      "L": "Guangzhou",
      "O": "system:kube-scheduler",
      "OU": "system"
    }
  ]
}

1.7.admin-csr.json

该配置是用于生成k8s管理客户端kubectl所需的kubeconfig时需要公钥和私钥所必须的证书配置文件

{
  "CN": "admin",
  "hosts": [],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "Guangdong",
      "L": "Guangzhou",
      "O": "system:masters",             
      "OU": "system"
    }
  ]
}

1.8.proxy-client-csr.json

kube-apiserver 的另一种访问方式就是使用 kubectl proxy 来代理访问, 而该证书就是用来支持SSL代理访问的. 在该种访问模式下, 我们是以http的方式发起请求到代理服务的, 此时, 代理服务会将该请求发送给 kube-apiserver, 在此之前, 代理会将发送给 kube-apiserver 的请求头里加入证书信息

{
    "CN": "aggregator",
    "hosts": [],
    "key": {
      "algo": "rsa",
      "size": 2048
    },
    "names": [
      {
        "C": "CN",
        "ST": "Guangdong",
        "L": "Guangzhou",
        "O": "system:masters",
        "OU": "System"
      }
    ]
  }

二.安装客户端相关软件及命令

该步骤主要是将部署过程中需要用的一些命令先进行安装，主要就是将以下二进制可执行命令拷贝到/usr/bin目录，该部分软件可以在其中一台master机器上进行安装，也可以在独立的机器上进行安装。

软件	用途
cfssl，cfssl-certinfo，cfssljson	用于生成安装所需的证书
cilium	用于查看cilium的安装状及卸载cilium的客户端
helm	用于安装charts的客户端，例如安装cilium,安装credn，安装ingress等
kubectl,kubectl-convert	k8s客户端软件，kubectl是管理k8s必需的的客户端软件

三.生成证书

将第一步所有的的配置文件放在csr-conf这样一个目录下，然后执行以下脚本生成证书

#!/bin/sh

etcd_cert_dir="install_etcd"  #存放etcd证书的目录
master_cert_dir="install_master/cert" #存放安装master所需证书的目录

[ -d $master_cert_dir ] || mkdir -p $master_cert_dir
[ -d $etcd_cert_dir ] || mkdir -p $etcd_cert_dir
[ -d client ] || mkdir -p client #client目录用于存放生成kubectl命令的配置及相关证书
[ -d ca ] || mkdir -p ca    #存放ca证书及私钥

echo "create ca.pem ca-key.pem======="
cfssl gencert -initca csr-conf/ca-csr.json | cfssljson -bare ca -
mv ca.pem ca-key.pem ca/
rm ca.csr

echo "create etcd.pem etcd-key.pem======="
cfssl gencert -ca=ca/ca.pem -ca-key=ca/ca-key.pem -config=csr-conf/ca-config.json -profile=kubernetes csr-conf/etcd-csr.json | cfssljson -bare $etcd_cert_dir/etcd
rm -f $etcd_cert_dir/etcd.csr

echo "create kube-apiserver.pem kube-apiserver-key.pem======="
cfssl gencert -ca=ca/ca.pem -ca-key=ca/ca-key.pem -config=csr-conf/ca-config.json -profile=kubernetes csr-conf/kube-apiserver-csr.json | cfssljson -bare $master_cert_dir/kube-apiserver
rm -f $master_cert_dir/kube-apiserver.csr

echo "create kube-scheduler.pem kube-scheduler-key.pem======="
cfssl gencert -ca=ca/ca.pem -ca-key=ca/ca-key.pem -config=csr-conf/ca-config.json -profile=kubernetes csr-conf/kube-scheduler-csr.json | cfssljson -bare $master_cert_dir/kube-scheduler
rm -f $master_cert_dir/kube-scheduler.csr

echo "create kube-controller-manager.pem kube-controller-manager-key.pem======="
cfssl gencert -ca=ca/ca.pem -ca-key=ca/ca-key.pem -config=csr-conf/ca-config.json -profile=kubernetes csr-conf/kube-controller-manager-csr.json | cfssljson -bare $master_cert_dir/kube-controller-manager
rm -f $master_cert_dir/kube-controller-manager.csr

echo "create proxy-client.pem proxy-client-key.pem======="
cfssl gencert -ca=ca/ca.pem -ca-key=ca/ca-key.pem -config=csr-conf/ca-config.json -profile=kubernetes csr-conf/proxy-client-csr.json  | cfssljson -bare $master_cert_dir/proxy-client
rm -f $master_cert_dir/proxy-client.csr

echo "create admin.pem admin-key.pem======="
cfssl gencert -ca=ca/ca.pem -ca-key=ca/ca-key.pem -config=csr-conf/ca-config.json -profile=kubernetes csr-conf/admin-csr.json | cfssljson -bare client/admin
rm -fv client/admin.csr

四.准备k8s配置文件

3.1.etcd.conf

10.16.120.81 的配置，每台机不一样

#[Member]
ETCD_NAME="etcd01"
ETCD_DATA_DIR="/opt/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="https://10.16.120.81:2380"
ETCD_LISTEN_CLIENT_URLS="https://10.16.120.81:2379,http://127.0.0.1:2379"

#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://10.16.120.81:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://10.16.120.81:2379"
ETCD_INITIAL_CLUSTER="etcd01=https://10.16.120.81:2380,etcd02=https://10.16.120.82:2380,etcd03=https://10.16.120.83:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"
ETCD_LISTEN_METRICS_URLS="http://0.0.0.0:2381"

10.16.120.82 的配置，每台机不一样

#[Member]
ETCD_NAME="etcd02"
ETCD_DATA_DIR="/opt/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="https://10.16.120.82:2380"
ETCD_LISTEN_CLIENT_URLS="https://10.16.120.82:2379,http://127.0.0.1:2379"

#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://10.16.120.82:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://10.16.120.82:2379"
ETCD_INITIAL_CLUSTER="etcd01=https://10.16.120.81:2380,etcd02=https://10.16.120.82:2380,etcd03=https://10.16.120.83:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"
ETCD_LISTEN_METRICS_URLS="http://0.0.0.0:2381"

10.16.120.83 的配置，每台机不一样

#[Member]
ETCD_NAME="etcd03"
ETCD_DATA_DIR="/opt/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="https://10.16.120.83:2380"
ETCD_LISTEN_CLIENT_URLS="https://10.16.120.83:2379,http://127.0.0.1:2379"

#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://10.16.120.83:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://10.16.120.83:2379"
ETCD_INITIAL_CLUSTER="etcd01=https://10.16.120.81:2380,etcd02=https://10.16.120.82:2380,etcd03=https://10.16.120.83:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"
ETCD_LISTEN_METRICS_URLS="http://0.0.0.0:2381"

3.2.kube-apiserver.conf

注意配置中的文件、证书路径。需要修改的地方主要就是etcd的IP,配置中的pem证书文件是在"三.生成证书" 时生成的，其中的token.csv 会在"五.准备kubeconfig配置文件"中生成

KUBE_APISERVER_OPTS="--enable-admission-plugins=NamespaceLifecycle,NodeRestriction,LimitRanger,ServiceAccount,DefaultStorageClass,ResourceQuota \
  --anonymous-auth=false \
  --secure-port=6443 \
  --authorization-mode=Node,RBAC \
  --runtime-config=api/all=true \
  --enable-bootstrap-token-auth \
  --service-cluster-ip-range=10.1.0.0/16 \
  --token-auth-file=/opt/kubernetes/conf/token.csv \
  --service-node-port-range=30000-50000 \
  --tls-cert-file=/opt/kubernetes/ssl/kube-apiserver.pem \
  --tls-private-key-file=/opt/kubernetes/ssl/kube-apiserver-key.pem \
  --client-ca-file=/opt/kubernetes/ssl/ca.pem \
  --kubelet-client-certificate=/opt/kubernetes/ssl/kube-apiserver.pem \
  --kubelet-client-key=/opt/kubernetes/ssl/kube-apiserver-key.pem \
  --kubelet-preferred-address-types=InternalIP,Hostname,InternalDNS,ExternalDNS \
  --service-account-issuer=https://kubernetes.default.svc \
  --service-account-key-file=/opt/kubernetes/ssl/ca-key.pem \
  --service-account-signing-key-file=/opt/kubernetes/ssl/ca-key.pem \
  --etcd-cafile=/opt/etcd/ssl/ca.pem \
  --etcd-certfile=/opt/etcd/ssl/etcd.pem \
  --etcd-keyfile=/opt/etcd/ssl/etcd-key.pem \
  --etcd-servers=https://10.16.120.81:2379,https://10.16.120.82:2379,https://10.16.120.83:2379 \
  --allow-privileged=true \
  --audit-log-maxage=5 \
  --audit-log-maxbackup=3 \
  --audit-log-maxsize=100 \
  --audit-log-path=/opt/kubernetes/logs/kube-apiserver-audit.log \
  --requestheader-allowed-names=aggregator \
  --requestheader-group-headers=X-Remote-Group \
  --requestheader-username-headers=X-Remote-User \
  --requestheader-extra-headers-prefix=X-Remote-Extra- \
  --requestheader-client-ca-file=/opt/kubernetes/ssl/ca.pem \
  --proxy-client-cert-file=/opt/kubernetes/ssl/proxy-client.pem \
  --proxy-client-key-file=/opt/kubernetes/ssl/proxy-client-key.pem \
  --v=4"

3.3.kube-controller-manager.conf

注意配置中的文件、证书路径，以及service和pod的网段，kubeconfig会在"五.准备kubeconfig配置文件"中生成

KUBE_CONTROLLER_MANAGER_OPTS="--v=2 \
  --kubeconfig=/opt/kubernetes/conf/kube-controller-manager.kubeconfig \
  --horizontal-pod-autoscaler-sync-period=10s \
  --service-cluster-ip-range=10.1.0.0/16 \
  --cluster-signing-cert-file=/opt/kubernetes/ssl/ca.pem \
  --cluster-signing-key-file=/opt/kubernetes/ssl/ca-key.pem \
  --allocate-node-cidrs=true \
  --cluster-cidr=10.2.0.0/16 \
  --cluster-signing-duration=175200h \
  --root-ca-file=/opt/kubernetes/ssl/ca.pem \
  --service-account-private-key-file=/opt/kubernetes/ssl/ca-key.pem \
  --leader-elect=true \
  --feature-gates=RotateKubeletServerCertificate=true \
  --controllers=*,bootstrapsigner,tokencleaner \
  --tls-cert-file=/opt/kubernetes/ssl/kube-controller-manager.pem \
  --tls-private-key-file=/opt/kubernetes/ssl/kube-controller-manager-key.pem \
  --use-service-account-credentials=true"

3.4.kube-scheduler.conf

注意配置中的文件路径，kueconfig会在"五.准备kubeconfig配置文件"中生成

KUBE_SCHEDULER_OPTS="--kubeconfig=/opt/kubernetes/conf/kube-scheduler.kubeconfig \
--leader-elect=true \
--v=2"

3.5.kubelet.yaml

其中10.1.0.2是安装conredns的IP，提前定义好次IP。/opt/kubernetes/ssl/ca.pem 为ca的证书路径，/run/systemd/resolve/resolv.conf为系统的resolved的dns配置路径，不配置此项会导致读取/etc/resolv.conf,而/etc/resolv.conf是/run/systemd/resolve/stub-resolv.conf的软连接，里面配置了本地缓存dns,127.0.0.1:53,会和k8s导致dns冲突

kind: KubeletConfiguration
apiVersion: kubelet.config.k8s.io/v1beta1
address: 0.0.0.0
port: 10250
readOnlyPort: 10255
cgroupDriver: systemd 
clusterDNS:
- 10.1.0.2
clusterDomain: cluster.local 
failSwapOn: false
authentication:
  anonymous:
    enabled: false
  webhook:
    cacheTTL: 2m0s
    enabled: true
  x509:
    clientCAFile: /opt/kubernetes/ssl/ca.pem 
authorization:
  mode: Webhook
  webhook:
    cacheAuthorizedTTL: 5m0s
    cacheUnauthorizedTTL: 30s
evictionHard:
  imagefs.available: 15%
  memory.available: 100Mi
  nodefs.available: 10%
  nodefs.inodesFree: 5%
maxOpenFiles: 2048000
maxPods: 200
resolvConf: /run/systemd/resolve/resolv.conf

3.6.containerd配置文件

containerd的配置文件，需要在worker上安装好containerd时,然后再执行containerd命令导出默认配置，并修改里面的镜像地址，也可以解压containerd的安装包，拷贝containerd的执行文件出来执行导出配置文件。

containerd config default | sudo tee /etc/containerd/config.toml
sed -i 's#SystemdCgroup.*#SystemdCgroup = true#' /etc/containerd/config.toml
sed -i 's#sandbox_image.*#sandbox_image = "registry.aliyuncs.com/google_containers/pause:3.8"#' /etc/containerd/config.toml

五.生成kubeconfig配置文件

kubeconfig的配置文件在安装kube-control-manger,kube-schedul,kubelet,以及配置kubectl客户端时都需要用到。该脚本中所使用到的路径与"三.生成证书"中所使用的的路径一致，如果路径有变动，需要两个脚本都修改一下存放路径。

#!/bin/bash

ca_dir="ca"  #存放ca证书的路径，与第二步生成证书时的路径一致
token_dir="install_master" #存放token.csv的路径
CONFIG_DIR="install_master/kubeconfig" #存放master端使用到kubeconfig的保存路径
worker_dir="install_worker/config"   #存放worker端使用到kubeconfig的保存路径
master_cert_dir="install_master/cert"   #存放maser端使用到的证书的路径，与第二步生成证书时的路径一致
client_dir="client" ##存放client生成的kubecofig以及client端的证书，与第二步生成证书时的路径一致

KUBE_APISERVER="https://yt-pcauto-k8s.pc.com.cn:6443"   #apiserver的地址

[ -d $worker_dir ] || mkdir -p $worker_dir


echo "create token ====="
cat > $token_dir/token.csv << EOF
$(head -c 16 /dev/urandom | od -An -t x | tr -d ' '),kubelet-bootstrap,10001,"system:bootstrappers"
EOF


echo "create kube-controller-manager.kubeconfig ====="
kubectl config set-cluster kubernetes \
        --certificate-authority=$ca_dir/ca.pem \
        --embed-certs=true \
        --server=${KUBE_APISERVER} \
        --kubeconfig=$CONFIG_DIR/kube-controller-manager.kubeconfig

kubectl config set-credentials system:kube-controller-manager \
        --client-certificate=$master_cert_dir/kube-controller-manager.pem \
        --client-key=$master_cert_dir/kube-controller-manager-key.pem \
        --embed-certs=true \
        --kubeconfig=$CONFIG_DIR/kube-controller-manager.kubeconfig

kubectl config set-context system:kube-controller-manager \
        --cluster=kubernetes \
        --user=system:kube-controller-manager \
        --kubeconfig=$CONFIG_DIR/kube-controller-manager.kubeconfig

kubectl config use-context system:kube-controller-manager --kubeconfig=$CONFIG_DIR/kube-controller-manager.kubeconfig


echo "create kube-scheduler.kubeconfig ====="
kubectl config set-cluster kubernetes \
        --certificate-authority=$ca_dir/ca.pem \
        --embed-certs=true \
        --server=${KUBE_APISERVER} \
        --kubeconfig=$CONFIG_DIR/kube-scheduler.kubeconfig

kubectl config set-credentials system:kube-scheduler \
        --client-certificate=$master_cert_dir/kube-scheduler.pem \
        --client-key=$master_cert_dir/kube-scheduler-key.pem \
        --embed-certs=true \
        --kubeconfig=$CONFIG_DIR/kube-scheduler.kubeconfig

kubectl config set-context system:kube-scheduler \
        --cluster=kubernetes \
        --user=system:kube-scheduler \
        --kubeconfig=$CONFIG_DIR/kube-scheduler.kubeconfig

kubectl config use-context system:kube-scheduler --kubeconfig=$CONFIG_DIR/kube-scheduler.kubeconfig


echo "create kubelet-bootstrap.kubeconfig ====="
TOKEN=$(awk -F "," '{print $1}' $token_dir/token.csv)
kubectl config set-cluster kubernetes \
          --certificate-authority=$ca_dir/ca.pem \
          --embed-certs=true \
          --server=${KUBE_APISERVER} \
          --kubeconfig=$worker_dir/kubelet-bootstrap.kubeconfig

kubectl config set-credentials kubelet-bootstrap \
          --token=${TOKEN} \
          --kubeconfig=$worker_dir/kubelet-bootstrap.kubeconfig

kubectl config set-context default \
          --cluster=kubernetes \
          --user=kubelet-bootstrap \
          --kubeconfig=$worker_dir/kubelet-bootstrap.kubeconfig

kubectl config use-context default --kubeconfig=$worker_dir/kubelet-bootstrap.kubeconfig


echo "create client kube.config ====="
kubectl config set-cluster kubernetes \
        --certificate-authority=$ca_dir/ca.pem \
        --embed-certs=true \
        --server=${KUBE_APISERVER} \
        --kubeconfig=$client_dir/kube.kubeconfig

kubectl config set-credentials admin \
        --client-certificate=$client_dir/admin.pem \
        --client-key=$client_dir/admin-key.pem \
        --embed-certs=true \
        --kubeconfig=$client_dir/kube.kubeconfig

kubectl config set-context kubernetes \
        --cluster=kubernetes \
        --user=admin \
        --kubeconfig=$client_dir/kube.kubeconfig

kubectl config use-context kubernetes --kubeconfig=$client_dir/kube.kubeconfig

六.准备启动脚本

6.1.etcd.service

etcd的启动脚本

[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target

[Service]
Type=notify
EnvironmentFile=-/opt/etcd/conf/etcd.conf
WorkingDirectory=/opt/etcd/
ExecStart=/opt/etcd/bin/etcd \
  --cert-file=/opt/etcd/ssl/etcd.pem \
  --key-file=/opt/etcd/ssl/etcd-key.pem \
  --trusted-ca-file=/opt/etcd/ssl/ca.pem \
  --peer-cert-file=/opt/etcd/ssl/etcd.pem \
  --peer-key-file=/opt/etcd/ssl/etcd-key.pem \
  --peer-trusted-ca-file=/opt/etcd/ssl/ca.pem \
  --peer-client-cert-auth \
  --client-cert-auth
Restart=on-failure
RestartSec=5
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

6.2.kube-apiserver.service

kube-apiserver的启动脚本

[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/kubernetes/kubernetes
After=etcd.service
Wants=etcd.service

[Service]
EnvironmentFile=-/opt/kubernetes/conf/kube-apiserver.conf
ExecStart=/opt/kubernetes/bin/kube-apiserver $KUBE_APISERVER_OPTS
Restart=on-failure
RestartSec=5
Type=notify
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

6.3.kube-controller-manager.service

kube-controller-manager的启动脚本

[Unit]
Description=Kubernetes Controller Manager
Documentation=https://github.com/kubernetes/kubernetes

[Service]
EnvironmentFile=-/opt/kubernetes/conf/kube-controller-manager.conf
ExecStart=/opt/kubernetes/bin/kube-controller-manager $KUBE_CONTROLLER_MANAGER_OPTS
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target

6.4.kube-scheduler.service

kube-scheduler的启动脚本

[Unit]
Description=Kubernetes Scheduler
Documentation=https://github.com/kubernetes/kubernetes

[Service]
EnvironmentFile=-/opt/kubernetes/conf/kube-scheduler.conf
ExecStart=/opt/kubernetes/bin/kube-scheduler $KUBE_SCHEDULER_OPTS
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target

6.5.kubelet.service

worker端kubelet的启动脚本

[Unit]
Description=Kubernetes Kubelet
Documentation=https://github.com/kubernetes/kubernetes
After=containerd.service
Requires=containerd.service

[Service]
ExecStart=/opt/kubernetes/bin/kubelet \
  --hostname-override=node-hostname \ #此处需要配置正确的节点的主机名
  --bootstrap-kubeconfig=/opt/kubernetes/conf/kubelet-bootstrap.kubeconfig \
  --cert-dir=/opt/kubernetes/ssl \
  --client-ca-file=/opt/kubernetes/ssl/ca.pem \
  --kubeconfig=/opt/kubernetes/conf/kubelet.kubeconfig \
  --config=/opt/kubernetes/conf/kubelet.yaml \
  --container-runtime-endpoint=unix:///var/run/containerd/containerd.sock \
  --v=2
Restart=on-failure
RestartSec=5
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

6.6.containerd启动脚本

在安装containerd时，解压cri-containerd-1.7.16-linux-amd64.tar.gz, tar zxvf cri-containerd-1.7.16-linux-amd64.tar.gz -C /就会在/etc/systemd/system/containerd.service 路径下有启动脚本

七.总结

建议将以上文件生成后统一放到一个目录，例如放到install_k8s的目录，然后将下载的软件也放在此目录，将生成证书的脚本和生成kubeconfig的脚本放在install_k8s目录下，在生成证书、配置文件、启动脚本以后，方便后边的安装步骤找对应的文件。