单master集群部署
一、 服务器环境及初始化
1、架构分析
| 集群角色 |
主机名 |
操作系统 |
IP地址 |
| master |
k8s-master |
OpenEuler24.03 |
192.168.92.101 |
| node |
k8s-node1 |
OpenEuler24.03 |
192.168.92.102 |
| node |
k8s-node2 |
OpenEuler24.03 |
192.168.92.103 |
2、初始化
所有节点都需要初始化!
2.1、清空Iptales默认规则及关闭防火墙
iptables -t nat -F
iptables -t filter -F
systemctl disable --now firewalld
2.2、关闭SELINUX
setenforce 0
sed -i 's/SELINUX=enforcing/SELINUX=disabled/' /etc/selinux/config
2.3、关闭Swap交换空间
swapoff -a
sed -i 's/.*swap.*/#&/' /etc/fstab
2.4、设置主机名
hostnamectl set-hostname k8s-master
hostnamectl set-hostname k8s-node1
hostnamectl set-hostname k8s-node2
2.5、编写hosts文件
cat <<EOF > /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.166.128 k8s-master
192.168.166.129 k8s-node1
192.168.166.130 k8s-node2
EOF
#拷贝到node节点
scp /etc/hosts 192.168.166.102:/etc
scp /etc/hosts 192.168.166.103:/etc
2.6、设置内核参数
cat <<EOF >> /etc/sysctl.conf
net.ipv4.ip_forward=1
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
modprobe br_netfilter
sysctl net.bridge.bridge-nf-call-ip6tables=1
sysctl net.bridge.bridge-nf-call-iptables=1
sysctl -p
二、安装Docker环境
所有节点都需要安装!
1、安装Docker
1.1、配置阿里源
cat <<EOF >> /etc/yum.repos.d/docker-ce.repo
docker-ce-stable
name=Docker CE Stable - $basearch
baseurl=https://mirrors.aliyun.com/docker-ce/linux/centos/9/x86_64/stable/
enabled=1
gpgcheck=1
gpgkey=https://mirrors.aliyun.com/docker-ce/linux/centos/gpg
EOF
1.2、安装docker
yum install -y docker-ce-28.0.3
1.3、启动docker
cat <<EOF >>/etc/docker/daemon.json
{
"registry-mirrors": [
"https://0vmzj3q6.mirror.aliyuncs.com",
"https://docker.m.daocloud.io",
"https://mirror.baidubce.com",
"https://dockerproxy.com",
"https://mirror.iscas.ac.cn",
"https://huecker.io",
"https://dockerhub.timeweb.cloud",
"https://noohub.ru",
"https://vlgh0kqj.mirror.aliyuncs.com"
]
}
EOF
systemctl daemon-reload
systemctl enable --now docker
2、安装cri-dockerd
下载地址:https://github.com/Mirantis/cri-dockerd/releases
yum install -y libcgroup
rpm -ivh cri-dockerd-0.3.8-3.el8.x86_64.rpm
#或者
yum localinstall cri-dockerd-0.3.8-3.el8.x86_64.rpm
修改CRI启动脚本
vim /usr/lib/systemd/system/cri-docker.service
ExecStart=/usr/bin/cri-dockerd --container-runtime-endpoint fd:// --pod-infra-container-image=registry.aliyuncs.com/google_containers/pause:3.9
启动cri
systemctl daemon-reload
systemctl enable --now cri-docker
三、安装kubeadm和kubectl
所有节点都需要安装!
1、配置yum源
cat <<EOF | tee /etc/yum.repos.d/kubernetes.repo
kubernetes
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes-new/core/stable/v1.28/rpm/
enabled=1
gpgcheck=1
gpgkey=https://mirrors.aliyun.com/kubernetes-new/core/stable/v1.28/rpm/repodata/repomd.xml.key
EOF
2、安装
yum install -y kubelet kubeadm kubectl
3、设置kubectl开机自启动
systemctl enable kubelet && systemctl start kubelet
四、部署Master节点
在k8s-master节点执行下述命令:
kubeadm init --apiserver-advertise-address=192.168.92.101 --image-repository=registry.aliyuncs.com/google_containers --kubernetes-version=v1.28.15 --pod-network-cidr=10.244.0.0/16 --service-cidr=10.96.0.0/12 --cri-socket=unix:///var/run/cri-dockerd.sock
命令解析:
--apiserver-advertise-address:指定 API Server 监听的 IP 地址。如果没有设置,则将使用默认的网络接口。
--image-repository:指定镜像仓库地址。默认值为"registry.k8s.io",但该仓库在中国无法访问,因此这里指定阿里云仓库。
--kubernetes-version:指定 Kubernetes 版本。
--pod-network-cidr:指定 Pod 网络的 CIDR 地址范围。
--service-cidr:指定 Service 网络的 CIDR 地址范围。
--cri-socket:指定 kubelet 连接容器运行时的 UNIX 套接字文件。
出问题后,集群还原
kubeadm reset --cri-socket=unix:///var/run/cri-dockerd.sock
保存证书文件
kubeadm join 192.168.92.101:6443 --token ew0hae.zvlbtmqviau62chk \
--discovery-token-ca-cert-hash sha256:b446dec4d6c3222792a0184cd5b3037d61fbcdb6aca0ff100c2fa35d166c7d77
#证书都是不一致的,注意查看。
配置管理集群文件
mkdir -p $HOME/.kube
cd /root/.kube
cp /etc/kubernetes/admin.conf ./config
#查看集群状态
kubectl get nodes
五、部署node节点
分别在k8s-node1和k8s-node2中执行:
kubeadm join 192.168.92.101:6443 --token ew0hae.zvlbtmqviau62chk \
--discovery-token-ca-cert-hash sha256:b446dec4d6c3222792a0184cd5b3037d61fbcdb6aca0ff100c2fa35d166c7d77 --cri-socket=unix:///var/run/cri-dockerd.sock
查看集群状态:
root@k8s-master \~\]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
k8s-master NotReady control-plane 2d16h v1.28.15
k8s-node1 NotReady \ 2d16h v1.28.15
k8s-node2 NotReady \ 2m14s v1.28.15
目前看到的是NotReady状态,是由于没有安装网络插件的原因。ROLES角色一栏显示"none",可以通过一下命令修改角色名称:
kubectl label node k8s-master node-role.kubernetes.io/master=master
kubectl label node k8s-node1 node-role.kubernetes.io/worker=worker
kubectl label node k8s-node2 node-role.kubernetes.io/worker=worker
## 六、部署网络插件
需要在master主机先创建calico的相关资源,然后查看所需镜像:
docker load -i calico.tar
docker load -i calico-apiserver.tar
**提交资源清单:**
wget https://raw.githubusercontent.com/projectcalico/calico/v3.26.0/manifests/tigera-operator.yaml \[root@k8s-master \~\]# kubectl create -f tigera-operator.yaml
wget https://raw.githubusercontent.com/projectcalico/calico/v3.26.0/manifests/custom-resources.yaml
##编辑网络信息
vim custom-resources.yaml
apiVersion: operator.tigera.io/v1
kind: Installation
metadata:
name: default
spec:
calicoNetwork:
ipPools:
- blockSize: 26
cidr: 10.244.0.0/16 # 修改此值,与"kubeadm init"命令中指定的 Pod 网络CIDR 地址范围保持一致
encapsulation: VXLANCrossSubnet
natOutgoing: Enabled
nodeSelector: all()
...
\[root@k8s-master \~\]# kubectl create -f custom-resources.yaml
**资源提交后,使用下述命令列出所需的镜像文件:**
\[root@k8s-master \~\]# kubectl -n calico-system describe pod \| grep "Image:" \| sort \| uniq
Image: docker.io/calico/cni:v3.26.0
Image: docker.io/calico/csi:v3.26.0
Image: docker.io/calico/kube-controllers:v3.26.0
Image: docker.io/calico/node-driver-registrar:v3.26.0
Image: docker.io/calico/node:v3.26.0
Image: docker.io/calico/pod2daemon-flexvol:v3.26.0
Image: docker.io/calico/typha:v3.26.0
\[root@k8s-master \~\]# kubectl -n calico-apiserver describe pod calico-apiserver-894947d4b-bqc7x \| grep -i 'image:'
Image: docker.io/calico/apiserver:v3.26.0
## 七、部署Dashboard
**下载Dashboard资源清单:**
wget https://raw.githubusercontent.com/kubernetes/dashboard/v2.7.0/aio/deploy/recommended.yaml
**修改Service类型:**
vim recommended.yaml
...
kind: Service
apiVersion: v1
metadata:
labels:
k8s-app: kubernetes-dashboard
name: kubernetes-dashboard
namespace: kubernetes-dashboard
spec:
type: NodePort # 指定 NodePort 类型
ports:
- port: 443
targetPort: 8443
nodePort: 30001 # 指定访问端口
selector:
k8s-app: kubernetes-dashboard
**在集群中创建资源:**
\[root@k8s-master \~\]# kubectl apply -f recommended.yaml
**查看 Pod 对象:**
\[root@k8s-master \~\]# kubectl get pods -n kubernetes-dashboard
所有 Pod 的状态都显示为"Running",说明 Dashboard 安装成功。在浏览器中访问 "https://\<节点 IP 地址\>:30001"即可看到WEB UI界面。
**创建一个服务账号并授予集群管理员权限:**
\[root@k8s-master \~\]# kubectl create serviceaccount admin-user -n kubernetes-dashboard
\[root@k8s-master \~\]# kubectl create clusterrolebinding admin-user --clusterrole=cluster-admin --serviceaccount=kubernetes-dashboard:admin-user
**根据服务账号创建 Token:**
\[root@k8s-master \~\]# kubectl create token admin-user -n kubernetes-dashboard
## 八、Metrics部署
系统资源的采集需要使用Metrics-server,能够采集节点和pod的内存、磁盘、CPU、网络的使用率!
**将k8s-master01的front-proxy-ca.crt文件复制到所有NODE节点**
\[root@k8s-master01 \~\]# scp /etc/kubernetes/pki/front-proxy-ca.crt k8s-node1:/etc/kubernetes/pki/
\[root@k8s-master01 \~\]# scp /etc/kubernetes/pki/front-proxy-ca.crt k8s-node2:/etc/kubernetes/pki/
#下载0.7版本的metrics
#下载地址:https://github.com/kubernetes-sigs/metrics-server/releases/download/v0.7.0/components.yaml
#或者网络方式安装:
\[root@k8s-master01 \~\]# kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/download/v0.7.0/components.yaml
#修改yaml文件镜像源及探针模式
\[root@k8s-master01 metrics\]# vim components.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
labels:
k8s-app: metrics-server
name: metrics-server
namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
labels:
k8s-app: metrics-server
rbac.authorization.k8s.io/aggregate-to-admin: "true"
rbac.authorization.k8s.io/aggregate-to-edit: "true"
rbac.authorization.k8s.io/aggregate-to-view: "true"
name: system:aggregated-metrics-reader
rules:
- apiGroups:
- metrics.k8s.io
resources:
- pods
- nodes
verbs:
- get
- list
- watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
labels:
k8s-app: metrics-server
name: system:metrics-server
rules:
- apiGroups:
- ""
resources:
- nodes/metrics
verbs:
- get
- apiGroups:
- ""
resources:
- pods
- nodes
verbs:
- get
- list
- watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
labels:
k8s-app: metrics-server
name: metrics-server-auth-reader
namespace: kube-system
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: extension-apiserver-authentication-reader
subjects:
- kind: ServiceAccount
name: metrics-server
namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
labels:
k8s-app: metrics-server
name: metrics-server:system:auth-delegator
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: system:auth-delegator
subjects:
- kind: ServiceAccount
name: metrics-server
namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
labels:
k8s-app: metrics-server
name: system:metrics-server
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: system:metrics-server
subjects:
- kind: ServiceAccount
name: metrics-server
namespace: kube-system
---
apiVersion: v1
kind: Service
metadata:
labels:
k8s-app: metrics-server
name: metrics-server
namespace: kube-system
spec:
ports:
- name: https
port: 443
protocol: TCP
targetPort: https
selector:
k8s-app: metrics-server
---
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
k8s-app: metrics-server
name: metrics-server
namespace: kube-system
spec:
selector:
matchLabels:
k8s-app: metrics-server
strategy:
rollingUpdate:
maxUnavailable: 0
template:
metadata:
labels:
k8s-app: metrics-server
spec:
containers:
- args:
- --cert-dir=/tmp
- --secure-port=10250
- --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
- --kubelet-use-node-status-port
- --metric-resolution=15s
- --kubelet-insecure-tls ##增加证书验证
image: registry.aliyuncs.com/google_containers/metrics-server:v0.7.2 ##修改为国内镜像源
imagePullPolicy: IfNotPresent
livenessProbe:
failureThreshold: 3
tcpSocket: ###修改探针模式
port: 10250 ###修改探测端口
periodSeconds: 10
name: metrics-server
ports:
- containerPort: 10250
name: https
protocol: TCP
readinessProbe:
failureThreshold: 3
tcpSocket: ###修改探针模式
port: 10250 ###修改探测端口
initialDelaySeconds: 20
periodSeconds: 10
resources:
requests:
cpu: 100m
memory: 200Mi
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop:
- ALL
readOnlyRootFilesystem: true
runAsNonRoot: true
runAsUser: 1000
seccompProfile:
type: RuntimeDefault
volumeMounts:
- mountPath: /tmp
name: tmp-dir
nodeSelector:
kubernetes.io/os: linux
priorityClassName: system-cluster-critical
serviceAccountName: metrics-server
volumes:
- emptyDir: {}
name: tmp-dir
---
apiVersion: apiregistration.k8s.io/v1
kind: APIService
metadata:
labels:
k8s-app: metrics-server
name: v1beta1.metrics.k8s.io
spec:
group: metrics.k8s.io
groupPriorityMinimum: 100
insecureSkipTLSVerify: true
service:
name: metrics-server
namespace: kube-system
version: v1beta1
versionPriority: 100