云原生高级——kubernetes的部署

一.构建harbor镜像仓库

1.部署docker本地仓库

配置阿里云源并下载 Docker 安装包

[root@harbor ~]# cat > /etc/yum.repos.d/docker.repo <<EOF
[docker]
name = docker
baseurl = https://mirrors.aliyun.com/docker-ce/linux/rhel/9.6/x86_64/stable/
gpgcheck = 0
EOF

[root@harbor ~]# dnf install httpd createrepo -y
#这是 HTTP 服务的根目录下的子目录，用来存放 RPM 包
[root@harbor ~]# mkdir  /var/www/html/docker/ -p
#修改默认端口，避免冲突
[root@harbor ~]# vim /etc/httpd/conf/httpd.conf
Listen 4444
[root@harbor ~]# systemctl enable --now httpd

下载 RPM 包并生成仓库元数据，并将仓库部署到本地中

作用：以前指向阿里云，现在指向本地仓库，内网传输比去外网下载快得多，多台机器可以共用这个本地源，能够批量下载。

#--downloadonly：只下载 RPM 包，不执行安装 --destdir：指定下载存放位置
[root@harbor ~]# dnf  install docker-ce --downloadonly  --destdir /mnt/ -y
[root@harbor ~]# mv /mnt/*.rpm /var/www/html/docker/
#扫描该目录下的所有 RPM，生成 repodata 目录
[root@harbor ~]# createrepo -v /var/www/html/docker/

#切换为本地私有仓库并安装
[root@harbor ~]# cat > /etc/yum.repos.d/docker.repo <<EOF
[docker]
name = docker
baseurl = http://172.25.254.254:4444/docker
gpgcheck = 0
EOF

2.安装docker在harbor仓库节点

#在100.10.20都需要以下步骤
[root@harbor ~]# dnf install docker-ce -y

#加载内核模块
[root@harbor ~]# echo br_netfilter > /etc/modules-load.d/docker_mod.conf
[root@harbor ~]# modprobe -a br_netfilter
[root@harbor ~]# vim /etc/sysctl.d/docker.conf
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward = 1

[root@harbor ~]# systemctl restart  systemd-modules-load.service
[root@harbor ~]# sysctl  --system

[root@harbor ~]# vim /lib/systemd/system/docker.service
#让 Docker 自动管理 iptables 规则
ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock --iptables=true
[root@harbor ~]# systemctl daemon-reload
[root@harbor ~]# systemctl enable --now docker

3.生成key

将密钥给其他的主机，在每台需要访问 Harbor 的 Docker 客户端上。

[root@harbor ~]# mkdir /data/certs -p
[root@harbor ~]# mkdir /data/certs -p
[root@harbor ~]# openssl req -newkey  rsa:4096 \
-nodes -sha256 -keyout /data/certs/timinglee.org.key \
-addext "subjectAltName = DNS:reg.timinglee.org" \
-x509 -days 365 -out /data/certs/timinglee.org.crt

You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [XX]:CN
State or Province Name (full name) []:Shannxi
Locality Name (eg, city) [Default City]:Xi'an
Organization Name (eg, company) [Default Company Ltd]:kubernetes
Organizational Unit Name (eg, section) []:harbor
Common Name (eg, your name or your server's hostname) []:reg.timinglee.org
Email Address []:admin@timinglee.org

4.编辑harbor配置文件

[root@harbor ~]# tar zxf  harbor-offline-installer-v2.5.4.tgz -C /opt/
[root@harbor ~]# cd /opt/harbor/
[root@harbor harbor]# ls
common.sh  harbor.v2.5.4.tar.gz  harbor.yml.tmpl  install.sh  LICENSE  prepare
[root@harbor harbor]# cp harbor.yml.tmpl harbor.yml
[root@harbor harbor]# vim harbor.yml
hostname: reg.timinglee.org
  certificate: /data/certs/timinglee.org.crt
  private_key: /data/certs/timinglee.org.key
harbor_admin_password: lee

#启动 Harbor 安装的命令
[root@harbor harbor]# ./install.sh --with-chartmuseum
#--with-chartmuseum 启用 ChartMuseum 组件

#编写启动脚本
[root@harbor ~]# vim /lib/systemd/system/harbor.service
[Unit]
Description=harbor with Docker Compose
Documentation=https://reg.timinglee.com/compose/
After=docker.service network-online.target
Requires=docker.service

[Service]
Type=oneshot
WorkingDirectory=/opt/harbor
ExecStart=/usr/bin/docker compose up -d
ExecStop=/usr/bin/docker compose down
RemainAfterExit=yes

[Install]
WantedBy=multi-user.target

#将docker的东西删掉观看开机启动是否成功
[root@harbor harbor]# docker compose  down
WARN[0000] /opt/harbor/docker-compose.yml: the attribute `version` is obsolete, it will be ignored, please remove it to avoid potential confusion
[+] down 4/6
 ⠇ Container registryctl       Stopping                1.9s
 ✔ Container nginx             Removed                 0.2s
 ✔ Container chartmuseum       Removed                 0.2s
 ✔ Container harbor-jobservice Removed                 0.1s
 ✔ Container harbor-portal     Removed                 0.1s
 ⠦ Container harbor-core       Stopping                1.7s               1.7s
 
 
[root@harbor ~]# systemctl enable --now harbor
Created symlink /etc/systemd/system/multi-user.target.wants/harbor.service → /usr/lib/systemd/system/harbor.service.

[root@harbor harbor]# docker compose  ps

harbor容器启动成功

5.启动并验证

在每一个主机下面要创建
[root@harbor harbor]# mkdir  /etc/docker/certs.d/reg.timinglee.org/ -p

#让 Docker 客户端能够通过 HTTPS 访问 reg.timinglee.org 这个 Harbor 仓库，而不报证书错误
[root@harbor harbor]# cp /data/certs/timinglee.org.crt  /etc/docker/certs.d/reg.timinglee.org/ca.crt

#让整个操作系统（curl、wget、浏览器等）都信任该证书
[root@harbor harbor]# cp /data/certs/timinglee.org.crt  /etc/pki/ca-trust/source/anchors/ca.cr

[root@harbor harbor]# vim /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
172.25.254.200     harbor reg.timinglee.org


[root@harbor harbor]# systemctl restart docker
[root@harbor harbor]# docker compose up -d

[root@harbor harbor]# docker login  reg.timinglee.org -u admin
Password:

WARNING! Your credentials are stored unencrypted in '/root/.docker/config.json'.
Configure a credential helper to remove this warning. See
https://docs.docker.com/go/credential-store/

Login Succeeded

在网页中访问harbor的IP就能进入harbor仓库

二.构建部署kubernetes所需主机

|-----------|-------------------|------------------|
| 主机名 | IP | 角色 |
| harbor | 172.25.254.250 | harbor仓库 |
| master | 172.25.254.100 | master，k8s集群控制节点 |
| node1 | 172.25.254.10 | worker，k8s集群工作节点 |
| node2 | 172.25.254.20 | worker，k8s集群工作节点 |

1.关闭swap（所有主机配置）

#方法一
systemctl disable --now  swap.target
systemctl mask swap.target
sed '/swap/s/^/#/g' -i /etc/fstab
#方法二
/dev/mapper/rhel-root   /                          xfs     defaults        0 0
UUID=0df96f31-df46-4b64-ab51-dd5a4e58445f /boot    xfs     defaults        0 0
/dev/mapper/rhel-home   /home                      xfs     defaults        0 0
#/dev/mapper/rhel-swap   none                      swap    defaults        0 0
#将最后一行注释掉
#弄好之后重启一下就可以了

2.配置可以使用harbor仓库

在100.19.20的主机上创建文件架
mkdir  /etc/docker/certs.d/reg.timinglee.org/ -p
#在harbor主机中分发证书到所有主机
[root@harbor ~]# for i in 100 10 20
> do
> scp /data/certs/timinglee.org.crt root@172.25.254.$i:/etc/docker/certs.d/reg.timinglee.org/ca.crt
> done

systemctl enable  docker
systemctl restart docker

3.所有主机配置docker加速器

cat >/etc/docker/daemon.json <<EOF
{
  "registry-mirrors":["https://reg.timinglee.org"]
}
EOF
systemctl restart docker

docker info
可以看到
Registry Mirrors:
  https://reg.timinglee.org/

4.所有主机彼此建立解析

vim /etc/hosts

127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
172.25.254.100     master
172.25.254.10      node1
172.25.254.20      node2
172.25.254.250     reg.timinglee.org

5.所有主机配置kubernetes安装源

 vim /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name = kubernetes
baseurl = https://mirrors.aliyun.com/kubernetes-new/core/stable/v1.35/rpm/
gpgcheck = 0


#检测
dnf list kubelet

#以上操作完成后重启主机检测swap分区
swapon  -s  没有任何输出表示ok

三.kubernetes的部署

1.安装cri-dockerd（所有主机中安装）

下载二进制包
[root@docker-node1 ~]# wget https://github.com/Mirantis/cri-dockerd/releases/download/v0.4.4/cri-dockerd-0.4.4.amd64.tgz

#解压二进制压缩包
[root@docker-node1 ~]# tar zxf cri-dockerd-0.4.4.amd64.tgz

[root@docker-node1 ~]# cd cri-dockerd/
[root@docker-node1 cri-dockerd]# ls
cri-dockerd  cri-docker.service  cri-docker.socket



#安装cri-docker命令到系统
[root@docker-node1 cri-dockerd]# install -o root -g root -m 0755 cri-dockerd  /usr/local/bin/cri-dockerd

[root@docker-node1 cri-dockerd]# ls -l /usr/local/bin/cri-dockerd
-rwxr-xr-x 1 root root 51638434 Aug 16 19:11 /usr/local/bin/cri-dockerd

#生成启动文件
[root@docker-node1 cri-dockerd]# cp cri-docker.s* /lib/systemd/system
cp: overwrite '/lib/systemd/system/cri-docker.service'? y
[root@docker-node1 cri-dockerd]# chmod  +x /lib/systemd/system/cri-docker.s*

#在添加时注意自己的版本，要与版本一致才能启动成功
[root@docker-node1 cri-dockerd]# vim /lib/systemd/system/cri-docker.service
ExecStart=/usr/local/bin/cri-dockerd --network-plugin=cni --pod-infra-container-image=reg.timinglee.org/k8s/pause:3.10.1 --container-runtime-endpoint fd://
ExecReload=/bin/kill -s HUP $MAINPID


#用ssh在其余主机部署cri-docker
[root@harbor cri-dockerd]# for i in 100 10 20; do scp cri-dockerd   root@172.25.254.$i:/usr/bin/cri-dockerd; done
 
[root@harbor cri-dockerd]# for i in 100 10 20; do scp /lib/systemd/system/cri-docker.s*   root@172.25.254.$i:/lib/systemd/system/; done


#启动服务
[root@docker-node1 cri-dockerd]# systemctl enable --now cri-docker.service
Created symlink /etc/systemd/system/multi-user.target.wants/cri-docker.service → /usr/lib/systemd/system/cri-docker.service.
[root@docker-node1 cri-dockerd]# systemctl status cri-docker.service

2.安装构建kubernetes 集群所需软件

#在master中
[root@master ~]dnf install kubelet kubeadm kubectl -y
[root@master ~]systemctl enable --now kubelet.service

#在node1，node2中
[root@k8s-node1 ~]# dnf install kubelet kubeadm  -y
[root@k8s-node1 ~]# systemctl enable --now kubelet.service

#也可以在harbor中将软件下载但不安装，存储到本地仓库中，通过ssh传输到master，node1，node2中
[root@harbor ~]# dnf install kubelet kubeadm kubectl --downloadonly --destdir /mnt/ -y
[root@harbor ~]# mv /mnt/*.rpm /var/www/html/docker/
[root@harbor ~]# createrepo -v /var/www/html/docker/
 
#master主机安装kubelet,kubeadm,kubectl
[root@harbor ~]#  ssh -l root 172.25.254.100 "dnf clean all;dnf install kubelet kubeadm kubectl -y"
 
#node主机安装kubelet,kubeadm
[root@harbor ~]# for i in 10 20; do ssh -l root 172.25.254.$i "dnf clean all;dnf install kubelet kubeadm kubectl -y;systemctl enable --now kubelet.service"; done

3.master节点中 kubectl 和kubeadm 补齐

[root@master ~]# echo "source <(kubectl completion bash)" >> ~/.bashrc
[root@master ~]# source  ~/.bashrc


##让补齐功能生效
[root@k8s-master mnt]# kubectl 命令后按2此tab键
alpha          (Commands for features in alpha)
annotate       (更新一个资源的注解)
api-resources  (Print the supported API resources on the server)
api-versions   (Print the supported API versions on the server, in the form of "group/version")
apply          (Apply a configuration to a resource by file name or stdin)
attach         (挂接到一个运行中的容器)
auth           (Inspect authorization)
autoscale      (Auto-scale a deployment, replica set, stateful set, or replication controller)
certificate    (Modify certificate resources)

[root@k8s-master ~]# kubeadm
certs       config      init        kubeconfig  token       version
completion  help        join        reset       upgrade

4.下载kubernetes集群所需镜像

查看所需镜像

[root@k8s-master ~]# kubeadm config images list

下载镜像

[root@master ~]# kubeadm config images pull \
--image-repository registry.aliyuncs.com/google_containers \
--kubernetes-version v1.35.8 \      #这里的版本号跟上面查到的一样
--cri-socket=unix:///var/run/cri-dockerd.sock

上传镜像到本地harbor

#验证是否能够进入docker容器
[root@master ~]# docker  login  reg.timinglee.org -u admin
Password:

WARNING! Your credentials are stored unencrypted in '/root/.docker/config.json'.
Configure a credential helper to remove this warning. See
https://docs.docker.com/go/credential-store/

Login Succeeded

#批量打上标签，为上传做准备
[root@master ~]# docker images  --format  "{{.Repository}}:{{.Tag}}" | awk -F "/" '/google/{system("docker tag "$0" reg.timinglee.org/k8s/"$3)}'
#上传镜像，为后续集群拉取做准备
[root@master ~]# docker images  --format  "{{.Repository}}:{{.Tag}}" | awk -F "/" '/timinglee/{system("docker push "$0)}'

四.在master中初始化kubernetes集群

1.在master中完成集群初始化

[root@master ~]# kubeadm init --pod-network-cidr=10.244.0.0/16 \
--image-repository reg.timinglee.org/k8s \
--kubernetes-version v1.35.8 \
--cri-socket=unix:///var/run/cri-dockerd.sock

#每个人不一样，其他主机加入本集群的凭证
kubeadm join 172.25.254.100:6443 --token tdwjoc.8d1yw3wl4r4tm6c4 \
        --discovery-token-ca-cert-hash sha256:6b5950ef2cdba85d6dfdb564ee90d4187fa3d341767dc9852cbdd5c9dee4f927

#如果忘记
[root@master ~]#  kubeadm token create --print-join-command
kubeadm join 172.25.254.100:6443 --token jl4ztx.cax3iysvu7onsh5s --discovery-token-ca-cert-hash sha256:6b5950ef2cdba85d6dfdb564ee90d4187fa3d341767dc9852cbdd5c9dee4f927

#如果初始化出问题
[root@master ~]# kubeadm reset  --cri-socket=unix:///var/run/cri-dockerd.sock #可以重置集群设定
#发现是无法连接，连接超时，可能是因为ip冲突，导致域名无法解析
#如果是初始化的问题，凭证无法添加，被卡住可以检查100上的防火墙

在初始化完成后末端常看

在初始化过程中出现这个报错是因为核心的数量不够，需要去设置里面添加cpu

2.添加kubernets环境变量到本机

[root@master ~]#  export KUBECONFIG=/etc/kubernetes/admin.conf > ~/.bash_profile 
[root@master ~]# source ~/.bash_profile 
[root@master ~]#  kubectl get nodes
NAME     STATUS     ROLES           AGE     VERSION
master   NotReady   control-plane   5m15s   v1.36.4

3.添加node节点到本集群

[root@node1 ~]#  kubeadm join 172.25.254.100:6443 --token jl4ztx.cax3iysvu7onsh5s --discovery-token-ca-cert-hash sha256:6b5950ef2cdba85d6dfdb564ee90d4187fa3d341767dc9852cbdd5c9dee4f927 --cri-socket=unix:///var/run/cri-dockerd.sock


#测试
[root@master ~]# kubectl get nodes		#可以看到集群中主机但是因为网络插件问题状态是NotReady
NAME     STATUS     ROLES           AGE     VERSION
master   NotReady   control-plane   5m16s   v1.35.3
node1    NotReady   <none>          66s     v1.35.3
node2    NotReady   <none>          58s     v1.35.3

#这里NotReady是因为没配网络

五.安装网络插件

#在mater中提前准备安装包
[root@k8s-master ~]# ls
flannel  kube-flannel-v0.28.9.tar.gz
[root@k8s-master ~]# tar zxf kube-flannel-v0.28.9.tar.gz

[root@k8s-master ~]# cd flannel/

[root@k8s-master flannel]# docker load  -i kube-flannel-v0.28.9.tar
Loaded image: ghcr.io/flannel-io/flannel-cni-plugin:v1.9.1-flannel3
Loaded image: ghcr.io/flannel-io/flannel:v0.28.9

#在进行传输时，要在harbor容器中创建flannel-io项目
[root@master ~]# docker tag ghcr.io/flannel-io/flannel-cni-plugin:v1.9.0-flannel1 reg.timinglee.org/flannel-io/flannel-cni-plugin:v1.9.0-flannel1
[root@master ~]# docker push  reg.timinglee.org/flannel-io/flannel-cni-plugin:v1.9.0-flannel1
The push refers to repository [reg.timinglee.org/flannel-io/flannel-cni-plugin]
2c8aa52d4746: Pushed
5aa68bbbc67e: Pushed
v1.9.0-flannel1: digest: sha256:b3d30c221113b30fea3e8a7fccb145e929b097d0319b9eeb6b5a591b10b5c671 size: 739

[root@master ~]# docker tag ghcr.io/flannel-io/flannel:v0.28.1 reg.timinglee.org/flannel-io/flannel:v0.28.1
[root@master ~]# docker push reg.timinglee.org/flannel-io/flannel:v0.28.1
The push refers to repository [reg.timinglee.org/flannel-io/flannel]
5668da16a30b: Pushed
5f70bf18a086: Pushed
00012e17b6cc: Pushed
9738bb9596cf: Pushed
b6233bc105d7: Pushed
03767449b95b: Pushed
9a7d8cf9ff51: Pushed
13a60c10faeb: Pushed
70dc5e033175: Pushed
256f393e029f: Pushed
v0.28.1: digest: sha256:e671adbc267460164555159210066d3304a43e3b5dd85cc0b5b6ad62e83aab52 size: 2414


[root@master ~]# vim kube-flannel.yml
image: flannel-io/flannel:v0.28.9
image: flannel-io/flannel-cni-plugin:v1.9.1-flannel3
image: flannel-io/flannel:v0.28.9

[root@k8s-master flannel]# grep -n  image: kube-flannel.yml
148:        image: flannel-io/flannel:v0.28.9
175:        image: flannel-io/flannel-cni-plugin:v1.9.1-flannel3
186:        image: flannel-io/flannel:v0.28.9




[root@master ~]# kubectl apply -f kube-flannel.yml
namespace/kube-flannel created
serviceaccount/flannel created
clusterrole.rbac.authorization.k8s.io/flannel created
clusterrolebinding.rbac.authorization.k8s.io/flannel created
configmap/kube-flannel-cfg created
daemonset.apps/kube-flannel-ds created

#如果是在前面镜像的提取错误，修改之后将kube-flannel.yml删掉重新生成
[root@k8s-master flannel]# kubectl delete -f kube-flannel.yml 


#测试
[root@k8s-master flannel]# kubectl get nodes
NAME         STATUS   ROLES           AGE   VERSION
k8s-master   Ready    control-plane   50m   v1.35.8
k8s-node1    Ready    <none>          39m   v1.35.8
k8s-node2    Ready    <none>          35m   v1.35.8