部署K8S

ninety_two2023-10-31 8:26

防火强的初始化：

[root@k8s-node-12 ~]# systemctl stop firewalld NetworkManager
[root@k8s-node-12 ~]# systemctl disable firewalld NetworkManager
Removed symlink /etc/systemd/system/multi-user.target.wants/NetworkManager.service.
Removed symlink /etc/systemd/system/multi-user.target.wants/firewalld.service.
Removed symlink /etc/systemd/system/dbus-org.freedesktop.nm-dispatcher.service.
Removed symlink /etc/systemd/system/network-online.target.wants/NetworkManager-wait-online.service.
Removed symlink /etc/systemd/system/dbus-org.fedoraproject.FirewallD1.service.
[root@k8s-node-12 ~]# sed -ri 's#(SELINUX=).*#\1disabled#' /etc/selinux/config
[root@k8s-node-12 ~]# setenforce 0
[root@k8s-node-12 ~]# iptables -F
[root@k8s-node-12 ~]# iptables -X
[root@k8s-node-12 ~]# iptables -Z
[root@k8s-node-12 ~]# iptables -P FORWARD ACCEPT

关闭swap：

k8s默认禁用swap功能
[root@k8s-node-11 ~]# swapoff -a
防止开机自动挂载swap分区
[root@k8s-node-11 ~]# sed -i '/ swap /  s/^\(.*\)$/#\1/g'  /etc/fstab

yum源的配置：

[root@k8s-node-11 yum.repos.d]# cat CentOS-Base.repo 
# CentOS-Base.repo
#
# The mirror system uses the connecting IP address of the client and the
# update status of each mirror to pick mirrors that are updated to and
# geographically close to the client.  You should use this for CentOS updates
# unless you are manually picking other mirrors.
#
# If the mirrorlist= does not work for you, as a fall back you can try the 
# remarked out baseurl= line instead.
#
#

[base]
name=CentOS-$releasever - Base
mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=os&infra=$infra
#baseurl=http://mirror.centos.org/centos/$releasever/os/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7

#released updates 
[updates]
name=CentOS-$releasever - Updates
mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=updates&infra=$infra
#baseurl=http://mirror.centos.org/centos/$releasever/updates/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7

#additional packages that may be useful
[extras]
name=CentOS-$releasever - Extras
mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=extras&infra=$infra
#baseurl=http://mirror.centos.org/centos/$releasever/extras/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7

#additional packages that extend functionality of existing packages
[centosplus]
name=CentOS-$releasever - Plus
mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=centosplus&infra=$infra
#baseurl=http://mirror.centos.org/centos/$releasever/centosplus/$basearch/
gpgcheck=1
enabled=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7

[root@k8s-node-11 yum.repos.d]# 

源配置具体步骤一：
[root@k8s-node-11 network-scripts]# curl -o /etc/yum.repos.d/CentOS-Base.repo https://mirrors.aliyun.com/repo/Centos-7.repo
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  2523  100  2523    0     0    287      0  0:00:08  0:00:08 --:--:--   777
[root@k8s-node-11 network-scripts]# curl -o /etc/yum.repos.d/epel.repo http://mirrors.aliyun.com/repo/epel-7.repo
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   664  100   664    0     0     77      0  0:00:08  0:00:08 --:--:--   162
[root@k8s-node-11 network-scripts]# sed -i '/aliyuncs/d' /etc/yum.repos.d/*.repo
[root@k8s-node-11 network-scripts]# yum clean all && yum makecache fast
已加载插件：fastestmirror, langpacks
正在清理软件源： base epel extras updates
Cleaning up list of fastest mirrors
已加载插件：fastestmirror, langpacks
Determining fastest mirrors
base                                                                                                                                                                                        | 3.6 kB  00:00:00     
epel                                                                                                                                                                                        | 4.7 kB  00:00:00     
extras                                                                                                                                                                                      | 2.9 kB  00:00:00     
updates                                                                                                                                                                                     | 2.9 kB  00:00:00     
(1/7): epel/x86_64/group_gz                                                                                                                                                                 |  99 kB  00:00:08     
(2/7): epel/x86_64/updateinfo                                                                                                                                                               | 1.0 MB  00:00:08     
(3/7): base/7/x86_64/group_gz                                                                                                                                                               | 153 kB  00:00:08     
(4/7): epel/x86_64/primary_db                                                                                                                                                               | 7.0 MB  00:00:01     
(5/7): base/7/x86_64/primary_db                                                                                                                                                             | 6.1 MB  00:00:09     
(6/7): updates/7/x86_64/primary_db                                                                                                                                                          |  24 MB  00:00:02     
(7/7): extras/7/x86_64/primary_db                                                                                                                                                           | 250 kB  00:00:12     
元数据缓存已建立
[root@k8s-node-11 network-scripts]#

源配置具体步骤二：
[root@k8s-master-10 ~]# curl -o /etc/yum.repos.d/CentOS-Base.repo https://mirrors.aliyun.com/repo/Centos-7.repo
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  2523  100  2523    0     0    288      0  0:00:08  0:00:08 --:--:--   783
[root@k8s-master-10 ~]# curl -o /etc/yum.repos.d/epel.repo http://mirrors.aliyun.com/repo/epel-7.repo
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   664  100   664    0     0     76      0  0:00:08  0:00:08 --:--:--   160
[root@k8s-master-10 ~]# sed -i '/aliyuncs/d' /etc/yum.repos.d/*.repo
[root@k8s-master-10 ~]# yum clean all && yum makecache fast
已加载插件：fastestmirror, langpacks
Repository base is listed more than once in the configuration
Repository updates is listed more than once in the configuration
Repository extras is listed more than once in the configuration
Repository centosplus is listed more than once in the configuration
Repository contrib is listed more than once in the configuration
正在清理软件源： base docker-ce-stable epel extras updates
Cleaning up list of fastest mirrors
已加载插件：fastestmirror, langpacks
Repository base is listed more than once in the configuration
Repository updates is listed more than once in the configuration
Repository extras is listed more than once in the configuration
Repository centosplus is listed more than once in the configuration
Repository contrib is listed more than once in the configuration
Determining fastest mirrors
base                                                                                                                                                                                        | 3.6 kB  00:00:00     
docker-ce-stable                                                                                                                                                                            | 3.5 kB  00:00:00     
epel                                                                                                                                                                                        | 4.7 kB  00:00:00     
extras                                                                                                                                                                                      | 2.9 kB  00:00:00     
updates                                                                                                                                                                                     | 2.9 kB  00:00:00     
(1/9): epel/x86_64/group_gz                                                                                                                                                                 |  99 kB  00:00:08     
(2/9): base/7/x86_64/group_gz                                                                                                                                                               | 153 kB  00:00:08     
(3/9): docker-ce-stable/7/x86_64/updateinfo                                                                                                                                                 |   55 B  00:00:08     
(4/9): docker-ce-stable/7/x86_64/primary_db                                                                                                                                                 | 117 kB  00:00:08     
(5/9): epel/x86_64/updateinfo                                                                                                                                                               | 1.0 MB  00:00:00     
(6/9): base/7/x86_64/primary_db                                                                                                                                                             | 6.1 MB  00:00:09     
(7/9): epel/x86_64/primary_db                                                                                                                                                               | 7.0 MB  00:00:00     
(8/9): extras/7/x86_64/primary_db                                                                                                                                                           | 250 kB  00:00:08     
(9/9): updates/7/x86_64/primary_db                                                                                                                                                          |  24 MB  00:00:10     
元数据缓存已建立
[root@k8s-master-10 ~]#

NTP配置：

[root@k8s-node-12 network-scripts]# yum install chrony -y
已加载插件：fastestmirror, langpacks
Loading mirror speeds from cached hostfile
软件包 chrony-3.4-1.el7.x86_64 已安装并且是最新版本
无须任何处理
[root@k8s-node-12 network-scripts]# systemctl start chronyd
[root@k8s-node-12 network-scripts]# systemctl enable chronyd
[root@k8s-node-12 network-scripts]# date
2023年 10月 24日 星期二 21:01:39 CST
[root@k8s-master-10 ~]# ntpdate -u ntp.aliyun.com
24 Oct 21:21:01 ntpdate[2672]: adjust time server 203.107.6.88 offset 0.005337 sec
[root@k8s-master-10 ~]# hwclock -w

修改Linux内核参数，开启数据包转发功能：

#容器跨主机通信，底层走的iptables，内核级别的数据包转发:
[root@k8s-master-10 ~]# cat <<EOF> /etc/sysctl.d/k8s.conf
> net.bridge.bridge-nf-call-ip6tables = 1
> net.bridge.bridge-nf-call-iptables = 1
> net.ipv4.ip_forward = 1
> vm.max_map_count=262144
> EOF

加载读取内核参数配置文件：
[root@k8s-node-12 network-scripts]# sysctl -p /etc/sysctl.d/k8s.conf
报错如下：
sysctl: cannot stat /proc/sys/net/bridge/bridge-nf-call-ip6tables: 没有那个文件或目录
sysctl: cannot stat /proc/sys/net/bridge/bridge-nf-call-iptables: 没有那个文件或目录
net.ipv4.ip_forward = 1
vm.max_map_count = 262144
解决方法：
root@k8s-node-12 network-scripts]# modprobe br_netfilter
[root@k8s-node-12 network-scripts]# sysctl -p
完美解决：
[root@k8s-node-12 network-scripts]# sysctl -p /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
vm.max_map_count = 262144

安装docker基础环境：

yum remove docker docker-common docker-selinux docker-engine -y
curl -o /etc/yum.repos.d/docker-ce.repo https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
yum makecache fast
yum list docker-ce --showduplicates
yum install docker-ce-19.03.15 docker-ce-cli-19.03.15 -y
mkdir -p /etc/docker/
vi daemon.json
{
  "registry-mirrors":["https://ms9glx6x.mirror.aliyuncs.com"],

  "exec-opts":["native.cgroupdriver=systemd"]
}

systemctl start docker && systemctl enable docker

安装K8S的初始化工具kubeadm命令（主节点执行）

# 安装k8s集群环境初始化的工具
# kubelet-1.19.3  #组件，增删改查pod在具体机器上，pod可以运行主节点上，node节点上；
# kubeadm-1.19.3  #k8s版本 1.19.3，自动拉取k8s基础组件镜像的一个工具；
# kubectl-1.19.3  #管理，维护k8s客户端和服务端交付的一个命令行工具；

所有机器执行

设置阿里云源
curl -o /etc/yum.repos.d/Centos-7.repo  http://mirrors.aliyun.com/repo/Centos-7.repo
curl -o /etc/yum.repos.d/docker-ce.repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo

cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes] 
name=Kubernetes 
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1 
gpgcheck=0 
repo_gpgcheck=0 
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg 
https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF

yum clean all && yum makecache

yum list kubeadm --showduplicates

yum install kubelet-1.19.3 kubeadm-1.19.3 kubectl-1.19.3 ipvsadm

k8s安装完毕后，设置所有节点的kubelet开机运行

#查看kubeadm版本信息，初始化k8s版本信息
[root@k8s-node-12 ~]#  kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.3", GitCommit:"1e11e4a2108024935ecfcb2912226cedeafd99df", GitTreeState:"clean", BuildDate:"2020-10-14T12:47:53Z", GoVersion:"go1.15.2", Compiler:"gc", Platform:"linux/amd64"}

设置开机自启
systemctl enable docker
systemctl enable kubelet

初始化k8s-master主节点（只在主节点执行）

kubeadm init \
--apiserver-advertise-address=192.168.208.128 \
--image-repository registry.aliyuncs.com/google_containers \
--kubernetes-version v1.19.3 \
--service-cidr=10.1.0.0/16 \
--pod-network-cidr=10.2.0.0/16 \
--service-dns-domain=cluster.local \
--ignore-preflight-errors=Swap \
--ignore-preflight-errors=NumCPU

k8s-master成功装好

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

===========================================================================================
#创建k8s集群配置文件
#制定了，默认的ssl证书在那？api-server的地址等
  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

===========================================================================================
#pod分布再多机器上，pod互相之间链接，得部署，集群网络，选用flannel网络插件
#安装使用即可。
You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

#使用如下命令，将k8s-node加入集群即可
Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 192.168.208.128:6443 --token n8zyxz.7w7kbbd1rjiygo21 \
    --discovery-token-ca-cert-hash sha256:e0f3fa4a1f0b2edd106828f586f7d9d78cb6ca51334f05456cc70a7ead7bceac

查看工作节点信息

[root@k8s-master-10 docker]# kubectl get nodes
NAME            STATUS     ROLES    AGE   VERSION
k8s-master-10   NotReady   master   26m   v1.19.3
[root@k8s-master-10 docker]# kubectl get nodes -owide
NAME            STATUS     ROLES    AGE   VERSION   INTERNAL-IP       EXTERNAL-IP   OS-IMAGE                KERNEL-VERSION           CONTAINER-RUNTIME
k8s-master-10   NotReady   master   27m   v1.19.3   192.168.208.128   <none>        CentOS Linux 7 (Core)   3.10.0-1160.el7.x86_64   docker://19.3.15
NotReady：未就绪，因为网络环境没配置；

其他节点加入集群

kubeadm join 192.168.208.128:6443 --token n8zyxz.7w7kbbd1rjiygo21 \
    --discovery-token-ca-cert-hash sha256:e0f3fa4a1f0b2edd106828f586f7d9d78cb6ca51334f05456cc70a7ead7bceac

加入集群后，查看工作节点信息。查看k8s集群用到了那些节点。

[root@k8s-master-10 docker]# kubectl get nodes -owide
NAME            STATUS     ROLES    AGE     VERSION   INTERNAL-IP       EXTERNAL-IP   OS-IMAGE                KERNEL-VERSION           CONTAINER-RUNTIME
k8s-master-10   NotReady   master   41m     v1.19.3   192.168.208.128   <none>        CentOS Linux 7 (Core)   3.10.0-1160.el7.x86_64   docker://19.3.15
k8s-node-11     NotReady   <none>   8m21s   v1.19.3   192.168.208.130   <none>        CentOS Linux 7 (Core)   3.10.0-1160.el7.x86_64   docker://19.3.15
k8s-node-12     NotReady   <none>   8m14s   v1.19.3   192.168.208.129   <none>        CentOS Linux 7 (Core)   3.10.0-1160.el7.x86_64   docker://19.3.15

如何让集群就绪？部署网络插件

#1、下载网络插件，配置文件，yaml以及配置文件
git clone --depth 1 https://github.com/coreos/flannel.git
#2、在k8s主节点上，应用这个yaml，基于yaml，创建具体的pod过程
#3、如果需要修改pod运行网络话，要修改配置文件
/root/flannel-master/Documentation/kube-flannel.yml

#创建k8s资源，都是写这种yml文件了；
[root@k8s-master-10 Documentation]# grep 'Network' -A 5 kube-flannel.yml 
      "Network": "10.2.0.0/16",
      "Backend": {
        "Type": "vxlan"
      }
    }
---
--
      hostNetwork: true
      priorityClassName: system-node-critical
      tolerations:
      - operator: Exists
        effect: NoSchedule
      serviceAccountName: flannel
[root@k8s-master-10 Documentation]

#修改第二处，夸主机的容器通信，最终不得走宿主机的物理网卡。
#告诉flannel物理网卡是谁
containers:
       - name: kube-flannel
         image: docker.io/flannel/flannel:v0.22.3
         command:
         - /opt/bin/flanneld
         args:
         - --ip-masq
         - --kube-subnet-mgr
         - --iface=ens33

基于kubectl命令，应用这个yml文件，读取、以及创建pod资源

配置k8s命令补缺（重要）

k8s命令太多，务必要配置补全
yum install bash-completion -y
source /usr/share/bash-completion/bash_completion
source <(kubectl completion bash)
echo "source <(kubectl completion bash)" >> ~/.bashrc

kubectl create -f ./kube-flannel.yml

查看当前机器的容器，关于flannel网络插件的进程

[root@k8s-master-10 Documentation]# docker ps | grep flannel
53844bcdd167   e23f7ca36333                                        "/opt/bin/flanneld -..."   8 minutes ago   Up 8 minutes             k8s_kube-flannel_kube-flannel-ds-mpgf6_kube-flannel_9cf25cba-14b6-4768-8ff0-6ec821f22769_0
4a86528f1db0   registry.aliyuncs.com/google_containers/pause:3.2   "/pause"                  9 minutes ago   Up 9 minutes             k8s_POD_kube-flannel-ds-mpgf6_kube-flannel_9cf25cba-14b6-4768-8ff0-6ec821f22769_0

status状态都是Ready，至此，所有机器，都走flannel进行集群通信了。

[root@k8s-master-10 Documentation]# kubectl get nodes 
NAME            STATUS   ROLES    AGE     VERSION
k8s-master-10   Ready    master   4h53m   v1.19.3
k8s-node-11     Ready    <none>   4h20m   v1.19.3
k8s-node-12     Ready    <none>   4h20m   v1.19.3
[root@k8s-master-10 Documentation]# kubectl get nodes -o wide
NAME            STATUS   ROLES    AGE     VERSION   INTERNAL-IP       EXTERNAL-IP   OS-IMAGE                KERNEL-VERSION           CONTAINER-RUNTIME
k8s-master-10   Ready    master   4h53m   v1.19.3   192.168.208.128   <none>        CentOS Linux 7 (Core)   3.10.0-1160.el7.x86_64   docker://19.3.15
k8s-node-11     Ready    <none>   4h21m   v1.19.3   192.168.208.130   <none>        CentOS Linux 7 (Core)   3.10.0-1160.el7.x86_64   docker://19.3.15
k8s-node-12     Ready    <none>   4h21m   v1.19.3   192.168.208.129   <none>        CentOS Linux 7 (Core)   3.10.0-1160.el7.x86_64   docker://19.3.15

创建一个Pod

[root@k8s-master-10 Documentation]# kubectl run linux0224-pod-1-nginx --image=nginx:1.14.1
pod/linux0224-pod-1-nginx created

查看pod信息

[root@k8s-master-10 Documentation]# kubectl get pods -owide
NAME                    READY   STATUS    RESTARTS   AGE    IP         NODE          NOMINATED NODE   READINESS GATES
linux0224-pod-1-nginx   1/1     Running   0          107s   10.2.1.2   k8s-node-11   <none>           <none>

pod的ip是k8s集群，才能访问的一个ip，无法在外部访问，外部访问得设置更多访问规则

[root@k8s-master-10 Documentation]# curl 10.2.1.2
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>
[root@k8s-master-10 Documentation]# 

[root@k8s-master-10 Documentation]# curl 10.2.1.2 -I
HTTP/1.1 200 OK
Server: nginx/1.14.1
Date: Sun, 29 Oct 2023 12:44:34 GMT
Content-Type: text/html
Content-Length: 612
Last-Modified: Tue, 06 Nov 2018 13:28:31 GMT
Connection: keep-alive
ETag: "5be196ff-264"
Accept-Ranges: bytes

修改页面显示信息

直接在容器上修改
[root@k8s-node-11 docker]# docker exec 9395505ba67d sh -c "echo '<meta charset-utf-8> 书季辛苦了~' > /usr/share/nginx/html/index.html "

[root@k8s-master-10 Documentation]# curl 10.2.1.2
<meta charset-utf-8> 书季辛苦了~

容器集群上修改，基于k8s命令
[root@k8s-master-10 Documentation]# kubectl exec linux0224-pod-1-nginx -- sh -c "echo '辛苦了书季，散会，下午好好消化下' >/usr/share/nginx/html/index.html"
[root@k8s-master-10 Documentation]# curl 10.2.1.2
辛苦了书季，散会，下午好好消化下
相关推荐
PyAIGCMaster5 分钟前
ubuntu装P104驱动
linux·运维·ubuntu
奈何不吃鱼6 分钟前
【Linux】ubuntu依赖安装的各种问题汇总
linux·运维·服务器
icy、泡芙8 分钟前
T527-----音频调试
linux·驱动开发·音视频
aherhuo11 分钟前
kubevirt网络
linux·云原生·容器·kubernetes
zzzhpzhpzzz20 分钟前
Ubuntu如何查看硬件型号
linux·运维·ubuntu
蜜獾云22 分钟前
linux firewalld 命令详解
linux·运维·服务器·网络·windows·网络安全·firewalld
陌北v124 分钟前
Docker Compose 配置指南
运维·docker·容器·docker-compose
只会copy的搬运工1 小时前
Jenkins 持续集成部署——Jenkins实战与运维（1）
运维·ci/cd·jenkins
o(╥﹏╥)1 小时前
linux（ubuntu ）卡死怎么强制重启
linux·数据库·ubuntu·系统安全
娶不到胡一菲的汪大东1 小时前
Ubuntu概述
linux·运维·ubuntu
热门推荐
01构建高效的广告投放系统：应用架构的设计与实现02ARM学习（31）编译器对overlay方式的支持03玄机平台应急响应—webshell查杀04Cube-Studio：开源大模型全链路一站式中台05校验 GPT-4 真实性的三个经典问题：快速区分 GPT-3.5 与 GPT-4，并提供免费测试网站06Docker 夺命连环 15 问07〔AI 绘画〕Stable Diffusion 之 解决绘制多人或面部很小的人物时面部崩坏问题 篇08基于大语言模型（LLM）的合成数据生成、策展和评估的综述09(欧拉)openEuler系统添加网卡文件配置流程、(欧拉)openEuler系统手动配置ipv6地址流程、(欧拉)openEuler系统网络管理说明10Ubuntu 20.04使用Livox mid 360 测试 FAST_LIO