k8s云原生yaml 文件编排web-MySQL项目和kubeadm搭高可用集群(三主两从一VIP)-63148详解

cat > mysql-rc.yaml <<EOF
apiVersion: v1
kind: ReplicationController             # 类型是副本控制器
metadata:                               
  name: mysql                           # RC的名称全局是唯一的，这里name是mysql
spec:
  replicas: 1                           # 副本数量1
  selector:                             # RC管理 拥有label={app:mysql}这个标签的 pod
    app: mysql
  template:                             # template代表下面开始定义一个pod
    metadata:
      labels:
        app: mysql                      # 这个pod拥有{app:mysql}这样一个标签
    spec:                               
      containers:                       # pod中的容器定义部分
      - name: mysql 
        image: mysql:5.7
        ports:
        - containerPort: 3306           # 容器监听的端口
        env:                            # 注入容器里面的环境变量
        - name: MYSQL_ROOT_PASSWORD
          value: "123456"
EOF

以下是YAML定义文件的关键说明：

kind 属性用于标识资源类型，例如这里的**ReplicationController**表示这是一个RC（副本控制器）。

**spec**部分定义了RC的核心属性：

**spec.selector**是Pod标签选择器，用于监控和管理具有指定标签的Pod实例
**replicas=1**表示集群中始终保持1个MySQL Pod实例运行

当运行的Pod数量少于replicas时，RC会根据spec.template中的Pod模板创建新实例。特别注意：

**spec.template.metadata.labels必须与spec.selector**匹配
若标签不匹配，RC会持续尝试创建无法匹配的Pod，导致无限循环

[1-1-1 将RC发布到k8s集群里面去](#1-1-1 将RC发布到k8s集群里面去)

复制代码

kubectl apply -f mysql-rc.yaml

[1-1-2 查看集群中的pods](#1-1-2 查看集群中的pods)

复制代码

[root@k8s-master ~]# kubectl get pods
NAME          READY   STATUS    RESTARTS   AGE
mysql-hk852   1/1     Running   0          8s

为了保证系统可用性，我们可以通过增加多个RC副本来实现负载均衡。不过需要注意的是，数据库通常采用单实例部署，因为数据库负载均衡需要额外配置数据同步机制，否则会导致数据不一致问题。这里提到的多副本方案仅作演示用途

kubectl scale rc mysql --replicas=5
检查 Pod 状态

kubectl get pods

现在启动了5个mysql，负载均衡完成。

[1-2 创建 mysql-svc.yaml](#1-2 创建 mysql-svc.yaml)

最后，我们创建一个与之关联的kubenetes Service-MYSQL 的定义文件（文件名为：mysql-svc.yaml），完整的内容和解析如下；

复制代码

cat > mysql-svc.yaml << EOF
apiVersion: v1
kind: Service              # 类型是service
metadata:                  
  name: mysql              # 这个service的全局唯一名称
spec:
  ports:
    - port: 3306           # service提供服务的端口号
  selector:
    app: mysql             # 把拥有{app:label}这个标签的pod应用到这个服务里面
EOF

其中metadata.name 是Service的服务名（ServiceName）；port属性则定义了Service的端口；sepc.selector确定了那些Pod副本（实例）对应到本服务。类似地，我们通过kubectl apply 命令创建Service对象。

[1-2-1 将services应用到集群中](#1-2-1 将services应用到集群中)

光有RC是不行的，RC不会提供服务的，需要一个Service。将它应用到集群里面去。

复制代码

kubectl apply -f mysql-svc.yaml

[1-2-2 查看services](#1-2-2 查看services)

复制代码

kubectl get svc

注意到，MySQL服务被分配了一个值为 10.100.130.176 的Cluster IP地址。随后，Kubernetes集群中其他新创建的Pod就可以通过Service的Cluster IP+端口号3306来连接和访问它了。但是要注意，外部是无法去访问这个Mysql的，3306只是对其他pod开放的端口号。回忆一下，之前的pod,node架构图就知道了。

通常，Cluster IP是在Service创建后由Kubernetes系统自动分配的，其他Pod无法预先知道某个Service的Cluster IP地址，因此需要一个服务发现机制来找到这个服务。为此，最初时，Kubernetes巧妙地使用了Linux环境变量（Environment Variable）来解决这个问题，后面会详细说明其机制。现在只需知道，根据Service的唯一名称，容器可以从环境变量中获取Service对应的Cluster IP地址和端口，从而发起TCP/IP连接请求。这个地方挺像Nacos里面一样的，只要应用名称就可以自动访问到。不用配置什么具体的ip什么的。

[1-3 附加：如何找到并进入mysql数据库中？](#1-3 附加：如何找到并进入mysql数据库中？)

如果需要查看mysql的表或者数据的话，可以这样来看；

首先：查看mysql在哪个集群中，当然单机架构就没事了；

kubectl get pods -o wide

可以看到mysql在node2节点上，我们就可以去node2集群中去找到mysql-4fxv6这个容器；

切换到node2节点

[root@k8s-node2 ~]# docker ps -a | grep mysql-4fxv6

过滤出来了两个容器，选第一个就行，因为第二个是属于一个控制器，容器名为：k8s_mysql_mysql-4fxv6_default_a6be3c5d-72ef-40d9-b6b7-83e31c4f5d86_0

进入该容器

[root@k8s-node1 ~]# docker exec -it k8s_mysql_mysql-4fxv6_default_a6be3c5d-72ef-40d9-b6b7-83e31c4f5d86_0 bash
进入数据库

mysql -uroot -p123456

这样就进入到数据库中了，可以自行查看表结构或者增删改查数据之类的了。

还有一种进入数据库容器的方法；

使用方法：

复制代码

 kubectl exec -it pod名字 -n名称空间 bash

实例：

复制代码

kubectl exec -it mysql-4fxv6 bash

为什么这样进入数据库，用工具连接不好吗？

答：因为mysql端口没有对外，使用navicat等工具并不能连接上。

[2、启动 tomcat 项目](#2、启动 tomcat 项目)

这个tomcat是书里面特定的，所以说我们要先去拉取一下这个镜像，这个镜像是作者已经制定好了的。

[2-1 拉取特定的tomcat镜像](#2-1 拉取特定的tomcat镜像)

复制代码

docker pull kubeguide/tomcat-app:v1

拉取完成之后，写yaml文件。

[2-2 创建 tomcat-rc.yaml](#2-2 创建 tomcat-rc.yaml)

复制代码

cat > tomcat-rc.yaml << EOF
apiVersion: v1
kind: ReplicationController
metadata:
  name: myweb
spec:
  replicas: 5
  selector:
    app: myweb
  template:
    metadata:
      labels:
        app: myweb
    spec:
      containers:
      - name: myweb
        image: kubeguide/tomcat-app:v1
        ports:
        - containerPort: 8080
        env:
        - name: MYSQL_SERVICE_HOST
          value: 'mysql'
        - name: MYSQL_SERVICE_PORT
          value: '3306'
EOF

注意：Tomcat 容器内，应用将使用环境变量 MYSQL_ SERVICE_ HOST 的值连接 MYSQL 务。更安全可靠的用法是使用服务的名称"mysql"。

[2-2-1 将RC发布到k8s集群里面去](#2-2-1 将RC发布到k8s集群里面去)

复制代码

kubectl apply -f tomcat-rc.yaml

这里面有5个副本就不用在多创建了，5个就够了，当然自己想要更多可以根据自己的情况创建。

[2-2-2 查看集群中的pods](#2-2-2 查看集群中的pods)

复制代码

[root@k8s-master ~]# kubectl get pods | grep myweb
myweb-56tzt   1/1     Running   0          7m37s
myweb-7h9t8   1/1     Running   0          7m37s
myweb-kzq69   1/1     Running   0          7m37s
myweb-tncrc   1/1     Running   0          7m37s
myweb-wprk4   1/1     Running   0          7m37s

可以看到已经创建了5个pod，接下来就该创建对外开放的端口也就是services了。

[2-3 创建 tomcat-svc.yaml](#2-3 创建 tomcat-svc.yaml)

复制代码

cat > tomcat-svc.yaml << EOF
apiVersion: v1
kind: Service
metadata:
  name: myweb
spec:
  type: NodePort
  ports:
    - port: 8080
      nodePort: 30001  # 对外暴露端口，使得k8s之外的人可以访问到这个服务
  selector:
    app: myweb
EOF

type=NodePort和nodePort=30001的两个属性表明此Service开启了NodePort方式的外网访问模式。在Kubernetes集群之外，比如在本机的浏览器里，可以通过30001这个端口访问myweb（对应到8080的虚端口上）。

[2-3-1 将services应用到集群中](#2-3-1 将services应用到集群中)

光有RC是不行的，RC不会提供服务的，需要一个Service。将它应用到集群里面去。

复制代码

kubectl apply -f tomcat-svc.yaml

[2-3-2 查看services](#2-3-2 查看services)

复制代码

kubectl get svc

30001为映射的端口，是可以对外访问的。

3、通过浏览器访问

复制代码

 #ip填你的master或者node的地址
http://172.16.11.221:30001

如果看不到这个网页，那么可能有几个原因：比如防火墙的问题，无法访问 30001 端口，或者因为你是通过代理上网的，浏览器错把虛拟机的IP地址当成远程地址了。可以在虛拟机上直接运行 curl 127.0.0.1:30001 来验证此端口是否能被访问，如果还是不能访问，那么这肯定不是机器的问题...

复制代码

#ip填你的master或者node的地址
http://172.16.11.221:30001/demo/

接下来可以尝试单击 "Add.." 按钮添加一条记录并提交，如下图所示，提交以后，数据就被写入 MySQL 数据库中了。

提交为submit，提交成功会显示如下图；

如果操作失败，可能是因为开启了过多的数据库副本，导致数据写入时无法确定目标数据库。

点击返回即可查看刚刚添加的数据

至此，我们已经完成了 Kubernetes 上部署 Tomcat 的示例。虽然这个例子并不复杂，但通过与传统分布式应用部署方式的对比，我们深刻体会到 Kubernetes 的强大之处------仅需简单的配置文件和几条命令就能完成整个集群的部署，这种便捷性令人印象深刻。

下一节我们将系统学习 Kubernetes 的基本概念和术语。在此之前，建议读者可以继续深入研究本例中的以下扩展内容：

了解 RC（Replication Controller）和 Service 等配置文件的格式规范

Kubernetes高可用集群部署

准备工作（所有节点都要做同样的操作）

服务器配置

高可用集群（三主两从一VIP）	主机名	ip地址	配置	需要用到的服务
主	k8s-master1	172.16.11.215	2C/2G/50G	kubeadm,docker,keepalived,haproxy
主	k8s-master2	172.16.11.216	2C/2G/50G	kubeadm,docker,keepalived,haproxy
主	k8s-master3	172.16.11.217	2C/2G/50G	kubeadm,docker,keepalived,haproxy
从	k8s-node1	172.16.11.218	2C/2G/50G	kubeadm,docker
从	k8s-node2	172.16.11.219	2C/2G/50G	kubeadm,docker
VIP	k8s-vip	172.16.11.220	2C/1G/50G	什么都不用装

关闭防火墙

one-dark-pro 复制代码

<span style="background-color:#282c34"><span style="color:#abb2bf"><code class="language-bash"><span style="color:#61afef">systemctl</span><span style="color:#98c379"> stop</span><span style="color:#98c379"> firewalld</span><span style="color:#abb2bf"> && </span><span style="color:#61afef">systemctl</span><span style="color:#98c379"> disable</span><span style="color:#98c379"> firewalld</span></code></span></span>

如果在线上服务器之类的不能关闭防火墙，那就需要开启几个端口；（这里说的是k8所用到的端口）

master节点:

规则	端口范围	作用	使用者
TCP	6443*	Kubernetes API server	All
TCP	2379-2380	etcd server client API	kube-apiserver, etcd
TCP	10250	Kubelet API	Self, Control plane
TCP	10251	kube-scheduler	Self
TCP	10252	kube-controller-manager	Self

node节点：

规则	端口范围	作用	使用者
TCP	10252	Kubelet API	Self, Control plane
TCP	30000-32767	NodePort Services**	All

关闭selinux

临时关闭selinux（沙河）如需永久关闭selinux需要修改为sed -i 's/^SELINUX=enforcing$/SELINUX=disabled/' /etc/selinux/config

one-dark-pro 复制代码

<span style="background-color:#282c34"><span style="color:#abb2bf"><code class="language-bash"><span style="color:#7f848e"><em>#临时关闭selinux</em></span>
<span style="color:#61afef">setenforce</span><span style="color:#d19a66"> 0</span>

<span style="color:#7f848e"><em>#永久关闭selinux</em></span>
<span style="color:#61afef">sed</span><span style="color:#d19a66"> -i</span><span style="color:#98c379"> 's/^SELINUX=enforcing$/SELINUX=permissive/'</span><span style="color:#98c379"> /etc/selinux/config</span></code></span></span>

关闭交换分区

one-dark-pro 复制代码

<span style="background-color:#282c34"><span style="color:#abb2bf"><code class="language-bash"><span style="color:#7f848e"><em>#临时关闭所有的交换分区</em></span>
<span style="color:#61afef">swapoff</span><span style="color:#d19a66"> -a</span>

<span style="color:#7f848e"><em>#永久关闭所有的交换分区</em></span>
<span style="color:#61afef">sed</span><span style="color:#d19a66"> -i</span><span style="color:#98c379"> '/swap/s/^\(.*\)$/#\1/g'</span><span style="color:#98c379"> /etc/fstab</span></code></span></span>

修改六台高可用集群的主机名：（每个主机限一条命令）

one-dark-pro 复制代码

<span style="background-color:#282c34"><span style="color:#abb2bf"><code class="language-bash"><span style="color:#abb2bf">[root@k8s-master1 ~]# hostnamectl set-hostname k8s-master1</span>
<span style="color:#abb2bf">[root@k8s-master2 ~]# hostnamectl set-hostname k8s-master2</span>
<span style="color:#abb2bf">[root@k8s-master3 ~]# hostnamectl set-hostname k8s-master3</span>
<span style="color:#abb2bf">[root@k8s-node1 ~]# hostnamectl set-hostname k8s-node1</span>
<span style="color:#abb2bf">[root@k8s-node2 ~]# hostnamectl set-hostname k8s-node2</span>
<span style="color:#abb2bf">[root@k8s-vip ~]# hostnamectl set-hostname k8s-vip</span></code></span></span>

所有节点都添加集群ip与主机名到hosts中：

one-dark-pro 复制代码

<span style="background-color:#282c34"><span style="color:#abb2bf"><code class="language-bash"><span style="color:#61afef">cat</span><span style="color:#abb2bf"> >> </span><span style="color:#98c379">/etc/hosts</span><span style="color:#abb2bf"> << </span><span style="color:#abb2bf">EOF</span> 
<span style="color:#98c379">172.16.11.215 k8s-master1</span>
<span style="color:#98c379">172.16.11.216 k8s-master2</span>
<span style="color:#98c379">172.16.11.217 k8s-master3</span>
<span style="color:#98c379">172.16.11.218 k8s-node1</span>
<span style="color:#98c379">172.16.11.219 k8s-node2</span>
<span style="color:#98c379">172.16.11.220 k8s-vip</span>
<span style="color:#abb2bf">EOF</span></code></span></span>

注意：ip一定要改成自己的ip，不要直接复制粘贴

六台机器进行时间同步

one-dark-pro 复制代码

<span style="background-color:#282c34"><span style="color:#abb2bf"><code class="language-bash"><span style="color:#7f848e"><em>#安裝同步时间命令</em></span>
<span style="color:#61afef">yum</span><span style="color:#98c379"> install</span><span style="color:#98c379"> ntpdate</span><span style="color:#d19a66"> -y</span>

<span style="color:#7f848e"><em>#同步时间</em></span>
<span style="color:#61afef">ntpdate</span><span style="color:#98c379"> cn.pool.ntp.org</span>

<span style="color:#7f848e"><em>#设置定时任务每五分钟同步一次时间</em></span>
<span style="color:#56b6c2">echo</span><span style="color:#98c379"> "*/5 * * * * root /usr/sbin/ntpdate cn.pool.ntp.org &>/dev/null"</span><span style="color:#abb2bf"> >> </span><span style="color:#98c379">/etc/crontab</span></code></span></span>

特殊说明:

如果是克隆虚拟机建议执行rm -rf /etc/udev/* 保证网卡UUID不同

六台都安装需要的一些命令：

添加centos源并将下载地址更换为阿里云地址

one-dark-pro 复制代码

<span style="background-color:#282c34"><span style="color:#abb2bf"><code class="language-bash"><span style="color:#7f848e"><em>#添加centos源</em></span>
<span style="color:#61afef">curl</span><span style="color:#d19a66"> -o</span><span style="color:#98c379"> /etc/yum.repos.d/CentOS-Base.repo</span><span style="color:#98c379"> https://mirrors.aliyun.com/repo/Centos-7.repo</span>

<span style="color:#7f848e"><em>#将下载地址更换为阿里云地址</em></span>
<span style="color:#61afef">sed</span><span style="color:#d19a66"> -i</span><span style="color:#d19a66"> -e</span><span style="color:#98c379"> '/mirrors.cloud.aliyuncs.com/d'</span><span style="color:#d19a66"> -e</span><span style="color:#98c379"> '/mirrors.aliyuncs.com/d'</span><span style="color:#98c379"> /etc/yum.repos.d/CentOS-Base.repo</span></code></span></span>

添加epel扩展源

one-dark-pro 复制代码

<span style="background-color:#282c34"><span style="color:#abb2bf"><code class="language-bash"><span style="color:#61afef">curl</span><span style="color:#d19a66"> -o</span><span style="color:#98c379"> /etc/yum.repos.d/epel.repo</span><span style="color:#98c379"> http://mirrors.aliyun.com/repo/epel-7.repo</span></code></span></span>

清除缓存

one-dark-pro 复制代码

<span style="background-color:#282c34"><span style="color:#abb2bf"><code class="language-bash"><span style="color:#61afef">yum</span><span style="color:#98c379"> clean</span><span style="color:#98c379"> all</span></code></span></span>

重新加载源缓存

one-dark-pro 复制代码

<span style="background-color:#282c34"><span style="color:#abb2bf"><code class="language-bash"><span style="color:#61afef">yum</span><span style="color:#98c379"> makecache</span></code></span></span>

升级yum并安装一些会用到的命令

one-dark-pro 复制代码

<span style="background-color:#282c34"><span style="color:#abb2bf"><code class="language-bash"><span style="color:#61afef">yum</span><span style="color:#d19a66"> -y</span><span style="color:#98c379"> update</span><span style="color:#abb2bf"> && </span><span style="color:#61afef">yum</span><span style="color:#d19a66"> -y</span><span style="color:#98c379"> install</span><span style="color:#98c379"> lrzsz</span><span style="color:#98c379"> wget</span><span style="color:#98c379"> conntrack</span><span style="color:#98c379"> ipvsadm</span><span style="color:#98c379"> ipset</span><span style="color:#98c379"> jq</span><span style="color:#98c379"> psmisc</span><span style="color:#98c379"> sysstat</span><span style="color:#98c379"> curl</span><span style="color:#98c379"> iptables</span><span style="color:#98c379"> net-tools</span><span style="color:#98c379"> libseccomp</span><span style="color:#98c379"> gcc</span><span style="color:#98c379"> gcc-c++</span><span style="color:#98c379"> yum-utils</span><span style="color:#98c379"> device-mapper-persistent-data</span><span style="color:#98c379"> lvm2</span><span style="color:#98c379"> bash-completion</span><span style="color:#98c379"> sshpass</span></code></span></span>

安装需要一些时间，就等待安装即可；

调整能打开文件数大小

one-dark-pro 复制代码

<span style="background-color:#282c34"><span style="color:#abb2bf"><code class="language-bash"><span style="color:#abb2bf">[root@k8s-master1 ~]# ulimit -SHn 65535</span>

<span style="color:#abb2bf">[root@k8s-master1 ~]# cat >> /etc/security/limits.conf << </span><span style="color:#abb2bf">EOF</span> 
<span style="color:#98c379">* soft nofile 655360</span>
<span style="color:#98c379">* hard nofile 131072</span>
<span style="color:#98c379">* soft nproc 65535</span>
<span style="color:#98c379">* hard nproc 655350</span>
<span style="color:#98c379">* soft memlock unlimited</span>
<span style="color:#98c379">* hard memlock unlimited</span>
<span style="color:#abb2bf">EOF</span></code></span></span>

补充说明:

ulimit用于限制shell启动进程所占用的资源，支持以下各种类型的限制：所创建的内核文件的大小、进程数据块的大小、shell

进程创建文件的大小、内存锁住的大小、常驻内存集的大小、打开文件描述符的数量、分配堆栈的最大大小、CPU

时间、单个用户的最大线程数、Shell 进程所能使用的最大虚拟内存。同时，它支持硬资源和软资源的限制。

hard:严格的设定，必定不能超过这个设定的数值 soft:警告的设定，可以超过这个设定值，但是若超过则有警告信息限制资源:

core -- 限制内核文件的大小

date -- 最大数据大小

fsize -- 最大文件大小

memlock -- 最大锁定内存地址空间

nofile -- 打开文件的最大数目

rss -- 最大持久设置大小

stack -- 最大栈大小

cpu -- 以分钟为单位的最多 CPU 时间

noproc -- 进程的最大数目（系统的最大进程数）

as -- 地址空间限制 maxlogins -- 此用户允许登录的最大数目

安装配置ipvsadm

one-dark-pro 复制代码

<span style="background-color:#282c34"><span style="color:#abb2bf"><code class="language-bash"><span style="color:#7f848e"><em>#安装所需的命令（上面已经全部安装了，可以不用管）</em></span>
<span style="color:#abb2bf">[root@k8s-master1 ~]# yum -y install ipvsadm ipset sysstat conntrack libseccomp</span>

<span style="color:#7f848e"><em>#加载ipvs相关模块</em></span>
<span style="color:#abb2bf">[root@k8s-master1 ~]# modprobe -- ip_vs</span>
<span style="color:#abb2bf">[root@k8s-master1 ~]# modprobe -- ip_vs_rr</span>
<span style="color:#abb2bf">[root@k8s-master1 ~]# modprobe -- ip_vs_wrr</span>
<span style="color:#abb2bf">[root@k8s-master1 ~]# modprobe -- ip_vs_sh</span>
<span style="color:#abb2bf">[root@k8s-master1 ~]# modprobe -- nf_conntrack_ipv4</span>
<span style="color:#abb2bf">[root@k8s-master1 ~]# cat > /etc/modules-load.d/ipvs.conf << </span><span style="color:#abb2bf">EOF</span>
<span style="color:#98c379">ip_vs</span>
<span style="color:#98c379">ip_vs_lc</span>
<span style="color:#98c379">ip_vs_wlc</span>
<span style="color:#98c379">ip_vs_rr</span>
<span style="color:#98c379">ip_vs_wrr</span>
<span style="color:#98c379">ip_vs_lblc</span>
<span style="color:#98c379">ip_vs_lblcr</span>
<span style="color:#98c379">ip_vs_dh</span>
<span style="color:#98c379">ip_vs_sh</span>
<span style="color:#98c379">ip_vs_nq</span>
<span style="color:#98c379">ip_vs_sed</span>
<span style="color:#98c379">ip_vs_ftp</span>
<span style="color:#98c379">ip_vs_sh</span>
<span style="color:#98c379">nf_conntrack_ipv4</span>
<span style="color:#98c379">ip_tables</span>
<span style="color:#98c379">ip_set</span>
<span style="color:#98c379">xt_set</span>
<span style="color:#98c379">ipt_set</span>
<span style="color:#98c379">ipt_rpfilter</span>
<span style="color:#98c379">ipt_REJECT</span>
<span style="color:#98c379">ipip</span>
<span style="color:#abb2bf">EOF</span>

<span style="color:#abb2bf">[root@k8s-master1 ~]# systemctl enable --now systemd-modules-load.service</span></code></span></span>

[高可用集群设置免密登录（在 master1 节点上操作）](#高可用集群设置免密登录（在 master1 节点上操作）)

在master1节点免密钥登录其他节点，安装过程中生成配置文件和证书均在master1上操作，集群管理也在master11上操作，阿里云或者AWS上需要单独一台kubectl服务器。

one-dark-pro 复制代码

<span style="background-color:#282c34"><span style="color:#abb2bf"><code class="language-bash"><span style="color:#7f848e"><em>#生成密钥</em></span>
<span style="color:#abb2bf">[root@k8s-master1 ~]# ssh-keygen -t rsa -P </span><span style="color:#98c379">''</span><span style="color:#abb2bf"> -f /root/.ssh/id_rsa &> /dev/null</span>
<span style="color:#7f848e"><em>#循环给高可用集群进行免密设置</em></span>
<span style="color:#abb2bf">[root@k8s-master1 ~]# </span><span style="color:#c678dd">for</span><span style="color:#e06c75"> i</span><span style="color:#c678dd"> in</span><span style="color:#98c379"> k8s-master1</span><span style="color:#98c379"> k8s-master2</span><span style="color:#98c379"> k8s-master3</span><span style="color:#98c379"> k8s-node1</span><span style="color:#98c379"> k8s-node2</span><span style="color:#abb2bf">;</span><span style="color:#c678dd">do</span><span style="color:#abb2bf"> sshpass -p </span><span style="color:#98c379">'123123'</span><span style="color:#abb2bf"> ssh-copy-id -o </span><span style="color:#e06c75">StrictHostKeyChecking</span><span style="color:#56b6c2">=</span><span style="color:#98c379">'no'</span><span style="color:#61afef"> -i</span><span style="color:#98c379"> .ssh/id_rsa.pub</span><span style="color:#e06c75"> $i</span><span style="color:#abb2bf">;</span><span style="color:#c678dd">done</span>

<span style="color:#7f848e"><em>#完成之后最好测试以下免密登录</em></span></code></span></span>

如下图就属于成功；

[部署 docker（所有节点都需要部署）](#部署 docker（所有节点都需要部署）)

one-dark-pro 复制代码

<span style="background-color:#282c34"><span style="color:#abb2bf"><code class="language-bash"><span style="color:#7f848e"><em>#安装docker所需的依赖包</em></span>
<span style="color:#abb2bf">[root@docker ~]# yum install -y yum-utils device-mapper-persistent-data lvm2	</span>

<span style="color:#7f848e"><em>#添加阿里云的docker镜像地址</em></span>
<span style="color:#abb2bf">[root@docker ~]# sudo yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo</span>
<span style="color:#abb2bf">&&#或者（二选一即可）</span>
<span style="color:#abb2bf">[root@docker ~]# wget https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo -O /etc/yum.repos.d/docker-ce.repo</span>

<span style="color:#7f848e"><em>#更新缓存，只处理新添加的yum源缓存</em></span>
<span style="color:#abb2bf">[root@docker ~]# yum makecache fast</span>

<span style="color:#7f848e"><em>#部署docker，默认安装最新版本</em></span>
<span style="color:#abb2bf">[root@docker ~]# yum install -y docker-ce-20.10.14 docker-ce-cli-20.10.14 containerd.io</span>

<span style="color:#7f848e"><em>#查看安装docker版本</em></span>
<span style="color:#abb2bf">[root@docker ~]# docker --version(</span><span style="color:#61afef">或者使用docker</span><span style="color:#98c379"> version</span><span style="color:#abb2bf">)</span>
<span style="color:#61afef">Docker</span><span style="color:#98c379"> version</span><span style="color:#98c379"> 20.10.14,</span><span style="color:#98c379"> build</span><span style="color:#98c379"> a224086</span>

<span style="color:#7f848e"><em>#加载docker配置</em></span>
<span style="color:#abb2bf">[root@docker ~]# systemctl daemon-reload</span>
<span style="color:#7f848e"><em>#启动docker服务并设置开机自启</em></span>
<span style="color:#abb2bf">[root@docker ~]# systemctl start docker && </span><span style="color:#61afef">systemctl</span><span style="color:#98c379"> enable</span><span style="color:#98c379"> docker</span>

<span style="color:#7f848e"><em>#查看docker可以安装的版本，也可以自己安装指定版本，yum -y install docker-ce-19.03.12.el7</em></span>
<span style="color:#abb2bf">[root@docker ~]# yum list docker-ce --showduplicates | </span><span style="color:#61afef">sort</span><span style="color:#d19a66"> -r</span></code></span></span>

给docker添加镜像加速器及cgroup并重启docker服务

one-dark-pro 复制代码

<span style="background-color:#282c34"><span style="color:#abb2bf"><code class="language-bash"><span style="color:#abb2bf">[root@docker ~]# mkdir -p /etc/docker</span>
<span style="color:#abb2bf">[root@docker ~]# tee /etc/docker/daemon.json <<-</span><span style="color:#abb2bf">'EOF'</span>
<span style="color:#98c379">{</span>
<span style="color:#98c379">  "registry-mirrors": ["https://mrlmpasq.mirror.aliyuncs.com"],</span>
<span style="color:#98c379">  "exec-opts": ["native.cgroupdriver=systemd"]</span>
<span style="color:#98c379">}</span>
<span style="color:#abb2bf">EOF</span>
<span style="color:#7f848e"><em>#由于新版kubelet建议使用systemd，所以可以把docker的CgroupDriver改成systemd</em></span>

<span style="color:#7f848e"><em>#重新加载docker配置</em></span>
<span style="color:#abb2bf">[root@docker ~]# systemctl daemon-reload</span>
<span style="color:#7f848e"><em>#重新启动docker服务</em></span>
<span style="color:#abb2bf">[root@docker ~]# systemctl restart docker</span></code></span></span>

docker部署完成

[部署 kubernetes（所有节点都要部署）](#部署 kubernetes（所有节点都要部署）)

配置相关的内核参数

将桥接的IPv4 流量传递到iptables 的链

one-dark-pro 复制代码

<span style="background-color:#282c34"><span style="color:#abb2bf"><code class="language-bash"><span style="color:#61afef">cat</span><span style="color:#abb2bf"> <<</span><span style="color:#abb2bf">EOF</span><span style="color:#abb2bf"> >> </span><span style="color:#98c379">/etc/sysctl.d/k8s.conf</span>
<span style="color:#98c379">net.ipv4.ip_forward = 1</span>
<span style="color:#98c379">net.bridge.bridge-nf-call-iptables = 1</span>
<span style="color:#98c379">net.bridge.bridge-nf-call-ip6tables = 1</span>
<span style="color:#98c379">fs.may_detach_mounts = 1</span>
<span style="color:#98c379">vm.overcommit_memory=1</span>
<span style="color:#98c379">vm.panic_on_oom=0</span>
<span style="color:#98c379">fs.inotify.max_user_watches=89100</span>
<span style="color:#98c379">fs.file-max=52706963</span>
<span style="color:#98c379">fs.nr_open=52706963</span>
<span style="color:#98c379">net.netfilter.nf_conntrack_max=2310720</span>
<span style="color:#98c379">net.ipv4.tcp_keepalive_time = 600</span>
<span style="color:#98c379">net.ipv4.tcp_keepalive_probes = 3</span>
<span style="color:#98c379">net.ipv4.tcp_keepalive_intvl =15</span>
<span style="color:#98c379">net.ipv4.tcp_max_tw_buckets = 36000</span>
<span style="color:#98c379">net.ipv4.tcp_tw_reuse = 1</span>
<span style="color:#98c379">net.ipv4.tcp_max_orphans = 327680</span>
<span style="color:#98c379">net.ipv4.tcp_orphan_retries = 3</span>
<span style="color:#98c379">net.ipv4.tcp_syncookies = 1</span>
<span style="color:#98c379">net.ipv4.tcp_max_syn_backlog = 16384</span>
<span style="color:#98c379">net.ipv4.ip_conntrack_max = 65536</span>
<span style="color:#98c379">net.ipv4.tcp_max_syn_backlog = 16384</span>
<span style="color:#98c379">net.ipv4.tcp_timestamps = 0</span>
<span style="color:#98c379">net.core.somaxconn = 16384</span>
<span style="color:#abb2bf">EOF</span>

<span style="color:#7f848e"><em>#让其生效</em></span>
<span style="color:#61afef">sysctl</span><span style="color:#d19a66"> --system</span></code></span></span>

[添加 k8s yum源](#添加 k8s yum源)

one-dark-pro 复制代码

<span style="background-color:#282c34"><span style="color:#abb2bf"><code class="language-bash"><span style="color:#abb2bf">[root@docker ~]# cat > /etc/yum.repos.d/kubernetes.repo << </span><span style="color:#abb2bf">EOF</span>
<span style="color:#98c379">[kubernetes]</span>
<span style="color:#98c379">name=Kubernetes</span>
<span style="color:#98c379">baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/</span>
<span style="color:#98c379">enabled=1</span>
<span style="color:#98c379">gpgcheck=0</span>
<span style="color:#98c379">repo_gpgcheck=0</span>
<span style="color:#98c379">gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg</span>
<span style="color:#abb2bf">EOF</span>

<span style="color:#7f848e"><em>#重新加载缓存</em></span>
<span style="color:#61afef">yum</span><span style="color:#98c379"> makecache</span><span style="color:#98c379"> fast</span></code></span></span>

[安装 kubeadm kubelet kubectl](#安装 kubeadm kubelet kubectl)

注：安装这几个版本不要用最新的，容易出问题

one-dark-pro 复制代码

<span style="background-color:#282c34"><span style="color:#abb2bf"><code class="language-bash"><span style="color:#61afef">yum</span><span style="color:#98c379"> install</span><span style="color:#d19a66"> -y</span><span style="color:#98c379"> kubeadm-1.20.0-0</span><span style="color:#98c379"> kubelet-1.20.0-0</span><span style="color:#98c379"> kubectl-1.20.0-0</span>

<span style="color:#7f848e"><em>#查看kubeadm版本</em></span>
<span style="color:#61afef">kubeadm</span><span style="color:#98c379"> version</span></code></span></span>

启动kubelet并设置开机自启

one-dark-pro 复制代码

<span style="background-color:#282c34"><span style="color:#abb2bf"><code class="language-bash"><span style="color:#61afef">systemctl</span><span style="color:#98c379"> enable</span><span style="color:#98c379"> kubelet</span><span style="color:#abb2bf"> && </span><span style="color:#61afef">systemctl</span><span style="color:#98c379"> start</span><span style="color:#98c379"> kubelet</span></code></span></span>

kubernetes强化tab（安装之后会tab可以补全命令及参数）

配置环境

one-dark-pro 复制代码

<span style="background-color:#282c34"><span style="color:#abb2bf"><code class="language-bash"><span style="color:#56b6c2">echo</span><span style="color:#98c379"> 'source  <(kubectl  completion  bash)'</span><span style="color:#abb2bf"> >> </span><span style="color:#98c379">~/.bashrc</span></code></span></span>

1、退出连接，重新连接；

2、或者bash更新环境就可以使用了。

[部署keepalived、haproxy --> 配置高可用（所有主节点部署）](#部署keepalived、haproxy --> 配置高可用（所有主节点部署）)

这里使用高可用和负载的组件为haproxy和keepalived，如果kubernets不是高可用架构，可以不做haproxy与keepalived。如果是公有云平台部署可以选择公用云自带的负载均衡来代替haproxy和keepalived，比如阿里云的SLB,或者腾讯云的ELB(大部分公有云不支持keepalived)。如果使用的是阿里云，kubectl控制端不能放在master节点，因为阿里云SLB有回环问题，也就是说SLB代理的服务器不能反向访问这个问题，但是腾讯云修复了这个问题。

再次提醒，所有主节点，不是所有节点；

安装 keepalived 和 haproxy

one-dark-pro 复制代码

<span style="background-color:#282c34"><span style="color:#abb2bf"><code class="language-bash"><span style="color:#61afef">yum</span><span style="color:#d19a66"> -y</span><span style="color:#98c379"> install</span><span style="color:#98c379"> keepalived</span><span style="color:#98c379"> haproxy</span></code></span></span>

[所有主节点修改haproxy配置](#所有主节点修改haproxy配置)

最后的server改为自己的主节点ip

one-dark-pro 复制代码

<span style="background-color:#282c34"><span style="color:#abb2bf"><code class="language-bash"><span style="color:#abb2bf">[root@k8s-master1 ~]# cat /etc/haproxy/haproxy.cfg</span>
<span style="color:#61afef">global</span>
<span style="color:#61afef">  maxconn</span><span style="color:#d19a66">  2000</span>
<span style="color:#61afef">  ulimit-n</span><span style="color:#d19a66">  16384</span>
<span style="color:#61afef">  log</span><span style="color:#d19a66">  127.0.0.1</span><span style="color:#98c379"> local0</span><span style="color:#98c379"> err</span>
<span style="color:#61afef">  stats</span><span style="color:#98c379"> timeout</span><span style="color:#98c379"> 30s</span>

<span style="color:#61afef">defaults</span>
<span style="color:#61afef">  log</span><span style="color:#98c379"> global</span>
<span style="color:#61afef">  mode</span><span style="color:#98c379">  http</span>
<span style="color:#61afef">  option</span><span style="color:#98c379">  httplog</span>
<span style="color:#61afef">  timeout</span><span style="color:#98c379"> connect</span><span style="color:#d19a66"> 5000</span>
<span style="color:#61afef">  timeout</span><span style="color:#98c379"> client</span><span style="color:#d19a66">  50000</span>
<span style="color:#61afef">  timeout</span><span style="color:#98c379"> server</span><span style="color:#d19a66">  50000</span>
<span style="color:#61afef">  timeout</span><span style="color:#98c379"> http-request</span><span style="color:#98c379"> 15s</span>
<span style="color:#61afef">  timeout</span><span style="color:#98c379"> http-keep-alive</span><span style="color:#98c379"> 15s</span>

<span style="color:#61afef">frontend</span><span style="color:#98c379"> monitor-in</span>
<span style="color:#61afef">  bind</span><span style="color:#e5c07b"> *</span><span style="color:#98c379">:33305</span>
<span style="color:#61afef">  mode</span><span style="color:#98c379"> http</span>
<span style="color:#61afef">  option</span><span style="color:#98c379"> httplog</span>
<span style="color:#61afef">  monitor-uri</span><span style="color:#98c379"> /monitor</span>

<span style="color:#61afef">frontend</span><span style="color:#98c379"> k8s-master</span>
<span style="color:#61afef">  bind</span><span style="color:#98c379"> 0.0.0.0:16443</span>
<span style="color:#61afef">  bind</span><span style="color:#98c379"> 127.0.0.1:16443</span>
<span style="color:#61afef">  mode</span><span style="color:#98c379"> tcp</span>
<span style="color:#61afef">  option</span><span style="color:#98c379"> tcplog</span>
<span style="color:#61afef">  tcp-request</span><span style="color:#98c379"> inspect-delay</span><span style="color:#98c379"> 5s</span>
<span style="color:#61afef">  default_backend</span><span style="color:#98c379"> k8s-master</span>

<span style="color:#61afef">backend</span><span style="color:#98c379"> k8s-master</span>
<span style="color:#61afef">  mode</span><span style="color:#98c379"> tcp</span>
<span style="color:#61afef">  option</span><span style="color:#98c379"> tcplog</span>
<span style="color:#61afef">  option</span><span style="color:#98c379"> tcp-check</span>
<span style="color:#61afef">  balance</span><span style="color:#98c379"> roundrobin</span>
<span style="color:#61afef">  default-server</span><span style="color:#98c379"> inter</span><span style="color:#98c379"> 10s</span><span style="color:#98c379"> downinter</span><span style="color:#98c379"> 5s</span><span style="color:#98c379"> rise</span><span style="color:#d19a66"> 2</span><span style="color:#98c379"> fall</span><span style="color:#d19a66"> 2</span><span style="color:#98c379"> slowstart</span><span style="color:#98c379"> 60s</span><span style="color:#98c379"> maxconn</span><span style="color:#d19a66"> 250</span><span style="color:#98c379"> maxqueue</span><span style="color:#d19a66"> 256</span><span style="color:#98c379"> weight</span><span style="color:#d19a66"> 100</span>
<span style="color:#61afef">  server</span><span style="color:#98c379"> k8s-master1</span><span style="color:#98c379">	172.16.11.215:6443</span><span style="color:#98c379">  check</span>
<span style="color:#61afef">  server</span><span style="color:#98c379"> k8s-master2</span><span style="color:#98c379">	172.16.11.216:6443</span><span style="color:#98c379">  check</span>
<span style="color:#61afef">  server</span><span style="color:#98c379"> k8s-master3</span><span style="color:#98c379">	172.16.11.217:6443</span><span style="color:#98c379">  check</span></code></span></span>

[所有主节点配置keepalived （一个一个配置）](#所有主节点配置keepalived （一个一个配置）)

k8s-master1节点配置

one-dark-pro 复制代码

<span style="background-color:#282c34"><span style="color:#abb2bf"><code class="language-bash"><span style="color:#abb2bf">[root@k8s-master1 ~]# vim /etc/keepalived/keepalived.conf </span>
<span style="color:#abb2bf">[root@k8s-master1 ~]# cat /etc/keepalived/keepalived.conf</span></code></span></span>

需要修改的地方有：（都需要根据自己的实际数值来改）

◎　interface #网卡名称

◎　mcast_src_ip #该节点的ip

◎　virtual_ipaddress #vip地址

one-dark-pro 复制代码

<span style="background-color:#282c34"><span style="color:#abb2bf"><code class="language-bash"><span style="color:#abb2bf">! </span><span style="color:#61afef">Configuration</span><span style="color:#98c379"> File</span><span style="color:#98c379"> for</span><span style="color:#98c379"> keepalived</span>
<span style="color:#61afef">global_defs</span><span style="color:#98c379"> {</span>
<span style="color:#61afef">    router_id</span><span style="color:#98c379"> LVS_DEVEL</span>
<span style="color:#61afef">script_user</span><span style="color:#98c379"> root</span>
<span style="color:#61afef">    enable_script_security</span>
<span style="color:#abb2bf">}</span>
<span style="color:#61afef">vrrp_script</span><span style="color:#98c379"> chk_apiserver</span><span style="color:#98c379"> {</span>
<span style="color:#61afef">    script</span><span style="color:#98c379"> "/etc/keepalived/check_apiserver.sh"</span>
<span style="color:#61afef">    interval</span><span style="color:#d19a66"> 5</span>
<span style="color:#61afef">    weight</span><span style="color:#d19a66"> -5</span>
<span style="color:#61afef">    fall</span><span style="color:#d19a66"> 2</span>  
<span style="color:#61afef">    rise</span><span style="color:#d19a66"> 1</span>
<span style="color:#abb2bf">}</span>
<span style="color:#61afef">vrrp_instance</span><span style="color:#98c379"> VI_1</span><span style="color:#98c379"> {</span>
<span style="color:#61afef">    state</span><span style="color:#98c379"> MASTER</span><span style="color:#7f848e"><em>					#高可用主1</em></span>
<span style="color:#61afef">    interface</span><span style="color:#98c379"> eth0</span><span style="color:#7f848e"><em>					#网卡名称</em></span>
<span style="color:#61afef">    mcast_src_ip</span><span style="color:#d19a66"> 172.16.11.215</span><span style="color:#7f848e"><em>		#该节点 IP</em></span>
<span style="color:#61afef">    virtual_router_id</span><span style="color:#d19a66"> 51</span>
<span style="color:#61afef">    priority</span><span style="color:#d19a66"> 101</span><span style="color:#7f848e"><em>				#设置最高级优先级</em></span>
<span style="color:#61afef">    advert_int</span><span style="color:#d19a66"> 2</span>
<span style="color:#61afef">    authentication</span><span style="color:#98c379"> {</span>
<span style="color:#61afef">        auth_type</span><span style="color:#98c379"> PASS</span>
<span style="color:#61afef">        auth_pass</span><span style="color:#98c379"> K8SHA_KA_AUTH</span>
<span style="color:#abb2bf">    }</span>
<span style="color:#61afef">    virtual_ipaddress</span><span style="color:#98c379"> {</span>
<span style="color:#61afef">        172.16.11.220</span><span style="color:#7f848e"><em>			#vip地址</em></span>
<span style="color:#abb2bf">    }</span>
<span style="color:#61afef">    track_script</span><span style="color:#98c379"> {</span>
<span style="color:#61afef">       chk_apiserver</span>
<span style="color:#abb2bf">    }</span>
<span style="color:#abb2bf">}</span></code></span></span>

k8s-master2节点配置

one-dark-pro 复制代码

<span style="background-color:#282c34"><span style="color:#abb2bf"><code class="language-bash"><span style="color:#abb2bf">[root@k8s-master2 ~]# vim /etc/keepalived/keepalived.conf </span>
<span style="color:#abb2bf">[root@k8s-master2 ~]# cat /etc/keepalived/keepalived.conf</span></code></span></span>

需要修改的地方有：（都需要根据自己的实际数值来改）

◎　interface #网卡名称

◎　mcast_src_ip #该节点的ip

◎　virtual_ipaddress #vip地址

one-dark-pro 复制代码

<span style="background-color:#282c34"><span style="color:#abb2bf"><code class="language-bash"><span style="color:#abb2bf">! </span><span style="color:#61afef">Configuration</span><span style="color:#98c379"> File</span><span style="color:#98c379"> for</span><span style="color:#98c379"> keepalived</span>
<span style="color:#61afef">global_defs</span><span style="color:#98c379"> {</span>
<span style="color:#61afef">    router_id</span><span style="color:#98c379"> LVS_DEVEL</span>
<span style="color:#61afef">script_user</span><span style="color:#98c379"> root</span>
<span style="color:#61afef">    enable_script_security</span>
<span style="color:#abb2bf">}</span>
<span style="color:#61afef">vrrp_script</span><span style="color:#98c379"> chk_apiserver</span><span style="color:#98c379"> {</span>
<span style="color:#61afef">    script</span><span style="color:#98c379"> "/etc/keepalived/check_apiserver.sh"</span>
<span style="color:#61afef">    interval</span><span style="color:#d19a66"> 5</span>
<span style="color:#61afef">    weight</span><span style="color:#d19a66"> -5</span>
<span style="color:#61afef">    fall</span><span style="color:#d19a66"> 2</span>  
<span style="color:#61afef">    rise</span><span style="color:#d19a66"> 1</span>
<span style="color:#abb2bf">}</span>
<span style="color:#61afef">vrrp_instance</span><span style="color:#98c379"> VI_1</span><span style="color:#98c379"> {</span>
<span style="color:#61afef">    state</span><span style="color:#98c379"> BACKUP</span><span style="color:#7f848e"><em>					#高可用从1</em></span>
<span style="color:#61afef">    interface</span><span style="color:#98c379"> eth0</span><span style="color:#7f848e"><em>					#网卡名称</em></span>
<span style="color:#61afef">    mcast_src_ip</span><span style="color:#d19a66"> 172.16.11.216</span><span style="color:#7f848e"><em>		#该节点 IP</em></span>
<span style="color:#61afef">    virtual_router_id</span><span style="color:#d19a66"> 51</span>
<span style="color:#61afef">    priority</span><span style="color:#d19a66"> 100</span><span style="color:#7f848e"><em>				#设置优先级</em></span>
<span style="color:#61afef">    advert_int</span><span style="color:#d19a66"> 2</span>
<span style="color:#61afef">    authentication</span><span style="color:#98c379"> {</span>
<span style="color:#61afef">        auth_type</span><span style="color:#98c379"> PASS</span>
<span style="color:#61afef">        auth_pass</span><span style="color:#98c379"> K8SHA_KA_AUTH</span>
<span style="color:#abb2bf">    }</span>
<span style="color:#61afef">    virtual_ipaddress</span><span style="color:#98c379"> {</span>
<span style="color:#61afef">        172.16.11.220</span><span style="color:#7f848e"><em>			#vip地址</em></span>
<span style="color:#abb2bf">    }</span>
<span style="color:#61afef">    track_script</span><span style="color:#98c379"> {</span>
<span style="color:#61afef">       chk_apiserver</span>
<span style="color:#abb2bf">    }</span>
<span style="color:#abb2bf">}</span></code></span></span>

k8s-master3节点配置

one-dark-pro 复制代码

<span style="background-color:#282c34"><span style="color:#abb2bf"><code class="language-bash"><span style="color:#abb2bf">[root@k8s-master3 ~]# vim /etc/keepalived/keepalived.conf </span>
<span style="color:#abb2bf">[root@k8s-master3 ~]# cat /etc/keepalived/keepalived.conf</span></code></span></span>

需要修改的地方有：（都需要根据自己的实际数值来改）

◎　interface #网卡名称

◎　mcast_src_ip #该节点的ip

◎　virtual_ipaddress #vip地址

one-dark-pro 复制代码

<span style="background-color:#282c34"><span style="color:#abb2bf"><code class="language-bash"><span style="color:#abb2bf">! </span><span style="color:#61afef">Configuration</span><span style="color:#98c379"> File</span><span style="color:#98c379"> for</span><span style="color:#98c379"> keepalived</span>
<span style="color:#61afef">global_defs</span><span style="color:#98c379"> {</span>
<span style="color:#61afef">    router_id</span><span style="color:#98c379"> LVS_DEVEL</span>
<span style="color:#61afef">script_user</span><span style="color:#98c379"> root</span>
<span style="color:#61afef">    enable_script_security</span>
<span style="color:#abb2bf">}</span>
<span style="color:#61afef">vrrp_script</span><span style="color:#98c379"> chk_apiserver</span><span style="color:#98c379"> {</span>
<span style="color:#61afef">    script</span><span style="color:#98c379"> "/etc/keepalived/check_apiserver.sh"</span>
<span style="color:#61afef">    interval</span><span style="color:#d19a66"> 5</span>
<span style="color:#61afef">    weight</span><span style="color:#d19a66"> -5</span>
<span style="color:#61afef">    fall</span><span style="color:#d19a66"> 2</span>  
<span style="color:#61afef">    rise</span><span style="color:#d19a66"> 1</span>
<span style="color:#abb2bf">}</span>
<span style="color:#61afef">vrrp_instance</span><span style="color:#98c379"> VI_1</span><span style="color:#98c379"> {</span>
<span style="color:#61afef">    state</span><span style="color:#98c379"> BACKUP</span><span style="color:#7f848e"><em>					#高可用从2</em></span>
<span style="color:#61afef">    interface</span><span style="color:#98c379"> ens33</span><span style="color:#7f848e"><em>					#网卡名称</em></span>
<span style="color:#61afef">    mcast_src_ip</span><span style="color:#d19a66"> 172.16.11.217</span><span style="color:#7f848e"><em>		#该节点 IP</em></span>
<span style="color:#61afef">    virtual_router_id</span><span style="color:#d19a66"> 51</span>
<span style="color:#61afef">    priority</span><span style="color:#d19a66"> 99</span><span style="color:#7f848e"><em>				#设置优先级</em></span>
<span style="color:#61afef">    advert_int</span><span style="color:#d19a66"> 2</span>
<span style="color:#61afef">    authentication</span><span style="color:#98c379"> {</span>
<span style="color:#61afef">        auth_type</span><span style="color:#98c379"> PASS</span>
<span style="color:#61afef">        auth_pass</span><span style="color:#98c379"> K8SHA_KA_AUTH</span>
<span style="color:#abb2bf">    }</span>
<span style="color:#61afef">    virtual_ipaddress</span><span style="color:#98c379"> {</span>
<span style="color:#61afef">        172.16.11.220</span><span style="color:#7f848e"><em>			#vip地址</em></span>
<span style="color:#abb2bf">    }</span>
<span style="color:#61afef">    track_script</span><span style="color:#98c379"> {</span>
<span style="color:#61afef">       chk_apiserver</span>
<span style="color:#abb2bf">    }</span>
<span style="color:#abb2bf">}</span></code></span></span>

[所有主节点编写健康检测脚本](#所有主节点编写健康检测脚本)

one-dark-pro 复制代码

<span style="background-color:#282c34"><span style="color:#abb2bf"><code class="language-bash"><span style="color:#abb2bf">[root@k8s-master1 ~]# vim /etc/keepalived/check_apiserver.sh </span>
<span style="color:#abb2bf">[root@k8s-master1 ~]# cat /etc/keepalived/check_apiserver.sh</span>
<span style="color:#7f848e"><em>#!/bin/bash</em></span>
<span style="color:#e06c75">err</span><span style="color:#56b6c2">=</span><span style="color:#98c379">0</span>
<span style="color:#c678dd">for</span><span style="color:#e06c75"> k</span><span style="color:#c678dd"> in</span><span style="color:#abb2bf"> $(</span><span style="color:#61afef">seq</span><span style="color:#d19a66"> 1</span><span style="color:#d19a66"> 3</span><span style="color:#abb2bf">);</span><span style="color:#c678dd">do</span>
<span style="color:#e06c75">    check_code</span><span style="color:#56b6c2">=</span><span style="color:#abb2bf">$(</span><span style="color:#61afef">pgrep</span><span style="color:#98c379"> haproxy</span><span style="color:#abb2bf">)</span>
<span style="color:#c678dd">    if</span><span style="color:#abb2bf"> [[ </span><span style="color:#e06c75">$check_code</span><span style="color:#56b6c2"> ==</span><span style="color:#98c379"> ""</span><span style="color:#abb2bf"> ]]; </span><span style="color:#c678dd">then</span>
<span style="color:#e06c75">        err</span><span style="color:#56b6c2">=</span><span style="color:#abb2bf">$(</span><span style="color:#61afef">expr</span><span style="color:#e06c75"> $err</span><span style="color:#98c379"> +</span><span style="color:#d19a66"> 1</span><span style="color:#abb2bf">)</span>
<span style="color:#61afef">        sleep</span><span style="color:#d19a66"> 1</span>
<span style="color:#c678dd">        continue</span>
<span style="color:#c678dd">    else</span>
<span style="color:#e06c75">        err</span><span style="color:#56b6c2">=</span><span style="color:#98c379">0</span>
<span style="color:#c678dd">        break</span>
<span style="color:#c678dd">    fi</span>
<span style="color:#c678dd">done</span>

<span style="color:#c678dd">if</span><span style="color:#abb2bf"> [[ </span><span style="color:#e06c75">$err</span><span style="color:#56b6c2"> !=</span><span style="color:#98c379"> "0"</span><span style="color:#abb2bf"> ]]; </span><span style="color:#c678dd">then</span>
<span style="color:#56b6c2">    echo</span><span style="color:#98c379"> "systemctl stop keepalived"</span>
<span style="color:#61afef">    /usr/bin/systemctl</span><span style="color:#98c379"> stop</span><span style="color:#98c379"> keepalived</span>
<span style="color:#56b6c2">    exit</span><span style="color:#d19a66"> 1</span>
<span style="color:#c678dd">else</span>
<span style="color:#56b6c2">    exit</span><span style="color:#d19a66"> 0</span>
<span style="color:#c678dd">fi</span>

<span style="color:#7f848e"><em>#给监测脚本添加执行权限</em></span>
<span style="color:#abb2bf">[root@k8s-master1 ~]# chmod +x /etc/keepalived/check_apiserver.sh</span></code></span></span>

[所有主节点启动 keepalived 与 haproxy](#所有主节点启动 keepalived 与 haproxy)

one-dark-pro 复制代码

<span style="background-color:#282c34"><span style="color:#abb2bf"><code class="language-bash"><span style="color:#7f848e"><em>#生效配置文件</em></span>
<span style="color:#abb2bf">[root@k8s-master1 ~]# systemctl daemon-reload</span>

<span style="color:#7f848e"><em>#启动并设置开机自启haproxy</em></span>
<span style="color:#abb2bf">[root@k8s-master1 ~]# systemctl enable --now haproxy</span>
<span style="color:#61afef">Created</span><span style="color:#98c379"> symlink</span><span style="color:#98c379"> from</span><span style="color:#98c379"> /etc/systemd/system/multi-user.target.wants/haproxy.service</span><span style="color:#98c379"> to</span><span style="color:#98c379"> /usr/lib/systemd/system/haproxy.service.</span>

<span style="color:#7f848e"><em>#启动并设置开机自启keepalived</em></span>
<span style="color:#abb2bf">[root@k8s-master1 ~]# systemctl enable --now keepalived</span>
<span style="color:#61afef">Created</span><span style="color:#98c379"> symlink</span><span style="color:#98c379"> from</span><span style="color:#98c379"> /etc/systemd/system/multi-user.target.wants/keepalived.service</span><span style="color:#98c379"> to</span><span style="color:#98c379"> /usr/lib/systemd/system/keepalived.service.</span></code></span></span>

k8s-master1主节点查看VIP

one-dark-pro 复制代码

<span style="background-color:#282c34"><span style="color:#abb2bf"><code class="language-bash"><span style="color:#7f848e"><em>#查看IP与vip的IP</em></span>
<span style="color:#abb2bf">[root@k8s-master1 ~]# hostname -I</span>
<span style="color:#61afef">172.16.11.215</span><span style="color:#d19a66"> 172.16.11.220</span><span style="color:#d19a66"> 172.17.0.1</span>

<span style="color:#7f848e"><em>#测试vip的16443端口是否通</em></span>
<span style="color:#abb2bf">[root@k8s-master1 ~]# telnet 172.16.11.220 16443</span>
<span style="color:#61afef">Trying</span><span style="color:#98c379"> 172.16.11.220...</span>
<span style="color:#61afef">Connected</span><span style="color:#98c379"> to</span><span style="color:#98c379"> 172.16.11.220.</span>
<span style="color:#61afef">Escape</span><span style="color:#98c379"> character</span><span style="color:#98c379"> is</span><span style="color:#98c379"> '^]'.</span>
<span style="color:#61afef">Connection</span><span style="color:#98c379"> closed</span><span style="color:#98c379"> by</span><span style="color:#98c379"> foreign</span><span style="color:#98c379"> host.</span></code></span></span>

[master节点初始化（所有 master节点操作，有单独在 master1节点操作的则在master1节点操作即可）](#master节点初始化（所有 master节点操作，有单独在 master1节点操作的则在master1节点操作即可）)

k8s-master1节点创建kubeadm-config.yaml配置文件如下：当然，也可以利用命令kubeadm config print init-defaults生成配置文件模板，然后进行修改：

需要自行修改的有：

◎　advertiseAddress #自己的master1节点IP

◎　name #自己的master1节点的名称

◎　certSANs #vip地址

◎　controlPlaneEndpoint #vip地址

◎　kubernetesVersion #kubernets版本

◎　podSubnet #pod网段

◎　serviceSubnet #service网段

one-dark-pro 复制代码

<span style="background-color:#282c34"><span style="color:#abb2bf"><code class="language-bash"><span style="color:#abb2bf">[root@k8s-master1 ~]# vim kubeadm-config.yaml</span>
<span style="color:#61afef">apiVersion:</span><span style="color:#98c379"> kubeadm.k8s.io/v1beta2</span>
<span style="color:#61afef">bootstrapTokens:</span>
<span style="color:#61afef">-</span><span style="color:#98c379"> groups:</span>
<span style="color:#61afef">  -</span><span style="color:#98c379"> system:bootstrappers:kubeadm:default-node-token</span>
<span style="color:#61afef">  token:</span><span style="color:#98c379"> 7t2weq.bjbawausm0jaxury</span><span style="color:#7f848e"><em>  #初始化集群使用的token</em></span>
<span style="color:#61afef">  ttl:</span><span style="color:#98c379"> 24h0m0s</span><span style="color:#7f848e"><em>    #token有效期</em></span>
<span style="color:#61afef">  usages:</span>
<span style="color:#61afef">  -</span><span style="color:#98c379"> signing</span>
<span style="color:#61afef">  -</span><span style="color:#98c379"> authentication</span>
<span style="color:#61afef">kind:</span><span style="color:#98c379"> InitConfiguration</span>
<span style="color:#61afef">localAPIEndpoint:</span>
<span style="color:#61afef">  advertiseAddress:</span><span style="color:#d19a66"> 172.16.11.215</span>
<span style="color:#61afef">  bindPort:</span><span style="color:#d19a66"> 6443</span>
<span style="color:#61afef">nodeRegistration:</span><span style="color:#7f848e"><em>   #集群节点的信息</em></span>
<span style="color:#61afef">  criSocket:</span><span style="color:#98c379"> /var/run/dockershim.sock</span>
<span style="color:#61afef">  name:</span><span style="color:#98c379"> k8s-master1</span>
<span style="color:#61afef">  taints:</span>
<span style="color:#61afef">  -</span><span style="color:#98c379"> effect:</span><span style="color:#98c379"> NoSchedule</span>
<span style="color:#61afef">    key:</span><span style="color:#98c379"> node-role.kubernetes.io/master</span>
<span style="color:#61afef">---</span>
<span style="color:#61afef">apiServer:</span>
<span style="color:#61afef">  certSANs:</span>
<span style="color:#61afef">  -</span><span style="color:#d19a66"> 172.16.11.220</span>
<span style="color:#61afef">  timeoutForControlPlane:</span><span style="color:#98c379"> 4m0s</span>
<span style="color:#61afef">apiVersion:</span><span style="color:#98c379"> kubeadm.k8s.io/v1beta2</span>
<span style="color:#61afef">certificatesDir:</span><span style="color:#98c379"> /etc/kubernetes/pki</span>
<span style="color:#61afef">clusterName:</span><span style="color:#98c379"> kubernetes</span>
<span style="color:#61afef">controlPlaneEndpoint:</span><span style="color:#98c379"> 172.16.11.220:16443</span><span style="color:#7f848e"><em>  #连接apiserver的地址</em></span>
<span style="color:#61afef">controllerManager:</span><span style="color:#98c379"> {}</span>
<span style="color:#61afef">dns:</span>
<span style="color:#56b6c2">  type</span><span style="color:#98c379">:</span><span style="color:#98c379"> CoreDNS</span>
<span style="color:#61afef">etcd:</span>
<span style="color:#61afef">  local:</span>
<span style="color:#61afef">    dataDir:</span><span style="color:#98c379"> /var/lib/etcd</span>
<span style="color:#61afef">imageRepository:</span><span style="color:#98c379"> registry.cn-hangzhou.aliyuncs.com/google_containers</span>
<span style="color:#61afef">kind:</span><span style="color:#98c379"> ClusterConfiguration</span>
<span style="color:#61afef">kubernetesVersion:</span><span style="color:#98c379"> v1.20.0</span><span style="color:#7f848e"><em>  #与kubernets版本对应 </em></span>
<span style="color:#61afef">networking:</span>
<span style="color:#61afef">  dnsDomain:</span><span style="color:#98c379"> cluster.local</span>
<span style="color:#61afef">  podSubnet:</span><span style="color:#98c379"> 172.16.10.1/18</span>
<span style="color:#61afef">  serviceSubnet:</span><span style="color:#98c379"> 172.17.0.1/16</span><span style="color:#7f848e"><em>  #pod,service与宿主机都不在同一个网段</em></span>
<span style="color:#61afef">scheduler:</span><span style="color:#98c379"> {}</span></code></span></span>

更新配置文件

one-dark-pro 复制代码

<span style="background-color:#282c34"><span style="color:#abb2bf"><code class="language-bash"><span style="color:#61afef">kubeadm</span><span style="color:#98c379"> config</span><span style="color:#98c379"> migrate</span><span style="color:#d19a66"> --old-config</span><span style="color:#98c379"> kubeadm-config.yaml</span><span style="color:#d19a66"> --new-config</span><span style="color:#98c379"> new.yaml</span></code></span></span>

将new.yaml文件复制到其他master节点，之后所有Master节点提前下载镜像，可以节省初始化时间

one-dark-pro 复制代码

<span style="background-color:#282c34"><span style="color:#abb2bf"><code class="language-bash"><span style="color:#61afef">scp</span><span style="color:#98c379"> new.yaml</span><span style="color:#98c379"> 172.16.11.216:/root/</span>
<span style="color:#61afef">scp</span><span style="color:#98c379"> new.yaml</span><span style="color:#98c379"> 172.16.11.217:/root/</span></code></span></span>

查看需要的镜像文件

one-dark-pro 复制代码

<span style="background-color:#282c34"><span style="color:#abb2bf"><code class="language-bash"><span style="color:#61afef">kubeadm</span><span style="color:#98c379"> config</span><span style="color:#98c379"> images</span><span style="color:#98c379"> list</span><span style="color:#d19a66"> --config</span><span style="color:#98c379"> /root/new.yaml</span></code></span></span>

所有master节点启动

one-dark-pro 复制代码

<span style="background-color:#282c34"><span style="color:#abb2bf"><code class="language-bash"><span style="color:#61afef">kubeadm</span><span style="color:#98c379"> config</span><span style="color:#98c379"> images</span><span style="color:#98c379"> pull</span><span style="color:#d19a66"> --config</span><span style="color:#98c379"> /root/new.yaml</span></code></span></span>

[master1节点初始化，初始化后生成对应的证书](#master1节点初始化，初始化后生成对应的证书)

one-dark-pro 复制代码

<span style="background-color:#282c34"><span style="color:#abb2bf"><code class="language-bash"><span style="color:#61afef">kubeadm</span><span style="color:#98c379"> init</span><span style="color:#d19a66"> --config</span><span style="color:#98c379"> /root/new.yaml</span><span style="color:#d19a66">  --upload-certs</span></code></span></span>

说明:生成的token有效期为2个小时，如果token过期后，可以采用一下方案解决；

Token过期后生成新的token：

kubeadm token create --print-join-command

Master需要生成--certificate-key

kubeadm init phase upload-certs --upload-certs

[master1节点配置环境变量，用于访问Kubernetes集群](#master1节点配置环境变量，用于访问Kubernetes集群)

one-dark-pro 复制代码

<span style="background-color:#282c34"><span style="color:#abb2bf"><code class="language-bash"><span style="color:#61afef">cat</span><span style="color:#abb2bf"> <<</span><span style="color:#abb2bf">EOF</span><span style="color:#abb2bf"> >> </span><span style="color:#98c379">/root/.bashrc</span>
<span style="color:#98c379">export KUBECONFIG=/etc/kubernetes/admin.conf</span>
<span style="color:#abb2bf">EOF</span>


<span style="color:#7f848e"><em>#让其生效</em></span>
<span style="color:#56b6c2">source</span><span style="color:#98c379"> /root/.bashrc</span></code></span></span>

[在 master1节点查看集群节点状态](#在 master1节点查看集群节点状态)

one-dark-pro 复制代码

<span style="background-color:#282c34"><span style="color:#abb2bf"><code class="language-bash"><span style="color:#61afef">kubectl</span><span style="color:#98c379"> get</span><span style="color:#98c379"> nodes</span></code></span></span>

截图

采用初始化安装方式，所有的系统组件均以容器的方式运行并且在kube-system命名空间内，此时可以查看Pod状态：

one-dark-pro 复制代码

<span style="background-color:#282c34"><span style="color:#abb2bf"><code class="language-bash"><span style="color:#61afef">kubectl</span><span style="color:#98c379"> get</span><span style="color:#98c379"> pods</span><span style="color:#d19a66"> -n</span><span style="color:#98c379"> kube-system</span><span style="color:#d19a66"> -o</span><span style="color:#98c379"> wide</span></code></span></span>

可以看到有两个READY的状态是0/1，这个不用担心，后面节点加入之后就好了。

他是执行的时候有一个告警：Warning FailedScheduling 7s (x3 over 2m7s) default-scheduler 0/1 nodes are available: 1 node(s) had taint {node.kubernetes.io/not-ready: }, that the pod didn't tolerate。

[其他master节点加入集群（master2 和 master3 节点操作）](#其他master节点加入集群（master2 和 master3 节点操作）)

one-dark-pro 复制代码

<span style="background-color:#282c34"><span style="color:#abb2bf"><code class="language-bash"><span style="color:#abb2bf">[root@k8s-master2 ~]# kubeadm join 172.16.11.220:16443 --token 7t2weq.bjbawausm0jaxury </span><span style="color:#56b6c2">\</span>
<span style="color:#61afef">   --discovery-token-ca-cert-hash</span><span style="color:#98c379"> sha256:f370e43a5b3218b7b6094980edd4fac3b6104a272e465767d3c78dfad8c62066</span><span style="color:#56b6c2"> \</span>
<span style="color:#d19a66">   --control-plane</span><span style="color:#d19a66"> --certificate-key</span><span style="color:#98c379"> 26270907ae9b940062bd9f5e4daad59e2804a4a5bdbfb61c7ebb0e6d2f5e5839</span></code></span></span>

one-dark-pro 复制代码

<span style="background-color:#282c34"><span style="color:#abb2bf"><code class="language-bash"><span style="color:#abb2bf">[root@k8s-master3 ~]# kubeadm join 172.16.11.220:16443 --token 7t2weq.bjbawausm0jaxury </span><span style="color:#56b6c2">\</span>
<span style="color:#61afef">   --discovery-token-ca-cert-hash</span><span style="color:#98c379"> sha256:f370e43a5b3218b7b6094980edd4fac3b6104a272e465767d3c78dfad8c62066</span><span style="color:#56b6c2"> \</span>
<span style="color:#d19a66">   --control-plane</span><span style="color:#d19a66"> --certificate-key</span><span style="color:#98c379"> 26270907ae9b940062bd9f5e4daad59e2804a4a5bdbfb61c7ebb0e6d2f5e5839</span></code></span></span>

加入成功，可以在master1上使用kubectl get nods来查看。

[Node节点配置（node1 和 node2 节点操作）](#Node节点配置（node1 和 node2 节点操作）)

Node节点上主要部署公司的一些业务应用，生产环境中不建议Master节点部署系统组件之外的其他Pod，测试环境可以允许Master节点部署Pod以节省系统资源。

将node节点添加到集群

one-dark-pro 复制代码

<span style="background-color:#282c34"><span style="color:#abb2bf"><code class="language-bash"><span style="color:#abb2bf">[root@k8s-node1 ~]# kubeadm join 172.16.11.220:16443 --token 7t2weq.bjbawausm0jaxury </span><span style="color:#56b6c2">\</span>
<span style="color:#61afef">    --discovery-token-ca-cert-hash</span><span style="color:#98c379"> sha256:f370e43a5b3218b7b6094980edd4fac3b6104a272e465767d3c78dfad8c62066</span></code></span></span>

one-dark-pro 复制代码

<span style="background-color:#282c34"><span style="color:#abb2bf"><code class="language-bash"><span style="color:#abb2bf">[root@k8s-node2 ~]# kubeadm join 172.16.11.220:16443 --token 7t2weq.bjbawausm0jaxury </span><span style="color:#56b6c2">\</span>
<span style="color:#61afef">    --discovery-token-ca-cert-hash</span><span style="color:#98c379"> sha256:f370e43a5b3218b7b6094980edd4fac3b6104a272e465767d3c78dfad8c62066</span></code></span></span>

加入成功，可以在master1上使用kubectl get nods来查看。

master1查看所有节点

one-dark-pro 复制代码

<span style="background-color:#282c34"><span style="color:#abb2bf"><code class="language-bash"><span style="color:#abb2bf"> [root@k8s-master1 ~]# kubectl get nodes</span></code></span></span>

截图

现在的集群状态都是NotReady表示不可达；这是因为还没有安装网络插件，下面我们来安装一下网络插件（caclico）

[Calico组件（只在 master1节点操作）【网络插件，用于连接其他节点】](#Calico组件（只在 master1节点操作）【网络插件，用于连接其他节点】)

Calico是一个纯三层的协议，为OpenStack虚拟机和Docker容器提供多主机间通信。Calico不使用重叠网络比如flannel和libnetwork重叠网络驱动，它是一个纯三层的方法，使用虚拟路由代替虚拟交换，每一台虚拟路由通过BGP协议传播可达信息（路由）到剩余数据中心。

拉取安装包（安装包内包含：Calico组件、Metrics组件、Dashboard组件）

one-dark-pro 复制代码

<span style="background-color:#282c34"><span style="color:#abb2bf"><code class="language-bash"><span style="color:#61afef">git</span><span style="color:#98c379"> clone</span><span style="color:#98c379"> https://github.com/dotbalo/k8s-ha-install.git</span></code></span></span>

如果拉取不下来，可以参考该地址下载：

one-dark-pro 复制代码

<span style="background-color:#282c34"><span style="color:#abb2bf"><code class="language-bash"><span style="color:#7f848e"><em>#拉取完成进入该目录</em></span>
<span style="color:#56b6c2">cd</span><span style="color:#98c379"> k8s-ha-install</span>

<span style="color:#7f848e"><em>#切换到manual-installation-v1.20.x分支</em></span>
<span style="color:#abb2bf">[root@k8s-master1 k8s-ha-install]# git checkout manual-installation-v1.20.x</span></code></span></span>

修改相关的配置文件

one-dark-pro 复制代码

<span style="background-color:#282c34"><span style="color:#abb2bf"><code class="language-bash"><span style="color:#7f848e"><em>#进入calico目录</em></span>
<span style="color:#abb2bf">[root@k8s-master1 k8s-ha-install]# cd calico/</span></code></span></span>

注意事项：

◎　需要改的就只要第一行的ip

one-dark-pro 复制代码

<span style="background-color:#282c34"><span style="color:#abb2bf"><code class="language-bash"><span style="color:#7f848e"><em>#将要修改的ip改为自己的ip，按照master节点顺序的改成自己的节点ip</em></span>
<span style="color:#abb2bf">[root@k8s-master1 calico]# sed -i </span><span style="color:#98c379">'s#etcd_endpoints: "http://<ETCD_IP>:<ETCD_PORT>"#etcd_endpoints: "https://172.16.11.215:2379,https://172.16.11.216:2379,https://172.16.11.217:2379"#g'</span><span style="color:#abb2bf"> calico-etcd.yaml</span>

<span style="color:#7f848e"><em>#设置临时环境变量ETCD_CA查看ca.crt文件并转化为base64格式取消换行符</em></span>
<span style="color:#abb2bf">[root@k8s-master1 calico]# </span><span style="color:#e06c75">ETCD_CA</span><span style="color:#56b6c2">=</span><span style="color:#98c379">`</span><span style="color:#61afef">cat</span><span style="color:#98c379"> /etc/kubernetes/pki/etcd/ca.crt </span><span style="color:#abb2bf">|</span><span style="color:#61afef"> base64</span><span style="color:#abb2bf"> |</span><span style="color:#61afef"> tr</span><span style="color:#d19a66"> -d</span><span style="color:#98c379"> '\n'`</span>
<span style="color:#7f848e"><em>#设置临时环境变量ETCD_CERT查看server.crt文件并转化为base64格式取消换行符</em></span>
<span style="color:#abb2bf">[root@k8s-master1 calico]# </span><span style="color:#e06c75">ETCD_CERT</span><span style="color:#56b6c2">=</span><span style="color:#98c379">`</span><span style="color:#61afef">cat</span><span style="color:#98c379"> /etc/kubernetes/pki/etcd/server.crt </span><span style="color:#abb2bf">|</span><span style="color:#61afef"> base64</span><span style="color:#abb2bf"> |</span><span style="color:#61afef"> tr</span><span style="color:#d19a66"> -d</span><span style="color:#98c379"> '\n'`</span>
<span style="color:#7f848e"><em>#设置临时环境变量ETCD_KEY查看server.key文件并转化为base64格式取消换行符</em></span>
<span style="color:#abb2bf">[root@k8s-master1 calico]# </span><span style="color:#e06c75">ETCD_KEY</span><span style="color:#56b6c2">=</span><span style="color:#98c379">`</span><span style="color:#61afef">cat</span><span style="color:#98c379"> /etc/kubernetes/pki/etcd/server.key </span><span style="color:#abb2bf">|</span><span style="color:#61afef"> base64</span><span style="color:#abb2bf"> |</span><span style="color:#61afef"> tr</span><span style="color:#d19a66"> -d</span><span style="color:#98c379"> '\n'`</span>

<span style="color:#7f848e"><em>#更换calico-etcd.yaml文件中的# etcd-key: null、# etcd-cert: null、# etcd-ca: null为指定值，临时的环境变量这这用。</em></span>
<span style="color:#abb2bf">[root@k8s-master1 calico]# sed -i </span><span style="color:#98c379">"s/# etcd-key: null/etcd-key: ${</span><span style="color:#e06c75">ETCD_KEY</span><span style="color:#98c379">}/g; s/# etcd-cert: null/etcd-cert: ${</span><span style="color:#e06c75">ETCD_CERT</span><span style="color:#98c379">}/g; s/# etcd-ca: null/etcd-ca: ${</span><span style="color:#e06c75">ETCD_CA</span><span style="color:#98c379">}/g"</span><span style="color:#abb2bf"> calico-etcd.yaml</span>

<span style="color:#7f848e"><em>#更换calico-etcd.yaml文件中的etcd_ca: ""#、etcd_cert: ""、etcd_key: "" 为指定值</em></span>
<span style="color:#abb2bf">[root@k8s-master1 calico]# sed -i </span><span style="color:#98c379">'s#etcd_ca: ""#etcd_ca: "/calico-secrets/etcd-ca"#g; s#etcd_cert: ""#etcd_cert: "/calico-secrets/etcd-cert"#g; s#etcd_key: "" #etcd_key: "/calico-secrets/etcd-key" #g'</span><span style="color:#abb2bf"> calico-etcd.yaml</span>

<span style="color:#7f848e"><em>#设置临时环境变量POD_SUBNET从kubernetes配置文件中查找自己的网关</em></span>
<span style="color:#abb2bf">[root@k8s-master1 calico]# </span><span style="color:#e06c75">POD_SUBNET</span><span style="color:#56b6c2">=</span><span style="color:#98c379">`</span><span style="color:#61afef">cat</span><span style="color:#98c379"> /etc/kubernetes/manifests/kube-controller-manager.yaml </span><span style="color:#abb2bf">|</span><span style="color:#61afef"> grep</span><span style="color:#98c379"> cluster-cidr= </span><span style="color:#abb2bf">|</span><span style="color:#61afef"> awk</span><span style="color:#d19a66"> -F=</span><span style="color:#98c379"> '{print $NF}'`</span>

<span style="color:#7f848e"><em>#注意下面的这个步骤是把calico-etcd.yaml文件里面的CALICO_IPV4POOL_CIDR下的网段改成自己的Pod网段，并打开注释，不用手动改，会用到上面的环境变量；</em></span>
<span style="color:#abb2bf">[root@k8s-master1 calico]# sed -i </span><span style="color:#98c379">'s@# - name: CALICO_IPV4POOL_CIDR@- name: CALICO_IPV4POOL_CIDR@g; s@#   value: "192.168.0.0/16"@  value: '"${</span><span style="color:#e06c75">POD_SUBNET</span><span style="color:#98c379">}"'@g'</span><span style="color:#abb2bf"> calico-etcd.yaml</span></code></span></span>

创建运行并查看容器状态

创建并运行yaml文件

one-dark-pro 复制代码

<span style="background-color:#282c34"><span style="color:#abb2bf"><code class="language-bash"><span style="color:#abb2bf">[root@k8s-maste1 calico]# kubectl apply -f calico-etcd.yaml</span></code></span></span>

查看创建的容器状态

one-dark-pro 复制代码

<span style="background-color:#282c34"><span style="color:#abb2bf"><code class="language-bash"><span style="color:#abb2bf">[root@k8s-master1 calico]# kubectl get pods -n kube-system</span></code></span></span>

同时，上面提到的两个coredns-54d67798b7-7w4k5容器没有运行起来的问题也成功的运行起来了。目标为：在 master1节点查看集群节点状态标题。

网络组件安装完成，可以再次查看一下节点网络连接状态了。

查看节点网络连接状态

one-dark-pro 复制代码

<span style="background-color:#282c34"><span style="color:#abb2bf"><code class="language-bash"><span style="color:#abb2bf">[root@k8s-master1 calico]# kubectl get nodes</span></code></span></span>

可以看到都连接成功了。

[Metrics部署（只在 master1节点操作）【用于查看其他节点的资源使用率】](#Metrics部署（只在 master1节点操作）【用于查看其他节点的资源使用率】)

在新版的Kubernetes中系统资源的采集均使用Metrics-server，可以通过Metrics采集节点和Pod的内存、磁盘、CPU和网络的使用率。

将Master1节点的front-proxy-ca.crt复制到所有Node节点

复制到node1节点上

one-dark-pro 复制代码

<span style="background-color:#282c34"><span style="color:#abb2bf"><code class="language-bash"><span style="color:#abb2bf">[root@k8s-master1 ~]# scp /etc/kubernetes/pki/front-proxy-ca.crt k8s-node1:/etc/kubernetes/pki/front-proxy-ca.crt</span></code></span></span>

复制到node2节点上

one-dark-pro 复制代码

<span style="background-color:#282c34"><span style="color:#abb2bf"><code class="language-bash"><span style="color:#abb2bf">[root@k8s-master1 ~]# scp /etc/kubernetes/pki/front-proxy-ca.crt k8s-node2:/etc/kubernetes/pki/front-proxy-ca.crt</span></code></span></span>

[安装metrics server](#安装metrics server)

one-dark-pro 复制代码

<span style="background-color:#282c34"><span style="color:#abb2bf"><code class="language-bash"><span style="color:#7f848e"><em>#进入上一步拉取的k8s-ha-install/metrics-server-0.4.x-kubeadm/目录</em></span>
<span style="color:#abb2bf">[root@k8s-master1 ~]# cd /root/k8s-ha-install/metrics-server-0.4.x-kubeadm/ </span>

<span style="color:#7f848e"><em>#根据该目录下的yaml文件创建容器</em></span>
<span style="color:#abb2bf">[root@k8s-master1 metrics-server-0.4.x-kubeadm]# kubectl apply -f comp.yaml</span></code></span></span>

查看节点状态

one-dark-pro 复制代码

<span style="background-color:#282c34"><span style="color:#abb2bf"><code class="language-bash"><span style="color:#abb2bf">[root@k8s-master1 metrics-server-0.4.x-kubeadm]# kubectl  top node</span>
<span style="color:#61afef">NAME</span><span style="color:#98c379">           CPU</span><span style="color:#abb2bf">(</span><span style="color:#61afef">cores</span><span style="color:#abb2bf">)   </span><span style="color:#98c379">CPU%</span><span style="color:#98c379">   MEMORY</span><span style="color:#abb2bf">(</span><span style="color:#61afef">bytes</span><span style="color:#abb2bf">)   </span><span style="color:#98c379">MEMORY%</span>   
<span style="color:#61afef">k8s-master1</span><span style="color:#98c379">   258m</span><span style="color:#98c379">         12%</span><span style="color:#98c379">    1125Mi</span><span style="color:#98c379">          65%</span>       
<span style="color:#61afef">k8s-master2</span><span style="color:#98c379">   224m</span><span style="color:#98c379">         11%</span><span style="color:#98c379">    1083Mi</span><span style="color:#98c379">          62%</span>       
<span style="color:#61afef">k8s-master3</span><span style="color:#98c379">   114m</span><span style="color:#98c379">         5%</span><span style="color:#98c379">     1047Mi</span><span style="color:#98c379">          60%</span>       
<span style="color:#61afef">k8s-node1</span><span style="color:#98c379">     61m</span><span style="color:#98c379">          3%</span><span style="color:#98c379">     860Mi</span><span style="color:#98c379">           50%</span>       
<span style="color:#61afef">k8s-node2</span><span style="color:#98c379">     62m</span><span style="color:#98c379">          3%</span><span style="color:#98c379">     889Mi</span><span style="color:#98c379">           51%</span></code></span></span>

如果还是top查看不到，可以查看一下metrics-server有没有起来，kubectl get pods -n kube-system | grep metrics，有问题排查之后就可以了。

k8s云原生yaml 文件编排web-MySQL项目和kubeadm搭高可用集群(三主两从一VIP)-63148详解

[1、启动 mysql 项目](#1、启动 mysql 项目)

[1-1 创建 mysql-rc.yaml](#1-1 创建 mysql-rc.yaml)

[1-2 创建 mysql-svc.yaml](#1-2 创建 mysql-svc.yaml)

[1-3 附加：如何找到并进入mysql数据库中？](#1-3 附加：如何找到并进入mysql数据库中？)

[2、启动 tomcat 项目](#2、启动 tomcat 项目)

[2-1 拉取特定的tomcat镜像](#2-1 拉取特定的tomcat镜像)

[2-2 创建 tomcat-rc.yaml](#2-2 创建 tomcat-rc.yaml)

[2-3 创建 tomcat-svc.yaml](#2-3 创建 tomcat-svc.yaml)

[高可用集群设置免密登录（在 master1 节点上操作）](#高可用集群设置免密登录（在 master1 节点上操作）)

[部署 docker（所有节点都需要部署）](#部署 docker（所有节点都需要部署）)

[部署 kubernetes（所有节点都要部署）](#部署 kubernetes（所有节点都要部署）)

[添加 k8s yum源](#添加 k8s yum源)

[安装 kubeadm kubelet kubectl](#安装 kubeadm kubelet kubectl)

[部署keepalived、haproxy --> 配置高可用（所有 主节点 部署）](#部署keepalived、haproxy --> 配置高可用（所有 主节点 部署）)

[所有 主节点 修改haproxy配置](#所有 主节点 修改haproxy配置)

[所有 主节点 配置keepalived （一个一个配置）](#所有 主节点 配置keepalived （一个一个配置）)

[所有 主节点 编写健康检测脚本](#所有 主节点 编写健康检测脚本)

[所有 主节点 启动 keepalived 与 haproxy](#所有 主节点 启动 keepalived 与 haproxy)

[master节点初始化（所有 master节点 操作，有单独在 master1节点 操作的则在master1节点操作即可）](#master节点初始化（所有 master节点 操作，有单独在 master1节点 操作的则在master1节点操作即可）)

[master1节点 初始化，初始化后生成对应的证书](#master1节点 初始化，初始化后生成对应的证书)

[master1节点 配置环境变量，用于访问Kubernetes集群](#master1节点 配置环境变量，用于访问Kubernetes集群)

[在 master1节点 查看集群节点状态](#在 master1节点 查看集群节点状态)

[其他master节点加入集群（master2 和 master3 节点操作）](#其他master节点加入集群（master2 和 master3 节点操作）)

[Node节点配置（node1 和 node2 节点操作）](#Node节点配置（node1 和 node2 节点操作）)

[Calico组件（只在 master1节点 操作）【网络插件，用于连接其他节点】](#Calico组件（只在 master1节点 操作）【网络插件，用于连接其他节点】)

[Metrics部署（只在 master1节点 操作）【用于查看其他节点的资源使用率】](#Metrics部署（只在 master1节点 操作）【用于查看其他节点的资源使用率】)

[安装metrics server](#安装metrics server)

[部署keepalived、haproxy --> 配置高可用（所有主节点部署）](#部署keepalived、haproxy --> 配置高可用（所有主节点部署）)

[所有主节点修改haproxy配置](#所有主节点修改haproxy配置)

[所有主节点配置keepalived （一个一个配置）](#所有主节点配置keepalived （一个一个配置）)

[所有主节点编写健康检测脚本](#所有主节点编写健康检测脚本)

[所有主节点启动 keepalived 与 haproxy](#所有主节点启动 keepalived 与 haproxy)

[master节点初始化（所有 master节点操作，有单独在 master1节点操作的则在master1节点操作即可）](#master节点初始化（所有 master节点操作，有单独在 master1节点操作的则在master1节点操作即可）)

[master1节点初始化，初始化后生成对应的证书](#master1节点初始化，初始化后生成对应的证书)

[master1节点配置环境变量，用于访问Kubernetes集群](#master1节点配置环境变量，用于访问Kubernetes集群)

[在 master1节点查看集群节点状态](#在 master1节点查看集群节点状态)

[Calico组件（只在 master1节点操作）【网络插件，用于连接其他节点】](#Calico组件（只在 master1节点操作）【网络插件，用于连接其他节点】)

[Metrics部署（只在 master1节点操作）【用于查看其他节点的资源使用率】](#Metrics部署（只在 master1节点操作）【用于查看其他节点的资源使用率】)