Ansible一键部署Kubernetes集群

一、环境准备

|------------|-----------------|--------|
| 主机 | ip地址 | 角色 |
| k8s-master | 192.168.252.141 | master |
| k8s-node1 | 192.168.252.142 | node |
| k8s-node2 | 192.168.252.143 | node |

二、实战

Ansible部署

主节点安装Ansible

yum -y install epel-release
yum -y install ansible
ansible --version

开启记录日志

vim /etc/ansible/ansible.cfg  
修改：
#log_path = /var/log/ansible.log ==> log_path = /var/log/ansible.log

去掉第一次连接ssh ask确认

vim /etc/ansible/ansible.cfg  
修改：
#host_key_checking = False  ==> host_key_checking = False

配置主机清单

vim /etc/ansible/hosts
内容：
[master]
192.168.252.141
[node]
192.168.252.142
192.168.252.143

[k8s:children]
master
node

[k8s:vars]
ansible_ssh_user=root
ansible_ssh_pass=syh2025659
ansible_ssh_port=22

k8s_version=1.20.2

测试连通性

ansible k8s -m ping

编排 Ansible playbook

mkdir /root/k8s-install-workspace
cd /root/k8s-install-workspace
mkdir -p ./install-k8s/{package,init,install-docker,install-k8s,master-init,install-cni,install-ipvs,master-join,node-join,install-ingress-nginx,install-nfs-provisioner,install-harbor,install-metrics-server,uninstall-k8s}/{files,templates,vars,tasks,handlers,meta,default}

tips:
package目中放的是提前导入的jar包 下面脚本需要直接提前导入这样子更快

我的网盘链接：

通过百度网盘分享的文件：k8s-install-workspace

链接：百度网盘 请输入提取码

提取码：ruj1

提前将package目录中的包导入进去

节点初始化

cd /root/k8s-install-workspace
vim ./install-k8s/init/files/hosts
内容：
192.168.252.141 k8s-master
192.168.252.142 k8s-node1
192.168.252.143 k8s-node2

准备脚本

脚本思路：

修改主机名-->配置hosts解析-->配置秘钥互相通信（通过expect来发送指令）-->对时间进行同步-->关闭防火墙-->关闭swap分区-->关闭selinux-->iptables 检查桥接流量-->启用ipvs的所需模块

cd /root/k8s-install-workspace
vim ./install-k8s/init/templates/init.sh
内容：
#!/usr/bin/env bash

### 【第一步】修改主机名
# 获取主机名
hostnamectl set-hostname $(grep `hostname -i` /tmp/hosts|awk '{print $2}')


### 【第二步】配置hosts
# 先删除
for line in `cat /tmp/hosts`
do
    sed -i "/$line/d" /etc/hosts
done
# 追加
cat /tmp/hosts >> /etc/hosts


### 【第三步】添加互信
# 先创建秘钥对
ssh-keygen -f ~/.ssh/id_rsa -P '' -q

# 安装expect
yum -y install expect -y

# 批量推送公钥
for line in `cat /tmp/hosts`
do

ip=`echo $line|awk '{print $1}'`
password={{ ansible_ssh_pass }}

expect <<-EOF

spawn ssh-copy-id -i /root/.ssh/id_rsa.pub $ip
expect {
    "(yes/no)?"
    {
        send "yes\n"
        expect "*assword:" { send "$password\n"}
    }
    "*assword:"
    {
        send "$password\n"
    }
}

expect eof
EOF
done


### 【第四步】时间同步
yum -y install ntpdate
ntpdate ntp.aliyun.com

### 【第五步】关闭防火墙
systemctl stop firewalld
systemctl disable firewalld


### 【第六步】关闭swap
# 临时关闭；关闭swap主要是为了性能考虑
swapoff -a
# 永久关闭
sed -ri 's/.*swap.*/#&/' /etc/fstab


### 【第七步】禁用SELinux
# 临时关闭
setenforce 0
# 永久禁用
sed -i 's/^SELINUX=enforcing$/SELINUX=disabled/' /etc/selinux/config


### 【第八步】允许 iptables 检查桥接流量
sudo modprobe br_netfilter
lsmod | grep br_netfilter

# 先删
rm -rf /etc/modules-load.d/k8s.conf

cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
overlay
br_netfilter
EOF

sudo modprobe overlay
sudo modprobe br_netfilter

rm -rf /etc/sysctl.d/k8s.conf

# 设置所需的 sysctl 参数，参数在重新启动后保持不变
cat > /etc/sysctl.d/k8s.conf <<EOF
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
vm.swappiness = 0
EOF

#加载模块
modprobe br_netfilter

#加载⽹桥过滤及内核转发配置⽂件
sysctl -p /etc/sysctl.d/k8s.conf

#配置ipvs功能
yum -y install ipset ipvsadm

cat <<EOF > /etc/modules-load.d/ipvs.conf 
ip_vs
ip_vs_lc
ip_vs_wlc
ip_vs_rr
ip_vs_wrr
ip_vs_lblc
ip_vs_lblcr
ip_vs_dh
ip_vs_sh
ip_vs_nq
ip_vs_sed
ip_vs_ftp
ip_vs_sh
nf_conntrack_ipv4
ip_tables
ip_set
xt_set
ipt_set
ipt_rpfilter
ipt_REJECT
ipip
EOF

cat > /etc/sysconfig/modules/ipvs.modules <<EOF
#!/bin/bash
modprobe -- ip_vs
modprobe -- ip_vs_rr
modprobe -- ip_vs_wrr
modprobe -- ip_vs_sh
modprobe -- nf_conntrack
EOF

chmod 755 /etc/sysconfig/modules/ipvs.modules
bash /etc/sysconfig/modules/ipvs.modules

# 应用 sysctl 参数而不重新启动
sudo sysctl --system

任务编排

将hosts发送给所有节点然后发送初始化节点的脚本然后进行执行

cd /root/k8s-install-workspace
vim ./install-k8s/init/tasks/main.yml
内容：
- name: cp hosts
  copy: src=hosts dest=/tmp/hosts
- name: init cp
  template: src=init.sh dest=/tmp/init.sh
- name: init install
  shell: sh /tmp/init.sh

安装docker

编写脚本

通过查询阿里云的安装docker-ce的方式进行安装docker然后配置加速器然后配置cgroup然后启动docker

cd /root/k8s-install-workspace
vim ./install-k8s/install-docker/files/install-docker.sh
内容：
#!/usr/bin/env bash

# step 1: 安装必要的一些系统工具
yum install -y yum-utils device-mapper-persistent-data lvm2
# Step 2: 添加软件源信息
yum-config-manager --add-repo https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
# Step 3
sed -i 's+download.docker.com+mirrors.aliyun.com/docker-ce+' /etc/yum.repos.d/docker-ce.repo
# Step 4: 更新并安装Docker-CE
yum makecache fast
yum -y install docker-ce
# Step 4: 开启Docker服务
systemctl start docker
systemctl enable docker
# 修改文件 /etc/docker/daemon.json，没有这个文件就创建
# 添加以下内容后，重启docker服务：
cat >/etc/docker/daemon.json<<EOF
{
        "registry-mirrors": [
                "https://hub.uuuadc.top",
                "https://docker.anyhub.us.kg",
                "https://dockerhub.jobcher.com",
                "https://dockerhub.icu",
                "https://docker.ckyl.me",
                "https://docker.awsl9527.cn"
        ],
        "exec-opts": ["native.cgroupdriver=systemd"]
}
EOF
# 重启
systemctl restart docker

# 查看
systemctl status docker containerd

任务编排

将脚本发送至所有节点然后执行脚本

cd /root/k8s-install-workspace
vim ./install-k8s/install-docker/tasks/main.yml
内容：
- name: install docker cp
  copy: src=install-docker.sh dest=/tmp/install-docker.sh
- name: install docker
  shell: sh /tmp/install-docker.sh

安装k8s并提前将镜像导入

检查k8s是否安装安装好的话直接下一步，阿里云上找repo源然后进行安装，可以更改版本下面的版本，然后将准备好的镜像tar包进行导入就可以了。

cd /root/k8s-install-workspace
vim ./install-k8s/install-k8s/templates/install-k8s.sh
内容：
#!/usr/bin/env bash

# 检查是否已经安装
yum list installed kubelet
if [ $? -eq 0 ];then
   exit 0
fi

cat > /etc/yum.repos.d/kubernetes.repo <<EOF
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=0
repo_gpgcheck=0
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF

yum install -y kubelet-1.20.2-0.x86_64 kubeadm-1.20.2-0.x86_64 kubectl-1.20.2-0.x86_64

DOCKER_CGROUPS=`docker info |grep 'Cgroup' | awk ' NR==1 {print $3}'`

cat > /etc/sysconfig/kubelet<<EOF
KUBELET_EXTRA_ARGS="--cgroup-driver=$DOCKER_CGROUPS --pod-infra-container-image=k8s.gcr.io/pause:3.2"
EOF

systemctl daemon-reload
systemctl enable kubelet && systemctl restart kubelet
systemctl is-active kubelet

#导入镜像
docker load < /tmp/kube-apiserver.tar
docker load < /tmp/kube-proxy.tar
docker load < /tmp/kube-controller-manager.tar
docker load < /tmp/kube-scheduler.tar
docker load < /tmp/etcd.tar
docker load < /tmp/pause.tar
docker load < /tmp/coredns.tar
docker load < /tmp/calico-cni.tar
docker load < /tmp/calico-kube-controller.tar
docker load < /tmp/calico-node.tar
docker load < /tmp/calico-pod.tar

任务编排

将tar包和执行脚本发送到各个节点的临时目录然后执行脚本

cd /root/k8s-install-workspace
vim ./install-k8s/install-k8s/tasks/main.yml
内容：
- name: copy tarISO
  copy: src=/root/k8s-install-workspace/install-k8s/package/ dest=/tmp/        #这个我是把提前准备好的包放到package里面了
- name: install k8s cp  
  template: src=install-k8s.sh dest=/tmp/install-k8s.sh
- name: install k8s
  shell: sh /tmp/install-k8s.sh

初始化k8s-master并应用网络插件

编写脚本

在主节点进行初始化然后将目录创建好然后从网上拉取calico的yml文件然后进行应用

cd /root/k8s-install-workspace
vim ./install-k8s/master-init/templates/master-init.sh
内容：
#!/usr/bin/env bash

# 判断是否已经初始化了
kubectl get nodes |grep -q `hostname` 1>&2 >/dev/null
if [ $? -eq 0 ];then
   exit 0
fi

ip=`hostname -i`


kubeadm init --kubernetes-version=v1.20.2 --pod-network-cidr=10.244.0.0/16 --apiserver-advertise-address=$ip


rm -rf $HOME/.kube
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

curl https://docs.projectcalico.org/v3.20/manifests/calico.yaml -O

kubectl apply -f calico.yaml

任务编排

将脚本发送到master节点进行执行即可

cd /root/k8s-install-workspace
vim ./install-k8s/master-init/tasks/main.yml
内容：
- name: k8s master init cp
  template: src=master-init.sh dest=/tmp/master-init.sh
- name: k8s master init
  shell: sh /tmp/master-init.sh

node节点加入k8s集群

通过ssh命令获取主节点初始化后的node加入集群节点的命令在node节点上执行进行node节点加入集群

cd /root/k8s-install-workspace
vim ./install-k8s/node-join/files/node-join.sh
内容：
#!/usr/bin/env bash

# 获取master ip，假设都是第一个节点为master
maser_ip=`head -1 /tmp/hosts |awk '{print $1}'`

# 判断节点是否加入
ssh $maser_ip "kubectl get nodes|grep -q `hostname`"
if [ $? -eq 0 ];then
 exit 0
fi

`ssh $maser_ip kubeadm token create --print-join-command`

任务编排

将脚本发送到所有node节点

cd /root/k8s-install-workspace
vim ./install-k8s/node-join/tasks/main.yml
内容：
- name: node join cp
  copy: src=node-join.sh dest=/tmp/node-join.sh
- name: node join
  shell: sh /tmp/node-join.sh

k8s 环境安装编排 roles

进行角色编排

cd /root/k8s-install-workspace
vim ./install-k8s/install-k8s.yaml
内容：
- hosts: k8s
  remote_user: root
  roles:
    - init
- hosts: k8s
  remote_user: root
  roles:
    - install-docker
- hosts: k8s
  remote_user: root
  roles:
    - install-k8s
- hosts: master
  remote_user: root
  roles:
    - master-init
- hosts: node
  remote_user: root
  roles:
    - node-join

启用剧本

cd /root/k8s-install-workspace/install-k8s
ansible-playbook  install-k8s.yaml