rke安装k8s - 技术栈

1、修改集群中各物理机主机名hostname文件

powershell 复制代码

# 查看
cat /etc/hostname
# 命令修改
hostnamectl  set-hostname  k8s-master

2、实现主机名与ip地址解析

powershell 复制代码

# 查看
 cat /etc/hosts
# 修改
vi /etc/hosts

3、配置ip_forward过滤机制

powershell 复制代码

# 修改
vi /etc/sysctl.conf
net.ipv4.ip_forward=1
net.bridge.bridge-nf-call-ip6tables=1
net.bridge.bridge-nf-call-iptables=1
# 查看
sysctl -p
# 执行sysctl -p 报错执行
modprobe br_netfilter

4、关闭防火墙

powershell 复制代码

# 停止放火墙
# systemctl  stop  firewalld
# 禁用防火墙
# systemctl  disable  firewalld
# 查看防火墙状态
# systemctl  status  firewalld
# 查看防火墙状态
# firewall-cmd --state

5、禁用swap

powershell 复制代码

# 查看swap
free -m

powershell 复制代码

# 查看swap文件
cat /etc/fstab

powershell 复制代码

# 注释掉fstab的swap配置
vi /etc/fstab
# 使配置文件生效
swapoff -a

6、添加时间同步

powershell 复制代码

# 查看时间
date
# 安装插件
yum  -y  install  update
# 设置
crontab -e
# 设置内容
0  */1  *  *  *  ntpdate  ntp.aliyun.com
# 查看
crontab  -l

7、关闭selinux

powershell 复制代码

# 查看
getenforce
# 查看
sestatus

powershell 复制代码

# 编辑配置文件
vi /etc/selinux/config
SELINUX=disabled

重启系统后

8、启用Cgroup；修改配置文件/etc/default/grub，启用cgroup内存限额功能,配置两个参数：

powershell 复制代码

vi /etc/default/grub
GRUB_CMDLINE_LINUX_DEFAULT="cgroup_enable=memory swapaccount=1"
GRUB_CMDLINE_LINUX="cgroup_enable=memory swapaccount=1"
# 更新grub
grub2-mkconfig -o /boot/grub2/grub.cfg

9、添加rancher用户

powershell 复制代码

# 添加用户
useradd -m rancher
# 添加至docker组
usermod  -aG  docker  rancher
# 添加用户密码
passwd rancher

powershell 复制代码

# 如果rancher用户没有root权限，可以使用visudo编辑/etc/hosts文件
visudo

根据需要可以选择下面四行中的一行：

powershell 复制代码

youuser ALL=(ALL) ALL 
%youuser ALL=(ALL) ALL 
youuser ALL=(ALL) NOPASSWD: ALL 
%youuser ALL=(ALL) NOPASSWD: ALL

第一行:允许用户youuser执行sudo命令(需要输入密码).

第二行:允许用户组youuser里面的用户执行sudo命令(需要输入密码).

第三行:允许用户youuser执行sudo命令,并且在执行的时候不输入密码.

第四行:允许用户组youuser里面的用户执行sudo命令,并且在执行的时候不输入密码.

参考：linux给普通用户添加sudo权限

10、配置ssh,需要切换到rancher用户

powershell 复制代码

# 切换至rancher用户
su rancher
# master生成ssh
ssh-keygen
# 将ssh密钥复制到其它node节点上
cd .ssh/
ssh-copy-id rancher@k8s-master
ssh-copy-id rancher@k8s-node1
ssh-copy-id rancher@k8s-node2

11、下载rke安装包

https://github.com/rancher/rke/releases

下载后上传到master上

切换至root用户执行

powershell 复制代码

su root
mv rke_linux-amd64 /usr/local/bin/rke && chmod +x /usr/local/bin/rke
rke --version

12、使用rancher用户创建rke安装k8s集群产生的配置文件

powershell 复制代码

# 在rancher的用户目录/root/rancher下创建集群部署cluster.yml文件
rke config --name cluster.yml
# 创建完之后，默认镜像是从dockerhub上拉取的，镜像需要替换成阿里云的仓库镜像

默认仓库配置如下：

官方配置参考

命令生成cluster.yml文件

powershell 复制代码

[+] Cluster Level SSH Private Key Path [~/.ssh/id_rsa]:               集群私钥路径：~/.ssh/id_rsa

[+] Number of Hosts [1]: 3                                     集群拥有几个节点：3

[+] SSH Address of host (1) [none]: 192.168.149.200                       第一个节点ip地址：192.168.149.200 

[+] SSH Port of host (1) [22]: 22                                        第一个节点端口：22

[+] SSH Private Key Path of host (192.168.149.200) [none]: ~/.ssh/id_rsa       第一个节点私钥路径：~/.ssh/id_rsa

[+] SSH User of host (192.168.149.200) [ubuntu]: rancher                   远程用户名：rancher

[+] Is host (192.168.149.200) a Control Plane host (y/n)? [y]: y               是否是k8s集群控制节点：y

[+] Is host (192.168.149.200) a Worker host (y/n)? [n]: n                    是否是k8s集群工作节点：n

[+] Is host (192.168.149.200) an etcd host (y/n)? [n]: n                     是否是k8s集群etcd节点：n

[+] Override Hostname of host (192.168.149.200) [none]:                   不覆盖现有主机：回车默认

[+] Internal IP of host (192.168.149.200) [none]:                         主机局域网地址：没有更改回车默认

[+] Docker socket path on host (192.168.149.200) [/var/run/docker.sock]: /var/run/docker.sock         主机上docker.sock路径：/var/run/docker.sock

[+] SSH Address of host (2) [none]: 192.168.149.205                         第二个节点ip地址：192.168.149.205

[+] SSH Port of host (2) [22]: 22                                          第二个节点远程端口：22

[+] SSH Private Key Path of host (192.168.149.205) [none]: ~/.ssh/id_rsa         第二个节点私钥路径：~/.ssh/id_rsa

[+] SSH User of host (192.168.149.205) [ubuntu]: rancher                     第二个节点远程用户名：rancher

[+] Is host (192.168.149.205) a Control Plane host (y/n)? [y]: n                 是否是k8s集群控制节点：n

[+] Is host (192.168.149.205) a Worker host (y/n)? [n]: y                      是否是k8s集群工作节点：y

[+] Is host (192.168.149.205) an etcd host (y/n)? [n]: n                       是否是k8s集群etcd节点：n

[+] Override Hostname of host (192.168.149.205) [none]:                    不覆盖现有主机：回车默认

[+] Internal IP of host (192.168.149.205) [none]:                            主机局域网地址：没有更改回车默认

[+] Docker socket path on host (192.168.149.205) [/var/run/docker.sock]: /var/run/docker.sock        主机上docker.sock路径：/var/run/docker.sock

[+] SSH Address of host (3) [none]: 192.168.149.210                             第三个节点ip地址：192.168.149.210

[+] SSH Port of host (3) [22]: 22                                              第三个节点远程端口：22

[+] SSH Private Key Path of host (192.168.149.210) [none]: ~/.ssh/id_rsa             第三个节点私钥路径：~/.ssh/id_rsa 

[+] SSH User of host (192.168.149.210) [ubuntu]: rancher                         第三个节点远程用户名：rancher

[+] Is host (192.168.149.210) a Control Plane host (y/n)? [y]: n                     是否是k8s集群控制节点：n

[+] Is host (192.168.149.210) a Worker host (y/n)? [n]: n                          是否是k8s集群工作节点：n

[+] Is host (192.168.149.210) an etcd host (y/n)? [n]: y                            是否是k8s集群etcd节点：y

[+] Override Hostname of host (192.168.149.210) [none]:                         不覆盖现有主机：回车默认

[+] Internal IP of host (192.168.149.210) [none]:                                主机局域网地址：没有更改回车默认

[+] Docker socket path on host (192.168.149.210) [/var/run/docker.sock]: /var/run/docker.sock  主机上docker.sock路径：/var/run/docker.sock

[+] Network Plugin Type (flannel, calico, weave, canal, aci) [canal]: calico                              网络插件类型：自选，我选择的是calico

[+] Authentication Strategy [x509]:                                                            认证策略形式：X509

[+] Authorization Mode (rbac, none) [rbac]: rbac                                                 认证模式：rbac

[+] Kubernetes Docker image [rancher/hyperkube:v1.25.9-rancher2]: rancher/hyperkube:v1.25.9-rancher2  k8s集群使用的docker镜像：rancher/hyperkube:v1.25.9-rancher2

[+] Cluster domain [cluster.local]: sbcinfo.com                                                   集群域名：默认即可    

[+] Service Cluster IP Range [10.43.0.0/16]:                                                     集群IP、server地址：默认即可

[+] Enable PodSecurityPolicy [n]:                                                             开启pod安全策略：n

[+] Cluster Network CIDR [10.42.0.0/16]:                                                       集群pod ip地址：默认即可

[+] Cluster DNS Service IP [10.43.0.10]:                                                        集群DNS ip地址：默认即可

[+] Add addon manifest URLs or YAML files [no]:                                                添加加载项清单url或yaml文件：回车默认即可或者no

13、开始安装集群，使用rancher用户

powershell 复制代码

# 开始执行部署前先确保个机器上的docker全部未运行，docker ps 和docker ps -a都有删除掉，清理所有不是从阿里云下载的镜像，确保集群部署使用镜像和环境的干净，然后执行下面的部署命令，观察日志直到成功
rke  up

成功部署截图：

master成功后的端口监听情况

查看监听端口命令netstat，如果没有则需使用root用户安装

yaml 复制代码

yum install net-tools

常见参数

-a (all)显示所有选项，默认不显示LISTEN相关

-t (tcp)仅显示tcp相关选项

-u (udp)仅显示udp相关选项

-n 拒绝显示别名，能显示数字的全部转化成数字。

-l 仅列出有在 Listen (监听) 的服務状态

-p 显示建立相关链接的程序名

-r 显示路由信息，路由表

-e 显示扩展信息，例如uid等

-s 按各个协议进行统计

-c 每隔一个固定时间，执行该netstat命令。

参考：CentOS下netstat 命令详解

worke工作节点镜像和启动情况截图

14、安装kubectl客户端管理工具

yaml 复制代码

cat <<EOF > /etc/yum.repos.d/kubernetes.repo

[kubernetes]

name=Kubernetes

baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64

enabled=1

gpgcheck=1

repo_gpgcheck=1

gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg

EOF

# 安装
yum install -y kubectl 
# kubectl的默认配置文件：  ~/.kube/config（这个文件是在root用户目录下的，这个config是文件不是目录）
# rke安装完成后将kube_config_cluster.yml复制并更名为config
# root用户
mkdir .kube
cd /root/.kube
cp /home/rancher/kube_config_cluster.yml config
# 更改完成后执行
kubectl get nodes

rke高可用k8s集群安装和实现手册
 CentOS7下，RKE部署k8s集群，及Helm Chart 安装Rancher高可用

测试可用的cluster.yml

yaml 复制代码

# If you intended to deploy Kubernetes in an air-gapped environment,
# please consult the documentation on how to configure custom RKE images.
nodes:
- address: 192.168.0.210
  port: "22"
  internal_address: ""
  role:
  - controlplane
  - etcd
  hostname_override: ""
  user: rancher
  docker_socket: /var/run/docker.sock
  ssh_key: ""
  ssh_key_path: ~/.ssh/id_rsa
  ssh_cert: ""
  ssh_cert_path: ""
  labels: {}
  taints: []
- address: 192.168.0.211
  port: "22"
  internal_address: ""
  role:
  - worker
  hostname_override: ""
  user: rancher
  docker_socket: /var/run/docker.sock
  ssh_key: ""
  ssh_key_path: ~/.ssh/id_rsa
  ssh_cert: ""
  ssh_cert_path: ""
  labels: {}
  taints: []
- address: 192.168.0.212
  port: "22"
  internal_address: ""
  role:
  - worker
  hostname_override: ""
  user: rancher
  docker_socket: /var/run/docker.sock
  ssh_key: ""
  ssh_key_path: ~/.ssh/id_rsa
  ssh_cert: ""
  ssh_cert_path: ""
  labels: {}
  taints: []
services:
  etcd:
    image: ""
    extra_args: {}
    extra_args_array: {}
    extra_binds: []
    extra_env: []
    win_extra_args: {}
    win_extra_args_array: {}
    win_extra_binds: []
    win_extra_env: []
    external_urls: []
    ca_cert: ""
    cert: ""
    key: ""
    path: ""
    uid: 0
    gid: 0
    snapshot: null
    retention: ""
    creation: ""
    backup_config: null
  kube-api:
    image: ""
    extra_args: {}
    extra_args_array: {}
    extra_binds: []
    extra_env: []
    win_extra_args: {}
    win_extra_args_array: {}
    win_extra_binds: []
    win_extra_env: []
    service_cluster_ip_range: 10.43.0.0/16
    service_node_port_range: ""
    pod_security_policy: false
    pod_security_configuration: ""
    always_pull_images: false
    secrets_encryption_config: null
    audit_log: null
    admission_configuration: null
    event_rate_limit: null
  kube-controller:
    image: ""
    extra_args: {}
    extra_args_array: {}
    extra_binds: []
    extra_env: []
    win_extra_args: {}
    win_extra_args_array: {}
    win_extra_binds: []
    win_extra_env: []
    cluster_cidr: 10.42.0.0/16
    service_cluster_ip_range: 10.43.0.0/16
  scheduler:
    image: ""
    extra_args: {}
    extra_args_array: {}
    extra_binds: []
    extra_env: []
    win_extra_args: {}
    win_extra_args_array: {}
    win_extra_binds: []
    win_extra_env: []
  kubelet:
    image: ""
    extra_args: {}
    extra_args_array: {}
    extra_binds: []
    extra_env: []
    win_extra_args: {}
    win_extra_args_array: {}
    win_extra_binds: []
    win_extra_env: []
    cluster_domain: cluster.local
    infra_container_image: ""
    cluster_dns_server: 10.43.0.10
    fail_swap_on: false
    generate_serving_certificate: false
  kubeproxy:
    image: ""
    extra_args: {}
    extra_args_array: {}
    extra_binds: []
    extra_env: []
    win_extra_args: {}
    win_extra_args_array: {}
    win_extra_binds: []
    win_extra_env: []
network:
  plugin: canal
  options: {}
  mtu: 0
  node_selector: {}
  update_strategy: null
  tolerations: []
authentication:
  strategy: x509
  sans: []
  webhook: null
addons: ""
addons_include: []
system_images:
  etcd: registry.cn-hangzhou.aliyuncs.com/rancher/mirrored-coreos-etcd:v3.5.6
  alpine: registry.cn-hangzhou.aliyuncs.com/rancher/rke-tools:v0.1.89
  nginx_proxy: registry.cn-hangzhou.aliyuncs.com/rancher/rke-tools:v0.1.89
  cert_downloader: registry.cn-hangzhou.aliyuncs.com/rancher/rke-tools:v0.1.89
  kubernetes_services_sidecar: registry.cn-hangzhou.aliyuncs.com/rancher/rke-tools:v0.1.89
  kubedns: registry.cn-hangzhou.aliyuncs.com/rancher/mirrored-k8s-dns-kube-dns:1.22.20
  dnsmasq: registry.cn-hangzhou.aliyuncs.com/rancher/mirrored-k8s-dns-dnsmasq-nanny:1.22.20
  kubedns_sidecar: registry.cn-hangzhou.aliyuncs.com/rancher/mirrored-k8s-dns-sidecar:1.22.20
  kubedns_autoscaler: registry.cn-hangzhou.aliyuncs.com/rancher/mirrored-cluster-proportional-autoscaler:1.8.6
  coredns: registry.cn-hangzhou.aliyuncs.com/rancher/mirrored-coredns-coredns:1.9.4
  coredns_autoscaler: registry.cn-hangzhou.aliyuncs.com/rancher/mirrored-cluster-proportional-autoscaler:1.8.6
  nodelocal: registry.cn-hangzhou.aliyuncs.com/rancher/mirrored-k8s-dns-node-cache:1.22.20
  kubernetes: registry.cn-hangzhou.aliyuncs.com/rancher/hyperkube:v1.26.7-rancher1
  flannel: registry.cn-hangzhou.aliyuncs.com/rancher/mirrored-flannel-flannel:v0.21.4
  flannel_cni: registry.cn-hangzhou.aliyuncs.com/rancher/flannel-cni:v0.3.0-rancher8
  calico_node: registry.cn-hangzhou.aliyuncs.com/rancher/mirrored-calico-node:v3.25.0
  calico_cni: registry.cn-hangzhou.aliyuncs.com/rancher/calico-cni:v3.25.0-rancher1
  calico_controllers: registry.cn-hangzhou.aliyuncs.com/rancher/mirrored-calico-kube-controllers:v3.25.0
  calico_ctl: registry.cn-hangzhou.aliyuncs.com/rancher/mirrored-calico-ctl:v3.25.0
  calico_flexvol: registry.cn-hangzhou.aliyuncs.com/rancher/mirrored-calico-pod2daemon-flexvol:v3.25.0
  canal_node: registry.cn-hangzhou.aliyuncs.com/rancher/mirrored-calico-node:v3.25.0
  canal_cni: registry.cn-hangzhou.aliyuncs.com/rancher/calico-cni:v3.25.0-rancher1
  canal_controllers: registry.cn-hangzhou.aliyuncs.com/rancher/mirrored-calico-kube-controllers:v3.25.0
  canal_flannel: registry.cn-hangzhou.aliyuncs.com/rancher/mirrored-flannel-flannel:v0.21.4
  canal_flexvol: registry.cn-hangzhou.aliyuncs.com/rancher/mirrored-calico-pod2daemon-flexvol:v3.25.0
  weave_node: registry.cn-hangzhou.aliyuncs.com/weaveworks/weave-kube:2.8.1
  weave_cni: registry.cn-hangzhou.aliyuncs.com/weaveworks/weave-npc:2.8.1
  pod_infra_container: registry.cn-hangzhou.aliyuncs.com/rancher/mirrored-pause:3.7
  ingress: registry.cn-hangzhou.aliyuncs.com/rancher/nginx-ingress-controller:nginx-1.7.0-rancher1
  ingress_backend: registry.cn-hangzhou.aliyuncs.com/rancher/mirrored-nginx-ingress-controller-defaultbackend:1.5-rancher1
  ingress_webhook: registry.cn-hangzhou.aliyuncs.com/rancher/mirrored-ingress-nginx-kube-webhook-certgen:v20230312-helm-chart-4.5.2-28-g66a760794
  metrics_server: registry.cn-hangzhou.aliyuncs.com/rancher/mirrored-metrics-server:v0.6.3
  windows_pod_infra_container: registry.cn-hangzhou.aliyuncs.com/rancher/mirrored-pause:3.7
  aci_cni_deploy_container: registry.cn-hangzhou.aliyuncs.com/noiro/cnideploy:5.2.7.1.81c2369
  aci_host_container: registry.cn-hangzhou.aliyuncs.com/noiro/aci-containers-host:5.2.7.1.81c2369
  aci_opflex_container: registry.cn-hangzhou.aliyuncs.com/noiro/opflex:5.2.7.1.81c2369
  aci_mcast_container: registry.cn-hangzhou.aliyuncs.com/noiro/opflex:5.2.7.1.81c2369
  aci_ovs_container: registry.cn-hangzhou.aliyuncs.com/noiro/openvswitch:5.2.7.1.81c2369
  aci_controller_container: registry.cn-hangzhou.aliyuncs.com/noiro/aci-containers-controller:5.2.7.1.81c2369
  aci_gbp_server_container: registry.cn-hangzhou.aliyuncs.com/noiro/gbp-server:5.2.7.1.81c2369
  aci_opflex_server_container: registry.cn-hangzhou.aliyuncs.com/noiro/opflex-server:5.2.7.1.81c2369
ssh_key_path: ~/.ssh/id_rsa
ssh_cert_path: ""
ssh_agent_auth: false
authorization:
  mode: rbac
  options: {}
ignore_docker_version: null
enable_cri_dockerd: null
kubernetes_version: ""
private_registries:
  - url: registry.cn-hangzhou.aliyuncs.com
    user: abc
    password: abc
    is_default: true
ingress:
  provider: ""
  options: {}
  node_selector: {}
  extra_args: {}
  dns_policy: ""
  extra_envs: []
  extra_volumes: []
  extra_volume_mounts: []
  update_strategy: null
  http_port: 0
  https_port: 0
  network_mode: ""
  tolerations: []
  default_backend: null
  default_http_backend_priority_class_name: ""
  nginx_ingress_controller_priority_class_name: ""
  default_ingress_class: null
cluster_name: ""
cloud_provider:
  name: ""
prefix_path: ""
win_prefix_path: ""
addon_job_timeout: 0
bastion_host:
  address: ""
  port: ""
  user: ""
  ssh_key: ""
  ssh_key_path: ""
  ssh_cert: ""
  ssh_cert_path: ""
  ignore_proxy_env_vars: false
monitoring:
  provider: ""
  options: {}
  node_selector: {}
  update_strategy: null
  replicas: null
  tolerations: []
  metrics_server_priority_class_name: ""
restore:
  restore: false
  snapshot_name: ""
rotate_encryption_key: false
dns: null