云服务器部署k8s集群

在两台不同厂商的云服务器上部署k8s集群，遇到一些问题。在此进行下总结。

首先要网络能够互通，我是通过添加虚拟网卡的方式

lsmod | grep ip_vs #检查是否有开启

#临时开启ip_vs

for i in (ls /lib/modules/(uname -r)/kernel/net/netfilter/ipvs|grep -o "^[^.]*");do echo i; /sbin/modinfo -F filename i >/dev/null 2>&1 && /sbin/modprobe $i; done

#永久开启

ls /lib/modules/$(uname -r)/kernel/net/netfilter/ipvs|grep -o "^[^.]*" >> /etc/modules

CENTOS7 增加虚拟网卡

确认内核是否有tun模块

modinfo tun

yum install tunctl -y

如果找不到

vim /etc/yum.repos.d/nux-misc.repo

nux-misc

name=Nux Misc

baseurl=http://li.nux.ro/download/nux/misc/el7/x86_64/

enabled=0

gpgcheck=1

gpgkey=http://li.nux.ro/download/nux/RPM-GPG-KEY-nux.ro

yum --enablerepo=nux-misc install tunctl

---

tunctl -t tap0 -u root

tap0 是虚拟网卡名字

ifconfig tap0 172.168.1.1 netmask 255.255.255.0 promisc

172.168.1.1 是ip地址

查看

ifconfig tap0

然后建立虚拟IP，如果有节点不是公网内网分离则不需要

#ifconfig查看网卡有哪些

#临时建立虚拟网卡，注意如果多个服务器都是公网内网分离，则都需要建立虚拟网卡

ifconfig eth1:0 公网IP netmask 255.255.255.0 up

#删除

ifconfig eth1:0 down

#永久建立

vim /etc/network/interfaces

auto eth1:0

iface eth1:0 inet static

name Ethernet alias LAN card

address 192.168.33.149

netmask 255.255.255.0

broadcast 192.168.33.255

/etc/init.d/networking restart

#ifconfig查看网卡有哪些
#临时建立虚拟网卡，注意如果多个服务器都是公网内网分离，则都需要建立虚拟网卡
ifconfig eth1:0 公网IP netmask 255.255.255.0 up
#删除
ifconfig eth1:0 down
 
#永久建立
vim  /etc/network/interfaces
auto eth1:0  
iface eth1:0 inet static  
name Ethernet alias LAN card  
address 192.168.33.149 
netmask 255.255.255.0  
broadcast 192.168.33.255
 
/etc/init.d/networking restart

观察参数修改是否生效命令：

sysctl --system

安装docker

安装k8s相关组件

修改kubelet的文件

vim /usr/lib/systemd/system/kubelet.service.d/10-kubeadm.conf

注意，这步很重要，如果不做，节点仍然会使用内网IP注册进集群

在末尾添加参数 --node-ip=公网IP

Note: This dropin only works with kubeadm and kubelet v1.11+

Service

Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf"

Environment="KUBELET_CONFIG_ARGS=--config=/var/lib/kubelet/config.yaml"

This is a file that "kubeadm init" and "kubeadm join" generates at runtime, populating the KUBELET_KUBEADM_ARGS variable dynamically

EnvironmentFile=-/var/lib/kubelet/kubeadm-flags.env

This is a file that the user can use for overrides of the kubelet args as a last resort. Preferably, the user should use

the .NodeRegistration.KubeletExtraArgs object in the configuration files instead. KUBELET_EXTRA_ARGS should be sourced from this file.

EnvironmentFile=-/etc/sysconfig/kubelet

ExecStart=

ExecStart=/usr/bin/kubelet KUBELET_KUBECONFIG_ARGS KUBELET_CONFIG_ARGS KUBELET_KUBEADM_ARGS KUBELET_EXTRA_ARGS --node-ip=公网ip

vim /usr/lib/systemd/system/kubelet.service.d/10-kubeadm.conf
 
# 注意，这步很重要，如果不做，节点仍然会使用内网IP注册进集群
# 在末尾添加参数 --node-ip=公网IP
 
# Note: This dropin only works with kubeadm and kubelet v1.11+
[Service]
Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf"
Environment="KUBELET_CONFIG_ARGS=--config=/var/lib/kubelet/config.yaml"
# This is a file that "kubeadm init" and "kubeadm join" generates at runtime, populating the KUBELET_KUBEADM_ARGS variable dynamically
EnvironmentFile=-/var/lib/kubelet/kubeadm-flags.env
# This is a file that the user can use for overrides of the kubelet args as a last resort. Preferably, the user should use
# the .NodeRegistration.KubeletExtraArgs object in the configuration files instead. KUBELET_EXTRA_ARGS should be sourced from this file.
EnvironmentFile=-/etc/sysconfig/kubelet
ExecStart=
ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS --node-ip=公网ip

创建master节点

部署dashboard时一直处于Pending状态

kubectl describe pod kubernetes-dashboard-5947dc95db-n42cs -n kubernetes-dashboard

1、node节点的taint（污点）和toleration（容忍）

使用kubeadm初始化的集群，出于安全考虑Pod不会被调度到Master Node上，也就是说Master Node不参与工作负载。这是因为当前的master节点被打上了node-role.kubernetes.io/master:NoSchedule的污点：

[root@master] ~$ kubectl describe nodes master.hanli.com |grep -E '(Roles|Taints)'
Roles:              master
Taints:             node-role.kubernetes.io/master:NoSchedul

因为这里搭建的是测试环境，内存资源不太足，可以选择去掉这个污点使master来承担一些工作负载：

先看一下taint命令的语法格式

kubectl taint node [node] key=value[effect]  

其中[effect] 可取值: [ NoSchedule | PreferNoSchedule | NoExecute ]

NoSchedule: 一定不能被调度

PreferNoSchedule: 尽量不要调度，实在没有地方调度的情况下，才考虑可以调度过来

NoExecute: 不仅不会调度, 还会立即驱逐Node上已有的Pod

现在去掉污点：key的值node-role.kubernetes.io，value为空

[root@master] ~$ kubectl taint nodes master.hanli.com node-role.kubernetes.io/master-
node/master.hanli.com untainted

修改后查看Taints字段状态：

1. [root@master] ~$ kubectl describe nodes master.hanli.com |grep -E '(Roles|Taints)'

2. Roles: <none>

3. Taints: <none>

如果要重新使master打上污点，即恢复Master 的不可调度状态，执行如下命令：

1. [root@master] ~$ kubectl taint nodes master.hanli.com node-role.kubernetes.io/master=:NoSchedule

2. node/master.hanli.com tainted

这里要注意的是如果你丢掉了"="，会报 error: at least one taint update is required错误

[root@master] ~$ kubectl taint nodes master.hanli.com node-role.kubernetes.io/master:NoSchedule

error: at least one taint update is required

除了直接去除污点，我们还可以在pod的yaml 配置文件中添加tolerations 字段来容忍 节点的污点

例如，如果你想让你的pod可以被调度到master节点上，在 pod 的 spec 中添加如下设置即可：

tolerations:

- key: "node-role.kubernetes.io/master"

operator: "Exists"

effect: "NoSchedule"