metric-server概述：

Metrics Server从kubelets收集资源指标，并通过Metrics API将它们暴露在Kubernetes apiserver中，以供HPA(Horizontal Pod Autoscaler)和VPA(Vertical Pod Autoscaler)使用。

Metrics API也可以通过kubectl top访问，从而更容易调试自动缩放管道。

参考链接:

https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/metrics-server

https://kubernetes.io/docs/tasks/debug/debug-cluster/resource-metrics-pipeline/

https://github.com/kubernetes-sigs/metrics-server

部署metric-server:

cat > deoloyment-metric-server.yaml <<'EOF'

apiVersion: rbac.authorization.k8s.io/v1

kind: ClusterRoleBinding

metadata:

labels:

kubernetes.io/cluster-service: "true"

addonmanager.kubernetes.io/mode: Reconcile

roleRef:

apiGroup: rbac.authorization.k8s.io

kind: ClusterRole

subjects:

kind: ServiceAccount

namespace: kube-system

apiVersion: rbac.authorization.k8s.io/v1

kind: RoleBinding

metadata:

namespace: kube-system

labels:

kubernetes.io/cluster-service: "true"

addonmanager.kubernetes.io/mode: Reconcile

roleRef:

apiGroup: rbac.authorization.k8s.io

kind: Role

subjects:

kind: ServiceAccount

namespace: kube-system

apiVersion: apiregistration.k8s.io/v1beta1

kind: APIService

metadata:

labels:

kubernetes.io/cluster-service: "true"

addonmanager.kubernetes.io/mode: Reconcile

spec:

service:

namespace: kube-system

group: metrics.k8s.io

version: v1beta1

insecureSkipTLSVerify: true

groupPriorityMinimum: 100

versionPriority: 100

apiVersion: v1

kind: ServiceAccount

metadata:

namespace: kube-system

labels:

kubernetes.io/cluster-service: "true"

addonmanager.kubernetes.io/mode: Reconcile

apiVersion: v1

kind: ConfigMap

metadata:

namespace: kube-system

labels:

kubernetes.io/cluster-service: "true"

addonmanager.kubernetes.io/mode: EnsureExists

data:

NannyConfiguration: |-

apiVersion: nannyconfig/v1alpha1

kind: NannyConfiguration

apiVersion: apps/v1

kind: Deployment

metadata:

namespace: kube-system

labels:

k8s-app: metrics-server

kubernetes.io/cluster-service: "true"

addonmanager.kubernetes.io/mode: Reconcile

version: v0.3.3

spec:

selector:

matchLabels:

k8s-app: metrics-server

version: v0.3.3

template:

metadata:

labels:

k8s-app: metrics-server

version: v0.3.3

annotations:

scheduler.alpha.kubernetes.io/critical-pod: ''

seccomp.security.alpha.kubernetes.io/pod: 'docker/default'

spec:

priorityClassName: system-cluster-critical

serviceAccountName: metrics-server

tolerations:

operator: Exists

containers:

name: metrics-server

metrics-server-amd64:v0.3.3

image: k8s151.oldboyedu.com:5000/metrics-server-amd64:v0.3.3

command:

/metrics-server
--metric-resolution=30s

These are needed for GKE, which doesn't support secure communication yet.

Remove these lines for non-GKE clusters, and when GKE supports token-based auth.

#- --kubelet-port=10255

#- --deprecated-kubelet-completely-insecure=true

--kubelet-insecure-tls
--kubelet-preferred-address-types=InternalIP,Hostname,InternalDNS,ExternalDNS,ExternalIP

ports:

containerPort: 443

protocol: TCP

name: metrics-server-nanny

image: registry.aliyuncs.com/google_containers/addon-resizer:1.8.5

image: k8s151.oldboyedu.com:5000/addon-resizer:1.8.5

resources:

limits:

cpu: 100m

memory: 300Mi

requests:

cpu: 5m

memory: 50Mi

env:

name: MY_POD_NAME

valueFrom:

fieldRef:

fieldPath: metadata.name

name: MY_POD_NAMESPACE

valueFrom:

fieldRef:

fieldPath: metadata.namespace

volumeMounts:

name: metrics-server-config-volume

mountPath: /etc/config

command:

/pod_nanny
--config-dir=/etc/config

#- --cpu=80m

--extra-cpu=0.5m

#- --memory=80Mi

#- --extra-memory=8Mi

--threshold=5
--deployment=metrics-server-v0.3.3
--container=metrics-server
--poll-period=300000
--estimator=exponential
--minClusterSize=2

Specifies the smallest cluster (defined in number of nodes)

resources will be scaled to.

#- --minClusterSize={{ metrics_server_min_cluster_size }}

volumes:

name: metrics-server-config-volume

configMap:

apiVersion: v1

kind: Service

metadata:

namespace: kube-system

labels:

addonmanager.kubernetes.io/mode: Reconcile

kubernetes.io/cluster-service: "true"

kubernetes.io/name: "Metrics-server"

spec:

selector:

k8s-app: metrics-server

ports:

port: 443

protocol: TCP

targetPort: https

apiVersion: rbac.authorization.k8s.io/v1

kind: ClusterRole

metadata:

labels:

kubernetes.io/cluster-service: "true"

addonmanager.kubernetes.io/mode: Reconcile

rules:

apiGroups:
""

resources:

pods
nodes
nodes/stats
namespaces

verbs:

get
list
watch
apiGroups:
"extensions"

resources:

deployments

verbs:

get
list
update
watch

apiVersion: rbac.authorization.k8s.io/v1

kind: ClusterRoleBinding

metadata:

labels:

kubernetes.io/cluster-service: "true"

addonmanager.kubernetes.io/mode: Reconcile

roleRef:

apiGroup: rbac.authorization.k8s.io

kind: ClusterRole

subjects:

kind: ServiceAccount

namespace: kube-system

EOF

面试题: 请一个Pod做了资源限制，当CPU资源使用率高达90%时如何解决？能否实现自动扩容呢？请说下思路？

---> hpa

Pod水平自动伸缩HPA案例:

1.编写资源清单

cat > deploy-tomcat.yaml <<'EOF'

apiVersion: extensions/v1beta1

kind: Deployment

metadata:

spec:

replicas: 1

template:

metadata:

labels:

app: oldboyedu-mysql

spec:

tolerations:

operator: Exists

containers:

name: mysql

image: k8s151.oldboyedu.com:5000/oldboyedu-db/mysql:5.7

ports:

containerPort: 3306

env:

name: MYSQL_ROOT_PASSWORD

value: '123456'

apiVersion: v1

kind: Service

metadata:

spec:

selector:

app: oldboyedu-mysql

ports:

port: 3306

targetPort: 3306

apiVersion: extensions/v1beta1

kind: Deployment

metadata:

spec:

replicas: 1

template:

metadata:

labels:

app: oldboyedu-tomcat-app

spec:

tolerations:

operator: Exists

containers:

name: myweb

image: jasonyin2020/tomcat-app:v1

image: k8s151.oldboyedu.com:5000/oldboyedu-tomcat/tomcat-app:v1

resources:

limits:

cpu: "100m"

requests:

cpu: "100m"

ports:

containerPort: 8080

env:

name: MYSQL_SERVICE_HOST

value: oldboyedu-mysql

name: MYSQL_SERVICE_PORT

value: '3306'

apiVersion: v1

kind: Service

metadata:

spec:

type: NodePort

selector:

app: oldboyedu-tomcat-app

ports:

port: 8080

targetPort: 8080

nodePort: 30888

EOF

2.部署tomcat应用

kubectl apply -f deploy-tomcat.yaml

3.测试

1.支持中文输出

cat >> /etc/mysql/my.cnf <<EOF

[mysql]

default-character-set = utf8

EOF

2.插入测试数据

INSERT INTO HPE_APP.T_USERS (USER_NAME,LEVEL) VALUES ('WangJianPing',888888888),('LiuDong',999999999999);

3.查看webUI

http://10.0.0.151:30888/demo

4.创建HPA规则

kubectl autoscale deployment oldboyedu-tomcat-app --max=10 --min=2 --cpu-percent=75

auther: Jason Yin

创建密钥对

ssh-keygen -t rsa -P "" -f /root/.ssh/id_rsa -q

声明你服务器密码，建议所有节点的密码均一致，否则该脚本需要再次进行优化

export mypasswd=yinzhengjie

定义主机列表

k8s_host_list=(ceph201 ceph202 ceph203)

配置免密登录，利用expect工具免交互输入

for i in ${k8s_host_list[@]};do

expect -c "

spawn ssh-copy-id -i /root/.ssh/id_rsa.pub root@$i

expect {

\"*yes/no*\" {send \"yes\r\"; exp_continue}

\"*password*\" {send \"$mypasswd\r\"; exp_continue}

done

EOF

[root@ceph201 ~]# sh password_free_login.sh

(3)让所有节点公用同一套密钥

[root@ceph201 ~]# scp -rp ~/.ssh ceph202:~

[root@ceph201 ~]# scp -rp ~/.ssh ceph203:~

3."ceph201"节点安装"ceph-deploy"工具，用于后期部署ceph集群

(1)准备国内的软件源（含基础镜像软件源和epel源）

[root@ceph201 ~]# curl -o /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-7.repo

[root@ceph201 ~]# curl -o /etc/yum.repos.d/epel.repo http://mirrors.aliyun.com/repo/epel-7.repo

(2)配置ceph软件源（注意哈，此步骤先试用清华源，之后在第9步骤会报错，切换为阿里源即可。）

[root@ceph201 ~]# cat > /etc/yum.repos.d/oldboyedu-ceph.repo <<'EOF'

[oldboyedu-ceph]

name=oldboyedu linux82 ceph 2022

baseurl=https://mirrors.tuna.tsinghua.edu.cn/ceph/rpm-octopus/el7/x86_64/

gpgcheck=0

enable=1

[oldboyedu-ceph-tools]

name=oldboyedu linux82 ceph tools

baseurl=https://mirrors.tuna.tsinghua.edu.cn/ceph/rpm-octopus/el7/noarch/

gpgcheck=0

enable=1

[ceph]

name=ceph

baseurl=http://mirrors.aliyun.com/ceph/rpm-jewel/el7/x86_64/

gpgcheck=0

priority=1

[ceph-noarch]

name=cephnoarch

baseurl=http://mirrors.aliyun.com/ceph/rpm-jewel/el7/noarch/

gpgcheck=0

priority=1

[ceph-source]

name=Ceph source packages

baseurl=http://mirrors.aliyun.com/ceph/rpm-jewel/el7/SRPMS

gpgcheck=0

EOF

(3)"ceph201"节点安装"ceph-deploy"工具，用于后期部署ceph集群

[root@ceph201 ~]# sed -ri 's#(keepcache=)0#\11#' /etc/yum.conf

[root@ceph201 ~]# yum -y install ceph-deploy

(4)将rpm软件源推送到其它节点

[root@ceph201 ~]# scp /etc/yum.repos.d/*.repo ceph202:/etc/yum.repos.d/

[root@ceph201 ~]# scp /etc/yum.repos.d/*.repo ceph203:/etc/yum.repos.d/

4."ceph201"节点ceph环境准备

(1)"ceph201"节点安装ceph环境

[root@ceph201 ~]# yum -y install ceph ceph-mon ceph-mgr ceph-radosgw ceph-mds ceph-osd

libyaml-0.1.4-11.el7_0.x86_64.rpm # 该模块可以不选择安装哈！

python-six-1.9.0-2.el7.noarch.rpm # 该模块可以不选择安装哈！

想要偷懒的同学可以使用局域网:

[root@ceph203 ~]# wget http://192.168.17.253/Kubernetes/day08-/softwares/oldboyedu-ceph.tar.gz

[root@ceph203 ~]# tar xf oldboyedu-ceph.tar.gz && cd ~/ceph-rpm && yum -y localinstall *.rpm

(3)在ceph201管理节点添加新增节点的OSD

[root@ceph201 ~]# cd /oldboyedu/ceph/cluster && ceph-deploy osd create ceph203 --data /dev/sdb

[root@ceph201 ~]# cd /oldboyedu/ceph/cluster && ceph-deploy osd create ceph203 --data /dev/sdc

(4)验证

ceph osd tree

ceph -s

(5)添加OSD注意事项

当我们添加OSD时会自动进行数据的负载均衡哟，因此建议大家如果真需要扩容时可以考虑在业务的低谷期进行哟。

使用ceph对接kubernetes:

1.在K8S集群中安装ceph的基础包环境

(1)创建软件源

curl -o /etc/yum.repos.d/epel.repo http://mirrors.aliyun.com/repo/epel-7.repo

cat > /etc/yum.repos.d/oldboyedu-ceph.repo <<EOF

[oldboyedu-ceph]

name=oldboyedu linux82 ceph 2022

baseurl=https://mirrors.tuna.tsinghua.edu.cn/ceph/rpm-octopus/el7/x86_64/

gpgcheck=0

enable=1

[oldboyedu-ceph-tools]

name=oldboyedu linux82 ceph tools

baseurl=https://mirrors.tuna.tsinghua.edu.cn/ceph/rpm-octopus/el7/noarch/

gpgcheck=0

enable=1

EOF

(2)master节点开启rpm软件包缓存

[root@k8s151.oldboyedu.com ~]# grep keepcache /etc/yum.conf

keepcache=1

[root@k8s151.oldboyedu.com ~]#

(3)安装ceph的基础包

[root@k8s151.oldboyedu.com ~]# yum -y install ceph-common

(4)将ceph软件包打包并下发到其它k8s node节点

[root@k8s151.oldboyedu.com ~]# mkdir k8s-ceph-common

[root@k8s151.oldboyedu.com ~]# find /var/cache/yum/ -type f -name "*.rpm" | xargs mv -t k8s-ceph-common

[root@k8s151.oldboyedu.com ~]# tar zcf oldboyedu-k8s-ceph-common.tar.gz k8s-ceph-common

[root@k8s151.oldboyedu.com ~]# scp oldboyedu-k8s-ceph-common.tar.gz k8s152.oldboyedu.com:~

[root@k8s151.oldboyedu.com ~]# scp oldboyedu-k8s-ceph-common.tar.gz k8s153.oldboyedu.com:~

(5)其它所有node节点安装ceph软件包

[root@k8s152.oldboyedu.com ~]# tar xf oldboyedu-k8s-ceph-common.tar.gz && yum -y localinstall k8s-ceph-common/*.rpm

[root@k8s153.oldboyedu.com ~]# tar xf oldboyedu-k8s-ceph-common.tar.gz && yum -y localinstall k8s-ceph-common/*.rpm

(6)将ceph集群的配置文件推送到k8s所有节点

[root@ceph201 ~]# scp /etc/ceph/ceph.conf 10.0.0.151:/etc/ceph/

[root@ceph201 ~]# scp /etc/ceph/ceph.conf 10.0.0.152:/etc/ceph/

[root@ceph201 ~]# scp /etc/ceph/ceph.conf 10.0.0.153:/etc/ceph/

参考连接:

https://kubernetes.io/docs/concepts/storage/volumes/#rbd

https://github.com/kubernetes/examples/tree/master/volumes/rbd

2.使用Ceph身份验证密钥

(1)如果提供了Ceph身份验证密钥，则该密钥应首先进行base64编码，然后将编码后的字符串放入密钥yaml中。

[root@ceph201 ~]# grep key /etc/ceph/ceph.client.admin.keyring |awk '{printf "%s", $NF}'|base64

QVFBTzlpTmplSXJZSmhBQS9EUGJYSURjYmYwSHR2Z1BxQUc3MUE9PQ==

[root@ceph201 ~]#

(2)创建Secret资源，注意替换key的值，要和ceph集群的key保持一致哟，上一步我已经取出来了。

[root@k8s151.oldboyedu.com ~]# cat > 01-oldboyedu-ceph-secret.yaml <<'EOF'

apiVersion: v1

kind: Secret

metadata:

type: "kubernetes.io/rbd"

data:

key: QVFBTzlpTmplSXJZSmhBQS9EUGJYSURjYmYwSHR2Z1BxQUc3MUE9PQ==

EOF

(3)在K8S集群中创建Secret资源，以便于后期Pod使用该资源。

[root@k8s151.oldboyedu.com ~]# kubectl apply -f 01-oldboyedu-ceph-secret.yaml

[root@k8s151.oldboyedu.com ~]# kubectl get secret

参考连接:

https://github.com/kubernetes/examples/blob/master/volumes/rbd/secret/ceph-secret.yaml

3.ceph集群创建k8s专用的存储池及镜像

[root@ceph201 ~]# ceph osd pool create k8s 128 128

[root@ceph201 ~]# rbd create -p k8s --size 2G --image-feature layering oldboyedu-linux

4.K8S使用ceph的rbd镜像案例

(1)创建kubernetes集群常用的资源池

[root@ceph201 ~]# ceph osd pool create k8s 128 128

pool 'k8s' created

[root@ceph201 ~]#

(2)在k8s资源池创建块设备，值得注意的是，我们可以指定该镜像的特性，否则由于内核版本过低而无法挂载哟

[root@ceph201 ~]# rbd create -p k8s --size 1024 --image-feature layering oldboyedu-linux

(3)创建K8S的资源，将MySQL的Pod持久化到ceph集群

[root@k8s151.oldboyedu.com ~]# cat > 02-oldboyedu-mysql-deploy.yml <<EOF

apiVersion: extensions/v1beta1

kind: Deployment

metadata:

spec:

replicas: 1

template:

metadata:

labels:

app: oldboyedu-web

spec:

volumes:

name: linux82-web

rbd:

指定ceph集群的mon节点

monitors:

'10.0.0.201:6789'
'10.0.0.202:6789'
'10.0.0.203:6789'

指定存储池

pool: k8s

指定rbd镜像名称

image: oldboyedu-linux

指定文件系统类型，其会自动帮咱们进行格式化哟~

fsType: xfs

是否可读，如果设置为true表示只读权限哟~

readOnly: false

指定访问ceph集群的用户

user: admin

指定访问集群的密钥资源，这个就是我们自己创建的secret局部资源，请一定要确保Pod和secret在同一个namespace，否则无法访问哟。

secretRef:

指定镜像格式，老版本不支持该字段，因此我这里也就先注释了

imageformat: "2"

指定镜像特性，老版本不支持该字段，因此我这里就先注释了

imagefeatures: "layering"

tolerations:

operator: Exists

containers:

name: web

image: k8s151.oldboyedu.com:5000/oldboyedu-web/nginx:1.20.1

volumeMounts:

name: linux82-web

mountPath: /usr/share/nginx/html

EOF

[root@k8s151.oldboyedu.com ~]# kubectl apply -f 02-oldboyedu-mysql-deploy.yml

参考连接:

https://github.com/kubernetes/examples/blob/master/volumes/rbd/rbd-with-secret.yaml

温馨提示:

(1)使用"ceph auth ls"指令可以查看身份验证状态的列表。

(2)创建的资源如下图所示，到时候可以直接参考"/oldboyedu/k8s-ceph"目录的yaml。

(3)这个软件包会出现一大堆错误，需要我们手动逐个去排查，对学员说是一次不错的历练。上课要演示完整的排查流程。

今日内容回顾:

metrics-server *****

为"kubectl top"和"hpa".

ceph集群搭建
ceph集群的基础使用
存储池
镜像
ceph集群的扩缩容
ceph的rbd和K8S集群集成案例 *****

k8s day08

image: registry.aliyuncs.com/google_containers/metrics-server-amd64:v0.3.3

These are needed for GKE, which doesn't support secure communication yet.

Remove these lines for non-GKE clusters, and when GKE supports token-based auth.

image: registry.aliyuncs.com/google_containers/addon-resizer:1.8.5

Specifies the smallest cluster (defined in number of nodes)

resources will be scaled to.

image: jasonyin2020/tomcat-app:v1

auther: Jason Yin

创建密钥对

声明你服务器密码，建议所有节点的密码均一致，否则该脚本需要再次进行优化

定义主机列表

配置免密登录，利用expect工具免交互输入

[ceph]

name=ceph

baseurl=http://mirrors.aliyun.com/ceph/rpm-jewel/el7/x86_64/

gpgcheck=0

priority=1

[ceph-noarch]

name=cephnoarch

baseurl=http://mirrors.aliyun.com/ceph/rpm-jewel/el7/noarch/

gpgcheck=0

priority=1

[ceph-source]

name=Ceph source packages

baseurl=http://mirrors.aliyun.com/ceph/rpm-jewel/el7/SRPMS

gpgcheck=0

libyaml-0.1.4-11.el7_0.x86_64.rpm # 该模块可以不选择安装哈！

python-six-1.9.0-2.el7.noarch.rpm # 该模块可以不选择安装哈！

指定ceph集群的mon节点

指定存储池

指定rbd镜像名称

指定文件系统类型，其会自动帮咱们进行格式化哟~

是否可读，如果设置为true表示只读权限哟~

指定访问ceph集群的用户

指定访问集群的密钥资源，这个就是我们自己创建的secret局部资源，请一定要确保Pod和secret在同一个namespace，否则无法访问哟。

指定镜像格式，老版本不支持该字段，因此我这里也就先注释了

imageformat: "2"

指定镜像特性，老版本不支持该字段，因此我这里就先注释了

imagefeatures: "layering"