一、实验
1.环境
(1)主机
表1 主机
|--------|--------------|--------|----------------|------------|
| 主机 | 架构 | 版本 | IP | 备注 |
| master | K8S master节点 | 1.29.0 | 192.168.204.8 | |
| node1 | K8S node节点 | 1.29.0 | 192.168.204.9 | |
| node2 | K8S node节点 | 1.29.0 | 192.168.204.10 | 已部署Kuboard |
(2)master节点查看集群
bash
1)查看node
kubectl get node
2)查看node详细信息
kubectl get node -o wide
(3)查看pod
bash
[root@master ~]# kubectl get pod -A
(4) 访问Kuboard
bash
http://192.168.204.10:30080/kuboard/cluster
查看节点
2.K8S 1.29版本 部署HELM
(1)查阅
bash
https://github.com/helm/helm/releases/tag/v3.14.4
目前最新版为v3.14.4
(2) 部署HELM
bash
1)安装 helm
//下载二进制 Helm client 安装包
helm-v3.14.4-linux-amd64.tar.gz
tar -zxvf helm-v3.14.4-linux-amd64.tar.gz
mv linux-amd64/helm /usr/local/bin/helm
helm version
//命令补全
source <(helm completion bash)
安装
(3)使用 helm 安装 Chart
bash
1)查阅
https://github.com/SonarSource/helm-chart-sonarqube
2)使用 helm 安装 Chart
//添加指定的 chart 仓库,
helm repo add sonarqube https://SonarSource.github.io/helm-chart-sonarqube
3) 下载指定版本
helm pull sonarqube/sonarqube --version 10.5.0+2748
查阅最新版本
安装
下载
(4)移动
bash
cd ~ && mkdir sonarqube
mv sonarqube-10.5.0+2748.tgz sonarqube/
cd sonarqube/;ls
3.搭建NFS
(1)检查并安装rpcbind和nfs-utils软件包
bash
[root@master ~]# rpm -q rpcbind nfs-utils
(2)创建目录并授权
bash
[root@master ~]# mkdir -p /opt/sonarqube
bash
[root@master opt]# chmod 777 sonarqube/
(3)打开nfs的配置文件
bash
[root@master opt]# vim /etc/exports
(4)配置文件
给所有网段用户赋予读写权限、同步内容、不压缩共享对象root用户权限
bash
......
/opt/sonarqube *(rw,sync,no_root_squash)
(5) 使NFS配置生效
bash
[root@master opt]# exportfs -r
(6)监听端口
bash
[root@master opt]# ss -antp | grep rpcbind
(7)查看共享
bash
[root@master opt]# showmount -e
其他节点查看
bash
[root@node1 ~]# showmount -e master
4.K8S 1.29版本安装nfs-provisioner
(1) 查阅
bash
https://github.com/kubernetes-sigs/nfs-subdir-external-provisioner/releases
(2)创建目录
bash
[root@master ~]# cd ~ && mkdir nfs-subdir-external-provisioner
[root@master ~]# cd nfs-subdir-external-provisioner/
(3)第一种方式下载
helm添加repo
bash
[root@master nfs-subdir-external-provisioner]# helm repo add nfs-subdir-external-provisioner https://kubernetes-sigs.github.io/nfs-subdir-external-provisioner
下载
bash
[root@master ~]# helm pull nfs-subdir-external-provisioner/nfs-subdir-external-provisioner
(5)第二种方式下载
查阅
bash
https://artifacthub.io/packages/helm/nfs-subdir-external-provisioner/nfs-subdir-external-provisioner
点击右边的install
弹出页面
点击右下角"this link"
(6)移动并解压(选择上面的第二种方式)
bash
[root@master ~]# mv nfs-subdir-external-provisioner-4.0.18.tgz nfs-subdir-external-provisioner
[root@master nfs-subdir-external-provisioner]# tar -xvf nfs-subdir-external-provisioner-4.0.18.tgz
(7)node节点导入镜像
导入本地
bash
[root@node1 ~]# docker load --input nfs-subdir-external-provisioner.tar
重新打标签
bash
[root@node1 ~]# docker tag k8s.gcr.io/sig-storage/nfs-subdir-external-provisioner:v4.0.2 registry.k8s.io/sig-storage/n
(8)master节点安装
bash
helm install nfs-subdir-external-provisioner nfs-subdir-external-provisioner/ --set nfs.server=192.168.204.8 --set nfs.path=/opt/sonarqube --set storageClass.name=nfs-client --set storageClass.defaultClass=true -n nfs-provisioner --create-namespace
(9)查看pod
详细查看
bash
[root@master ~]# kubectl describe pod nfs-subdir-external-provisioner-567b586d45-xz8r6 -n nfs-provisioner
Name: nfs-subdir-external-provisioner-567b586d45-xz8r6
Namespace: nfs-provisioner
Priority: 0
Service Account: nfs-subdir-external-provisioner
Node: node1/192.168.204.9
Start Time: Sat, 27 Apr 2024 19:38:39 +0800
Labels: app=nfs-subdir-external-provisioner
pod-template-hash=567b586d45
release=nfs-subdir-external-provisioner
Annotations: cni.projectcalico.org/containerID: 8f4479951e36de27cc21dcce8b7bf11a34eb838107d4457c6ca352acbf69399e
cni.projectcalico.org/podIP: 10.244.166.167/32
cni.projectcalico.org/podIPs: 10.244.166.167/32
Status: Running
IP: 10.244.166.167
IPs:
IP: 10.244.166.167
Controlled By: ReplicaSet/nfs-subdir-external-provisioner-567b586d45
Containers:
nfs-subdir-external-provisioner:
Container ID: docker://9c18e809cc7179a55d66a1886b6addbd034841a6010fd07c4b4049449ab79814
Image: registry.k8s.io/sig-storage/nfs-subdir-external-provisioner:v4.0.2
Image ID: docker://sha256:932b0bface75b80e713245d7c2ce8c44b7e127c075bd2d27281a16677c8efef3
Port: <none>
Host Port: <none>
State: Running
Started: Sat, 27 Apr 2024 19:38:41 +0800
Ready: True
Restart Count: 0
Environment:
PROVISIONER_NAME: cluster.local/nfs-subdir-external-provisioner
NFS_SERVER: 192.168.204.8
NFS_PATH: /opt/sonarqube
Mounts:
/persistentvolumes from nfs-subdir-external-provisioner-root (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-t248d (ro)
Conditions:
Type Status
PodReadyToStartContainers True
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
nfs-subdir-external-provisioner-root:
Type: NFS (an NFS mount that lasts the lifetime of a pod)
Server: 192.168.204.8
Path: /opt/sonarqube
ReadOnly: false
kube-api-access-t248d:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Pulled 88s kubelet Container image "registry.k8s.io/sig-storage/nfs-subdir-external-provisioner:v4.0.2" already present on machine
Normal Created 88s kubelet Created container nfs-subdir-external-provisioner
Normal Started 87s kubelet Started container nfs-subdir-external-provisioner
Normal Scheduled 81s default-scheduler Successfully assigned nfs-provisioner/nfs-subdir-external-provisioner-567b586d45-xz8r6 to node1
(10)Kuboard查看
工作负载
容器组
详细信息
5.K8S 1.29版本 部署Sonarqube(第一种方式)
(1)解压
bash
[root@master ~]# cd sonarqube/
[root@master sonarqube]# ls
[root@master sonarqube]# tar -xvf sonarqube-10.5.0+2748.tgz
(2)修改values.yaml文件
bash
[root@master sonarqube]# cd sonarqube/
[root@master sonarqube]# vim values.yaml
......
# 全局搜索"service"关键字
service:
type: NodePort # 类型修改为NodePort
externalPort: 9000
internalPort: 9000
nodePort: 30090 # NodePort对外暴露的端口
......
persistence:
enabled: true #设置为true
......
storageClass: nfs-client #设置为当前集群默认的StorageClass
修改前:
修改后:
(3)创建一个安装Sonarqube用的名字空间
bash
[root@master sonarqube]# kubectl create ns sonarqube
(4)chart安装Sonarqube
bash
[root@master sonarqube]# helm install sonarqube ./sonarqube -n sonarqube
NAME: sonarqube
LAST DEPLOYED: Sat Apr 27 20:12:09 2024
NAMESPACE: sonarqube
STATUS: deployed
REVISION: 1
NOTES:
1. Get the application URL by running these commands:
export NODE_PORT=$(kubectl get --namespace sonarqube -o jsonpath="{.spec.ports[0].nodePort}" services sonarqube-sonarqube)
export NODE_IP=$(kubectl get nodes --namespace sonarqube -o jsonpath="{.items[0].status.addresses[0].address}")
echo http://$NODE_IP:$NODE_PORT
WARNING:
Please note that the SonarQube image runs with a non-root user (uid=1000) belonging to the root group (guid=0). In this way, the chart can support arbitrary user ids as recommended in OpenShift.
Please visit https://docs.openshift.com/container-platform/4.14/openshift_images/create-images.html#use-uid_create-images for more information.
WARNING: The embedded PostgreSQL is intended for evaluation only, it is DEPRECATED, and it will be REMOVED in a future release.
Please visit https://artifacthub.io/packages/helm/sonarqube/sonarqube#production-use-case for more information.
(5)输入命令
bash
[root@master sonarqube]# export NODE_PORT=$(kubectl get --namespace sonarqube -o jsonpath="{.spec.ports[0].nodePort}" services sonarqube-sonarqube)
[root@master sonarqube]# export NODE_IP=$(kubectl get nodes --namespace sonarqube -o jsonpath="{.items[0].status.addresses[0].address}")
[root@master sonarqube]# echo http://$NODE_IP:$NODE_PORT
http://192.168.204.8:30090
(6)node节点拉取postgresql镜像
bash
[root@node2 ~]# docker pull docker.io/bitnami/postgresql:11.14.0-debian-10-r22
#也可以使用替代愿镜像:m.daocloud.io/docker.io/bitnami/postgresql:11.14.0-debian-10-r22
Kuboard容器组查看
(7) node节点拉取sonarqube镜像
node2拉取镜像
bash
[root@node2 ~]# docker pull sonarqube:10.5.0-community
node2节点导出镜像
bash
[root@node2 ~]# docker save -o sonarqube.tar sonarqube:10.5.0-community
复制Docker镜像到node1节点
bash
[root@node2 ~]# scp sonarqube.tar root@node1:~
node1节点导入Docker镜像
bash
[root@node1 ~]# docker load -i sonarqube.tar
(8) 查看服务
(9)查看卷
bash
[root@master sonarqube]# cd /opt/sonarqube/
[root@master sonarqube]# ls
Kuboard查看
(10)HELM更新配置文件
bash
[root@master sonarqube]# helm upgrade -f sonarqube/values.yaml sonarqube ./sonarqube -n sonarqube
(11)删除项目
bash
[root@master sonarqube]# helm uninstall sonarqube -n sonarqube
5.K8S 1.29版本 部署Sonarqube(第二种方式)
(1)创建NFS
postgresql
bash
[root@master opt]# cd ~
[root@master ~]# mkdir -p /opt/postgre
[root@master ~]# cd /opt
[root@master opt]# chmod 777 postgre/
[root@master opt]# vim /etc/exports
[root@master opt]# exportfs -r
[root@master opt]# showmount -e
Export list for master:
/opt/postgre *
/opt/sonar *
/opt/sonarqube *
/opt/nexus *
/opt/k8s *
sonarqube
bash
[root@master sonarqube]# cd ~
[root@master ~]# mkdir -p /opt/sonar
[root@master ~]#
[root@master ~]# cd /opt
[root@master opt]# chmod 777 sonar/
[root@master opt]# vim /etc/exports
[root@master opt]# exportfs -r
[root@master opt]# showmount -e
Export list for master:
/opt/sonar *
/opt/sonarqube *
/opt/nexus *
/opt/k8s *
(2)创建postgresql的pv
bash
[root@master ~]# vim pv-postgre.yaml
bash
apiVersion: v1
kind: PersistentVolume
metadata:
name: pv-postgre
spec:
capacity:
storage: 5Gi #配置容量大小
volumeMode: Filesystem
accessModes:
- ReadWriteOnce #配置访问策略为只允许一个节点读写
persistentVolumeReclaimPolicy: Retain #配置回收策略,Retain为手动回收
storageClassName: "pv-postgre" #配置为nfs
nfs:
path: /opt/postgre #配置nfs服务端的共享路径
server: 192.168.204.8 #配置nfs服务器地址
(3)生成资源
bash
[root@master ~]# kubectl apply -f pv-postgre.yaml
(4)查看pv
bash
[root@master ~]# kubectl get pv
(5)拉取镜像
node2
bash
[root@node2 ~]# docker pull postgres:11.4
(6) 导出镜像
bash
[root@node2 ~]# docker save -o postgres.tar postgres:11.4
(7)复制Docker镜像到node1节点
bash
[root@node2 ~]# scp postgres.tar root@node1:~
(8)node1节点导入Docker镜像
bash
[root@node1 ~]# docker load -i postgres.tar
(9)部署postgresql
bash
[root@master ~]# vim postgre.yaml
bash
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: postgre-pvc
namespace: sonarqube
spec:
accessModes:
- ReadWriteOnce
storageClassName: "pv-postgre"
resources:
requests:
storage: 2Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: postgres-sonar
labels:
app: postgres-sonar
namespace: sonarqube
spec:
replicas: 1
selector:
matchLabels:
app: postgres-sonar
template:
metadata:
labels:
app: postgres-sonar
spec:
containers:
- name: postgres-sonar
image: postgres:11.4
imagePullPolicy: IfNotPresent
ports:
- containerPort: 5432
env:
- name: POSTGRES_DB
value: "sonarDB"
- name: POSTGRES_USER
value: "sonarUser"
- name: POSTGRES_PASSWORD
value: "123456"
resources:
limits:
cpu: 1000m
memory: 2048Mi
requests:
cpu: 500m
memory: 1024Mi
volumeMounts:
- name: data
mountPath: /var/lib/postgresql/data
volumes:
- name: data
persistentVolumeClaim:
claimName: postgre-pvc
---
apiVersion: v1
kind: Service
metadata:
name: postgres-sonar
namespace: sonarqube
labels:
app: postgres-sonar
spec:
ports:
- port: 5432
protocol: TCP
targetPort: 5432
selector:
app: postgres-sonar
(10) 生成资源
bash
[root@master ~]# kubectl apply -f postgre.yaml
(11) 查看pv,pvc
bash
[root@master ~]# kubectl get pv
[root@master ~]# kubectl get pvc -n sonarqube
(12) 查看pod,svc
bash
[root@master ~]# kubectl get pod.svc -n sonarqube
(13)Kuboard查看
工作负载
容器组
服务
(14)创建sonarqube的pv
bash
[root@master ~]# vim pv-sonar.yaml
bash
apiVersion: v1
kind: PersistentVolume
metadata:
name: pv-sonar
spec:
capacity:
storage: 10Gi #配置容量大小
volumeMode: Filesystem
accessModes:
- ReadWriteOnce #配置访问策略为只允许一个节点读写
persistentVolumeReclaimPolicy: Retain #配置回收策略,Retain为手动回收
storageClassName: "pv-sonar" #配置为nfs
nfs:
path: /opt/sonar #配置nfs服务端的共享路径
server: 192.168.204.8 #配置nfs服务器地址
(15)生成资源
bash
[root@master ~]# kubectl apply -f pv-sonar.yaml
(16)查看pv
bash
[root@master ~]# kubectl get pv
(17)拉取镜像
node1
bash
[root@node1 ~]# docker pull sonarqube:lts
(18) 导出镜像
bash
[root@node1 ~]# docker save -o sonar.tar sonarqube:lts
(19)复制Docker镜像到node1节点
bash
[root@node1 ~]# scp sonar.tar root@node2:~
(20)node1节点导入Docker镜像
bash
[root@node2 ~]# docker load -i sonar.tar
(21) 部署sonarqube
bash
[root@master ~]# vim sonar.yaml
bash
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: sonarqube-pvc
namespace: sonarqube
spec:
accessModes:
- ReadWriteOnce
storageClassName: "pv-sonar"
resources:
requests:
storage: 5Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: sonarqube
labels:
app: sonarqube
namespace: sonarqube
spec:
replicas: 1
selector:
matchLabels:
app: sonarqube
template:
metadata:
labels:
app: sonarqube
spec:
initContainers:
- name: init-sysctl
image: busybox
imagePullPolicy: IfNotPresent
command: ["sysctl", "-w", "vm.max_map_count=262144"]
securityContext:
privileged: true
containers:
- name: sonarqube
image: sonarqube:lts
ports:
- containerPort: 9000
env:
- name: SONARQUBE_JDBC_USERNAME
value: "sonarUser"
- name: SONARQUBE_JDBC_PASSWORD
value: "123456"
- name: SONARQUBE_JDBC_URL
value: "jdbc:postgresql://postgres-sonar:5432/sonarDB" #postgres-sonar改成集群IP
livenessProbe:
httpGet:
path: /sessions/new
port: 9000
initialDelaySeconds: 60
periodSeconds: 30
readinessProbe:
httpGet:
path: /sessions/new
port: 9000
initialDelaySeconds: 60
periodSeconds: 30
failureThreshold: 6
resources:
limits:
cpu: 2000m
memory: 2048Mi
requests:
cpu: 1000m
memory: 1024Mi
volumeMounts:
- mountPath: /opt/sonarqube/conf
name: data
subPath: conf
- mountPath: /opt/sonarqube/data
name: data
subPath: data
- mountPath: /opt/sonarqube/extensions
name: data
subPath: extensions
volumes:
- name: data
persistentVolumeClaim:
claimName: sonarqube-pvc
---
apiVersion: v1
kind: Service
metadata:
name: sonarqube
namespace: sonarqube
labels:
app: sonarqube
spec:
type: NodePort
ports:
- name: sonarqube
port: 9000
targetPort: 9000
nodePort: 30090
protocol: TCP
selector:
app: sonarqube
(22) 生成资源
bash
[root@master ~]# kubectl apply -f sonar.yaml
(23) 查看pv,pvc
bash
[root@master ~]# kubectl get pv
[root@master ~]# kubectl get pvc -n sonarqube
(24)查看pod,svc
bash
[root@master ~]# kubectl get pod,svc -n sonarqube
(25)部署ingress
bash
[root@master ~]# vim ingress-sonar.yaml
bash
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: ingress-sonar
namespace: sonarqube
spec:
ingressClassName: "nginx"
rules:
- host: sonarqube.site
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: sonarqube
port:
number: 9000
(26)生成资源
bash
[root@master ~]# kubectl apply -f ingress-sonar.yaml
(27)查看ingress
bash
[root@master ~]# kubectl get ingress -n sonarqube
(28)详细查看
bash
[root@master ~]# kubectl describe ingress ingress-sonar -n sonarqube
Name: ingress-sonar
Labels: <none>
Namespace: sonarqube
Address: 10.101.23.182
Ingress Class: nginx
Default backend: <default>
Rules:
Host Path Backends
---- ---- --------
sonarqube.site
/ sonarqube:9000 (10.244.166.129:9000)
Annotations: <none>
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Sync 72s (x2 over 86s) nginx-ingress-controller Scheduled for sync
Normal Sync 72s (x2 over 86s) nginx-ingress-controller Scheduled for sync
(29)Kuboard查看
应用路由
详细信息
(30)master节点修改hosts
bash
[root@master ~]# vim /etc/hosts
(31)查看
ingress-nginx-controller对外暴露端口为31820
(32)curl测试
bash
[root@master ~]# curl sonarqube.site:31820
(33)物理机修改hosts
(34)访问系统
bash
http://sonarqube.site:31820
(35)输入用户名和密码
bash
账号:admin
密码:admin
(36) 设置新密码
弹出
修改
(37)进入系统
(38)其他方式的Sonarqube部署
可以参考本人博客:
持续集成交付CICD:CentOS 7 安装 Sonarqube9.6-CSDN博客
二、问题
1.chart安装Sonarqube报错
(1)报错
bash
Error: INSTALLATION FAILED: cannot load values.yaml: error converting YAML to JSON: yaml: line 67: mapping values are not allowed in this context
(2)原因
格式错误。
(3)解决方法
修改配置文件。
修改前:
修改后:
2.K8S 部署sonarqube报错
(1)报错
查看pod,svc
查看deploy
(2) 原因分析
JDBC连接postgresql失败,value: "jdbc:postgresql://postgres-sonar:5432/sonarDB"中的postgres-sonar需要写入集群IP。
(3)解决方法
修改配置文件:
bash
[root@master ~]# kubectl edit deploy sonarqube -n sonarqube
修改后:
成功:
bash
[root@master ~]# kubectl get pod,svc -n sonarqube