云原生Kubernetes: K8S 1.29版本 部署Sonarqube

一、实验

1.环境

（1）主机

表1 主机

|--------|--------------|--------|----------------|------------|
| 主机 | 架构 | 版本 | IP | 备注 |
| master | K8S master节点 | 1.29.0 | 192.168.204.8 | |
| node1 | K8S node节点 | 1.29.0 | 192.168.204.9 | |
| node2 | K8S node节点 | 1.29.0 | 192.168.204.10 | 已部署Kuboard |

（2）master节点查看集群

1）查看node
kubectl get node
 
2)查看node详细信息
kubectl get node -o wide
 

（3）查看pod

[root@master ~]# kubectl get pod -A

(4) 访问Kuboard

http://192.168.204.10:30080/kuboard/cluster

查看节点

2.K8S 1.29版本 部署HELM

（1）查阅

https://github.com/helm/helm/releases/tag/v3.14.4

目前最新版为v3.14.4

(2) 部署HELM

1）安装 helm 
//下载二进制 Helm client 安装包
helm-v3.14.4-linux-amd64.tar.gz
 
tar -zxvf helm-v3.14.4-linux-amd64.tar.gz
mv linux-amd64/helm /usr/local/bin/helm
helm version
 
//命令补全
source <(helm completion bash)

安装

（3）使用 helm 安装 Chart

1）查阅
https://github.com/SonarSource/helm-chart-sonarqube

2）使用 helm 安装 Chart
//添加指定的 chart 仓库，
helm repo add sonarqube https://SonarSource.github.io/helm-chart-sonarqube

3) 下载指定版本
helm pull sonarqube/sonarqube --version 10.5.0+2748       

查阅最新版本

安装

下载

（4）移动

cd ~ && mkdir sonarqube

mv sonarqube-10.5.0+2748.tgz sonarqube/

cd sonarqube/;ls

3.搭建NFS

（1）检查并安装rpcbind和nfs-utils软件包

[root@master ~]# rpm -q rpcbind nfs-utils

（2）创建目录并授权

[root@master ~]# mkdir -p /opt/sonarqube

[root@master opt]# chmod 777 sonarqube/

(3)打开nfs的配置文件

[root@master opt]# vim /etc/exports

（4）配置文件

给所有网段用户赋予读写权限、同步内容、不压缩共享对象root用户权限

......
/opt/sonarqube *(rw,sync,no_root_squash)

(5) 使NFS配置生效

[root@master opt]# exportfs -r

（6）监听端口

[root@master opt]# ss -antp | grep rpcbind

（7）查看共享

[root@master opt]# showmount -e

其他节点查看

[root@node1 ~]# showmount -e master

4.K8S 1.29版本安装nfs-provisioner

(1) 查阅

https://github.com/kubernetes-sigs/nfs-subdir-external-provisioner/releases

（2）创建目录

[root@master ~]# cd ~ && mkdir nfs-subdir-external-provisioner
[root@master ~]# cd nfs-subdir-external-provisioner/

（3）第一种方式下载

helm添加repo

[root@master nfs-subdir-external-provisioner]# helm repo add nfs-subdir-external-provisioner https://kubernetes-sigs.github.io/nfs-subdir-external-provisioner

下载

[root@master ~]# helm pull nfs-subdir-external-provisioner/nfs-subdir-external-provisioner

（5）第二种方式下载

查阅

https://artifacthub.io/packages/helm/nfs-subdir-external-provisioner/nfs-subdir-external-provisioner

点击右边的install

弹出页面

点击右下角"this link"

（6）移动并解压（选择上面的第二种方式）

[root@master ~]# mv nfs-subdir-external-provisioner-4.0.18.tgz nfs-subdir-external-provisioner

[root@master nfs-subdir-external-provisioner]# tar -xvf nfs-subdir-external-provisioner-4.0.18.tgz 

（7）node节点导入镜像

导入本地

[root@node1 ~]# docker load --input nfs-subdir-external-provisioner.tar

重新打标签

[root@node1 ~]# docker tag k8s.gcr.io/sig-storage/nfs-subdir-external-provisioner:v4.0.2 registry.k8s.io/sig-storage/n

（8）master节点安装

helm install nfs-subdir-external-provisioner nfs-subdir-external-provisioner/ --set nfs.server=192.168.204.8 --set nfs.path=/opt/sonarqube --set storageClass.name=nfs-client --set storageClass.defaultClass=true -n nfs-provisioner --create-namespace 

（9）查看pod

详细查看

[root@master ~]# kubectl describe pod nfs-subdir-external-provisioner-567b586d45-xz8r6 -n nfs-provisioner
Name:             nfs-subdir-external-provisioner-567b586d45-xz8r6
Namespace:        nfs-provisioner
Priority:         0
Service Account:  nfs-subdir-external-provisioner
Node:             node1/192.168.204.9
Start Time:       Sat, 27 Apr 2024 19:38:39 +0800
Labels:           app=nfs-subdir-external-provisioner
                  pod-template-hash=567b586d45
                  release=nfs-subdir-external-provisioner
Annotations:      cni.projectcalico.org/containerID: 8f4479951e36de27cc21dcce8b7bf11a34eb838107d4457c6ca352acbf69399e
                  cni.projectcalico.org/podIP: 10.244.166.167/32
                  cni.projectcalico.org/podIPs: 10.244.166.167/32
Status:           Running
IP:               10.244.166.167
IPs:
  IP:           10.244.166.167
Controlled By:  ReplicaSet/nfs-subdir-external-provisioner-567b586d45
Containers:
  nfs-subdir-external-provisioner:
    Container ID:   docker://9c18e809cc7179a55d66a1886b6addbd034841a6010fd07c4b4049449ab79814
    Image:          registry.k8s.io/sig-storage/nfs-subdir-external-provisioner:v4.0.2
    Image ID:       docker://sha256:932b0bface75b80e713245d7c2ce8c44b7e127c075bd2d27281a16677c8efef3
    Port:           <none>
    Host Port:      <none>
    State:          Running
      Started:      Sat, 27 Apr 2024 19:38:41 +0800
    Ready:          True
    Restart Count:  0
    Environment:
      PROVISIONER_NAME:  cluster.local/nfs-subdir-external-provisioner
      NFS_SERVER:        192.168.204.8
      NFS_PATH:          /opt/sonarqube
    Mounts:
      /persistentvolumes from nfs-subdir-external-provisioner-root (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-t248d (ro)
Conditions:
  Type                        Status
  PodReadyToStartContainers   True 
  Initialized                 True 
  Ready                       True 
  ContainersReady             True 
  PodScheduled                True 
Volumes:
  nfs-subdir-external-provisioner-root:
    Type:      NFS (an NFS mount that lasts the lifetime of a pod)
    Server:    192.168.204.8
    Path:      /opt/sonarqube
    ReadOnly:  false
  kube-api-access-t248d:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type    Reason     Age   From               Message
  ----    ------     ----  ----               -------
  Normal  Pulled     88s   kubelet            Container image "registry.k8s.io/sig-storage/nfs-subdir-external-provisioner:v4.0.2" already present on machine
  Normal  Created    88s   kubelet            Created container nfs-subdir-external-provisioner
  Normal  Started    87s   kubelet            Started container nfs-subdir-external-provisioner
  Normal  Scheduled  81s   default-scheduler  Successfully assigned nfs-provisioner/nfs-subdir-external-provisioner-567b586d45-xz8r6 to node1

（10）Kuboard查看

工作负载

容器组

详细信息

5.K8S 1.29版本 部署Sonarqube（第一种方式）

（1）解压

[root@master ~]# cd sonarqube/
[root@master sonarqube]# ls
[root@master sonarqube]# tar -xvf sonarqube-10.5.0+2748.tgz 

（2）修改values.yaml文件

[root@master sonarqube]# cd sonarqube/
[root@master sonarqube]# vim values.yaml 
......
# 全局搜索"service"关键字
service:
  type: NodePort  # 类型修改为NodePort
  externalPort: 9000
  internalPort: 9000
  nodePort: 30090 # NodePort对外暴露的端口
......
persistence:
  enabled: true  #设置为true
......
  storageClass: nfs-client  #设置为当前集群默认的StorageClass

修改前：

修改后：

（3）创建一个安装Sonarqube用的名字空间

[root@master sonarqube]# kubectl create ns sonarqube

（4）chart安装Sonarqube

[root@master sonarqube]# helm install sonarqube ./sonarqube -n sonarqube
NAME: sonarqube
LAST DEPLOYED: Sat Apr 27 20:12:09 2024
NAMESPACE: sonarqube
STATUS: deployed
REVISION: 1
NOTES:
1. Get the application URL by running these commands:
  export NODE_PORT=$(kubectl get --namespace sonarqube -o jsonpath="{.spec.ports[0].nodePort}" services sonarqube-sonarqube)
  export NODE_IP=$(kubectl get nodes --namespace sonarqube -o jsonpath="{.items[0].status.addresses[0].address}")
  echo http://$NODE_IP:$NODE_PORT
WARNING: 
         Please note that the SonarQube image runs with a non-root user (uid=1000) belonging to the root group (guid=0). In this way, the chart can support arbitrary user ids as recommended in OpenShift.
         Please visit https://docs.openshift.com/container-platform/4.14/openshift_images/create-images.html#use-uid_create-images for more information.

WARNING: The embedded PostgreSQL is intended for evaluation only, it is DEPRECATED, and it will be REMOVED in a future release.
         Please visit https://artifacthub.io/packages/helm/sonarqube/sonarqube#production-use-case for more information.

（5）输入命令

[root@master sonarqube]# export NODE_PORT=$(kubectl get --namespace sonarqube -o jsonpath="{.spec.ports[0].nodePort}" services sonarqube-sonarqube)
[root@master sonarqube]# export NODE_IP=$(kubectl get nodes --namespace sonarqube -o jsonpath="{.items[0].status.addresses[0].address}")
[root@master sonarqube]# echo http://$NODE_IP:$NODE_PORT
http://192.168.204.8:30090

（6）node节点拉取postgresql镜像

[root@node2 ~]# docker pull docker.io/bitnami/postgresql:11.14.0-debian-10-r22

#也可以使用替代愿镜像：m.daocloud.io/docker.io/bitnami/postgresql:11.14.0-debian-10-r22 

Kuboard容器组查看

(7) node节点拉取sonarqube镜像

node2拉取镜像

[root@node2 ~]# docker pull sonarqube:10.5.0-community

node2节点导出镜像

[root@node2 ~]# docker save -o sonarqube.tar sonarqube:10.5.0-community

复制Docker镜像到node1节点

[root@node2 ~]# scp sonarqube.tar root@node1:~

node1节点导入Docker镜像

[root@node1 ~]# docker load -i sonarqube.tar 

(8) 查看服务

(9)查看卷

[root@master sonarqube]# cd /opt/sonarqube/
[root@master sonarqube]# ls

Kuboard查看

（10）HELM更新配置文件

[root@master sonarqube]# helm upgrade -f sonarqube/values.yaml sonarqube ./sonarqube -n sonarqube

（11）删除项目

[root@master sonarqube]# helm uninstall sonarqube -n sonarqube

5.K8S 1.29版本 部署Sonarqube（第二种方式）

（1）创建NFS

postgresql

[root@master opt]# cd ~
[root@master ~]#  mkdir -p /opt/postgre
[root@master ~]# cd /opt
[root@master opt]# chmod 777 postgre/
[root@master opt]# vim /etc/exports
[root@master opt]# exportfs -r
[root@master opt]# showmount -e
Export list for master:
/opt/postgre   *
/opt/sonar     *
/opt/sonarqube *
/opt/nexus     *
/opt/k8s       *

sonarqube

[root@master sonarqube]# cd ~
[root@master ~]# mkdir -p /opt/sonar
[root@master ~]# 
[root@master ~]# cd /opt
[root@master opt]# chmod 777 sonar/
[root@master opt]# vim /etc/exports
[root@master opt]# exportfs -r
[root@master opt]# showmount -e
Export list for master:
/opt/sonar     *
/opt/sonarqube *
/opt/nexus     *
/opt/k8s       *

（2）创建postgresql的pv

[root@master ~]# vim pv-postgre.yaml

apiVersion: v1
kind: PersistentVolume
metadata:
  name: pv-postgre
spec:
  capacity:
    storage: 5Gi    #配置容量大小
  volumeMode: Filesystem
  accessModes:
    - ReadWriteOnce     #配置访问策略为只允许一个节点读写
  persistentVolumeReclaimPolicy: Retain  #配置回收策略，Retain为手动回收
  storageClassName: "pv-postgre"       #配置为nfs
  nfs:
    path: /opt/postgre   #配置nfs服务端的共享路径
    server: 192.168.204.8    #配置nfs服务器地址

（3）生成资源

[root@master ~]# kubectl apply -f pv-postgre.yaml 

（4）查看pv

[root@master ~]# kubectl get pv

（5）拉取镜像

node2

[root@node2 ~]# docker pull postgres:11.4

(6) 导出镜像

[root@node2 ~]# docker save -o postgres.tar postgres:11.4

（7）复制Docker镜像到node1节点

[root@node2 ~]# scp postgres.tar root@node1:~ 

（8）node1节点导入Docker镜像

[root@node1 ~]# docker load -i postgres.tar 

(9)部署postgresql

[root@master ~]# vim postgre.yaml

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: postgre-pvc
  namespace: sonarqube
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: "pv-postgre"
  resources:
    requests:
      storage: 2Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: postgres-sonar
  labels:
    app: postgres-sonar
  namespace: sonarqube
spec:
  replicas: 1
  selector:
    matchLabels:
      app: postgres-sonar
  template:
    metadata:
      labels:
        app: postgres-sonar
    spec:
      containers:
        - name: postgres-sonar
          image: postgres:11.4
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 5432
          env:
            - name: POSTGRES_DB
              value: "sonarDB"
            - name: POSTGRES_USER
              value: "sonarUser"
            - name: POSTGRES_PASSWORD
              value: "123456"
          resources:
            limits:
              cpu: 1000m
              memory: 2048Mi
            requests:
              cpu: 500m
              memory: 1024Mi
          volumeMounts:
            - name: data
              mountPath: /var/lib/postgresql/data
      volumes:
        - name: data
          persistentVolumeClaim:
            claimName: postgre-pvc
---
apiVersion: v1
kind: Service
metadata:
  name: postgres-sonar
  namespace: sonarqube
  labels:
    app: postgres-sonar
spec:
  ports:
    - port: 5432
      protocol: TCP
      targetPort: 5432
  selector:
    app: postgres-sonar

(10) 生成资源

[root@master ~]# kubectl apply -f postgre.yaml 

(11) 查看pv,pvc

[root@master ~]# kubectl get pv

[root@master ~]# kubectl get pvc -n sonarqube

(12) 查看pod,svc

[root@master ~]#  kubectl get pod.svc -n sonarqube

(13)Kuboard查看

工作负载

容器组

服务

（14）创建sonarqube的pv

[root@master ~]# vim pv-sonar.yaml

apiVersion: v1
kind: PersistentVolume
metadata:
  name: pv-sonar
spec:
  capacity:
    storage: 10Gi    #配置容量大小
  volumeMode: Filesystem
  accessModes:
    - ReadWriteOnce     #配置访问策略为只允许一个节点读写
  persistentVolumeReclaimPolicy: Retain  #配置回收策略，Retain为手动回收
  storageClassName: "pv-sonar"       #配置为nfs
  nfs:
    path: /opt/sonar   #配置nfs服务端的共享路径
    server: 192.168.204.8    #配置nfs服务器地址

（15）生成资源

[root@master ~]# kubectl apply -f pv-sonar.yaml 

（16）查看pv

[root@master ~]# kubectl get pv

（17）拉取镜像

node1

[root@node1 ~]# docker pull sonarqube:lts

(18) 导出镜像

[root@node1 ~]# docker save -o sonar.tar sonarqube:lts

（19）复制Docker镜像到node1节点

[root@node1 ~]# scp sonar.tar root@node2:~ 

（20）node1节点导入Docker镜像

[root@node2 ~]# docker load -i sonar.tar 

(21) 部署sonarqube

[root@master ~]# vim sonar.yaml

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: sonarqube-pvc
  namespace: sonarqube
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: "pv-sonar"
  resources:
    requests:
      storage: 5Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: sonarqube
  labels:
    app: sonarqube
  namespace: sonarqube
spec:
  replicas: 1
  selector:
    matchLabels:
      app: sonarqube
  template:
    metadata:
      labels:
        app: sonarqube
    spec:
      initContainers:
        - name: init-sysctl
          image: busybox
          imagePullPolicy: IfNotPresent
          command: ["sysctl", "-w", "vm.max_map_count=262144"]
          securityContext:
            privileged: true
      containers:
        - name: sonarqube
          image: sonarqube:lts
          ports:
            - containerPort: 9000
          env:
            - name: SONARQUBE_JDBC_USERNAME
              value: "sonarUser"
            - name: SONARQUBE_JDBC_PASSWORD
              value: "123456"
            - name: SONARQUBE_JDBC_URL
              value: "jdbc:postgresql://postgres-sonar:5432/sonarDB"  #postgres-sonar改成集群IP
          livenessProbe:
            httpGet:
              path: /sessions/new
              port: 9000
            initialDelaySeconds: 60
            periodSeconds: 30
          readinessProbe:
            httpGet:
              path: /sessions/new
              port: 9000
            initialDelaySeconds: 60
            periodSeconds: 30
            failureThreshold: 6
          resources:
            limits:
              cpu: 2000m
              memory: 2048Mi
            requests:
              cpu: 1000m
              memory: 1024Mi
          volumeMounts:
            - mountPath: /opt/sonarqube/conf
              name: data
              subPath: conf
            - mountPath: /opt/sonarqube/data
              name: data
              subPath: data
            - mountPath: /opt/sonarqube/extensions
              name: data
              subPath: extensions
      volumes:
        - name: data
          persistentVolumeClaim:
            claimName: sonarqube-pvc
---
apiVersion: v1
kind: Service
metadata:
  name: sonarqube
  namespace: sonarqube
  labels:
    app: sonarqube
spec:
  type: NodePort
  ports:
    - name: sonarqube
      port: 9000
      targetPort: 9000
      nodePort: 30090
      protocol: TCP
  selector:
    app: sonarqube

(22) 生成资源

[root@master ~]# kubectl apply -f sonar.yaml 

(23) 查看pv,pvc

[root@master ~]# kubectl get pv

[root@master ~]#  kubectl get pvc -n sonarqube

(24)查看pod,svc

[root@master ~]# kubectl get pod,svc -n sonarqube

（25）部署ingress

[root@master ~]# vim ingress-sonar.yaml

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress-sonar
  namespace: sonarqube
spec:
  ingressClassName: "nginx"
  rules:
  - host: sonarqube.site
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: sonarqube
            port:
              number: 9000

(26)生成资源

[root@master ~]# kubectl apply -f ingress-sonar.yaml 

（27）查看ingress

[root@master ~]# kubectl get ingress -n sonarqube

（28）详细查看

[root@master ~]# kubectl describe ingress ingress-sonar -n sonarqube
Name:             ingress-sonar
Labels:           <none>
Namespace:        sonarqube
Address:          10.101.23.182
Ingress Class:    nginx
Default backend:  <default>
Rules:
  Host            Path  Backends
  ----            ----  --------
  sonarqube.site  
                  /   sonarqube:9000 (10.244.166.129:9000)
Annotations:      <none>
Events:
  Type    Reason  Age                From                      Message
  ----    ------  ----               ----                      -------
  Normal  Sync    72s (x2 over 86s)  nginx-ingress-controller  Scheduled for sync
  Normal  Sync    72s (x2 over 86s)  nginx-ingress-controller  Scheduled for sync

(29）Kuboard查看

应用路由

详细信息

（30）master节点修改hosts

[root@master ~]# vim /etc/hosts

（31）查看

ingress-nginx-controller对外暴露端口为31820

（32）curl测试

[root@master ~]# curl sonarqube.site:31820

（33）物理机修改hosts

（34）访问系统

http://sonarqube.site:31820

（35）输入用户名和密码

账号：admin

密码：admin

(36) 设置新密码

弹出

修改

（37）进入系统

（38）其他方式的Sonarqube部署

可以参考本人博客：

持续集成交付CICD：CentOS 7 安装 Sonarqube9.6-CSDN博客

二、问题

1.chart安装Sonarqube报错

（1）报错

Error: INSTALLATION FAILED: cannot load values.yaml: error converting YAML to JSON: yaml: line 67: mapping values are not allowed in this context

（2）原因

格式错误。

（3）解决方法

修改配置文件。

修改前：

修改后：

2.K8S 部署sonarqube报错

（1）报错

查看pod,svc

查看deploy

(2) 原因分析

JDBC连接postgresql失败，value: "jdbc:postgresql://postgres-sonar:5432/sonarDB"中的postgres-sonar需要写入集群IP。

（3）解决方法

修改配置文件：

[root@master ~]# kubectl edit deploy sonarqube  -n sonarqube

修改后：

成功：

[root@master ~]# kubectl get pod,svc -n sonarqube