K8S Helm 安装ingress-nginx/ingress-nginx

crabdave1232023-12-29 15:56

安装ingress-nginx/ingress-nginx

添加helm仓库

复制代码 
  - [root@k8s-master ~]# helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
  - [root@k8s-master ~]# helm repo update

下载安装包

  • helm pull ingress-nginx/ingress-nginx
    • 解压
      • tar -zxvf ingress-nginx-4.0.1.tgz

备份并修改 values.yaml 文件

  • 修改controller的镜像地址

    controller:
    name: /
    enableAnnotationValidations: false
    image:
    ## Keep false as default for now!
    chroot: false
    registry: docker.io
    image: willdockerhub/ingress-nginx-controller
    ## for backwards compatibility consider setting the full image url via the repository value below
    ## use either current default registry/image or repository format or installing chart by providing the values.yaml will fail
    ## repository:
    tag: "v1.0.0"
    digest: sha256:0851b34f69f69352bf168e6ccf30e1e20714a264ab1ecd1933e4d8c0fc3215c6
    pullPolicy: IfNotPresent
    # www-data -> uid 101
    runAsUser: 101
    allowPrivilegeEscalation: true

    • 修改 hostNetwork 的值为 true：

is merged hostNetwork: true ## Use host ports 80 and 443

    • dnsPolicy的值改为: ClusterFirstWithHostNet

to keep resolving names inside the k8s network, use ClusterFirstWithHostNet. dnsPolicy: ClusterFirstWithHostNet

    • kind类型更改为：DaemonSet

      -- Use a DaemonSet or Deployment

      kind: DaemonSet

      -- Annotations to be added to the controller Deployment or DaemonSet

    • kube-webhook-certgen的镜像地址改为国内仓库地址

      patch:
      enabled: true
      image:
      registry: registry.cn-hangzhou.aliyuncs.com
      image: google_containers/kube-webhook-certgen
      ## for backwards compatibility consider setting the full image url via the repository value below
      ## use either current default registry/image or repository format or installing chart by providing the values.yaml will fail
      ## repository:
      tag: "v20231011-8b53cabe0"
      #digest: sha256:a7943503b45d552785aa3b5e457f169a5661fb94d82b8a3373bcd9ebaf9aac80
      digest: sha256:488fc1dcc9269161ac781ffb5df0a9751cb64693bf195fe76e57f211db332dd9
      pullPolicy: IfNotPresent

安装

  • 使用当前目录中的values.yaml文件

    • helm install ingress-nginx -n ingress-nginx .

  • 指定values.yaml文件，并使用debug参数，方便调试异常

    • helm upgrade --install ingress-nginx ingress-nginx/ingress-nginx -n ingress-nginx --values values.yaml --debug
      • upgrade --install 会导致重新下载包
        • 如果不想重新下载，直接使用install

  • 安装成功，并给了一个创建ingress的例子

    NOTES:
    The ingress-nginx controller has been installed.
    It may take a few minutes for the LoadBalancer IP to be available.
    You can watch the status by running 'kubectl --namespace ingress-nginx get services -o wide -w ingress-nginx-controller'

    An example Ingress that makes use of the controller:
    apiVersion: networking.k8s.io/v1
    kind: Ingress
    metadata:
    name: example
    namespace: foo
    spec:
    ingressClassName: nginx
    rules:
    - host: www.example.com
    http:
    paths:
    - pathType: Prefix
    backend:
    service:
    name: exampleService
    port:
    number: 80
    path: /
    # This section is only required if TLS is to be enabled for the Ingress
    tls:
    - hosts:
    - www.example.com
    secretName: example-tls

    If TLS is enabled for the Ingress, a Secret containing the certificate and key must also be provided:

    apiVersion: v1
    kind: Secret
    metadata:
    name: example-tls
    namespace: foo
    data:
    tls.crt: <base64 encoded cert>
    tls.key: <base64 encoded key>
    type: kubernetes.io/tls

  • 处于pending状态的ingress-nginx-controller

    [root@k8s-master01 ingress-nginx]# kubectl get svc -n ingress-nginx
    NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
    ingress-nginx-controller LoadBalancer 10.99.163.44 <pending> 80:31268/TCP,443:31052/TCP 4m54s
    ingress-nginx-controller-admission ClusterIP 10.100.131.12 <none> 443/TCP 4m54s

    [root@k8s-master01 ingress-nginx]# kubectl --namespace ingress-nginx get services -o wide -w ingress-nginx-controller
    NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
    ingress-nginx-controller LoadBalancer 10.99.163.44 <pending> 80:31268/TCP,443:31052/TCP 94s app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx

  • 给节点打标签ingress=true

    • 需要给节点打上刚刚设置的标签ingress=true，让 Pod 调度到指定的节点，比如调度到 master 节点

      给master节点打上标签 ingress=ture

      [root@k8s-master ingress-nginx]# kubectl label node master1 ingress=true
      node/master1 labeled

      k8s默认集群中，出于安全考虑，默认配置下Kubernetes不会将Pod调度到Master节点。测试环境无所谓，所以执行下面命令去除master的污点：

      [root@k8s-master ingress-nginx]# kubectl taint node master1 node-role.kubernetes.io/master-

参考：

How do I set up a pod to allow scheduling on the master · Issue #1814 · k3s-io/k3s · GitHub How do I set up a pod to allow scheduling on the master

K8S 将 pod 调度到指定 nodes 上运行 - 知乎 K8S 将 pod 调度到指定 nodes 上运行

复制代码 
[root@master01 rocketmq]# kubectl describe nodes | grep Taints
Taints:             node-role.kubernetes.io/control-plane:NoSchedule
Taints:             <none>
Taints:             <none>
[root@master01 rocketmq]# kubectl taint nodes --all node-role.kubernetes.io/control-plane-
node/master01 untainted
taint "node-role.kubernetes.io/control-plane" not found
taint "node-role.kubernetes.io/control-plane" not found
[root@master01 rocketmq]# kubectl describe nodes | grep Taints
Taints:             <none>
Taints:             <none>
Taints:             <none>
[root@master01 rocketmq]# kubectl get nodes
NAME       STATUS   ROLES           AGE   VERSION
master01   Ready    control-plane   44d   v1.28.2
node02     Ready    <none>          44d   v1.28.2
node03     Ready    <none>          44d   v1.28.2
[root@master01 rocketmq]# kubectl describe nodes | grep Taints
Taints:             <none>
Taints:             <none>
Taints:             <none>
[root@master01 rocketmq]# kubectl get pods -A -owide |grep ingress
ingress-nginx            ingress-nginx-controller-bc96v                           1/1     Running            0                  41s     yourip     master01   <none>           <none>

k8s命令（pod相关（驱逐、强制删除）、让Master当Node用、修改nodeport端口范围、修改k8sDNS、运行nslookup容器）_kubectl drain-CSDN博客

复制代码 
#将 Master 也当作 Node 使用
kubectl taint node nodename node-role.kubernetes.io/master-

[root@app01 home]# kubectl taint node app01 node-role.kubernetes.io/master-
node/app01 untainted
[root@app01 home]# 

#将 Master 恢复成 Master Only 状态
kubectl taint node nodename node-role.kubernetes.io/master="":NoSchedule
  
[root@app01 rabbitmq]# kubectl taint node app01  node-role.kubernetes.io/master="":NoSchedule
node/app01 tainted
[root@app01 rabbitmq]#
    • 本次直接将标签打到了node02上
      • 所以配置的域名要指向node02节点的ip地址
      • ingress-nginx-controller对应的pod会自动部署到node02节点上

配置访问hosts

  • 192.168.221.132 harbor.david.org

  • 测试

    • 用浏览器访问没问，curl缺少header中的一些参数，所以报308错误

    [root@k8s-master01 harbor]# curl harbor.david.org

    <html> <head><title>308 Permanent Redirect</title></head> <body>

    308 Permanent Redirect

    nginx
    </body> </html>

configmap权限异常处理

  • 查看ingress-nginx-controller日志

    • root@k8s-master01 harbor\]# kubectl logs -n ingress-nginx ingress-nginx-controller-4nl8l

      • E1122 05:49:57.933496 7 leaderelection.go:329] error initially creating leader election record: configmaps is forbidden: User "system:serviceaccount:ingress-nginx:ingress-nginx" cannot create resource "configmaps" in API group "" in the namespace "ingress-nginx"
        • 需要create和update权限，修改role即可

          • root@k8s-master01 nginx\]# kubectl edit clusterrole -n ingress-nginx ingress-nginx * 添加 * - create * - update * 保存并退出，再查看ingress-nginx-controller日志 * \[root@k8s-master01 harbor\]# kubectl logs -n ingress-nginx ingress-nginx-controller-4nl8l

      rules:

      • apiGroups:
        • ""
          resources:
        • configmaps
        • endpoints
        • nodes
        • pods
        • secrets
        • namespaces
          verbs:
        • list
        • watch
        • create
        • update

异常调试

  • 安装加上debug参数 展示详细安装过程，方便排查错误

  • client.go:779: [debug] ingress-nginx-admission-create: Jobs active: 0, jobs failed: 0, jobs succeeded: 0

  • 提示job有异常，先找到这个job

    [root@k8s-master01 ingress-nginx]# kubectl get job -A
    NAMESPACE NAME COMPLETIONS DURATION AGE
    ingress-nginx ingress-nginx-admission-create 0/1 8s 8s
    ingress-nginx ingress-nginx-admission-patch 0/1 69m 69m

    [root@k8s-master01 ingress-nginx]# kubectl get pods -A
    ingress-nginx ingress-nginx-admission-create-r7bps

  • 有个job对应的pod有异常

    [root@k8s-master01 ingress-nginx]# kubectl describe pods -n ingress-nginx ingress-nginx-admission-create-r7bps

    Normal BackOff 5s (x2 over 28s) kubelet Back-off pulling image "registry.aliyuncs.com/google_containers/kube-webhook-certgen:v1.5.1@sha256:a7943503b45d552785aa3b5e457f169a5661fb94d82b8a3373bcd9ebaf9aac80"

提示镜像有异常

Failed to pull image "registry.cn-hangzhou.aliyuncs.com/google_containers/kube-webhook-certgen:v20231011-8b53cabe0@sha256:a7943503b45d552785aa3b5e457f169a5661fb94d82b8a3373bcd9ebaf9aac80": rpc error: code = NotFound desc = failed to pull and unpack image "registry.cn-hangzhou.aliyuncs.com/google_containers/kube-webhook-certgen@sha256:a7943503b45d552785aa3b5e457f169a5661fb94d82b8a3373bcd9ebaf9aac80": failed to resolve reference "registry.cn-hangzhou.aliyuncs.com/google_containers/kube-webhook-certgen@sha256:a7943503b45d552785aa3b5e457f169a5661fb94d82b8a3373bcd9ebaf9aac80": registry.cn-hangzhou.aliyuncs.com/google_containers/kube-webhook-certgen@sha256:a7943503b45d552785aa3b5e457f169a5661fb94d82b8a3373bcd9ebaf9aac80: not found

修改了镜像还不管用，还需要修改sha256

使用ctr下载该镜像

root@k8s-master01 ingress-nginx\]# ctr -n=k8s.io images pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-webhook-certgen:v20231011-8b53cabe0 ##### 找到该镜像 \[root@k8s-master01 ingress-nginx\]# ctr -n=k8s.io images list registry.cn-hangzhou.aliyuncs.com/google_containers/kube-webhook-certgen:v20231011-8b53cabe0 application/vnd.docker.distribution.manifest.list.v2+json sha256:488fc1dcc9269161ac781ffb5df0a9751cb64693bf195fe76e57f211db332dd9 22.1 MiB linux/amd64,linux/arm/v7,linux/arm64,linux/s390x io.cri-containerd.image=managed ##### 把sha256字符串加到values.yaml里 digest: sha256:488fc1dcc9269161ac781ffb5df0a9751cb64693bf195fe76e57f211db332dd9 ##### 再次安装即可 *

相关推荐
网硕互联的小客服1 小时前
服务器经常出现蓝屏是什么原因导致的？如何排查和修复？
运维·服务器·stm32·单片机·网络安全
喜欢吃豆1 小时前
从零构建MCP服务器：FastMCP实战指南
运维·服务器·人工智能·python·大模型·mcp
海外空间恒创科技1 小时前
恒创科技：香港站群服务器做seo站群优化效果如何
运维·服务器·科技
搬码临时工1 小时前
内网服务器怎么设置公网远程访问？ windows桌面连接和Linux自带SSH外网异地跨网用完整步骤教程
运维·服务器·ssh
Rudon滨海渔村2 小时前
解决阿里云ubuntu内存溢出导致vps死机无法访问 - 永久性增加ubuntu的swap空间 - 阿里云Linux实例内存溢出(OOM)问题修复方案
linux·运维·ubuntu
苹果醋32 小时前
Vue3组合式API应用：状态共享与逻辑复用最佳实践
java·运维·spring boot·mysql·nginx
A-刘晨阳2 小时前
【Linux】Redis 6.2.6 的二进制部署【适用于多版本】
linux·运维·redis
蓝纹绿茶2 小时前
【Mac】实现Docker下载安装【正在逐步完善】
macos·docker·容器
2401_861615283 小时前
跨平台的ARM 和 x86 Docker 镜像：汇编语言实验环境搭建
linux·汇编·ubuntu·docker·容器
自由游戏开发者4 小时前
用U盘启动制作centos系统最常见报错，系统卡住无法继续问题（手把手）
linux·运维·centos
热门推荐
01集群聊天服务器---MySQL数据库的建立02Java学习第十五部分——MyBatis03Coze扣子平台完整体验和实践（附国内和国际版对比）04基于odoo17的设计模式详解---单例模式05使用Ruby接入实时行情API教程06扣子（coze）实战|我用扣子搭建了一个自动分析小红薯笔记内容的AI应用|详细步骤拆解07《深入设计模式》模式结构汇总08基于odoo17的设计模式详解---装饰模式09DeepSeek各版本说明与优缺点分析10Java类变量（静态变量）