【微服务部署】06-日志集成

Tiger_shl2023-08-31 16:46

文章目录

    • [1. EFK日志三件套集成](#1. EFK日志三件套集成)
      • [1.1 核心组件](#1.1 核心组件)
      • [1.2 部署](#1.2 部署)
    • [2. Exceptionless日志系统](#2. Exceptionless日志系统)
      • [2.1 Exceptionless核心特性](#2.1 Exceptionless核心特性)
      • [2.2 Exceptionless部署文件](#2.2 Exceptionless部署文件)
      • [2.3 K8s中使用Exceptionless](#2.3 K8s中使用Exceptionless)

1. EFK日志三件套集成

1.1 核心组件

  • Elasticsearch（存储）
  • Fluentd（收集器）
  • Kibana（数据看板）

每个Kubernetes节点上可以运行一个Fluentd实例，这个实例会去收集所有Pod打印出来的日志流。Fluentd可以做到无侵入的收集日志；

多个Kubernetes节点组成了Kubernetes集群，每个Fluentd收集到的日志都推送到Elasticsearch中。Elasticsearch可以单独部署，也可以部署在Kuberbetes集群中；

Kibana负责管理Elasticsearch，并作为数据看板

1.2 部署

需要部署Elasticsearch、Fluentd、Kibana

Elasticsearch

复制代码 
apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: elasticsearch
  namespace: default
  labels:
    tag: elasticsearch
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: elasticsearch
    spec:
      containers:
      - name: elasticsearch
        image: "elasticsearch:7.5.2"
        imagePullPolicy: IfNotPresent
        env:
        - name: discovery.type
          value: "single-node"
        volumeMounts:
          - mountPath: "/usr/share/elasticsearch/data"
            name: elasticsearch-storage
        ports:
        - containerPort: 9200
        - containerPort: 9300
        resources:
            limits:
              cpu: 1000m
              memory: 2048Mi
            requests:
              cpu: 100m
              memory: 512Mi
      terminationGracePeriodSeconds: 10
      volumes:
        - name: elasticsearch-storage
          hostPath:
              path: "/d/k8s/volumes/elasticsearch/data"
              type: DirectoryOrCreate
      restartPolicy: Always

--- 

  
apiVersion: v1
kind: Service
metadata:
  name: elasticsearch
  namespace: default
  labels:
    tag: "elasticsearch"
spec:
  type: NodePort
  ports:
  - nodePort: 30007
    port: 9200
    targetPort: 9200
    name: "9200"
    protocol: TCP
  - nodePort: 30008
    port: 9300
    targetPort: 9300
    name: "9300"
    protocol: TCP
  selector:
    app: elasticsearch

Fluentd

复制代码 
apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: fluentd
  namespace: default
  labels:
    tag: fluentd
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: fluentd
    spec:
      containers:
      - name: fluentd
        image: "witskeeper/fluentd-es:68"
        imagePullPolicy: IfNotPresent
        env:
        - name: discovery.type
          value: "single-node"
        volumeMounts:
          - mountPath: "/fluentd/etc/fluent.conf"
            name: fluentd-config
            subPath: fluent.conf
          - mountPath: "/var/log"
            name: containers-logs
          - mountPath: "/var/lib/docker/containers"
            name: docker-logs
        ports:
        - containerPort: 24224
        resources:
            limits:
              cpu: 1000m
              memory: 2048Mi
            requests:
              cpu: 100m
              memory: 512Mi
      terminationGracePeriodSeconds: 10
      volumes:
        - name: fluentd-config
          configMap:
              name: fluentd-config
        - name: containers-logs
          hostPath:
              path: "/var/log"
              type: DirectoryOrCreate
        - name: docker-logs
          hostPath:
            path: /var/lib/docker/containers
      restartPolicy: Always

--- 

  
apiVersion: v1
kind: Service
metadata:
  name: fluentd
  namespace: default
  labels:
    tag: "fluentd"
spec:
  type: NodePort
  ports:
  - nodePort: 30011
    port: 24224
    targetPort: 24224
    protocol: TCP
  selector:
    app: fluentd

Kibana

复制代码 
apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: kibana
  namespace: default
  labels:
    tag: kibana
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: kibana
    spec:
      containers:
      - name: kibana
        image: "kibana:6.8.6"
        imagePullPolicy: IfNotPresent
        env:
        - name: discovery.type
          value: "single-node"
        - name: XPACK_MONITORING_ENABLED
          value: "true"
        ports:
        - containerPort: 5601
        resources:
            limits:
              cpu: 1000m
              memory: 2048Mi
            requests:
              cpu: 100m
              memory: 512Mi
      terminationGracePeriodSeconds: 10
      restartPolicy: Always

--- 

  
apiVersion: v1
kind: Service
metadata:
  name: kibana
  namespace: default
  labels:
    tag: "kibana"
spec:
  type: NodePort
  ports:
  - nodePort: 30009
    port: 5601
    targetPort: 5601
    protocol: TCP
  selector:
    app: kibana

2. Exceptionless日志系统

2.1 Exceptionless核心特性

  • 友好的界面
  • 内置日志分类看板
  • 按团队和项目管理应用
  • 支持Web hook发送异常通知
  • 基于.Net Core开源

Exceptionless由UI、Job、Api三个镜像组成。背后依赖了Redis存储和文件存储；

文件存储是用来存储应用提交的日志文件，然后通过Job将这些文件推送到Elasticsearch中，用户可以通过UI和Api从dashboard上查询异常日志；

应用程序通过Api推送异常日志

2.2 Exceptionless部署文件

复制代码 
apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: exceptionless-api
  namespace: default
  labels:
    tag: exceptionless-api
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: exceptionless-api
    spec:
      containers:
      - name: exceptionless-api
        image: "exceptionless/api:6.0.0"
        imagePullPolicy: IfNotPresent
        env:
        - name: EX_AppMode
          value: "Production"
        - name: EX_BaseURL
          value: http://localhost:30013
        - name: EX_ConnectionStrings__Cache
          value: provider=redis
        - name: EX_ConnectionStrings__Elasticsearch
          value: server=http://elasticsearch:9200;enable-size-plugin=false
        #- name: EX_ConnectionStrings__Email
        #  value: smtps://user:password@smtp.host.com:587
        - name: EX_ConnectionStrings__MessageBus
          value: provider=redis
        - name: EX_ConnectionStrings__Queue
          value: provider=redis
        - name: EX_ConnectionStrings__Redis
          value: server=redis,abortConnect=false
        - name: EX_ConnectionStrings__Storage
          value: provider=folder;path=/app/storage
        - name: EX_RunJobsInProcess
          value: 'false'
        volumeMounts:
          - mountPath: "/app/storage"
            name: exceptionless-storage
        ports:
        - containerPort: 80
        resources:
            limits:
              cpu: 1000m
              memory: 2048Mi
            requests:
              cpu: 100m
              memory: 128Mi
      terminationGracePeriodSeconds: 10
      volumes:
        - name: exceptionless-storage
          hostPath:
              path: "/d/k8s/volumes/exceptionless" // 本地存储目录
              type: DirectoryOrCreate
      restartPolicy: Always

---

apiVersion: v1
kind: Service
metadata:
  name: exceptionless-api
  namespace: default
  labels:
    tag: "exceptionless-api"
spec:
  type: NodePort
  ports:
  - nodePort: 30012
    port: 80
    targetPort: 80
    protocol: TCP
  selector:
    app: exceptionless-api

---

apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: exceptionless-job
  namespace: default
  labels:
    tag: exceptionless-job
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: exceptionless-job
    spec:
      containers:
      - name: exceptionless-job
        image: "exceptionless/job:6.0.0"
        imagePullPolicy: IfNotPresent
        env:
        - name: EX_AppMode
          value: "Production"
        - name: EX_BaseURL
          value: http://localhost:30013
        - name: EX_ConnectionStrings__Cache
          value: provider=redis
        - name: EX_ConnectionStrings__Elasticsearch
          value: server=http://elasticsearch:9200;enable-size-plugin=false
        #- name: EX_ConnectionStrings__Email
        #  value: smtps://user:password@smtp.host.com:587
        - name: EX_ConnectionStrings__MessageBus
          value: provider=redis
        - name: EX_ConnectionStrings__Queue
          value: provider=redis
        - name: EX_ConnectionStrings__Redis
          value: server=redis,abortConnect=false
        - name: EX_ConnectionStrings__Storage
          value: provider=folder;path=/app/storage
        - name: EX_RunJobsInProcess
          value: 'false'
        volumeMounts:
          - mountPath: "/app/storage"
            name: exceptionless-job-storage
        ports:
        - containerPort: 80
        resources:
            limits:
              cpu: 1000m
              memory: 2048Mi
            requests:
              cpu: 100m
              memory: 128Mi
      terminationGracePeriodSeconds: 10
      volumes:
        - name: exceptionless-job-storage
          hostPath:
              path: "/d/k8s/volumes/exceptionless"
              type: DirectoryOrCreate
      restartPolicy: Always

---

apiVersion: v1
kind: Service
metadata:
  name: exceptionless-job
  namespace: default
  labels:
    tag: "exceptionless-job"
spec:
  type: NodePort
  ports:
  - nodePort: 30014
    port: 80
    targetPort: 80
    protocol: TCP

---
apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: exceptionless-ui
  namespace: default
  labels:
    tag: exceptionless-ui
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: exceptionless-ui
    spec:
      containers:
      - name: exceptionless-ui
        image: "exceptionless/ui:2.8.1497"
        imagePullPolicy: IfNotPresent
        env:
        - name: AppMode
          value: "Production"
        - name: EX_ApiUrl
          value: http://localhost:30012
        ports:
        - containerPort: 80
        resources:
            limits:
              cpu: 1000m
              memory: 2048Mi
            requests:
              cpu: 100m
              memory: 128Mi
      terminationGracePeriodSeconds: 10
      restartPolicy: Always

---

apiVersion: v1
kind: Service
metadata:
  name: exceptionless-ui
  namespace: default
  labels:
    tag: "exceptionless-ui"
spec:
  type: NodePort
  ports:
  - nodePort: 30013
    port: 80
    targetPort: 80
    protocol: TCP
  selector:
    app: exceptionless-ui

// Program
public static int Main(string[] args)
{
    Log.Logger = new LoggerConfiguration().ReadFrom.Configuration(Configuration)
   .MinimumLevel.Debug()
   .Enrich.FromLogContext()
   .WriteTo.Console(new RenderedCompactJsonFormatter())
   //.WriteTo.Fluentd("localhost", 30011, tag: "geektime-ordering-api", restrictedToMinimumLevel: Serilog.Events.LogEventLevel.Debug)
   .CreateLogger();
   // 将Configuration中的exceptionless读取出来,并绑定到它默认的配置上去
    Configuration.GetSection("exceptionless").Bind(Exceptionless.ExceptionlessClient.Default.Configuration);
    try
    {
        Log.Information("Starting web host");
        CreateHostBuilder(args).Build().Run();
        return 0;
    }
    catch (Exception ex)
    {
        Log.Fatal(ex, "Host terminated unexpectedly");
        return 1;
    }
    finally
    {
        Log.CloseAndFlush();
    }
}


// 配置文件
"exceptionless": {
    "ApiKey": "PSaokqmXC8T4xgWal9I0atA8TlYG7Ytz65JvxAL3",
    "ServerUrl": "http://localhost:30012"
  },

// startup
public void Configure(IApplicationBuilder app, IWebHostEnvironment env)
{
	app.UseExceptionless();// 放在所有中间件最前面.为了能捕获到所有异常
}

2.3 K8s中使用Exceptionless

配置文件

复制代码 
 "exceptionless": {
    "ApiKey": "PSaokqmXC8T4xgWal9I0atA8TlYG7Ytz65JvxAL3" // 需要是当前项目的秘钥
  },

因为serverUrl地址是共享的，地址被配置在env.txt中

要将配置引用进来,需要在deployment.yaml中注册serverUrl为环境变量

通过EFK收集应用的全量日志,Exceptionless收集异常日志

相关推荐
绝无仅有4 分钟前
某大厂MySQL面试之SQL注入触点发现与SQLMap测试
后端·面试·github
绝无仅有10 分钟前
某互联网大厂的面试go语言从基础到实战的经验和总结
后端·面试·github
澡点睡觉10 分钟前
【golang长途旅行第38站】工厂模式
开发语言·后端·golang
小蒜学长13 分钟前
基于SpringBoot+Vue的健身房管理系统的设计与实现（代码+数据库+LW）
java·数据库·vue.js·spring boot·后端
这里有鱼汤16 分钟前
你以为 FastAPI 足够强？其实 Litestar 能让你的项目更轻量高效
后端·python
失散1318 分钟前
分布式专题——2 深入理解Redis线程模型
java·数据库·redis·分布式·架构
菜鸟谢32 分钟前
windows vscode go 编译速度慢问题
后端
Victor3561 小时前
Redis（51）如何监控Redis哨兵的状态？
后端
止观止1 小时前
GitHub App 架构解析与最佳实践
架构·github
IT_陈寒1 小时前
Python性能优化：5个被低估的魔法方法让你的代码提速50%
前端·人工智能·后端
热门推荐
012025 年高教社杯全国大学生数学建模竞赛C 题 NIPT 的时点选择与胎儿的异常判定 完整成品思路模型代码分享，全网首发高质量！！！022025年数学建模国赛C题超详细解题思路032025全国大学生数学建模C题保姆级思路模型（持续更新）：NIPT 的时点选择与胎儿的异常判定042025高教社杯国赛数学建模选题建议+初步分析052025国赛B题保姆级教程思路分析 碳化硅外延层厚度的确定06不再让Windows更新！&Edge游戏助手卸载及关闭自动更新07UV安装并设置国内源08KGG转MP3工具|非KGM文件|解密音频09奈飞工厂官网，国内Netflix影视在线看|中文网页电脑版入口10（E题｜AI 辅助智能体测）2025年高教杯全国大学生数学建模国赛解题思路|完整代码论文集合