【微服务部署】06-日志集成

Tiger_shl2023-08-31 16:46

文章目录

    • [1. EFK日志三件套集成](#1. EFK日志三件套集成)
      • [1.1 核心组件](#1.1 核心组件)
      • [1.2 部署](#1.2 部署)
    • [2. Exceptionless日志系统](#2. Exceptionless日志系统)
      • [2.1 Exceptionless核心特性](#2.1 Exceptionless核心特性)
      • [2.2 Exceptionless部署文件](#2.2 Exceptionless部署文件)
      • [2.3 K8s中使用Exceptionless](#2.3 K8s中使用Exceptionless)

1. EFK日志三件套集成

1.1 核心组件

  • Elasticsearch（存储）
  • Fluentd（收集器）
  • Kibana（数据看板）

每个Kubernetes节点上可以运行一个Fluentd实例，这个实例会去收集所有Pod打印出来的日志流。Fluentd可以做到无侵入的收集日志；

多个Kubernetes节点组成了Kubernetes集群，每个Fluentd收集到的日志都推送到Elasticsearch中。Elasticsearch可以单独部署，也可以部署在Kuberbetes集群中；

Kibana负责管理Elasticsearch，并作为数据看板

1.2 部署

需要部署Elasticsearch、Fluentd、Kibana

Elasticsearch

apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: elasticsearch
  namespace: default
  labels:
    tag: elasticsearch
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: elasticsearch
    spec:
      containers:
      - name: elasticsearch
        image: "elasticsearch:7.5.2"
        imagePullPolicy: IfNotPresent
        env:
        - name: discovery.type
          value: "single-node"
        volumeMounts:
          - mountPath: "/usr/share/elasticsearch/data"
            name: elasticsearch-storage
        ports:
        - containerPort: 9200
        - containerPort: 9300
        resources:
            limits:
              cpu: 1000m
              memory: 2048Mi
            requests:
              cpu: 100m
              memory: 512Mi
      terminationGracePeriodSeconds: 10
      volumes:
        - name: elasticsearch-storage
          hostPath:
              path: "/d/k8s/volumes/elasticsearch/data"
              type: DirectoryOrCreate
      restartPolicy: Always

--- 

  
apiVersion: v1
kind: Service
metadata:
  name: elasticsearch
  namespace: default
  labels:
    tag: "elasticsearch"
spec:
  type: NodePort
  ports:
  - nodePort: 30007
    port: 9200
    targetPort: 9200
    name: "9200"
    protocol: TCP
  - nodePort: 30008
    port: 9300
    targetPort: 9300
    name: "9300"
    protocol: TCP
  selector:
    app: elasticsearch

Fluentd

apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: fluentd
  namespace: default
  labels:
    tag: fluentd
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: fluentd
    spec:
      containers:
      - name: fluentd
        image: "witskeeper/fluentd-es:68"
        imagePullPolicy: IfNotPresent
        env:
        - name: discovery.type
          value: "single-node"
        volumeMounts:
          - mountPath: "/fluentd/etc/fluent.conf"
            name: fluentd-config
            subPath: fluent.conf
          - mountPath: "/var/log"
            name: containers-logs
          - mountPath: "/var/lib/docker/containers"
            name: docker-logs
        ports:
        - containerPort: 24224
        resources:
            limits:
              cpu: 1000m
              memory: 2048Mi
            requests:
              cpu: 100m
              memory: 512Mi
      terminationGracePeriodSeconds: 10
      volumes:
        - name: fluentd-config
          configMap:
              name: fluentd-config
        - name: containers-logs
          hostPath:
              path: "/var/log"
              type: DirectoryOrCreate
        - name: docker-logs
          hostPath:
            path: /var/lib/docker/containers
      restartPolicy: Always

--- 

  
apiVersion: v1
kind: Service
metadata:
  name: fluentd
  namespace: default
  labels:
    tag: "fluentd"
spec:
  type: NodePort
  ports:
  - nodePort: 30011
    port: 24224
    targetPort: 24224
    protocol: TCP
  selector:
    app: fluentd

Kibana

apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: kibana
  namespace: default
  labels:
    tag: kibana
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: kibana
    spec:
      containers:
      - name: kibana
        image: "kibana:6.8.6"
        imagePullPolicy: IfNotPresent
        env:
        - name: discovery.type
          value: "single-node"
        - name: XPACK_MONITORING_ENABLED
          value: "true"
        ports:
        - containerPort: 5601
        resources:
            limits:
              cpu: 1000m
              memory: 2048Mi
            requests:
              cpu: 100m
              memory: 512Mi
      terminationGracePeriodSeconds: 10
      restartPolicy: Always

--- 

  
apiVersion: v1
kind: Service
metadata:
  name: kibana
  namespace: default
  labels:
    tag: "kibana"
spec:
  type: NodePort
  ports:
  - nodePort: 30009
    port: 5601
    targetPort: 5601
    protocol: TCP
  selector:
    app: kibana

2. Exceptionless日志系统

2.1 Exceptionless核心特性

  • 友好的界面
  • 内置日志分类看板
  • 按团队和项目管理应用
  • 支持Web hook发送异常通知
  • 基于.Net Core开源

Exceptionless由UI、Job、Api三个镜像组成。背后依赖了Redis存储和文件存储；

文件存储是用来存储应用提交的日志文件，然后通过Job将这些文件推送到Elasticsearch中，用户可以通过UI和Api从dashboard上查询异常日志；

应用程序通过Api推送异常日志

2.2 Exceptionless部署文件

apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: exceptionless-api
  namespace: default
  labels:
    tag: exceptionless-api
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: exceptionless-api
    spec:
      containers:
      - name: exceptionless-api
        image: "exceptionless/api:6.0.0"
        imagePullPolicy: IfNotPresent
        env:
        - name: EX_AppMode
          value: "Production"
        - name: EX_BaseURL
          value: http://localhost:30013
        - name: EX_ConnectionStrings__Cache
          value: provider=redis
        - name: EX_ConnectionStrings__Elasticsearch
          value: server=http://elasticsearch:9200;enable-size-plugin=false
        #- name: EX_ConnectionStrings__Email
        #  value: smtps://user:password@smtp.host.com:587
        - name: EX_ConnectionStrings__MessageBus
          value: provider=redis
        - name: EX_ConnectionStrings__Queue
          value: provider=redis
        - name: EX_ConnectionStrings__Redis
          value: server=redis,abortConnect=false
        - name: EX_ConnectionStrings__Storage
          value: provider=folder;path=/app/storage
        - name: EX_RunJobsInProcess
          value: 'false'
        volumeMounts:
          - mountPath: "/app/storage"
            name: exceptionless-storage
        ports:
        - containerPort: 80
        resources:
            limits:
              cpu: 1000m
              memory: 2048Mi
            requests:
              cpu: 100m
              memory: 128Mi
      terminationGracePeriodSeconds: 10
      volumes:
        - name: exceptionless-storage
          hostPath:
              path: "/d/k8s/volumes/exceptionless" // 本地存储目录
              type: DirectoryOrCreate
      restartPolicy: Always

---

apiVersion: v1
kind: Service
metadata:
  name: exceptionless-api
  namespace: default
  labels:
    tag: "exceptionless-api"
spec:
  type: NodePort
  ports:
  - nodePort: 30012
    port: 80
    targetPort: 80
    protocol: TCP
  selector:
    app: exceptionless-api

---

apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: exceptionless-job
  namespace: default
  labels:
    tag: exceptionless-job
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: exceptionless-job
    spec:
      containers:
      - name: exceptionless-job
        image: "exceptionless/job:6.0.0"
        imagePullPolicy: IfNotPresent
        env:
        - name: EX_AppMode
          value: "Production"
        - name: EX_BaseURL
          value: http://localhost:30013
        - name: EX_ConnectionStrings__Cache
          value: provider=redis
        - name: EX_ConnectionStrings__Elasticsearch
          value: server=http://elasticsearch:9200;enable-size-plugin=false
        #- name: EX_ConnectionStrings__Email
        #  value: smtps://user:password@smtp.host.com:587
        - name: EX_ConnectionStrings__MessageBus
          value: provider=redis
        - name: EX_ConnectionStrings__Queue
          value: provider=redis
        - name: EX_ConnectionStrings__Redis
          value: server=redis,abortConnect=false
        - name: EX_ConnectionStrings__Storage
          value: provider=folder;path=/app/storage
        - name: EX_RunJobsInProcess
          value: 'false'
        volumeMounts:
          - mountPath: "/app/storage"
            name: exceptionless-job-storage
        ports:
        - containerPort: 80
        resources:
            limits:
              cpu: 1000m
              memory: 2048Mi
            requests:
              cpu: 100m
              memory: 128Mi
      terminationGracePeriodSeconds: 10
      volumes:
        - name: exceptionless-job-storage
          hostPath:
              path: "/d/k8s/volumes/exceptionless"
              type: DirectoryOrCreate
      restartPolicy: Always

---

apiVersion: v1
kind: Service
metadata:
  name: exceptionless-job
  namespace: default
  labels:
    tag: "exceptionless-job"
spec:
  type: NodePort
  ports:
  - nodePort: 30014
    port: 80
    targetPort: 80
    protocol: TCP

---
apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: exceptionless-ui
  namespace: default
  labels:
    tag: exceptionless-ui
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: exceptionless-ui
    spec:
      containers:
      - name: exceptionless-ui
        image: "exceptionless/ui:2.8.1497"
        imagePullPolicy: IfNotPresent
        env:
        - name: AppMode
          value: "Production"
        - name: EX_ApiUrl
          value: http://localhost:30012
        ports:
        - containerPort: 80
        resources:
            limits:
              cpu: 1000m
              memory: 2048Mi
            requests:
              cpu: 100m
              memory: 128Mi
      terminationGracePeriodSeconds: 10
      restartPolicy: Always

---

apiVersion: v1
kind: Service
metadata:
  name: exceptionless-ui
  namespace: default
  labels:
    tag: "exceptionless-ui"
spec:
  type: NodePort
  ports:
  - nodePort: 30013
    port: 80
    targetPort: 80
    protocol: TCP
  selector:
    app: exceptionless-ui

// Program
public static int Main(string[] args)
{
    Log.Logger = new LoggerConfiguration().ReadFrom.Configuration(Configuration)
   .MinimumLevel.Debug()
   .Enrich.FromLogContext()
   .WriteTo.Console(new RenderedCompactJsonFormatter())
   //.WriteTo.Fluentd("localhost", 30011, tag: "geektime-ordering-api", restrictedToMinimumLevel: Serilog.Events.LogEventLevel.Debug)
   .CreateLogger();
   // 将Configuration中的exceptionless读取出来,并绑定到它默认的配置上去
    Configuration.GetSection("exceptionless").Bind(Exceptionless.ExceptionlessClient.Default.Configuration);
    try
    {
        Log.Information("Starting web host");
        CreateHostBuilder(args).Build().Run();
        return 0;
    }
    catch (Exception ex)
    {
        Log.Fatal(ex, "Host terminated unexpectedly");
        return 1;
    }
    finally
    {
        Log.CloseAndFlush();
    }
}


// 配置文件
"exceptionless": {
    "ApiKey": "PSaokqmXC8T4xgWal9I0atA8TlYG7Ytz65JvxAL3",
    "ServerUrl": "http://localhost:30012"
  },

// startup
public void Configure(IApplicationBuilder app, IWebHostEnvironment env)
{
	app.UseExceptionless();// 放在所有中间件最前面.为了能捕获到所有异常
}

2.3 K8s中使用Exceptionless

配置文件

 "exceptionless": {
    "ApiKey": "PSaokqmXC8T4xgWal9I0atA8TlYG7Ytz65JvxAL3" // 需要是当前项目的秘钥
  },

因为serverUrl地址是共享的，地址被配置在env.txt中

要将配置引用进来,需要在deployment.yaml中注册serverUrl为环境变量

通过EFK收集应用的全量日志,Exceptionless收集异常日志

相关推荐
机器之心1 小时前
图学习新突破：一个统一框架连接空域和频域
人工智能·后端
思忖小下1 小时前
梳理你的思路(从OOP到架构设计)_简介设计模式
设计模式·架构·eit
.生产的驴2 小时前
SpringBoot 对接第三方登录 手机号登录 手机号验证 微信小程序登录 结合Redis SaToken
java·spring boot·redis·后端·缓存·微信小程序·maven
顽疲2 小时前
springboot vue 会员收银系统 含源码 开发流程
vue.js·spring boot·后端
机器之心2 小时前
AAAI 2025｜时间序列演进也是种扩散过程？基于移动自回归的时序扩散预测模型
人工智能·后端
Yvemil73 小时前
《开启微服务之旅：Spring Boot Web开发》（二）
前端·spring boot·微服务
hanglove_lucky3 小时前
本地摄像头视频流在html中打开
前端·后端·html
维李设论3 小时前
Node.js的Web服务在Nacos中的实践
前端·spring cloud·微服务·eureka·nacos·node.js·express
皓木.4 小时前
(自用)配置文件优先级、SpringBoot原理、Maven私服
java·spring boot·后端
i7i8i9com5 小时前
java 1.8+springboot文件上传+vue3+ts+antdv
java·spring boot·后端
热门推荐
01〔AI 绘画〕Stable Diffusion 之 解决绘制多人或面部很小的人物时面部崩坏问题 篇02YOLOv8训练好模型后，追加轮数继续训练、或者提前终止训练，缩减训练轮数03玄机平台应急响应—webshell查杀04机器人编程怎么入门？05HCIA-datacom数通题库和录播视频资料06Docker 夺命连环 15 问07SQL 主从数据库实时备份08校验 GPT-4 真实性的三个经典问题：快速区分 GPT-3.5 与 GPT-4，并提供免费测试网站09(欧拉)openEuler系统添加网卡文件配置流程、(欧拉)openEuler系统手动配置ipv6地址流程、(欧拉)openEuler系统网络管理说明10VSCode插件 —— Cody AI （免费AI助手！）