springboot 实现kafka多源配置

文章目录

背景

实际开发中,不同的topic可能来自不同的集群,所以就需要配置不同的kafka数据源,基于springboot自动配置的思想,最终通过配置文件的配置,自动生成生产者及消费者的配置。

核心配置

自动化配置类

java 复制代码
import com.example.kafka.autoconfig.CustomKafkaDataSourceRegister;
import com.example.kafka.autoconfig.kafkaConsumerConfig;
import org.springframework.beans.BeansException;
import org.springframework.beans.factory.BeanFactory;
import org.springframework.beans.factory.BeanFactoryAware;
import org.springframework.beans.factory.config.SmartInstantiationAwareBeanPostProcessor;
import org.springframework.boot.autoconfigure.condition.ConditionalOnWebApplication;
import org.springframework.boot.context.properties.EnableConfigurationProperties;
import org.springframework.context.annotation.Configuration;
import org.springframework.context.annotation.Import;
import org.springframework.kafka.annotation.EnableKafka;

@EnableKafka
@Configuration(
        proxyBeanMethods = false
)
@ConditionalOnWebApplication
@EnableConfigurationProperties({kafkaConsumerConfig.class})
@Import({CustomKafkaDataSourceRegister.class})
public class MyKafkaAutoConfiguration implements BeanFactoryAware, SmartInstantiationAwareBeanPostProcessor {
    public MyKafkaAutoConfiguration() {
    }

    public void setBeanFactory(BeanFactory beanFactory) throws BeansException {
        beanFactory.getBean(CustomKafkaDataSourceRegister.class);
    }
}

注册生产者、消费者核心bean到spring

复制代码
public void afterPropertiesSet() {
        Map<String, ConsumerConfigWrapper> factories = kafkaConsumerConfig.getFactories();
        if (factories != null && !factories.isEmpty()) {
            factories.forEach((factoryName, consumerConfig) -> {
                KafkaProperties.Listener listener = consumerConfig.getListener();
                Integer concurrency = consumerConfig.getConcurrency();
                // 创建监听容器工厂
                ConcurrentKafkaListenerContainerFactory<String, String> containerFactory = createKafkaListenerContainerFactory(consumerConfig.buildProperties(), listener, concurrency);
                // 注册到容器
                if (!beanFactory.containsBean(factoryName)) {
                    beanFactory.registerSingleton(factoryName, containerFactory);
                }
            });
        }
        Map<String, KafkaProperties.Producer> templates = kafkaProducerConfig.getTemplates();
        if (!ObjectUtils.isEmpty(templates)) {
            templates.forEach((templateName, producerConfig) -> {
                //registerBean(beanFactory, templateName, KafkaTemplate.class, propertyValues);
                //注册spring bean的两种方式
                registerBeanWithConstructor(beanFactory, templateName, KafkaTemplate.class, producerFactoryValues(producerConfig.buildProperties()));
            });
        }
    }

配置spring.factories

yml 复制代码
org.springframework.boot.autoconfigure.EnableAutoConfiguration=\
com.example.kafka.MyKafkaAutoConfiguration

yml配置

复制代码
spring:
  kafka:
    multiple:
      consumer:
        factories:
          test-factory:
            key-deserializer: org.apache.kafka.common.serialization.StringDeserializer
            value-deserializer: org.apache.kafka.common.serialization.StringDeserializer
            bootstrap-servers: 192.168.56.112:9092
            group-id: group_a
            concurrency: 25
            fetch-min-size: 1048576
            fetch-max-wait: 3000
            listener:
              type: batch
            properties:
              spring-json-trusted-packages: '*'
        key-deserializer: org.apache.kafka.common.serialization.StringDeserializer
        value-deserializer: org.apache.kafka.common.serialization.StringDeserializer
        auto-offset-reset: latest
      producer:
        templates:
          test-template:
            bootstrap-servers: 192.168.56.112:9092
            key-serializer: org.apache.kafka.common.serialization.StringSerializer
            value-serializer: org.apache.kafka.common.serialization.StringSerializer
        key-serializer: org.apache.kafka.common.serialization.StringSerializer
        value-serializer: org.apache.kafka.common.serialization.StringSerializer

使用

源码仓库

https://github.com/fafeidou/shield

相关推荐
allenXer4 小时前
Spring Boot测试全景指南:JUnit 5 + Testcontainers实现单元与集成测试
spring boot·微服务·log4j
熟悉的新风景6 小时前
springboot项目或其他项目使用@Test测试项目接口配置-spring-boot-starter-test
java·spring boot·后端
亿牛云爬虫专家7 小时前
Kafka与Flink打造流式数据采集方案:以二手房信息为例
flink·kafka·数据采集·爬虫代理·数据处理·二手房·定时抓取
军军君0112 小时前
基于Springboot+UniApp+Ai实现模拟面试小工具三:后端项目基础框架搭建上
前端·vue.js·spring boot·面试·elementui·微信小程序·uni-app
简放13 小时前
Spring-AI系列-AI模型API
spring boot·aigc·openai
泉城老铁13 小时前
Gitee上开源主流的springboot框架一探究竟
spring boot·后端·架构
Yang's14 小时前
四种高效搭建SpringBoot项目的方式详解
java·spring boot·后端
bing_15814 小时前
在Spring Boot 开发中 Bean 的声明和依赖注入最佳的组合方式是什么?
java·spring boot·后端·bean
别来无恙14915 小时前
Spring Boot + MyBatis 实现用户登录功能详解(基础)
spring boot·后端·mybatis
TinpeaV15 小时前
Elasticsearch8 Windows安装教程
windows·spring boot·后端·elasticsearch