kafka入门(三):kafka多线程消费

kafka消费积压

如果生产者发送消息的速度过快,或者是消费者处理消息的速度太慢,那么就会有越来越多的消息无法及时消费,也就是消费积压。

消费积压时,可以使用多线程消费,提高消费速度。

kafka多线程消费的代码:

复制代码
public class ThirdMultiConsumerThreadDemo {
    public static final String BROKER_LIST = "localhost:9092";
    public static final String TOPIC = "myTopic1";
    public static final String GROUP_ID = "group.demo";


    public static void main(String[] args) {
        Properties props = initConfig();
        KafkaConsumerThread consumerThread = new KafkaConsumerThread(props, TOPIC,
                Runtime.getRuntime().availableProcessors());
        consumerThread.start();
    }


    /***
     * kafka配置
     * @return
     */
    public static Properties initConfig() {
        Properties props = new Properties();
        props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG,
                StringDeserializer.class.getName());
        props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG,
                StringDeserializer.class.getName());
        props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, BROKER_LIST);
        props.put(ConsumerConfig.GROUP_ID_CONFIG, GROUP_ID);
        props.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, true);
        return props;
    }


    /**
     * kafka消费者线程
     */
    public static class KafkaConsumerThread extends Thread {
        private KafkaConsumer<String, String> kafkaConsumer;
        private ExecutorService executorService;
        private int threadNumber;

        public KafkaConsumerThread(Properties props, String topic, int threadNumber) {
            kafkaConsumer = new KafkaConsumer<>(props);
            kafkaConsumer.subscribe(Collections.singletonList(topic));
            this.threadNumber = threadNumber;
            executorService = new ThreadPoolExecutor(threadNumber, threadNumber,
                    0L, TimeUnit.MILLISECONDS, new ArrayBlockingQueue<>(1000),
                    new ThreadPoolExecutor.CallerRunsPolicy());
        }

        @Override
        public void run() {
            try {
                while (true) {
                    ConsumerRecords<String, String> records =
                            kafkaConsumer.poll(Duration.ofMillis(100));
                    if (!records.isEmpty()) {
                        executorService.submit(new RecordsHandler(records));
                    }
                }
            } catch (Exception e) {
                log.error("run error", e);
            } finally {
                kafkaConsumer.close();
            }
        }

    }

    /**
     * 处理消息
     */
    public static class RecordsHandler extends Thread {
        public final ConsumerRecords<String, String> records;

        public RecordsHandler(ConsumerRecords<String, String> records) {
            this.records = records;
        }

        @Override
        public void run() {
            //处理records.
            for (ConsumerRecord<String, String> record : records) {
                System.out.println("==========>record:"+record.value() + ",thread:" + Thread.currentThread().getName());
            }
        }
    }

}

发送消息后,使用多线程消息,运行结果如下:

复制代码
==========>record:{"id":"1234","name":"lin"},thread:pool-1-thread-1
==========>record:{"id":"5678","name":"chen"},thread:pool-1-thread-2
==========>record:{"id":"91011","name":"wu"},thread:pool-1-thread-3

参考资料:

《深入理解Kafka:核心设计与实践原理》

相关推荐
用户3610588626122 天前
Kafka源码深度剖析:KafkaProducer最终如何处理响应
kafka
goehou3 天前
Kafka 的默认配置适合演示,不适合生产
kafka·消息队列·幂等·ai agent
ClouGence3 天前
开源数据库管理工具 CloudDM 发布新版本,支持 GreptimeDB 数据源
数据库·kafka
用户3610588626124 天前
KafkaProducer 源码深度剖析:筛选可以发送消息的 Broker
kafka
西木Qi4 天前
Kafka消息中间件单机版安装配置
分布式·kafka
飞奔的瓜牛5 天前
Kafka 生产实践指南:使用要点、性能优化与运维维护
中间件·性能优化·kafka
qq_4523962310 天前
第二篇:《CommitLog + ConsumeQueue:一条日志解决 Kafka 多分区随机写的难题》
分布式·kafka
我不会起名字32212 天前
消费回调固定 return nil:错误传播与 Kafka 位点提交的区别
后端·mongodb·kafka·go·im·故障排查
qq_4523962312 天前
第十篇:《Kafka 安全加固:SASL、SSL、ACL 与多租户隔离》
安全·kafka·ssl
用户36105886261213 天前
KafkaProducer 元数据管理:send() 卡住时,它在等什么
kafka