kafka如何获取topic一天的消息量

背景

有时候我们想要统计某个topic一天的消息量大小,在监控不完善的情况下我们可以如何统计呢?

java实现

我们可以基于kafka提供的client自己去实现

首先引入client依赖

xml 复制代码
        <dependency>
            <groupId>org.apache.kafka</groupId>
            <artifactId>kafka-clients</artifactId>
            <version>3.5.0</version>
        </dependency>

具体实现代码

java 复制代码
    public static void main(String[] args) {
    
        String bootstrapServers = "kafka-小奏技术-001.com:9092,kafka-小奏技术-002.com:9092,kafka-小奏技术-003.com:9092";
        String topicName = "小奏技术-topic";

        Properties props = new Properties();
        props.put(AdminClientConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrapServers);

        try (AdminClient adminClient = AdminClient.create(props)) {
            long endTime = System.currentTimeMillis();
            // 24 hours ago
            long startTime = endTime - 24 * 60 * 60 * 1000; 

            // Get topic partitions
            List<TopicPartition> partitions = getTopicPartitions(adminClient, topicName);

            // Get offsets for start time
            Map<TopicPartition, Long> startOffsets = getOffsetsForTime(adminClient, partitions, startTime);

            // Get offsets for end time (current time)
            Map<TopicPartition, Long> endOffsets = getOffsetsForTime(adminClient, partitions, endTime);

            // Calculate total message count
            long totalMessages = calculateMessageCount(startOffsets, endOffsets);

            System.out.println("Total messages in the last 24 hours for topic '" + topicName + "': " + totalMessages);

        } catch (Exception e) {
            e.printStackTrace();
        }
    }

    private static List<TopicPartition> getTopicPartitions(AdminClient adminClient, String topicName) throws ExecutionException, InterruptedException {
        DescribeTopicsResult describeTopicsResult = adminClient.describeTopics(Collections.singletonList(topicName));
        Map<String, TopicDescription> topicDescriptionMap = describeTopicsResult.all().get();
        TopicDescription topicDescription = topicDescriptionMap.get(topicName);

        List<TopicPartition> partitions = new ArrayList<>();
        for (TopicPartitionInfo partitionInfo : topicDescription.partitions()) {
            partitions.add(new TopicPartition(topicName, partitionInfo.partition()));
        }
        return partitions;
    }

    private static Map<TopicPartition, Long> getOffsetsForTime(AdminClient adminClient, List<TopicPartition> partitions, long timestamp) throws ExecutionException, InterruptedException {
        Map<TopicPartition, OffsetSpec> timestampsToSearch = new HashMap<>();
        for (TopicPartition partition : partitions) {
            timestampsToSearch.put(partition, OffsetSpec.forTimestamp(timestamp));
        }

        ListOffsetsResult offsetsForTimes = adminClient.listOffsets(timestampsToSearch);
        Map<TopicPartition, ListOffsetsResult.ListOffsetsResultInfo> offsetsResultMap = offsetsForTimes.all().get();

        Map<TopicPartition, Long> resultOffsets = new HashMap<>();
        for (Map.Entry<TopicPartition, ListOffsetsResult.ListOffsetsResultInfo> entry : offsetsResultMap.entrySet()) {
            resultOffsets.put(entry.getKey(), entry.getValue().offset());
        }

        return resultOffsets;
    }

    private static long calculateMessageCount(Map<TopicPartition, Long> startOffsets, Map<TopicPartition, Long> endOffsets) {
        long totalMessages = 0;
        for (TopicPartition partition : startOffsets.keySet()) {
            Long startOffset = startOffsets.get(partition);
            Long endOffset = endOffsets.get(partition);

            if (startOffset != null && endOffset != null) {
                totalMessages += endOffset - startOffset;
            }
        }
        return totalMessages;
    }

具体的实现逻辑大致如下

  1. 获取topic的所有partition
  2. 获取partition在开始时间点的offset
  3. 获取partition在结束时间点的offset
  4. 计算offset差值即为当前时间段的消息量

总结

代码实现还是比较简单的,就是获取到topic的所有partition的偏移量,然后累加就行

我们也可以基于kafka暴露的JMX指标˙中kafka.server:type=BrokerTopicMetrics,name=MessagesInPerSec,topic=([-.\w]+) 来计算

相关推荐
kangkang-43 分钟前
PC端基于SpringBoot架构控制无人机(二):MavLink协议
java·spring boot·后端·无人机
麦兜*1 小时前
Spring Boot秒级冷启动方案:阿里云FC落地实战(含成本对比)
java·spring boot·后端·spring·spring cloud·系统架构·maven
喷火龙8号2 小时前
MSC中的Model层:数据模型与数据访问层设计
后端·架构
5ycode2 小时前
dify项目结构说明与win11本地部署
后端·开源
LaoZhangAI2 小时前
GPT-image-1 API如何传多图:开发者完全指南
前端·后端
fouryears_234173 小时前
深入拆解Spring核心思想之一:IoC
java·后端·spring
codervibe3 小时前
从 0 到 1,我如何独立开发一套教务系统(EduCore)
后端
error_cn3 小时前
podman安全性设置
后端
一个热爱生活的普通人3 小时前
Go 泛型终极指南:告别 interface{},写出更安全、更强大的代码!
后端·go
求知摆渡3 小时前
从零开始搭建typecho
前端·后端