Kafka批量消费部分处理成功时的手动提交方案

Kafka批量消费部分处理成功时的手动提交方案

当使用Kafka批量消费时，如果500条消息中只有部分处理成功，需要谨慎处理偏移量提交以避免消息丢失或重复消费。以下是几种处理方案示例：

方案1：记录成功消息并提交最后成功偏移量

ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(100));
Map<TopicPartition, OffsetAndMetadata> offsetsToCommit = new HashMap<>();

for (ConsumerRecord<String, String> record : records) {
    try {
        // 处理消息
        processMessage(record);
        
        // 记录成功处理的偏移量
        offsetsToCommit.put(
            new TopicPartition(record.topic(), record.partition()),
            new OffsetAndMetadata(record.offset() + 1) // 提交下一条要消费的偏移量
        );
    } catch (Exception e) {
        log.error("处理消息失败: {}", record, e);
        // 可以选择继续处理下一条或中断批量处理
    }
}

// 手动提交成功处理的偏移量
if (!offsetsToCommit.isEmpty()) {
    consumer.commitSync(offsetsToCommit);
}

方案2：按分区处理并提交

ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(100));

records.partitions().forEach(partition -> {
    List<ConsumerRecord<String, String>> partitionRecords = records.records(partition);
    long lastSuccessOffset = -1;
    
    for (ConsumerRecord<String, String> record : partitionRecords) {
        try {
            processMessage(record);
            lastSuccessOffset = record.offset();
        } catch (Exception e) {
            log.error("处理消息失败: {}", record, e);
            break; // 分区内遇到错误则停止处理该分区剩余消息
        }
    }
    
    if (lastSuccessOffset >= 0) {
        consumer.commitSync(Collections.singletonMap(
            partition,
            new OffsetAndMetadata(lastSuccessOffset + 1)
        ));
    }
});

方案3：使用事务处理

// 需要配置生产者 transactional.id 和 enable.idempotence=true
KafkaProducer<String, String> producer = new KafkaProducer<>(props);
producer.initTransactions();

ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(100));

try {
    producer.beginTransaction();
    
    Map<TopicPartition, OffsetAndMetadata> offsetsToCommit = new HashMap<>();
    
    for (ConsumerRecord<String, String> record : records) {
        try {
            // 处理消息并可能产生新的消息
            ProcessingResult result = processMessage(record);
            
            // 发送处理结果到下游主题
            producer.send(new ProducerRecord<>("output-topic", result.getKey(), result.getValue()));
            
            // 记录偏移量
            offsetsToCommit.put(
                new TopicPartition(record.topic(), record.partition()),
                new OffsetAndMetadata(record.offset() + 1)
            );
        } catch (Exception e) {
            log.error("处理消息失败: {}", record, e);
            // 可以选择继续或中断
        }
    }
    
    // 提交偏移量到事务
    producer.sendOffsetsToTransaction(offsetsToCommit, consumer.groupMetadata());
    producer.commitTransaction();
} catch (Exception e) {
    producer.abortTransaction();
    throw e;
}

方案4：使用死信队列(DLQ)处理失败消息

ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(100));
Map<TopicPartition, OffsetAndMetadata> offsetsToCommit = new HashMap<>();
KafkaProducer<String, String> dlqProducer = new KafkaProducer<>(dlqProps);

for (ConsumerRecord<String, String> record : records) {
    try {
        processMessage(record);
        offsetsToCommit.put(
            new TopicPartition(record.topic(), record.partition()),
            new OffsetAndMetadata(record.offset() + 1)
        );
    } catch (Exception e) {
        log.error("处理消息失败，发送到DLQ: {}", record, e);
        // 发送失败消息到死信队列
        dlqProducer.send(new ProducerRecord<>("dlq-topic", record.key(), record.value()));
        // 仍然提交偏移量，因为失败消息已转移到DLQ
        offsetsToCommit.put(
            new TopicPartition(record.topic(), record.partition()),
            new OffsetAndMetadata(record.offset() + 1)
        );
    }
}

if (!offsetsToCommit.isEmpty()) {
    consumer.commitSync(offsetsToCommit);
}
dlqProducer.close();

注意事项

1. 幂等性：确保消息处理是幂等的，以防需要重新处理
2. 性能考虑：频繁的小批量提交会影响吞吐量
3. 错误处理策略：根据业务需求决定是跳过失败消息、重试还是停止处理
4. 监控：记录失败消息和提交的偏移量以便排查问题
5. 事务边界：使用事务时注意事务大小和超时问题

选择哪种方案取决于您的具体业务需求、消息重要性以及对一致性的要求。