Kafka broker 写消息的过程

Producer → Kafka Broker → Replication → Consumer
               |
          Partition chosen (by key or round-robin)
               |
          Message appended to end of log (commit log)

上面的流程是kafka 写操作的大体流程。

kafka 不会特意保留message 在内存中，而是直接写入了disk。

那么消费的时候，如果是最近 produced 的message，即使写入了disk，还是会保留在OS的page cache中。 OS page cache 即使是flush 到了disk，也不会被立刻清除，这个是OS 自己的机制。 所以consumer 读最近生产的消息的时候，还是可以从page cache中读取出来的，而不用从disk 中读取。

1. Kafka Broker Receives the Message

   决定partition

2. Message is Appended to a Partition Log

   The selected broker appends the message to the end of the partition log (a sequential write).

   Kafka writes messages to disk efficiently using write-ahead logs (WAL).

3. Replication (for Fault Tolerance)

   Kafka waits for acknowledgments based on the producer's acks setting:

   acks=0: Producer doesn't wait.

   acks=1: Wait for leader only.

   acks=all: Wait for all replicas to acknowledge.

4. Message is Made Available to Consumers

   Once the acks is fulfilled, it becomes available for consumers.

   Consumers read sequentially using offsets.

   Kafka keeps messages for a retention period (e.g., 7 days), regardless of whether they've been consumed.