Sobes.tech
Senior

If Kafka crashes, what happens to event delivery?

sobes.tech AI

Answer from AI

If Kafka goes down (becomes unavailable), event sending depends on the producer settings and system architecture:

  • Buffering on the producer side: Many Kafka clients can buffer messages locally and retry sending until the broker becomes available.
  • Synchronous or asynchronous sending mode: In synchronous mode, the send call blocks until acknowledgment; in asynchronous mode, messages can accumulate in a queue.
  • Retry and timeout settings: The producer can be configured for a certain number of retry attempts.
  • Message loss: If the buffer overflows or the application does not handle errors, messages may be lost.

Example in Go using sarama (Kafka client):

producer, err := sarama.NewSyncProducer(brokers, config)
if err != nil {
    // handle producer creation error
}

msg := &sarama.ProducerMessage{Topic: "topic", Value: sarama.StringEncoder("event data")}
partition, offset, err := producer.SendMessage(msg)
if err != nil {
    // handle send error (e.g., Kafka is unavailable)
    // retry or log as needed
}

Thus, when Kafka crashes, events will not be sent immediately, and without additional logic, they may be lost. To ensure reliability, buffering, queues, or message backup systems are often used.