Senior
If Kafka crashes, what happens to event delivery?
sobes.tech AI
Answer from AI
If Kafka goes down (becomes unavailable), event sending depends on the producer settings and system architecture:
- Buffering on the producer side: Many Kafka clients can buffer messages locally and retry sending until the broker becomes available.
- Synchronous or asynchronous sending mode: In synchronous mode, the send call blocks until acknowledgment; in asynchronous mode, messages can accumulate in a queue.
- Retry and timeout settings: The producer can be configured for a certain number of retry attempts.
- Message loss: If the buffer overflows or the application does not handle errors, messages may be lost.
Example in Go using sarama (Kafka client):
producer, err := sarama.NewSyncProducer(brokers, config)
if err != nil {
// handle producer creation error
}
msg := &sarama.ProducerMessage{Topic: "topic", Value: sarama.StringEncoder("event data")}
partition, offset, err := producer.SendMessage(msg)
if err != nil {
// handle send error (e.g., Kafka is unavailable)
// retry or log as needed
}
Thus, when Kafka crashes, events will not be sent immediately, and without additional logic, they may be lost. To ensure reliability, buffering, queues, or message backup systems are often used.