How was Kafka configured to handle a large stream of events? One topic with partitions or many topics?

For processing a large stream of events in Kafka, it is usually preferable to use a single topic with a large number of partitions rather than multiple topics. This is because partitions allow for scalable processing and provide parallelism, while maintaining message order within each partition.

Advantages of a single topic with partitions:

• Easier to manage configuration and monitoring.
• Allows for efficient load balancing among consumers.
• Simplifies data processing and aggregation.

Multiple topics can complicate architecture and lead to unnecessary overhead in management and monitoring.

Therefore, for a large event stream, it is recommended to use one topic with a sufficient number of partitions to ensure the necessary level of parallelism and performance.