Sobes.tech
Senior

Describe the scheme of parallel subscription extension: topic, partition key, number of partitions, and consumer instances.

sobes.tech AI

Answer from AI

Parallel subscription extension in messaging systems (e.g., Kafka) is built on the following components:

  • Topic — a logical category of messages. All messages with the same topic are published to one topic.
  • Partitioning key — a value that determines which partition a message will go to. Messages with the same key will always go to the same partition, ensuring order processing for that key.
  • Number of partitions — the topic is divided into several partitions for parallelism. The more partitions, the higher the potential degree of parallelism.
  • Consumer instances — the number of consumers reading messages from the topic. Usually, the number of active consumers in a group does not exceed the number of partitions, as one partition can only be processed by one consumer at a time.

Thus, parallel subscription extension is achieved by distributing messages across partitions using the partitioning key and processing these partitions with different consumer instances. This allows scaling processing and maintaining message order for each key.

Example: a topic with 4 partitions, partitioning key — userId, 3 consumer instances. Messages with the same userId will go to one partition and be processed by one consumer, while different userIds will be distributed across different partitions and consumers.

Describe the scheme of parallel subscription… - sobes.tech