What is the difference between using a Consumer Group and not using one in Kafka?

In Apache Kafka, a Consumer Group is a group of consumers that jointly read data from a topic, distributing partitions among themselves.

Without a Consumer Group:

• Each consumer reads all messages from the topic independently.
• If multiple consumers read from the same topic, each will receive all messages (duplication).

With a Consumer Group:

• Consumers in the group divide the topic's partitions among themselves, ensuring parallel processing without duplication.
• If there are N consumers and M partitions in the group, each partition is assigned to one consumer.
• It allows scaling of processing and provides load balancing.

Example in Go using a Consumer Group:

import (
    "context"
    "github.com/segmentio/kafka-go"
)

func consumeGroup() {
    r := kafka.NewReader(kafka.ReaderConfig{
        Brokers: []string{"localhost:9092"},
        GroupID: "my-group",
        Topic:   "my-topic",
    })

    for {
        m, err := r.ReadMessage(context.Background())
        if err != nil {
            break
        }
        fmt.Printf("message at offset %d: %s = %s\n", m.Offset, string(m.Key), string(m.Value))
    }
    r.Close()
}

Thus, a Consumer Group ensures coordination and scalability of message consumption in Kafka.