How to avoid resending the same message in Kafka if the source event's status has not changed?

To avoid resending the same message in Kafka if the status of the original event hasn't changed, you can implement idempotency at the message sending level. Here are some approaches:

1. Store the state of sent messages — before sending, check if a message with the same status has already been sent. For example, use a database or cache to store the last status of the event.

2. Use unique message keys — Kafka supports idempotent sending when using keys and enabling producer idempotency. If the status hasn't changed, do not send a new message with the same key.

3. Check the status before sending — in the application logic, compare the current status with the previous one and send a message only if it has changed.

Example in Python using a dictionary to store statuses:

sent_status = {}

def send_event(event_id, status, producer):
    if sent_status.get(event_id) == status:
        print("Status hasn't changed, message not sent")
        return
    # Send message
    producer.send('topic', key=event_id.encode(), value=status.encode())
    sent_status[event_id] = status

This way, you control that messages are sent only when the status changes, avoiding duplication.