How to ensure message delivery completeness control between two Kafka clusters via a security gateway (without modifying the gateway)?
sobes.tech AI
Answer from AI
To control the completeness of message delivery between two Kafka clusters through a security gateway that cannot be modified, the following approaches can be used:
-
Message identification and acknowledgments: Include unique message identifiers and use acknowledgment mechanisms at the application level. For example, the sender can store the IDs of sent messages, and the receiver can confirm receipt.
-
Using Kafka Connect with Source and Sink connectors: If the gateway allows traffic, connectors can be configured for data replication with offset checking.
-
Monitoring offsets and metrics: Track offsets in both clusters to ensure messages are not lost.
-
Implementing resend logic at the client level: If delivery confirmation is absent, resend the message.
-
Using external storage to monitor the state: For example, store message delivery status in a database or distributed cache.
Since the gateway cannot be changed, all control mechanisms should be implemented at the application and Kafka client level, ensuring idempotency and the ability to resend messages.