How to organize access for external systems to the Kafka cluster?

External systems' access to a Kafka cluster is usually organized through Kafka broker network interfaces. The main methods are:

1. Configuring listeners — Kafka configuration specifies addresses and ports where brokers accept connections. A separate listener is allocated for external systems, for example PLAINTEXT://0.0.0.0:9092 or SSL://0.0.0.0:9093.

2. Authentication and authorization — for security, SASL (e.g., SCRAM, GSSAPI) and/or SSL with certificates are used, along with ACLs (Access Control Lists) to restrict permissions.

3. Network settings — external systems must have network access to brokers (VPN, firewall, NAT, routing).

4. Using proxy or REST API — if direct access to Kafka is undesirable, Kafka REST Proxy or specialized services can be used.

Example of listener configuration in server.properties:

listeners=PLAINTEXT://0.0.0.0:9092,SSL://0.0.0.0:9093
advertised.listeners=PLAINTEXT://your.external.ip:9092,SSL://your.external.ip:9093
listener.security.protocol.map=PLAINTEXT:PLAINTEXT,SSL:SSL

Thus, external systems connect to Kafka at specified addresses and ports, considering security settings.