What message format should be used in Kafka for high-load services: JSON or binary?

For high-load services in Kafka, it is preferable to use a binary message format, such as Protocol Buffers, Avro, or Thrift. Binary formats provide smaller message sizes and faster serialization/deserialization compared to JSON, which is critical under high load and large data volumes.

JSON is convenient for debugging and development, but due to its text format, it is bulkier and slower to process. In Go, you can use, for example, Avro with the linkedin/goavro library or Protocol Buffers with the official package google.golang.org/protobuf.

Example of using Protocol Buffers:

// Generated code from a .proto file
import (
    "google.golang.org/protobuf/proto"
)

msg := &MyMessage{Field1: "value", Field2: 123}
data, err := proto.Marshal(msg)
if err != nil {
    // handle error
}
// data can be sent to Kafka

Thus, binary format enhances performance and reduces network and disk space load.