What to do with MDC context in Kafka?

MDC (Mapped Diagnostic Context) is used for storing logging context, such as request IDs, user information, etc. When working with Kafka, a problem arises: during asynchronous message processing, MDC is not automatically passed between threads.

To preserve the MDC context when working with Kafka, the usual approach is:

1. Before sending a message:

   • Add the necessary MDC values to the Kafka message headers.

2. Upon receiving a message:

   • Extract the values from the headers and restore the MDC context for the current thread.

This allows logs related to processing a specific message to retain their context and be linked.

Example in Java:

// Before sending
ProducerRecord<String, String> record = new ProducerRecord<>(topic, key, value);
MDC.getCopyOfContextMap().forEach((k, v) -> record.headers().add(k, v.getBytes(StandardCharsets.UTF_8)));
producer.send(record);

// Upon receiving
ConsumerRecord<String, String> record = ...;
Map<String, String> contextMap = new HashMap<>();
record.headers().forEach(header -> contextMap.put(header.key(), new String(header.value(), StandardCharsets.UTF_8)));
MDC.setContextMap(contextMap);

// Then log with the restored MDC

There are also libraries and wrappers for Kafka Producer/Consumer that automate this process.

It is important to remember to clear the MDC after processing a message to avoid context leaks between threads.