If a pod in Kubernetes crashes after successfully booking a hotel but before saving the booking ID, how to restore consistency?

In a situation where a pod in Kubernetes crashes after successfully booking a hotel but before saving the booking ID, there is a data consistency problem — the booking was made, but the information was not saved.

To restore consistency, the following approaches can be applied:

1. Idempotency of operations and retries

   • Make the operation of saving the booking ID idempotent so that repeated executions do not cause duplication.
   • Upon pod restart or service rerun, check if the booking has already been made, and if so — save the ID.

2. Using distributed transactions or sagas

   • Break the process into steps (sagas), where each step can be compensated.
   • If saving the ID fails, perform a compensating operation — cancel the booking.

3. External state storage

   • Save the intermediate booking state in a reliable storage (e.g., database or message queue).
   • Upon pod restart, read the state and continue the process from the last successful step.

Example in C# (simplified):

// Check if the booking ID is already saved
var bookingId = await bookingRepository.GetBookingIdAsync(userId);
if (bookingId == null)
{
    // If not, try to save
    bookingId = await hotelService.BookHotelAsync(details);
    await bookingRepository.SaveBookingIdAsync(userId, bookingId);
}
// Continue working with bookingId

Thus, the key is to make operations resilient to failures and ensure the possibility of re-execution without losing data consistency.