We all know that only those who do nothing make no mistakes. Tell us about a memorable fail in production: what went wrong, how you fixed it, and what lessons you learned?

One of the memorable failures in production occurred due to incorrect error handling in a Go service. As a result, under certain conditions, the application crashed, leading to part of the functionality being unavailable.

What went wrong:

• There was a mistake in the code: the function returned an error, but it was not properly checked.
• Because of this, the error was not handled, and the application continued to run in an incorrect state.

How it was fixed:

• Quickly identified the problem through logs and monitoring.
• Made corrections by adding error checks and proper handling with response return to the client.
• Deployed the patch via CI/CD pipeline with minimal downtime.

Lessons learned:

• Always check and handle errors, especially in critical places.
• Implement automated tests covering error scenarios.
• Use monitoring and alerts for quick problem detection.

Example of error checking in Go:

result, err := someFunction()
if err != nil {
    log.Errorf("Error executing someFunction: %v", err)
    return err
}
// continue working with result

This experience emphasized the importance of discipline in error handling and quality testing.