Have you ever experienced secret rotation, External Secrets update, and the service crashing?

Yes, such cases do happen. For example, when secrets are rotated and External Secrets updates them in the cluster, the service may crash if:

• The new secret contains an error (e.g., incorrect format or missing fields).
• The service does not handle secret updates gracefully and requires a restart.
• During the secret update, the service loses access to necessary data, leading to errors.

To avoid crashes, it is important to:

• Validate new secrets before applying.
• Use smooth configuration update mechanisms (e.g., reload without restart).
• Provide fallback or cache previous secrets in case of errors.

Example: if External Secrets updates the database password and the service cannot reconnect with the new password, it may crash. Error handling for connection issues and rollback options should be considered.