Sobes.tech
Middle+

How will Kubernetes update secrets if I have six pod replicas with the same secret?

sobes.tech AI

Answer from AI

Kubernetes does not update secrets directly while the pod is running. Pods use a snapshot of the secret that existed at the time of their launch.

To have pods "see" the updated secret, they need to be restarted or recreated. Common deployment strategies for this include:

  • Rolling Update: Old pods are gradually replaced with new ones with the updated secret. This is the standard strategy for Deployments.

  • Recreate: All old pods are deleted simultaneously, and new ones are created. This results in a short service downtime.

  • Blue/Green Deployment: A completely new version of the application (with the updated secret) is deployed in parallel with the old one. After verification, all traffic is switched to the new version, and the old one is removed.

  • Canary Release: A new version of the application (with the updated secret) is deployed to a small subset of users, gradually increasing traffic to it.

When using ConfigMaps and Secrets mounted as files, changes in the secret can be automatically updated inside an already running pod, but this occurs with some delay (kubernetes.io/docs/concepts/configuration/secret/#using-secrets-as-files-from-a-pod). However, to use a secret, for example, in environment variables, the pod needs to be restarted.

Example of updating a Deployment using Rolling Update:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
spec:
  replicas: 6
  selector:
    matchLabels:
      app: my-app
  strategy:
    type: RollingUpdate # Standard strategy
    rollingUpdate:
      maxUnavailable: 1 # Maximum number of unavailable pods during update
      maxSurge: 1       # Maximum number of new pods created above the desired replicas
  template:
    metadata:
      labels:
        app: my-app
    spec:
      containers:
      - name: my-container
        image: my-image:latest
        env:
        - name: MY_SECRET_VALUE
          valueFrom:
            secretKeyRef:
              name: my-secret # Secret name
              key: my-key     # Secret key

After changing the my-secret secret, you need to update the Deployment (for example, by changing the image version or adding an annotation) to trigger a Rolling Update:

kubectl apply -f deployment.yaml

Kubernetes will detect changes in the Deployment spec and start creating new pods that will access the updated secret. Old pods will be terminated.