What is your experience with secret storage sidecars…

Worked with the sidecar pattern for injecting secrets into containerized applications. Main solutions encountered:

Vault Agent Injector (HashiCorp Vault):
- Deployed as a controller in Kubernetes, intercepts pod creation, looks for Vault annotations.
- Automatically adds a Volume and Vault Agent sidecar container to the pod.
- Vault Agent handles authentication to Vault, retrieves secrets, and writes them to the pod's filesystem (via Volume).
- The application inside the pod reads secrets from the mounted Volume.
- Advantages: centralized secret management, dynamic retrieval, automatic rotation, integration with various Vault authentication backends.
Kubernetes Secrets Store CSI Driver:
- Allows applications in Kubernetes to access secrets stored in external secret stores (Azure Key Vault, GCP Secret Manager, AWS Secrets Manager, HashiCorp Vault, etc.) as volumes.
- A sidecar container (Specific Provider) runs as part of the same pod or as a separate agent and interacts with the external store via gRPC.
- Retrieved secrets are mounted into the pod as a temporary file storage (Secrets Store CSI Driver Volume).
- The application reads secrets from the filesystem.
- Advantages: unified interface for accessing different secret stores in Kubernetes, ability to mount not only Secrets/Configs but also certificates, keys.

The general approach minimizes storing secrets directly in the application image or Kubernetes manifests, delegating their retrieval and update to the sidecar. This enhances security by separating secret handling logic from application logic and simplifies secret management in dynamic environments like Kubernetes.

# Example of using Vault Agent Injector annotations in a Pod manifest
apiVersion: v1
kind: Pod
metadata:
  name: my-app
  annotations:
    vault.hashicorp.com/agent-inject: "true"
    vault.hashicorp.com/role: "internal-app"
    vault.hashicorp.com/agent-inject-template-db-config: |
      {{- with secret "database/creds/my-app" -}}
      DB_USERNAME="{{ .Data.username }}"
      DB_PASSWORD="{{ .Data.password }}"
      {{- end -}}
spec:
  containers:
    - name: app
      image: my-app-image
      # The application will read secrets from /vault/secrets/db-config
      volumeMounts:
        - name: injected-secrets
          mountPath: "/vault/secrets"
  volumes:
    - name: injected-secrets
      # Volume to be created and filled by Vault Agent
      emptyDir:
        # Using tmpfs for in-memory secret storage
        medium: Memory

# Example of using Secrets Store CSI Driver
apiVersion: v1
kind: Pod
metadata:
  name: my-app-csi
spec:
  serviceAccountName: workload-identity-sa # Example for cloud providers
  containers:
    - name: app
      image: my-app-image
      volumeMounts:
      - name: secret-volume
        mountPath: "/mnt/secrets-store"
        readOnly: true
  volumes:
    - name: secret-volume
      csi:
        driver: secrets-store.csi.k8s.io
        readOnly: true
        volumeAttributes:
          secretProviderClass: "azure-keyvault-csi" # Reference to SecretProviderClass
---
apiVersion: secrets-store.csi.k8s.io/v1
kind: SecretProviderClass
metadata:
  name: azure-keyvault-csi
spec:
  provider: azure # Specifying the provider
  parameters:
    usePodIdentity: "false" # Using managed Azure AD identities
    useVMManagedIdentity: "true"
    userAssignedIdentityID: "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx" # Managed identity ID
    keyvaultName: "my-keyvault" # Azure Key Vault name
    tenantId: "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx" # Azure AD tenant ID
    objects: |
      array:
        - |
          objectName: my-database-password # Secret name in Key Vault
          objectType: secret
          objectVersion: "" # Optional secret version
          fieldName: DB_PASSWORD # File name where the secret will be written in the Volume

Using sidecars for secrets ensures separation of concerns and simplifies integration with secret management systems.

What is your experience with secret storage sidecars?