Secrets Management

Kubernetes Secret Fundamentals

A Kubernetes Secret is an API object that stores small pieces of sensitive data such as passwords, tokens, and TLS certificates. Secrets are base64-encoded in etcd by default — base64 is encoding, not encryption. Anyone with read access to the Secret object or direct etcd access can trivially decode the values.

Creating Secrets

Copy# Imperative: from literal values (base64-encoded automatically)
kubectl create secret generic db-credentials \
  --from-literal=username=myapp \
  --from-literal=password='S!B\*d$zDsb' \
  -n production

# Imperative: from files (file content becomes the value)
kubectl create secret generic tls-secret \
  --from-file=tls.crt=./server.crt \
  --from-file=tls.key=./server.key \
  -n production

# Declarative YAML — values must be base64-encoded manually
# echo -n "mysecret" | base64  →  bXlzZWNyZXQ=

CopyapiVersion: v1
kind: Secret
metadata:
  name: db-credentials
  namespace: production
  annotations:
    reloader.stakater.com/match: "true"   # trigger pod restart on change (Reloader)
type: Opaque                              # most common type — arbitrary key/value pairs
immutable: true                           # GA 1.21: prevents accidental modification; requires delete+recreate to update
data:
  username: bXlhcHA=                      # base64("myapp")
  password: U0FCXCpkJHpEc2I=             # base64("S!B\*d$zDsb")
stringData:                               # plaintext convenience field — K8s encodes it
  jdbc-url: "jdbc:postgresql://db:5432/mydb?user=myapp&password=S!B\\*d$zDsb"
  # stringData is write-only; reading the secret returns data (base64) not stringData

Secret Types Reference

imagePullSecret for Private Registries

Copy# Create imagePullSecret from docker login credentials
kubectl create secret docker-registry my-registry-creds \
  --docker-server=registry.example.com \
  --docker-username=robot-account \
  --docker-password="$REGISTRY_TOKEN" \
  --docker-email=noreply@example.com \
  -n production

Copy# Reference in pod spec
spec:
  imagePullSecrets:
  - name: my-registry-creds
  containers:
  - name: app
    image: registry.example.com/myapp:latest

# Or attach to ServiceAccount so all pods using it inherit the secret
apiVersion: v1
kind: ServiceAccount
metadata:
  name: app-sa
  namespace: production
imagePullSecrets:
- name: my-registry-creds

Consuming Secrets: Env Vars vs Volume Mounts

Environment Variables

Copyenv:
# Method 1: individual key reference
- name: DB_PASSWORD
  valueFrom:
    secretKeyRef:
      name: db-credentials
      key: password
      optional: false          # pod fails to start if secret/key missing

# Method 2: envFrom — all keys from a secret become env vars
envFrom:
- secretRef:
    name: db-credentials       # ALL keys in secret become env vars
    optional: false
- configMapRef:
    name: app-config

Volume Mounts

Copyvolumes:
- name: secret-vol
  secret:
    secretName: db-credentials
    defaultMode: 0400            # read-only for owner only
    items:                       # optional: select specific keys and rename paths
    - key: password
      path: db/password          # mounted as /secrets/db/password
      mode: 0400

containers:
- name: app
  volumeMounts:
  - name: secret-vol
    mountPath: /secrets
    readOnly: true

Env Var vs Volume Mount Trade-offs

Projected Volume (Multiple Sources)

Copy# Combine secrets, configmaps, service account token, and downward API in one volume
volumes:
- name: projected
  projected:
    sources:
    - secret:
        name: db-credentials
        items:
        - key: password
          path: db-password
    - configMap:
        name: app-config
    - serviceAccountToken:
        path: token
        expirationSeconds: 3600   # bound token, 1h expiry
        audience: "https://kubernetes.default.svc"
    - downwardAPI:
        items:
        - path: namespace
          fieldRef:
            fieldPath: metadata.namespace

RBAC for Secrets

Secrets require the most restrictive RBAC posture of any Kubernetes object. Key principles:

• Never grant list on secrets cluster-wide. list returns all secret objects including their data field values. A subject with list on secrets in a namespace can read every secret value in that namespace.
• Grant get on specific secrets by name where possible using resourceNames.
• Audit secret access regularly — secret reads should be infrequent and expected.

Copy# Minimal: allow a specific workload to read only its own secrets
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: app-secret-reader
  namespace: production
rules:
- apiGroups: [""]
  resources: ["secrets"]
  resourceNames: ["db-credentials", "api-keys"]  # only these secrets
  verbs: ["get"]              # no list, no watch, no update

---
# What NOT to do:
rules:
- apiGroups: [""]
  resources: ["secrets"]
  verbs: ["get", "list", "watch"]  # list = read ALL secret values in namespace

etcd Encryption at Rest

etcd stores all Kubernetes objects including Secrets. Without encryption at rest, anyone who accesses the etcd data files (disk snapshots, backup tapes, cloud provider storage) can read all secret values. Encryption at rest protects against:

• Stolen etcd backup files
• Cloud provider support access to underlying storage
• Physical disk theft in on-premises deployments

It does not protect against: compromised API server (which decrypts on read), compromised etcd network connection (use TLS for that), or a compromised kube-apiserver process.

Encryption Providers

Envelope Encryption Architecture (KMS)

WRITE path (secret creation):
  kube-apiserver → generate random DEK (data encryption key, 32 bytes)
                 → encrypt secret data with DEK (AES-GCM)
                 → call KMS provider plugin → KMS (AWS KMS / GCP KMS / Vault)
                 → KMS encrypts DEK with KEK (key encryption key) → returns encrypted DEK
                 → store: {encrypted_DEK + encrypted_data} in etcd
                 → DEK is NOT stored anywhere in plaintext

READ path (secret retrieval):
  kube-apiserver → read {encrypted_DEK + encrypted_data} from etcd
                 → call KMS → decrypt DEK with KEK
                 → decrypt data with DEK
                 → return plaintext secret to caller
                 → (KMS v2: DEK cached in memory — avoids KMS call on every read)

KEY ROTATION:
  KMS v1: re-encrypt all secrets (expensive full re-encrypt pass)
  KMS v2: rotate KEK in KMS; old DEKs automatically re-wrapped on next write

EncryptionConfiguration

CopyapiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
- resources:
  - secrets                    # encrypt Secret objects
  - configmaps                 # optionally encrypt ConfigMaps too
  providers:
  - kms:                       # first provider = used for NEW writes
      name: aws-kms
      endpoint: unix:///var/run/kmsplugin/socket.sock  # KMS plugin socket
      cachesize: 1000           # DEK cache size (KMS v2)
      timeout: 3s
  - identity: {}               # fallback for reading old unencrypted data
                               # IMPORTANT: keep identity last to read legacy objects
                               # Remove identity only after all secrets are re-encrypted

Copy# Apply encryption config to kube-apiserver
# Add flag: --encryption-provider-config=/etc/kubernetes/encryption-config.yaml

# After enabling, re-encrypt all existing secrets (they were written unencrypted)
kubectl get secrets --all-namespaces -o json | \
  kubectl replace -f -
# This reads all secrets (decrypts with identity provider) and rewrites them
# (encrypts with the first provider = KMS)

# Verify a secret is encrypted in etcd
ETCDCTL_API=3 etcdctl get \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  /registry/secrets/default/my-secret | xxd | head
# Should show binary/encrypted data, not readable YAML

External Secret Stores

The most secure pattern is to never store secrets in Kubernetes etcd at all. External secret stores provide centralized management, fine-grained access control, rotation automation, and full audit trails.

External Secrets Operator (ESO)

External Secrets Operator (ESO) is a Kubernetes operator that reads from external secret stores and creates Kubernetes Secrets from them. It runs as a controller and syncs secrets on a configurable interval.

External Secret Store          ESO Controller              Kubernetes
(Vault / AWS SM / GCP SM)      (Operator)                 Cluster
─────────────────────          ──────────────             ──────────────────
                                reads SecretStore    →    SecretStore/ClusterSecretStore CRD
                                polls ExternalSecret →    ExternalSecret CRD
reads secret value         ←   calls store API
                                creates/updates K8s  →    Secret (managed by ESO)
                                Secret every
                                refreshInterval

SecretStore — defines the external backend

Copy# Namespace-scoped SecretStore
apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: aws-secrets-manager
  namespace: production
spec:
  provider:
    aws:
      service: SecretsManager
      region: us-east-1
      auth:
        jwt:                          # IRSA authentication via projected SA token
          serviceAccountRef:
            name: eso-sa              # SA annotated with IAM role ARN
---
# Cluster-scoped (accessible from all namespaces)
apiVersion: external-secrets.io/v1beta1
kind: ClusterSecretStore
metadata:
  name: vault-backend
spec:
  provider:
    vault:
      server: "https://vault.example.com"
      path: "secret"
      version: "v2"
      auth:
        kubernetes:
          mountPath: "kubernetes"
          role: "eso-role"
          serviceAccountRef:
            name: eso-sa
            namespace: external-secrets

ExternalSecret — maps store keys to Kubernetes Secret

CopyapiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: db-credentials
  namespace: production
spec:
  refreshInterval: "1h"              # sync every hour; use "0" to disable auto-sync
  secretStoreRef:
    name: aws-secrets-manager        # references the SecretStore above
    kind: SecretStore                # or ClusterSecretStore
  target:
    name: db-credentials             # name of the K8s Secret to create/update
    creationPolicy: Owner            # ESO owns the secret; deleted when ExternalSecret is deleted
    # creationPolicy: Merge          # merge into existing secret (ESO adds keys, doesn't own)
    # creationPolicy: None           # ESO does not create the secret (external process creates it)
    deletionPolicy: Retain           # keep K8s Secret if ExternalSecret is deleted
    # deletionPolicy: Delete         # delete K8s Secret when ExternalSecret is deleted
    template:                        # optional: template the secret format
      type: kubernetes.io/dockerconfigjson
      data:
        .dockerconfigjson: |
          {"auths":{"registry.example.com":{"auth":"{{ .token | b64enc }}"}}}
  data:
  - secretKey: password              # key in the resulting K8s Secret
    remoteRef:
      key: prod/db/credentials       # path in the external store
      property: password             # specific field within the store entry (JSON)
      version: "AWSCURRENT"          # version/stage (AWS Secrets Manager)
  - secretKey: username
    remoteRef:
      key: prod/db/credentials
      property: username
  dataFrom:                          # import all keys from a store path
  - extract:
      key: prod/app/config           # all JSON keys become Secret data keys

ESO Status and Debugging

Copy# Check sync status
kubectl get externalsecret -n production
# NAME             STORE                REFRESH-INTERVAL   STATUS   READY
# db-credentials   aws-secrets-manager  1h                 SecretSynced   True

kubectl describe externalsecret db-credentials -n production
# Shows: last sync time, conditions, errors

# Force immediate refresh
kubectl annotate externalsecret db-credentials \
  force-sync=$(date +%s) -n production

Secrets Store CSI Driver

The Secrets Store CSI Driver (SSCD) mounts secrets from external stores directly into pods as files, bypassing Kubernetes Secrets entirely. Secrets are fetched at pod startup, mounted as tmpfs volumes, and optionally synced to Kubernetes Secrets for env var consumption.

Pod startup sequence:
  1. Scheduler places pod on node
  2. kubelet calls CSI driver for the SecretProviderClass volume
  3. CSI driver authenticates to external store (Vault/AWS SM) using pod's SA token
  4. CSI driver fetches secrets and mounts them as tmpfs files in the pod
  5. Pod starts with secret files available at /mnt/secrets/
  6. (Optional) CSI driver creates K8s Secret for env var consumption

ROTATION:
  CSI driver re-fetches secrets on each pod rotation period (default: 2 minutes)
  File content is updated in place — no pod restart needed for file changes
  K8s Secret sync is also updated — Reloader needed for env var rotation

SecretProviderClass

CopyapiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: aws-secrets
  namespace: production
spec:
  provider: aws
  parameters:
    objects: |
      - objectName: "prod/db/credentials"
        objectType: "secretsmanager"
        jmesPath:
          - path: username
            objectAlias: db-username
          - path: password
            objectAlias: db-password
      - objectName: "/prod/api/key"
        objectType: "ssmparameter"
        objectAlias: api-key
  secretObjects:                       # optionally sync to K8s Secret for env var use
  - data:
    - key: password
      objectName: db-password
    - key: username
      objectName: db-username
    secretName: db-credentials-synced  # K8s Secret created/updated by CSI driver
    type: Opaque

Copy# Reference in pod spec
volumes:
- name: secrets-store
  csi:
    driver: secrets-store.csi.k8s.io
    readOnly: true
    volumeAttributes:
      secretProviderClass: aws-secrets

containers:
- name: app
  volumeMounts:
  - name: secrets-store
    mountPath: /mnt/secrets
    readOnly: true
  env:                                 # use synced K8s Secret for env vars if needed
  - name: DB_PASSWORD
    valueFrom:
      secretKeyRef:
        name: db-credentials-synced
        key: password

Sealed Secrets (GitOps)

Sealed Secrets (Bitnami) allows encrypting Kubernetes Secrets so they can be safely committed to Git repositories. The controller holds the private decryption key; the encrypted SealedSecret resource is safe to store in source control.

Developer workflow:
  1. Developer creates a plain K8s Secret YAML locally
  2. kubeseal CLI fetches controller's public cert from the cluster
  3. kubeseal encrypts the secret: SealedSecret YAML (safe for Git)
  4. Developer commits SealedSecret.yaml to Git repo
  5. GitOps operator (ArgoCD/Flux) applies SealedSecret to cluster
  6. Sealed Secrets controller decrypts → creates K8s Secret

Security model:
  - SealedSecret is encrypted with the controller's RSA public key
  - Only the controller (holding the private key) can decrypt
  - Encryption is namespaced (by default): a SealedSecret for "production"
    cannot be decrypted/used in "staging"
  - Controller private key must be backed up separately

Copy# Install kubeseal CLI
brew install kubeseal

# Fetch the controller's public key (one-time or periodically)
kubeseal --fetch-cert \
  --controller-name=sealed-secrets-controller \
  --controller-namespace=kube-system \
  > pub-sealed-secrets.pem

# Create a plain secret and pipe to kubeseal
kubectl create secret generic db-credentials \
  --from-literal=password=mysecret \
  --dry-run=client -o yaml | \
  kubeseal --cert pub-sealed-secrets.pem \
           --scope namespace-wide \
  > sealed-db-credentials.yaml

# Commit sealed-db-credentials.yaml to Git — safe to store
# DO NOT commit the plain secret YAML

# Apply to cluster (ArgoCD/Flux does this automatically)
kubectl apply -f sealed-db-credentials.yaml

Copy# SealedSecret format (safe to commit to Git)
apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
  name: db-credentials
  namespace: production
spec:
  encryptedData:
    password: AgBy3i4OJSWK+PiTySYZZA9rO43cGDEq...  # encrypted value
  template:
    metadata:
      name: db-credentials
      namespace: production
    type: Opaque

IRSA & Workload Identity

IRSA (IAM Roles for Service Accounts) and Workload Identity eliminate the need for static cloud credentials entirely. Instead of storing AWS access keys or GCP service account keys as Kubernetes Secrets, pods authenticate to cloud services using their Kubernetes ServiceAccount identity, which is federated to a cloud IAM role.

AWS IRSA

Pod (SA: my-app)
  ├── Projected volume: /var/run/secrets/eks.amazonaws.com/serviceaccount/token
  │   (audience-bound token, 1h expiry)
  └── STS call: AssumeRoleWithWebIdentity(token, role_arn)
        ↓
  AWS IAM validates: token signed by cluster OIDC provider?
        ↓
  Returns temporary credentials (AccessKeyId, SecretAccessKey, SessionToken)
        ↓
  AWS SDK uses temp creds automatically — no static keys stored anywhere

Copy# 1. Create IAM role with trust policy allowing the SA (done once by platform team)
# Trust policy: allow sts:AssumeRoleWithWebIdentity from specific SA

# 2. Annotate the ServiceAccount with the IAM role ARN
kubectl annotate serviceaccount my-app \
  eks.amazonaws.com/role-arn=arn:aws:iam::123456789012:role/my-app-role \
  -n production

# 3. No other changes needed — AWS SDK automatically detects IRSA token

CopyapiVersion: v1
kind: ServiceAccount
metadata:
  name: my-app
  namespace: production
  annotations:
    eks.amazonaws.com/role-arn: "arn:aws:iam::123456789012:role/my-app-role"
    eks.amazonaws.com/token-expiration: "3600"   # token expiry seconds (default 86400)

GCP Workload Identity

Copy# Bind K8s SA to GCP Service Account
gcloud iam service-accounts add-iam-policy-binding \
  my-gsa@my-project.iam.gserviceaccount.com \
  --role=roles/iam.workloadIdentityUser \
  --member="serviceAccount:my-project.svc.id.goog[production/my-app]"

# Annotate K8s SA
kubectl annotate serviceaccount my-app \
  iam.gke.io/gcp-service-account=my-gsa@my-project.iam.gserviceaccount.com \
  -n production

Azure Workload Identity

Copy# Azure AD Workload Identity (AZWI)
# Create federated credential between K8s SA and Azure Managed Identity
az identity federated-credential create \
  --name my-app-federated \
  --identity-name my-app-identity \
  --resource-group my-rg \
  --issuer "https://oidc.prod-aks.azure.com/my-cluster-id/" \
  --subject "system:serviceaccount:production:my-app"

# Annotate K8s SA
kubectl annotate serviceaccount my-app \
  azure.workload.identity/client-id="$CLIENT_ID" \
  -n production

Secret Rotation Patterns

Secrets should be rotated regularly: database passwords, API keys, and TLS certificates all have recommended rotation windows. The challenge in Kubernetes is rotating without downtime.

Zero-Downtime Rotation with Rolling Restart

Step 1: Update the external store (or K8s Secret) with the new credential value
        (keep old value accessible during transition — dual credentials)

Step 2: ESO / CSI driver syncs new value to K8s Secret (within refreshInterval)
        OR: update K8s Secret directly

Step 3: Trigger rolling restart of affected Deployments:
        kubectl rollout restart deployment/my-app -n production
        (new pods start with new secret; old pods still run with old)

Step 4: Verify new pods are healthy and processing correctly

Step 5: Revoke old credential in the external store / database

Step 6: Remove dual-credential support from application if applicable

Reloader — Automatic Restart on Secret Change

Copy# Install Stakater Reloader
helm upgrade --install reloader stakater/reloader \
  --namespace reloader \
  --create-namespace

Copy# Annotate Deployment to auto-restart when referenced secret changes
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
  annotations:
    reloader.stakater.com/auto: "true"        # watch all secrets/configmaps in pod spec
    # OR: only watch specific secrets:
    secret.reloader.stakater.com/reload: "db-credentials,api-keys"

cert-manager for TLS Certificate Rotation

Copy# cert-manager automatically rotates TLS certs before expiry
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: my-tls-cert
  namespace: production
spec:
  secretName: my-tls-secret          # cert-manager creates and rotates this Secret
  duration: 2160h                    # 90 days
  renewBefore: 360h                  # renew 15 days before expiry
  issuerRef:
    name: letsencrypt-prod
    kind: ClusterIssuer
  dnsNames:
  - my-app.example.com

Secret Scanning in CI/CD

Preventing secrets from being committed to source control is far easier than responding to a secret leak. Implement scanning at multiple stages of the development lifecycle.

Pre-commit with detect-secrets

Copy# Install detect-secrets
pip install detect-secrets

# Generate baseline (current known false positives)
detect-secrets scan > .secrets.baseline

# In .pre-commit-config.yaml:
# repos:
# - repo: https://github.com/Yelp/detect-secrets
#   hooks:
#   - id: detect-secrets
#     args: ['--baseline', '.secrets.baseline']

# Run manually
detect-secrets scan --baseline .secrets.baseline

git-secrets (AWS Labs)

Copy# Install and configure
git secrets --install    # install hooks in current repo
git secrets --register-aws  # add AWS secret patterns (access keys, etc.)

# Scan entire repo history
git secrets --scan-history

TruffleHog — deep history scan

Copy# Scan entire git history for high-entropy strings and known secret patterns
trufflehog git https://github.com/myorg/myrepo \
  --only-verified \
  --json | jq '.SourceMetadata.Data.Git.commit'

# In CI/CD pipeline (GitHub Actions example):
# - uses: trufflesecurity/trufflehog@main
#   with:
#     path: ./
#     base: ${{ github.event.repository.default_branch }}
#     head: HEAD

Kubernetes Secret scanning

Copy# Find secrets with empty/trivial values
kubectl get secrets --all-namespaces -o json | \
  jq -r '.items[] | select(.data | length == 0) | "\(.metadata.namespace)/\(.metadata.name)"'

# Find secrets not referenced by any pod (orphaned secrets)
# Use kube-score or Trivy Operator for automated checks

# Check for secrets in ConfigMaps (should never happen)
kubectl get configmaps --all-namespaces -o json | \
  jq -r '.items[].data | to_entries[] | select(.value | test("password|secret|token|key"; "i")) | .key'

Anti-Patterns

Metrics & Alerts

Key Metrics

Alerts

Copygroups:
- name: secrets.rules
  rules:

  - alert: ExternalSecretSyncFailing
    expr: |
      rate(externalsecrets_sync_calls_total{status="error"}[5m]) > 0
    for: 5m
    annotations:
      summary: "ExternalSecret {{ $labels.name }}/{{ $labels.namespace }} failing to sync"
      description: "Check ESO logs and external store connectivity. Secret may be stale."
    labels:
      severity: high

  - alert: KMSEncryptionLatencyHigh
    expr: |
      histogram_quantile(0.99,
        rate(apiserver_storage_transformation_duration_seconds_bucket{transformation_type="encrypt"}[5m])
      ) > 0.5
    for: 2m
    annotations:
      summary: "KMS encryption P99 > 500ms — API server latency degraded"
    labels:
      severity: warning

  - alert: SecretReadByUnexpectedServiceAccount
    # Implement via audit log stream:
    # verb=get, resource=secrets, user.username NOT IN allowlist
    annotations:
      summary: "Secret read by unexpected principal: {{ $labels.user }}"
    labels:
      severity: high

  - alert: TLSCertificateExpiringSoon
    expr: |
      (certmanager_certificate_expiration_timestamp_seconds - time()) / 86400 < 14
    annotations:
      summary: "TLS certificate expires in < 14 days: {{ $labels.name }}"
    labels:
      severity: warning

Runbooks

1. ExternalSecret stuck in error state: Check kubectl describe externalsecret <name> -n <ns> for the error condition. Common causes: ESO SA lacks IAM permission to read the secret (add IAM policy), secret path is wrong (typo in key), external store is unavailable (check VPC connectivity, firewall). Force resync: kubectl annotate externalsecret <name> force-sync=$(date +%s) -n <ns>.
2. KMS plugin unavailable — API server degraded: Kubernetes will queue writes and fail reads that require decryption. New secrets cannot be created. Diagnose: check KMS plugin DaemonSet/pod logs (kubectl logs -l app=aws-encryption-provider -n kube-system). Check KMS service availability in the cloud console. If KMS is down and API server is operational, existing readable secrets are cached; new creates fail. Restore KMS service or fail over to a backup KMS key.
3. Secret leaked — immediate response: (1) Identify what was leaked: exact secret name and key, when it was created, who has read access (audit log). (2) Immediately rotate the credential at the source (database, API provider). (3) Update the secret in Kubernetes and restart dependent pods. (4) Revoke the old credential. (5) Investigate how the leak occurred (env var in logs, git commit, overly broad RBAC). (6) File incident report.
4. Pod cannot start due to secret not found: kubectl describe pod <name> will show secret "x" not found. Check if the secret exists: kubectl get secret <name> -n <ns>. If using ESO: check ExternalSecret sync status. If using CSI driver: check SecretProviderClass and CSI driver pod logs. Create the secret manually if needed for emergency recovery, then fix the root cause.
5. Secret rotation causing application errors: Identify which pods are failing (check logs/events). If pods are using env vars, they need a restart to pick up new secret values — trigger kubectl rollout restart deployment/<name>. If Reloader is installed, verify the annotation is set correctly. For CSI-mounted secrets: file content should update automatically; verify the app re-reads the file (not caching at startup).

Best Practices

1. Never store secrets in environment variables for sensitive values — use volume mounts. Env vars are exposed in /proc/self/environ, process listings, core dumps, and accidentally in application logs. Volume-mounted secrets have a smaller exposure surface. For secrets that must be env vars (legacy apps), use secretKeyRef, not envFrom on the whole secret.
2. Enable etcd encryption at rest with KMS envelope encryption as the minimum standard. File-based encryption (secretbox/aesgcm) improves over plain-text but the key is on disk alongside the data. KMS envelope encryption keeps the key in a managed service (AWS KMS, GCP KMS, Vault) with independent audit logs and access controls. This is the correct production standard.
3. Use an external secret store (Vault, AWS Secrets Manager, GCP Secret Manager) for all production secrets. External stores provide: centralized audit logs, fine-grained access policies per secret, automated rotation, version history, and lease-based TTLs. Kubernetes Secrets should be a sync target (via ESO or CSI driver), not the source of truth.
4. Use IRSA/Workload Identity to eliminate static cloud credentials entirely. Any AWS access key, GCP service account key, or Azure client secret stored in Kubernetes is a long-lived credential that can be compromised. With IRSA/Workload Identity, pods authenticate using their Kubernetes ServiceAccount identity federated to cloud IAM — no static credentials stored anywhere.
5. Enforce least-privilege RBAC on secrets: get only, specific names, no list/watch. Grant get on specific secret names via resourceNames. Never grant list or watch on secrets — these return all secret values in the namespace. Audit all RBAC grants on secrets quarterly.
6. Implement pre-commit secret scanning and CI pipeline scanning. Install detect-secrets or git-secrets as a pre-commit hook in every repository. Add TruffleHog to CI pipelines to scan new commits. Rotate any secret that passes through a CI log or appears in a commit. Make secret scanning a blocking CI gate.
7. Automate secret rotation and enforce maximum TTLs. Manual rotation is error-prone and often forgotten. Use cert-manager for TLS, Vault's dynamic secrets for database credentials, and ESO's refreshInterval for external secrets. Enforce maximum TTLs via policy: no API key older than 90 days, no database password older than 30 days. Alert on approaching expiry.
8. For GitOps workflows, use Sealed Secrets and back up the controller private key. Plain Kubernetes Secrets cannot be safely stored in Git. Sealed Secrets enables GitOps without compromising security. Always maintain an offline backup of the Sealed Secrets controller private key in a separate secure vault — without it, all encrypted secrets become permanently inaccessible if the controller is lost.