Encryption at Rest

Why Encrypt at Rest

By default, all Kubernetes objects — including Secrets — are stored in etcd as plaintext (base64-encoded, which is encoding not encryption). Anyone with read access to the etcd data directory or etcd API can read every Secret in the cluster.

Attack Vectors That Encryption at Rest Mitigates

How etcd Encryption Works

Kubernetes API server encrypts resources before writing them to etcd, and decrypts them after reading. etcd itself has no knowledge of the encryption — it stores opaque encrypted bytes.

What Gets Encrypted

The EncryptionConfiguration specifies which resource types to encrypt. You can encrypt any API resource, but these are the most security-relevant:

Encryption Providers

KMS v2 Envelope Encryption

KMS v2 (GA in Kubernetes 1.29) uses envelope encryption — a two-tier key hierarchy that keeps the KMS workload manageable while maintaining strong security properties.

KMS Plugin Architecture

The API server communicates with the KMS via a local gRPC socket. The KMS plugin (a separate process) translates between the Kubernetes KMS gRPC protocol and the cloud KMS API.

Copy# kube-apiserver communicates via Unix socket to KMS plugin
# KMS plugin process runs on the control plane node

──── kube-apiserver ────────────────────────────────────────────────────
#  │
#  │  gRPC over Unix socket: /var/run/kms-plugin/socket.sock
#  │  Protocol: k8s.io/kms/v2beta1
#  ▼
──── KMS plugin process ────────────────────────────────────────────────
#  │
#  │  Cloud SDK / Vault API
#  ▼
──── External KMS ──────────────────────────────────────────────────────
#  AWS KMS / GCP Cloud KMS / Azure Key Vault / HashiCorp Vault

KMS Plugin Examples

EncryptionConfiguration

Full Configuration with All Providers

Copy# /etc/kubernetes/encryption-config.yaml
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration

resources:
# ── Secrets: highest priority, KMS v2 ────────────────────────────────
- resources:
  - secrets
  providers:
  # First provider = used for new writes
  - kms:
      apiVersion: v2            # KMS v2 (GA in 1.29)
      name: aws-kms-key-1
      endpoint: unix:///var/run/kms-plugin/socket.sock
      cachesize: 1000           # Max cached DEKs
      timeout: 3s
  # identity as last fallback allows reading pre-encryption data
  # REMOVE after all secrets re-encrypted
  - identity: {}

# ── ConfigMaps: secretbox (local key) ────────────────────────────────
- resources:
  - configmaps
  providers:
  - secretbox:
      keys:
      - name: key2
        secret: c2VjcmV0Ym94a2V5Zm9yY29uZmlnbWFwc3h5ejEyMzQ1Njc=
        # 32-byte base64-encoded random key
        # Generate: head -c 32 /dev/urandom | base64
  - identity: {}

# ── Everything else: identity (unencrypted) ──────────────────────────
# Add other resource types above as needed

secretbox Provider (Local Key)

Copy# Generate a 32-byte base64-encoded key for secretbox
head -c 32 /dev/urandom | base64

# secretbox provider configuration
resources:
- resources:
  - secrets
  providers:
  - secretbox:
      keys:
      - name: key1          # Arbitrary name; appears in etcd prefix
        secret: <base64-32-bytes>
  - identity: {}           # Read fallback for pre-encryption data

aesgcm Provider

Copy# aesgcm uses 16-byte (AES-128) or 32-byte (AES-256) keys
# Generate AES-256 key:
head -c 32 /dev/urandom | base64

providers:
- aesgcm:
    keys:
    - name: key1
      secret: <base64-32-bytes>
- identity: {}

KMS v2 Provider (Full)

Copyproviders:
- kms:
    apiVersion: v2
    name: my-kms-key            # Logical name (appears in etcd prefix)
    endpoint: unix:///var/run/kmsplugin/socket.sock
    cachesize: 1000             # Number of DEKs to cache (default 1000)
    timeout: 3s                 # gRPC call timeout (default 3s)
- identity: {}                  # Read fallback; remove after re-encryption

Enabling Encryption

Enabling encryption for an existing cluster requires a careful sequence to avoid downtime and data loss. The API server must be restarted, which is momentarily disruptive in single-API-server setups.

Copy# Step 2: kubeadm API server manifest additions
# /etc/kubernetes/manifests/kube-apiserver.yaml
spec:
  containers:
  - command:
    - kube-apiserver
    # ... existing flags ...
    - --encryption-provider-config=/etc/kubernetes/encryption-config.yaml
    volumeMounts:
    - name: encryption-config
      mountPath: /etc/kubernetes/encryption-config.yaml
      readOnly: true
  volumes:
  - name: encryption-config
    hostPath:
      path: /etc/kubernetes/encryption-config.yaml
      type: File

Copy# Step 4: Re-encrypt all existing Secrets
# This reads each secret and writes it back — triggers encryption on write

kubectl get secrets --all-namespaces -o json | \
  kubectl replace -f -

# Re-encrypt ConfigMaps if configured
kubectl get configmaps --all-namespaces -o json | \
  kubectl replace -f -

# For large clusters, process namespace by namespace to avoid API server overload
for ns in $(kubectl get namespaces -o jsonpath='{.items[*].metadata.name}'); do
  echo "Processing namespace: $ns"
  kubectl get secrets -n "$ns" -o json | kubectl replace -f - || true
  sleep 1
done

Key Rotation

Key rotation for local providers (secretbox, aesgcm) requires a precise sequence. KMS v2 key rotation is handled by the KMS provider — you rotate the KEK in AWS/GCP/Azure and the plugin transparently re-wraps DEKs.

Local Key Rotation (secretbox / aesgcm)

Copy# Phase 1: Add new key as first (new writes use new key)
providers:
- secretbox:
    keys:
    - name: key2            # NEW — first = used for writes
      secret: <new-key-base64>
    - name: key1            # OLD — kept for reading existing data
      secret: <old-key-base64>

# Phase 2: After re-encryption, remove old key
providers:
- secretbox:
    keys:
    - name: key2            # Only new key remains
      secret: <new-key-base64>

KMS v2 Key Rotation

Copy# AWS KMS: rotate the CMK (creates new key material, old still valid for decrypt)
aws kms enable-key-rotation --key-id <key-id>

# GCP Cloud KMS: create a new key version
gcloud kms keys versions create \
  --key my-k8s-key \
  --keyring my-keyring \
  --location global

# Set new version as primary
gcloud kms keys versions get-public-key 1 \
  --key my-k8s-key --keyring my-keyring --location global

# After KMS rotation: force API server to re-derive DEK from new KEK
# Restart API server OR wait for DEK expiry (1h default)

# Re-encrypt all secrets so they use new DEK wrapped by new KEK
kubectl get secrets --all-namespaces -o json | kubectl replace -f -

Verifying Encryption

After enabling encryption, verify that etcd actually stores encrypted data — not just that the API server accepted the configuration.

Copy# Create a test secret
kubectl create secret generic encryption-test \
  --from-literal=test-key=test-value \
  -n default

# Read directly from etcd using etcdctl
# (must run on a control plane node with etcd certs)
ETCDCTL_API=3 etcdctl \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  get /registry/secrets/default/encryption-test

# Expected output if encrypted (secretbox):
# /registry/secrets/default/encryption-test
# k8s:enc:secretbox:v1:key1:<binary ciphertext>

# If you see plaintext JSON/protobuf: encryption is NOT working
# k8s:enc: prefix MUST be present

Copy# Alternative: pipe through hexdump to inspect binary prefix
ETCDCTL_API=3 etcdctl \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  get /registry/secrets/default/encryption-test | hexdump -C | head -5

# Look for the "k8s:enc:" ASCII prefix in the hex output
# 6b 38 73 3a 65 6e 63 3a  = "k8s:enc:"

Copy# Check API server logs for encryption provider initialization
kubectl logs -n kube-system kube-apiserver-<node-name> | \
  grep -i "encrypt\|kms\|provider"

# Expected log line on successful KMS v2 init:
# "KMS plugin initialized" or "encryption provider" messages

# Verify encryption config is loaded
kubectl get --raw /healthz/etcd

# Check KMS plugin health
kubectl get --raw /healthz/kms-providers

etcd Disk-Level Encryption

Kubernetes-level encryption (via EncryptionConfiguration) and disk-level encryption (LUKS/dm-crypt) are complementary, not mutually exclusive. Disk encryption protects against physical disk theft; Kubernetes encryption protects against etcd API compromise and backup theft.

etcd TLS Configuration

Regardless of encryption at rest, etcd communications must be TLS-encrypted in transit. This is separate from encryption at rest but equally important.

Copy# etcd flags for mutual TLS (peer and client)

# Client-to-server TLS (kube-apiserver → etcd)
--cert-file=/etc/kubernetes/pki/etcd/server.crt
--key-file=/etc/kubernetes/pki/etcd/server.key
--trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt
--client-cert-auth=true

# Peer TLS (etcd member ↔ etcd member)
--peer-cert-file=/etc/kubernetes/pki/etcd/peer.crt
--peer-key-file=/etc/kubernetes/pki/etcd/peer.key
--peer-trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt
--peer-client-cert-auth=true

LUKS Full Disk Encryption for etcd Data Directory

Copy# Create LUKS encrypted volume for etcd data
# (Run before etcd is initialized)

# 1. Create LUKS container on device /dev/sdb
cryptsetup luksFormat --type luks2 \
  --cipher aes-xts-plain64 \
  --key-size 512 \
  --hash sha256 \
  --pbkdf argon2id \
  /dev/sdb

# 2. Open the LUKS volume
cryptsetup luksOpen /dev/sdb etcd-data

# 3. Format the plaintext device
mkfs.ext4 /dev/mapper/etcd-data

# 4. Mount as etcd data directory
mount /dev/mapper/etcd-data /var/lib/etcd

# 5. Add to /etc/crypttab for auto-open on boot
# etcd-data  /dev/sdb  /etc/etcd-luks.key  luks

# 6. Key stored in HSM or cloud KMS — NOT on the same disk

Node Storage Security

Encryption at rest for etcd is only one layer. Data also lands on node storage through container filesystems, emptyDir volumes, and node-local resources.

emptyDir and tmpfs

Copy# emptyDir defaults to node disk — data survives pod restarts on same node
# Use medium: Memory (tmpfs) for sensitive temporary data
spec:
  containers:
  - name: app
    volumeMounts:
    - name: scratch
      mountPath: /tmp/secrets
  volumes:
  - name: scratch
    emptyDir:
      medium: Memory   # tmpfs — data in RAM, never written to disk
      sizeLimit: 64Mi

Secret Volume Default Mode

Copy# Secret volumes are projected to tmpfs on the node by default (since K8s 1.3)
# They do NOT write to the node's disk filesystem
# However, container images and writable layers ARE on disk

volumes:
- name: db-creds
  secret:
    secretName: db-credentials
    defaultMode: 0400   # Read-only for owner only (pod UID)

Container Image Layer Encryption (OCI)

OCI image encryption (ocicrypt) allows container image layers to be encrypted. This is distinct from Kubernetes-level encryption and is relatively uncommon — it protects image content at rest in the registry and on node disk.

Managed Kubernetes Encryption

Amazon EKS

Copy# Enable Secrets encryption with AWS KMS CMK at cluster creation
aws eks create-cluster \
  --name my-cluster \
  --kubernetes-version 1.29 \
  --resources-vpc-config subnetIds=subnet-xxx,securityGroupIds=sg-xxx \
  --encryption-config '[{"resources":["secrets"],"provider":{"keyArn":"arn:aws:kms:us-east-1:123456789012:key/xxx"}}]'

# Enable on existing cluster
aws eks associate-encryption-config \
  --cluster-name my-cluster \
  --encryption-config '[{"resources":["secrets"],"provider":{"keyArn":"arn:aws:kms:..."}}]'

# Verify encryption config
aws eks describe-cluster --name my-cluster \
  --query 'cluster.encryptionConfig'

GKE (Google Kubernetes Engine)

Copy# GKE uses Google-managed encryption by default (AES-256)
# For CMEK (Customer-Managed Encryption Keys):
gcloud container clusters create my-cluster \
  --database-encryption-key projects/my-project/locations/global/keyRings/my-ring/cryptoKeys/my-key

# GKE Application-layer secrets encryption (using Cloud KMS)
gcloud container clusters update my-cluster \
  --database-encryption-key projects/my-project/locations/global/keyRings/my-ring/cryptoKeys/my-key

AKS (Azure Kubernetes Service)

Copy# AKS encrypts etcd at rest with Microsoft-managed keys by default
# For customer-managed keys (CMK):
az aks create \
  --resource-group myRG \
  --name myAKS \
  --enable-encryption-at-host

# Azure Disk Encryption for node OS/data disks
az aks update \
  --resource-group myRG \
  --name myAKS \
  --enable-disk-encryption

Metrics, Alerts & Runbooks

Key Metrics

Alerts

Copy# Alert: KMS plugin unhealthy
- alert: KMSPluginUnhealthy
  expr: up{job="kube-apiserver"} == 1 and absent(kms_plugin_rpc_duration_seconds)
  for: 5m
  severity: critical
  annotations:
    summary: "KMS plugin not reachable — new Secret writes will fail"

# Alert: High KMS call latency
- alert: KMSHighLatency
  expr: histogram_quantile(0.99, kms_plugin_rpc_duration_seconds_bucket) > 1
  for: 5m
  annotations:
    summary: "KMS gRPC p99 latency exceeds 1s — API server write latency elevated"

# Alert: Encryption transformation errors
- alert: EncryptionTransformationErrors
  expr: increase(apiserver_storage_transformation_operations_total{status="failed"}[5m]) > 0
  severity: critical
  annotations:
    summary: "Encryption/decryption failures — data may be inaccessible"

# Alert: DEK cache full
- alert: DEKCacheNearFull
  expr: apiserver_envelope_encryption_dek_cache_fill_percent > 90
  annotations:
    summary: "DEK cache over 90% full — consider increasing cachesize"

Runbooks

Best Practices