Service Accounts

Section 06 › 05 Last updated: 2025 ~30 min read

On this page

  1. ServiceAccount Model
  2. The default ServiceAccount Risk
  3. Token Evolution: Legacy vs Bound
  4. Projected ServiceAccount Tokens
  5. TokenRequest API
  6. automountServiceAccountToken
  7. Token Inside the Pod
  8. OIDC Discovery & Federation
  9. RBAC for Service Accounts
  10. Service Account Patterns
  11. TokenReview API
  12. Workload Identity (Cloud)
  13. Auditing Service Accounts
  14. Metrics & Alerts
  15. Best Practices
Coverage checklist
  • ServiceAccount: namespace-scoped, auto-created default
  • default SA risks: auto-mounted, shared by all pods
  • Legacy SA tokens: permanent, no expiry, stored as Secret
  • Bound projected tokens (1.20+ default): audience, expiry, pod-binding
  • LegacyServiceAccountTokenNoAutoGeneration (1.24)
  • LegacyServiceAccountTokenTracking (1.26)
  • LegacyServiceAccountTokenCleanUp (1.28)
  • Projected SA token volume YAML
  • TokenRequest API: programmatic token creation
  • automountServiceAccountToken: false on SA and pod
  • In-pod token path: /var/run/secrets/kubernetes.io/serviceaccount/
  • Token fields: namespace, ca.crt, token
  • JWT payload: iss, sub, aud, exp, kubernetes claims
  • OIDC discovery endpoint for SA tokens
  • Federation to cloud IAM (IRSA/Workload Identity cross-ref)
  • RBAC: one SA per workload; avoid default SA
  • SA for operators: namespace vs cluster scope trade-offs
  • Cross-namespace SA: RoleBinding can reference SA from other namespace
  • TokenReview API: validate tokens externally
  • SA token abuse patterns: container escape, lateral movement
  • Audit SA tokens: find unused, find overly-broad
  • kubectl auth can-i --as=system:serviceaccount:ns:name
  • rakkess / rbac-lookup for SA audit
  • 5 metrics, 4 alerts, 5 runbooks, 8 best practices

ServiceAccount Model

A ServiceAccount (SA) is a namespace-scoped Kubernetes object that provides an identity for processes running inside pods. When a pod makes API server calls, it authenticates using the ServiceAccount token that is (optionally) mounted into the pod.

ServiceAccount (namespace-scoped)
  ├── name: my-app
  ├── namespace: production
  ├── automountServiceAccountToken: false   (opt-in for pods that need API access)
  └── imagePullSecrets: [...]               (inherited by pods using this SA)

Pod uses SA:
  spec.serviceAccountName: my-app
    → kubelet mounts token into pod at /var/run/secrets/kubernetes.io/serviceaccount/
    → pod uses token to authenticate to kube-apiserver
    → RBAC authorization: is system:serviceaccount:production:my-app allowed?

Authentication identity string:
  system:serviceaccount::
  e.g.: system:serviceaccount:production:my-app

ServiceAccount Object

apiVersion: v1
kind: ServiceAccount
metadata:
  name: my-app
  namespace: production
  annotations:
    # Cloud workload identity annotations (see Workload Identity section)
    eks.amazonaws.com/role-arn: "arn:aws:iam::123456789012:role/my-app-role"
automountServiceAccountToken: false   # disable auto-mount; opt pods in explicitly
imagePullSecrets:                     # all pods using this SA inherit these
- name: private-registry-creds

Every Namespace Gets a default ServiceAccount

When a namespace is created, Kubernetes automatically creates a ServiceAccount named default. Any pod that does not specify a serviceAccountName automatically uses the default SA. This is the root cause of many SA security issues.

The default ServiceAccount Risk

The default ServiceAccount is shared by all pods that don't specify one. In a busy namespace, dozens of unrelated workloads may share the same default SA token. If any one of those workloads is compromised, the attacker gets a token that has all permissions granted to default SA across all RBAC bindings targeting it. Worse: many operators and Helm charts (incorrectly) bind ClusterRoles to the default SA, progressively widening its permissions over time.

Harden the Default ServiceAccount

# Apply to every namespace on cluster creation or via admission policy
kubectl patch serviceaccount default \
  -n production \
  --patch '{"automountServiceAccountToken": false}'

# Verify: all namespaces should have default SA with automount disabled
kubectl get serviceaccounts --all-namespaces \
  -o jsonpath='{range .items[?(@.metadata.name=="default")]}{.metadata.namespace}: automount={.automountServiceAccountToken}{"\n"}{end}'
# Enforce via Kyverno: all new namespaces get default SA with automount=false
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: disable-default-sa-automount
spec:
  rules:
  - name: disable-automount-default-sa
    match:
      any:
      - resources:
          kinds: ["ServiceAccount"]
    mutate:
      patchStrategicMerge:
        metadata:
          name: default
        automountServiceAccountToken: false

Token Evolution: Legacy vs Bound

Legacy Tokens (pre-1.22 default)

Before Kubernetes 1.22, service account tokens were stored as Secret objects and auto-mounted into pods. These tokens had critical security weaknesses:

PropertyLegacy TokenBound Projected Token
StorageSecret object in etcdGenerated on-demand by kubelet
ExpiryNever expiresConfigurable (default 1 hour)
AudienceAny audienceBound to specific audience
Pod bindingNot bound to a podBound to specific pod UID
Node bindingNot bound to a nodeBound to node
RotationNever rotatesAuto-rotated before expiry
RevocationDelete the SecretInvalidated when pod or SA is deleted
Risk if stolenValid forever until Secret deletedValid only until expiry (≤1h default)

Legacy Token Deprecation Timeline

VersionChange
1.20Bound tokens introduced as default for projected volumes; legacy tokens still auto-created as Secrets
1.22Bound projected tokens become the default for new pods
1.24LegacyServiceAccountTokenNoAutoGeneration: legacy token Secrets no longer auto-created for new SAs
1.26LegacyServiceAccountTokenTracking: API server tracks which legacy tokens were recently used
1.28LegacyServiceAccountTokenCleanUp: unused legacy tokens auto-invalidated after 1 year
1.29+Manual creation of legacy token Secrets still supported but strongly discouraged
Manually created legacy token Secrets still work — and are still dangerous. You can still manually create a Secret with type: kubernetes.io/service-account-token and it will be populated with a non-expiring token. This pattern is used by some CI/CD tools and external monitoring systems. Audit for these: kubectl get secrets --all-namespaces --field-selector type=kubernetes.io/service-account-token. Migrate them to bound tokens via the TokenRequest API.

Projected ServiceAccount Tokens

Bound projected tokens are the modern, secure way to mount SA tokens into pods. They are generated by the kubelet via the TokenRequest API and mounted as projected volumes. They are automatically rotated by the kubelet before expiry.

spec:
  serviceAccountName: my-app
  volumes:
  - name: kube-api-token
    projected:
      sources:
      - serviceAccountToken:
          path: token
          expirationSeconds: 3600        # default; min 600s (10 min); kubelet rotates at 80% of TTL
          audience: "https://kubernetes.default.svc.cluster.local"  # audience claim in JWT
      - configMap:
          name: kube-root-ca.crt         # CA cert for verifying API server
          items:
          - key: ca.crt
            path: ca.crt
      - downwardAPI:
          items:
          - path: namespace
            fieldRef:
              fieldPath: metadata.namespace
  containers:
  - name: app
    volumeMounts:
    - name: kube-api-token
      mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      readOnly: true
kubelet auto-rotates projected tokens at 80% of their TTL. For a 3600s (1h) token, the kubelet fetches a new token at ~2880s (48 min), before the old one expires. The file at /var/run/secrets/kubernetes.io/serviceaccount/token is updated in place. Applications using the Kubernetes client-go library automatically re-read the token file — they do not need a restart. Applications using static token caching must re-read the file periodically.

TokenRequest API

The TokenRequest API (GA 1.20) allows creating bound tokens for a ServiceAccount programmatically, for any duration and audience. This is used by kubelet internally for projected volumes, and can be used by external systems that need short-lived tokens.

# Create a bound token for a SA via kubectl
kubectl create token my-app \
  --namespace production \
  --duration 3600s \
  --audience "https://kubernetes.default.svc"
# Returns a JWT that expires in 1h, bound to the SA
# TokenRequest API call (used by external authenticators, CSI drivers, OIDC proxies)
apiVersion: authentication.k8s.io/v1
kind: TokenRequest
metadata:
  name: my-app
  namespace: production
spec:
  audiences:
  - "https://kubernetes.default.svc.cluster.local"
  - "vault"                               # custom audience for Vault auth
  expirationSeconds: 7200
  boundObjectRef:                         # optionally bind to a specific pod
    kind: Pod
    name: my-app-xyz
    uid: "abc-123-..."                    # token invalidated if pod UID changes
# RBAC required to create tokens for a SA:
# verb: create, resource: serviceaccounts/token, resourceNames: [sa-name]
# Example: allow a CI system to create tokens for a specific SA
kubectl create role token-creator \
  --verb=create \
  --resource=serviceaccounts/token \
  --resource-name=my-app \
  -n production

automountServiceAccountToken

The automountServiceAccountToken field controls whether a SA token is automatically mounted into pods. It can be set at the ServiceAccount level (applies to all pods using the SA) or at the pod level (overrides the SA setting).

SA SettingPod SettingResult
true (default)not setToken mounted
truetrueToken mounted
truefalseToken NOT mounted (pod overrides SA)
falsenot setToken NOT mounted
falsetrueToken mounted (pod overrides SA)
falsefalseToken NOT mounted
# Pattern: disable at SA level, enable only for pods that need API access
apiVersion: v1
kind: ServiceAccount
metadata:
  name: api-client
  namespace: production
automountServiceAccountToken: false    # default OFF

---
# Pod that needs API access: explicitly opt in
spec:
  serviceAccountName: api-client
  automountServiceAccountToken: true   # explicit opt-in for this specific pod

---
# Pod that doesn't need API access: inherit SA default (false)
spec:
  serviceAccountName: api-client
  # automountServiceAccountToken not set → inherits false from SA
  # result: no token mounted → pod cannot call kube-apiserver at all
Most application pods should not have API server access at all. Web servers, workers, batch jobs, and most microservices have no reason to call the Kubernetes API. Setting automountServiceAccountToken: false on the SA (or pod) reduces the blast radius of a container compromise — the attacker cannot use the mounted token to enumerate cluster resources or escalate privileges.

Token Inside the Pod

When a token is mounted, three files appear at /var/run/secrets/kubernetes.io/serviceaccount/:

FileContentsUsage
tokenJWT bearer token for the SASent as Authorization: Bearer <token> header to API server
ca.crtCluster CA certificateUsed to verify the API server's TLS certificate
namespaceCurrent namespace name (plain text)Used by client libraries to determine the pod's namespace

JWT Payload Structure

# Decode a projected token to inspect its claims
cat /var/run/secrets/kubernetes.io/serviceaccount/token | \
  cut -d. -f2 | base64 -d 2>/dev/null | jq .
{
  "aud": ["https://kubernetes.default.svc.cluster.local"],
  "exp": 1735689600,              // expiration timestamp
  "iat": 1735686000,              // issued-at timestamp
  "iss": "https://kubernetes.default.svc.cluster.local",
  "kubernetes.io": {
    "namespace": "production",
    "node": {
      "name": "node-1",
      "uid": "abc-123"
    },
    "pod": {
      "name": "my-app-xyz-abc",
      "uid": "def-456"            // token invalidated when this pod is deleted
    },
    "serviceaccount": {
      "name": "my-app",
      "uid": "ghi-789"
    },
    "warnafter": 1735688400       // kubelet will rotate token after this time
  },
  "nbf": 1735686000,
  "sub": "system:serviceaccount:production:my-app"
}

Using the Token from Within a Pod

# From inside a pod: call the API server using the mounted token
TOKEN=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)
APISERVER=https://kubernetes.default.svc
CACERT=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt
NAMESPACE=$(cat /var/run/secrets/kubernetes.io/serviceaccount/namespace)

# List pods in the same namespace
curl -s --cacert $CACERT \
  -H "Authorization: Bearer $TOKEN" \
  "$APISERVER/api/v1/namespaces/$NAMESPACE/pods" | jq '.items[].metadata.name'

OIDC Discovery & Federation

Kubernetes exposes an OIDC discovery endpoint that allows external systems (AWS IAM, GCP IAM, Vault, etc.) to verify SA tokens without calling the Kubernetes API. This is the foundation for IRSA and Workload Identity.

OIDC Discovery Endpoint:
  https://{apiserver}/.well-known/openid-configuration
  → returns issuer URL, JWKS URI, supported algorithms

JWKS Endpoint:
  https://{apiserver}/openid/v1/jwks
  → returns the public keys used to sign SA tokens

Verification flow (e.g., AWS STS AssumeRoleWithWebIdentity):
  1. Pod presents SA JWT to AWS STS
  2. AWS STS reads iss claim from JWT header
  3. AWS STS fetches JWKS from {iss}/.well-known/openid-configuration
  4. AWS STS verifies JWT signature against JWKS public keys
  5. AWS STS verifies aud claim matches configured value
  6. AWS STS issues temporary credentials if trust policy matches sub claim
# Inspect cluster OIDC discovery (from within cluster)
curl -s https://kubernetes.default.svc/.well-known/openid-configuration | jq

# From external systems: the issuer URL is configured on the API server
# --service-account-issuer=https://oidc.eks.us-east-1.amazonaws.com/id/CLUSTER_ID

# For EKS: the issuer is the EKS OIDC endpoint
# For GKE: https://container.googleapis.com/v1/projects/.../serviceAccounts
# For self-managed: configure --service-account-issuer with a publicly reachable URL
The OIDC issuer URL must be publicly reachable for external federation. For IRSA and similar patterns, the cloud provider's IAM service must be able to fetch the JWKS from your cluster's OIDC endpoint. For EKS and GKE this is handled automatically. For self-managed clusters, you must expose the JWKS endpoint publicly (or use a separate OIDC provider like Dex or auth0) and configure --service-account-issuer with the public URL.

RBAC for Service Accounts

The core principle is one dedicated ServiceAccount per workload, each with minimal RBAC permissions. See 01-rbac.html for full RBAC documentation. Key SA-specific patterns:

Dedicated SA with Minimal Permissions

# Pattern: every workload gets its own SA
apiVersion: v1
kind: ServiceAccount
metadata:
  name: payment-service
  namespace: production
automountServiceAccountToken: false    # explicitly opt pods in
---
# Only grant what's needed — this SA manages its own ConfigMaps only
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: payment-service-config
  namespace: production
rules:
- apiGroups: [""]
  resources: ["configmaps"]
  resourceNames: ["payment-config", "payment-feature-flags"]
  verbs: ["get", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: payment-service-config
  namespace: production
subjects:
- kind: ServiceAccount
  name: payment-service
  namespace: production
roleRef:
  kind: Role
  name: payment-service-config
  apiGroup: rbac.authorization.k8s.io

SA for Cluster-Wide Operators

# Operator SA in its own namespace with ClusterRole
apiVersion: v1
kind: ServiceAccount
metadata:
  name: my-operator
  namespace: my-operator-system
automountServiceAccountToken: true     # operator needs API access
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: my-operator
subjects:
- kind: ServiceAccount
  name: my-operator
  namespace: my-operator-system        # SA namespace must be specified
roleRef:
  kind: ClusterRole
  name: my-operator                    # defined in rbac.html examples
  apiGroup: rbac.authorization.k8s.io

Cross-Namespace SA Reference

# Grant a SA from namespace A access to namespace B's resources
# The SA subject can reference a different namespace than the binding
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: monitoring-pod-reader
  namespace: production               # binding is in production
subjects:
- kind: ServiceAccount
  name: prometheus                    # SA is in monitoring namespace
  namespace: monitoring               # SA's namespace explicitly specified
roleRef:
  kind: ClusterRole
  name: view
  apiGroup: rbac.authorization.k8s.io

Service Account Patterns

Pattern: SA for Init Containers vs Main Container

# If init container needs API access but main container doesn't,
# disable automount at pod level and add projected volume only to init container.
# However, Kubernetes doesn't support per-container SA tokens natively.
# Workaround: use TokenRequest API from an init container to fetch a scoped token,
# store it in an emptyDir, and have the main container use that file.
spec:
  serviceAccountName: init-job-sa    # SA with init permissions
  automountServiceAccountToken: true # needed for init
  initContainers:
  - name: setup
    image: kubectl:latest
    command: ["/bin/sh", "-c",
      "kubectl create token limited-sa --duration=300s > /shared/app-token"]
    volumeMounts:
    - name: shared
      mountPath: /shared
  containers:
  - name: app
    env:
    - name: KUBE_TOKEN_FILE
      value: /shared/app-token        # use limited scoped token
    volumeMounts:
    - name: shared
      mountPath: /shared
  volumes:
  - name: shared
    emptyDir:
      medium: Memory                  # tmpfs — not written to disk

Pattern: SA Token for External Vault Auth

# Pod requests a token with custom audience for Vault
volumes:
- name: vault-token
  projected:
    sources:
    - serviceAccountToken:
        path: token
        expirationSeconds: 7200
        audience: "vault"             # Vault configured with this audience

# Vault Kubernetes auth method verifies the token:
# vault write auth/kubernetes/login \
#   role=my-app \
#   jwt="$(cat /var/run/secrets/vault/token)"

Pattern: Minimal SA for Read-Only Workloads

# Workloads that don't need API access at all
# (web servers, workers, batch jobs — the majority of workloads)

# 1. Disable default SA automount in the namespace
kubectl patch serviceaccount default -n production \
  -p '{"automountServiceAccountToken": false}'

# 2. Don't create a dedicated SA for pods that don't need API access
# They'll use 'default' SA with automount=false → no token at all

# 3. Only create dedicated SAs for workloads that need API access
kubectl create serviceaccount my-operator -n production

TokenReview API

The TokenReview API allows external systems (or admission webhooks) to validate a Kubernetes SA token and determine the associated identity. This is used by Vault's Kubernetes auth method, OIDC proxies, and custom authentication systems.

# TokenReview request
apiVersion: authentication.k8s.io/v1
kind: TokenReview
spec:
  token: "eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9..."
  audiences:                          # optional: validate against specific audiences
  - "https://kubernetes.default.svc"
# Validate a token using kubectl
kubectl create -f - <
# RBAC needed to call TokenReview
rules:
- apiGroups: ["authentication.k8s.io"]
  resources: ["tokenreviews"]
  verbs: ["create"]
# The system:auth-delegator ClusterRole grants this
# Used by: extension API servers, webhook authenticators, Vault K8s auth

Workload Identity (Cloud)

Workload Identity patterns (IRSA for AWS, GCP Workload Identity, Azure Workload Identity) use SA tokens with custom audiences to federate Kubernetes identity to cloud IAM roles — eliminating static cloud credentials entirely. See 04-secrets-management.html#irsa for the full implementation details.

Key points for the SA side of these patterns:

Auditing Service Accounts

Find All SA RBAC Permissions

# List all roles bound to a specific SA
rbac-lookup prometheus -n monitoring -k sa

# Check what a specific SA can do
kubectl auth can-i --list \
  --as=system:serviceaccount:production:my-app \
  -n production

# Find SAs with cluster-admin binding
kubectl get clusterrolebindings -o json | \
  jq -r '.items[] | select(.roleRef.name=="cluster-admin") |
  "\(.metadata.name): \(.subjects[].kind)/\(.subjects[].namespace // "cluster")/\(.subjects[].name)"'

# Find all SAs that can list secrets
kubectl get rolebindings,clusterrolebindings --all-namespaces -o json | \
  jq -r '.items[] | select(.subjects[]?.kind=="ServiceAccount") |
  "\(.metadata.namespace)/\(.metadata.name) → \(.roleRef.name)"'

Find Legacy (Non-Expiring) SA Token Secrets

# Find all legacy SA token secrets (should migrate to bound tokens)
kubectl get secrets --all-namespaces \
  --field-selector type=kubernetes.io/service-account-token \
  -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name,SA:.metadata.annotations.kubernetes\.io/service-account\.name'

# Check when legacy tokens were last used (1.26+ with LegacyServiceAccountTokenTracking)
kubectl get secrets --all-namespaces \
  --field-selector type=kubernetes.io/service-account-token \
  -o json | jq -r '.items[] |
  "\(.metadata.namespace)/\(.metadata.name): last-used=\(.metadata.annotations["kubernetes.io/legacy-token-last-used"] // "never")"'

Access Matrix with rakkess

# Full access matrix for a SA
rakkess --sa my-app -n production

# Compare permissions of two SAs
rakkess --sa my-app -n production > /tmp/my-app-perms.txt
rakkess --sa other-app -n staging > /tmp/other-app-perms.txt
diff /tmp/my-app-perms.txt /tmp/other-app-perms.txt

Detect SA Token Abuse via Audit Log

# Find API calls made by a specific SA (from audit log)
cat audit.log | jq -c 'select(
  .user.username == "system:serviceaccount:production:my-app"
) | {verb, resource: .objectRef.resource, namespace: .objectRef.namespace, responseCode}'

# Find SAs making unexpected API calls (e.g., listing secrets)
cat audit.log | jq -c 'select(
  (.user.username | startswith("system:serviceaccount:")) and
  .objectRef.resource == "secrets" and
  .verb == "list"
)'

# Find pods that have accessed the K8s API (via user-agent of client-go)
cat audit.log | jq -c 'select(
  .userAgent | test("kube-apiserver-admission|kubectl") | not
) | {user: .user.username, verb, resource: .objectRef.resource}'

Metrics & Alerts

Key Metrics

MetricSourceWhat It Tells You
serviceaccount_legacy_tokens_totalkube-apiserver (1.26+)Count of legacy (non-expiring) tokens still in use — should trend to 0
serviceaccount_stale_tokens_totalkube-apiserver (1.26+)Legacy tokens not used recently — candidates for cleanup
apiserver_authentication_attempts_total{result="success",username=~"system:serviceaccount:.*"}kube-apiserverRate of SA token auth requests; spikes from unexpected SAs
token_request_duration_seconds{namespace}kube-apiserverTokenRequest API latency — degradation impacts pod startup
rest_client_requests_total{user_agent,verb,resource}client-goAPI calls from specific workloads — detect runaway controllers

Alerts

groups:
- name: serviceaccount.rules
  rules:

  - alert: LegacyServiceAccountTokensInUse
    expr: serviceaccount_legacy_tokens_total > 0
    for: 24h
    annotations:
      summary: "{{ $value }} legacy (non-expiring) SA tokens still in use"
      description: "Migrate to bound projected tokens. Find via: kubectl get secrets --field-selector type=kubernetes.io/service-account-token"
    labels:
      severity: warning

  - alert: DefaultServiceAccountUsed
    # Implement via audit log: user.username=system:serviceaccount:*:default
    # and verb != "get" (some read-only access may be acceptable)
    annotations:
      summary: "Pod using default ServiceAccount made API call: {{ $labels.namespace }}"
    labels:
      severity: warning

  - alert: ServiceAccountAPICallRateHigh
    expr: |
      rate(apiserver_request_total{user=~"system:serviceaccount:.*"}[5m]) > 100
    for: 5m
    annotations:
      summary: "SA {{ $labels.user }} making >100 API calls/sec — possible runaway controller"
    labels:
      severity: warning

  - alert: UnexpectedSecretsListBySA
    # Via audit log: verb=list, resource=secrets, user=system:serviceaccount:*
    annotations:
      summary: "SA {{ $labels.user }} listed secrets — verify this is expected"
    labels:
      severity: high

Runbooks

  1. Legacy token still in use: Identify the SA and which pods use it (kubectl get pods --all-namespaces -o json | jq -r '.items[] | select(.spec.serviceAccountName=="<sa>") | "\(.metadata.namespace)/\(.metadata.name)"'). Check if the pod uses the legacy Secret directly (volumes[].secret.secretName) or the projected volume. If legacy Secret: update pod spec to use projected volume. If the SA is used by an external system (CI/CD, monitoring), migrate to TokenRequest API. Delete the legacy Secret after migration.
  2. Default SA making API calls: Identify which pods are using the default SA (kubectl get pods -n <ns> -o jsonpath='{range .items[?(!@.spec.serviceAccountName)]}{.metadata.name}{"\n"}{end}'). Create dedicated SAs with minimal RBAC for workloads that need API access. Disable default SA automount. This is a configuration fix, not an incident — unless the default SA has elevated RBAC permissions (escalate in that case).
  3. SA token stolen (container compromise): If you suspect a SA token was exfiltrated: (1) For bound tokens: wait for expiry (≤1h); (2) For legacy tokens: delete the Secret immediately; (3) Delete the SA and recreate to invalidate all existing tokens; (4) Audit API calls made with the token via audit log; (5) Check for any new RoleBindings or ClusterRoleBindings created by the compromised SA; (6) Rotate any credentials the SA had access to.
  4. Pod fails to start — cannot mount SA token: Check events: kubectl describe pod <name>. If error is "failed to create token" the TokenRequest API may be unavailable (API server issues). If error is "SA not found", the specified SA doesn't exist — create it. If automountServiceAccountToken was explicitly set to true but SA set it to false, pod spec overrides take effect — verify the spec.
  5. SA with excessive permissions discovered: Identify all RoleBindings/ClusterRoleBindings for the SA. Determine which permissions are actually needed (check audit log for what the SA's pods actually call). Remove overly broad bindings. Create narrow replacements. Test workloads still function. Schedule quarterly permission review for all operator/controller SAs.

Best Practices

  1. Create one dedicated ServiceAccount per workload — never use the default. Name it after the workload it serves (payment-service, my-operator, prometheus). This ensures RBAC permissions are scoped to exactly the workload that needs them, and a compromise in one workload doesn't inherit permissions intended for another.
  2. Disable automountServiceAccountToken on the default SA in every namespace. Patch the default SA immediately after namespace creation: kubectl patch serviceaccount default -p '{"automountServiceAccountToken":false}'. Enforce this with a Kyverno ClusterPolicy that mutates new ServiceAccounts named default. This prevents all unspecified pods from receiving an API token they don't need.
  3. Set automountServiceAccountToken: false on dedicated SAs; opt pods in explicitly. Even for dedicated SAs, disable automount at the SA level and enable it only on the specific pod specs that need API access. This adds a second line of defense — if the pod spec is misconfigured to use the wrong SA, it still won't get a token.
  4. Migrate all legacy SA token Secrets to bound projected tokens. Audit with kubectl get secrets --field-selector type=kubernetes.io/service-account-token. For pods: switch to projected volumes. For external systems: use the TokenRequest API to obtain short-lived tokens programmatically. Delete legacy token Secrets after migration. Target: zero legacy tokens in production.
  5. Use the minimal token TTL that your application can handle. The default 3600s (1h) is appropriate for most workloads. For high-security workloads, consider 600s (10 min). The kubelet rotates tokens at 80% of TTL automatically — applications using client-go automatically benefit. Custom applications must re-read the token file periodically; check their re-read interval before reducing TTL.
  6. Use Workload Identity (IRSA/GKE/Azure) instead of static cloud credentials. Any pod that needs to call AWS, GCP, or Azure APIs should use Workload Identity — not a static credential stored in a Kubernetes Secret. See Secrets Management: IRSA for implementation. Workload Identity tokens are short-lived, automatically rotated, and have full IAM audit trails.
  7. Audit SA permissions quarterly using rakkess and rbac-lookup. Run rakkess --sa <name> -n <ns> for every operator/controller SA. Identify any permissions that are no longer needed (check against current audit log usage). Permission creep is common in long-running clusters — operators accumulate RBAC rules from multiple Helm chart versions.
  8. Alert on API calls by the default SA and on legacy token usage. Both are canaries for misconfiguration or an ongoing compromise. A properly configured cluster should show zero API calls from default SAs (except for specific intended exceptions) and zero legacy token usage. Treat any alert on these as requiring immediate investigation, not just monitoring.