Service Accounts
On this page
- ServiceAccount Model
- The default ServiceAccount Risk
- Token Evolution: Legacy vs Bound
- Projected ServiceAccount Tokens
- TokenRequest API
- automountServiceAccountToken
- Token Inside the Pod
- OIDC Discovery & Federation
- RBAC for Service Accounts
- Service Account Patterns
- TokenReview API
- Workload Identity (Cloud)
- Auditing Service Accounts
- Metrics & Alerts
- Best Practices
Coverage checklist
- ServiceAccount: namespace-scoped, auto-created default
- default SA risks: auto-mounted, shared by all pods
- Legacy SA tokens: permanent, no expiry, stored as Secret
- Bound projected tokens (1.20+ default): audience, expiry, pod-binding
- LegacyServiceAccountTokenNoAutoGeneration (1.24)
- LegacyServiceAccountTokenTracking (1.26)
- LegacyServiceAccountTokenCleanUp (1.28)
- Projected SA token volume YAML
- TokenRequest API: programmatic token creation
- automountServiceAccountToken: false on SA and pod
- In-pod token path: /var/run/secrets/kubernetes.io/serviceaccount/
- Token fields: namespace, ca.crt, token
- JWT payload: iss, sub, aud, exp, kubernetes claims
- OIDC discovery endpoint for SA tokens
- Federation to cloud IAM (IRSA/Workload Identity cross-ref)
- RBAC: one SA per workload; avoid default SA
- SA for operators: namespace vs cluster scope trade-offs
- Cross-namespace SA: RoleBinding can reference SA from other namespace
- TokenReview API: validate tokens externally
- SA token abuse patterns: container escape, lateral movement
- Audit SA tokens: find unused, find overly-broad
- kubectl auth can-i --as=system:serviceaccount:ns:name
- rakkess / rbac-lookup for SA audit
- 5 metrics, 4 alerts, 5 runbooks, 8 best practices
ServiceAccount Model
A ServiceAccount (SA) is a namespace-scoped Kubernetes object that provides an identity for processes running inside pods. When a pod makes API server calls, it authenticates using the ServiceAccount token that is (optionally) mounted into the pod.
ServiceAccount (namespace-scoped)
├── name: my-app
├── namespace: production
├── automountServiceAccountToken: false (opt-in for pods that need API access)
└── imagePullSecrets: [...] (inherited by pods using this SA)
Pod uses SA:
spec.serviceAccountName: my-app
→ kubelet mounts token into pod at /var/run/secrets/kubernetes.io/serviceaccount/
→ pod uses token to authenticate to kube-apiserver
→ RBAC authorization: is system:serviceaccount:production:my-app allowed?
Authentication identity string:
system:serviceaccount::
e.g.: system:serviceaccount:production:my-app
ServiceAccount Object
apiVersion: v1
kind: ServiceAccount
metadata:
name: my-app
namespace: production
annotations:
# Cloud workload identity annotations (see Workload Identity section)
eks.amazonaws.com/role-arn: "arn:aws:iam::123456789012:role/my-app-role"
automountServiceAccountToken: false # disable auto-mount; opt pods in explicitly
imagePullSecrets: # all pods using this SA inherit these
- name: private-registry-creds
Every Namespace Gets a default ServiceAccount
When a namespace is created, Kubernetes automatically creates a ServiceAccount named default. Any pod that does not specify a serviceAccountName automatically uses the default SA. This is the root cause of many SA security issues.
The default ServiceAccount Risk
default SA token. If any one of those workloads is compromised, the attacker gets a token that has all permissions granted to default SA across all RBAC bindings targeting it. Worse: many operators and Helm charts (incorrectly) bind ClusterRoles to the default SA, progressively widening its permissions over time.
Harden the Default ServiceAccount
# Apply to every namespace on cluster creation or via admission policy
kubectl patch serviceaccount default \
-n production \
--patch '{"automountServiceAccountToken": false}'
# Verify: all namespaces should have default SA with automount disabled
kubectl get serviceaccounts --all-namespaces \
-o jsonpath='{range .items[?(@.metadata.name=="default")]}{.metadata.namespace}: automount={.automountServiceAccountToken}{"\n"}{end}'
# Enforce via Kyverno: all new namespaces get default SA with automount=false
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
name: disable-default-sa-automount
spec:
rules:
- name: disable-automount-default-sa
match:
any:
- resources:
kinds: ["ServiceAccount"]
mutate:
patchStrategicMerge:
metadata:
name: default
automountServiceAccountToken: false
Token Evolution: Legacy vs Bound
Legacy Tokens (pre-1.22 default)
Before Kubernetes 1.22, service account tokens were stored as Secret objects and auto-mounted into pods. These tokens had critical security weaknesses:
| Property | Legacy Token | Bound Projected Token |
|---|---|---|
| Storage | Secret object in etcd | Generated on-demand by kubelet |
| Expiry | Never expires | Configurable (default 1 hour) |
| Audience | Any audience | Bound to specific audience |
| Pod binding | Not bound to a pod | Bound to specific pod UID |
| Node binding | Not bound to a node | Bound to node |
| Rotation | Never rotates | Auto-rotated before expiry |
| Revocation | Delete the Secret | Invalidated when pod or SA is deleted |
| Risk if stolen | Valid forever until Secret deleted | Valid only until expiry (≤1h default) |
Legacy Token Deprecation Timeline
| Version | Change |
|---|---|
| 1.20 | Bound tokens introduced as default for projected volumes; legacy tokens still auto-created as Secrets |
| 1.22 | Bound projected tokens become the default for new pods |
| 1.24 | LegacyServiceAccountTokenNoAutoGeneration: legacy token Secrets no longer auto-created for new SAs |
| 1.26 | LegacyServiceAccountTokenTracking: API server tracks which legacy tokens were recently used |
| 1.28 | LegacyServiceAccountTokenCleanUp: unused legacy tokens auto-invalidated after 1 year |
| 1.29+ | Manual creation of legacy token Secrets still supported but strongly discouraged |
type: kubernetes.io/service-account-token and it will be populated with a non-expiring token. This pattern is used by some CI/CD tools and external monitoring systems. Audit for these: kubectl get secrets --all-namespaces --field-selector type=kubernetes.io/service-account-token. Migrate them to bound tokens via the TokenRequest API.
Projected ServiceAccount Tokens
Bound projected tokens are the modern, secure way to mount SA tokens into pods. They are generated by the kubelet via the TokenRequest API and mounted as projected volumes. They are automatically rotated by the kubelet before expiry.
spec:
serviceAccountName: my-app
volumes:
- name: kube-api-token
projected:
sources:
- serviceAccountToken:
path: token
expirationSeconds: 3600 # default; min 600s (10 min); kubelet rotates at 80% of TTL
audience: "https://kubernetes.default.svc.cluster.local" # audience claim in JWT
- configMap:
name: kube-root-ca.crt # CA cert for verifying API server
items:
- key: ca.crt
path: ca.crt
- downwardAPI:
items:
- path: namespace
fieldRef:
fieldPath: metadata.namespace
containers:
- name: app
volumeMounts:
- name: kube-api-token
mountPath: /var/run/secrets/kubernetes.io/serviceaccount
readOnly: true
/var/run/secrets/kubernetes.io/serviceaccount/token is updated in place. Applications using the Kubernetes client-go library automatically re-read the token file — they do not need a restart. Applications using static token caching must re-read the file periodically.
TokenRequest API
The TokenRequest API (GA 1.20) allows creating bound tokens for a ServiceAccount programmatically, for any duration and audience. This is used by kubelet internally for projected volumes, and can be used by external systems that need short-lived tokens.
# Create a bound token for a SA via kubectl
kubectl create token my-app \
--namespace production \
--duration 3600s \
--audience "https://kubernetes.default.svc"
# Returns a JWT that expires in 1h, bound to the SA
# TokenRequest API call (used by external authenticators, CSI drivers, OIDC proxies)
apiVersion: authentication.k8s.io/v1
kind: TokenRequest
metadata:
name: my-app
namespace: production
spec:
audiences:
- "https://kubernetes.default.svc.cluster.local"
- "vault" # custom audience for Vault auth
expirationSeconds: 7200
boundObjectRef: # optionally bind to a specific pod
kind: Pod
name: my-app-xyz
uid: "abc-123-..." # token invalidated if pod UID changes
# RBAC required to create tokens for a SA:
# verb: create, resource: serviceaccounts/token, resourceNames: [sa-name]
# Example: allow a CI system to create tokens for a specific SA
kubectl create role token-creator \
--verb=create \
--resource=serviceaccounts/token \
--resource-name=my-app \
-n production
automountServiceAccountToken
The automountServiceAccountToken field controls whether a SA token is automatically mounted into pods. It can be set at the ServiceAccount level (applies to all pods using the SA) or at the pod level (overrides the SA setting).
| SA Setting | Pod Setting | Result |
|---|---|---|
| true (default) | not set | Token mounted |
| true | true | Token mounted |
| true | false | Token NOT mounted (pod overrides SA) |
| false | not set | Token NOT mounted |
| false | true | Token mounted (pod overrides SA) |
| false | false | Token NOT mounted |
# Pattern: disable at SA level, enable only for pods that need API access
apiVersion: v1
kind: ServiceAccount
metadata:
name: api-client
namespace: production
automountServiceAccountToken: false # default OFF
---
# Pod that needs API access: explicitly opt in
spec:
serviceAccountName: api-client
automountServiceAccountToken: true # explicit opt-in for this specific pod
---
# Pod that doesn't need API access: inherit SA default (false)
spec:
serviceAccountName: api-client
# automountServiceAccountToken not set → inherits false from SA
# result: no token mounted → pod cannot call kube-apiserver at all
automountServiceAccountToken: false on the SA (or pod) reduces the blast radius of a container compromise — the attacker cannot use the mounted token to enumerate cluster resources or escalate privileges.
Token Inside the Pod
When a token is mounted, three files appear at /var/run/secrets/kubernetes.io/serviceaccount/:
| File | Contents | Usage |
|---|---|---|
token | JWT bearer token for the SA | Sent as Authorization: Bearer <token> header to API server |
ca.crt | Cluster CA certificate | Used to verify the API server's TLS certificate |
namespace | Current namespace name (plain text) | Used by client libraries to determine the pod's namespace |
JWT Payload Structure
# Decode a projected token to inspect its claims
cat /var/run/secrets/kubernetes.io/serviceaccount/token | \
cut -d. -f2 | base64 -d 2>/dev/null | jq .
{
"aud": ["https://kubernetes.default.svc.cluster.local"],
"exp": 1735689600, // expiration timestamp
"iat": 1735686000, // issued-at timestamp
"iss": "https://kubernetes.default.svc.cluster.local",
"kubernetes.io": {
"namespace": "production",
"node": {
"name": "node-1",
"uid": "abc-123"
},
"pod": {
"name": "my-app-xyz-abc",
"uid": "def-456" // token invalidated when this pod is deleted
},
"serviceaccount": {
"name": "my-app",
"uid": "ghi-789"
},
"warnafter": 1735688400 // kubelet will rotate token after this time
},
"nbf": 1735686000,
"sub": "system:serviceaccount:production:my-app"
}
Using the Token from Within a Pod
# From inside a pod: call the API server using the mounted token
TOKEN=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)
APISERVER=https://kubernetes.default.svc
CACERT=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt
NAMESPACE=$(cat /var/run/secrets/kubernetes.io/serviceaccount/namespace)
# List pods in the same namespace
curl -s --cacert $CACERT \
-H "Authorization: Bearer $TOKEN" \
"$APISERVER/api/v1/namespaces/$NAMESPACE/pods" | jq '.items[].metadata.name'
OIDC Discovery & Federation
Kubernetes exposes an OIDC discovery endpoint that allows external systems (AWS IAM, GCP IAM, Vault, etc.) to verify SA tokens without calling the Kubernetes API. This is the foundation for IRSA and Workload Identity.
OIDC Discovery Endpoint:
https://{apiserver}/.well-known/openid-configuration
→ returns issuer URL, JWKS URI, supported algorithms
JWKS Endpoint:
https://{apiserver}/openid/v1/jwks
→ returns the public keys used to sign SA tokens
Verification flow (e.g., AWS STS AssumeRoleWithWebIdentity):
1. Pod presents SA JWT to AWS STS
2. AWS STS reads iss claim from JWT header
3. AWS STS fetches JWKS from {iss}/.well-known/openid-configuration
4. AWS STS verifies JWT signature against JWKS public keys
5. AWS STS verifies aud claim matches configured value
6. AWS STS issues temporary credentials if trust policy matches sub claim
# Inspect cluster OIDC discovery (from within cluster)
curl -s https://kubernetes.default.svc/.well-known/openid-configuration | jq
# From external systems: the issuer URL is configured on the API server
# --service-account-issuer=https://oidc.eks.us-east-1.amazonaws.com/id/CLUSTER_ID
# For EKS: the issuer is the EKS OIDC endpoint
# For GKE: https://container.googleapis.com/v1/projects/.../serviceAccounts
# For self-managed: configure --service-account-issuer with a publicly reachable URL
--service-account-issuer with the public URL.
RBAC for Service Accounts
The core principle is one dedicated ServiceAccount per workload, each with minimal RBAC permissions. See 01-rbac.html for full RBAC documentation. Key SA-specific patterns:
Dedicated SA with Minimal Permissions
# Pattern: every workload gets its own SA
apiVersion: v1
kind: ServiceAccount
metadata:
name: payment-service
namespace: production
automountServiceAccountToken: false # explicitly opt pods in
---
# Only grant what's needed — this SA manages its own ConfigMaps only
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: payment-service-config
namespace: production
rules:
- apiGroups: [""]
resources: ["configmaps"]
resourceNames: ["payment-config", "payment-feature-flags"]
verbs: ["get", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: payment-service-config
namespace: production
subjects:
- kind: ServiceAccount
name: payment-service
namespace: production
roleRef:
kind: Role
name: payment-service-config
apiGroup: rbac.authorization.k8s.io
SA for Cluster-Wide Operators
# Operator SA in its own namespace with ClusterRole
apiVersion: v1
kind: ServiceAccount
metadata:
name: my-operator
namespace: my-operator-system
automountServiceAccountToken: true # operator needs API access
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: my-operator
subjects:
- kind: ServiceAccount
name: my-operator
namespace: my-operator-system # SA namespace must be specified
roleRef:
kind: ClusterRole
name: my-operator # defined in rbac.html examples
apiGroup: rbac.authorization.k8s.io
Cross-Namespace SA Reference
# Grant a SA from namespace A access to namespace B's resources
# The SA subject can reference a different namespace than the binding
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: monitoring-pod-reader
namespace: production # binding is in production
subjects:
- kind: ServiceAccount
name: prometheus # SA is in monitoring namespace
namespace: monitoring # SA's namespace explicitly specified
roleRef:
kind: ClusterRole
name: view
apiGroup: rbac.authorization.k8s.io
Service Account Patterns
Pattern: SA for Init Containers vs Main Container
# If init container needs API access but main container doesn't,
# disable automount at pod level and add projected volume only to init container.
# However, Kubernetes doesn't support per-container SA tokens natively.
# Workaround: use TokenRequest API from an init container to fetch a scoped token,
# store it in an emptyDir, and have the main container use that file.
spec:
serviceAccountName: init-job-sa # SA with init permissions
automountServiceAccountToken: true # needed for init
initContainers:
- name: setup
image: kubectl:latest
command: ["/bin/sh", "-c",
"kubectl create token limited-sa --duration=300s > /shared/app-token"]
volumeMounts:
- name: shared
mountPath: /shared
containers:
- name: app
env:
- name: KUBE_TOKEN_FILE
value: /shared/app-token # use limited scoped token
volumeMounts:
- name: shared
mountPath: /shared
volumes:
- name: shared
emptyDir:
medium: Memory # tmpfs — not written to disk
Pattern: SA Token for External Vault Auth
# Pod requests a token with custom audience for Vault
volumes:
- name: vault-token
projected:
sources:
- serviceAccountToken:
path: token
expirationSeconds: 7200
audience: "vault" # Vault configured with this audience
# Vault Kubernetes auth method verifies the token:
# vault write auth/kubernetes/login \
# role=my-app \
# jwt="$(cat /var/run/secrets/vault/token)"
Pattern: Minimal SA for Read-Only Workloads
# Workloads that don't need API access at all
# (web servers, workers, batch jobs — the majority of workloads)
# 1. Disable default SA automount in the namespace
kubectl patch serviceaccount default -n production \
-p '{"automountServiceAccountToken": false}'
# 2. Don't create a dedicated SA for pods that don't need API access
# They'll use 'default' SA with automount=false → no token at all
# 3. Only create dedicated SAs for workloads that need API access
kubectl create serviceaccount my-operator -n production
TokenReview API
The TokenReview API allows external systems (or admission webhooks) to validate a Kubernetes SA token and determine the associated identity. This is used by Vault's Kubernetes auth method, OIDC proxies, and custom authentication systems.
# TokenReview request
apiVersion: authentication.k8s.io/v1
kind: TokenReview
spec:
token: "eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9..."
audiences: # optional: validate against specific audiences
- "https://kubernetes.default.svc"
# Validate a token using kubectl
kubectl create -f - <
# RBAC needed to call TokenReview
rules:
- apiGroups: ["authentication.k8s.io"]
resources: ["tokenreviews"]
verbs: ["create"]
# The system:auth-delegator ClusterRole grants this
# Used by: extension API servers, webhook authenticators, Vault K8s auth
Workload Identity (Cloud)
Workload Identity patterns (IRSA for AWS, GCP Workload Identity, Azure Workload Identity) use SA tokens with custom audiences to federate Kubernetes identity to cloud IAM roles — eliminating static cloud credentials entirely. See 04-secrets-management.html#irsa for the full implementation details.
Key points for the SA side of these patterns:
- The SA must have the cloud-specific annotation (e.g.,
eks.amazonaws.com/role-arn) - The pod spec should set
serviceAccountNameexplicitly — never rely on default SA for cloud identity - The projected volume mounts a token with the cloud-specific audience (e.g.,
sts.amazonaws.com) - The cloud provider's admission webhook injects this volume automatically when the SA annotation is present (for EKS/GKE managed clusters)
- The SA's RBAC within Kubernetes should still be minimal — Workload Identity grants cloud IAM permissions, not K8s API permissions
Auditing Service Accounts
Find All SA RBAC Permissions
# List all roles bound to a specific SA
rbac-lookup prometheus -n monitoring -k sa
# Check what a specific SA can do
kubectl auth can-i --list \
--as=system:serviceaccount:production:my-app \
-n production
# Find SAs with cluster-admin binding
kubectl get clusterrolebindings -o json | \
jq -r '.items[] | select(.roleRef.name=="cluster-admin") |
"\(.metadata.name): \(.subjects[].kind)/\(.subjects[].namespace // "cluster")/\(.subjects[].name)"'
# Find all SAs that can list secrets
kubectl get rolebindings,clusterrolebindings --all-namespaces -o json | \
jq -r '.items[] | select(.subjects[]?.kind=="ServiceAccount") |
"\(.metadata.namespace)/\(.metadata.name) → \(.roleRef.name)"'
Find Legacy (Non-Expiring) SA Token Secrets
# Find all legacy SA token secrets (should migrate to bound tokens)
kubectl get secrets --all-namespaces \
--field-selector type=kubernetes.io/service-account-token \
-o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name,SA:.metadata.annotations.kubernetes\.io/service-account\.name'
# Check when legacy tokens were last used (1.26+ with LegacyServiceAccountTokenTracking)
kubectl get secrets --all-namespaces \
--field-selector type=kubernetes.io/service-account-token \
-o json | jq -r '.items[] |
"\(.metadata.namespace)/\(.metadata.name): last-used=\(.metadata.annotations["kubernetes.io/legacy-token-last-used"] // "never")"'
Access Matrix with rakkess
# Full access matrix for a SA
rakkess --sa my-app -n production
# Compare permissions of two SAs
rakkess --sa my-app -n production > /tmp/my-app-perms.txt
rakkess --sa other-app -n staging > /tmp/other-app-perms.txt
diff /tmp/my-app-perms.txt /tmp/other-app-perms.txt
Detect SA Token Abuse via Audit Log
# Find API calls made by a specific SA (from audit log)
cat audit.log | jq -c 'select(
.user.username == "system:serviceaccount:production:my-app"
) | {verb, resource: .objectRef.resource, namespace: .objectRef.namespace, responseCode}'
# Find SAs making unexpected API calls (e.g., listing secrets)
cat audit.log | jq -c 'select(
(.user.username | startswith("system:serviceaccount:")) and
.objectRef.resource == "secrets" and
.verb == "list"
)'
# Find pods that have accessed the K8s API (via user-agent of client-go)
cat audit.log | jq -c 'select(
.userAgent | test("kube-apiserver-admission|kubectl") | not
) | {user: .user.username, verb, resource: .objectRef.resource}'
Metrics & Alerts
Key Metrics
| Metric | Source | What It Tells You |
|---|---|---|
serviceaccount_legacy_tokens_total | kube-apiserver (1.26+) | Count of legacy (non-expiring) tokens still in use — should trend to 0 |
serviceaccount_stale_tokens_total | kube-apiserver (1.26+) | Legacy tokens not used recently — candidates for cleanup |
apiserver_authentication_attempts_total{result="success",username=~"system:serviceaccount:.*"} | kube-apiserver | Rate of SA token auth requests; spikes from unexpected SAs |
token_request_duration_seconds{namespace} | kube-apiserver | TokenRequest API latency — degradation impacts pod startup |
rest_client_requests_total{user_agent,verb,resource} | client-go | API calls from specific workloads — detect runaway controllers |
Alerts
groups:
- name: serviceaccount.rules
rules:
- alert: LegacyServiceAccountTokensInUse
expr: serviceaccount_legacy_tokens_total > 0
for: 24h
annotations:
summary: "{{ $value }} legacy (non-expiring) SA tokens still in use"
description: "Migrate to bound projected tokens. Find via: kubectl get secrets --field-selector type=kubernetes.io/service-account-token"
labels:
severity: warning
- alert: DefaultServiceAccountUsed
# Implement via audit log: user.username=system:serviceaccount:*:default
# and verb != "get" (some read-only access may be acceptable)
annotations:
summary: "Pod using default ServiceAccount made API call: {{ $labels.namespace }}"
labels:
severity: warning
- alert: ServiceAccountAPICallRateHigh
expr: |
rate(apiserver_request_total{user=~"system:serviceaccount:.*"}[5m]) > 100
for: 5m
annotations:
summary: "SA {{ $labels.user }} making >100 API calls/sec — possible runaway controller"
labels:
severity: warning
- alert: UnexpectedSecretsListBySA
# Via audit log: verb=list, resource=secrets, user=system:serviceaccount:*
annotations:
summary: "SA {{ $labels.user }} listed secrets — verify this is expected"
labels:
severity: high
Runbooks
- Legacy token still in use: Identify the SA and which pods use it (
kubectl get pods --all-namespaces -o json | jq -r '.items[] | select(.spec.serviceAccountName=="<sa>") | "\(.metadata.namespace)/\(.metadata.name)"'). Check if the pod uses the legacy Secret directly (volumes[].secret.secretName) or the projected volume. If legacy Secret: update pod spec to use projected volume. If the SA is used by an external system (CI/CD, monitoring), migrate to TokenRequest API. Delete the legacy Secret after migration. - Default SA making API calls: Identify which pods are using the default SA (
kubectl get pods -n <ns> -o jsonpath='{range .items[?(!@.spec.serviceAccountName)]}{.metadata.name}{"\n"}{end}'). Create dedicated SAs with minimal RBAC for workloads that need API access. Disable default SA automount. This is a configuration fix, not an incident — unless the default SA has elevated RBAC permissions (escalate in that case). - SA token stolen (container compromise): If you suspect a SA token was exfiltrated: (1) For bound tokens: wait for expiry (≤1h); (2) For legacy tokens: delete the Secret immediately; (3) Delete the SA and recreate to invalidate all existing tokens; (4) Audit API calls made with the token via audit log; (5) Check for any new RoleBindings or ClusterRoleBindings created by the compromised SA; (6) Rotate any credentials the SA had access to.
- Pod fails to start — cannot mount SA token: Check events:
kubectl describe pod <name>. If error is "failed to create token" the TokenRequest API may be unavailable (API server issues). If error is "SA not found", the specified SA doesn't exist — create it. If automountServiceAccountToken was explicitly set to true but SA set it to false, pod spec overrides take effect — verify the spec. - SA with excessive permissions discovered: Identify all RoleBindings/ClusterRoleBindings for the SA. Determine which permissions are actually needed (check audit log for what the SA's pods actually call). Remove overly broad bindings. Create narrow replacements. Test workloads still function. Schedule quarterly permission review for all operator/controller SAs.
Best Practices
- Create one dedicated ServiceAccount per workload — never use the default. Name it after the workload it serves (
payment-service,my-operator,prometheus). This ensures RBAC permissions are scoped to exactly the workload that needs them, and a compromise in one workload doesn't inherit permissions intended for another. - Disable automountServiceAccountToken on the default SA in every namespace. Patch the default SA immediately after namespace creation:
kubectl patch serviceaccount default -p '{"automountServiceAccountToken":false}'. Enforce this with a Kyverno ClusterPolicy that mutates new ServiceAccounts nameddefault. This prevents all unspecified pods from receiving an API token they don't need. - Set automountServiceAccountToken: false on dedicated SAs; opt pods in explicitly. Even for dedicated SAs, disable automount at the SA level and enable it only on the specific pod specs that need API access. This adds a second line of defense — if the pod spec is misconfigured to use the wrong SA, it still won't get a token.
- Migrate all legacy SA token Secrets to bound projected tokens. Audit with
kubectl get secrets --field-selector type=kubernetes.io/service-account-token. For pods: switch to projected volumes. For external systems: use the TokenRequest API to obtain short-lived tokens programmatically. Delete legacy token Secrets after migration. Target: zero legacy tokens in production. - Use the minimal token TTL that your application can handle. The default 3600s (1h) is appropriate for most workloads. For high-security workloads, consider 600s (10 min). The kubelet rotates tokens at 80% of TTL automatically — applications using client-go automatically benefit. Custom applications must re-read the token file periodically; check their re-read interval before reducing TTL.
- Use Workload Identity (IRSA/GKE/Azure) instead of static cloud credentials. Any pod that needs to call AWS, GCP, or Azure APIs should use Workload Identity — not a static credential stored in a Kubernetes Secret. See Secrets Management: IRSA for implementation. Workload Identity tokens are short-lived, automatically rotated, and have full IAM audit trails.
- Audit SA permissions quarterly using rakkess and rbac-lookup. Run
rakkess --sa <name> -n <ns>for every operator/controller SA. Identify any permissions that are no longer needed (check against current audit log usage). Permission creep is common in long-running clusters — operators accumulate RBAC rules from multiple Helm chart versions. - Alert on API calls by the default SA and on legacy token usage. Both are canaries for misconfiguration or an ongoing compromise. A properly configured cluster should show zero API calls from default SAs (except for specific intended exceptions) and zero legacy token usage. Treat any alert on these as requiring immediate investigation, not just monitoring.