Section 11

Admission Flow

Overview

Every write request to the Kubernetes API server passes through the admission chain before it is persisted to etcd. This chain enforces policy, injects defaults, and validates correctness. Understanding it is essential for debugging mysterious rejections and for building admission webhooks.

Full Admission Sequence

kubectl / controller / service
        │
        │  POST/PUT/PATCH /api/v1/...
        ▼
┌─────────────────────────────────────────────────────────┐
│                     API Server                          │
│                                                         │
│  1. AUTHENTICATION                                      │
│     ├── Bearer token (JWT) → TokenReview               │
│     ├── Client certificate → x509 validation           │
│     └── Anonymous (if enabled)                         │
│              │ identity: username, groups, extra        │
│              ▼                                          │
│  2. AUTHORIZATION (RBAC)                                │
│     ├── SubjectAccessReview: can <identity> <verb>      │
│     │   <resource> in <namespace>?                      │
│     └── Result: Allow / Deny                           │
│              │ allowed                                  │
│              ▼                                          │
│  3. MUTATING ADMISSION WEBHOOKS                         │
│     ├── API server looks up MutatingWebhookConfiguration│
│     ├── For each matching webhook (parallel):           │
│     │   ├── POST https://webhook-svc/mutate             │
│     │   │   {AdmissionReview request}                   │
│     │   └── Response: {allowed, patch (JSONPatch)}      │
│     ├── Patches applied in webhook order               │
│     └── Result: mutated object                         │
│              │                                          │
│              ▼                                          │
│  4. OBJECT SCHEMA VALIDATION                            │
│     └── OpenAPI v3 schema check (unknown fields, types) │
│              │                                          │
│              ▼                                          │
│  5. VALIDATING ADMISSION WEBHOOKS                       │
│     ├── For each matching webhook (parallel):           │
│     │   ├── POST https://webhook-svc/validate           │
│     │   └── Response: {allowed, status (error message)} │
│     └── ANY denial → entire request rejected           │
│              │                                          │
│              ▼                                          │
│  6. PERSIST TO etcd                                     │
│     └── Object written with resourceVersion             │
└─────────────────────────────────────────────────────────┘
        │
        ▼
   201 Created / 200 OK / 422 Unprocessable Entity

Authentication Methods

# Check how a request authenticated
kubectl auth whoami
# Name:   alice
# Groups: [system:authenticated]

# Service account inside cluster
# Bearer: token from /var/run/secrets/kubernetes.io/serviceaccount/token
# → API server calls TokenReview to validate JWT signature

# Client certificate (kubeconfig)
# API server validates against cluster CA
# CN = username, O = groups

# Anonymous
# Only if --anonymous-auth=true on API server (disabled in hardened clusters)

Mutating Webhook — Istio Sidecar Injection

The most common mutating webhook in production is the Istio sidecar injector. Here is the exact sequence:

kubectl apply pod.yaml
        │
        ▼
API Server receives POST /api/v1/namespaces/production/pods
        │
        │  Check MutatingWebhookConfiguration "istio-sidecar-injector"
        │  Rules: pods CREATE, namespaces with label istio-injection=enabled
        │
        ▼
POST https://istiod.istio-system.svc:443/inject
Body: AdmissionReview {
  request: {
    uid: "abc-123",
    kind: {group:"", version:"v1", kind:"Pod"},
    object: <original pod JSON>,
    operation: "CREATE"
  }
}

        ▼
istiod responds:
AdmissionReview {
  response: {
    uid: "abc-123",
    allowed: true,
    patchType: "JSONPatch",
    patch: base64(<JSONPatch array>)
  }
}

JSONPatch adds:
  - initContainers: [istio-init]          (iptables rules)
  - containers: [istio-proxy]             (Envoy sidecar)
  - volumes: [istio-envoy, istio-token]
  - annotations: {sidecar.istio.io/status: ...}

MutatingWebhookConfiguration

apiVersion: admissionregistration.k8s.io/v1
kind: MutatingWebhookConfiguration
metadata:
  name: istio-sidecar-injector
webhooks:
- name: sidecar-injector.istio.io
  clientConfig:
    service:
      name: istiod
      namespace: istio-system
      path: /inject
      port: 443
    caBundle: <base64-encoded-CA>       # must match webhook server's TLS cert
  rules:
  - apiGroups: [""]
    apiVersions: ["v1"]
    resources: ["pods"]
    operations: ["CREATE"]
  namespaceSelector:
    matchLabels:
      istio-injection: "enabled"        # only inject in labelled namespaces
  objectSelector:
    matchExpressions:
    - key: sidecar.istio.io/inject
      operator: NotIn
      values: ["false"]
  failurePolicy: Fail                   # Fail = reject pod if webhook unreachable
  sideEffects: None
  admissionReviewVersions: ["v1"]
  reinvocationPolicy: IfNeeded          # re-run if another webhook modified the object

ValidatingWebhookConfiguration — Kyverno Policy

apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
metadata:
  name: kyverno-resource-validating-webhook-cfg
webhooks:
- name: validate.kyverno.svc
  clientConfig:
    service:
      name: kyverno-svc
      namespace: kyverno
      path: /validate/fail
    caBundle: <base64-encoded-CA>
  rules:
  - apiGroups: ["*"]
    apiVersions: ["*"]
    resources: ["pods", "deployments", "statefulsets"]
    operations: ["CREATE", "UPDATE"]
  failurePolicy: Ignore               # allow if Kyverno is down (production safety)
  namespaceSelector:
    matchExpressions:
    - key: kubernetes.io/metadata.name
      operator: NotIn
      values: ["kube-system", "kyverno"]
  sideEffects: None
  timeoutSeconds: 10

Failure Modes and Debugging

failurePolicy: Fail vs Ignore

failurePolicy: Fail (default for most webhooks)
  → If the webhook server is unreachable or returns 5xx:
    the original request is REJECTED
  → Protects against policy bypass when webhook is down
  → Risk: webhook outage blocks all pod creation

failurePolicy: Ignore
  → If webhook is unreachable: request proceeds as if webhook approved
  → Safer for availability, but policy can be bypassed during webhook downtime
  → Use for non-critical webhooks (metrics, labelling)

Common Webhook Failures

# Error: "failed calling webhook ... connection refused"
# → Webhook server pod is down
kubectl get pods -n <webhook-namespace>
kubectl logs -n <webhook-namespace> -l app=<webhook> --tail=50

# Error: "x509: certificate signed by unknown authority"
# → caBundle in WebhookConfiguration doesn't match webhook server's TLS cert
# cert-manager cainjector should keep this in sync automatically
kubectl get mutatingwebhookconfiguration <name> \
  -o jsonpath='{.webhooks[0].clientConfig.caBundle}' | \
  base64 -d | openssl x509 -noout -subject -issuer

# Error: "admission webhook ... timed out"
# → Webhook server is too slow (default timeout: 10s)
# Check webhook server resource usage; increase timeoutSeconds

# Dry-run to test what webhooks would do without persisting
kubectl apply -f pod.yaml --dry-run=server

# See which webhooks fired for a resource
kubectl get events -n <namespace> | grep webhook

Admission Chain Timing

Total admission chain overhead (typical):
  Authentication:          1–5ms
  RBAC authorization:      1–3ms
  Mutating webhooks:       5–50ms per webhook (network call)
  Schema validation:       1–5ms
  Validating webhooks:     5–50ms per webhook (network call)
  etcd write:              5–50ms (fsync)
  ─────────────────────────────────
  Total:                   20–200ms (1–3 webhooks)
  Total (many webhooks):   200ms–1s

Impact: latency is additive for each webhook in the chain.
Keep webhook logic fast (<10ms). Use caching inside webhook servers.
← Previous
Service Account Token