Webhook Flow
Overview
Traces the complete admission webhook invocation path — from the API server receiving a request, through the mutating and validating webhook chains, to the final object being written to etcd or rejected.
Webhook Types
MutatingAdmissionWebhook
→ Called BEFORE object is stored
→ Can modify (mutate) the object — e.g., inject sidecar, set defaults
→ All mutating webhooks run, then the final merged object is used
ValidatingAdmissionWebhook
→ Called AFTER all mutations are done
→ CANNOT modify the object — only Allow or Deny
→ All validating webhooks run in parallel; any Deny rejects the request
Order: mutating (serial) → schema validation → validating (parallel)Full Webhook Invocation Sequence
kubectl API Server Mutating WH 1 Mutating WH 2 Validating WH etcd
│ │ │ │ │ │
│─POST Pod ────►│ │ │ │ │
│ │ │ │ │ │
│ ┌─── Authentication + Authorization ──────┐ │ │
│ └────────────────────────────────────────┘ │ │
│ │ │ │ │ │
│ ┌─── Mutating Webhook Chain ───────────────────────┐ │ │
│ │ │ │ │
│ │ API Server: iterate MutatingWebhookConfiguration│ │ │
│ │ rules that match: CREATE pods │ │ │
│ │ │ │ │
│ │──AdmissionReview POST ───────────────────────────►│ │
│ │ {operation:CREATE, │ │ │
│ │ object: {pod spec...}, │ │ │
│ │ uid: req-abc} │ │ │ │
│ │◄── AdmissionResponse ────────────────────────────│ │
│ │ {allowed:true, │ │ │
│ │ patch: [ │ │ │
│ │ {op:add, path:/spec/containers/-, │ │ │
│ │ value:{name:istio-proxy,...}} │ │ │
│ │ ]} │ │ │ │
│ │ │ │ │
│ │ [API Server applies JSON patch to object] │ │ │
│ │ │ │ │
│ │──AdmissionReview POST ──────────────────────────────► │
│ │ (object with istio-proxy injected) │ │ │
│ │◄── AdmissionResponse (allowed:true, no patch) ────── │
│ │ │ │ │
│ └───────────────────────────────────────────────────┘ │ │
│ │ │ │ │ │
│ ┌─── Schema Validation ──────────────────────────────┐ │
│ │ Validate final mutated object against CRD schema │ │
│ └────────────────────────────────────────────────────┘ │
│ │ │ │ │ │
│ ┌─── Validating Webhook Chain ──────────────────────┐│ │
│ │ ││ │
│ │ All validating webhooks called in PARALLEL ││ │
│ │──AdmissionReview POST ─────────────────────────────► │
│ │ (fully mutated object) ││ │
│ │◄── AdmissionResponse {allowed:true} ─────────────── │
│ │ ││ │
│ └───────────────────────────────────────────────────┘│ │
│ │ │ │ │ │
│ │──WRITE ───────────────────────────────────────────────────►│
│◄── 201 Created│ │ │ │ │AdmissionReview Request and Response
// Request sent by API server to webhook
{
"apiVersion": "admission.k8s.io/v1",
"kind": "AdmissionReview",
"request": {
"uid": "705ab4f5-6393-11e8-b7cc-42010a800002",
"kind": {"group": "", "version": "v1", "resource": "pods"},
"resource": {"group": "", "version": "v1", "resource": "pods"},
"requestKind": {"group": "", "version": "v1", "resource": "pods"},
"name": "payments-api-xxx",
"namespace": "production",
"operation": "CREATE",
"userInfo": {
"username": "alice",
"groups": ["dev-team", "system:authenticated"]
},
"object": { ... pod spec ... },
"oldObject": null,
"dryRun": false
}
}// Mutating webhook response — with JSON patch
{
"apiVersion": "admission.k8s.io/v1",
"kind": "AdmissionReview",
"response": {
"uid": "705ab4f5-6393-11e8-b7cc-42010a800002",
"allowed": true,
"patchType": "JSONPatch",
"patch": "W3sib3AiOiJhZGQiLCAicGF0aCI6ICIvc3BlYy9jb250YWluZXJzLy0iLCAidmFsdWUiOiB7Im5hbWUiOiAiaXN0aW8tcHJveHkifX1d"
// base64-encoded JSON patch array:
// [{"op":"add","path":"/spec/containers/-","value":{"name":"istio-proxy",...}}]
}
}// Validating webhook response — deny
{
"apiVersion": "admission.k8s.io/v1",
"kind": "AdmissionReview",
"response": {
"uid": "705ab4f5-6393-11e8-b7cc-42010a800002",
"allowed": false,
"status": {
"code": 403,
"message": "Pod must have resource requests set on all containers"
}
}
}MutatingWebhookConfiguration
apiVersion: admissionregistration.k8s.io/v1
kind: MutatingWebhookConfiguration
metadata:
name: istio-sidecar-injector
annotations:
# cert-manager can inject the CABundle automatically
cert-manager.io/inject-ca-from: istio-system/istio-webhook-cert
webhooks:
- name: sidecar-injector.istio.io
clientConfig:
service:
name: istiod
namespace: istio-system
path: /inject
port: 443
caBundle: <base64-CA> # API server uses this to verify webhook TLS
rules:
- operations: ["CREATE"]
apiGroups: [""]
apiVersions: ["v1"]
resources: ["pods"]
scope: "Namespaced"
namespaceSelector:
matchLabels:
istio-injection: enabled # only namespaces with this label
objectSelector:
matchExpressions:
- key: sidecar.istio.io/inject
operator: NotIn
values: ["false"]
failurePolicy: Fail # Fail = block CREATE if webhook unreachable
# Ignore = allow CREATE if webhook unreachable
sideEffects: None # required if dryRun requests should be honoured
timeoutSeconds: 10 # 1-30s, default 10s
admissionReviewVersions: ["v1", "v1beta1"]
reinvocationPolicy: IfNeeded
# IfNeeded: run again if another mutating webhook modified the object
# Never: run at most onceValidatingWebhookConfiguration
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
metadata:
name: kyverno-resource-validating-webhook-cfg
webhooks:
- name: validate.kyverno.svc
clientConfig:
service:
name: kyverno-svc
namespace: kyverno
path: /validate/fail
caBundle: <base64-CA>
rules:
- operations: ["CREATE", "UPDATE", "DELETE"]
apiGroups: ["*"]
apiVersions: ["*"]
resources: ["*/*"]
scope: "*"
namespaceSelector:
matchExpressions:
- key: kubernetes.io/metadata.name
operator: NotIn
values: ["kube-system", "kyverno"]
failurePolicy: Fail
matchPolicy: Equivalent # match old and new API versions
sideEffects: NoneOnDryRun
timeoutSeconds: 10
admissionReviewVersions: ["v1"]Webhook Failure Modes
failurePolicy: Fail (default for most security webhooks)
→ If webhook is unreachable, times out, or returns HTTP error:
API server BLOCKS the request (returns 500)
→ Safest for security enforcement — no bypass via webhook outage
→ Risk: if webhook crashes, all matching API calls fail
failurePolicy: Ignore
→ If webhook is unreachable or errors: API server ALLOWS the request
→ Used for non-critical mutation (add nice-to-have defaults)
→ Safer for cluster availability, weaker security guarantee
timeoutSeconds: 10 (default)
→ Webhook must respond within this window
→ Short timeout → fast fail
→ Long timeout → slow rejection experience for users
Recommendation:
Security-critical webhooks (OPA, Kyverno): Fail + PodDisruptionBudget + HPA
Mutation webhooks (Istio inject): Fail but ensure HA deployment
Non-critical defaults: IgnoreWebhook Performance
Webhook latency adds directly to kubectl apply response time.
Typical overhead:
Istio sidecar injector: 5-30ms
Kyverno policy check: 10-50ms
OPA Gatekeeper: 10-100ms
cert-manager CA inject: 1-5ms
Total with 5 webhooks: 50-200ms added to every CREATE/UPDATE
Optimization:
1. Narrow rules (namespaceSelector, objectSelector) — skip objects that don't need it
2. Run webhooks as in-cluster services (not external) — no inter-VPC latency
3. Set realistic timeoutSeconds (5s not 30s)
4. Monitor webhook latency: API server exposes histogram metrics
apiserver_admission_webhook_admission_duration_secondsDebugging Webhooks
# List all mutating and validating webhooks
kubectl get mutatingwebhookconfigurations
kubectl get validatingwebhookconfigurations
# Show which webhooks match a resource
kubectl get mutatingwebhookconfigurations -o json | \
jq '.items[].webhooks[] | select(.rules[].resources[] | contains("pods")) | .name'
# Dry-run to see if webhook blocks (without writing to etcd)
kubectl apply --dry-run=server -f pod.yaml
# Webhook rejection details are in the error from kubectl
kubectl apply -f pod.yaml
# Error: pods "payments-api" is forbidden:
# [kyverno-resource-validating-webhook-cfg.validate.kyverno.svc]
# resource requests not set
# API server logs show webhook calls
kubectl logs -n kube-system -l component=kube-apiserver --tail=200 | \
grep webhook
# Check webhook endpoint reachability from API server
kubectl run debug --image=curlimages/curl --rm -it -- \
curl -k https://kyverno-svc.kyverno.svc.cluster.local/validate/fail
# Webhook latency metrics
kubectl get --raw /metrics | grep admission_webhook_admission_duration
# Namespace bypass (skip webhooks entirely for break-glass)
# Label a namespace to bypass Kyverno:
kubectl label namespace break-glass kyverno.io/exclude=enabledChicken-and-Egg: Webhook Bootstrapping
Problem: cert-manager webhook validates cert-manager's own CRDs.
If cert-manager crashes, its webhook blocks cert-manager from restarting.
Solutions:
1. failurePolicy: Ignore for the cert-manager webhook itself
(accepted risk: invalid cert-manager resources slip through during outage)
2. Use --enable-certificate-owner-ref=true so webhook cert Secret
is owned by the Certificate — deleted and recreated on restart
3. Emergency: delete the webhook temporarily
kubectl delete mutatingwebhookconfigurations cert-manager-webhook
→ restart cert-manager
→ cert-manager recreates its webhook on startup
4. Use cainjector which injects CA bundle independently of webhook availabilityRelated
- 03 — Admission Flow — where webhooks sit in the full admission chain
- 02 — Admission Controllers — webhook-based policy enforcement
- 07 — Certificate Management — webhook TLS cert management