Admission Controllers

Admission controllers are plugins that intercept API requests to the kube-apiserver after authentication and authorization but before objects are persisted to etcd. They sit in the final two stages of the seven-stage request pipeline (see kube-apiserver §Request Pipeline) and are the last defense against misconfigured or policy-violating objects entering the cluster.

Where Admission Sits in the Request Pipeline

Two distinct phases run in order:

1. Mutating admission — plugins may modify the object (add defaults, inject sidecars, set labels). All mutating plugins run, then the mutated object is re-validated against the OpenAPI schema.
2. Validating admission — plugins may only approve or reject; they cannot modify. If any validating plugin rejects, the entire request fails.

Built-in Admission Plugins

Enable plugins with --enable-admission-plugins and disable with --disable-admission-plugins on kube-apiserver. The default set (Kubernetes 1.29+) includes most of the plugins below.

Dynamic Admission: Webhooks

The two dynamic webhook plugins — MutatingAdmissionWebhook and ValidatingAdmissionWebhook — allow cluster operators to extend admission without modifying kube-apiserver. The webhook configurations are themselves Kubernetes API objects that can be applied, updated, or deleted without restarting the API server.

MutatingWebhookConfiguration

apiVersion: admissionregistration.k8s.io/v1
kind: MutatingWebhookConfiguration
metadata:
  name: my-mutating-webhook
webhooks:
  - name: pod-defaults.example.com          # Must be fully qualified domain; used as sort key
    admissionReviewVersions: ["v1", "v1beta1"]
    clientConfig:
      service:                              # Webhook runs as an in-cluster Service
        namespace: webhook-system
        name: my-webhook-svc
        port: 443
        path: /mutate
      caBundle:          # CA that signed the webhook's TLS cert
    rules:
      - apiGroups:   [""]
        apiVersions: ["v1"]
        resources:   ["pods"]
        operations:  ["CREATE", "UPDATE"]
        scope:       "Namespaced"           # Namespaced | Cluster | *
    namespaceSelector:                      # Only apply to matching namespaces
      matchExpressions:
        - key: kubernetes.io/metadata.name
          operator: NotIn
          values: ["kube-system", "webhook-system"]
    objectSelector:                         # Only apply to matching objects
      matchLabels:
        inject-sidecar: "true"
    failurePolicy: Fail                     # Fail | Ignore
    sideEffects: None                       # None | NoneOnDryRun | Some | Unknown
    timeoutSeconds: 10                      # 1–30, default 10
    reinvocationPolicy: IfNeeded           # Never | IfNeeded
    matchPolicy: Equivalent                # Exact | Equivalent (convert to canonical version before matching)

ValidatingWebhookConfiguration

apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
metadata:
  name: my-validating-webhook
webhooks:
  - name: pod-policy.example.com
    admissionReviewVersions: ["v1"]
    clientConfig:
      service:
        namespace: opa-system
        name: opa
        port: 443
        path: /v1/admit
      caBundle: 
    rules:
      - apiGroups:   ["apps"]
        apiVersions: ["v1"]
        resources:   ["deployments"]
        operations:  ["CREATE", "UPDATE"]
    namespaceSelector:
      matchExpressions:
        - key: policy-enforced
          operator: In
          values: ["true"]
    failurePolicy: Fail
    sideEffects: None
    timeoutSeconds: 15

AdmissionReview Payload

The API server serializes the object into an AdmissionReview request, posts it to the webhook's HTTPS endpoint, and reads the response.

{
  "apiVersion": "admission.k8s.io/v1",
  "kind": "AdmissionReview",
  "request": {
    "uid": "705ab4f5-6393-11e8-b7cc-42010a800002",
    "kind": {"group":"","version":"v1","kind":"Pod"},
    "resource": {"group":"","version":"v1","resource":"pods"},
    "subResource": "",
    "requestKind": {"group":"","version":"v1","kind":"Pod"},
    "requestResource": {"group":"","version":"v1","resource":"pods"},
    "name": "my-pod",
    "namespace": "default",
    "operation": "CREATE",
    "userInfo": {
      "username": "alice",
      "groups": ["system:authenticated"]
    },
    "object": { ... },        // new object (CREATE/UPDATE)
    "oldObject": { ... },     // old object (UPDATE/DELETE)
    "dryRun": false,
    "options": { ... }
  }
}

// Mutating: allow + patch
{
  "apiVersion": "admission.k8s.io/v1",
  "kind": "AdmissionReview",
  "response": {
    "uid": "705ab4f5-...",   // MUST echo request uid
    "allowed": true,
    "patchType": "JSONPatch",
    "patch": "W3sib3AiOiJhZGQiLCJwYXRoIjoiL..."
    // base64-encoded JSON Patch array:
    // [{"op":"add","path":"/spec/containers/-",
    //   "value":{...sidecar...}}]
  }
}

// Validating: reject
{
  "response": {
    "uid": "705ab4f5-...",
    "allowed": false,
    "status": {
      "code": 403,
      "message": "image must be from approved registry"
    }
  }
}

Failure Policy

sideEffects Field

The sideEffects field tells the API server whether the webhook makes external changes (writes to databases, creates resources, calls external APIs). This matters for dry-run requests.

If sideEffects is Some or Unknown, the API server will NOT call the webhook for dry-run requests, which means kubectl apply --dry-run=server may give incorrect results for that webhook's rules.

Reinvocation Policy

Mutating webhooks run concurrently, so webhook B may not see the changes made by webhook A. The reinvocationPolicy: IfNeeded field instructs the API server to re-run all mutating webhooks a second time if any webhook modified the object in the first pass. Built-in mutating plugins also participate in reinvocation.

matchPolicy

CRDs and API groups may have multiple versions. matchPolicy: Equivalent means the API server will call your webhook even when the request arrives in a different API version than what your rule specifies, as long as the versions are recognized as equivalent (i.e., they share a common storage version). Exact only matches the literal API version in the rule.

namespaceSelector and objectSelector

These two fields filter which requests reach the webhook, reducing load and avoiding recursive self-calls:

# Exempt kube-system and your own webhook namespace
namespaceSelector:
  matchExpressions:
    - key: kubernetes.io/metadata.name
      operator: NotIn
      values: ["kube-system", "kube-public", "cert-manager", "webhook-system"]

# Only call webhook for pods with the opt-in label
objectSelector:
  matchLabels:
    sidecar.istio.io/inject: "true"

ValidatingAdmissionPolicy (CEL-based, GA 1.30)

ValidatingAdmissionPolicy (VAP) provides in-process policy evaluation using Common Expression Language (CEL) — no external webhook call, no TLS, no network round-trip. Cross-reference: API Model §CEL in CRDs and ValidatingAdmissionPolicy.

Three-Object Model

apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicy
metadata:
  name: require-resource-limits
spec:
  failurePolicy: Fail
  matchConstraints:
    resourceRules:
      - apiGroups:   [""]
        apiVersions: ["v1"]
        resources:   ["pods"]
        operations:  ["CREATE", "UPDATE"]
  variables:
    - name: containers
      expression: "object.spec.containers"
  validations:
    - expression: >
        variables.containers.all(c,
          has(c.resources) &&
          has(c.resources.limits) &&
          has(c.resources.limits.cpu) &&
          has(c.resources.limits.memory)
        )
      message: "All containers must have CPU and memory limits"
      reason: Invalid
    - expression: >
        variables.containers.all(c,
          has(c.resources) &&
          has(c.resources.requests) &&
          has(c.resources.requests.cpu)
        )
      message: "All containers must have CPU requests"
  auditAnnotations:
    - key: "require-limits-violation"
      valueExpression: >
        "Pod " + object.metadata.name + " missing resource limits"

---
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicyBinding
metadata:
  name: require-resource-limits-binding
spec:
  policyName: require-resource-limits
  validationActions: [Deny]            # Deny | Warn | Audit
  matchResources:
    namespaceSelector:
      matchLabels:
        enforce-limits: "true"

validationActions

CEL Expression Context Variables

PodSecurity Admission

The PodSecurity admission plugin (GA 1.25, replaces deprecated PodSecurityPolicy) enforces the Pod Security Standards via namespace labels. No webhook needed — it is built into kube-apiserver as a built-in plugin.

Pod Security Standard Levels

Namespace Label Configuration

apiVersion: v1
kind: Namespace
metadata:
  name: my-app
  labels:
    # Format: pod-security.kubernetes.io/: 
    # MODE: enforce | audit | warn
    # LEVEL: privileged | baseline | restricted
    # VERSION: latest | v1.29 (pin to specific version)
    pod-security.kubernetes.io/enforce: restricted
    pod-security.kubernetes.io/enforce-version: v1.29
    pod-security.kubernetes.io/audit: restricted
    pod-security.kubernetes.io/audit-version: v1.29
    pod-security.kubernetes.io/warn: restricted
    pod-security.kubernetes.io/warn-version: v1.29

Exemptions

The PodSecurity plugin supports exemptions configured in --admission-control-config-file:

apiVersion: apiserver.config.k8s.io/v1
kind: AdmissionConfiguration
plugins:
  - name: PodSecurity
    configuration:
      apiVersion: pod-security.admission.config.k8s.io/v1
      kind: PodSecurityConfiguration
      defaults:
        enforce: "baseline"
        enforce-version: "latest"
        audit: "restricted"
        warn: "restricted"
      exemptions:
        usernames: ["system:serviceaccount:kube-system:replicaset-controller"]
        runtimeClasses: []
        namespaces: ["kube-system", "kube-public", "monitoring"]

NodeRestriction Plugin

The NodeRestriction plugin is a critical security boundary. When a kubelet authenticates as system:node:<nodeName> (group system:nodes), this plugin limits what that credential can do:

• May only modify its own Node object (not other nodes)
• May only modify Pods bound to itself (not pods on other nodes)
• Cannot set arbitrary labels — only labels with a specific prefix (node-restriction.kubernetes.io/) plus a limited set of well-known labels
• Cannot modify taints on other nodes
• Cannot read Secrets or ConfigMaps not referenced by pods on its node (enforced by the Node authorizer, which works alongside this plugin)

ResourceQuota Admission

The ResourceQuota admission plugin works in conjunction with the ResourceQuota controller (see kube-controller-manager §Controllers). The plugin:

1. Lists all ResourceQuota objects in the request's namespace
2. For each quota that covers the requested resource, reads the current usage from the quota's status
3. Adds the request's resource delta to the current usage
4. Rejects the request if usage would exceed the hard limit
5. After etcd persist, the quota controller updates the actual usage in the status

Webhook TLS and Certificate Management

All webhook endpoints must be HTTPS. The caBundle field in the webhook configuration holds the PEM-encoded CA that signed the webhook server's TLS certificate. Common patterns for managing this:

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: webhook-tls
  namespace: webhook-system
spec:
  secretName: webhook-tls-secret
  dnsNames:
    - my-webhook-svc.webhook-system.svc
    - my-webhook-svc.webhook-system.svc.cluster.local
  issuerRef:
    name: cluster-ca-issuer
    kind: ClusterIssuer

# Init container generates TLS cert
openssl req -x509 -newkey rsa:4096 \
  -keyout /tls/tls.key -out /tls/tls.crt \
  -days 365 -nodes \
  -subj "/CN=my-webhook.webhook-system.svc"

# Patch webhook config with CA
kubectl patch mutatingwebhookconfigurations \
  my-webhook -p "{\"webhooks\":[{\"name\":
  \"my-webhook.example.com\",
  \"clientConfig\":{\"caBundle\":
  \"$(base64 -w0 /tls/tls.crt)\"}}]}"

Developing a Webhook

Go Webhook Server Pattern

package main

import (
    "encoding/json"
    "fmt"
    "net/http"

    admissionv1 "k8s.io/api/admission/v1"
    corev1 "k8s.io/api/core/v1"
    metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)

func mutatePods(w http.ResponseWriter, r *http.Request) {
    var review admissionv1.AdmissionReview
    if err := json.NewDecoder(r.Body).Decode(&review); err != nil {
        http.Error(w, err.Error(), http.StatusBadRequest)
        return
    }

    var pod corev1.Pod
    json.Unmarshal(review.Request.Object.Raw, &pod)

    // Build JSON Patch
    patch := []map[string]interface{}{}
    if _, ok := pod.Labels["app"]; !ok {
        patch = append(patch, map[string]interface{}{
            "op":    "add",
            "path":  "/metadata/labels/app",
            "value": pod.Name,
        })
    }
    patchBytes, _ := json.Marshal(patch)
    patchType := admissionv1.PatchTypeJSONPatch

    response := &admissionv1.AdmissionReview{
        TypeMeta: metav1.TypeMeta{
            APIVersion: "admission.k8s.io/v1",
            Kind:       "AdmissionReview",
        },
        Response: &admissionv1.AdmissionResponse{
            UID:       review.Request.UID,
            Allowed:   true,
            PatchType: &patchType,
            Patch:     patchBytes,
        },
    }

    w.Header().Set("Content-Type", "application/json")
    json.NewEncoder(w).Encode(response)
}

func main() {
    mux := http.NewServeMux()
    mux.HandleFunc("/mutate", mutatePods)
    mux.HandleFunc("/healthz", func(w http.ResponseWriter, r *http.Request) {
        fmt.Fprintln(w, "ok")
    })
    http.ListenAndServeTLS(":8443", "/tls/tls.crt", "/tls/tls.key", mux)
}

controller-runtime Webhook Pattern

The controller-runtime library (used by kubebuilder and operator-sdk) provides a higher-level webhook framework:

// Implement the webhook.Defaulter interface for mutating
type PodDefaulter struct{}

func (d *PodDefaulter) Default(ctx context.Context, obj runtime.Object) error {
    pod := obj.(*corev1.Pod)
    if pod.Labels == nil {
        pod.Labels = map[string]string{}
    }
    if _, ok := pod.Labels["app"]; !ok {
        pod.Labels["app"] = pod.Name
    }
    return nil
}

// Implement webhook.Validator for validating
type PodValidator struct{}

func (v *PodValidator) ValidateCreate(ctx context.Context, obj runtime.Object) (admission.Warnings, error) {
    pod := obj.(*corev1.Pod)
    for _, c := range pod.Spec.Containers {
        if c.Resources.Limits == nil {
            return nil, fmt.Errorf("container %q has no resource limits", c.Name)
        }
    }
    return nil, nil
}

// Register in main
mgr.GetWebhookServer().Register("/mutate-v1-pod", &webhook.Admission{
    Handler: admission.DefaultingWebhookFor(scheme, &PodDefaulter{}),
})

ImagePolicyWebhook

The ImagePolicyWebhook plugin is distinct from ValidatingAdmissionWebhook. It uses a separate configuration file (--admission-control-config-file) and a different ImageReview API rather than AdmissionReview. It is specifically designed for image admission decisions and predates the more general webhook framework.

# /etc/kubernetes/admission-config.yaml
apiVersion: apiserver.config.k8s.io/v1
kind: AdmissionConfiguration
plugins:
  - name: ImagePolicyWebhook
    configuration:
      imagePolicy:
        kubeConfigFile: /etc/kubernetes/image-policy-webhook.kubeconfig
        allowTTL: 50
        denyTTL: 50
        retryBackoff: 500
        defaultAllow: false   # fail closed if webhook unavailable

Most modern deployments use ValidatingAdmissionWebhook with OPA Gatekeeper or Kyverno for image policy instead, as those have richer policy models and are easier to operate.

Policy Engines: OPA Gatekeeper vs Kyverno

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: require-non-root
spec:
  validationFailureAction: Enforce   # Enforce | Audit
  background: true                   # also audit existing resources
  rules:
    - name: check-runAsNonRoot
      match:
        any:
          - resources:
              kinds: [Pod]
      validate:
        message: "Pods must run as non-root"
        pattern:
          spec:
            =(initContainers):
              - =(securityContext):
                  runAsNonRoot: true
            containers:
              - securityContext:
                  runAsNonRoot: true

apiVersion: templates.gatekeeper.sh/v1
kind: ConstraintTemplate
metadata:
  name: k8sallowedrepos
spec:
  crd:
    spec:
      names:
        kind: K8sAllowedRepos
      validation:
        openAPIV3Schema:
          type: object
          properties:
            repos:
              type: array
              items:
                type: string
  targets:
    - target: admission.k8s.gatekeeper.sh
      rego: |
        package k8sallowedrepos
        violation[{"msg": msg}] {
          container := input.review.object.spec.containers[_]
          satisfied := [good | repo := input.parameters.repos[_]
                         good := startswith(container.image, repo)]
          not any(satisfied)
          msg := sprintf("container <%v> has unapproved image <%v>", [container.name, container.image])
        }
---
apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sAllowedRepos
metadata:
  name: allowed-repos
spec:
  enforcementAction: deny
  match:
    kinds:
      - apiGroups: [""]
        kinds: ["Pod"]
  parameters:
    repos:
      - "gcr.io/my-org/"
      - "registry.k8s.io/"

kube-apiserver Admission Configuration

Key Flags

AdmissionConfiguration File

apiVersion: apiserver.config.k8s.io/v1
kind: AdmissionConfiguration
plugins:
  - name: EventRateLimit
    path: /etc/kubernetes/event-rate-limit.yaml
  - name: PodSecurity
    configuration:
      apiVersion: pod-security.admission.config.k8s.io/v1
      kind: PodSecurityConfiguration
      defaults:
        enforce: "baseline"
        enforce-version: "latest"
        warn: "restricted"
        warn-version: "latest"
      exemptions:
        namespaces: ["kube-system"]

Webhook Performance Considerations

Admission webhooks add latency to every write request. In high-throughput clusters, poorly configured webhooks are a common cause of API server performance degradation.

Debugging Webhook Issues

Check which webhooks are registered

kubectl get mutatingwebhookconfigurations -o wide
kubectl get validatingwebhookconfigurations -o wide

# Describe to see rules and selectors
kubectl describe mutatingwebhookconfigurations my-webhook

Admission error in kubectl output

Error from server: error when creating "pod.yaml":
  admission webhook "pod-defaults.example.com" denied the request:
  image must be from approved registry

# Trace the rejection
kubectl apply -f pod.yaml -v=8 2>&1 | grep -E "Response|webhook"

Webhook timing out

# Check webhook pod logs
kubectl logs -n webhook-system deploy/my-webhook --tail=50

# Check webhook endpoint reachability from apiserver
# (useful if apiserver is on control plane node)
curl -k https://my-webhook-svc.webhook-system.svc.cluster.local:443/healthz

# Check apiserver audit log for admission errors
# (requires audit logging enabled with level >= Metadata)
grep "admission webhook" /var/log/kubernetes/audit.log | tail -20

Prometheus Metrics

Alerting Rules

groups:
  - name: admission-webhook
    rules:
      - alert: AdmissionWebhookHighLatency
        expr: |
          histogram_quantile(0.99,
            rate(apiserver_admission_webhook_admission_duration_seconds_bucket[5m])
          ) > 5
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Admission webhook p99 latency {{ $labels.name }} > 5s"

      - alert: AdmissionWebhookRejectionSpike
        expr: |
          rate(apiserver_admission_webhook_request_total{rejected="true"}[5m]) > 10
        for: 2m
        labels:
          severity: warning
        annotations:
          summary: "Webhook {{ $labels.name }} rejecting > 10 req/s"

      - alert: AdmissionWebhookFailOpen
        expr: |
          rate(apiserver_admission_webhook_fail_open_count[5m]) > 0
        for: 1m
        labels:
          severity: warning
        annotations:
          summary: "Webhook {{ $labels.name }} failing open (Ignore policy triggered)"

      - alert: AdmissionWebhookUnavailable
        expr: |
          rate(apiserver_admission_webhook_request_total{code=~"5.."}[5m]) > 1
        for: 3m
        labels:
          severity: critical
        annotations:
          summary: "Webhook {{ $labels.name }} returning 5xx errors"

Troubleshooting Runbooks

# 1. Identify which webhook rejected the request
kubectl apply -f resource.yaml 2>&1
# Output: admission webhook "webhook-name.example.com" denied the request: ...

# 2. Find the webhook config
kubectl get mutatingwebhookconfigurations,validatingwebhookconfigurations \
  | grep webhook-name

# 3. Inspect the rules
kubectl describe validatingwebhookconfigurations webhook-name

# 4. Check if webhook pod is running
kubectl get pods -A | grep webhook

# 5. Read webhook server logs
kubectl logs -n webhook-system deploy/webhook-server --tail=100

# 6. Temporarily change to warn/ignore for diagnosis
kubectl patch validatingwebhookconfigurations webhook-name \
  --type=json \
  -p='[{"op":"replace","path":"/webhooks/0/failurePolicy","value":"Ignore"}]'

# Symptom: webhook pod is in Pending/Failed, and all new pods fail admission

# 1. Confirm the deadlock
kubectl get pods -n webhook-system
# Pod is Pending or in ImagePullBackOff

kubectl describe pod new-test-pod 2>&1 | grep "admission webhook"
# Error references the webhook that's down

# 2. Check namespaceSelector — is the webhook exempting itself?
kubectl get mutatingwebhookconfigurations my-webhook -o yaml \
  | grep -A 10 namespaceSelector

# 3. Emergency: temporarily disable the problematic webhook
kubectl delete mutatingwebhookconfigurations my-webhook
# OR patch failurePolicy to Ignore
kubectl patch mutatingwebhookconfigurations my-webhook \
  --type=json \
  -p='[{"op":"replace","path":"/webhooks/0/failurePolicy","value":"Ignore"}]'

# 4. Let webhook pod start, then re-enable
# 5. Fix webhook config to exempt its own namespace

# Symptom: pods rejected with "violates PodSecurity policy"

# 1. Check namespace labels
kubectl get namespace my-app --show-labels

# 2. Test in warn mode first (without changing enforce)
kubectl label namespace my-app \
  pod-security.kubernetes.io/warn=restricted \
  pod-security.kubernetes.io/warn-version=v1.29
kubectl apply -f pod.yaml
# Look for Warning headers in output

# 3. Identify the specific violations
# The rejection message lists which fields violate which standard
# Error: "restricted" policy check failed: containers must not set
#   securityContext.allowPrivilegeEscalation = true

# 4. Fix the pod spec
# Add to container:
# securityContext:
#   allowPrivilegeEscalation: false
#   runAsNonRoot: true
#   seccompProfile:
#     type: RuntimeDefault
#   capabilities:
#     drop: ["ALL"]

# 5. Or use a less restrictive level for this namespace
kubectl label namespace my-app \
  pod-security.kubernetes.io/enforce=baseline --overwrite

# Symptom: "exceeded quota: team-quota, requested: cpu=500m,
#   used: cpu=1900m, limited: cpu=2"

# 1. Check quota usage
kubectl describe resourcequota -n my-namespace

# 2. Find what's consuming quota
kubectl top pods -n my-namespace
kubectl get pods -n my-namespace -o custom-columns=\
  "NAME:.metadata.name,CPU-REQ:.spec.containers[0].resources.requests.cpu"

# 3. Delete unused resources
kubectl delete pod old-debug-pod -n my-namespace

# 4. Or increase the quota (requires cluster admin)
kubectl edit resourcequota team-quota -n my-namespace

# 5. Check for pods stuck Terminating (consuming quota)
kubectl get pods -n my-namespace | grep Terminating
kubectl delete pod stuck-pod -n my-namespace --force --grace-period=0

# Symptom: Error with "ValidatingAdmissionPolicy" in the message

# 1. List active policies and bindings
kubectl get validatingadmissionpolicies
kubectl get validatingadmissionpolicybindings

# 2. Check which policy triggered
kubectl describe validatingadmissionpolicybinding require-limits-binding

# 3. Test a policy manually with dry-run
kubectl apply -f pod.yaml --dry-run=server

# 4. Audit mode — see violations without enforcing
kubectl patch validatingadmissionpolicybinding require-limits-binding \
  --type=merge \
  -p '{"spec":{"validationActions":["Audit"]}}'

# 5. Check audit annotations
kubectl get events -n my-namespace | grep "ValidatingAdmissionPolicy"

# 6. Read policy CEL expressions
kubectl get validatingadmissionpolicies require-resource-limits -o yaml \
  | grep -A 5 expression

Production Best Practices

1. Exempt webhook namespaces from their own webhooks. Use namespaceSelector to exclude the namespace where webhook pods run. This prevents bootstrapping deadlocks.
2. Use failurePolicy: Fail only for security-critical webhooks. All other webhooks should use Ignore or have strong HA guarantees (≥3 replicas, PDB, proper readiness probe).
3. Set sideEffects: None on all webhooks that don't call external systems. This enables correct dry-run behavior and is required in some Kubernetes versions.
4. Set narrow namespaceSelector and objectSelector. Broad rules that match all objects add latency to every write. Use label-based opt-in where possible.
5. Keep webhook latency under 100ms p99. The API server's default request timeout is 60s, but users expect fast feedback. Use informers to cache state inside the webhook server rather than calling the API on each request.
6. Use reinvocationPolicy: IfNeeded for mutating webhooks that need to see the results of other mutating webhooks (e.g., sidecar injectors that depend on labels set by another webhook).
7. Prefer ValidatingAdmissionPolicy (CEL) over webhooks for pure validation. VAP has zero network overhead, survives webhook outages, and is auditable via API objects. Use webhooks only for mutations or complex multi-step validation requiring external data.
8. Rotate webhook TLS certificates before expiry. Use cert-manager with short-lived certificates (90 days) and automatic rotation. An expired cert causes all matched requests to fail.
9. Test webhooks with --dry-run=server in CI. This catches admission failures before production deployments.
10. Monitor apiserver_admission_webhook_admission_duration_seconds. Alert at p99 > 1s for individual webhooks. A slow webhook is invisible to users until it becomes a complete timeout.