kube-controller-manager

What kube-controller-manager Does

kube-controller-manager is a collection of independent control loops (controllers) compiled into one binary and launched as a single process. Each controller watches specific API objects and reconciles them toward their desired state. They share an in-process informer cache, so all 30+ controllers together make only a single set of watch streams to kube-apiserver.

The Reconciliation Loop Pattern

Every controller in kube-controller-manager follows the same fundamental observe → compare → act pattern. Understanding this pattern explains why Kubernetes converges to desired state even after partial failures.

Key invariants that make this work safely:

• Reads from informer cache (resourceVersion=0), not live API — avoids thundering-herd on kube-apiserver
• Writes go through kube-apiserver with resourceVersion for optimistic concurrency
• Status is written separately via /status subresource — controller owns status, user owns spec
• Work queue deduplication — rapid spec changes produce one reconcile, not many
• Exponential backoff on errors — failed reconciles retry with increasing delays

Shared Informer Architecture

All controllers inside kube-controller-manager share a single SharedInformerFactory. This means that even though 30+ controllers all need to watch Pods, there is only one Pod watch stream to kube-apiserver, and one copy of the Pod cache in memory. Each controller registers its own event handlers against the shared informer.

Complete Controller Inventory

As of Kubernetes 1.30, kube-controller-manager runs the following controllers. Each is a separate goroutine with its own work queue and informer event handlers.

Workload Controllers

Node Controllers

Storage Controllers

Network Controllers

Namespace, RBAC, and Security Controllers

Garbage Collection and Lifecycle Controllers

Deep Dive: Node Lifecycle Controller

The Node Lifecycle Controller is arguably the most operationally important controller — it determines when failed nodes trigger pod eviction and how quickly your cluster recovers from node failures.

Deep Dive: Garbage Collector

The Garbage Collector (GC) is the most architecturally complex controller. It maintains an in-memory owner-dependency graph (a DAG) of all API objects and propagates deletions through the graph.

# ownerReference chain: Deployment → ReplicaSet → Pod
kubectl get pod my-pod-abc12 -o jsonpath='{.metadata.ownerReferences}'
# [{apiVersion:apps/v1, kind:ReplicaSet, name:my-deploy-6799fc88d8, uid:..., controller:true}]

kubectl get rs my-deploy-6799fc88d8 -o jsonpath='{.metadata.ownerReferences}'
# [{apiVersion:apps/v1, kind:Deployment, name:my-deploy, uid:..., controller:true}]

# Delete deployment with background GC (default)
kubectl delete deployment my-deploy
# Deployment deleted immediately; RS and Pods deleted asynchronously by GC

# Delete with foreground GC (wait for dependents)
kubectl delete deployment my-deploy --cascade=foreground

# Orphan — delete deployment but keep ReplicaSets and Pods
kubectl delete deployment my-deploy --cascade=orphan

# Check if GC is processing (look for rapid object deletions)
kubectl get events --field-selector reason=SuccessfulDelete | head -20

# Detect pods with dangling ownerReferences
kubectl get pods -A -o json | jq -r '
  .items[] |
  select(.metadata.ownerReferences != null) |
  "\(.metadata.namespace)/\(.metadata.name): owner=\(.metadata.ownerReferences[0].name)"
' | head -20

Deep Dive: Deployment Controller

The Deployment controller orchestrates rolling updates by managing ReplicaSets. Understanding its algorithm is essential for debugging slow rollouts and unexpected pod terminations.

# Rolling update algorithm
# Given: maxSurge=1, maxUnavailable=1, replicas=3
#
# Initial state: RS-v1 has 3 pods (all Ready)
# After spec.template change:
#
# Step 1: Create RS-v2 with 0 replicas
# Step 2: Scale RS-v2 to 1 (maxSurge=1, total=4=3+1)
# Step 3: Wait for RS-v2 pod[0] to be Ready
# Step 4: Scale RS-v1 to 2 (maxUnavailable=1, so min Ready=2)
# Step 5: Scale RS-v2 to 2 → wait → scale RS-v1 to 1
# Step 6: Scale RS-v2 to 3 → wait → scale RS-v1 to 0
# RS-v1 kept at 0 replicas (not deleted) for rollback

kubectl rollout status deployment/my-deploy
kubectl rollout history deployment/my-deploy
kubectl rollout undo deployment/my-deploy          # roll back to previous RS
kubectl rollout undo deployment/my-deploy --to-revision=2  # roll back to specific revision

# Key status fields
kubectl get deploy my-deploy -o yaml | grep -A 30 "status:"
# status:
#   availableReplicas: 3     # Ready pods with minReadySeconds met
#   readyReplicas: 3          # pods passing readinessProbe
#   replicas: 3               # total pods (including not ready)
#   updatedReplicas: 3        # pods with current template
#   observedGeneration: 5     # last spec generation processed
#   conditions:
#   - type: Available          # availableReplicas >= minAvailable
#     status: "True"
#   - type: Progressing        # rollout is making progress
#     status: "True"
#   - type: ReplicaFailure     # pod creation failing
#     status: "False"

# Detect stalled rollout
kubectl get deploy my-deploy -o jsonpath='{.status.conditions[?(@.type=="Progressing")].message}'

Deep Dive: HPA Controller

The Horizontal Pod Autoscaler controller runs its own sync loop independently from the event-driven controllers. It polls metrics at a fixed interval and calculates desired replicas using a ratio formula.

desiredReplicas = ceil(currentReplicas * (currentMetricValue / desiredMetricValue))

# Example: target CPU = 50%, current CPU = 80%, current replicas = 3
# desiredReplicas = ceil(3 * (80 / 50)) = ceil(4.8) = 5

# Scale-up: immediate (no stabilization by default)
# Scale-down: 5-minute stabilization window (--horizontal-pod-autoscaler-downscale-stabilization)
# This prevents flapping on temporary load spikes

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: my-app
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: my-app
  minReplicas: 2
  maxReplicas: 20
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 60
  - type: Resource
    resource:
      name: memory
      target:
        type: AverageValue
        averageValue: 500Mi
  - type: Pods
    pods:
      metric:
        name: http_requests_per_second
      target:
        type: AverageValue
        averageValue: "100"
  behavior:
    scaleDown:
      stabilizationWindowSeconds: 300  # 5-minute cooldown on scale-down
      policies:
      - type: Percent
        value: 10
        periodSeconds: 60              # max 10% pods removed per minute
    scaleUp:
      stabilizationWindowSeconds: 0    # immediate scale-up
      policies:
      - type: Pods
        value: 4
        periodSeconds: 60              # max 4 pods added per minute

Configuration and Key Flags

# kube-controller-manager static pod (kubeadm)
spec:
  containers:
  - command:
    - kube-controller-manager
    - --allocate-node-cidrs=true
    - --cluster-cidr=10.244.0.0/16
    - --service-cluster-ip-range=10.96.0.0/12
    - --cluster-signing-cert-file=/etc/kubernetes/pki/ca.crt
    - --cluster-signing-key-file=/etc/kubernetes/pki/ca.key
    - --controllers=*,bootstrapsigner,tokencleaner  # * = all default, plus extras
    - --kubeconfig=/etc/kubernetes/controller-manager.conf
    - --leader-elect=true
    - --root-ca-file=/etc/kubernetes/pki/ca.crt
    - --service-account-private-key-file=/etc/kubernetes/pki/sa.key
    - --use-service-account-credentials=true
    - --node-monitor-grace-period=40s
    - --pod-eviction-timeout=5m0s
    - --node-eviction-rate=0.1
    - --concurrent-deployment-syncs=5
    - --concurrent-replicaset-syncs=5
    - --concurrent-gc-syncs=20

Leader Election

kube-controller-manager uses a Lease object in kube-system namespace for leader election. Only the leader runs the controllers; standby instances loop on lease renewal attempts.

# Check current leader
kubectl -n kube-system get lease kube-controller-manager -o yaml
# .spec.holderIdentity: "master-1_..."
# .spec.acquireTime: "2024-01-15T10:30:00Z"
# .spec.renewTime: "2024-01-15T11:45:30Z"   # updated every leaseDuration/3
# .spec.leaseDurationSeconds: 15

# Leader election logs
kubectl -n kube-system logs kube-controller-manager-master-1 | grep -i "leader"
# I0115 10:30:00 leaderelection.go:258] successfully acquired lease kube-system/kube-controller-manager

# Lease parameters (via flags):
# --leader-elect-lease-duration=15s    # time a non-leader waits to acquire
# --leader-elect-renew-deadline=10s    # time leader has to renew before losing
# --leader-elect-retry-period=2s       # retry period for acquiring/renewing

Metrics and Alerting

# Scrape metrics
curl -sk https://localhost:10257/metrics \
  --cert /etc/kubernetes/pki/controller-manager-client.crt \
  --key /etc/kubernetes/pki/controller-manager-client.key

# Key metrics
workqueue_depth{name="deployment"}                    # items in work queue
workqueue_adds_total{name="deployment"}               # total items added
workqueue_queue_duration_seconds{name="deployment"}   # time items wait in queue
workqueue_work_duration_seconds{name="deployment"}    # time to process each item
workqueue_retries_total{name="deployment"}            # retries due to errors

controller_runtime_reconcile_total{controller="deployment",result="success"}
controller_runtime_reconcile_total{controller="deployment",result="error"}
controller_runtime_reconcile_errors_total{controller="deployment"}

# Node lifecycle
node_collector_evictions_total{zone="us-east-1a"}     # pods evicted
node_collector_unhealthy_nodes_in_zone{zone="us-east-1a"}

# GC
garbage_collector_attempt_to_delete_queue_consumers   # GC worker count
garbage_collector_dirty_processing_latency_seconds

# HPA
horizontal_pod_autoscaler_metadata_generation          # HPA reconcile count

groups:
- name: kube-controller-manager
  rules:
  - alert: KubeControllerManagerDown
    expr: absent(up{job="kube-controller-manager"} == 1)
    for: 5m
    labels:
      severity: critical
    annotations:
      summary: "kube-controller-manager is down"
      description: "No kube-controller-manager instance is active. Self-healing is suspended."

  - alert: KubeControllerManagerHighWorkQueueDepth
    expr: workqueue_depth{job="kube-controller-manager",name=~"deployment|replicaset"} > 100
    for: 10m
    labels:
      severity: warning
    annotations:
      summary: "Controller work queue depth is high"
      description: "{{ $labels.name }} controller queue depth is {{ $value }}."

  - alert: KubeControllerManagerReconcileErrors
    expr: |
      rate(workqueue_retries_total{job="kube-controller-manager"}[5m]) > 1
    for: 10m
    labels:
      severity: warning
    annotations:
      summary: "High controller reconcile error rate"
      description: "Controller {{ $labels.name }} is retrying at {{ $value }}/s."

  - alert: KubeNodeEvictionHigh
    expr: rate(node_collector_evictions_total[5m]) > 0.5
    for: 5m
    labels:
      severity: warning
    annotations:
      summary: "High node eviction rate"
      description: "{{ $value }} pods/s evicted in zone {{ $labels.zone }}."

  - alert: KubeControllerManagerLeaderElectionLost
    expr: |
      increase(leader_election_master_status{job="kube-controller-manager"}[5m]) == 0
      and on(instance) leader_election_master_status == 0
    for: 5m
    labels:
      severity: critical
    annotations:
      summary: "No kube-controller-manager leader"

Troubleshooting

# Check if kube-controller-manager is running
kubectl -n kube-system get pod -l component=kube-controller-manager
kubectl -n kube-system logs kube-controller-manager-master-1 | tail -50

# Check work queue depth
kubectl get --raw /metrics | grep 'workqueue_depth{name="deployment"}'

# Check Deployment conditions
kubectl describe deploy my-deploy | grep -A 5 "Conditions:"

# Check ReplicaSet events
kubectl get events --field-selector involvedObject.kind=ReplicaSet

# Common causes:
# - kube-controller-manager in CrashLoopBackOff (check logs)
# - Resource quota exhausted (ResourceQuota controller rejecting pods)
# - Image pull secret missing (pod creation succeeds but pod stuck)
# - RBAC: controller SA missing permissions to create pods

# Check node conditions
kubectl describe node failed-node | grep -A 10 "Conditions:"

# Check node taints (NoExecute should be added)
kubectl get node failed-node -o jsonpath='{.spec.taints}'

# Check pod toleration seconds
kubectl get pod my-pod -o jsonpath='{.spec.tolerations}'

# Check eviction rate limit metrics
kubectl get --raw /metrics | grep node_collector_evictions_total

# Force-evict a pod (bypasses normal eviction flow)
kubectl delete pod my-pod --grace-period=0 --force

# If node-monitor-grace-period not met yet, wait or reduce it
# (changing requires restarting kube-controller-manager)

# Find what finalizers are blocking deletion
kubectl get <resource> <name> -o jsonpath='{.metadata.finalizers}'

# Find all objects with finalizers
kubectl get all -A -o json | jq -r '
  .items[] |
  select(.metadata.finalizers != null and (.metadata.finalizers | length) > 0) |
  "\(.kind) \(.metadata.namespace)/\(.metadata.name): \(.metadata.finalizers)"
'

# Emergency: manually remove a stuck finalizer (DANGEROUS — only if controller is dead)
kubectl patch <resource> <name> --type=json \
  -p '[{"op":"remove","path":"/metadata/finalizers"}]'

# Check if the relevant controller is running (e.g., pvc-protection requires running kcm)
kubectl -n kube-system logs kube-controller-manager-master-1 | grep -i "pvc-protection"

# Check GC work queue
kubectl get --raw /metrics | grep garbage_collector

# Check if GC is processing
kubectl -n kube-system logs kube-controller-manager-master-1 | grep -i "garbage"

# Find orphaned ReplicaSets (no ownerReference or owner missing)
kubectl get rs -A -o json | jq -r '
  .items[] |
  select(.metadata.ownerReferences == null or (.metadata.ownerReferences | length) == 0) |
  "\(.metadata.namespace)/\(.metadata.name)"
'

# Check GC concurrent syncs (increase if cluster has many objects)
# --concurrent-gc-syncs=20 (default); increase to 50+ for very large clusters

# Check HPA status
kubectl describe hpa my-app
# Look for: "unable to get metrics", "failed to get CPU utilization"
# Conditions section shows why scaling is blocked

# Check metrics-server is running and healthy
kubectl top pods -n kube-system
kubectl -n kube-system get pods -l k8s-app=metrics-server

# Check if resource requests are set (required for CPU% metric)
kubectl get pod my-pod -o jsonpath='{.spec.containers[0].resources.requests}'

# Check HPA events
kubectl get events --field-selector involvedObject.kind=HorizontalPodAutoscaler

# Check if min/max bounds are blocking
kubectl get hpa my-app -o jsonpath='{.spec.minReplicas},{.spec.maxReplicas},{.status.currentReplicas}'

Production Best Practices