Control Plane — Complete Internal Architecture

What Is the Control Plane?

The control plane is the set of processes that implement the desired-state management loop of the cluster. It never runs your workloads; it only watches, decides, and instructs. Every cluster mutation flows through the control plane, and every node periodically reports status back to it.

The Five Core Components

Control Plane Architecture Diagram

Figure 1: Control plane component communication topology. All external and internal traffic routes through kube-apiserver. etcd is only accessible by apiserver. The scheduler and controllers only talk to the apiserver — never directly to nodes.

Communication Matrix

Understanding who talks to whom, over what protocol, on what port, authenticated how is critical for firewall rules, mTLS policy, and audit log interpretation.

Component Roles — Internal Mechanics

kube-apiserver — The API Gateway

The apiserver is the only component that reads from and writes to etcd. It is completely stateless: all state lives in etcd. Multiple apiserver replicas can run simultaneously because they share etcd as the source of truth.

Internal request lifecycle (full detail in 04-kubernetes-api-model.html §Request Lifecycle):

The apiserver also proxies requests to webhooks (MutatingAdmissionWebhook, ValidatingAdmissionWebhook) and to aggregated API servers (metrics-server, custom API servers via APIService). It handles Watch via long-lived HTTP/2 streaming connections — see the Informer architecture in 04-kubernetes-api-model.html §Informers.

Key apiserver flags

# Minimal production-relevant flags
kube-apiserver \
  --advertise-address=                  # IP used by kubelet to reach apiserver
  --bind-address=0.0.0.0                            # Listen on all interfaces
  --secure-port=6443                                # HTTPS port (default)
  --etcd-servers=https://etcd-0:2379,https://etcd-1:2379,https://etcd-2:2379
  --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt
  --etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt
  --etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key
  --client-ca-file=/etc/kubernetes/pki/ca.crt
  --tls-cert-file=/etc/kubernetes/pki/apiserver.crt
  --tls-private-key-file=/etc/kubernetes/pki/apiserver.key
  --authorization-mode=Node,RBAC
  --enable-admission-plugins=NodeRestriction,PodSecurity,MutatingAdmissionWebhook,ValidatingAdmissionWebhook
  --service-cluster-ip-range=10.96.0.0/12
  --service-node-port-range=30000-32767
  --allow-privileged=false
  --audit-log-path=/var/log/kubernetes/audit.log
  --audit-policy-file=/etc/kubernetes/audit-policy.yaml
  --feature-gates=...                               # feature flags
  --max-requests-inflight=800                       # concurrency limit
  --max-mutating-requests-inflight=400
  --request-timeout=60s

etcd — The Ground Truth

etcd is a distributed key-value store using the Raft consensus algorithm. It guarantees strong consistency — every read reflects the most recently committed write (given quorum). Kubernetes stores every API object as a protobuf-encoded value at a path like /registry/{group}/{resource}/{namespace}/{name}.

Raft requires a quorum of ⌊n/2⌋ + 1 members. A 3-member cluster tolerates 1 failure (quorum = 2). A 5-member cluster tolerates 2 failures (quorum = 3). Always run an odd number of members.

# Inspect etcd leader
ETCDCTL_API=3 etcdctl \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/peer.crt \
  --key=/etc/kubernetes/pki/etcd/peer.key \
  endpoint status --write-out=table

# Output columns: ENDPOINT, ID, VERSION, DB SIZE, IS LEADER, RAFT TERM, RAFT INDEX

kube-scheduler — Two-Phase Placement

The scheduler watches the apiserver for Pods in Pending state with spec.nodeName == "". For each such pod, it runs a two-phase algorithm:

The scheduler uses the Scheduling Framework (introduced v1.15, stable v1.19) — a plugin-based system where all logic is expressed as plugins implementing extension points: PreFilter, Filter, PostFilter, PreScore, Score, NormalizeScore, Reserve, Permit, PreBind, Bind, PostBind.

# Check scheduling failures
kubectl get events --field-selector reason=FailedScheduling
kubectl describe pod    # "Events" section shows scheduler decisions

# Enable verbose scheduler logs
kube-scheduler --v=10  # Logs filter/score results per node

# Inspect scheduler metrics
curl -sk https://localhost:10259/metrics | grep scheduler_

kube-controller-manager — The Reconciliation Engine

The kube-controller-manager runs 30+ controllers in a single binary with a single process. Each controller is an independent goroutine running a reconciliation loop:

// Generic reconciliation loop pattern (all controllers follow this)
for {
    desired := readDesiredState(apiserver)  // via Informer cache
    actual  := readActualState(apiserver)   // via Informer cache
    if desired != actual {
        makeChanges(apiserver)              // Create/Update/Delete child objects
    }
    // Sleep or wait for next Informer event
}

The key controllers and what they manage:

cloud-controller-manager — Cloud Integration

Introduced to decouple cloud-provider-specific logic from the core Kubernetes binary. Before its introduction (pre-v1.6), every cloud provider patched their logic directly into kube-controller-manager and kubelet, creating a monolithic and slow release cycle.

The CCM runs three cloud-specific controllers:

• Node controller: Annotates new Nodes with cloud metadata (instance type, zone, region, provider ID). Deletes Node objects when cloud instance is terminated.
• Route controller: Programs cloud routing tables so pod-to-pod traffic flows across nodes (GCP VPC native routing, AWS VPC CNI, etc.).
• Service controller: Watches Services of type: LoadBalancer. Creates/deletes cloud load balancers. Updates status.loadBalancer.ingress with the allocated IP/hostname.

High Availability Control Plane

Stacked vs External etcd Topology

apiserver HA — Stateless Horizontal Scale

Since the apiserver is stateless (all state in etcd), you can run any number of replicas. A TCP load balancer (or VIP via keepalived/haproxy) distributes client connections across all healthy apiserver instances. There is no leader election for apiservers — all replicas serve reads and writes concurrently.

# Verify HA apiserver (kubeadm)
kubectl get pods -n kube-system -l component=kube-apiserver

# Check which apiserver is serving your request
kubectl config view --minify -o jsonpath='{.clusters[0].cluster.server}'

# Each kubeconfig points to the VIP; inspect apiserver endpoints
kubectl get endpoints kubernetes -n default

Scheduler and Controller-Manager Leader Election

Unlike the apiserver, the scheduler and controller-manager are NOT safe to run as multiple active instances — two schedulers could assign the same pod to two nodes. They use Kubernetes Lease-based leader election:

The leader election mechanism:

1. Each replica tries to acquire a Lease object in the kube-system namespace.
2. The Lease has a leaseDurationSeconds (default 15s) and a renewDeadlineSeconds (default 10s).
3. The leader continuously renews the Lease by updating renewTime.
4. If the leader fails to renew within leaseDuration, another replica acquires the Lease and becomes leader.
5. Non-leaders sleep and periodically attempt to acquire the Lease.

# Check scheduler leader
kubectl get lease kube-scheduler -n kube-system -o yaml
# holderIdentity: kube-scheduler-node1_abc-uuid

# Check controller-manager leader
kubectl get lease kube-controller-manager -n kube-system -o yaml

# Watch for leadership changes
kubectl get lease -n kube-system --watch

# Controller-manager leader election flags
kube-controller-manager \
  --leader-elect=true \
  --leader-elect-lease-duration=15s \
  --leader-elect-renew-deadline=10s \
  --leader-elect-retry-period=2s

Static Pods — How Control Plane Components Run

On clusters provisioned by kubeadm, all control plane components (apiserver, etcd, scheduler, controller-manager) run as static pods. Static pods are managed directly by the kubelet on the node — not by the apiserver or any controller.

The kubelet's --pod-manifest-path (default: /etc/kubernetes/manifests/) contains YAML files. The kubelet watches this directory and creates/restarts pods whenever files are added, modified, or deleted.

# Static pod manifests location (kubeadm)
ls /etc/kubernetes/manifests/
# etcd.yaml
# kube-apiserver.yaml
# kube-controller-manager.yaml
# kube-scheduler.yaml

# Static pods appear as mirror pods in the API server
# (prefixed with node name, e.g. kube-apiserver-control-plane-1)
kubectl get pods -n kube-system

# Editing a static pod manifest immediately restarts the component
# (kubelet detects inotify change and recreates the pod)
vim /etc/kubernetes/manifests/kube-apiserver.yaml

Component Startup Order and Dependencies

PKI and Certificate Architecture

Every communication in the control plane uses TLS. The control plane PKI is a hierarchy of Certificate Authorities managed by kubeadm (or manually for more complex setups).

# View all cluster certificates and their expiry
kubeadm certs check-expiration

# Renew all certificates (kubeadm)
kubeadm certs renew all

# View cert details
openssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -text | grep -A 5 "Subject Alternative"
openssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -dates

# Check controller-manager cert
openssl x509 -in /etc/kubernetes/pki/apiserver-etcd-client.crt -noout -text | grep "Subject:"

Health Check Endpoints and Monitoring

Every control plane component exposes health check endpoints. These are polled by load balancers, monitoring systems, and readiness probes in the static pod manifests.

# Check apiserver health from within cluster
kubectl get --raw /healthz
kubectl get --raw /readyz
kubectl get --raw /livez
kubectl get --raw /readyz?verbose   # Shows each check's status

# Individual readiness checks
kubectl get --raw /readyz/poststarthook/rbac/bootstrap-roles
kubectl get --raw /readyz/etcd

# From node directly
curl -sk https://localhost:6443/healthz --cert /etc/kubernetes/pki/admin.crt --key /etc/kubernetes/pki/admin.key

# Check scheduler and controller-manager from CP node
curl -sk https://127.0.0.1:10259/healthz
curl -sk https://127.0.0.1:10257/healthz

# etcd health check
ETCDCTL_API=3 etcdctl endpoint health \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/healthcheck-client.crt \
  --key=/etc/kubernetes/pki/etcd/healthcheck-client.key

Critical Metrics to Monitor

Troubleshooting the Control Plane

apiserver Not Responding

# Step 1: Check if the static pod is running on CP node
ssh cp-node-1
crictl ps | grep kube-apiserver

# Step 2: Check kubelet logs (manages static pods)
journalctl -u kubelet --since "10 minutes ago" | grep apiserver

# Step 3: Check apiserver logs directly
kubectl logs -n kube-system kube-apiserver-cp-node-1
# Or directly via crictl:
crictl logs $(crictl ps --name kube-apiserver -q)

# Step 4: Check manifest for syntax errors
cat /etc/kubernetes/manifests/kube-apiserver.yaml | python3 -c "import sys,yaml;yaml.safe_load(sys.stdin)"

# Step 5: Verify etcd connectivity
ETCDCTL_API=3 etcdctl endpoint health ...
# If etcd is down → apiserver will not serve writes (reads from cache possible)

# Step 6: Check certificates
openssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -dates
kubeadm certs check-expiration

Scheduler Not Scheduling Pods

# Check if scheduler is running
kubectl get pods -n kube-system -l component=kube-scheduler

# Check leader election
kubectl get lease kube-scheduler -n kube-system

# Check scheduler logs for specific pod
kubectl logs -n kube-system kube-scheduler-cp-node-1 | grep "Failed to schedule"
kubectl logs -n kube-system kube-scheduler-cp-node-1 | grep 

# Get scheduling failure events
kubectl get events --field-selector reason=FailedScheduling --all-namespaces

# Describe the unscheduled pod
kubectl describe pod    # Look at "Events:" section

# Common causes:
# - Insufficient CPU/memory on all nodes
# - Taint with no toleration
# - NodeSelector/Affinity that matches no nodes
# - PVC cannot be bound (VolumeBinding plugin failure)

Controller Not Reconciling

# Check controller-manager status
kubectl get pods -n kube-system -l component=kube-controller-manager
kubectl logs -n kube-system kube-controller-manager-cp-node-1 --tail=100

# Check leader
kubectl get lease kube-controller-manager -n kube-system

# Check work queue metrics (if prometheus is available)
kubectl port-forward -n kube-system kube-controller-manager-cp-node-1 10257:10257
curl -sk https://localhost:10257/metrics | grep workqueue

# Check for throttling / rate limiting
kubectl logs ... | grep "Throttling request"

# Check RBAC permissions for specific controller SA
kubectl auth can-i list deployments --as=system:serviceaccount:kube-system:deployment-controller

etcd Issues

# etcd cannot reach quorum
# Symptom: apiserver returns 503 for writes, etcd logs show "failed to reach quorum"
ETCDCTL_API=3 etcdctl endpoint status --write-out=table ...

# etcd database size too large
ETCDCTL_API=3 etcdctl endpoint status ...  # check DB SIZE column
# Compact and defragment:
ETCDCTL_API=3 etcdctl compact $(etcdctl endpoint status --write-out=json | jq '.[0].Status.header.revision')
ETCDCTL_API=3 etcdctl defrag

# etcd slow writes / high fsync latency
# Check disk performance:
iostat -x 1 5
# etcd is sensitive to fsync latency; use SSDs; avoid NFS/network storage

# Check etcd alarms
ETCDCTL_API=3 etcdctl alarm list
ETCDCTL_API=3 etcdctl alarm disarm  # After resolving the root cause

Upgrading the Control Plane

Control plane upgrades must be done before upgrading worker nodes. The supported pattern is to upgrade one minor version at a time (e.g., 1.29 → 1.30, not 1.29 → 1.31).

# Using kubeadm (standard upgrade flow)

# 1. Upgrade kubeadm on first CP node
apt-get update && apt-get install -y kubeadm=1.30.0-1.1

# 2. Verify upgrade plan
kubeadm upgrade plan

# 3. Apply upgrade to first CP node (upgrades apiserver, scheduler, controller-manager, etcd)
kubeadm upgrade apply v1.30.0

# 4. Upgrade kubeadm on remaining CP nodes
# Then on each additional CP node:
kubeadm upgrade node

# 5. Upgrade kubelet and kubectl on CP nodes
apt-get install -y kubelet=1.30.0-1.1 kubectl=1.30.0-1.1
systemctl daemon-reload && systemctl restart kubelet

# 6. Then upgrade worker nodes (drain → upgrade kubelet → uncordon)

Production Control Plane Checklist

Dependency Graph and Next Files