Cluster Architecture Overview

This document is the architectural map of a Kubernetes cluster — every component, its role, how it communicates with other components, what ports it uses, and how the whole system maintains consistency under failures. Read this before diving into any individual component's deep-dive.

1 · The Two Planes

A Kubernetes cluster is logically split into two planes:

• Control Plane — makes decisions (scheduling, reconciliation, admission), stores state. Brains of the cluster.
• Data Plane (Worker Nodes) — executes workloads. Runs the actual Pods that serve traffic.

In a production cluster these planes run on separate physical/virtual machines. The control plane should never run user workloads — use NoSchedule taints.

2 · Control Plane Components In Depth

2.1 kube-apiserver

The API server is the only stateful, central component that all other components talk to. It is the single source of truth gateway — no component talks to another component directly (exception: kubelet can be called back by the API server for exec/logs/port-forward, and etcd is called only by the API server).

2.2 etcd

etcd is the cluster's brain memory. Every Kubernetes object (Pod spec, Deployment state, Secret value, Node status, Service endpoint) is persisted as a key-value entry in etcd. The kube-apiserver is the only component that reads from or writes to etcd directly.

2.3 kube-scheduler

The scheduler watches for Pods with spec.nodeName == "" (unscheduled) and writes a nodeName into the Pod spec via a binding API call. It runs a filter → score → bind pipeline for every Pod.

2.4 kube-controller-manager

A single binary that embeds 30+ individual controllers, each running its own reconciliation loop. All controllers use the same watch-and-reconcile pattern.

2.5 cloud-controller-manager

Separates cloud-provider-specific logic from the core Kubernetes code. Manages: cloud load balancers (LoadBalancer type Services), cloud node metadata (instance types, zones), and cloud routes (for overlay-less networking on GCE/AWS).

3 · Worker Node Components In Depth

3.1 kubelet

The kubelet is the primary node agent. It is the bridge between the Kubernetes API and the container runtime. It watches the API server for Pods assigned to its node, drives the CRI to create/destroy containers, runs probes, and reports status back.

3.2 kube-proxy

kube-proxy watches Service and EndpointSlice objects and programs the host's networking stack (iptables rules, IPVS virtual servers, or nftables) to implement the Service virtual IP abstraction.

3.3 Container Runtime (CRI)

The CRI-compliant runtime (containerd or CRI-O) receives instructions from the kubelet via gRPC, pulls images, manages container lifecycle, and interacts with the OCI low-level runtime (runc).

3.4 CNI Plugin

The CNI plugin (Calico, Cilium, Flannel, Weave, etc.) is invoked by the kubelet via the CRI when a Pod sandbox is created. It assigns a Pod IP, creates the veth pair, and programs routes. CNI is not a running daemon — it is a set of binary executables in /opt/cni/bin/ called by the runtime.

4 · Component Communication Matrix

A critical production security concept: understanding exactly which components talk to which, over which port, authenticated with which certificate.

5 · Complete Port Reference

6 · High Availability Architecture

6.1 Stacked vs. External etcd

There are two HA topologies for the control plane:

6.2 Quorum Math and Failure Tolerance

6.3 Controller/Scheduler Leader Election

While the API server is active-active (all replicas serve traffic), the scheduler and controller-manager use leader election to ensure only one instance makes decisions at a time. Leader election is implemented using a coordination.k8s.io/v1/Lease object in the kube-system namespace.

# Inspect leader election leases
kubectl get leases -n kube-system
# NAME                      HOLDER                    AGE
# kube-controller-manager   node1.cluster.internal    45d
# kube-scheduler            node1.cluster.internal    45d

# Who currently holds the scheduler lease?
kubectl get lease kube-scheduler -n kube-system -o jsonpath='{.spec.holderIdentity}'

# Leader election config (in controller-manager flags)
# --leader-elect=true (default)
# --leader-elect-lease-duration=15s  (time a lease is valid)
# --leader-elect-renew-deadline=10s  (how long leader retries to renew)
# --leader-elect-retry-period=2s    (how often non-leader tries to acquire)

7 · Network Topology

A Kubernetes cluster uses four distinct IP address spaces:

7.1 Kubernetes Networking Requirements

The Kubernetes networking model mandates:

1. Every Pod gets a unique, cluster-routable IP (no NAT for Pod-to-Pod traffic within the cluster)
2. Every Pod can reach every other Pod by IP, regardless of which node they are on
3. Every node can reach every Pod IP
4. Pods see their own IP as their IP (no masquerading for inbound)

CNI plugins are responsible for implementing these rules. They achieve this via: VXLAN overlays (Flannel), BGP routing (Calico), eBPF dataplane (Cilium), etc.

8 · Failure Domains and Topology

8.1 Multi-AZ Worker Node Distribution

In cloud deployments, worker nodes should be spread across at least 3 availability zones. Kubernetes provides topology-aware scheduling to enforce this.

# Spread Pods evenly across AZs (topologySpreadConstraints)
spec:
  topologySpreadConstraints:
  - maxSkew: 1                                  # Max imbalance between zones
    topologyKey: topology.kubernetes.io/zone   # Use the zone label on nodes
    whenUnsatisfiable: DoNotSchedule           # Fail scheduling rather than violate
    labelSelector:
      matchLabels:
        app: web
  - maxSkew: 1
    topologyKey: kubernetes.io/hostname        # Also spread across nodes within a zone
    whenUnsatisfiable: ScheduleAnyway          # Best-effort node spread

8.2 Control Plane AZ Distribution

# Verify control plane nodes are spread across AZs
kubectl get nodes -l node-role.kubernetes.io/control-plane \
  -o custom-columns='NAME:.metadata.name,ZONE:.metadata.labels.topology\.kubernetes\.io/zone'

# Example output (good):
# NAME              ZONE
# cp-node-1         us-east-1a
# cp-node-2         us-east-1b
# cp-node-3         us-east-1c

9 · Cluster Sizing Reference

10 · Cloud vs Bare Metal Architecture Differences

11 · Component Startup Order

When bootstrapping or recovering a cluster, components must start in the correct order:

1. etcd — must be healthy first; API server cannot start without etcd
2. kube-apiserver — reads/writes etcd; all other components depend on it
3. kube-controller-manager — starts watching API server; leader election requires API server
4. kube-scheduler — starts watching API server; leader election requires API server
5. cloud-controller-manager (if present) — starts after API server
6. kubelet on each node — registers with API server; starts DaemonSet Pods (CoreDNS, CNI agent, kube-proxy)
7. kube-proxy — DaemonSet Pod, starts after kubelet is ready
8. CoreDNS — Deployment, starts after kube-proxy is programming Service rules
9. CNI agent (if daemon-based, e.g., Calico node, Cilium agent) — DaemonSet, started by kubelet

# Monitor startup sequence on a control plane node
journalctl -u etcd -f &
journalctl -u kube-apiserver -f &
journalctl -u kube-controller-manager -f &
journalctl -u kube-scheduler -f &

# For kubeadm clusters (static Pods in /etc/kubernetes/manifests/):
# These are started by kubelet as static Pods — no systemd units
ls /etc/kubernetes/manifests/
# etcd.yaml  kube-apiserver.yaml  kube-controller-manager.yaml  kube-scheduler.yaml

# Check static Pod health
kubectl get pods -n kube-system | grep -E "etcd|apiserver|scheduler|controller"

12 · Certificate Topology

Kubernetes uses mutual TLS for all inter-component communication. Understanding the certificate chain is essential for troubleshooting and for manually bootstrapping clusters.

Kubernetes PKI hierarchy (typical kubeadm cluster):

/etc/kubernetes/pki/
├── ca.crt / ca.key
│   └── Cluster CA — signs:
│       ├── apiserver.crt (server cert for kube-apiserver TLS)
│       ├── apiserver-kubelet-client.crt (API server → kubelet client cert)
│       ├── controller-manager.conf (embedded client cert for controller-manager)
│       ├── scheduler.conf (embedded client cert for scheduler)
│       └── kubelet client certs (in /var/lib/kubelet/pki/ on each node)
│
├── etcd/ca.crt / etcd/ca.key
│   └── etcd CA (separate) — signs:
│       ├── etcd/server.crt (etcd server TLS)
│       ├── etcd/peer.crt (etcd peer replication TLS)
│       └── apiserver-etcd-client.crt (API server → etcd client cert)
│
├── front-proxy-ca.crt / front-proxy-ca.key
│   └── Front-proxy CA — signs:
│       └── front-proxy-client.crt (used for API aggregation layer)
│
└── sa.pub / sa.key
    └── Service Account signing key pair
        (sa.key signs SA tokens; sa.pub used by API server to verify them)

Default certificate validity: 1 year (kubeadm)
CA validity: 10 years
Auto-rotation: kubelet client certs rotate automatically via
  RotateKubeletClientCertificate feature (enabled by default since 1.8)

# Check certificate expiry dates on a control plane node
kubeadm certs check-expiration

# Rotate certificates manually (kubeadm clusters)
kubeadm certs renew all

# Check a specific cert
openssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -dates
# notAfter=Jun  1 00:00:00 2026 GMT

# Check kubelet node cert (on each worker node)
openssl x509 -in /var/lib/kubelet/pki/kubelet-client-current.pem -noout -subject -dates

13 · Single-Node Development Clusters

For local development, all components run on a single machine. The control plane and worker role are combined.

# Create a multi-node kind cluster (3 nodes: 1 control-plane, 2 workers)
cat <<EOF | kind create cluster --name dev --config=-
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
- role: worker
- role: worker
networking:
  podSubnet: "10.244.0.0/16"
  serviceSubnet: "10.96.0.0/12"
EOF

# Load a local image into kind (no registry needed)
kind load docker-image myapp:dev --name dev

14 · Cluster Health Checks

# ---- Overall cluster health ----
kubectl get nodes -o wide                    # All nodes, status, version
kubectl get pods -n kube-system              # All control plane + system Pods

# ---- API server health ----
kubectl get --raw /healthz                   # "ok"
kubectl get --raw /healthz/etcd              # "ok"
kubectl get --raw /readyz                    # Checks all health indicators
kubectl get --raw /livez                     # Liveness check

# ---- etcd health (on control plane node) ----
ETCDCTL_API=3 etcdctl \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/healthcheck-client.crt \
  --key=/etc/kubernetes/pki/etcd/healthcheck-client.key \
  endpoint health

ETCDCTL_API=3 etcdctl ... endpoint status --write-out=table
# Shows: DB size, Raft index, leader, version

# ---- Component health (modern) ----
kubectl get --raw /api/v1/nodes | jq .
kubectl describe node <cp-node> | grep -A5 "Conditions:"

# ---- Scheduler / controller-manager ----
kubectl get --raw /healthz    # through API server aggregation
# or directly on the pod:
kubectl exec -n kube-system kube-scheduler-<hash> -- wget -O- http://localhost:10259/healthz
kubectl exec -n kube-system kube-controller-manager-<hash> -- wget -O- http://localhost:10257/healthz

# ---- Event stream for recent issues ----
kubectl get events --sort-by='.lastTimestamp' -A | tail -30

15 · Architecture Production Checklist

Next Files

References

• Kubernetes architecture docs — kubernetes.io/docs/concepts/architecture/
• kubeadm HA docs — kubernetes.io/docs/setup/production-environment/tools/kubeadm/high-availability/
• etcd documentation — etcd.io/docs/
• Kubernetes scalability thresholds — github.com/kubernetes/community/blob/master/sig-scalability/configs-and-limits/thresholds.md
• PKI certificates documentation — kubernetes.io/docs/setup/best-practices/certificates/
• Production Kubernetes — Josh Rosso, Craig Tracey et al., O'Reilly 2021