Worker Node Overview

A worker node (simply called a "node" in Kubernetes) is a machine — physical or virtual — that runs containerized workloads. Nodes provide the compute, memory, storage, and networking substrate on which pods execute. The control plane (see Control Plane Overview) decides where to run pods; nodes do the actual work of running them.

Every worker node runs a fixed set of components. Understanding what each component does, how they communicate, and how they share resources on the node is the foundation for diagnosing failures, tuning performance, and hardening security.

Required Node Components

Node Internal Architecture

Component Ports and Endpoints

The Node Object in the API

A node is represented as a Node object in the Kubernetes API. The node itself (via kubelet) populates most of the status fields; the control plane (via the Node Lifecycle Controller — see kube-controller-manager) manages taints and conditions.

# Inspect a node's full status
kubectl get node worker-1 -o yaml

apiVersion: v1
kind: Node
metadata:
  name: worker-1
  labels:
    kubernetes.io/arch: amd64
    kubernetes.io/hostname: worker-1
    kubernetes.io/os: linux
    node.kubernetes.io/instance-type: m5.2xlarge   # set by CCM
    topology.kubernetes.io/region: us-east-1
    topology.kubernetes.io/zone: us-east-1a
  annotations:
    node.alpha.kubernetes.io/ttl: "0"
    volumes.kubernetes.io/controller-managed-attach-detach: "true"
spec:
  podCIDR: 10.244.3.0/24          # Assigned by controller-manager
  podCIDRs:
    - 10.244.3.0/24
  providerID: aws:///us-east-1a/i-0abc123def456   # Set by CCM
  taints:                          # Added by controller; tolerations on pods
    - effect: NoSchedule
      key: node.kubernetes.io/not-ready
status:
  addresses:
    - address: 10.0.1.42
      type: InternalIP
    - address: worker-1
      type: Hostname
  allocatable:                     # Capacity minus system reserved
    cpu: "7800m"
    memory: "29Gi"
    pods: "110"
    hugepages-2Mi: "0"
    ephemeral-storage: "90Gi"
  capacity:                        # Raw hardware capacity
    cpu: "8"
    memory: "32Gi"
    pods: "110"
    ephemeral-storage: "100Gi"
  conditions:
    - type: MemoryPressure
      status: "False"
    - type: DiskPressure
      status: "False"
    - type: PIDPressure
      status: "False"
    - type: Ready
      status: "True"
      reason: KubeletReady
      message: "kubelet is posting ready status"
      lastHeartbeatTime: "2026-05-16T12:00:00Z"
      lastTransitionTime: "2026-05-15T08:00:00Z"
  nodeInfo:
    kernelVersion: 5.15.0-1053-aws
    osImage: Ubuntu 22.04.3 LTS
    containerRuntimeVersion: containerd://1.7.13
    kubeletVersion: v1.29.3
    kubeProxyVersion: v1.29.3
    architecture: amd64
    operatingSystem: linux

Capacity vs Allocatable

Two distinct resource measurements exist at the node level. The difference is critical for scheduling and bin-packing:

allocatable = capacity
  - kube-reserved
  - system-reserved
  - eviction-threshold

Reservation Example

# kubelet configuration (KubeletConfiguration)
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
kubeReserved:
  cpu: "200m"
  memory: "512Mi"
  ephemeral-storage: "2Gi"
systemReserved:
  cpu: "200m"
  memory: "512Mi"
  ephemeral-storage: "2Gi"
evictionHard:
  memory.available: "200Mi"
  nodefs.available: "5%"
  nodefs.inodesFree: "5%"
  imagefs.available: "10%"

On an 8-core / 32Gi node, this yields:

• Allocatable CPU: 8000m − 200m − 200m = 7600m
• Allocatable memory: 32Gi − 512Mi − 512Mi − 200Mi eviction = ~30.8Gi

Node Conditions

kubelet sets node conditions to communicate health to the control plane. The Node Lifecycle Controller (in kube-controller-manager) reads these to taint and eventually evict pods from unhealthy nodes.

Node Registration

When kubelet starts on a new machine, it registers the node with the API server. This is the bootstrap sequence:

1. TLS bootstrap — kubelet uses a bootstrap token to authenticate and sends a CertificateSigningRequest (CSR) for its node identity certificate (system:node:<nodeName>).
2. Certificate issuance — The CertificateApproval controller or auto-approver signs the CSR. kubelet saves the resulting certificate and rotates it before expiry.
3. Node creation — kubelet POSTs a Node object with capacity, nodeInfo, and an uninitialized taint if cloud integration is enabled.
4. CCM initialization — The cloud-controller-manager's Node Controller (see cloud-controller-manager) reads cloud metadata, sets the providerID, region/zone labels, and removes the uninitialized taint.
5. CNI setup — The network plugin is invoked by kubelet when the first pod is created on the node, setting up the node's pod network.

How a Pod Runs on a Node

Once a pod is scheduled and its spec.nodeName is set, the execution chain on the worker node is:

1. kubelet watch — kubelet's informer detects a pod object with spec.nodeName == thisNode and no status.phase set.
2. Admit pod — kubelet runs local admission: checks resource fit against allocatable, checks node selectors/taints. If it cannot admit, sets status.phase = Failed.
3. Volume setup — kubelet's volume manager coordinates with the CSI node plugin to attach and mount any required PersistentVolumes and ConfigMap/Secret volumes.
4. CNI invocation — kubelet calls the container runtime (containerd), which calls the CNI plugin binary. CNI creates the network namespace, veth pair, assigns the pod IP from the node's podCIDR, and programs routes.
5. Pull image — containerd pulls the container image via the image store (OCI registry). If the image is already present and imagePullPolicy is IfNotPresent, the pull is skipped.
6. Create and start containers — containerd calls runc (or another OCI runtime) to set up Linux namespaces (PID, net, mount, UTS, IPC), cgroups, and execute the container entrypoint.
7. Init containers first — If the pod has init containers, they run serially to completion before app containers start.
8. Probes — kubelet runs liveness, readiness, and startup probes per the pod spec. Failed liveness probes trigger container restarts. Failed readiness probes remove the pod's IP from Endpoints/EndpointSlices.
9. Status reporting — kubelet PATCHes the pod's status subresource and the node's status with heartbeats every nodeStatusUpdateFrequency (default: 10s).

Linux Kernel Primitives Used by Nodes

Kubernetes node functionality is built directly on Linux kernel features. Understanding these makes container behavior predictable:

Node Specialization

Not all nodes are equal. Production clusters commonly segment nodes by function:

Quick Node Diagnostics

# Overview of all nodes
kubectl get nodes -o wide

# Node conditions and events
kubectl describe node worker-1

# Resource usage across nodes
kubectl top nodes

# Pods running on a specific node
kubectl get pods --all-namespaces --field-selector spec.nodeName=worker-1

# kubelet logs (systemd)
journalctl -u kubelet -f --since "10 minutes ago"

# containerd logs
journalctl -u containerd -f

# kube-proxy logs
kubectl logs -n kube-system -l k8s-app=kube-proxy --tail=50

# Node resource allocations
kubectl describe node worker-1 | grep -A10 "Allocated resources"

# Check cgroup driver consistency (must match between kubelet and containerd)
kubectl get node worker-1 -o jsonpath='{.status.nodeInfo.containerRuntimeVersion}'
crictl info | grep cgroupDriver

Key Node Metrics

Alerting Rules

groups:
  - name: worker-node
    rules:
      - alert: NodeNotReady
        expr: kube_node_status_condition{condition="Ready",status="true"} == 0
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "Node {{ $labels.node }} not Ready for 2m"

      - alert: NodeMemoryPressure
        expr: kube_node_status_condition{condition="MemoryPressure",status="true"} == 1
        for: 1m
        labels:
          severity: warning
        annotations:
          summary: "Node {{ $labels.node }} under memory pressure"

      - alert: NodeDiskPressure
        expr: kube_node_status_condition{condition="DiskPressure",status="true"} == 1
        for: 1m
        labels:
          severity: warning
        annotations:
          summary: "Node {{ $labels.node }} under disk pressure"

      - alert: NodeHighCPU
        expr: |
          (1 - avg by(instance)(rate(node_cpu_seconds_total{mode="idle"}[5m]))) > 0.9
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "Node {{ $labels.instance }} CPU > 90% for 10m"

      - alert: ContainerOOMKilled
        expr: increase(container_oom_events_total[5m]) > 0
        labels:
          severity: warning
        annotations:
          summary: "Container {{ $labels.container }} in pod {{ $labels.pod }} OOM killed"

Production Best Practices

1. Always configure kube-reserved and system-reserved. Without reservations, the scheduler can completely exhaust node resources, causing kubelet and containerd to be OOM-killed — which looks like a node failure from the control plane's perspective.
2. Disable the kubelet read-only port (--read-only-port=0). It exposes pod and node information without authentication and is a CIS benchmark finding.
3. Use cgroups v2. It provides better memory accounting, I/O limits, and pressure notifications. Ensure both kubelet (cgroupDriver: systemd) and containerd use the same cgroup driver — mismatch causes pod failures.
4. Enforce node registration constraints. Use --node-labels in kubelet config to set immutable labels at registration time. Use NodeRestriction admission (see Admission Controllers §NodeRestriction) to prevent kubelets from labeling themselves with arbitrary values.
5. Set maxPods based on node size. The default of 110 is appropriate for small nodes. Large nodes can host more, but increasing beyond 250 requires careful CNI and kubelet tuning.
6. Monitor node conditions with kube-state-metrics and alert on any pressure condition. By the time a node reaches disk pressure, it has already started evicting pods — catch the trend early.
7. Use node taints to dedicate nodes to specific workloads. System DaemonSets should tolerate all taints; application workloads should only be scheduled on appropriately labeled and tainted nodes.
8. Enable kubelet certificate rotation (rotateCertificates: true). Node certificates expire; automatic rotation prevents node-level authentication failures after ~1 year.