kube-scheduler

What kube-scheduler Does

kube-scheduler watches the API server for Pods whose spec.nodeName is empty (unbound Pods) and assigns each to the most suitable node by writing a Binding object — a POST to /api/v1/namespaces/{ns}/pods/{name}/binding. Once bound, kubelet on the target node picks up the Pod via its own watch stream and starts it.

Scheduling Framework and Extension Points

The Scheduling Framework (introduced in 1.15, stable in 1.19) replaces the old predicates/priorities model. Every scheduling decision is decomposed into 12 ordered extension points, each implemented by one or more plugins.

Built-in Scheduler Plugins

Filter Plugins

Score Plugins

Scheduling Queue Internals

The scheduler maintains three internal queues that govern when and in what order Pods are evaluated:

Two-Phase Deep Dive: Filter and Score

Filter Phase

Each Filter plugin's Filter(pod, nodeInfo) method runs for every (pod, feasible_node) pair. Runs are parallelized across nodes (default 16 goroutines). A node is included in the feasible set only if all plugins return success.

# See which plugins rejected a pod and why
kubectl describe pod my-pod | grep -A 20 "Events:"
# Events will show: "0/5 nodes are available: 2 node(s) had taint, 3 Insufficient memory"

# Count of filter failures per plugin (Prometheus)
scheduler_framework_extension_point_duration_seconds{extension_point="Filter",status="Error"}

# Enable verbose scheduler logs (very noisy — use only in non-prod)
# Add to kube-scheduler pod: --v=10

The percentageOfNodesToScore optimization: on clusters with >100 nodes, once enough feasible nodes are found (default: min(50, number_of_nodes) to a max of 100%), the scheduler stops evaluating remaining nodes. This is a heuristic — good enough in practice, can be disabled for precise scheduling.

apiVersion: kubescheduler.config.k8s.io/v1
kind: KubeSchedulerConfiguration
percentageOfNodesToScore: 0   # 0 = use adaptive default based on cluster size
profiles:
- schedulerName: default-scheduler

Score Phase

Each Score plugin returns 0–100 per feasible node. The final score for each node is the weighted sum: Σ(plugin_weight × normalized_score). NormalizeScore brings raw scores within a plugin into [0,100] range. The node with the highest final score wins. Ties broken by random selection (to prevent hot nodes).

# Example: default score plugin weights
# (from default-scheduler profile in KubeSchedulerConfiguration)
plugins:
  score:
    enabled:
    - name: NodeResourcesBalancedAllocation
      weight: 1
    - name: ImageLocality
      weight: 1
    - name: InterPodAffinity
      weight: 1
    - name: NodeResourcesFit       # LeastAllocated strategy
      weight: 1
    - name: NodeAffinity
      weight: 1
    - name: PodTopologySpread
      weight: 2                    # higher weight = stronger influence
    - name: TaintToleration
      weight: 1

Node Affinity, Pod Affinity, Taints and Tolerations

Node Affinity

spec:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:   # HARD — Filter
        nodeSelectorTerms:
        - matchExpressions:
          - key: kubernetes.io/arch
            operator: In
            values: [amd64]
          - key: node-role/gpu
            operator: Exists
      preferredDuringSchedulingIgnoredDuringExecution:  # SOFT — Score
      - weight: 80
        preference:
          matchExpressions:
          - key: topology.kubernetes.io/zone
            operator: In
            values: [us-east-1a]

Pod Affinity and Anti-Affinity

spec:
  affinity:
    podAntiAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:  # HARD anti-affinity
      - labelSelector:
          matchLabels:
            app: my-app
        topologyKey: kubernetes.io/hostname  # no two pods on same node
      preferredDuringSchedulingIgnoredDuringExecution:  # SOFT spread across zones
      - weight: 100
        podAffinityTerm:
          labelSelector:
            matchLabels:
              app: my-app
          topologyKey: topology.kubernetes.io/zone

Taints and Tolerations

# Taint a node (effect: NoSchedule | PreferNoSchedule | NoExecute)
kubectl taint nodes worker-1 dedicated=gpu:NoSchedule
kubectl taint nodes worker-1 node.kubernetes.io/not-ready:NoExecute

# Remove a taint
kubectl taint nodes worker-1 dedicated=gpu:NoSchedule-

# Pod tolerates the taint
spec:
  tolerations:
  - key: dedicated
    operator: Equal
    value: gpu
    effect: NoSchedule
  - key: node.kubernetes.io/not-ready   # tolerate node-not-ready for 300s
    operator: Exists
    effect: NoExecute
    tolerationSeconds: 300

Topology Spread Constraints

Introduced in 1.14 (GA 1.19). Replaces complex pod anti-affinity patterns for even distribution. More efficient than InterPodAffinity as it uses pre-aggregated per-topology-domain counts.

spec:
  topologySpreadConstraints:
  - maxSkew: 1                         # max allowed difference between any two domains
    topologyKey: topology.kubernetes.io/zone
    whenUnsatisfiable: DoNotSchedule   # hard constraint (Filter)
    labelSelector:
      matchLabels:
        app: my-app
    matchLabelKeys:                    # k8s 1.27+ — auto-exclude old revisions
    - pod-template-hash
  - maxSkew: 2
    topologyKey: kubernetes.io/hostname
    whenUnsatisfiable: ScheduleAnyway  # soft constraint (Score)
    labelSelector:
      matchLabels:
        app: my-app

Preemption and PriorityClass

When no node can fit a pending pod after filtering, the DefaultPreemption plugin (running in the PostFilter extension point) tries to evict lower-priority pods to make room.

apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: high-priority
value: 1000000         # 0 = lowest; 1,000,000,000 = max (system-reserved)
globalDefault: false
preemptionPolicy: PreemptLowerPriority  # PreemptLowerPriority | Never
description: "Critical production workloads"
---
# Use in pod spec
spec:
  priorityClassName: high-priority

Built-in system PriorityClasses:

Preemption Algorithm

1. Collect candidate nodes where evicting some pods would make room for the pending pod.
2. For each candidate node: identify the minimum set of pods to evict (lowest priority, fewest disruptions, no PDB violations).
3. Pick the node that minimizes disruption (fewest pods evicted, respect PDB).
4. Evict chosen pods (graceful termination via deletionTimestamp). Pending pod is NOT immediately scheduled — it waits for evicted pods to terminate, then re-enters the scheduling cycle.

Scheduler Profiles and Multiple Schedulers

A single kube-scheduler binary can run multiple profiles, each with its own schedulerName, plugin configuration, and plugin weights. Pods opt into a profile via spec.schedulerName.

apiVersion: kubescheduler.config.k8s.io/v1
kind: KubeSchedulerConfiguration
leaderElection:
  leaderElect: true
clientConnection:
  kubeconfig: /etc/kubernetes/scheduler.conf
profiles:
# Profile 1: default balanced scheduler
- schedulerName: default-scheduler
  plugins:
    score:
      enabled:
      - name: NodeResourcesFit
        weight: 1
      - name: PodTopologySpread
        weight: 2
  pluginConfig:
  - name: NodeResourcesFit
    args:
      scoringStrategy:
        type: LeastAllocated
        resources:
        - name: cpu
          weight: 1
        - name: memory
          weight: 1

# Profile 2: bin-packing for batch jobs
- schedulerName: bin-pack-scheduler
  plugins:
    score:
      enabled:
      - name: NodeResourcesFit
        weight: 1
  pluginConfig:
  - name: NodeResourcesFit
    args:
      scoringStrategy:
        type: MostAllocated   # pack pods tightly
        resources:
        - name: cpu
          weight: 1
        - name: memory
          weight: 1

# Pod using the bin-pack profile
spec:
  schedulerName: bin-pack-scheduler
  containers:
  - name: batch-job
    image: my-batch:latest

Gang Scheduling and the Permit Extension Point

Gang scheduling (also called co-scheduling) ensures all pods in a group are scheduled together — or none are. This is critical for ML training jobs (all workers must start simultaneously) and MPI workloads.

The Permit extension point enables this: a plugin can return Wait instead of Allow. The pod is tentatively assigned to a node but held in a waiting state until the plugin explicitly allows it (or a timeout expires).

# Coscheduling via the scheduling-plugins project (sig-scheduling)
# https://github.com/kubernetes-sigs/scheduler-plugins
apiVersion: scheduling.sigs.k8s.io/v1alpha1
kind: PodGroup
metadata:
  name: my-ml-job
spec:
  minMember: 4           # all 4 workers must be schedulable before any binds
  minResources:
    cpu: "4"
    memory: 8Gi
---
spec:
  schedulerName: scheduler-plugins-scheduler
  labels:
    scheduling.sigs.k8s.io/pod-group: my-ml-job

The Permit plugin tracks how many pods in the group have been tentatively placed. Once minMember is reached, all waiting pods are released to Bind simultaneously. If the timeout expires before all members are placed, waiting pods are rejected (→ Unreserve) and returned to the queue.

Leader Election

In HA setups, multiple kube-scheduler replicas run but only one is active. Leadership is managed via a Lease object in the kube-system namespace. The details are identical to the kube-controller-manager pattern — see 00-control-plane-overview.html § Leader Election.

# Check current scheduler leader
kubectl -n kube-system get lease kube-scheduler -o yaml
# .spec.holderIdentity shows the active scheduler pod

# Scheduler logs showing leader election
kubectl -n kube-system logs kube-scheduler-master-1 | grep -i "leader"
# I0115 10:30:00 leaderelection.go:248] attempting to acquire leader lease...
# I0115 10:30:01 leaderelection.go:258] successfully acquired lease kube-system/kube-scheduler

Configuration Reference

apiVersion: kubescheduler.config.k8s.io/v1
kind: KubeSchedulerConfiguration
# Leader election
leaderElection:
  leaderElect: true
  leaseDuration: 15s
  renewDeadline: 10s
  retryPeriod: 2s
  resourceNamespace: kube-system
  resourceName: kube-scheduler

# API server connection
clientConnection:
  kubeconfig: /etc/kubernetes/scheduler.conf
  qps: 50
  burst: 100

# Performance tuning
percentageOfNodesToScore: 0     # 0 = adaptive (recommended)
parallelism: 16                 # goroutines for filter/score

# Health / metrics
healthzBindAddress: 0.0.0.0:10259
metricsBindAddress: 0.0.0.0:10259

Metrics and Alerting

# Scrape scheduler metrics
curl -sk https://localhost:10259/metrics \
  --cert /etc/kubernetes/pki/scheduler-client.crt \
  --key /etc/kubernetes/pki/scheduler-client.key | grep scheduler_

# Key metrics
scheduler_scheduling_attempt_duration_seconds{result="scheduled"}   # e2e scheduling latency
scheduler_scheduling_attempt_duration_seconds{result="unschedulable"}
scheduler_pending_pods{queue="active"}       # pods waiting to be scheduled
scheduler_pending_pods{queue="backoff"}      # pods in backoff
scheduler_pending_pods{queue="unschedulable"}# pods with no node found
scheduler_preemption_attempts_total          # total preemption attempts
scheduler_preemption_victims                 # total pods evicted by preemption
scheduler_framework_extension_point_duration_seconds{extension_point="Filter"}
scheduler_framework_extension_point_duration_seconds{extension_point="Score"}
scheduler_volume_scheduling_stage_error_reason  # volume binding errors

groups:
- name: kube-scheduler
  rules:
  - alert: KubeSchedulerDown
    expr: absent(up{job="kube-scheduler"} == 1)
    for: 5m
    labels:
      severity: critical
    annotations:
      summary: "kube-scheduler is down"
      description: "No kube-scheduler instance is up. Pods cannot be scheduled."

  - alert: KubeSchedulerHighPendingPods
    expr: scheduler_pending_pods{queue="unschedulable"} > 50
    for: 15m
    labels:
      severity: warning
    annotations:
      summary: "High number of unschedulable pods"
      description: "{{ $value }} pods have been unschedulable for >15m."

  - alert: KubeSchedulerHighLatency
    expr: |
      histogram_quantile(0.99,
        rate(scheduler_scheduling_attempt_duration_seconds_bucket{result="scheduled"}[5m])
      ) > 5
    for: 10m
    labels:
      severity: warning
    annotations:
      summary: "Scheduler p99 latency > 5s"
      description: "P99 scheduling latency is {{ $value }}s."

  - alert: KubeSchedulerPreemptionHigh
    expr: rate(scheduler_preemption_victims[5m]) > 10
    for: 5m
    labels:
      severity: warning
    annotations:
      summary: "High preemption rate"
      description: "Scheduler is evicting >10 pods/s. Check PriorityClass assignments."

Troubleshooting Scheduling Issues

# Full scheduling failure reason
kubectl describe pod my-pod | grep -A 30 "Events:"
# Common messages:
# "0/5 nodes are available: 2 Insufficient cpu, 3 node(s) had taint..."
# "0/5 nodes are available: 5 node(s) didn't match pod's node affinity/selector"
# "0/5 nodes are available: 5 pod has unbound immediate PersistentVolumeClaims"

# Check node resources and allocatable
kubectl describe nodes | grep -A 8 "Allocated resources"

# Check if nodes are cordoned
kubectl get nodes | grep SchedulingDisabled

# Check taints on nodes
kubectl get nodes -o custom-columns=NAME:.metadata.name,TAINTS:.spec.taints

# Check if the pod's resource requests are too high
kubectl get pod my-pod -o jsonpath='{.spec.containers[*].resources}'

# Check PVC status
kubectl get pvc
kubectl describe pvc my-pvc
# Look for: "waiting for first consumer to be created before binding"
# This is normal for WaitForFirstConsumer StorageClass — PVC binds AFTER scheduler picks node

# Check StorageClass binding mode
kubectl get storageclass standard -o jsonpath='{.volumeBindingMode}'
# Immediate = PVC bound before scheduling (may conflict with zone affinity)
# WaitForFirstConsumer = PVC bound after scheduling (node zone determines PVC zone)

# Volume scheduling logs
kubectl -n kube-system logs kube-scheduler-master-1 | grep -i "volume"

# Check existing topology spread
kubectl get pods -l app=my-app -o wide | awk '{print $7}' | sort | uniq -c

# Check if topologySpreadConstraints are set
kubectl get deploy my-deploy -o jsonpath='{.spec.template.spec.topologySpreadConstraints}'

# Check zone labels on nodes
kubectl get nodes --show-labels | grep topology.kubernetes.io/zone

# If using old pod anti-affinity instead:
# Migrate to topologySpreadConstraints for better performance and flexibility

# Check scheduler logs for spread scoring
kubectl -n kube-system logs kube-scheduler-master-1 -v=4 | grep TopologySpread

# Check scheduling attempt duration
kubectl get --raw /metrics | grep scheduler_scheduling_attempt_duration

# Common causes of slow scheduling:
# 1. InterPodAffinity with large pod counts — migrate to topologySpreadConstraints
# 2. percentageOfNodesToScore too high — reduce for large clusters
# 3. Too many Score plugins with high weight — profile and trim
# 4. Custom webhook in Permit extension point — check latency

# Increase parallelism for large clusters
# In KubeSchedulerConfiguration:
parallelism: 32  # default 16, increase for 1000+ node clusters

# Check if scheduler leader is healthy
kubectl -n kube-system get lease kube-scheduler -o yaml | grep holderIdentity

# Check if priority classes are set correctly
kubectl get priorityclass
kubectl get pod high-priority-pod -o jsonpath='{.spec.priority}'

# Check if PDB is blocking preemption
kubectl get pdb -A
kubectl describe pdb my-pdb | grep -E "Allowed Disruptions|Status"

# Check scheduler logs for preemption decisions
kubectl -n kube-system logs kube-scheduler-master-1 | grep -i preempt

# Check if preemptionPolicy is set to Never
kubectl get priorityclass low-priority -o jsonpath='{.preemptionPolicy}'

Scheduling Sequence: Pod to Bound

Production Best Practices